akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Documentation/examples of building LibTorch with make, not cmake

username: Leon_Fedden
cooked: <p>Hey all, sorry if this has already been covered, I tried googling/searching the forums but couldn’t find any relevant results.</p>
<p>I’m looking at integrating a model into a JUCE project, which uses a tool called the Projucer as the build system. The most analogous tool in terms of syntax etc would plain old Makefiles, and I was wondering if anyone knew of any examples or documentation of Make being used to build LibTorch, which in turn I could use to translate to a Projucer project.</p>
<p>The documentation denotes this on CMake:</p>
<blockquote>
<p>The LibTorch distribution encompasses a collection of shared libraries, header files and CMake build configuration files. While CMake is not a requirement for depending on LibTorch, it is the recommended approach and will be well supported into the future.</p>
</blockquote>
<p>Thanks for reading!</p>

username: tom
cooked: <p>For building LibTorch with make, no, and I’d think you’d see limited interest in supporting building libtorch without cmake.</p>
<p>Now, using a prebuilt libtorch in other programs is pretty straightforward even without CMake, I have written a trivial C++ program running a model on an image and displaying (the sample program shown on my slides  <a href="https://lernapparat.de/pytorch-jit-android/" rel="nofollow noopener">https://lernapparat.de/pytorch-jit-android/</a>) and compiled that with</p>
<pre><code class="lang-auto">g++ run.cpp -I /usr/local/lib/python3.6/dist-packages/torch/lib/include/  -L /usr/local/lib/python3.6/dist-packages/torch/lib/ -ltorch -lcaffe2 -lc10 -lpthread -lX11
</code></pre>
<p>(where the pthread possibly and X11certainly are CImg dependencies rather than those of PyTorch).</p>
<p>Best regards</p>
<p>Thomas</p>

username: ShigekiKarita
cooked: <p>I’m using libtorch inside anaconda. Maybe my makefile gives you some hints <a href="https://github.com/ShigekiKarita/libtorch-cxx-examples/blob/master/mnist/Makefile" rel="nofollow noopener">https://github.com/ShigekiKarita/libtorch-cxx-examples/blob/master/mnist/Makefile</a></p>

username: Edwardmark
cooked: <p>How to use the libtorch <a href="https://download.pytorch.org/libtorch/nightly/cu92/libtorch-win-shared-with-deps-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/nightly/cu92/libtorch-win-shared-with-deps-latest.zip</a> using make?<br>
Thanks.</p>

username: Edwardmark
cooked: <p>How you solve the problem, please help me out, thanks.<br>
Bests,<br>
Edward.</p>

username: Edwardmark
cooked: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="32741">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>CMake</p>
</blockquote>
</aside>
<p>How to build libtorch with cuda? How to write the makefile? Thanks.</p>

username: Sagar_Raikar
cooked: <p>I was able to integrate the model with JUCE project. Using traced model for C++.</p>
Multiplying Tensors

username: Jack_Rolph
cooked: <p>I have to perform the operation:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bdd05dfd4ce9230d1281114691fce70ba56840b.gif" alt="CodeCogsEqn (1)" data-base62-sha1="3YuqJsAMKQZVFqmHMLcx2bkX0cr" width="107" height="24"></p>
<p>I have two tensors that I want to multiply in this way, <code>A</code> and <code>B</code>:</p>
<pre><code class="lang-auto">A = torch.randn(B,C,j,i)
B = torch.randn(B,C,j,k)
C = AxB
C.shape = torch.Size(B,C,j,i,k)
</code></pre>
<p>Where I have assumed the shape of the output tensor based on the indices of the equation.</p>
<p>What is the function that I should use to produce the desired result?</p>

username: ptrblck
cooked: <p>Would this code work?</p>
<pre><code class="lang-python">B, C, i, j, k = 2, 3, 4, 5, 6

A = torch.randn(B,C,j,i)
B = torch.randn(B,C,j,k)

C = torch.matmul(A.unsqueeze(4), B.unsqueeze(3))
print(C.shape)
</code></pre>
How to use pixelShuffle?

username: mathematics
cooked: <p>As official doc gives example to upscale the image</p>
<pre><code class="lang-auto">pixel_shuffle = nn.PixelShuffle(3)
input = torch.randn(1, 9, 4, 4)
output = pixel_shuffle(input)
# torch.Size([1, 1, 12, 12])
</code></pre>
<p>As it mutilplies by input, I wanted to use 2 or any other input , giving eror</p>
<blockquote>
<p>RuntimeError: invalid argument 2: size ‘[1 x 2 x 2 x 2 x 4 x 4]’ is invalid for input with 144 elements at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/THStorage.c:41</p>
</blockquote>
<p>I was using on torch size of<br>
<code>torch.randn(1, 512, 6,6)</code><br>
I wanted to upsale it it<code> (1, 3, 128, 128)</code> by only using pixelShuffle,<br>
But when i use upscaling except 2 all other fails.</p>
<p>At last of upscaling it generates<br>
<code>torch.Size([50, 2, 96, 96])</code></p>
<p>Help how can i use PixelShuffle to generate image of shape nc=3 and desired height and width.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You won’t be able to convert 512 channels to 3 given the formula from <a href="https://pytorch.org/docs/stable/nn.html?highlight=pixelshuffle#torch.nn.PixelShuffle">the docs</a>.</p>
<pre><code class="lang-python">input [N, L=C*factor**2, H_in, W_in]
output [N, C, H_out=H_in*factor, W_out=W_in*factor]
</code></pre>
<p>For your setup you would need to calculate:</p>
<pre><code class="lang-auto">3 * factor**2 = 512
factor**2 = 512/3
factor = sqrt(512/3)
factor ~= 13.07 # not an int and thus not usable
</code></pre>
Train, test, validation streamlined

username: ianweb100
cooked: <p>I have successfully managed to break down the dataset into three parts ( train, test and oval). What is bugging me is how long was the way I found to do it using data loader. No need to say, I am a beginner user of PyTorch. What i am wondering is if it does not exist a more straightforward method like Sklearn, train test split.</p>
<p>Please see my code below.</p>
<pre><code>#load the data from the path the data is only separated into three classes
train_dataset = torchvision.datasets.ImageFolder(
root= data_path,
transform = data_transforms,
target_transform=None)

# Load the data as one large batch
loader = DataLoader(
train_dataset,
batch_size=1000,
num_workers=0,
shuffle=True)

# Separate into train val
trainloader, valloader = torch.utils.data.random_split(loader.dataset, (144 , 48))

#Load train and val
 train_loader = DataLoader( #has 144
trainloader,
batch_size=16,
num_workers=0,
shuffle=True)

val_loader = DataLoader( # has 48
valloader,
batch_size=1,
num_workers=0,
shuffle=False)

#Separate val into test and val
testloader, valloader = torch.utils.data.random_split(val_loader.dataset, (40, 8))

#Load test and val
test_loader = DataLoader( # has 40
testloader,
batch_size=1,
num_workers=0,
shuffle=False)

val_loader = DataLoader( # has 8
valloader,
batch_size=1,
num_workers=0,
shuffle=False)</code></pre>

username: ptrblck
cooked: <p>You could use <code>sklearn.model_selection.train_test_split</code> directly by using the dataset indices (just create the indices using <code>np.arange</code>).<br>
Once you have the indices for each dataset split, you could either use <code>Subset</code> and wrap the original dataset with the corresponding indices, or use a <code>SubsetRandomSampler</code> and pass these samplers to each <code>DataLoader</code>.</p>
How do you properly load weights while training?

username: alx
cooked: <p>Hi, I’d like to load the best weights of my model instead of stopping the whole training when overfitting.</p>
<p>Right now, I’m doing something similar to:</p>
<pre><code class="lang-auto">if overfit is True:
     model.load_state_dict(torch.load(path_of_best_weights))
</code></pre>
<p>However, when I look at my accuracy I don’t see any difference. It seems like its continuing the training as if nothing happened.</p>
<p>The only thing that worked was to reload the whole model and then load on my best weights:</p>
<pre><code class="lang-auto">model = models.resnet18(pretrained=True)
model.load_state_dict(torch.load(path_of_best_weights))
</code></pre>
<p>Is there a better way of doing this?</p>

username: ptrblck
cooked: <p>That’s a weird issue, as both approaches should be equal, if you are loading the <code>state_dict</code> from a file.<br>
Just to make sure there is no error in the first approach, since you mentioned “similar to”:<br>
you should either load the <code>state_dict</code> from a file or create a <code>copy.deepcopy</code> of the <code>state_dict</code>.<br>
Otherwise the <code>state_dict</code> will hold the references to the parameters and you will just load the same parameters.</p>

username: alx
cooked: <p>Interesting, I started digging into backing up the optimizer as well because I wasn’t. But even then, its still not working.</p>
<p>If I asked you to revert back to how a model was at epoch x how would you do it?</p>
<p>I was even thinking about emptying the gpu and re-adding the model with the state-dict because it works perfectly when I restart my notebook but not when load the state dict while I’m training.</p>
<p>Even if I do something like:</p>
<pre><code class="lang-auto">if epoch == 50:
   new_model = models.resnet18(pretrained=True)
   new_model.load_state_dict(torch.load('weights_from_epoch_40'))
   new_model.to(DEVICE)
</code></pre>
<p>Then epoch 51 will have the same accuracy as if I never loaded the weights from epoch 40</p>

username: ptrblck
cooked: <p>Could you post your notebook, so that we could have a look?</p>
BatchNorm in RNN Step

username: fulltopic
cooked: <p>Hi,</p>
<p>I had pre-trained a RNN with BatchNormalization with collected data. In training, the input was in sizes of {batchSize, seqLen, others} so that the num_features of BatchNorm layer is seqLen (a fixed number). Then I want to transfer the RNN into A3C and train it online.  In this case, the input is {1, 1, others} in real-time action generation, and the num_features of BatchNorm should be 1. But the BatchNorm(num_features = seqLen) could still be used in offline network updating.</p>
<p>How to deal with the case? Could I just ignore the BatchNorm layer in real-time forward? Do I have to train the net in step-by-step way?</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <p>If you are dealing with a different number of features for the batchnorm layer, you could add a condition into your model, which would then pick the right batchnorm layer for the current input.</p>
<p>I’m not familiar with your use case, but maybe it would also make sense to slice and copy the batchnorm layer parameters and buffers from the <code>seqLen</code> use case to the single feature use case?</p>

username: fulltopic
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>. “slice and copy batchnorm layer parameters and buffers” works.</p>
<p>Is there any official/widely-used way to do “slice and copy batchnorm layer” instead of manipulating “weights/bias” tensor values directly?</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <p>You could use something like:</p>
<pre><code class="lang-python">small_bn = nn.BatchNorm2d(1)
with torch.no_grad():
    small_bn.weight = nn.Parameter(large_bn.weight[0:1]) # or take the mean, slice at another pos?
    # same for bias, running_mean and running_var
</code></pre>
<p>I don’t know, if that would make sense at all for your use case, but you might consider it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Empty tensor from cross entropy loss function

username: zeus
cooked: <p>My input to the cross entropy loss function is <strong>torch.Size([69856, 21])</strong> and target is <strong>torch.Size([69856])</strong> and output is <strong>torch.Size([])</strong>.<br>
When I mention <strong>nn.CrossEntropyLoss(reduce=None)</strong> it is giving empty tensor<br>
when I mention <strong>nn.CrossEntropyLoss(reduce=False)</strong> it gives correct output shape but values are <strong>Nan</strong>.<br>
Please help me solve this issue.</p>

username: KFrank
cooked: <p>Hi Zeus!</p>
<aside class="quote no-group" data-username="zeus" data-post="1" data-topic="82401" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/a4c791/48.png" class="avatar"> zeus:</div>
<blockquote>
<p>My input to the cross entropy loss function is <strong>torch.Size([69856, 21])</strong> and target is <strong>torch.Size([69856])</strong> and output is <strong>torch.Size([])</strong>.</p>
</blockquote>
</aside>
<p>Could you post a simplified, but complete and runnable script that<br>
illustrates this issue, together with the output of the script?</p>
<p>Best.</p>
<p>K. Frank</p>

username: zeus
cooked: <p>My predicted scores are full of <strong>nan</strong>. That’s why it is returning empty tensors.</p>
How to get alpha values of focal loss for multiclass classification?

username: Umair_Javaid
cooked: <p>I am sure the gamma value of focal loss is fixed(=2) but I am confused about how to get alpha values of focal loss for multiclass classification? Is the alpha value fixed or not?<br>
I do not understand the following line:</p>
<blockquote>
<p>It may be set by inverse class frequency</p>
</blockquote>
user_suspended: yes
Change first convolution layer of Densenet

username: sampa
cooked: <p>I want to convert first convolution layer from 3 channel to 4 channel.</p>
<pre><code class="lang-auto">DenseNet(
  (features): Sequential(
    (conv0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)        
    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)     
    (relu0): ReLU(inplace=True)
</code></pre>
<p>It should be like</p>
<pre><code class="lang-auto">DenseNet(
  (features): Sequential(
    (conv0): Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)        
    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)     
    (relu0): ReLU(inplace=True)
</code></pre>
<p>I did</p>
<pre><code class="lang-auto">model= models.densenet121(pretrained=True)
                model.conv0 = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
                set_parameter_requires_grad(model, feature_extract)
                num_ftrs = model.classifier.in_features
                model.classifier = nn.Linear(num_ftrs, num_classes)
</code></pre>
<p>But it is not working</p>

username: ptrblck
cooked: <p>What kind of error are you seeing?</p>

username: sampa
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.  It is showing RuntimeError: Given groups=1, weight of size 64 3 7 7, expected input[16, 4, 224, 224] to have 3 channels, but got 4 channels instead<br>
I have changed the convolution layer as</p>
<pre><code class="lang-auto">model.conv0 = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
</code></pre>
<p>But When I print the model, it is showing</p>
<pre><code class="lang-auto">(conv0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) 
</code></pre>

username: futscdav
cooked: <p>The <code>conv0</code> is under the Sequential <code>features</code>, not an attribute of the model itself.</p>
<p>I think you might be able to swap it with indexing it like you would a list<br>
<code>model.features[0] = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)</code></p>

username: sampa
cooked: <p><a class="mention" href="/u/futscdav">@futscdav</a> Thanks. Its working.</p>
Use of KLDivLoss on samples of unknown distribution?

username: steffenN
cooked: <p>Hi,</p>
<p>i have an output of a NN that is supposed to follow a certain (normal in this case) distribution. From what i understand KLDivLoss measures the distance between two distributions, so i guess this is what i need to use. However, my outputs are not propabilities! And i think that is what KLDivLoss needs. You could consider the output to be samples of a distribution and i want this distribution to be guassian (or maybe something else later on).<br>
So my question is: Is there a function in PyTorch i can use to get the distribution(/propabilities) from the samples so that i can use KLDivLoss and still backpropagate through it?<br>
Using a histogramm came to mind first, but from what i found, torch.histc is not able backpropagate…</p>
<p>Would anyone know a good approach to this?</p>
Can I load a model with the same layer parameters but design my own forward function?

username: xueluli
cooked: <p>I designed a model that has two different kinds of inputs: x1 and x2. x2 will only pass through part of the model and its output is used for some constraints. I saved the model after training and try to reuse it in the test stage. However, I only need input x1 during the test stage. Is there any way that I can load the layer parameters of the saved model but with different contents in the “forward” function? It seems that torch.load(model.state_dict) cannot give me satisfying results since it requires the same forward function.</p>
<p>Thanks very much!</p>

username: ptrblck
cooked: <p>The forward method is independent from the <code>state_dict</code> and you should be able to manipulate this method as you wish.<br>
<code>state_dict = torch.load(path_to_state_dict)</code> will only load all parameters and buffers in an <code>OrderedDict</code> without any knowledge about the <code>forward</code> method.<br>
Do you see any error using this approach?</p>

username: xueluli
cooked: <p>Yes, I use the commands as below:</p>
<p>model_path = os.path.join(args.checkpoint,‘model_epoch_{}.pth’.format(str(ep)))<br>
checkpoint = torch.load(model_path)<br>
model = Unet_BiCLSTM()<br>
model.load_state_dict(checkpoint[“model”].state_dict())</p>
<p>where Unet_BiCLSTM() is my network which only have different forward function as the pretrained model. Then I obtain mistakes as below:</p>
<p>RuntimeError: Error(s) in loading state_dict for Unet_BiCLSTM:<br>
Missing key(s) in state_dict: “feature_ext.conv0.weight”, “feature_ext.norm0.weight”, “feature_ext.norm0.bias”, “feature_ext.norm0.running_mean”, “feature_ext.norm0.running_var”, …</p>
<p>Is there something wrong about it?</p>
<p>Thanks very much!</p>

username: dunefox
cooked: <p>You could use strict=false in the state_dict, that might help - if I understand correctly.</p>
<p>Edit: But that would mean that the relevant weights aren’t loaded at all since the model expects these layers to be present… are the layers named the same? Can you load them back into the original model? Or do you mean by ‘different contents in the forward function’ that you’re changing the model architecture?</p>

username: xueluli
cooked: <p>Yeah, I can load them back to the  model. I copied the contents in the original model, and change the contents in the forward function by reducing one input, and removing the corresponding line in the forward function which uses the same layer as other inputs. During the training, this input is used to pass through a part of the network and its output is constrained to provide some regularization to the network. In the test stage, this process is not used anymore, so I want to remove it by changing the forward function.</p>
Sparse linear solver that works with autograd

username: Patrickens
cooked: <p>Dear PyTorch people, what a dream of library this is!</p>
<p>I have a set of linear systems of the following form:</p>
<p><strong><sup>1</sup>A(v) • <sup>1</sup>x = <sup>1</sup>b(v)</strong><br>
<strong><sup>2</sup>A(v) • <sup>2</sup>x = <sup>2</sup>b(v,<sup>1</sup>x)</strong><br>
<strong><sup>3</sup>A(v) • <sup>3</sup>x = <sup>3</sup>b(v,<sup>1</sup>x,<sup>2</sup>x)</strong><br>
<strong>…</strong></p>
<p>The <strong><sup>n</sup>A</strong> matrices is constructed from several component <strong><sup>n</sup>A<sub>i</sub></strong> matrices multiplied by an element of vector <strong>v</strong>: <strong><sup>1</sup>A = <sup>1</sup>A<sub>a</sub> * v<sub>a</sub> + <sup>1</sup>A<sub>b</sub> * v<sub>b</sub> + <sup>1</sup>A<sub>c</sub> * v<sub>c</sub>…</strong> The <strong><sup>n</sup>A<sub>i</sub></strong> matrices are large, sparse and there are many of them. I would thus like to store the matrices as sparse tensors. I was wondering whether there were any plans to implement a sparse linear solver (direct, since I guess iterative would be impossible) that was amenable to autograd? I need autograd to compute the Jacobian <strong>dx/dv</strong> in the end.</p>
<p>For now I have it working with dense systems, but I was wondering whether there would be a version of lu_solve that works with backpropagation, since I would like to store the factorization of <strong>A</strong> for use down the road. Discussed in issue: <a href="https://github.com/pytorch/pytorch/issues/22620" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/22620</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>As mentioned in the issue, we would be happy to add this.<br>
Do you know what the gradient of the output wrt LU_data would be? Or do you have a reference where we can find that formula?</p>

username: Patrickens
cooked: <p>Oeh, sorry I misinterpreted, I thought you guys were already working on it. I unfortunately do not know of such a publication.</p>
<p>On another note: are there any plans to add a torch.sparse x torch.sparse matrix multiplication method? It would not have to work with autograd, but when constructing my system this would save tons of memory.</p>

username: albanD
cooked: <p>Unfortunately no one is working on this at the moment no <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  The github issue is here so that if someone wants to contribute it, we will be happy to help and accept a PR <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>For sparsexsparse mm, I think it is on the roadmap for sparse. There is an issue discussing this as well that will have more informations,</p>

username: Patrickens
cooked: <p>Thanks a bunch! I am now kind of working around it by doing all sparse x sparse multiplications as sparse x dense and subsequently turning the result back into sparse. All of the resulting tensors are static, so do not need to be autogradable.</p>
How to free-up model activations

username: shagunsodhani
cooked: <p>I am trying a distillation-like setup where predictions from one model (say model X) are used as target for another model (say model Y). Since model X is pre-trained, I compute the logits under <code>torch.eval</code> context and train model Y with it.</p>
<p>In this case, would Pytorch maintain the computation graph (and intermediate activations) corresponding to model X or free up the computational graph as soon as the forward pass for model X is complete? Is there a way to free the memory held by computational graph for model X?</p>

username: albanD
cooked: <p>If you don’t want any autograd related operations to happen while you forward your model X, you can run the forward in a no_grad block:</p>
<pre><code class="lang-auto">with torch.no_grad():
   out = model_X(inp)
</code></pre>

username: shagunsodhani
cooked: <p>Thanks for the prompt reply. I am wondering how to avoid storing the intermediate activations (for model X) and not just the .grad attribute</p>

username: albanD
cooked: <p>Running in no grad mode will disable all the autograd. So no graph will be created and no intermediate activation will be saved.</p>

username: shagunsodhani
cooked: <p>Perfect! I will try it out and if everything works, I will come back to mark it as the accepted answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
My gpu is working slower than cpu

username: vamsi_balijepally
cooked: <p>my code is here<br>
import torch<br>
torch.cuda.is_available() // it was True<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
from torch.utils.data import DataLoader<br>
from torchvision import datasets,transforms<br>
import pandas as pd<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
import seaborn as sn<br>
from sklearn.metrics import confusion_matrix</p>
<p>trans = transforms.ToTensor()<br>
train_data = datasets.CIFAR10(root = “C:/Users/vamsi”,train = True,download = True,transform = trans)<br>
test_data = datasets.CIFAR10(root = “C:/Users/vamsi”,train = False,download = True,transform = trans)</p>
<p>class ConvolutionalNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3,6,5,1)<br>
self.conv2 = nn.Conv2d(6,16,3,1)<br>
self.fc1 = nn.Linear(16<em>6</em>6,120)<br>
self.fc2 = nn.Linear(120,84)<br>
self.fc3 = nn.Linear(84,10)</p>
<pre><code>def forward(self,x):
    x = F.relu(self.conv1(x))
    x = F.max_pool2d(x,2,2)
    x = F.relu(self.conv2(x))
    x = F.max_pool2d(x,2,2)
    x = F.relu(self.fc1(x.view(-1,16*6*6)))
    x = F.relu(self.fc2(x))
    x = F.log_softmax(self.fc3(x),dim =1)
    return x
</code></pre>
<p>model = ConvolutionalNetwork()<br>
model.cuda()<br>
next(model.parameters()).is_cuda // it was true<br>
criterion = torch.nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters(),lr = 0.001)</p>
<p>start_time = time.time()<br>
<span class="hashtag">#trackers</span><br>
epochs = 13<br>
train_correct = []<br>
test_correct = []<br>
train_loss = []<br>
test_loss = []</p>
<p>for i in range(epochs):<br>
trn_corr = 0<br>
tst_corr = 0<br>
for b,(x_train,y_train) in enumerate(train_load):<br>
b += 1<br>
y_pred = model.forward(x_train.cuda())<br>
loss = criterion(y_pred,y_train.cuda())<br>
pred = torch.max(y_pred.data ,1)[1]<br>
trn_corr += (pred == y_train.cuda()).sum()<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>    if b%500 == 0:
        
        print(f'EPOCH:{i} BATCH:{b} acuracy:{trn_corr.item()*100/(b*10)} loss:{loss.item()}')
        
        
train_loss.append(loss)
train_correct.append(trn_corr)


with torch.no_grad():
    for b,(x_test,y_test) in enumerate(test_load):
        y_val = model.forward(x_test.cuda().type(torch.float32))
        predict = torch.max(y_val.data,1)[1]
        tst_corr += (predict == y_test.cuda().type(torch.float32)).sum()
        loss = criterion(y_val,y_test.cuda())
        
    
test_loss.append(loss)
test_correct.append(tst_corr)
</code></pre>
<p>print(f’total time taken = {(start_time-time.time())/60} ')</p>

username: Rohan_Kumar
cooked: <p>Pls format the code correctly, so people could help you with more ease.</p>
Fastest way to get the minim,element wise,between two tensors

username: Cosmin1
cooked: <p>Currently i m doing something like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44cf08f497c1748424399f8414565e83fcb26cc0.png" data-download-href="https://discuss.pytorch.org/uploads/default/44cf08f497c1748424399f8414565e83fcb26cc0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44cf08f497c1748424399f8414565e83fcb26cc0.png" alt="image" data-base62-sha1="9OI2nAoTNO69JMzgp5fmpnXWyB2" width="690" height="82" data-dominant-color="232425"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1233×148 3.82 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You could use the elementwise comparison using <code>torch.min</code> and <code>torch.max</code> by passing two tensors:</p>
<blockquote>
<p>min(input, other, out=None) → Tensor<br>
Each element of the tensor <code>input</code> is compared with the corresponding element of the tensor<br>
<code>other</code> and an element-wise minimum is taken.<br>
The resulting tensor is returned.</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">a = torch.randn(10, 2)
b = torch.randn(10, 2)
torch.min(a, b)
</code></pre>

username: Cosmin1
cooked: <p>Thank you for the response,but it doesen t seem to work like that using libtorch.</p>
Libtorch in linux cluster much lower speed than my local machine

username: Yue_Li
cooked: <p>Hi,  when I am running a small CNN model in C++ using libtorch on linux clusters, i find its speed is much slower than on my local machine. Does anyone know the reason?  The network input is 2x128x128, on my local machine, it takes only 0.002 seconds to finish the forward. However on linux clusters, it takes rougly 0.3 seconds to finish the forward. Does this issue have anything to do with loading dynamic libraries or multi thread, I mean since the input and network are very simple, will multi thread cause slower speed?</p>

username: ptrblck
cooked: <p>Are you using the CPU only or also a GPU?<br>
In the latter case, are you synchronizing the code? Since CUDA operations are asynchronous, you would have to synchronize the code via <code>torch.cuda.synchronize()</code> to get a valid profiling result.<br>
If you are using a GPU, do both machines have the same GPU, CUDA, and cudnn version?</p>
<p>Also, how did you install PyTorch on these machines? Did you build from source or installed a binary?<br>
Are both PyTorch versions equal?</p>

username: Yue_Li
cooked: <p>Hi, Thank you for helping!</p>
<p>For both cases, I am using CPU only.</p>
<p>I installed from a binary for both cases. In my local machine, what I used is libtorch for Mac. While on linux clusters, I have tried different versions of libtorch and observed this issue always. Also I noticed that on clusters, running time at different times are quite different. Sometimes, it takes 0.4 seconds. Sometimes, it takes maybe only 0.03 seconds for a same input. In addition, after i reduce the model size from 90M to 4 M, the running speed on my local machine improved. However, on clusters, the running speed seems almost unchanged. It feels just like on clusters, there is some extra processing that takes some time. I am so confused.</p>

username: ptrblck
cooked: <p>A 10x speedup sounds strange.<br>
Are you seeing these 0.4s and 0.03s runs randomly or is the first pass slower than the rest?</p>

username: Yue_Li
cooked: <p>Hi, sorry for the late reply.<br>
First regarding your question, I was seeing these 0.4s and 0.03s almost in every runs.<br>
Later, i found that this issue can be fixed by adding the following sentence   at::init_num_threads();<br>
Thanks again for your kind help!</p>
Select a sequential subset of Dataset

username: verance
cooked: <p>Hello, I would like to implement a loop where iteration by iteration I select part of the dataset and train on it.<br>
What is the best way to select the first x samples of my dataset? Something like this: <code>train_dataset[0:300], train_dataset[300:600]</code></p>
<p>Another idea is to make that choice random but without repeating numbers at each iteration. So the idea is to select <code>x</code> random numbers from <code>0, len(train_dataset)</code> and remove them after each iteration. But for that, I would need to slice the train_dataset based on one array, <code>train_dataset[[1, 2, 3]]</code>, for example.</p>
<p>Any idea on how to do something like this?</p>
<p>Regards</p>

username: Kushaj
cooked: <p>You can do this by using dataloader. Pass <code>bs=300</code> and <code>shuffle=False</code> for your first option. For your second option you can set <code>shuffle=True</code>.</p>
<p>Also, if you do not want to use dataloader. Then you use this.</p>
<ol>
<li>If you create a pytorch dataset. Then you can access the underlying data using <code>dataset.data[0:300]</code>. In this way you can select your x samples.</li>
<li>To get random samples.
<ul>
<li>First use <code>torch.randperm(len(dataset))</code>. This will give you a random permutation of integers which you will use as indexes.</li>
<li>Now you can take the samples based on this list.</li>
</ul>
</li>
</ol>
How to choose adaptive lambda for regularizer?

username: Janashree_Ananthan
cooked: <p>Hai,<br>
Can anyone provide me the way of using adaptive regularizer?<br>
say</p>
<pre><code class="lang-auto">reg += torch.sum(torch.mul(w,w)-a)
</code></pre>
<p>I need the <strong>a</strong> to be trainable, how should i do that?<br>
Thanks in advance!</p>

username: Kushaj
cooked: <p>If you want <code>a</code> to be a parameter of the model wrap it around <code>nn.Parameter</code>. In your <code>nn.Module</code> add this line: <code>self.a = nn.Parameter(torch.tensor(0.5), requires_grad=True)</code>, where 0.5 is the initialization value.</p>
Add softmax layer to Resnet50

username: pulpaul
cooked: <p>Hi there,</p>
<p>I would like to add a Softmax layer to the end of a custom Resnet50 that i have trained. I wana add it for validation. My ResNet50 is:</p>
<pre><code class="lang-auto">    model = torchvision.models.resnet50(pretrained=True)
    model.fc = nn.Sequential(
        nn.Linear(model.fc.in_features, 1000),
        nn.Linear(1000, 128),
        nn.Linear(128, 64),
        nn.Linear(64, 8)
    )
</code></pre>
<p>How could I do it?</p>
<p>Thanks</p>

username: Isaac_Kargar
cooked: <p>You want to get your trained model without softmax and feed it an image and get its output and apply softmax on the result, right? if so, you can use sth like: <code>nn.Softmax()(model(image))</code> or <code>F.softmax(model(image))</code>. it will give you the probability for 8 classes.</p>

username: Kushaj
cooked: <p>You can use <code>nn.Sequential</code> to add your layer.</p>
<pre><code class="lang-python">model.fc = nn.Sequential(
    *model.fc,
    nn.Softmax(),
)
</code></pre>
Generating Data in Parallel

username: bing
cooked: <p>Hi Guys,</p>
<p>I am trying to generate data in parallel following <a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" rel="nofollow noopener">this</a> tutorial.</p>
<p>This tutorial first assumes that my dataset should be in this format-<br>
<code>training_generator = SomeSingleCoreGenerator('some_training_set_with_labels.pt')</code></p>
<p>I have never stored data in this format, mine data is in<br>
Dataset<br>
ClassA<br>
ClassB…</p>
<p>format.</p>
<p>How to do it in the above format, so I can proceed to follow the tutorial in the required format?</p>

username: albanD
cooked: <p>I think this example refers to the case where you use the builting <code>torch.utils.data.Dataset</code> and <code>torch.utils.data.Dataloader</code> to build your dataset loader.<br>
The dataloader in particular will give you a generator like this.</p>

username: bing
cooked: <p>Okay, I have a doubt. Before following the tutorial, I was doing the data parallelism using the official Pytorch:DATA PARALLELISM -(<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#optional-data-parallelism" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#optional-data-parallelism</a>) tutorial.<br>
It basically wraps up the model in <code>nn.DataParallel(model)</code>.<br>
So <code>nn.DataParallel(model)</code> doesn’t have any effect on loading the input data in parallel and data is still being loaded using a single core only?</p>

username: albanD
cooked: <p>The DataParallel module is to run your model in parrallel on multiple GPUs. This is not related to loading data in parallel !</p>
Equivalent of Keras max_norm constraint in Pytorch

username: kmt
cooked: <p>I’m trying to implement the equivalent of the Keras max_norm constraint in my Pytorch convnet.<br>
" <code>maxnorm(m)</code>  will, if the L2-Norm of your weights exceeds  <code>m</code> , scale your whole weight matrix by a factor that reduces the norm to  <code>m</code> . " It can also constrain the norm of every convolutional filter which is what I want to do.</p>
<p>(Fuller explanation of max_norm here:<br>
<a href="https://stats.stackexchange.com/questions/257996/what-is-maxnorm-constraint-how-is-it-useful-in-convolutional-neural-networks" rel="nofollow noopener">https://stats.stackexchange.com/questions/257996/what-is-maxnorm-constraint-how-is-it-useful-in-convolutional-neural-networks</a> )</p>
<p>I found this answer <a href="https://discuss.pytorch.org/t/kernel-constraint-similar-to-the-one-implemented-in-keras/49936" class="inline-onebox">Kernel Constraint similar to the one implemented in Keras</a> in the forums but I don’t understand how the clamp implements max_norm, or how to use it to constrain the norm for individual convolutional filters.</p>
Cutomize VGG11 to train

username: jasg
cooked: <p>I would like to customize the VGG 11 model so that it has an input channel of 1, that is, the images to train are grayscale, please help me</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>I think easiest way is too define a super class and change only first layer, something like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision

class MyVGG(nn.Module):
    def __init__(self, in_channel=1):
        super(MyVGG, self).__init__()
        
        # use PyTorch implementation
        self.model = torchvision.models.vgg11()
        # replace first layer of model with your own desired
        self.model.features[0] = nn.Conv2d(1, 64, 3, 1, 1)

    def forward(self, x):
        return self.model(x)

myvgg = MyVGG()
</code></pre>
<p>Another approach is to use the source code of models <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py" rel="nofollow noopener">here</a> and change <a href="https://github.com/pytorch/vision/blob/12b551e7a7232d829df0f01ae9f6c56305571dfc/torchvision/models/vgg.py#L65" rel="nofollow noopener">this line</a> to <code>in_channels = 1</code>.</p>
<p>This <a href="https://discuss.pytorch.org/t/modify-resnet-or-vgg-for-single-channel-grayscale/22762">thread</a> may also help you.</p>
<p>bests</p>

username: jasg
cooked: <p>Thanks its usefull for me</p>
__call__ signature of nn.Module

username: vpj
cooked: <p><a href="https://github.com/lab-ml/labml/blob/master/labml/helpers/pytorch/module.py" rel="nofollow noopener">https://github.com/lab-ml/labml/blob/master/labml/helpers/pytorch/module.py</a></p>
<p>This is a class I wrote than subclass <code>nn.Module</code> and let modules implement <code>__call__</code> instead of <code>forward</code>. I did this to get better <em>type checking</em> and <em>type inference</em> in python.</p>
<p>This module uses <code>__init_subclass__</code> set the <code>__call__</code> method of it’s subclasses to <code>forward</code> so that it doesn’t mess up with <code>nn.Module</code> hooks and stuff. I’ve been using this for a while and hasn’t encountered problems.</p>
<p>Are there implications that I haven’t seen?</p>

username: albanD
cooked: <p>Hi,</p>
<p>So this does not call the original <code>__call__</code> anymore?</p>

username: vpj
cooked: <p>It does, basically we use <code>__init_subclass__</code> to rename your implementation of <code>__call__</code> to <code>forward</code></p>

username: albanD
cooked: <p>But the original implementation of <code>__call__</code> then calls <code>forward</code>. Doesn’t that lead to infinite recursion?</p>

username: vpj
cooked: <p>Sorry I din’t see your comment.</p>
<p>No it doesn’t call itself. Only the subclass implementation of <strong>call</strong> is renamed.</p>
Does performance drop due to default adding of parameters gradients?

username: braindotai
cooked: <p>Just a question out of curiosity, in pytorch the current gradients of the parameters is automatically added to the previous ones, therefore we have to set all the gradients to zero after each update.<br>
So is it true that by doing that it does cause performance regression since we are doing two unwanted stuff repeatedly - 1. Set the previous gradients to zero and - 2. Then add the current gradients to the previous gradients.<br>
Also is there a way to stop the default addition of gradients.</p>

username: albanD
cooked: <p>The thing is that you need somewhere to store the gradients. And in modern systems, adding into an existing buffer is much faster than allocating a new one then copying the values into it.<br>
So this is not really a perf regression: re-using memory is actually a better way to do this.</p>
<p>If you really want to not do that, you can set the <code>.grad</code> fields to <code>None</code> instead of zeroing them out and a new Tensor to store the gradients will be set back every time.</p>

username: braindotai
cooked: <p>So that’s what it is, thanks for your time.</p>
Change Tensor in forward pass

username: Awais_Jafar
cooked: <p>I’m trying modify a tensor in forward block to copy one value to another for a tensor in forward pass for a neural network.<br>
Something like</p>
<pre><code class="lang-auto">
def forward(self, X):
        x = self.fc(X)
        x = self.modify_tensor(x)
        return self.fc2(X)

def modify_tensor(self, X):
        idx = random.sample(range(X.shape[0), k=1)
        X[idx] = X[idx+1]
        return X
</code></pre>
<p>First I’m not sure if it’s correct since I would want the gradients to be calculated according to the new value. Secondly, this does work (not sure if correctly but it runs) but the backward pass for such an operation takes forever. I’m not sure what is happening. is there a better way?</p>

username: albanD
cooked: <p>For the correctness, yes it will work. Keep in mind that the values that you override will get 0 gradients ars they are not used anymore.</p>
<p>For the speed, it should not change much.<br>
Can you give sample Tensors (generated with torch.rand) for example to make a runnable example that shows the slow down?</p>
EmbeddingBag of vocab size

username: Amy12xx
cooked: <p>Why does EmbeddingBag expect length of vocab as number of embeddings?</p>
<p><strong>num_embeddings</strong> - size of the dictionary of embeddings</p>
<p><a href="https://pytorch.org/docs/stable/nn.html?highlight=embeddingbag#torch.nn.EmbeddingBag" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html?highlight=embeddingbag#torch.nn.EmbeddingBag</a></p>
<p>I’m following the example here <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html</a></p>

username: vainaijr
cooked: <p>maybe it create a representation for each word, because computer do not understand word, computer understand numbers, so when we do,</p>
<pre><code class="lang-auto">x = nn.EmbeddingBag(num_embeddings=10, embedding_dim=3, mode='sum')
list(x.parameters())
</code></pre>
<p>it give,</p>
<pre><code class="lang-auto">[Parameter containing:
 tensor([[ 0.8823, -0.3787,  0.8360],
         [-1.4388, -0.6124, -1.6967],
         [ 0.4632,  0.6406,  0.1272],
         [-0.8657, -2.0807, -0.9140],
         [-0.3749, -0.5471, -0.5424],
         [ 0.9730,  0.5713,  0.4584],
         [-1.3402,  0.1033, -1.4363],
         [-0.1600, -0.3686, -0.2954],
         [ 1.1288, -0.1282, -1.0070],
         [ 0.8220, -0.0371, -0.7206]], requires_grad=True)]
</code></pre>
<p>this means that our vocabulary has 10 words, and each of those words are represented by an array of 3 floating point numbers, so, for a computer, a word would mean these 3 floating point numbers.</p>
<p>so if our vocabulary had 10 words like</p>
<pre><code class="lang-auto">apple orange banana grape juice fruit pineapple strawberry mango watermelon
</code></pre>
<p>then for computer,</p>
<pre><code class="lang-auto">apple
</code></pre>
<p>would mean</p>
<pre><code class="lang-auto">[ 0.8823, -0.3787,  0.8360]
</code></pre>
<p>and</p>
<pre><code class="lang-auto">orange
</code></pre>
<p>would mean</p>
<pre><code class="lang-auto">[-1.4388, -0.6124, -1.6967],
</code></pre>
<p>if we want to change the representation of any of these words, then update their embedding.</p>
<p>plus embeddingbag would give us a <code>sum</code> (or we could even get <code>mean</code>) of these embeddings, that is, if we do,</p>
<pre><code class="lang-auto">x(torch.LongTensor([[0, 1]]))
</code></pre>
<p>then we will get,</p>
<pre><code class="lang-auto">tensor([[-0.5565, -0.9911, -0.8608]], grad_fn=&lt;EmbeddingBagBackward&gt;)
</code></pre>
<p>that is sum of first two arrays</p>

username: Amy12xx
cooked: <p>Yes, I understand it now, thank you. “EmbeddingBag” threw me off, but I see it is similar to having an Embedding layer (so it is of vocab size to index into) except that we can perform an aggregation over it.</p>
Error with torch.scatter_

username: Amit17
cooked: <p>Hi,I am very new to pytorch and I am exploring the torch library.I am getting the following issue with scatter_<br>
import torch</p>
<p>one_hot = torch.zeros(10,10,dtype=torch.float64)</p>
<p>one_hot.shape</p>
<p>a=torch.Tensor([3.,5.,2.,1.,4.,6.,7.,8.,9.,10.])<br>
one_hot.scatter_(1,a.unsqueeze(1),1)<br>
RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 one_hot.scatter_(1,a.unsqueeze(1),1)</p>
<p>RuntimeError: index 4656722015779028992 is out of bounds for dimension 1 with size 10</p>

username: vainaijr
cooked: <p>maybe it expect index as type long, so,</p>
<pre><code class="lang-auto">a = torch.tensor([3,5,2,1,4,6,7,8,9])
</code></pre>
<p>would work</p>

username: Amit17
cooked: <aside class="quote no-group" data-username="vainaijr" data-post="2" data-topic="82310">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vainaijr/48/49167_2.png" class="avatar"> vainaijr:</div>
<blockquote>
<p><code>a = torch.tensor([3,5,2,1,4,6,7,8,9])</code></p>
</blockquote>
</aside>
<p>hi, it worked replaced  I guess the issue was with float dtype.</p>