akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Bilinear is slower than nearest after QAT

username: KevinZ1992
cooked: <p>In my origin model,the upsample part is</p>
<pre><code class="lang-auto">F.interpolate(l7, scale_factor=2.0, mode='bilinear', align_corners=True)
</code></pre>
<p>，when i get QAT model.pt and tried it on android ,the inference time of the model.pt is slow, just similar to the float.pt<br>
So，i changed the upsample part just like</p>
<pre><code class="lang-auto">F.interpolate(l7, scale_factor=2.0, mode='nearest')
</code></pre>
<p>the inference time is speed up.<br>
But the result of segmentation model is too bad.<br>
Why bilinear is slower than nearest after QAT?<br>
Is there anyone can explain and give some suggestions.<br>
Thx</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Actually, I do not know about the QAT, but <code>nearest</code> is always faster than <code>bilinear</code>. In <code>bilinear</code> a transformation need to be computed meanwhile nearest is just copy/paste without any computation (almost).</p>
<p>Although in large tensors, <code>linear</code> is possibly preferred even in term of speed.</p>
<p>Bests</p>

username: KevinZ1992
cooked: <p>Quantization-aware training (QAT) is the quantization method that typically results in the highest accuracy.<br>
You are right, <code>nearest</code>  is always faster than  <code>bilinear</code>.<br>
I test it on android, the difference less than 5ms ,but it’s more than about 100ms for quanted model.</p>
Equivalent of Keras's `mask` arg in the forward pass of an RNN

username: henry-prior
cooked: <p>I’m looking for the equivalent of the  <code>mask</code>  arg for the forward pass from Keras’s RNN implementation in PyTorch. It seems that PyTorch has the  <code>pack_padded_sequence</code>  function to skip over the end of padded sequences, but Keras has some additional functionality that I need, which is the ability to skip RNN computations at arbitrary time points and propagate the previous time step to the next time step according to the  <code>mask</code>  arg.</p>

username: aponcedeleonch
cooked: <p>Let me know if you wind a solution or workaround to this. I am actually struggling with exactly the same.</p>
How to debug nan in loss issue?

username: lev
cooked: <p>Hi all,<br>
I am not posting the actual code since it is a lot and cannot be reduced that easy to the problem. But essentially I am having</p>
<p>output = net(input) which is a batchsize x 1 tensor.<br>
I calculate the mean of output over the batchsize and my loss function is</p>
<p>loss = (mean - 10).pow(2).</p>
<p>So I am trying to have a mean of my network output = 10. First 2 iterations the loss goes down, in the third iteration it suddenly goes up to a million and then nan.</p>
<p>How can I debug such an issue (the definition of the loss was a simplification)?</p>

username: Atul_Kumar
cooked: <p>I think problem is with your loss function. You should formulate it differently and add more constraint otherwise gradient gonna explode. or try abs(mean-10)</p>

username: lev
cooked: <p>But why? What is the problem with it? (Btw. I did the same in TensorFlow, at least I hope it is the same, and received no error)</p>

username: lfolle
cooked: <p>The problem might be, that your loss has no upper bound. So in case your <code>mean</code> has high value, the resulting loss will be extremely large and possibly resulting in an <code>Nan</code>.</p>
<p>You might be able to alleviate this by limiting the loss.<br>
Either by clipping the loss to a maximum value or by limiting the range of the <code>mean</code> values by using a sigmoid prior to entering the loss calculation.</p>

username: lev
cooked: <p>You are right, the loss is suddenly growing. But it starts properly:<br>
The sequence is: 1600, 1550, 1520, 600000, 5e17, nan<br>
I am wondering why it suddenly goes up like that.<br>
If I clip the loss, wont my gradient be 0 and training collapse?<br>
Sigmoid seems an interesting idea, I will try that, thanks.</p>

username: lfolle
cooked: <p>Yes you are right, I meant clipping the gradient. This should avoid extreme steps during the optimization.<br>
You might also try to lower your learning rate to see if the current learining rate is to high.</p>

username: lev
cooked: <p>With much reduced learning rate it works out but too slow and convergence is not as good as it should be (comparing to TensorFlow version).<br>
Ok I can try clipping the gradient, thanks.<br>
I now tried Sigmoid and the result was that loss went down for many iterations (hundreds) and then again suddendly went up in shortest time (10 iterations) and gave nan again. I am more and more confused.</p>

username: lev
cooked: <p>Thank you once more for you advice. The clipping did the trick in so far that it does converge. Value is not very good though.<br>
Still I am puzzled why this gradient explodes so suddenly. But it’s good to control it</p>
Problems using a ResNet pretained model

username: learningpytorch
cooked: <p>I am trying to use a pretrained ResNet model but I get this error:<br>
RuntimeError: size mismatch, m1: [30 x 2048], m2: [9216 x 1024] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283</p>
<p>model.fc = nn.Sequential(nn.Linear(9216,1024), nn.ReLU(),nn.Dropout(0.4),nn.Linear(1024,10))</p>
<p>Full Notebook can be found here: <a href="https://colab.research.google.com/drive/1k40NNulSIS6ANagopSPBH4Xty_Cw39qC#scrollTo=9J6VCm2y8tXn" rel="nofollow noopener">https://colab.research.google.com/drive/1k40NNulSIS6ANagopSPBH4Xty_Cw39qC#scrollTo=9J6VCm2y8tXn</a></p>
<p>Please I would really appreciate a bit of help with this issue since I am a new practitioner. Thanks in advanced.</p>

username: ptrblck
cooked: <p>It looks like you already corrected the number of input features to <code>2048</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: learningpytorch
cooked: <p>Yes <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> Thanks for your help! However, I am still getting an error that seems to be in the test loader and cannot find where…if you have a clue I will appreciate it a lot.</p>

username: ptrblck
cooked: <p>Which error are you getting now?</p>

username: learningpytorch
cooked: <p>I got this: RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same</p>
<p>In the Google Colab link of my first post this message can be seen at the very end of the notebook.</p>
<p>Thanks a million for your kindness.</p>

username: ptrblck
cooked: <p>This error points to a device mismatch between your input tensor and the model parameters.<br>
You should push <code>X_test</code> to the device (same as with the training data) via: <code>X_test = X_test.cuda()</code>.</p>

username: SANTOSH_S
cooked: <p>Hi <a class="mention" href="/u/learningpytorch">@learningpytorch</a><br>
I did see your colab code.<br>
You have used <code>train_loader</code> instead of <code>valid_loader</code> for testing the performance. In the validation you’ll have to move the <code>X_test, y_test</code> to CUDA to solve the issue.</p>

username: learningpytorch
cooked: <p>Thanks a million for your kindness!</p>

username: learningpytorch
cooked: <p>Thanks a million for your help <a class="mention" href="/u/santosh_s">@SANTOSH_S</a>! It is important for people like me who are not really experienced to receive feedback from experts like you and <a class="mention" href="/u/ptrblck">@ptrblck</a>. I really appreciate it.</p>
What is np.where's equivalent in libtorch?

username: Kiki_Rizki_Arpiandi
cooked: <p>I can get the index of matrix elements where the values hold particular condition, by:</p>
<pre><code class="lang-auto">import numpy as np 
matrix = np.random.randint(0,8,(4,4)) 
inds =np.where(matrix&lt;5) 
</code></pre>
<p>How do I achieve the same result using <code>torch::where(.)</code> ?<br>
It’s seem that <code>torch::where(.)</code> has different api with <code>np.where(.)</code> .</p>

username: tot0k
cooked: <p>I installed the excellent <a href="https://github.com/dpilger26/NumCpp" rel="nofollow noopener">NumCpp</a> library, which is a header only implementation of numpy for C++. Works fine for me.</p>
Optimizer lr changing after loading state dict

username: bluesky314
cooked: <p>After I load my optimiser state dict when a previously run session with a different lr, the new optimizer’s lr also changes.</p>
<p>eg)</p>
<pre><code class="lang-auto">lr=0.01
opt = torch.optim.Adam(model.parameters(), lr=lr, betas=(0.9, 0.999), eps=1e-08, weight_decay=weight_decay) 
for groups in opt.param_groups: print(groups['lr']); break
opt.load_state_dict(torch.load(load_saved)['optimizer']
for groups in opt.param_groups: print(groups['lr']); break
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">0.01
0.05
</code></pre>
<p>Is this the correct behavior? I am reloading previous stats but lowering lr to continue training</p>

username: ptrblck
cooked: <p>I assume the learning rate was stored in the <code>state_dict</code> and thus loaded in:</p>
<pre><code class="lang-python">opt.load_state_dict(torch.load(load_saved)['optimizer']
</code></pre>
<p>which I would consider the correct behavior to restore the previous session.<br>
If you want to change it after loading the <code>state_dict</code>, you could change it directly in the <code>param_group</code>.</p>
<p>Let me know, if I misunderstood the question.</p>
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1) when using batchNorm

username: meryem_janati
cooked: <p>I got the error above when I use the custom batchNorm exactly at <a href="https://github.com/ptrblck/pytorch_misc/blob/31ac50c415f16cf7fec277dbdba72b9fb4d732d3/batch_norm_manual.py#L63" rel="nofollow noopener">this line</a><br>
when I checked the size of the input in my model I get <code>torch.Size([0])</code> during the training  and <code>torch.Size([1000, 3 ,32, 32])</code> at test time knowing that the model train with no error w/o batchNorm or with the batchNorm of torch and gives good accuracy. Can you please explain to me how is that possible if the input size is 0 and how to make it work with the custom batchNorm. thank you</p>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self, conv1_dim=100, conv2_dim=150, conv3_dim=250, conv4_dim=500):
        super(Net, self).__init__()
        self.conv4_dim = conv4_dim

        self.conv1 = nn.Conv2d(3, conv1_dim, 5, stride=1, padding=2)
        self.conv2 = nn.Conv2d(conv1_dim, conv2_dim, 3, stride=1, padding=2)
        self.conv3 = nn.Conv2d(conv2_dim, conv3_dim, 3, stride=1, padding=2)
        self.conv4 = nn.Conv2d(conv3_dim, conv4_dim, 3, stride=1, padding=2)

        self.pool = nn.MaxPool2d(2, 2)

        self.fc1 = nn.Linear(conv4_dim * 3 * 3, 270) # 3x3 is precalculated and written, you need to do it if you want to change the # of filters
        self.fc2 = nn.Linear(270, 150)
        self.fc3 = nn.Linear(150, 10)

        self.normalize1 = mbn.MyBatchNorm2d(conv1_dim)
        self.normalize2 = mbn.My.BatchNorm2d(conv2_dim)
        self.normalize3 = mbn.MyBatchNorm2d(conv3_dim)
        self.normalize4 = mbn.MyBatchNorm2d(conv4_dim)

    def forward(self, x):
        print(x.size())
        x = self.pool(F.relu(self.normalize1((self.conv1(x))))) # first convolutional then batch normalization then relu then max pool
        x = self.pool(F.relu(self.normalize2((self.conv2(x)))))
        x = self.pool(F.relu(self.normalize3((self.conv3(x)))))
        x = self.pool(F.relu(self.normalize4((self.conv4(x)))))

        x = x.view(-1, self.conv4_dim * 3 * 3) # flattening the features
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)

        return x
</code></pre>
<p>for loss I tested with both nll_loss and cross_entropy</p>

username: ptrblck
cooked: <p>This line of code checks the size of the channel dimension, while you input seems to be empty.<br>
Make sure to pass valid inputs to the method, as empty inputs are not supported in this manual batch norm version.</p>
How to get Segmentation Mask of Pascal VOC Dataset

username: Haider_Ali_Shuvo
cooked: <p>I am very new to image segmentation .<br>
I tried loading Pascal VOC datasets using torchvision dataset , But in return it gives original image and segmentation image of different colours. But for an image of  &lt;C,H,W&gt; what i need is &lt;H,W&gt; masks to let me know which pixel belongs to which class .<br>
How do i get the segmentation mask ?</p>
Replace "Variable" with torch.Tensor

username: codeknight11
cooked: <p>I am working on someone’s old codebase and I noticed they have used <code>Variable</code> at a lot of places.<br>
I know that <code>Variable</code> has been deprecated.<br>
What changes can I make to the code to replace Variable with Tensor?</p>
<pre><code class="lang-auto">for n, p in model.named_parameters():
	# retrieve the consolidated mean and fisher information.
	n = n.replace('.', '__')
	mean = getattr(model, '{}_estimated_mean'.format(n))
	fisher = getattr(model, '{}_estimated_fisher'.format(n))
	
	# wrap mean and fisher in variables.
	# mean = Variable(mean)
	# fisher = Variable(fisher)
	mean = torch.Tensor(mean, requires_grad=True)
	fisher = torch.Tensor(fisher, requires_grad=True)  
</code></pre>
<p>Are the changes in last 4 lines correct? I’ve replaced <code>Variable(mean)</code> with <code>torch.Tensor(mean, requires_grad=True)</code>  and so on…</p>
<p>Also I would like to know if wrapping a tensor inside <code>Variable</code> used to detach it from the computation graph? If yes, then I would like to do the same here.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can replace calls like <code>Variable(foo)</code> by <code>foo</code>.<br>
And you can replace <code>Variable(bar, requires_grad=True)</code> by <code>bar.requires_grad_()</code>.</p>
Require_grad vs requires_grad

username: AshviniKSharma
cooked: <p>What is the difference between require_grad and requires_grad? I am trying to freeze layers of the network. But during the implementation of the freeze function, I used require_grad = False for freezing layers, and now when I am checking it with requires_grad it says layers are not frozen i.e. result in requires_grad=True? Is there any fundamental difference between the two implementations?</p>
<pre><code class="lang-auto">model.freeze()
for name,param in model.named_parameters():
    print(name,param.requires_grad)
</code></pre>
<p>Code of model.freeze():</p>
<pre><code class="lang-auto">def freeze(self):
        # To freeze the residual layers
        for param in self.network.parameters():
            param.require_grad = False
        for param in self.network.fc.parameters():
            param.require_grad = True
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p><code>require_grad</code> is not a pytorch thing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> So it has no effect.<br>
Note that you can set any attribute you want on Tensor, this is why you didn’t see any error (param.foo = 3 would work as well).</p>
Save preds as numpy

username: alameer
cooked: <p>I am trying to save the predictions from my code as CSV file but getting this error message. Can someone please help? Thanks</p>
<pre><code class="lang-auto">  output = model(images)
   preds = output.sum(dim=[1,2,3])
   np.save('output.csv', preds)
</code></pre>
<p>I am getting this error message:</p>
<p>RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.</p>

username: albanD
cooked: <p>Hi,</p>
<p>As mentioned in the error, you will need to detach to avoid giving a Tensor that requires_grad=True to numpy: <code>np.save('output.csv', preds.detach())</code></p>

username: alameer
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="85810">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>preds.detach()</p>
</blockquote>
</aside>
<p>Thanks! That has got rid of the error message, but the file that is getting saved as csv is empty for some reason. Any idea why please?</p>

username: albanD
cooked: <p>Not sure. What is the content of the Tensor you give to it? You can try to convert it to numpy before passing it with <code>pred.detach().numpy()</code> ?</p>

username: alameer
cooked: <p>Many Thanks for your help! When i run the training model the file is saving the tensor which consist of 16 values. However the values seem to get over written at each step and the final CSV final only has 3 values. Any idea why I am not getting all the values? Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>I would guess that <code>np.save()</code> overrides the file every time and that the last batch from the dataloader is of size 3 (because dataset_len % batch_size = 3) and so you get this.<br>
I am not sure if there is an option to append in np.save but a solution is to accumulate all the values in a Tensor and write it at the end.<br>
Or write to the file yourself and make sure to append to it.</p>
Pytorch Tensorboard NameError

username: keunwoo
cooked: <p>I’m trying to use torch.utils.tensorboard.<br>
But when I call ‘writer.add_scalar’, it crashes with NameError: name ‘workspace’ is not defined</p>
<p>The problem is in tensorboard/writer.py, line 293, in add_scalar<br>
scalar_value = workspace.FetchBlob(scalar_value)</p>
<p>How can I fix this issue?<br>
I already installed ‘future’, but it did not help.</p>

username: Prashant_Kalikotay
cooked: <p>Sharing the code could fetch you better replies.<img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

username: keunwoo
cooked: <p>The issue solve. I was putting weird value in the writer.</p>

username: Duayt
cooked: <p>Im facing the same problem on the workspace name, could you share me your solution? what were the weird logging value you had.</p>
<p>Thanks</p>
Expand size of tensor

username: wall-e
cooked: <p>Hi everyone,</p>
<p>I have feature maps with size: [1,100,20,30]. I want to scan over the feature maps with 10 filters with size of each filter: [1,1,5,5] to generate a new tensor with size: [1,100,20,30,1,10]. Last values of the size: 1 and 10 are the result values of the scanning process.</p>
<p>Could anyone give me an advice how to do that?<br>
Thank you.</p>

username: Scott_Hoang
cooked: <p>So you have an input with this format Batch x Channels x Width x Height and you want to apply a 5x5 kernel to generate an output of [BxCxWxHx1x10] or [Bx10*CxWxH]<br>
can you not use Conv2d? here is my code for it</p>
<pre><code class="lang-auto">a = torch.rand(1,100,20,30)
cnv = torch.nn.Conv2d(in_channels=100, out_channels=100*10, kernel_size=5, stride=1, padding=2)
b = cnv(a).view(1,100,20,30,1,10)
</code></pre>

username: wall-e
cooked: <p>The values of filter are set by myself, not a learnable kernel. For only one filter, I tried with:</p>
<pre><code class="lang-auto">a = torch.randn(1, 100, 20, 30) # Batch x Channels x Height x Width
kernel = kernel.view(1,1,5,5).repeat(100,1,1,1)
output = F.conv2d(a,kerkel.float(),group=100,padding=2)
</code></pre>
<p>and It can work. But I want to apply 10 filters to generate an output of [1, 100, 20, 30, 1, 10], it means that the output is a tensor of 1 batch of 20x30 of 1x10 vector of 100 channels.</p>
Bug: optimizer got an empty parameter list

username: springtostring
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d288943052361d5787bd6d35a850f738922635db.png" data-download-href="https://discuss.pytorch.org/uploads/default/d288943052361d5787bd6d35a850f738922635db" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d288943052361d5787bd6d35a850f738922635db.png" alt="image" data-base62-sha1="u2sT29AfXFYS9c3OXBrvjFUGt4v" width="690" height="124" data-dominant-color="F3F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×166 4.49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am trying to build Logistic matrix factorization.When I run it,the error says that :optimizer got an empty parameter list.I am unfamiliar to PyTorch and I don’t know what causes the error.Can you give me some suggestions?Thank you!<br>
this is my code</p>
<pre><code class="lang-auto">class LMF(nn.Module):
    def __init__(self, C, R_test , n_factor):
       '''
       :param C:
       :param R_test:
       :param n_factor:
       '''
       super(LMF, self).__init__()
       self.n_factor = n_factor
       self.C = torch.tensor(C)
       self.R = torch.tensor(R_test)
       self.n_user = C.shape[0]
       self.n_item = C.shape[1]
       self.X = torch.randn([self.n_user, self.n_factor], requires_grad=True)
       self.Y = torch.randn([self.n_item, self.n_factor], requires_grad=True)
       self.user_biases = torch.randn((self.n_user, 1), requires_grad=True)
       self.item_biases = torch.randn((self.n_item, 1), requires_grad=True)


    def forward(self):

        A=torch.mm(self.X,self.Y.T)
        A=A+self.user_biases
        A=A+self.item_biases.T
        loss=self.C.dot(A) - (1+C).dot(torch.log((1+torch.exp(A))))
        return -loss

    def MPR(self):
        rank = 0.0
        R_sum = torch.sum(self.R)
        R_hat=torch.mm(self.X , self.Y.T)
        R_hat_rank = torch.argsort(torch.argsort(-R_hat, axis=1))
        A = R * (R_hat_rank / self.n_item)
        rank = torch.sum(A)/ R_sum
        return rank.item()

''''''''''''''''
''''''''''''''''
''''''''''''''''
model=LMF(C,R_test,10)
learning_rate=0.01
optimizer=torch.optim.Adam(params = model.parameters() , lr = learning_rate)

</code></pre>

username: harsha_g
cooked: <p>You should use <code>torch.nn.Parameter</code> or <code>torch.nn.ParameterList</code> to specify your parameters.</p>

username: springtostring
cooked: <p>thank you for your help</p>
CUDA extension undefined symbol (I do import torch before the extension)

username: MartinBagge
cooked: <p>Okay, so I have a question about build a CUDA extension.</p>
<p>I have an undefined symbol error and it seems to me that my cpp file cannot see the function inside the cu file even though the cpp has the header of the function.<br>
But since the cpp file is not included in the cu file, how does the system know that the header inside the cpp is implemented in the cu file? How it is getting called?</p>
<p>The reason for my error is most likely lack of knowledge which I hope someone where can provide.<br>
I have of cause read the tutorial about extensions again and again and read the code examples on github but even though I mirror everything just with my own implementations I have undefined symbol on during runtime. And yes I do import torch before the extension.</p>

username: aashaka
cooked: <p>Were you able to figure this out?</p>
ERROR: No matching distribution found

username: dachosen1
cooked: <p>I can’t install torch using tox.ini nor using pip install .  is there a way to fix this error</p>
<h2>Option 1</h2>
<p>requiremenst.txt: content</p>
<pre><code class="lang-auto">torch===1.5.1
torchvision===0.6.1
</code></pre>
<pre><code class="lang-auto">
py37 installdeps: -rrequirements.txt

ERROR: Could not find a version that satisfies the requirement torch===1.5.1 (from -r requirements.txt (line 11)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.5.1 (from -r requirements.txt (line 11))

=================================== log end ===================================
ERROR: could not install deps [-rrequirements.txt]; v = InvocationError("'C:\\Users\\ander\\OneDrive\\Projects\\Common-Voice\\model\\.tox\\py37\\Scripts\\pip.EXE' install -rrequirements.txt", 1)
___________________________________ summary ___________________________________
ERROR:   py37: could not install deps [-rrequirements.txt]; v = InvocationError("'C:\\Users\\ander\\OneDrive\\Projects\\Common-Voice\\model\\.tox\\py37\\Scripts\\pip.EXE' install -rrequirements.txt", 1)

</code></pre>
<h2>Option 2</h2>
<p>requirements.txt: content</p>
<pre><code class="lang-auto">torch===1.5.1 f https://download.pytorch.org/whl/torch_stable.html
torchvision===0.6.1 f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>Error log:</p>
<pre><code class="lang-auto">py37 installdeps: -rrequirements.txt
ERROR: invocation failed (exit code 1), logfile: C:\Users\ander\OneDrive\Projects\Common-Voice\model\.tox\py37\log\py37-1.log
================================== log start ==================================
ERROR: Invalid requirement: 'torch===1.5.1 f https://download.pytorch.org/whl/torch_stable.html' (from line 11 of requirements.txt)

=================================== log end ===================================
ERROR: could not install deps [-rrequirements.txt]; v = InvocationError("'C:\\Users\\ander\\OneDrive\\Projects\\Common-Voice\\model\\.tox\\py37\\Scripts\\pip.EXE' install -rrequirements.txt", 1)
___________________________________ summary ___________________________________
ERROR:   py37: could not install deps [-rrequirements.txt]; v = InvocationError("'C:\\Users\\ander\\OneDrive\\Projects\\Common-Voice\\model\\.tox\\py37\\Scripts\\pip.EXE' install -rrequirements.txt", 1)

</code></pre>
Converted pytorch model doesn't work

username: lugookeke
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/bc4f62cdebd25f4cda01be6462692809dee524c2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/bc4f62cdebd25f4cda01be6462692809dee524c2" title="22%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc4f62cdebd25f4cda01be6462692809dee524c2_2_690x323.jpeg" alt="22%20PM" data-base62-sha1="qRRQzYi3kCHBtJwBn3fsaxipLq2" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc4f62cdebd25f4cda01be6462692809dee524c2_2_690x323.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc4f62cdebd25f4cda01be6462692809dee524c2_2_1035x484.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc4f62cdebd25f4cda01be6462692809dee524c2_2_1380x646.jpeg 2x" data-dominant-color="AFAFB1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">22%20PM</span><span class="informations">2055×964 389 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>hey guys as you can see in the image I converted a pytorch model to onnx, then to coreML but my app crashes once I launch it on my phone, the problem has to do with the input and output names , cormel has input referring to an image and output referring to a string, idk how to implement that when converting from pytorch, any help would be much appreciated.</p>
<p>Thanks</p>

username: RIP_Dude
cooked: <p>Converting quantized model to ONNX, isn’t supported fully yet. I believe it’s an experimental feature??</p>
<p>It could be a few things, but I’d have suggest you test the PyTorch, then ONNX, and finally the CoreML. Pin down the problem.</p>
<p>Can you <strong>share the Jupiter notebook</strong> on the left?</p>
<p>I want to try it to make sure it’s not an issue related to the size of your input dummy variable.</p>
BatchNorm2d RuntimeError

username: saluei
cooked: <p>I have simple model like this<br>
model input (3   32  32)<br>
Sequential(<br>
(conv1_A): adap_conv2d(3, 12, kernel_size=(11, 11), stride=(1, 1), padding=(5, 5), dilation=(False,False))<br>
(conv1_A_batch_norm): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(conv1_A_leaky): LeakyReLU(negative_slope=0.1, inplace=True)<br>
)<br>
without BatchNorm2d everything is OK , when I add BatchNorm2d I get the runtime error:<br>
RuntimeError: dilation should be greater than zero, but got [0, 0]<br>
I don’t use dilation anywhere why this error happen? and what that say???<br>
thank’s for any help</p>

username: chetan_patil
cooked: <p>The dilation of <code>(1,1)</code> means no dilation.<br>
Replace <code>(False,False)</code> to <code>(1,1)</code>  .</p>

username: saluei
cooked: <aside class="quote no-group" data-username="saluei" data-post="1" data-topic="86294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saluei/48/44594_2.png" class="avatar"> saluei:</div>
<blockquote>
<p>adap_conv2d</p>
</blockquote>
</aside>
<p>Thank you , problem solved<br>
in my inherited class from Conv2d (named adap_conv2d) parameter disorder in constructor caused this problem, your hint solved that</p>
2D max-pooling of complex-valued data separated in real-valued channels

username: fabigro
cooked: <p>Hi all,</p>
<p>I have complex-valued data where the real and imaginary components are separated into two real-valued channels. Is it possible to apply 2d max-pooling considering the absolute value of the complex value while keeping the channels separated in the output?</p>
<p>Thank you!</p>

username: chetan_patil
cooked: <p>Hi <a class="mention" href="/u/fabigro">@fabigro</a>,<br>
It would be easier if you give a working example.</p>
Hinge loss in PyTorch

username: blade
cooked: <p>I was wondering if there is an equivalent for <code>tf.compat.v1.losses.hinge_loss</code> in PyTorch? Is <code>torch.nn.HingeEmbeddingLoss</code> the equivalent function? Thanks!</p>
<p><strong>Edits:</strong><br>
I implemented the Hinge Loss function from the <a href="https://en.wikipedia.org/wiki/Hinge_loss" rel="nofollow noopener">definition</a> as below:</p>
<pre><code class="lang-auto">class HingeLoss(torch.nn.Module):

    def __init__(self):
        super(HingeLoss, self).__init__()
        self.relu = nn.ReLU()

    def forward(self, output, target):
        all_ones = torch.ones_like(target)
        labels = 2 * target - all_ones
        losses = all_ones - torch.mul(output.squeeze(1), labels)

        return torch.norm(self.relu(losses))
</code></pre>
<p>However, this gives a different value than <code>torch.nn.HingeEmbeddingLoss</code>, as I have explained in <a href="https://discuss.pytorch.org/t/developing-a-demo-classifier/86160">this example</a>.</p>
Combine two tensors of same dimension to get a final output tensor using trainable weights

username: Kushagra_Bhatia
cooked: <p>While working on a problem related to question-answering(MRC), I have implemented two different architectures that independently give two tensors (probability distribution over the tokens). Both the tensors are of dimension (batch_size,512). I wish to obtain the final output of the form (batch_size,512). How can I combine the two tensors using trainable weights and then train the model on the final prediction?</p>
<p>Additional Information:</p>
<p>So in the forward function of my NN model, I have used BERT model to encode the 512 tokens. These encodings are 768 dimensional. These are then passed to a nn.Linear(768,1) to output a tensor of shape (batch_size,512,1). Apart from this, I have another model built on top of the BERT encodings that also yields a tensor of shape (batch_size, 512, 1). I wish to combine these two tensors to finally get a tensor of shape (batch_size, 512, 1) which can be trained against the output logits of the same shape using CrossEntropyLoss. Also, since both the output tensors are first normalized (softmax) then combined, please suggest a way to compute the CrossEntropyLoss using the final output tensor.</p>
<p>Please share the PyTorch code snippet if possible.</p>
Hooks for functions?

username: kuonlp
cooked: <p>I’ve seen that the hook methods are only available for classes that inherit from nn.Module. Is there any easy/simple way to use/do something like the hooks but in functions such as torch.cat or torch.nn.functional.interpolate? I’m particularly interested in tracing the output of these functions.</p>

username: ptrblck
cooked: <p>Not that I’m aware of, but since you will get the output directly using the functional API, you could simple store this output instead of using hooks or wouldn’t it work for your use case?</p>

username: googlebot
cooked: <p>If you don’t use JIT, why not do simple monkey patching.</p>
<pre><code class="lang-auto">_cat = torch.cat
def cat(*a):
  return _cat(*a)
torch.cat = cat
</code></pre>
How to make two different network?

username: oasjd7
cooked: <pre><code class="lang-auto">net1 = resnet50().cuda()
net2 = resnet50().cuda()
</code></pre>
<p>I want to net1 and net2 have same architecture but different each other.<br>
input -&gt; net1 -&gt; output1<br>
input -&gt; net2 -&gt; output2<br>
output1 and output2 is exactly same…why net1 and net2 are the same?</p>

username: ebarsoum
cooked: <p>How resnet50 was constructed? Is this from TorchVision models?</p>
BCELoss from scratch

username: Bixqu
cooked: <p>How can I define BCEloss for intermediary layers of an autoencoder ? I can’t use the already existing function, since that one expects that one of the inputs be labels, not variables. Any ideas?</p>

username: smth
cooked: <p>just write it out with autograd ops, i.e. torch.* operations.</p>
<pre><code class="lang-python">def example_loss(x, t):
   loss = ((x - t)^^2).sum()
   return loss
</code></pre>

username: Bixqu
cooked: <p>Thanks but that’s MSE loss. That one is easy to implement. I was asking about cross entropy.</p>
<p>Would</p>
<pre><code>CE_Loss = - (y*torch.log(a) + (1-y)*torch.log(1-a)).sum().mean()
</code></pre>
<p>give the same thing?</p>

username: tom
cooked: <p>Hi <a class="mention" href="/u/bixqu">@Bixqu</a>,</p>
<p>in principle yes. Generally, it is recommended that you apply a small regularization for the logarithm.<br>
If you try:</p>
<pre><code class="lang-python">lossfunc = torch.nn.BCELoss()
for eps in [1e-11,1e-12,1e-13]:
    pred=Variable(torch.FloatTensor([1.0,0.0]))
    label=Variable(torch.FloatTensor([0.0,1.0]))
    ownloss=-(label*pred.clamp(min=eps).log()+(1-label)*(1-pred).clamp(min=eps).log()).mean()
    print("Reg: {} BCELoss: {:.4f} own BCELoss {:.4f}".format(eps, lossfunc(pred, label).data[0], ownloss.data[0]))
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">Reg: 1e-11 BCELoss: 27.6310 own BCELoss 25.3284
Reg: 1e-12 BCELoss: 27.6310 own BCELoss 27.6310
Reg: 1e-13 BCELoss: 27.6310 own BCELoss 29.9336
</code></pre>
<p>(If you are a friend of multilabel, you might sum over the labels and only take the mean over the batch or somesuch.)</p>
<p>Disclaimer: I don’t have any actual knowledge of the implementation of BCELoss.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Bixqu
cooked: <p>Thanks for the reply. I want to apply this as the loss function of an autoencoder so I think it should work.</p>

username: Filippo_Portera
cooked: <p>Hello everybody,<br>
do you know where I can find the exact implementation of the nn.BCELoss() of PyTorch?<br>
I’m trying to implement a binary classificator with a slightly modified loss.<br>
Thanks a lot and best regards,<br>
filippo portera</p>

username: harsha_g
cooked: <p>This one <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py#L448" rel="nofollow noopener">here</a>?</p>

username: Filippo_Portera
cooked: <p>Thanks but does it refer to F.binary_cross_entropy()?<br>
I couldn’t find the detailed implementation.<br>
The lowest level I reached is:</p>
<p>torch._C._nn.binary_cross_entropy()</p>

username: harsha_g
cooked: <p>The <code>forward</code> of <code>nn.BCELoss</code> directs to <code>F.binary_cross_entropy()</code> which further takes you to <code>torch._C._nn.binary_cross_entropy()</code> (the lowest you’ve reached).</p>

username: ptrblck
cooked: <p>You can find the CPU implementation of the forward method of <code>binary_cross_entropy</code> <a href="https://github.com/pytorch/pytorch/blob/ba98c0e38ca76a8e03d27e20a1d2bfc324d502ca/aten/src/ATen/native/Loss.cpp#L127-L161">here</a> (and the backward right below it).</p>
AttributeError: 'DataLoader' object has no attribute '_dataset_kind'

username: lpkoh
cooked: <p>Hi,</p>
<p>I am currently using Python 3.7, and installed with:<br>
python -m pip install torch==1.3.0 torchvision==0.4.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/torch_stable.html</a> --user</p>
<p>When i tried to run a github repo, I get the error:</p>
<h1>AttributeError: ‘DataLoader’ object has no attribute ‘_dataset_kind’</h1>
<p>This happens in dataloader.py:<br>
Specifically:</p>
<p>Traceback (most recent call last):<br>
File “run_classifier.py”, line 244, in <br>
main()<br>
File “run_classifier.py”, line 226, in main<br>
ypred, yprob, ystd = classify(model, train_data, args)<br>
File “run_classifier.py”, line 139, in classify<br>
for i, data in tqdm(enumerate(text), total=len(text)):<br>
File “C:\Users\Koh Ling Ping\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py”, line 278, in <strong>iter</strong><br>
return _MultiProcessingDataLoaderIter(self)<br>
File “C:\Users\Koh Ling Ping\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py”, line 636, in <strong>init</strong><br>
super(_MultiProcessingDataLoaderIter, self).<strong>init</strong>(loader)<br>
File “C:\Users\Koh Ling Ping\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py”, line 303, in <strong>init</strong><br>
self._dataset_kind = loader._dataset_kind<br>
AttributeError: ‘DataLoader’ object has no attribute ‘_dataset_kind’</p>
<p>I tried to switch torch versions 1.2.0 and 1.1.0, and torchvision versions as well. Can someone advise on this issue?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Can you verify that your dataloader properly calls <code>super().__init__()</code> during its initialization?<br>
It looks like some fields that are populated during the <code>DataLoader</code> class initialization are not set in your case.</p>

username: kaushil24
cooked: <p>Have you found a solution yet ? I ran into the same trouble while running <a href="https://github.com/NVIDIA/sentiment-discovery" rel="nofollow noopener">NVIDIA/sentiment-discovery</a></p>

username: harsha_g
cooked: <p><a class="mention" href="/u/kaushil24">@kaushil24</a> Did you try <a href="https://github.com/NVIDIA/sentiment-discovery/issues/63#issuecomment-549809161" rel="nofollow noopener">this</a>?</p>
Pytorch for loop inefficience

username: alex_gilabert
cooked: <p>Hello</p>
<p>I have the next code and im wondering how i could implement somehow the last loop. Here comes an explanation of the code</p>
<pre><code class="lang-auto">features_list = torch.empty( (0, 1000), dtype=torch.float ).cuda()
</code></pre>
<p>Firstly we create an empty tensor of 1000 colummns (output of VGG16 conv_layer5_3).</p>
<p>Batch size of the dataloader = 8</p>
<pre><code class="lang-auto">    counter = 0                                                     
    model.cuda()
    for i, data in enumerate(dataloader, 0):
        #Extraccion de Tensores
        input, label = data                                             
        input, label = input.to(device), label.to(device)              
        n,c ,h,w = input.size()                                         
        outputs = model(input)   
</code></pre>
<p>this outputs torch tensor has dims: torch.Size([8, 1000])</p>
<pre><code class="lang-auto">        if (i == 0):                                                    
            features_list = torch.cat( (features_list, outputs[0].view(1,-1)), 0)

</code></pre>
<p>For the very first iteration we add the first tensor that comes in the mini-batch.</p>
<pre><code class="lang-auto">        dist_tensores = torch.cdist(outputs, features_list, p=2.0) 
</code></pre>
<p>This distance dims depends on the iteration but firstly are:<br>
1º torch.Size([8, 1])<br>
2º torch.Size([8, 6])<br>
3º torch.Size([8, 11])<br>
…</p>
<pre><code class="lang-auto">AVG = 60
</code></pre>
<pre><code class="lang-auto">        activation = torch.gt(dist_tensores, AVG, out=torch.cuda.FloatTensor(len(outputs), len(features_list)))
</code></pre>
<p>Depends on the iteration, always [8, X] being X the number of tensors in  <code>feature_list</code></p>
<pre><code class="lang-auto">        counter = len(features_list)
        activation_list = torch.sum(activation, dim=0)

</code></pre>
<pre><code class="lang-auto">        for x in range(len(activation)):
          if (torch.sum(activation[x], dim=0) == counter):
            features_list = torch.cat( (features_list, outputs[x].view(1,-1)), 0)
</code></pre>
<p>If the add of all the positions in dim 0 equals the number of rows of  <code>feature_list</code>  we proceed to add the tensor via  <code>torch.cat</code> .</p>
<p>My issue is that i don<code>t know how to add the tensor  </code>outputs[x]`  if im not ussing a loop. I think there are “for loops” created for this or maybe its just a vectorial comparision.</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/replacing-a-for-loop-with-indexing/17730/8">here</a>.</p>
UnicodeDecodeError: 'gbk' codec can't decode byte 0xaa in position 120: illegal multibyte sequence

username: 111215
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a734dcfced85d5fe56787875958899b72694786e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a734dcfced85d5fe56787875958899b72694786e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a734dcfced85d5fe56787875958899b72694786e_2_690x371.png" alt="image" data-base62-sha1="nRaZMDabuCf9ROanXalB6hvcAyO" width="690" height="371" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a734dcfced85d5fe56787875958899b72694786e_2_690x371.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a734dcfced85d5fe56787875958899b72694786e_2_1035x556.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/7/a734dcfced85d5fe56787875958899b72694786e.png 2x" data-dominant-color="272A2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×736 86.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
An error occurred generating the corresponding 2007_train.txt: UnicodeDecodeError: ‘GBK’ codec can’t decode byte 0xAA in position 120: illegal multibyte sequence</p>
<p>The 2007_train.txt file should generate the location parameters and labels of the target. What I generated was an empty file.</p>

username: ptrblck
cooked: <p>This error might be raised, if unknown characters for the current encoding are found and changing the encoding might solve the issue as described <a href="https://stackoverflow.com/questions/40997603/unicodedecodeerror-gbk-codec-cant-decode-byte-when-read-json-contains-chines">here</a>.</p>
<p>I’m not sure, if this error is raised by Keras or TensorFlow, which you are apparently using, but if so you might get a better answer on StackOverflow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Dropout while testing

username: NeelayS
cooked: <p>Hey everyone !<br>
I was wondering if there was a way I could introduce dropout into a network while using it for inference.<br>
I do not have access to the network definition which may or may not have a dropout layer.</p>

username: ptrblck
cooked: <p>How are you loading the model, if you don’t have the model definition?<br>
Even if the model was completely stored (instead of its <code>state_dict</code>), you would have to have the source files, no?<br>
If you are using a scripted model, then you should at least be able to see all modules.</p>
<p>Anyway, you if you don’t have access to the model and cannot check its layers, you could call <code>model.train()</code> before using the test method. This would however also set all batchnorm layers to trianing mode, which is usually wrong.</p>
<p>If you have access to the modules, you could call <code>train()</code> on the dropout layers only.</p>
Rotated MNIST for regression

username: ASJ
cooked: <p>Hi everyone,</p>
<p>I’d like to create a (random) rotated version of MNIST for regression, so that the target is the angle. Any ideas?</p>
<p>I have tried subclassing torchvision.datasets.MNIST and overriding getitem, but a new rotation would be applied every time I access the data.</p>

username: ptrblck
cooked: <p>Your approach of deriving from <code>MNIST</code> and applying the rotation in the <code>__getitem__</code> method seems to be valid.<br>
You could store the angle values in a <code>list</code> and apply the corresponding angle in <code>__getitem__</code>.<br>
This would make sure that a particular sample for an <code>index</code> gets the same rotation using the <code>index</code>.</p>

username: ASJ
cooked: <p>Thanks for your reply.</p>
<p>Yeah, storing the angles in a list would do the trick. I’m afraid this solution (overriding <code>__getitem__</code>) is not efficient though.</p>
<p>Do you have ideas of how I could rotate the input and save the corresponding target beforehand?<br>
So that I could combine this pre-processing with other transforms.</p>
<p>For context, this is my current piece of code:</p>
<pre><code class="lang-auto">class RotatedMNIST(MNIST):
  def __getitem__(self, i):
    input, target = super(RotatedMNIST, self).__getitem__(i)
    target = random.normalvariate(0, 3.14/4)
    input = TF.rotate(input, target * (180 / 3.14))
    input = transforms.ToTensor()(input)
    return input, target

trainloader = DataLoader(RotatedMNIST('./data', train=True, download=True), batch_size=batch_size, shuffle=True)
</code></pre>

username: ptrblck
cooked: <p>Do you see a significant slowdown using this approach or why do you think it’s not efficient?<br>
You could of course rotate all images beforehand and store these rotated tensors.<br>
Then in your custom <code>Dataset</code>, you could load all tensors and preprocess them with the other augmentation methods.<br>
I’m not sure how much time this would save.</p>
How to constrict each column of weight matrix sum to 1

username: you_shin
cooked: <p>I have a linear layer which is nn.Linear(2000,50,bias=False). The weight matrix is 50x2000. I want to constrict each column of the weight matrix sum to 1. Specifically, w[0][0] + w[1][0] + … + w[49][0] = 1. How can I do this in pytorch ?</p>

username: harsha_g
cooked: <p>Did you try this?<br>
<a href="https://discuss.pytorch.org/t/contraining-weights-to-sum-to-1/20609/2">https://discuss.pytorch.org/t/contraining-weights-to-sum-to-1/20609/2</a></p>

username: you_shin
cooked: <p>Thank you. At first, I did not understand the answer in it. But now I get it.</p>