akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
D Loss goes to zero after 4/5 epoch

username: banikr
cooked: <p>I am using image to image synthesis and using GAN architecture. For discriminator network I am calculating real and fake loss individually and backpropagating them. Then I am evaluating the sum of the real and fake loss to check the network performance. But after 4/5 epochs the LossD goes to zero which is according to this <a href="https://github.com/soumith/ganhacks" class="inline-onebox" rel="noopener nofollow ugc">GitHub - soumith/ganhacks: starter from "How to Train a GAN?" at NIPS2016</a> is a symptom of early failure detection. As he says to check the norm of the gradients how do I do that?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0db5f02e37aad1a61efb34a8b88c5eccadd2f223.png" data-download-href="https://discuss.pytorch.org/uploads/default/0db5f02e37aad1a61efb34a8b88c5eccadd2f223" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0db5f02e37aad1a61efb34a8b88c5eccadd2f223.png" alt="image" data-base62-sha1="1Xi0usdIZpiZoxWnC9gar4dR6aT" width="690" height="150" data-dominant-color="3C3C3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1700×371 43.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch.cat in backpropagate

username: Ali_Mirzaeyan
cooked: <p>Hi<br>
I was testing how torch.cat will affect the backpropagated loss. I write this piece of code but I’m not sure why no gradient will reach the inputs. any clue what might be wrong ?</p>
<pre><code class="lang-auto">x = Variable(torch.tensor([1.]), requires_grad=True)
y = Variable(torch.tensor([1.]), requires_grad=True)
target = Variable(torch.tensor([3., 3.]))
z = torch.cat([x,y])
loss = Variable( torch.mean(z - target), requires_grad = True)
loss.backward()

</code></pre>
<p>now I’m checking the gradient of each one of the variables x, and y but their Values are None !! what is going on</p>
<pre><code class="lang-auto">x.grad == None
True
y.grad == None
True
</code></pre>

username: Ali_Mirzaeyan
cooked: <aside class="quote no-group quote-modified" data-username="Ali_Mirzaeyan" data-post="1" data-topic="46572">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_mirzaeyan/48/6565_2.png" class="avatar"> Ali_Mirzaeyan:</div>
<blockquote>
<p>I added retain_grad() after torch.cat, as explained here <a href="https://stackoverflow.com/questions/55942423/pytorch-backpropagating-from-sum-of-matrix-elements-to-leaf-variable" class="inline-onebox" rel="noopener nofollow ugc">python - Pytorch: backpropagating from sum of matrix elements to leaf variable - Stack Overflow</a>,  and apparently it works now.</p>
</blockquote>
</aside>
<pre><code class="lang-auto">x = Variable(torch.tensor([1.]), requires_grad=True)
y = Variable(torch.tensor([1.]), requires_grad=True)
target = Variable(torch.tensor([3., 3.]))
z = torch.cat([x,y])
z.retain_grad()
loss = Variable( torch.mean(z - target), requires_grad = True)
loss.backward()
</code></pre>
Reproducibility and performance

username: BramVanroy
cooked: <p>The <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">documentation</a> states:</p>
<blockquote>
<p>Deterministic mode can have a performance impact, depending on your model.</p>
</blockquote>
<p>My question is, what is meant by performance here. Processing speed or model quality (i.e. minimal loss)?</p>

username: ptrblck
cooked: <p>Performance relates to the processing speed in this case. E.g. if you disable <code>cudnn.benchmark</code>, your execution might get slower (depending on the model).</p>

username: BramVanroy
cooked: <p>Thank you for the confirmation. I feel that this may be clarified in the documentation, but leaving it answered here is fine too.</p>

username: ptrblck
cooked: <p>It might be confusing others too, so feel free to create a GitHub issue so that it can be discussed (and fixed) there, too. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Binary cross entropy error

username: Rojin
cooked: <p>I am implementing a variational autoencoder. The input has both positive and negative numbers and it is NOT between 0 and 1. I did a log2 transform before training the model. I was wondering if binary_cross_entropy is a good choice in this case? because after the first iteration it gives me the following error:</p>
<pre><code class="lang-auto">RuntimeError: Assertion `x &gt;= 0. &amp;&amp; x &lt;= 1.' failed. input value should be between 0~1, but got nan at /Users/distiller/project/conda/conda-bld/pytorch_1556653492823/work/aten/src/THNN/generic/BCECriterion.c:60
</code></pre>

username: ptrblck
cooked: <p>I’m not sure if your transformation returns output in <code>[0, 1]</code>, but the error points to a NaN value.<br>
Could you try to check, why the output contains a NaN?</p>

username: Rojin
cooked: <p>Thanks for the help!</p>
<p>So I did some modifications to the code and now I’m not getting the error anymore.</p>

username: Rojin
cooked: <p>So I was able to get rid of the error by adding this to the decoder sigmoid output:</p>
<pre><code class="lang-auto">torch.clamp(F.sigmoid(self.fc6(h)), 0, 1)

</code></pre>
<p>But still, I wanted to see if that is the right thing to do?</p>

username: ptrblck
cooked: <p>That’s strange, since <code>F.sigmoid</code> should only yield values in the range <code>[0, 1]</code>, so that the <code>clamp</code> operation shouldn’t change anything.<br>
Are you sure the <code>clamp</code> fixed this issue?</p>

username: Rojin
cooked: <p>So I noticed that the problem was because I wan not using sigmoid at the end, and you are right it is good now without needing clamp <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
V100 is too slow for training

username: otutay
cooked: <p>I have some issues with v100 GPU. A tiny-yolov2 classification example takes around 25 seconds for one mini-batch(batch size 50). It consumes around 2.5 GB of ram and GPU utilization hovering around 0.<br>
I’m using windows server 12 with the v100 and all of the installation is done via anaconda environment.<br>
Python = 3.7<br>
pytorch = 1.0</p>
<p>Same code runs in 2-2.2 second on one p2000 GPU and on a 1080 1.8 seconds. Both p2000 and 1080 are on a ubuntu 16.04 machine with anaconda, pytorch 1.0, python 3.7.</p>
<p>one weird issue is that in windows machine nvidia-smi does not show the Python Processes even if the GPU ram is fully utilized by increasing the batch size.</p>
<p>this is somewhat similar with the</p><aside class="quote" data-post="1" data-topic="11011">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/antspy/48/3475_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/slow-pytorch-on-v100/11011">Slow pytorch on V100</a> 
  </div>
  <blockquote>
    I have installed pytorch on a Tesla V100. The problem is that it is way too slow; after 16 hours, not even one epoch of imagenet was completed! I believe this should be much faster in normal circumstances. 
I have cuda9 installed and the driver version is NVIDIA-SMI 384.98, which I think it is good enough. I have also the most NCLL2 and the most recent version of cuDNN. 
What else should I check to make sure the V100 is configured properly? And how to fix the problem? 
Thank you!
  </blockquote>
</aside>

<p>want to open a new post to get some attention.<br>
thanks for your help</p>

username: otutay
cooked: <p>The issue is related with the BIOS options. It is solved right now.</p>

username: Keshav_Kolluru
cooked: <p>Could you detail the BIOS issue you have faced? I am also having a similar issue of very slow training times on V100 and have reinstalled the entire software stack from the OS to CudNN but to no effect. I would like to know what I can check in the BIOS to see if everything is properly set.</p>

username: otutay
cooked: <p>In my case, there is a bios option about performance mode of the system. I selected as high performance mode. I think, it arranges pcie bus.  I dont remember the exact name.</p>
Loss function that penalizes uncommon values (frequency-based)

username: trougnouf
cooked: <p>I would like to make a loss function that assigns a loss based on the frequency each item appears.<br>
For example if it gets a tensor [1,6,1,2,4,9,4,4] where 1 appears twice and 4 three times, the loss could be (1/2 + 1/1 + 1/2 + 1/1 + 1/3 + 1/1 + 1/3 + 1/3) / 8 = 0,625 , whereas [1,2,3,4,5,6,7,8] would have a loss of 1.<br>
I have been struggling with this idea and how it could be implemented in a way that works with backpropagation. Does it seem feasible and what should I look into?</p>
Mutual information as custom loss function

username: Morpheus_Hsieh
cooked: <p>Hi all, I would like to use mutual information as a custom loss function. Is there any built-in functions that I can use to achieve that? Thanks!</p>

username: smth
cooked: <p>there isn’t an inbuilt loss, but you can simply create a loss function that takes <code>Variable</code>s x and y and computes the mutual information. The backward is automatically computed using autograd.</p>
<pre><code class="lang-auto">def mutual_info_loss(x, y):
   # implement it using torch.* operations, etc., such as torch.log(x)
   return loss

# later...
loss.backward()
</code></pre>

username: Josiane_Rodrigues
cooked: <p>Hi <a class="mention" href="/u/morpheus_hsieh">@Morpheus_Hsieh</a>,</p>
<p>I need to use mutual information as a loss function as well, but I’m having difficulty. Were you able to implement?</p>

username: Morpheus_Hsieh
cooked: <p>Hi, sorry I can’t help you there. It’s been a long time and I can’t find the code. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=6" title=":sweat:" class="emoji" alt=":sweat:"></p>

username: GM_039
cooked: <p>Hi smth,</p>
<p>I am facing issue in implementing mutual_info_loss using torch function. Can you please help me in implementing it. Any leads will be appreciated.<br>
Thank you</p>
Add extra layer to vGG

username: SHASHANK_KUMAR_MISHR
cooked: <p>How to add exra two layer to existing VGG?Please provide the code sample.</p>

username: sai_tharun
cooked: <p>i didnt get you,give us an example</p>

username: SHASHANK_KUMAR_MISHR
cooked: <p><a class="mention" href="/u/sai_tharun">@sai_tharun</a> In trained Vgg contains 1000 classes but if I want to restrict it to 10 classes how to do that?</p>

username: sai_tharun
cooked: <p>vgg.classifier[6] = nn.Linear(in_features, 10)</p>
Where is the source code for Batchnorm2d?

username: Hotaekhan
cooked: <p>Hello,</p>
<p>recently, I tried to implement the batchnorm 2d layer with the scale layer.<br>
the scale layer is doing like below<br>
y = (scale * x + shift)^power</p>
<p>The batchnorm2d layer consists of two consecutive scale layers as I know.</p>
<ol>
<li>
<p>nomalization(zero mean and unit variance)<br>
hat(x) = (x-E[x]) / sqrt(Var[X] + eps)</p>
</li>
<li>
<p>affine transformation.<br>
y = gamma * hat(x) + beta</p>
</li>
</ol>
<p>I found the batchnorm2d has four parameters. running_mean, running_var, weight, bias.</p>
<p>So…I implemented like below</p>
<ol>
<li>
<p>the first layer do normalization<br>
scale = 1 / sqrt(running_var+eps)<br>
shift = running_mean * -1.0<br>
power = 1.0</p>
</li>
<li>
<p>and second layer do the affine transformation<br>
scale = weights<br>
shift = bias<br>
power = 1.0</p>
</li>
</ol>
<p>I think it looks well but I got the wrong values from my implementation.<br>
So, I want to know how to operate the batchnorm2d in evaluation mode.<br>
Where can I find?</p>
GPU (cuda) support under WSL

username: meeotch
cooked: <p>Is it the case that using pytorch with Cuda support under WSL (Windows Subsystem for Linux) is currently impossible, as suggested by someone near the bottom of this thread?   <a href="https://github.com/pytorch/pytorch/issues/6098" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/6098</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>As pointed out in the thread, the problem is that the WSL itself does not support GPU (issue tracked <a href="https://github.com/Microsoft/WSL/issues/829">here</a>) so there is nothing we can do until this is done.</p>

username: erogol
cooked: <p>There I share a hacky way to enable CUDA with WSL <a href="http://www.erogol.com/using-windows-wsl-for-deep-learning-development/" rel="nofollow noopener">http://www.erogol.com/using-windows-wsl-for-deep-learning-development/</a></p>

username: albanD
cooked: <p>Is the gist to use a windows python install from within the WSL?</p>
Backpropagating through scatter add

username: stefdoerr
cooked: <p>Hey, I am trying to sum values in a Tensor based on the index of the output array.<br>
Here is an example</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
import numpy as np

data = Variable(torch.from_numpy(np.array([3, 2, 1, 8, 7], dtype=np.float32)), requires_grad=True)
idx = Variable(torch.from_numpy(np.array([0, 0, 1, 1, 2], dtype=np.int64)), requires_grad=False)
</code></pre>
<p>So I would like to obtain a <code>Variable</code> with values <code>[5, 9, 7]</code> which I can backpropagate through.</p>
<p>I tried doing it with</p>
<pre><code class="lang-auto">out = Variable(torch.zeros(3), requires_grad=True)
out.scatter_add_(0, idx, data)
</code></pre>
<p>but this won’t work as I am doing an in-place operation on a leaf <code>Variable</code>.</p>
<p>What is the correct way of doing this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>A simple fix is just to <code>clone</code> out before performing the scatter on it.</p>

username: stefdoerr
cooked: <p>Ok that worked! But I don’t get the logic behind it.</p>

username: albanD
cooked: <p>A “leaf Variable” is a Variable created by the user for which you want the gradients.<br>
The thing is that the original value could be needed by other stuff when performing the backward pass without pytorch knowing about it and so some computed gradients would be wrong.</p>

username: stefdoerr
cooked: <p>Nevermind. This works</p>
<pre><code class="lang-auto">out = Variable(torch.zeros(3), requires_grad=True).clone()
out = out.scatter_add_(0, idx, data)
out = out.scatter_add_(0, idx, data)
</code></pre>

username: albanD
cooked: <p>Also you can use the non-inplace version:<br>
<code>out3 = out.scatter_add(0, idx, data).scatter_add(0, idx2, data2)</code></p>

username: stefdoerr
cooked: <p>Oops, sorry I deleted my previous post and cannot restore it. Thanks in any case!</p>

username: Jeeyung_Kim
cooked: <p>Hi,</p>
<p>If I want to use some part of values in <code>out</code> by using scatter, for example, I would choose 1,2,3 index of <code>out</code>, is it possible to use backpropagate?</p>
<p>Thank you in advance</p>

username: albanD
cooked: <p>The scatter/gather ops are differentiable, so it will work</p>
How to load a model from pytorch to caffe2 with c++?

username: Hotaekhan
cooked: <p>Hello,</p>
<p>I have tried to load the trained model from the pytorch into caffe2 or TensorRT.<br>
I tried…</p>
<ol>
<li>
<p>First, export onnx format then read it in caffe2.<br>
-&gt; It is ok for python. But I can’t figure out how to do it for c++. I meant I can’t find caffe2 api for onnx backend.</p>
</li>
<li>
<p>First, export onnx format then read it in TensorRT.<br>
-&gt; My model includes CNN and LSTM layer like this (CNN-Pool-CNN-Pool-Reshape-LSTM).<br>
-&gt; The input shape of pytorch’s LSTM is (seq x batch x feat). So I reshaped the tensor from Pool layer from (batch x channel x 1 x width) to (seq x batch x feat).<br>
-&gt; But, TensorRT doesn’t support moving the batch dim.</p>
</li>
</ol>
<p>I think case 2) is related in TensorRT. But case 1) is related in pytorch/caffe2.<br>
Any helps?</p>
<p>Thanks</p>

username: alex.veuthey
cooked: <p>I don’t know about TensorRT, but for Caffe2, you can save the Caffe2 model in predict and init files, like shown in <a href="https://github.com/onnx/tutorials/blob/master/tutorials/OnnxCaffe2Import.ipynb" rel="nofollow noopener">this tutorial</a>. Then I guess you can import them in the C++ API? Haven’t tried it but that seems like the way to go…</p>

username: Hotaekhan
cooked: <p>Thanks for your reply.<br>
I read that tutorial. And…yes I tried to same thing in the C++.<br>
But I can’t find any sample code or tutorial in the C++ for caffe2.<br>
I have found the <a href="https://caffe2.ai/doxygen-c/html/classes.html" rel="nofollow noopener">C++ api list for caffe2</a>. but…looks little messy…<img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>oh, I found <a href="https://github.com/leonardvandriel/caffe2_cpp_tutorial" rel="nofollow noopener">the tutorial</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
I get the fixed output in the test phase when I set model.eval()

username: shengyudingli
cooked: <p>I use nn.BatchNorm2d in my model. When I test my model and set model.eval()  (which will just affect BN and Dropout layer), I get the approximately fixed outputs, which means I feed different inputs to the network, but the outputs are the same. I don’t know why, I sincerely hope the someone can help me solving this confusion.</p>

username: ptrblck
cooked: <p>Maybe your model might just have learned the “mean prediction value” and thus would need some more training.<br>
Could you check the bias values of the output layer and compare them to the output values?<br>
I’ve seen some cases where my model get stuck during training and just predicted the mean of all target values (facial key points).</p>

username: shengyudingli
cooked: <p>Thank you very much, I will check it. By the way, if I remove this statement (model.eval() ), I can get the right result. Do you think I have to set model.eval() in the test phase?</p>

username: ptrblck
cooked: <p>You would have to call <code>model.eval()</code> to change the behavior of some layers like <code>nn.BatchNorm</code> and <code>nn.Dropout</code>. However, if your validation works better in train mode, this might be a sign that the running estimates of your batchnorm layers are not really useful. Which batch size are you using during training and evaluation? Could you try to change the <code>momentum</code> of the batchnorm layers or remove them completely?</p>

username: shengyudingli
cooked: <p>The batch sizes are 17 in both train and evaluation phase, Do you think <code>nn.BatchNorm</code> should works on big training data?</p>

username: ptrblck
cooked: <p>Depending on the stats of your dataset, you might also try to use <code>nn.GroupNorm</code> which should perform better on smaller batches.</p>

username: dawn
cooked: <p>I have the same issuie, and my model is bert, which is just linear layer, droupout layer and layernorm layer, the result is always the same when model is eval mode, how can I debug this？</p>
torch.Tensor with uncertain length of one dimension

username: whisht
cooked: <p>I am programming a custom optimizer, and the legth of first dimension is  not certain because last batch have no enough data to bulid a batch. So the initialization of weights with fixed batch_size do not satisfy the last batch for <code>torch.matmul</code>.<br>
So, I wonder whether I can create a tensor where  the lenght of some dimension could be variable, like variable length list.</p>
Trouble using an LSTM in a seq2seq model

username: raconma
cooked: <p>I’m doing a project starting from the seq2seq translation tutorial and I was trying to change the GRU rnn to an LSTM one but I’m having some problems with the dimension in the Encoder. I have tryied a lot of changes but I’m kind of new to this and I don’t know where I’m messing up.<br>
Encoder:</p>
<pre><code class="lang-auto">class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)

        self.lstm = nn.LSTM(hidden_size, int(hidden_size), num_layers=1, batch_first=True, bidirectional=True)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        output, hidden = self.lstm(output, hidden)
        return output, hidden

    def initHidden(self):
        return (torch.zeros(1, 1, self.hidden_size, device=device),torch.zeros(1, 1, self.hidden_size, device=device))
</code></pre>
<hr>
<p>And I’m getting the following error:</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
4 attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words, dropout_p=0.1).to(device)<br>
5<br>
----&gt; 6 trainIters(encoder1, attn_decoder1, 75000, print_every=5000)</p>
<p>9 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_hidden_size(self, hx, expected_hidden_size, msg)<br>
170         # type: (Tensor, Tuple[int, int, int], str) → None<br>
171         if hx.size() != expected_hidden_size:<br>
 → 172             raise RuntimeError(msg.format(expected_hidden_size, tuple(hx.size())))<br>
173<br>
174     def check_forward_args(self, input, hidden, batch_sizes):</p>
<p>RuntimeError: Expected hidden[0] size (2, 1, 256), got (1, 1, 256)</p>
</blockquote>
<p>Hope you can help me.</p>
Implementatoin of A Tensor Indexing Operation

username: JaydenZoZ
cooked: <p>Thinking an operation from the paper <a href="https://arxiv.org/pdf/1903.12174.pdf" rel="nofollow noopener">TensorMask: A Foundation for Dense Object Segmentation</a> for days.</p>
<p>Given 4-d Tensor of shape (V, U, Y, X), how to implement</p>
<p><code>F(v, u, y, x) = F(v, u, y+v', x+u')</code></p>
<p>Where <code>v' = v - int(v / 2), u' = u - int(u / 2)</code> and <code>v, u, y, x</code> are normal indexes.</p>
<p>Thanks in advance.</p>
How to load partial optimizer checkpoint

username: AIJoris
cooked: <p>Hi,</p>
<p>I am trying fine-tune a model with an additional module compared to the pre-trained model (similar to <a href="https://discuss.pytorch.org/t/load-optimizer-for-partial-parameters/2617">this</a> post). Loading the model state dict works fine using the strict=False option. However, for the optimizer I get the following error:</p>
<p><code>ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group</code></p>
<p>How do I load a state from an optimizer for which the new model has an additional module?</p>
<p>Thanks!</p>
Float precision of torch

username: singleroc
cooked: <p>Hello, Dear Forumers,</p>
<p>I met a precision problem in pytorch when add an EPS to a float number. Does anyone know why the following example give different (comparasion, minus) result.</p>
<blockquote>
<blockquote>
<blockquote>
<p>a = torch.ones(1, dtype=torch.float) * 2304<br>
b = a + 0.0001<br>
a, b, a==b, a - b<br>
(tensor([2304.]), tensor([2304.]), tensor([1], dtype=torch.uint8), tensor([0.]))</p>
<p>a = torch.ones(1, dtype=torch.float) * 100<br>
b = a + 0.0001<br>
a, b, a==b, a - b<br>
(tensor([100.]), tensor([100.0000991821]), tensor([0], dtype=torch.uint8), tensor([-9.9182128906e-05]))</p>
</blockquote>
</blockquote>
</blockquote>

username: MariosOreo
cooked: <p>Hello,</p>
<p>I found that:</p>
<pre><code class="lang-auto">a = torch.ones(1, dtype=torch.float) * 2047
a + 0.0001
&gt;&gt; tensor([2047.0001])
</code></pre>
<p>It seems that <strong>2048</strong> is a boundary, I think it must related to the floating point expression in binary format, it clips the significand part that causes the precision loss (if I am wrong, correct me please).<br>
<strong>But</strong> I cannot get the accurate idea. Does anyone have accurate explanations?</p>

username: singleroc
cooked: <p>Hi, thanks for your reply. Numpy seems not to have such an issue.</p>
<blockquote>
<blockquote>
<blockquote>
<p>a = np.ones(1, dtype=np.float32)<br>
a.dtype<br>
dtype(‘float32’)<br>
a = 2048<br>
a + 0.0001<br>
2048.0001<br>
a + 0.00001<br>
2048.00001</p>
</blockquote>
</blockquote>
</blockquote>

username: tom
cooked: <p><code>0.0001</code> isn’t a multiple of a negative power of two, so it cannot be represented exactly. There is a <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format" rel="nofollow noopener">detailed discussion of precision in the Wikipedia article</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
_nms.so undefined symbol __cudapopcallconfiguration

username: UsamaMaq
cooked: <p>Hello<br>
I am trying to use <a href="https://github.com/yhenon/pytorch-retinanet" rel="nofollow noopener">this</a> repo. But i am getting this error<br>
<code>File "/home/maq/PycharmProjects/pytorch-retinanet/visualize.py", line 106, in &lt;module&gt; main(coco_path, path_to_model) File "/home/maq/PycharmProjects/pytorch-retinanet/visualize.py", line 53, in main retinanet = torch.load(path_to_model) File "/home/maq/PycharmProjects/pytorch4python2/pytorch-retinanet/local/lib/python2.7/site-packages/torch/serialization.py", line 358, in load return _load(f, map_location, pickle_module) File "/home/maq/PycharmProjects/pytorch4python2/pytorch-retinanet/local/lib/python2.7/site-packages/torch/serialization.py", line 542, in _load result = unpickler.load() File "/home/maq/PycharmProjects/pytorch-retinanet/model.py", line 9, in &lt;module&gt; from lib.nms.pth_nms import pth_nms File "/home/maq/PycharmProjects/pytorch-retinanet/lib/nms/pth_nms.py", line 2, in &lt;module&gt; from ._ext import nms File "/home/maq/PycharmProjects/pytorch-retinanet/lib/nms/_ext/nms/__init__.py", line 3, in &lt;module&gt; from ._nms import lib as _lib, ffi as _ffi ImportError: /home/maq/PycharmProjects/pytorch-retinanet/lib/nms/_ext/nms/_nms.so: undefined symbol: __cudaPopCallConfiguration</code></p>
<p>I tried to troubleshoot it using this <a href="https://rusty1s.github.io/pytorch_geometric/build/html/notes/installation.html" rel="nofollow noopener">link</a>. All the paths added. In FAQ point 7 of link</p>
<p>Result in python environment in which repo is being compiled:<br>
$ python -c “import torch; print(torch.version.cuda)”<br>
9.0.176</p>
<p>$nvcc --version<br>
nvcc: NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2017 NVIDIA Corporation<br>
Built on Fri_Sep__1_21:08:03_CDT_2017<br>
Cuda compilation tools, release 9.0, V9.0.176</p>
<p>$  python -c “import torch; print(torch.<strong>version</strong>)”<br>
0.4.1</p>
<p>$python -c “import torch; print(torch.cuda.is_available())”<br>
True</p>
<p>I am using ubuntu 16.04.</p>
<p>PS: Outside environment in terminal. Result of all the commands are same except<br>
$  python -c “import torch; print(torch.<strong>version</strong>)”<br>
which result 1.0.1.post2. But i am running repo in environment i dont think this should matter.</p>
Scheduling batch size in DataLoader

username: dpernes
cooked: <p>I have a sequence classification problem that I am trying to solve using some kind of “curriculum learning”, which I describe in the following. My dataset consists of sequences and (in the particular problem I am trying to solve) the larger the sequences are, the harder they are to classify correctly.</p>
<p>Therefore, I am scheduling the sequence length in the dataset so that sequences are initially small and their length increases progressively during training. Each mini-batch obtained from the DataLoader has the shape <code>(N, L_i, D)</code>, where <code>N</code> is the batch size, <code>L_i</code> is the sequence length at the <code>i</code>-th epoch and <code>D</code> is the dimension of the data.</p>
<p>In order to use the GPU memory efficiently, I would like to use a variable batch size, which is initially larger and decreases along the training epochs, to compensate for the extra memory required by the larger sequences. I thought it would be enough to do the following immediately before each epoch starts:</p>
<p><code>train_loader.batch_size = N_i</code>,</p>
<p>where <code>N_i</code> would be the desired batch size for the <code>i</code>-th epoch. However, apparently, the DataLoader does not allow the batch size to be changed after initialization, since the following error appears:</p>
<p><code>ValueError: batch_size attribute should not be set after DataLoader is initialized</code></p>
<p>Is there any workaround for this limitation?</p>

username: ptrblck
cooked: <p>Re-initializing the <code>DataLoader</code> after each epoch might be a valid alternative, as it should be quite cheap and therefore shouldn’t impact the performance.<br>
Would that work for you or would that mess with your curriculum learning schedule?</p>

username: dpernes
cooked: <p>Yes, that does the job, thank you! I thought it would be a bit more expensive than it actually is <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
TorchVision.utils.save_images() ValueError: unknown file extension

username: rim_zayani
cooked: <p>Hello,<br>
I’m trying to save the output image of my autoencoder.<br>
Here is some of my code:</p>
<pre><code class="lang-auto">def train(train_dl, inp, out, model):   
    optimizer=torch.optim.Adam(model.parameters(), lr=args.learning_rate)
    model.train()
    train_loss=0.0
    for idx, (data, label) in enumerate(train_dl):
        data, label= Variable(data), Variable(label)
        out, mu, var=model(data)
        loss=loss_fct(data, out, mu, var)
        train_loss+=loss.data[0]
        model.zero_grad()
        loss.backward()
        optimizer.step()
    save_image(data[0], inp)
    save_image(out[0], out)
    av_loss= train_loss / len(train_dl.dataset)     
    return mu, var, av_loss
</code></pre>
<pre><code class="lang-auto">if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Autoencoder')
    parser.add_argument('-oimg','--output-img', type=str, default='output12.png', metavar='',
                    help='output image (default: output12.png)')
    parser.add_argument('-iimg','--input-img', type=str, default='input12.png', metavar='',
                    help='input image (default: input12.png)')
    parser.add_argument('-outp','--output', type=str, default='out.txt', metavar='',
                    help='output file (default: out.txt)')
    args = parser.parse_args()
</code></pre>
<pre><code class="lang-auto">transform1=transforms.Compose([transforms.ToTensor()])
train1=ImageFolder("/home/uz/zaianir/scratch/DB/sentinel_data1", transform1)
train_dl=torch.utils.data.DataLoader(train1, batch_size=args.batch_size, shuffle=True)
model=VAE() 
mu, var, loss= train(train_dl, args.input_img, args.output_img, model)
</code></pre>
<p>But I get this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/uz/zaianir/scratch/vae_sentinel1.py", line 112, in &lt;module&gt;
    mu, var, loss= train(train_dl, args.input_img, args.output_img, model)
  File "/home/uz/zaianir/scratch/vae_sentinel1.py", line 84, in train
    save_image(out[0], out)
  File "/home/uz/zaianir/scratch/.conda/envs/myenv/lib/python3.6/site-packages/torchvision/utils.py", line 104, in save_image
    im.save(filename)
  File "/home/uz/zaianir/scratch/.conda/envs/myenv/lib/python3.6/site-packages/PIL/Image.py", line 1989, in save
    raise ValueError('unknown file extension: {}'.format(ext))
ValueError: unknown file extension:
</code></pre>
<p>Can you please help me?<br>
Thank you very much.</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/torchvision/utils.html#torchvision.utils.save_image" rel="nofollow noopener">save_image</a> expects a <code>filename</code> as the second argument, while it seems you are passing the tensor again to it.</p>
LSTM accuracy doesn't increase

username: cane95
cooked: <p>Hello,</p>
<p>I’m implementing RNN for video classification, but the accuracy doesn’t increase. I use pretrained resnet-50 as features extractor to feed LSTM. In particular I’m worried that I’m messing up the forward step of the network, here is my code</p>
<pre><code class="lang-auto">    def forward(self, x, seq):
        state = self._init_state(b_size=len(seq))

        y = []
        for i in range(len(seq)):
            y.append(self.resnet(x[i]))

        y = torch.nn.utils.rnn.pad_sequence(y)
        pack = torch.nn.utils.rnn.pack_padded_sequence(y, seq, batch_first=False)
        z, _ = self.lstm(pack, state)
        z = nn.utils.rnn.pad_packed_sequence(z, batch_first=False)

        t = []
        for i in range(len(seq)):
            t.append( z[0][seq[i]-1,i,:] )

        t = torch.stack(t,0)
        out = self.classifier(t)
        out = self.out(out)

        return out
</code></pre>
<p>I attach also the modified collate_fn function of the dataloader</p>
<pre><code class="lang-auto">def my_collate(batch):
    frames, l = zip(*batch)
    lengths = []
    for i in range(len(frames)):
        lengths.append(frames[i].shape[0])

    perm_idx = sorted(range(len(lengths)), key=lengths.__getitem__, reverse=True)

    frames_out = [frames[i] for i in perm_idx]
    l_out = [l[i] for i in perm_idx]
    lengths_out = [lengths[i] for i in perm_idx]


    return frames_out, torch.LongTensor(l_out), lengths_out
</code></pre>
<p>Do you think there is a bug in my implementation? Thanks</p>
Calculate batched jvp (R-op)

username: alexander_langley
cooked: <p>One user posted his code for an R-op <a href="https://discuss.pytorch.org/t/adding-functionality-hessian-and-fisher-information-vector-products/23295">here</a>, but with no explanation of how to use it. Using the approach described in <a href="https://j-towns.github.io/2017/06/12/A-new-trick.html" rel="nofollow noopener">this blog</a> I can calculate a Jacobian-vector product for a single vector. What I’d like to be able to do is to calculate it for an entire batch of vectors (some matrix). Here is the code I’m using for the single vector case:</p>
<pre><code class="lang-auto">mlp = MLP()
batch_size = 1
x = torch.randn(batch_size, 25)
v = torch.randn(batch_size, mlp.n_params())
y = mlp(x)
u = torch.ones_like(y, requires_grad=True)

vjp = torch.autograd.grad(y, mlp.parameters(), grad_outputs=u, create_graph=True)
vjp_flat = torch.cat([grad.view(-1) for grad in vjp])
jvp = torch.autograd.grad(vjp_flat, u, grad_outputs=v, retain_graph=False)
</code></pre>
<p>I’ve tried making u and v matrices (making the first dimension larger than 1) and the results are incorrect. I’m validating the jvp by checking it against a Jacobian obtained by sequentially calling backward on every element of the network output.</p>
<p>Does anyone know how I can accomplish this? I run into needing to compute this quantity quite a bit.</p>

username: ybj14
cooked: <p>My code here <a href="https://gist.github.com/ybj14/7738b119768af2fe765a2d63688f5496" rel="nofollow noopener">jvp.py</a> might help. Since you are dealing with output-params jacobian, maybe you need to be more careful with dimensions.</p>

username: alexander_langley
cooked: <p>Thanks, that seems to work pretty well. It seems like it’s possible to get a batch of jvps where there are multiple Jacobians, but only one vector, but when the vector is batched then the result is a batch of sums. Is that correct?</p>
<p>So, supposing that we have 3 vectors, this code will calculate a batch of sums of the 3 products?</p>

username: ybj14
cooked: <p>I think you are right. In your case where the v is collection of multiple vectors, your might need to tweak the components of ujp to get the correct dimension. Or you can use get_jvp for the weight of each module and then combine them together.</p>
How can I operate on a tensor slice guided by a index group with repeated index inside?

username: JTiC
cooked: <p>When I try to operate a tensor guided by a index with repeated index, it would only responses to the last operation. For example:</p>
<pre><code class="lang-auto">A = torch.FloatTensor([1,2,3,4])
index = torch.LongTensor([1,2,1,2])
add_value = torch.FloatTensor([1,2,3,4])
A[index]+= add_value
print(A)
# tensor([ 1.,  5.,  7.,  4.])
</code></pre>
<p>you can find that the output tensor came from [1, 2+3, 3+4, 4].<br>
However, what I want to get is [1, 2+1+3, 3+2+4, 4].<br>
What should I do?<br>
Thanks!<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: ptrblck
cooked: <p><code>tensor.scatter_add</code> should yield this result:</p>
<pre><code class="lang-python">A.scatter_add(0, index, add_value)
</code></pre>

username: JTiC
cooked: <p>Thank you! It works!</p>
Is there an equivalent of `tf.nn.embedding_lookup` in Caffe2 Python API?

username: shalijiang
cooked: <p>I’m trying to find an equivalent of tf.nn.embedding_lookup` in Caffe2.<br>
Seems there is a C++ interface, but couldn’t find the python interface.</p>
<p>BTW, does this link (<a href="https://caffe2.ai/doxygen-python/html/annotated.html" rel="nofollow noopener">https://caffe2.ai/doxygen-python/html/annotated.html</a>) contain all python APIs for Caffe2? Seems I couldn’t find the documentation of certain functions, e.g.,  <code>core.C.create_db</code></p>
Good model and loss function for multi class output

username: sai_tharun
cooked: <p>output labels size: [32,2,10], where 32 is batch size, in this every image belongs to one of 10 subclasses from 2 classes. i have used Vgg and added one more classifier to it and used NLLLoss function but after trainiing the model for 50 epochs,my loss seems to be not decreasing much, it is around 3.5.<br>
can anyone tell me which model should i use and the loss function.</p>
<p>below is my code:</p>
<pre><code class="lang-auto">loss_fn=CrossEntropyLoss()
loss_epoch_arr = []
max_epochs = 50

min_loss = 1000
batch_size = 64
loss_for_model={}

n_iters = np.ceil(9000/batch_size)
for lr in [0.01,0.01,0.05,0.009]:
    
    for w_d in [0.001,0.0001,0.0005,0.01]:
        
        opt = optim.Adam(model.parameters(), lr=lr,weight_decay=w_d,eps=0.1)    
        for epoch in range(max_epochs):
        
            for i, data in enumerate(train_loader, 0):
        
                images, labels = data
                images, labels = images.to(device), labels.to(device)
                        
                opt.zero_grad()
                out = custom_vgg.forward(images,model)
                out_classifier,out_classifier1 = out  
                labels1,labels2 = labels[:, 0], labels[:, 1] 
                
                targetnp=labels1.cpu().numpy()
                targetnp1 = labels2.cpu().numpy()
                idxs=np.where(targetnp&gt;0)[1]
                idxs1=np.where(targetnp1&gt;0)[1]
                new_targets=torch.LongTensor(idxs)
                new_targets1=torch.LongTensor(idxs1)
                new_targets = new_targets.to(device)
                new_targets1 = new_targets1.to(device)          
                loss_classifier = loss_fn(out_classifier, new_targets)
                loss_classifier1 = loss_fn(out_classifier1,new_targets1)
                loss = loss_classifier + loss_classifier1
                loss.backward()
                opt.step()
                
                if min_loss &gt; loss.item():
                    min_loss = loss.item()
                    best_model = copy.deepcopy(model.state_dict())
                    print('Min loss %0.2f' % min_loss)
                
                if i % 100 == 0:
                    print('Iteration: %d/%d, Loss: %0.2f' % (i, n_iters, loss.item()))
                del images, labels, out, out_classifier, out_classifier1
                torch.cuda.empty_cache()                
            loss_epoch_arr.append(loss.item())
           loss_for_model["f{lr} + '_'+f{w_d}"] = min_loss
plt.plot(loss_epoch_arr)
plt.show()
</code></pre>
Using resnet101/densenet pretrained on Food101

username: Mona_Jalal
cooked: <p>I wonder if resnet/densenet pretrained on food101 dataset exists in native pytorch or should I use something like below?</p>
<p><a href="https://github.com/lidan1/multidensenet/commit/9b8029aed5a13b028683e5e7df24c93cbea5183e" class="onebox" target="_blank" rel="nofollow noopener">https://github.com/lidan1/multidensenet/commit/9b8029aed5a13b028683e5e7df24c93cbea5183e</a></p>
Compile problem with libtorch

username: GloomyGhost-Mosquito
cooked: <p>Development environment:</p>
<ul>
<li>Clion</li>
<li>Windows10</li>
<li>MinGW/lib/gcc/x86_64-w64-mingw32/8.1.0</li>
</ul>
<p>Error message:</p>
<pre><code class="lang-cpp">[ 50%] Building CXX object CMakeFiles/A.dir/main.cpp.obj
In file included from D:/Program Files/MinGW/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/cstddef:49,
                 from X:/libtorch_cpu/include/torch/csrc/api/include/torch/cuda.h:5,
                 from X:/libtorch_cpu/include/torch/csrc/api/include/torch/all.h:3,
                 from X:/libtorch_cpu/include/torch/csrc/api/include/torch/torch.h:3,
                 from Z:\GitCode\A\main.cpp:1:
D:/Program Files/MinGW/lib/gcc/x86_64-w64-mingw32/8.1.0/include/c++/x86_64-w64-mingw32/bits/c++config.h:257:27: error: #if with no expression
 #if _GLIBCXX_USE_CXX11_ABI
</code></pre>

username: GloomyGhost-Mosquito
cooked: <p>OK,I find there is this option</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f9d966c61fee5d13e68f96c907bf6257b3525c82.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="zEgFzSg3f3eTyjDdn6vsN9vZs3M" width="607" height="211"></p>
<p>All thing worked!</p>
Questions about conv2d in caffe2 and fbgemm

username: Ryan95
cooked: <p>Hi, in <a href="https://github.com/pytorch/pytorch/blob/master/caffe2/quantization/server/conv_dnnlowp_op.cc#L1510" rel="nofollow noopener">caffe2/quantization/server/conv_dnnlowp_op.cc</a>, i find it says “only 1-3d convolution supported by NHWC”, so only 3d convolution can be implemented and accelerated by fbgemm? how to apply fbgemm to 2d convolution?</p>

username: dskhudia
cooked: <p>This means 1d, 2d and 3d are supported for NHWC storage. 2d convolutions (for low precision) are accelerated by fbgemm when the engine used is DNNLOWP.</p>
How to add an extra input to a CNN?

username: kdlsw
cooked: <p>Hi, I am new to this. There is this problem I have encountered, I was trying to classify the pixels in the input image into classes, sort of like segmentation, using CNN. The “interested” pixels usually locate in the top right corner, but the input pictures are too big, which I have to slice them in patches, by doing this, each input patch loses its “which region of the whole picture it’s from” information. How should I manually add this info into the input, in order to let the CNN taking position into account? Thank you.</p>

username: ptrblck
cooked: <p>You could try to add additional channels containing the coordinate information as described in <a href="https://arxiv.org/pdf/1807.03247.pdf" rel="nofollow noopener">this paper from Uber</a>.<br>
Here is also the corresponding video explanation:<br>
</p><div class="lazyYT" data-youtube-id="8yFQc6elePA" data-youtube-title="An Intriguing Failing of Convolutional Neural Networks and the CoordConv..." data-width="480" data-height="270" data-parameters="feature=oembed&amp;wmode=opaque"></div>
Stack tensor onto itself along the dimension

username: jgamper
cooked: <p>Hello,</p>
<p>I have a tensor of the shape <code>(batch_size, bag_size, num_channels, height, width)</code> for multiple instance learning. Where each batch consists of a certain number of bags (batch size), each bag consists of a certain number of images (bag size).</p>
<p>I would like to take that tensor, and stack all of the images from every bag along one dimension, resulting in a tensor <code>(batch_size * bag_size, num_channels, height, width)</code>, without mixing the data. It seems that <code>view</code> is not be the right tool here?</p>
<pre><code class="lang-auto">H = cnn_model(x.view(-1, num_channels, width, height)) # x of shape (batch_size, bag_size, num_channels, height, width)
</code></pre>
<p>When I compute the loss as a simple sum of outputs,  weighting just with one bag in the batch, while zero weighting the rest of the bags. Then calling <code>backward()</code>, the resulting gradients are still with respect to every input. What would be the solution, how could I otherwise restack <code>(batch_size, bag_size, num_channels, height, width)</code> into  <code>(batch_size * bag_size, num_channels, height, width)</code>?</p>
<pre><code class="lang-auto">weights = torch.zeros(H.size())
weights.data[1,:] = 1
loss = torch.dot(weights.view(-1), H.view(-1))
loss.backward()
</code></pre>

username: ptrblck
cooked: <p>Since the flattened dimensions are consecutive, the elements won’t be mixed.<br>
Here is a small example to show what the operation is doing:</p>
<pre><code class="lang-python">batch_size, bag_size, num_channels, height, width = 2, 3, 2, 2, 2
x = torch.arange(batch_size*bag_size*num_channels*height*width).view(
    batch_size, bag_size, num_channels, height, width)
print(x)
x = x.view(-1, num_channels, height, width)
print(x)
</code></pre>
Network automatically split the input sequence

username: Rickyim
cooked: <p>Very strange that my model automatically split the sequence and call lines in <code>forward()</code> multiple times.<br>
This is my model:</p>
<blockquote>
<p>input_dim = 2<br>
output_dim = 10<br>
target_dim = 10</p>
<p>class LSTMClassifier(nn.Module):</p>
<pre><code>def __init__(self, input_dim, output_dim, target_dim):
    super(LSTMClassifier, self).__init__()
    self.input_dim = input_dim
    self.output_dim = output_dim
    self.target_dim = target_dim
    self.lstm = nn.LSTM(self.input_dim, self.output_dim)
    self.out2score = nn.Linear(self.output_dim, self.target_dim)

def forward(self, input):
    _, lstm_out = self.lstm(input)
    #hidden_state
    lstm_out = lstm_out[0].squeeze()
    score = self.out2score(lstm_out) 
    outclass = F.log_softmax(score, dim = 1)
    return outclass
</code></pre>
</blockquote>
<p>also I am using Gpus</p>
<blockquote>
<p><span class="hashtag-raw">#define</span> a net<br>
net = LSTMClassifier(input_dim, output_dim, target_dim)<br>
loss_function = nn.NLLLoss()<br>
optimizer = optim.Adam(net.parameters(), lr=0.01)<br>
<span class="hashtag-raw">#using</span> Gpus<br>
if torch.cuda.is_available():</p>
<pre><code>print("cuda is available...")
if torch.cuda.device_count() &gt; 1:
    print('using %d Gpus'%torch.cuda.device_count())
    net=nn.DataParallel(net)
net.cuda()
</code></pre>
<p>else:<br>
print(‘cuda disabled’)</p>
</blockquote>
<p>When I call <code>net( )</code>with data shape<code>[6, 10, 2]</code>where <code>6</code> is the sequence length in LSTM. I see from the vscode debug console that the self.lstm() is called <strong>three times</strong> and each time with sequence length of 2. So the final output size becomes <code>[30, 10]</code>, instead of <code>[10, 10]</code>, which is what I want.</p>
<p>Any suggestions? Thanks!</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> splits the input batch in dim0 and pushes each chunk onto a GPU.<br>
Each GPU will call the <code>forward</code> method thereafter using the current data chunk.<br>
In your case it looks like you are using 3 GPUs.<br>
Since you are apparently using data samples with the batch size in dim1, this will also yield a wrong behavior, since the data should be usually chunked in the batch dimension.<br>
You could permute your data and use <code>batch_first=True</code> in your RNN layers or permute the data chunk to the original shape in the <code>forward</code> method.</p>