akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Contributing to QA/Testing of PyTorch

username: pjot726
cooked: <p>Hello Everyone,</p>
<p>I am a newcomer to PyTorch, and I found this project very interesting. I am interested in knowing if there is a testing/QA process used by the PyTorch project, and if there is, how one could contribute to it. I have some experience with PyTest, if it is relevant.</p>

username: ptrblck
cooked: <p>You can find the tests <a href="https://github.com/pytorch/pytorch/tree/master/test" rel="nofollow noopener">here</a>.<br>
A good start to get familiar with them could be to have a look in <a href="https://github.com/pytorch/pytorch/blob/master/test/test_nn.py" rel="nofollow noopener">test_nn.py</a>.</p>
<p>Please let us know, if you have some ideas on how to improve them. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
VGG input issue

username: Erdi
cooked: <p>Hi all</p>
<p>When using transfer learning with VGG (probably others), i saw lots of pytorch kernels out there starting with</p>
<p>''VGG-16 Takes 224x224 images as input, so we resize all of them ‘’</p>
<p>in the transformation section. But it seems to me this statement is a bit misleading. VGG has 7x7 Adaptive Adaptive Pooling before the classifier:<br>
class VGG(nn.Module):</p>
<pre><code>def __init__(self, features, num_classes=1000, init_weights=True):
    super(VGG, self).__init__()
    self.features = features
    self.avgpool = nn.AdaptiveAvgPool2d((7, 7))
    self.classifier = nn.Sequential(
        nn.Linear(512 * 7 * 7, 4096),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(4096, 4096),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(4096, num_classes),
    )
    if init_weights:
        self._initialize_weights()

def forward(self, x):
    x = self.features(x)
    x = self.avgpool(x)
    x = torch.flatten(x, 1)
    x = self.classifier(x)
    return x
</code></pre>
<p>so no matter what input is given, it is rescaled to 512x7x7 before the linear layer. What do you guys think? Am I missing something? I found this point important because resizing from say 100x100 to 224x224 (resize 256 crop 224 etc) may easily pollute the model.</p>

username: ptrblck
cooked: <p>The adaptive pooling layer relaxes the restriction of 224x224 inputs, but note that the kernels were trained using this resolution.<br>
If your input size is quite different from the original used one, your performance might be worse.</p>

username: Erdi
cooked: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> so the model will work for any input size but since the original model is trained with 224x224, probably we dont expect a high performance.</p>

username: ptrblck
cooked: <p>Yes, that would be the basic assumption.<br>
However, your model might still work fine, if the spatial size does not change very much or if you finetune it on the new size.</p>
How to deal with the limited action space in Reinforcement Learning?

username: zikun_liu
cooked: <p>Hi, I’m new in reinforcement learning and I’m recently using A3C for continuous control.</p>
<p>My action i.e the output of the actor network is a vector with size(64*1) and these 64 values are used as the probabilities, so all of them should be in the scale of (0,1).  I use the normal distribution with the decayed variance to choose the real action and calculate the log-probabilities, but if I do so, the action will be out of range.</p>
<p>I tried to clip the action between 0 and 1, but that would destroy the normal distribution and lead to some bad results. Have you ever had the same problem? Any advice? Thanks!</p>
How do I recover the 3D structure of a layer after a Fully Connected layer (or a flatten layer)?

username: pinocchio
cooked: <p>I want to apply 1 fully connected layer first and then start apply convolutions. But I noticed that the 3D structure might be lost. Is there a way to recover that 3D structure and apply 2d convolutions or am I doomed to only apply 1d convolutions thereafter?</p>
<hr>
<hr>
<p>Cross posted:</p>
<ul>
<li><a href="https://forums.fast.ai/t/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer-or-a-flatten-layer/52489" rel="nofollow noopener">https://forums.fast.ai/t/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer-or-a-flatten-layer/52489</a></li>
<li><a href="https://discuss.pytorch.org/t/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer-or-a-flatten-layer/53313" class="inline-onebox">How do I recover the 3D structure of a layer after a Fully Connected layer (or a flatten layer)?</a></li>
<li><a href="https://ai.stackexchange.com/questions/13950/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer" rel="nofollow noopener">https://ai.stackexchange.com/questions/13950/how-do-i-recover-the-3d-structure-of-a-layer-after-a-fully-connected-layer</a></li>
</ul>

username: albanD
cooked: <p>Hi,</p>
<p>You can unflatten after the fully connected layer:</p>
<pre><code class="lang-auto">original_size = input.size()
flat_input = input.view(original_size[0], -1)
flat_ouput = linear(flat_input)
output = flat_output.view(original_size)
</code></pre>

username: pinocchio
cooked: <p>I want to build an unflatten layer. How would one do that?</p>
<p>e.g. of the flatten layer:</p>
<pre><code class="lang-auto">#mini class to add a flatten layer to the ordered dictionary
class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0),-1)
</code></pre>
<p>from your code I assume the only way is to do this:</p>
<pre><code class="lang-auto">#mini class to add an unflatten layer to the ordered dictionary
class UnFlatten(nn.Module):
    def forward(self, flat_ouput, original_size):
        return flat_ouput.view(original_size)
</code></pre>
Can something be done to fix this hierarchy?

username: vainaijr
cooked: <p>torch.nn.modules.module.Module</p>
<p>shouldn’t it be something like</p>
<p>torch.nn.modules.Module</p>
<p>or maybe just torch.nn.modules</p>
<p>why are there multiple keywords for same name in the hierarchy?</p>
<p>similarly there is</p>
<p>torch.nn.modules.activation.torch</p>
<p>multiple same names in the hierarchy, isn’t this a bug that needs to be fixed?</p>

username: ajhanwar
cooked: <p>I’m pretty sure the hierarchy is for encapsulating certain functions.<br>
For example, with <code>torch.nn.modules.module</code>, folder <code>modules</code> contains <code>module.py</code> which in itself contains a <code>Module</code> class.</p>
<p>Feel free to check out the repository as it provides much better context as to why the functions are organized in the manner they are: <a href="https://github.com/pytorch/pytorch/tree/master/torch/nn/modules" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/torch/nn/modules</a></p>
Fold/unfold to get patches

username: noornk
cooked: <p>Hi All.<br>
I have a dataset of 3328x2560, and 4084x3328 images, and I want to get patches of size 256x256. How can I use fold and unfold, to get patches and then put them back together as the original image?<br>
Thanks for your help!</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">this post</a> which gives an example of this use case.</p>

username: noornk
cooked: <p>Thanks for your reply.<br>
My images are 2-dimensions, and I fail to adapt the code. I can do the unfold, but I can not put them back together. Could you please help me on that?</p>

username: ptrblck
cooked: <p>Sure!<br>
Could you post the code you are using to unfold your input?<br>
Note that you would have to pad the inputs of 4084 to 4096 to get non overlapping patches.</p>

username: noornk
cooked: <p>I just do as you did in the mentioned post, so it’ll be:</p>
<blockquote>
<p>x = image<br>
kh, kw = 256, 256 # kernel size<br>
dh, dw = 256, 256 # stride<br>
patches = x.unfold(0, kc, dc).unfold(1, kh, dh)<br>
unfold_shape = patches.size()<br>
patches = patches.contiguous().view(patches.size(0), -1, kc, kh) <span class="hashtag-raw">#unsure</span><br>
print(patches.shape)</p>
</blockquote>
<p>Although I quite don’t understand the line with “unsure” tag.<br>
And yes I know, but is there a way to do it without adding padding? like maybe having overlapping patches?</p>

username: ptrblck
cooked: <p>I don’t think there is an easy way of changing the stride for the last window.<br>
Here is the code for both input sizes:</p>
<pre><code class="lang-python"># 3328x2560
x = torch.randn(3328, 2560)
kh, kw = 256, 256 # kernel size
dh, dw = 256, 256 # stride
patches = x.unfold(0, kh, dh).unfold(1, kw, dw)
unfold_shape = patches.size()

patches = patches.contiguous().view(-1, kh, kw)
print(patches.shape)

# Reshape back
patches_orig = patches.view(unfold_shape)
output_h = unfold_shape[0] * unfold_shape[2]
output_w = unfold_shape[1] * unfold_shape[3]
patches_orig = patches_orig.permute(0, 2, 1, 3).contiguous()
patches_orig = patches_orig.view(output_h, output_w)

# Check for equality
print((patches_orig == x).all())


# 4084x3328
x = torch.randn(4084, 3328)
kh, kw = 256, 256 # kernel size
dh, dw = 256, 256 # stride
patches = x.unfold(0, kh, dh).unfold(1, kw, dw)
unfold_shape = patches.size()

patches = patches.contiguous().view(-1, kh, kw)
print(patches.shape)

# Reshape back
patches_orig = patches.view(unfold_shape)
output_h = unfold_shape[0] * unfold_shape[2]
output_w = unfold_shape[1] * unfold_shape[3] # you will lose some pixels in w
patches_orig = patches_orig.permute(0, 2, 1, 3).contiguous()
patches_orig = patches_orig.view(output_h, output_w)

# Check for equality using slicing, since w is now smaller in x
print((patches_orig == x[:output_h, :output_w]).all())
</code></pre>
<p>As you can see, you’ll lose some pixels in dim1 for the second input shape.</p>
How does pytorch DistributedDataParallel synchronize gradient calculation between different nodes?

username: qiuyang
cooked: <p>I know pytorch DistributedDataParallel use torch.distributed.Reducer class to ensure different GPUs finish gradients calculation in one node and then start ring all reduce process.But how to ensure other nodes finish gradients calculation?For example,there are three nodes.The first nodes finished gradient calculation and prepared to start all reduce.How does the first node know whether other nodes finished gradient calculation?</p>
How to understand torch.expand_as operation?

username: ronghui
cooked: <pre><code class="lang-auto">a = torch.rand(2, 3)
b = torch.rand(4, 3)
c = b.expand_as(a)
print(c)

a = torch.rand(2, 3)
b = torch.rand(2, 4)
c = b.expand_as(a)
print(c)
</code></pre>
<p>I dont know why it does not work. What is the right way to figure it out?</p>

username: MariosOreo
cooked: <p>Hello <a class="mention" href="/u/ronghui">@ronghui</a>,</p>
<p>From the document,</p>
<blockquote>
<p>Expand this tensor to the same size as  <code>other</code> .  <code>self.expand_as(other)</code>  is equivalent to  <code>self.expand(other.size())</code> .</p>
</blockquote>
<p>And the <code>.expand()</code> operation involves some broadcasting semantics <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">here</a>. I think you could not “expand” a large size tensor to a smaller one. So the code snippet above will not work.</p>
<p>If I am wrong, please correct me. Thanks.</p>

username: ronghui
cooked: <p>So you are saying two tensors must be “broadcastable” before we can apply expand_as on them?</p>

username: MariosOreo
cooked: <p>After some try, it is just my own view that the <code>.expand()</code> operation maybe involves broadcasting semantics.</p>

username: ZucchiniTang
cooked: <p>print out a,b will be easy for understanding.<br>
Example 1)<br>
output:</p>
<pre><code class="lang-auto">a: tensor([[0.9619, 0.0384, 0.7012],
        [0.5561, 0.3637, 0.9272]])
b: tensor([[0.5986, 0.2582, 0.6261],
        [0.6928, 0.9175, 0.6737],
        [0.9951, 0.8568, 0.6015],
        [0.7922, 0.5019, 0.8162]])
</code></pre>
<p>the reason is 1) the size of b is bigger than a’s,  you can not expand b by a. 2) the dimension is not match, to output different c, you can size of b to (2, 2, 3) or others. Shown as below,</p>
<pre><code class="lang-auto">a = torch.rand(2, 3)
b = torch.rand(2,2, 3)
print('a:',a)
print('b:',b)
c = a.expand_as(b)
print('c:',c)
</code></pre>
<p>outputs:</p>
<pre><code class="lang-auto">a: tensor([[0.4748, 0.5521, 0.7741],
        [0.0785, 0.2785, 0.5222]])
b: tensor([[[0.7777, 0.3046, 0.8019],
         [0.7398, 0.1424, 0.6398]],

        [[0.9034, 0.8937, 0.8674],
         [0.1737, 0.3192, 0.4451]]])
c: tensor([[[0.4748, 0.5521, 0.7741],
         [0.0785, 0.2785, 0.5222]],

        [[0.4748, 0.5521, 0.7741],
         [0.0785, 0.2785, 0.5222]]])
</code></pre>
<p>Example 2) is same problem with example 1.</p>
Replace duplicate value of tensor based on scores from another

username: IjlalBaig
cooked: <p>Hi,</p>
<p>Consider the following example:</p>
<pre><code class="lang-auto">idxs = [[0, 1, 2, 1], [3, 2, 1, 3]]
scores = [[0.5000, 0.8000, 0.5000, 0.5000], [0.5000, 0.5000, 0.5000, 0.9000]]
</code></pre>
<p>Within a batch of <code>idxs</code>, I would like to only retain values with the largest corresponding <code>score</code> entry and replace the rest with <code>-1</code>.</p>
<p>So I would expect an output:</p>
<p><code>out = [[ 0, 1, 2, -1], [ -1, 2, 1, 3]]</code></p>
<p>How can I achieve something like this?</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>Your output does not match your description I think.<br>
Don’t you expect: <code>[[ -1, 1, -1, -1], [-1, -1, -1, 3]]</code> ?</p>

username: IjlalBaig
cooked: <p>Hi albanD,</p>
<p>For the first batch [0, 1, 2, 1], only 1s are duplicate, so the index with larger score remains and the other 1 is replaced by -1, similarly in the second batch [3, 2, 1, 3], 3s are duplicate, so only 3s with the smaller scores are replaced</p>

username: albanD
cooked: <p>Ho I missed the part about duplicates in your explanation.<br>
I’m afraid you won’t have a specialized function to do this, so you will have to do it by handm checking each index.</p>

username: IjlalBaig
cooked: <p>Thanks for the reply.<br>
I do have an idea of how I want to implement this, but it is limited to just one batch. If I flatten the all batches and then use that function, the search space for comparison will be too large; since each duplicate value will be compared with all the other values in flattened tensor. Is there away I can apply a function to all batches individually?</p>

username: albanD
cooked: <p>You can do a outer for loop and just look at <code>input = full_input.select(0, batch_idx)</code> every time.<br>
Note that <code>select()</code> does not copy memory and so any <strong>inplace</strong> change of <code>input</code> will be reflected into <code>full_input</code> !</p>

username: IjlalBaig
cooked: <p>I was hoping to avoid using a for loop, but I guess it is inevitable. Thanks for all your help.</p>
The exactly same codes in two different projects result a Tensor to different device?

username: feiyuhuahuo
cooked: <p>I am confused about which device would a Tensor comes from when creating a new Tensor?<br>
I am using a piece of exactly same code in two different projects. In one project, ‘self.priors = torch.Tensor(prior_data).view(-1, 4)’, self.priors is on GPU. But  for the other project, self.priors is on CPU.<br>
What’s the reason? The two pieces of code are completely same.</p>

username: albanD
cooked: <p>This is most likely because either <a href="https://pytorch.org/docs/stable/torch.html#torch.set_default_tensor_type" rel="nofollow noopener">set_default_tensor_type</a> or <a href="https://pytorch.org/docs/stable/torch.html#torch.set_default_dtype" rel="nofollow noopener">set_default_dtype</a> has been called in one project and not the other.</p>

username: feiyuhuahuo
cooked: <p>Oh, yes. I missed ‘set_default_tensor_type’ in one project. Thank you.</p>
A question about torch.log

username: shenwei101623
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e07cb6c4d327e3ebcc13a22662c3d824ad51ad49.jpeg" alt="test" data-base62-sha1="w1U9pocgO6BlZnM6ECoACWyG4Wt" width="570" height="179"><br>
Why does this happen?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This happens because log is only defined for strictly positive inputs. And so we return -inf for all values &lt;= 0 as it is the limit at 0.</p>
The model is trained fine but when testing it is showing error

username: Sainadh086
cooked: <p>I trained the model with RNN and Added some hidden weights. It is working fine on testing and validation but when coming to testing it is not working I did same preprocessing and checked several times but I not find how to solve the error. [<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ed463c27e1f0431cea041413d4f308fd3225ac9d.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed463c27e1f0431cea041413d4f308fd3225ac9d" title="Screenshot%20(171)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ed463c27e1f0431cea041413d4f308fd3225ac9d_2_690x388.png" alt="Screenshot%20(171)" data-base62-sha1="xR1DzrXZEAzOxeTieUWclQ0t8f3" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ed463c27e1f0431cea041413d4f308fd3225ac9d_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ed463c27e1f0431cea041413d4f308fd3225ac9d_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ed463c27e1f0431cea041413d4f308fd3225ac9d_2_1380x776.png 2x" data-dominant-color="E3E3E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(171)</span><span class="informations">1920×1080 158 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> ]</p>

username: Sainadh086
cooked: <p>You can view my code in<br>
<aside class="onebox googledrive">
  <header class="source">
      <a href="https://drive.google.com/file/d/0B-Gg3CeHstknZHdQc3o1VWpmNWk1R2piUEgzS2pqWVlfTUUw/view?usp=sharing" target="_blank" rel="nofollow noopener">drive.google.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/0B-Gg3CeHstknZHdQc3o1VWpmNWk1R2piUEgzS2pqWVlfTUUw/view?usp=sharing" target="_blank" rel="nofollow noopener"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/0B-Gg3CeHstknZHdQc3o1VWpmNWk1R2piUEgzS2pqWVlfTUUw/view?usp=sharing" target="_blank" rel="nofollow noopener">Sentiment Analysis by Cherukumalli Sainadh.ipynb</a></h3>

<p>Colaboratory notebook</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: albanD
cooked: <p>You should check the <code>labels</code> values for your test set to make sure they are correct.<br>
Also you can run the same code on the CPU to get a better error message.</p>
Difference and use of the padding and output padding in ConvTranspose2d

username: Rishav_Sapahia
cooked: <p>What’s the difference between padding and output padding in <code>torch.nn.ConvTranspose2d(in_channels, out_channels, kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1, padding_mode='zeros')</code></p>
<p>I am trying it to apply to a input of 256×32×32 to obtain output of 256×64×64 with filter size 2*2 and stride 2 but couldn’t understand the role of different padding here.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The Theano tutorial on convolution should have all the information you need <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html" rel="nofollow noopener">here</a>. It is quite clear with a lot of illustrations.</p>
Problem about num_workers and tensorboard

username: JoshTang
cooked: <p>Hi,</p>
<p>When I use num_workers &gt; 0, I found that it will also create the multi number of tensorboard files. However, it only writes the information in the first creation files.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5fe49273ffe3fac2a72a12e8ab5141abc9459dbd.png" data-download-href="https://discuss.pytorch.org/uploads/default/5fe49273ffe3fac2a72a12e8ab5141abc9459dbd" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5fe49273ffe3fac2a72a12e8ab5141abc9459dbd.png" alt="1" data-base62-sha1="dGj4x4L8oLBKs9DpmOaq1aALvXL" width="690" height="100" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1104×161 4.31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2c62a9d3d1b1366195dea90b5e7021cae4035631.png" alt="%E5%9C%96%E7%89%87" data-base62-sha1="6kEoqFqMrTuxNYq5TOPXvTywga5" width="585" height="185"></p>
<p>Since it doesn’t show any error, I’m not sure which part of the code should provide for your reference. What can I do to prevent it create lots of files?</p>

username: albanD
cooked: <p>Hi,</p>
<p>How do you use tensorboard? Where do you initialize it?</p>
Propagating back to a CNN model through an embedding layer

username: oldcricket
cooked: <p>Hi, I have a CNN model, which produces the index of an object in a vocabulary. The index is then embedded (by nn.Embedding) and fed into a language model for further processing. What I need here is end-to-end learning. However, the problem seems to be that the learning error cannot be propagated back to the CNN model through the embedding layer. Since the language model is pre-trained, I cannot make any change to the word embedding (i.e. nn.Embedding) in this case. Anyone has experience on that? Thanks a lot for the help in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The operation that goes from your first network output to the index is non differentiable (or it’s gradient is 0 almost everywhere). So you cannot backprop through that operation.</p>
<p>A possible solution could be to do a soft indexing on your embedding layer based on the continuous output of your first network. But that is not supported by Embedding so you will need to do this by hand with a matrix vector product using the weights of the Embedding layer.</p>
Training accuracy is decreasing and validation is constant but very less then training accuracy

username: Manali
cooked: <p>I am working on crop disease with 20k images. I have imbalance dataset, so downsampled it but when I applied the resnet model on this I am getting training accuracy initially with  86%  but later epochs it is decreasing and val acc to 7% is constant with different batch_size, weight_decay. I have augmented data with brightness, rescaled, etc.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could it be that the model overfits on your training set given that you only have 20k images and use a resnet? Does a very simple model give similar behavior?</p>
How can I accumute the total loss if using a closure for the optimizer?

username: Fractale
cooked: <p>I was doing  <code>total_loss += loss.detach()</code> after  <code>optimizer.step()</code>  but now I use a closure.<br>
How can I have access access to the loss now?</p>
<p>I use the  <code>total_loss</code>  to know when to stop the learning process.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I would use a variable from a common parent scope to store it.<br>
Reset it before the training and each closure function modify this variable by adding to it. For example by making <code>total_loss</code> a global variable if you use python 2.7 or <code>nonlocal</code> in python 3.5+.</p>

username: Fractale
cooked: <p>Dear <a class="mention" href="/u/alband">@albanD</a> thank you for your reply. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=6" title=":grinning:" class="emoji" alt=":grinning:"><br>
Does the <code>total_loss</code> will still be accurate for optimizer that require a closure?</p>
<p>I didn’t really understand why a closure is necessary for certain optimizer<br>
but from my current understanding the optimizer will run the closure multiple times each step.<br>
Does the <code>total_loss</code> will still be a valid way to stop training in that case?</p>
<p>Thank you so much</p>

username: albanD
cooked: <p>Hi,</p>
<p>That will depend on your problem definition and the optimizer.<br>
In some case, you may only want to use the first eval of the closure by the optimizer.<br>
In other cases, you can just average over all the evaluations as it is still valid information.<br>
That would depend on your application.</p>

username: JTiC
cooked: <p>Hi, I look through the questions about the closure, but I still don’t understand why to use closure. Do you have some classicial and simple examples? For example, if you want to do xxx, you have to use closure, or other examples like this. Thank you so much!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Optimizers that need to evaluate the function multiple times will require closure. For example <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.LBFGS" rel="nofollow noopener">LBFGS</a>.</p>
How I can use Ignite as a metric class in my training loop

username: Aiman_Mutasem-bellh
cooked: <p>Dear friends,</p>
<p>How I can use Ignite just as a metric class in my training loop to calculate the (accuracy, precision, recall)?</p>
<p>Regards,<br>
Aiman</p>

username: ptrblck
cooked: <p>If you are using Ignite for training, have a look at the <a href="https://pytorch.org/ignite/quickstart.html#code" rel="nofollow noopener">Quickstart guide</a> showing an example usage of some metrics.<br>
Or would you like to use some Ignite snippets in isolation?</p>

username: Aiman_Mutasem-bellh
cooked: <p>Dear ptrblck,</p>
<p>Thank you for replay. Actually, the Accuracy and Loss class worked fine with me, but I can’t calculate the precision, recall, and F1 as isolated classes in my training loop.</p>
<p>I tried as below code:</p>
<pre><code class="lang-auto">def get_precision(output, trg):

    #output = torch.tensor(output) #predicted
    #trg = torch.tensor(trg) #output
                         
    precision = Precision(output_transform=thresholded_output_transform, average=False)
    #binary_accuracy = Accuracy(thresholded_output_transform)
    precision.update((output, trg))
    epoch_precision = precision.compute()
        
    return epoch_precision
</code></pre>
<p>and I called in the training section like</p>
<pre><code class="lang-auto"> precision = get_precision(output, trg)
</code></pre>
<p>I get a tensor array like this</p>
<pre><code class="lang-auto">tensor([0.4250, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
       dtype=torch.float64)
</code></pre>
<p>I think this is not a correct value to use directly.</p>
<p>The training class as :</p>
<details>
<summary>
Summary</summary>
<pre><code class="lang-auto">ef train(model, iterator, optimizer, criterion, clip):
    
    model.train()
    
    epoch_loss = 0
    epoch_accuracy  = 0 
    epoch_precision = 0 
    
    for i, batch in enumerate(iterator):
        
        src = batch.src
        trg = batch.trg
        
        optimizer.zero_grad()
        
        output = model(src, trg[:,:-1])

        
        output = output.contiguous().view(-1, output.shape[-1])
        trg = trg[:,1:].contiguous().view(-1)       

        loss = criterion(output, trg)
        accuracy = get_accuracy (output, trg)
        precision = get_precision(output, trg)
        
        loss.backward()
        
        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
        
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_accuracy  +=  accuracy 
        epoch_precision += precision 
    return epoch_loss / len(iterator), epoch_accuracy  / len(iterator), epoch_precision / len(iterator)
</code></pre>
</details>

username: Aiman_Mutasem-bellh
cooked: <p>I found the solution, just I set  the average value = True.</p>
Torch.lstsq didn't work

username: YeongHwa_Jin
cooked: <p>I want to use torch.lstsq(), but I got a this error msg.</p>
<p>" module ‘torch’ has no attribute ‘lstsq’ "</p>
<p>Why does this error occur?</p>

username: ptrblck
cooked: <p>This method was introduces in PyTorch <code>1.2.0</code>, so make sure you are using the latest stable release.<br>
You can find the install instructions on the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">website</a>.</p>

username: YeongHwa_Jin
cooked: <p>Oh, I see. Thanks to your reply!</p>
Train and test a model in parallel

username: JTiC
cooked: <p>As we know, in PyTorch, model.train() and model.eval() are needed when training or testing. Can we train and test a model in parallel?For example, training data (batch_size=128) and testing data(batch_size=16)   is concat and a dataloader 128+16 is gotten. And the training loss computation is called, for the first 128 samples and the testing loss is computed based on the last 16 samples? Is this a good way to do NAS or AutoML?</p>
Grad flow for tensor reshape operation

username: Liven_Zhu
cooked: <pre><code class="lang-auto">import torch
data = torch.tensor([[1.,2.,3.,4.], [5.,6.,7.,8.]], requires_grad=True)
batch = data.shape[0]
t_data = data.reshape(batch, 2, 2)
tf_data = torch.zeros((batch, 3, 2, 2))
for i in range(batch):
    tf_data[i] = t_data[i].expand_as(tf_data[i])
loss = torch.sum(tf_data)
print('tf_data\n', tf_data)
print('data\n', data)
tf_data.register_hook(print)
data.register_hook(print)
loss.backward()
</code></pre>
<p>I want to know the gradient flow for some tensor operation like reshape or expand_as, so I wrote above code to test. I got below output.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3b3d91956dd17184f585ddcf20b710651619e827.png" alt="image" data-base62-sha1="8s451AxNjQrlfyyFPexqBPWGWp1" width="637" height="499"><br>
the grad of data and tf_data is all 3 and 1 no matter how I changed data, can anyone explain the gradient flow for this case in pytorch? How does the pytorch calculate the gradient for this case?</p>

username: ilyes
cooked: <p>Well, the <code>tf_data</code> grads are very logical since  <code>loss = sum (tf_data)</code> .No matter what your code looks like before, and since <code>loss = tf_data[0,0,0,0] + tf_data[0,0,0,1] + ... + tf_data[1,2,1,1]</code> the grad will be just a differentiation with respect to each element which means 1.</p>
<p>Now for <code>data</code> grads, this line <code>tf_data[i] = t_data[i].expand_as(tf_data[i])</code> is basically repeating (by stacking) <code>t_data[i]</code> ( shape = ( 2, 2) in this case)  a number of times until it reaches <code>tf_data[i]</code> dimensions ( (3, 2, 2) in this case). means you are repeating 3 times.</p>
<p>So <code>loss = sum(tf_data) = 3 * sum (t_data) = 3 * sum (data)</code> means whatever your code looks like before ( without changing dimensions) you will get <code>grads = 3</code>.</p>
<p><code>3 * sum (t_data) = 3 * sum (data)</code> because summing two tensors with same elements and different shapes, will result in the same result.</p>

username: Liven_Zhu
cooked: <p>Got it !!! Thank you.</p>
About Transformer in Pytorch 1.2

username: bemorept
cooked: <p>Hi, All.</p>
<p>New Transformer functions in 1.2 are just for seq2seq application ?<br>
Is It no relation with image?</p>
<p>Why is not “Translator” but “Transformer” ?</p>
<p>Thanks in advance.<br>
Best,<br>
<span class="mention">@bemoregt</span>.</p>

username: 1-800-BAD-CODE
cooked: <p>Although the name may be confusing for those in the image processing world (where ‘transformations’ are commonly used) the Transformer is an architecture for machine translation.</p>
<p>‘Translator’ is a general term; ‘Transformer’ is the name of a popular machine translation architecture and the implementation that you see in <code>torch.nn.modules.transformer.py</code>.</p>
Get local size in distributed training

username: meijieru
cooked: <p>Anyway to get number of procedures per node in distributed training?<br>
In horovod we could use <code>hvd.local_size()</code>, but I found no alternative in distributed module.<br>
Thanks.</p>

username: pietern
cooked: <p>There is no concept of local processes and remote processes; only the total number of processes through <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.get_world_size" rel="nofollow noopener"><code>torch.distributed.get_world_size()</code></a>.</p>
<p>To understand if we need add this: what do you need this for?</p>

username: meijieru
cooked: <p>For example, I want to fully use all CPU for data loading without overhead.<br>
Thus I want to divide the <code>number_workers</code> by local size.</p>

username: meijieru
cooked: <p><a class="mention" href="/u/pietern">@pietern</a> Any solution for this? Thanks.</p>
Pytorch C++ Frontend is Crashing on the loading the Data

username: reself
cooked: <p>Hello Guys,</p>
<p>i need help! I’m trying to setup Pytorch(LibTorch) in Windows 10.<br>
I was trying to use the Mnist Example on the Webpage from Pytorch -&gt; <a href="https://pytorch.org/cppdocs/frontend.html" rel="nofollow noopener">https://pytorch.org/cppdocs/frontend.html</a></p>
<p>Its is crashing when i try to Load my dataset in a specific folder.</p>
<pre><code class="lang-auto">	auto dataset = torch::data::datasets::MNIST("/mnist")                  &lt;-------------------
		.map(torch::data::transforms::Normalize&lt;&gt;(0.5, 0.5))
		.map(torch::data::transforms::Stack&lt;&gt;());
</code></pre>
<p>It crashed right in the first line with the Error Message:</p>
<pre><code class="lang-auto">Unhandled exception at 0x00007FFFC2B9A839 in testingmnist.exe: Microsoft C++ exception: c10::Error at memory location 0x000000B3FCAFE9C0.
</code></pre>
<p>The Folder is there i testet another location a fixed location and not a relative location.</p>
<p>When i try to debug i go into the functions so it dont crashes immidiatly.</p>
<p>I hope you guys can help me if you need more Information go and Ask.<br>
Thank you for taking time to read this.</p>
<p>Lars</p>

username: reself
cooked: <p>Ok new update it is failing always when it trys to do some Memory things.</p>
<p>The Exception is throwed by the kernel32.dll</p>
<p>I hope that will help.</p>
<p>Is it a Windows only problem ?</p>

username: peterjc123
cooked: <p>Could you please try the debug version and post the stack trace here?</p>

username: reself
cooked: <p>Ok i got it fixed now. The problem was the Dataset that you download from the mnist site. Dont OPEN int with Winrar or other things. I opened it on Linux with tar and the all works. So Windows is fucking up the data.</p>
Build a recommendation from DLRN?

username: Etienne-Lamy
cooked: <p>Dear all,</p>
<p>I just tested the Facebook code for recommendation. The DLRN tool has good results (Kaggle ads challenge) in predicting whether an ad will be clicked, given the user info.</p>
<p>But I don’t understand, how would an ad recommendation come out of this ? Does one have to make a generative network to find an ad that the customer is susceptible to click ?</p>
<p>Indeed, the DLRN network is a discriminative network, not a generative one.</p>
<p>Thanks,</p>
DCGAN : How to improve Generator and Discriminator Loss during training?

username: Marion2
cooked: <p>I trained a <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">DCGAN from Pytorch tutorial</a> that I changed a little to generate 1024 px output but since I did that Generator and Discriminator Loss during training isn’t good as you can see it here :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/586c09f16e516155a17f6b68667164bde88aa9b3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/586c09f16e516155a17f6b68667164bde88aa9b3" title="10"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/586c09f16e516155a17f6b68667164bde88aa9b3_2_690x342.jpeg" alt="10" data-base62-sha1="cCdulOMl4BNzN1QWMi9RLX5bSJt" width="690" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/586c09f16e516155a17f6b68667164bde88aa9b3_2_690x342.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/5/586c09f16e516155a17f6b68667164bde88aa9b3_2_1035x513.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/586c09f16e516155a17f6b68667164bde88aa9b3.jpeg 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">10</span><span class="informations">1336×664 70.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you have some advises to improve it ?</p>
<p>Thanks for your help ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>I’m not sure if it’s that easy so scale up the resolution of the generated images.<br>
E.g. in <a href="https://research.nvidia.com/publication/2017-10_Progressive-Growing-of" rel="nofollow noopener"> Progressive Growing of GANs for Improved Quality, Stability, and Variation</a> the authors used “low-res” layers and stacked more layers on top of them after pretraining to increase the resolution.</p>
<p>I doubt an output of 1024x1024 will work out of the box using the DCGAN.</p>

username: Marion2
cooked: <p>Now I doubt it too, thank you for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to optimize code to run in GPU?

username: li0218
cooked: <p>I find a example about “wrap padding” in</p>
<blockquote>
<p><a href="https://github.com/pytorch/pytorch/issues/3858" class="inline-onebox" rel="noopener nofollow ugc">Wrap padding · Issue #3858 · pytorch/pytorch · GitHub</a></p>
</blockquote>
<p>and I modify the code a little to make some dimension “wrap padding” and some padding with zeros.</p>
<blockquote>
<p>def pad_circular_nd2(x: torch.Tensor, pad: int, dim, dim0) → torch.Tensor:<br>
“”"<br>
:param x: shape [H, W]<br>
:param pad: int &gt;= 0<br>
:param dim: the dimension over which the tensors are padded<br>
:return:<br>
“”"</p>
<pre><code>if isinstance(dim, int):
    dim = [dim]
if isinstance(dim0, int):
    dim0 = [dim0]

for d in dim:
    if d &gt;= len(x.shape):
        raise IndexError(f"dim {d} out of range")

    idx = tuple(slice(0, None if s != d else pad, 1) for s in range(len(x.shape)))
    x = torch.cat([x, x[idx]], dim=d)

    idx = tuple(slice(None if s != d else -2 * pad, None if s != d else -pad, 1) for s in range(len(x.shape)))
    x = torch.cat([x[idx], x], dim=d)
    pass

x0 = torch.zeros(x.size()).double().cuda()
for d in dim0:
    if d &gt;= len(x.shape):
        raise IndexError(f"dim {d} out of range")

    idx = tuple(slice(0, None if s != d else pad, 1) for s in range(len(x.shape)))
    x = torch.cat([x, x0[idx]], dim=d)

    idx = tuple(slice(None if s != d else -2 * pad, None if s != d else -pad, 1) for s in range(len(x.shape)))
    x = torch.cat([x0[idx], x], dim=d)
    pass


return x.cuda()
</code></pre>
</blockquote>
<p>However this “wrap padding” runs on CPU, though I expect it runs on GPU and makes the training much slower. Is there any way to fix it?</p>

username: ptrblck
cooked: <p>It should run on the GPU, if you pass <code>x</code> as a CUDATensor.<br>
How did you check this operation runs on the CPU?</p>
<p>There also seems to be a <code>mode='circular'</code> now in <code>F.pad</code>.</p>

username: li0218
cooked: <p>Hi ptrblck,<br>
The CPU consume is about 70% for a i9 CPU and without this “wrap padding” the CPU consume is close to 0. So I guess I might make something wrong and the code pass the data between CPU and GPU.</p>

username: li0218
cooked: <p>The F.pad circular mode works.  Thank you!</p>
Is there any "Auto-Augmentation package" for "Transfer Learning"?

username: bemorept
cooked: <p>Hi, All.</p>
<p>Is there any “Auto-Augmentation package” for “Transfer Learning”?</p>
<p>I wonder …</p>
<p>Thanks.<br>
<span class="mention">@bemoregt</span>.</p>

username: bemorept
cooked: <p>I think that Most “Auto-Augmentation packages” are suitable for “From the scratch training”.</p>
<p>May be…</p>

username: ptrblck
cooked: <p>What do you mean by “Auto-Augmentation package”?<br>
Would the augmentations be selected automatically somehow?</p>
How does one store Pytorch models without having to store the parameters?

username: pinocchio
cooked: <p>I want to only store the layers block and the computation graph.</p>
<p>To start off I want it to work for Sequential models. I want to be able to store the Sequential model architecture without having to store the parameters. Is that possible?</p>
<p>After that, is it generalizable for arbitrary models? I only want to store the shape and types of the pytorch graph. I don’t need the parameters.</p>

username: ptrblck
cooked: <p>I think the easiest way would be to store the model class definition, if you don’t want the parameters.</p>

username: pinocchio
cooked: <p>hmmm what does that mean? how does one do that?</p>
<p>I’ve been just saving the string of the Sequential model for the moment…but then I need to parse the string to extract things from it or even worse use an eval function…</p>
CPU is at 100% even when training is on GPU

username: AravindDoss
cooked: <p>I am training a model in GPU (80% load). But my CPU is at 100% load (all time). I want to know whether there exist command to limit the CPU usage. (Like restricting the core count, percentage…)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a5d5f52c60c2fec7e5fbd57ae4d9929e18905a99.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5d5f52c60c2fec7e5fbd57ae4d9929e18905a99" title="Screenshot%20from%202019-08-13%2012-08-08"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5d5f52c60c2fec7e5fbd57ae4d9929e18905a99_2_501x500.png" alt="Screenshot%20from%202019-08-13%2012-08-08" data-base62-sha1="nF3bsKTN8sklCIFnYeVKa2i5pJn" width="501" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5d5f52c60c2fec7e5fbd57ae4d9929e18905a99_2_501x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/a5d5f52c60c2fec7e5fbd57ae4d9929e18905a99_2_751x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/a5d5f52c60c2fec7e5fbd57ae4d9929e18905a99.png 2x" data-dominant-color="735D6A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-08-13%2012-08-08</span><span class="informations">766×763 97.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jGsch
cooked: <p>Reduce the <code>num_workers</code> used when you create your dataloader.</p>
<p><code>num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)</code><br>
<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader</a></p>

username: spanev
cooked: <p>What model are you training?</p>

username: AravindDoss
cooked: <p>Recurrent model… GRU.</p>

username: AravindDoss
cooked: <p>The issues was due to DataLoader as <a class="mention" href="/u/jgsch">@jGsch</a> said.</p>
Collecting gradients for multiple losses?

username: Goldname
cooked: <p>I am trying to do something like this (simplified version of my code):</p>
<pre><code class="lang-auto">for x in range(1,1000):
  output = model(data)
#Change Data
  loss = loss + F.nll_loss(output, target)

# Calculate gradients of model in backward pass
loss.backward()

# Collect gradients
final_result = final_result + myvar.grad.data
</code></pre>
<p>The problem is that a significant number of temporary variables are causing me to run out of GPU memory. Hence, is this next piece of code logically equivalent?</p>
<pre><code class="lang-auto">for x in range(1,1000):
  output = model(data)
  loss = F.nll_loss(output, target)
#Change data
# Calculate gradients of model in backward pass
  loss.backward(retain_graph=True)

# Collect gradients
  final_result = final_result + myvar.grad.data
  del loss
  del other_variables
</code></pre>
<p>If I am understand how .backward and .grad.data works correctly, then it should be equivalent. However, this is not the case for me and I’m currently looking for the bug.</p>

username: ptrblck
cooked: <p>The addition of <code>final_result = final_result + myvar.grad.data</code> won’t work, if you don’t zero out the gradients in each iteration.<br>
Currently you are accumulating:</p>
<pre><code class="lang-python">final_result = (grad0) + (grad0+grad1) + (grad0+grad1+grad2) + ...
</code></pre>
<p>since <code>loss.backward</code> will already accumulate the gradients.<br>
Another approach would be to let <code>loss.backward()</code> accumulate the gradients automatically and just assigning <code>final_result</code> after the loop.</p>
<pre><code class="lang-python">
# 1
torch.manual_seed(2809)
model = nn.Linear(1, 2, bias=False)
loss = 0.
for _ in range(1000):
    x = torch.randn(1, 1)
    target = torch.randint(0, 2, (1,))
    output = model(x)
    loss = loss + F.nll_loss(output, target)
    
loss.backward()
final_grad1 = model.weight.grad

# 2
torch.manual_seed(2809)
model = nn.Linear(1, 2, bias=False)
for _ in range(1000):
    x = torch.randn(1, 1)
    target = torch.randint(0, 2, (1,))
    output = model(x)
    loss = F.nll_loss(output, target)
    loss.backward()
    
final_grad2 = model.weight.grad

print(torch.allclose(final_grad1, final_grad2))
&gt; True
</code></pre>

username: Goldname
cooked: <p>Is this still true that the loss.backward was accumulating the gradients despite me doing “del loss” during each loop?</p>

username: ptrblck
cooked: <p>The deletion of the loss shouldn’t make a difference, as the gradients were already accumulated.</p>

username: Goldname
cooked: <p>Sorry, One thing I forgot to add: I am doing model.zero_grad() after each iteration. Does this zero out the myvar.grad.data?</p>

username: ptrblck
cooked: <p><code>model.zero_grad</code> will zero out the gradients of all internal parameters.<br>
If you’ve registered <code>self.myvar = nn.Parameter(...)</code>, it should be also zeroed out.</p>