akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Are there any logs from PyTorch building?

username: Darek554
cooked: <p>Hi.<br>
I’ve started building pytorch from source in developer mode, but i had to leave home for a while and i’ve set PC shutdown in an hour - when building process was on 95% stage - and now i don’t know whether compilation was successful <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: tom
cooked: <p>The easiest thing might be to re-run the thing. Typically cmake (and/or ccache if set up) should make it a breeze if it has been done before (see CONTRIBUTING.md for details).</p>

username: Darek554
cooked: <p>Ok, i’ll try, i thought that if i rerun build pytorch will be building few hours again, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tom
cooked: <p>So my variant of your problem is: when I compile (64-bit) PyTorch on the RaspPi (with 4GB), most of the time, I can use 4 cores (-j4). So I compile with that until I hit OOM, then I run it with -j1 through the large source files, hit ^C when I get bored and go back to -j4 for the remainder. Seems to work reasonably well. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Numba cuda and pytorch distribution may conflict

username: torch_torch
cooked: <p>I’m trying to use numba and pytorch distribution simultaneously. When I create new tensor on GPU, I got cuda initialization error:<br>
Here is my code:</p>
<pre><code class="lang-auto">import torch
import numpy as np
import torch.distributed as dist
from torch.multiprocessing import Process
from numba import cuda
import os
def func(idx):
    a = torch.randn([10, 10, 10], device=‘cuda’)
return
def init_process(idx, size, fn, backend=‘NCCL’):
    os.environ[‘MASTER_ADDR’] = ‘127.0.0.1’
    os.environ[‘MASTER_PORT’] = ‘29500’
    dist.init_process_group(backend, rank=idx, world_size=size)
    fn(idx)
    return
if  **name**  == “ **main** ”:
    processes = []
    for idx in range(2):
        p = Process(target=init_process, args=(idx, 2, func))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>Here is error report:</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=47 error=3 : initialization error
Process Process-2:
Traceback (most recent call last):
File “/home/sss/anaconda3/envs/torch_new/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/home/sss/anaconda3/envs/torch_new/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/sss/Desktop/Experiment/test.py”, line 14, in init_process
fn(idx)
File “/home/sss/Desktop/Experiment/test.py”, line 8, in func
a = torch.randn([10, 10, 10], device=‘cuda’)
File “/home/sss/anaconda3/envs/torch_new/lib/python3.8/site-packages/torch/cuda/ **init** .py”, line 190, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:47
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=47 error=3 : initialization error
Process Process-1:
Traceback (most recent call last):
File “/home/sss/anaconda3/envs/torch_new/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/home/sss/anaconda3/envs/torch_new/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/sss/Desktop/Experiment/test.py”, line 14, in init_process
fn(idx)
File “/home/sss/Desktop/Experiment/test.py”, line 8, in func
a = torch.randn([10, 10, 10], device=‘cuda’)
File “/home/sss/anaconda3/envs/torch_new/lib/python3.8/site-packages/torch/cuda/ **init** .py”, line 190, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:47
</code></pre>
<p>When I comment “from numba import cuda”, there is no error reported. Since I really need numba and cuda, I can’t comment them. Can anyone solve my problems?</p>

username: tom
cooked: <p>You could either see which version of cuda numba uses see if PyTorch offers that, too (or vice versa) or you could self-compile one or the other or both.</p>

username: torch_torch
cooked: <p>Thank you for the reply.<br>
I compiled a new numba for pytorch. It worked !<br>
Thank you !</p>
Incorrect results after loading saved quantized model

username: deepak_mangla
cooked: <p>I have quantized and saved the model using this code -</p>
<pre><code class="lang-auto">model = Net()
model.load_state_dict(torch.load('model.pth', map_location=torch.device('cpu')))                                
model.qconfig = torch.quantization.default_qconfig              
model.eval()
torch.quantization.prepare(model, inplace=True)
evaluate(model)
torch.quantization.convert(model, inplace=True)
model.eval()
x = evaluate(model)
torch.save(model.state_dict(), 'model_q.pth')
</code></pre>
<p>and loading the model like this -</p>
<pre><code class="lang-auto">model2 = Net()
torch.qconfig = torch.quantization.default_qconfig              
model2.eval()
torch.quantization.prepare(model2, inplace=True)
torch.quantization.convert(model2, inplace=True)
model2.eval()
model2.load_state_dict(torch.load('model_q.pth'))
xQ = evaluate(model2)
</code></pre>
<p>Now x and xQ are different. I checked the parameters of both ‘model’ and ‘model2’. Parameters are same.</p>
<pre><code class="lang-auto">for i in range(len(list(model.parameters()))):
    print(np.equal(list(model.parameters())[i].detach().numpy(), list(model2.parameters())[i].detach().numpy()))
</code></pre>
<p>All parameters are equal.</p>
<p>Is there anything incorrect with my method of loading or saving? or some bug in PyTorch?</p>

username: deepak_mangla
cooked: <p>I  compared output of all layers of original model and loaded  model. I found output of BatchNorm2d was different.</p>
<p>The problems is Pytorch wasn’t saving ‘scale’ and ‘zero_point’ of unfused QuantizedBatchNorm in checkpoints. Two solutions -</p>
<ul>
<li>Save these values as pickle when saving model. While loading model, load pickle and add scale and zero point to  QuantizedBatchNorm layers.</li>
<li>Fuse BatchNorm with Convolution.</li>
</ul>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/deepak_mangla">@deepak_mangla</a>, thanks for the report.  I created <a href="https://github.com/pytorch/pytorch/issues/43774" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/43774</a> to verify correct behavior.  Please let us know if you have a repro on a toy model.</p>
Duplicated Multi-branch results

username: Erica_Zheng
cooked: <p>I’m using “ModuleList” to create multiple branches, but it turns out all branches output the same result. <a href="https://github.com/zzzheng/ONE-implementation-mini" rel="nofollow noopener">Mini repo</a>. Thank you!</p>
<pre><code class="lang-auto">    def _make_branches(self, *layers):
        branches = []
        for i in range(self.num_branches):
            branch = nn.Sequential(*layers)
            branches.append(branch)
            branches = nn.ModuleList(branches)
        return branches
</code></pre>

username: ptrblck
cooked: <p>The <code>_make_branches</code> method seems to take already initialized modules and creates <code>nn.ModuleList</code>s using them.<br>
Since each of the passed <code>layers</code> will contain the same parameters, it’s expected that the output would be the same.<br>
If you want to create branches with new layers, you would have to initialize new modules before adding them to the <code>nn.ModuleList</code>s.</p>

username: Erica_Zheng
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the explanation.<br>
I wonder Is it possible to initialize dynamic number of new modules?</p>

username: ptrblck
cooked: <p>If you would like to initialize a specific number of layers given a passed argument to the model, you could replace <code>*layers</code> with the initialization of new modules:</p>
<pre><code class="lang-python">    def _make_branches(self, num_layers):
        branches = []
        for i in range(num_layers):
            branch = nn.Sequential(nn.Linear(1, 1)) # replace this with any new initialization
            branches.append(branch)
            branches = nn.ModuleList(branches)
        return branches
</code></pre>

username: Erica_Zheng
cooked: <p>Great advice. Thank you so much!</p>
Copying indexed slices of a tensor to an empty tensor

username: Anej_Sterle
cooked: <p>Hi guys, I’m new here and I have a problem. I have a tensor with data <code>mat</code> and an empty tensor of the same shape. I also have a 1D index tensor ‘ind’, which has indices of slices I want to copy. What I want to do is to look at index of a slice I want and put it in the same spot in the empy tensor.</p>
<pre><code class="lang-auto">mat = torch.arange(0,16).reshape(4,2,2)
empty = torch.zeros((4,2,2))
ind = torch.tensor([0,2])
</code></pre>
<p>The idea is to do something like this ( which I know is wrong, I’m new to this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:">  ):</p>
<pre><code class="lang-auto">empty[ind] = mat[ind]
</code></pre>
<p>so here the empty matrix should have first and third slices, while second and fourth slices should<br>
remain empty ( or in this case filled with zeros ):</p>
<pre><code class="lang-auto">torch.tensor( [ [ [ 0, 1 ],
                         [ 2, 3 ] ],
                       [ [ 0 , 0 ],
                         [ 0 , 0 ] ],
                        [ [ 8, 9 ],
                         [10, 11 ] ],
                         [ [ 0 , 0 ],
                         [ 0 , 0 ] ] ] )
</code></pre>
<p>How do I solve this?<br>
Thank you for your answers.</p>

username: ptrblck
cooked: <p>Your code seems to be working fine, if you get rid of the type mismatch error by calling <code>float()</code> on <code>mat</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-python">mat = torch.arange(0,16).reshape(4,2,2).float()
empty = torch.zeros((4,2,2))
ind = torch.tensor([0,2])
empty[ind] = mat[ind]
print(empty)
&gt; tensor([[[ 0.,  1.],
           [ 2.,  3.]],

          [[ 0.,  0.],
           [ 0.,  0.]],

          [[ 8.,  9.],
           [10., 11.]],

          [[ 0.,  0.],
           [ 0.,  0.]]])
</code></pre>

username: Anej_Sterle
cooked: <p>Thank you so much! I think I was being too vague, with calling tensor as an index. Now the code is working properly <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
A problem regarding training on ImageNet dataset

username: yichao_Liu
cooked: <p>Hi,<br>
I just use NAS to search a architecture on ImageNet, but I have a searching result like 50% acc, testing result always 1%. I don’t why this happens. I checked the dataset, there is no mismatch for different classes. I also train the transofrmed architecture searched on CIFAR10 and 100 successfully. But still got 1% acc on testset of ImageNet. It’s very weird. Is it a overfitting problem? Can this overfitting lead to 1% acc ?! I’m pleased if anyone can help me. Thank you so much.</p>
How to add pytorch as a requirement for a windows python package

username: George3d6
cooked: <p>I have a python package which depends on pytorch and which I’d like windows users to be able to install via pip (the specific package is: <a href="https://github.com/mindsdb/lightwood" rel="nofollow noopener">https://github.com/mindsdb/lightwood</a>, but I don’t think this is very relevant to my question).</p>
<p>What are the best practices for going about this ?</p>
<p>Are there some project I could use as examples ?</p>
<p>It seems like the pypi hosted version of torch &amp; torchvision aren’t windows compatible and the “getting started” section suggests installing from the custom pytorch repository, but beyond that I’m not sure what the ideal solution would be to incorporate this as part of a setup script.</p>

username: Shisho_Sama
cooked: <p>Pytorch is available on windows. did you check <a href="http://Pytorch.org" rel="nofollow noopener">Pytorch.org</a>?</p>

username: George3d6
cooked: <p>Hmh, maybe I didn’t phrase my question correctly.</p>
<p>Could you tell me what you thought I was asking :?</p>

username: Shisho_Sama
cooked: <blockquote>
<p>It seems like the pypi hosted version of torch &amp; torchvision aren’t windows compatible and the “getting started” section suggests installing from the custom pytorch repository, but beyond that I’m not sure what the ideal solution would be to incorporate this as part of a setup script.</p>
</blockquote>
<p>Reading  this, I thought you dont know about <code>Pytorch.org</code> that provides 100% windows compatible pip packages which you can use!<br>
You can find all the packages here is as well <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>

username: George3d6
cooked: <p>Yes, the issue with that is that you can’t list them as a dependency, since pip will complain about downloading stuff from sources other than pypi and the version that pytroch has on pypi doesn’t seem to work on windows.</p>

username: peterjc123
cooked: <p>Please see my answers here: <a href="https://github.com/pytorch/pytorch/issues/32485#issuecomment-577113812" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32485#issuecomment-577113812</a>.</p>

username: George3d6
cooked: <p>I did see the answer, problem being that I can’t do that for a pypi package, setup.py doesn’t support find-links anymore and if I use the <code>@</code> syntax or some other syntax to set the repository it will complain about “Can’t depend on packages not hosted on pypi”.</p>
<p>The workaround I have right now is a manual install on windows as part of the setup script by calling a pip subprocess and trying the various versions: <a href="https://github.com/mindsdb/lightwood/blob/master/setup.py#L41" rel="nofollow noopener">https://github.com/mindsdb/lightwood/blob/master/setup.py#L41</a></p>
<p>But it doesn’t look like it’s particularly best practiy to me.</p>

username: WaterKnight
cooked: <p>Hi, did you find a better solution?</p>
Is there a Pytorch implementation of RAdam + LookAhead?

username: pinocchio
cooked: <p>I was looking at <a href="https://arxiv.org/abs/1907.08610v1" rel="nofollow noopener">the new paper by Geoffrey Hinton</a> and was wondering if it’s been implemented. Look at the article:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="32" height="32">
      <a href="https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d" target="_blank" title="12:11AM - 20 August 2019" rel="nofollow noopener">Medium – 20 Aug 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/330;"><img src="https://miro.medium.com/max/1171/1*Ll7kbj7FKVzJk-bLqdR53g.jpeg" class="thumbnail"></div>

<h3><a href="https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d" target="_blank" rel="nofollow noopener">New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam +...</a></h3>

<p>A new paper in part by the famed deep learning researcher Geoffrey Hinton introduces the LookAhead optimizer(“LookAhead optimizer: k steps…</p>

  <p><span class="label1">Reading time: 9 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>Cross posted:</p>
<ul>
<li><a href="https://www.quora.com/unanswered/Is-there-a-Pytorch-implementation-of-RAdam-LookAhead" rel="nofollow noopener">https://www.quora.com/unanswered/Is-there-a-Pytorch-implementation-of-RAdam-LookAhead</a></li>
</ul>

username: LeviViana
cooked: <p>Here: <a href="https://github.com/mgrankin/over9000" rel="nofollow noopener">https://github.com/mgrankin/over9000</a></p>

username: Tony-Y
cooked: <p><a href="https://github.com/lonePatient/lookahead_pytorch/issues/1#issuecomment-523809288" rel="nofollow noopener">RAdam+LA</a></p>
<pre><code class="lang-python">from radam import RAdam
from optimizer import Lookahead
base_optim = RAdam(model.parameters(),lr = 0.001) 
optimizer =  Lookahead(base_optim, k=5, alpha=0.5)
</code></pre>
<p><a href="https://github.com/LiyuanLucasLiu/RAdam/blob/master/radam.py" rel="nofollow noopener">RAdam</a> | <a href="https://github.com/cybertronai/pytorch-lamb/blob/master/pytorch_lamb/lamb.py" rel="nofollow noopener">LAMB</a> | <a href="https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20" rel="nofollow noopener">Ralamb</a> | <a href="https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py" rel="nofollow noopener">Lookahead</a></p>

username: drevicko
cooked: <p>The RAdam link seems broken (a 404 on github - maybe it’s not public?)</p>
<p>Anyway, <a href="https://github.com/LiyuanLucasLiu/RAdam" rel="nofollow noopener">this link</a> works.</p>
Query regarding AdaptiveAvgPool3d

username: ParthTrehan
cooked: <p>I have an input tensor like x = torch.rand(2,3,13,224,224) and after passing through this tensor from my model I get an output shape as torch.Size([2, 512, 13, 14, 14]). I then further want to reduce 14x14 to 1x1 so it looks like this torch.Size([2, 512, 13, 1, 1]) and then have a max of the 13 features to get an output tensor like torch.Size([2, 512, 1, 1, 1]).</p>
<p>I am pretty new to PyTorch and I found way to do this,</p>
<p>gap = torch.nn.AdaptiveAvgPool3d((None,1,1))<br>
x = model(torch.rand(2,3,13,224,224))<br>
x = gap(x)<br>
x = torch.max(x, 2, keepdim=True)[0]</p>
<p>I want to know if this is the correct way to do this or not. I want to reduce all the 14x14 values to 1x1 using average pooling, my question is am i using AdaptiveAvgPool3d in a correct way or should I use AdaptiveAvgPool2d in some way for reducing 14x14?</p>

username: ptrblck
cooked: <p>Your code looks alright.<br>
Alternatively, since your desired output shape after the average pooling operation is <code>[batch_size, 512, 13, 1, 1]</code>, you could also directly apply the <code>mean()</code> operation on dim3 and dim4 in a similar way as you’ve already done for <code>torch.max</code> (but this should yield the same output as your current approach).</p>
To minimize gpu memory usage, how should I sum all the losses?

username: oasjd7
cooked: <p>To minimize gpu memory usage, how should I sum all the losses?</p>
<pre><code class="lang-auto">for epoch in range(epochs):
      for step, data in enumerate(dataloader):
          ...
          total_loss = criterion(input, target) # 1st loss

          second_loss= criterion(input2, target2).item() # 2nd loss
          total_loss += second_loss.item()
          del second_loss

          third_loss = criterion(input3, target3).item() # 3rd loss
          total_Loss += third_loss.item()
          del third_loss
          ...
          optimizer.zero_grad()
          total_loss.backward()
          optimizer.step()

</code></pre>

username: ptrblck
cooked: <p>In your current approach you are adding <code>second_loss</code> and <code>third_loss</code> as Python floats to the <code>total_loss</code> (detached floating point constants), which won’t have any effect on the backward pass.<br>
I’m not familiar with your use case, but if you want these losses to participate in the gradient calculation, you shouldn’t call <code>.item()</code> on them.</p>
<p>Are you running out of memory, if you are adding these losses directly to <code>total_loss</code>, which would be weird as I would assume they should use very little memory compared to the complete model and the intermediate tensors (but it also depends, which <code>criterion</code> you are using)?</p>
Named Tensor Tutorial linked wrong

username: RaLo4
cooked: <p>Hey,<br>
just wanted to let you know that,<br>
this banner for the <strong>Named Tensor tutorial</strong> on the <a href="https://pytorch.org/tutorials/" rel="noopener nofollow ugc">pytorch tutorials page</a> links to the <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="noopener nofollow ugc">memory format tutorial</a> instead of the <a href="https://pytorch.org/tutorials/intermediate/named_tensor_tutorial.html" rel="noopener nofollow ugc">named tensor tutorial</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c61cd127c0c17f6d8a59b4acaa42700b6bf4b787.png" data-download-href="https://discuss.pytorch.org/uploads/default/c61cd127c0c17f6d8a59b4acaa42700b6bf4b787" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c61cd127c0c17f6d8a59b4acaa42700b6bf4b787_2_690x175.png" alt="image" data-base62-sha1="sgAhjGnUuuph1pI4kN0zGiV5qF9" width="690" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c61cd127c0c17f6d8a59b4acaa42700b6bf4b787_2_690x175.png, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c61cd127c0c17f6d8a59b4acaa42700b6bf4b787.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c61cd127c0c17f6d8a59b4acaa42700b6bf4b787.png 2x" data-dominant-color="ECECEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">871×222 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This link works fine though:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/022b2fcd59d5067bff5dd19a72cfafc78de93f93.png" alt="image" data-base62-sha1="jbtT6tcwugzSiT4Y69krRDgXzt" width="280" height="73"></p>

username: ptrblck
cooked: <p>Thanks for reporting this issue!<br>
<a class="mention" href="/u/alband">@albanD</a> do you know, who might be able to fix the links on the website?</p>

username: albanD
cooked: <p>Not sure,<br>
Opening an issue on the website repo might do it? <a href="https://github.com/pytorch/pytorch.github.io">https://github.com/pytorch/pytorch.github.io</a></p>
<p>I’ll also point this to the relevant person to see if there is a better way.</p>
<p>Thanks for the feedback <a class="mention" href="/u/ralo4">@RaLo4</a> !</p>

username: Brianjo
cooked: <p>Hey thanks all. I’m fixing this now. <a href="https://github.com/pytorch/tutorials/pull/1140" rel="nofollow noopener">https://github.com/pytorch/tutorials/pull/1140</a></p>
<p>Cheers,</p>
<p>Brianjo</p>

username: ptrblck
cooked: <p>Thanks! I was looking for the repo of the website, but apparently missed the <code>pytorch.github.io</code> one <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Learning time increase over-time

username: David_Silva
cooked: <p>Hello,</p>
<p>I am training several networks at the same time. With the passing of episodes the training time takes longer. From the 10th to the 90th episode the total training time of all networks goes from 6.42 seconds to 13.7 seconds. What may be the reason for this increase? Thank you in advance</p>
<pre><code class="lang-auto"># dqn class
    def updateSARSD(self, state, action, reward, next_state, done):
        self.remember(state, action, reward, next_state, int(done) )
        # Optimize the target network
        self.learn()

    def learn(self):

        if self.memory.mem_cntr &lt; self.batch_size * 2:
            return

        self.q_eval.optimizer.zero_grad()
        self.replace_target_network()

        states, actions, rewards, next_states, dones = self.sample_memory()

        indices = np.arange(self.batch_size)
        q_pred = self.q_eval.forward(states)[indices,actions]
        q_next = self.q_next.forward(next_states).max(dim=1)[0]
        q_next[dones] = 0.0
        q_target = rewards +  self.gamma * q_next

        loss = self.q_eval.loss(q_target, q_pred).to(self.q_eval.device)
        loss.backward()
        self.q_eval.optimizer.step()

        self.learn_step_counter += 1
        self.decrement_epsilon()

        return loss

    def replace_target_network(self):

        if self.learn_step_counter % self.replace_target_cnt == 0:
            self.q_next.load_state_dict(self.q_eval.state_dict())

    def remember(self, *args):
        self.memory.push(*args)

    def sample_memory(self):

        state, action, reward, new_state, done = self.memory.sample(self.batch_size)

        states = torch.tensor(state, dtype=torch.float, device=self.q_eval.device)
        rewards = torch.tensor(reward, dtype=torch.float, device=self.q_eval.device)
        dones = torch.tensor(done)
        actions = torch.tensor(action)
        next_states = torch.tensor(new_state, dtype=torch.float, device=self.q_eval.device)

        return states, actions, rewards, next_states, dones

# memory_buffer class

    def push(self, state, action, reward, new_state, done):

        index = self.mem_cntr % self.mem_size
        self.state_memory[index] = state
        self.action_memory[index] = action
        self.reward_memory[index] = reward
        self.new_state_memory[index] = new_state
        self.terminal_memory[index] = done
        self.mem_cntr += 1

    def sample(self, batch_size):

        max_mem = min(self.mem_cntr, self.mem_size)
        batch = np.random.choice(max_mem, batch_size, replace=False) # replace = False to not repeat any memory

        states = self.state_memory[batch]
        actions = self.action_memory[batch]
        rewards = self.reward_memory[batch]
        next_states = self.new_state_memory[batch]
        dones = self.terminal_memory[batch]

        return states, actions, rewards, next_states, dones
</code></pre>

username: iffiX
cooked: <p>Have you detached the “action” tensor before you storing it with <code>remember</code> called by <code>updateSARSD</code>?</p>
Just open-sourced AdverTorch, a Pytorch based Toolbox for Adversarial Robustness Research

username: gwding
cooked: <p>Hi folks,</p>
<p>We (Borealis AI, Canada) have just open-sourced AdverTorch, a pytorch based toolbox for adversarial robustness research. It is still work in progress, but contains basic functionalities on attacks, defenses, BPDA, and examples of adversarial training. You’re welcomed to try it out, and let us know for questions and suggestions.</p>
<p>The repo is at <a href="https://github.com/borealisai/advertorch" rel="nofollow noopener">https://github.com/borealisai/advertorch</a></p>

username: vmirly1
cooked: <p>That’s great! Thanks for sharing!</p>

username: Alon
cooked: <p>Thanks for this, I was looking for a simple way to apply adversarial samples during training!<br>
I know it is intended for adversarial robustness, but do you think it can help classification accuracy in a similar way that AdvProp (<a href="https://arxiv.org/pdf/1911.09665v1.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1911.09665v1.pdf</a>) does?</p>
How to understand GPU status and training speed

username: Yi_Zhang
cooked: <p>Hi,</p>
<p>I have done some experiments on multi-gpu training, and I feel a bit confused about the relationship between gpu status and training speed.<br>
My original expectation was the slowest GPU would be the bottleneck for training speed, since local parameters need to sync in each step. But my experiment result proves I was wrong. Anyone can explain why the busiest GPU doesn’t slow down training speed as expected?</p>
<p>I’m using all_reduce to sync parameters:</p>
<pre><code class="lang-auto">for param in model.parameters():
     if param.requires_grad and param.grad is not None:
             torch.distributed.all_reduce(param.grad.data, 
                                          op=torch.distributed.ReduceOp.SUM)
</code></pre>
<p>I measured GPU-Util, and my guess is higher GPU-Util means the GPU is busier and should be slower for training same size of batches. More experiment result for training the same dataset:</p>
<p>test 1: 4 GPU with about 95% GPU-Util - training time is 35 sec<br>
test 2: 2 GPU with 0% GPU-Util, 2 GPU with 90% GPU-Util - training time is 18 sec<br>
test 3: 3 GPU with 0% GPU-Util, 1 GPU with 97% GPU-Util - training time is 15 sec<br>
test 4: 4 GPU with about 0% GPU-Util - training time is 10 sec</p>
<p>If the slowest GPU was the bottleneck, then training time of test 2 and test 3 should be similar as test 1. But how to understand this result?<br>
Please also let me know if you notice any mistake in my experiment.</p>
<p>Thanks.</p>

username: mrshenli
cooked: <p>One reason might be that CUDA GPU shows 100% utilization when running NCCL collective communications, even if it is actually block waiting for other peers to join and doing nothing. So the GPU utilization number cannot faithfully represent how busy a GPU is.</p>

username: Yi_Zhang
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a>, thanks for reply. Then do you know what could be a better way to check a GPU’s status? For example, to compare training speed on each GPU when using multi-gpu training?</p>

username: mrshenli
cooked: <p>One option might be using nvprof and then visualize the result. It will show time consumed by different comp and comm ops. See the following links:</p>
<ol>
<li><a href="https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/" rel="nofollow noopener">https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/</a></li>
<li><a href="https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/" rel="nofollow noopener">https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/</a></li>
</ol>
<p>We are also working on extending autograd profiler to work with DDP, but we don’t have a target date for it yet.</p>
A Wild Tensor Appears after `to()`

username: chwan-rice
cooked: <p>The following code runs out of memory on my machine.</p>
<pre><code class="lang-python">import torch

a = torch.ones(2000, 1000, 1000)
b = a.clone()
a.to('cuda:0')
del a
torch.cuda.empty_cache()
b.to('cuda:0')
</code></pre>
<p>However, the following code works well</p>
<pre><code class="lang-python">import torch

a = torch.ones(2000, 1000, 1000)
b = a.clone()
a = a.to('cuda:0')
del a
torch.cuda.empty_cache()
b.to('cuda:0')
</code></pre>
<p>Their difference is line 5. For the first code, I have no idea what to do to release the memory. Can anyone help me? Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>.to()</code> operation is out of place, so the first code does weird stuff for sure because it send data to the GPU and then discard it instantly.<br>
The second one actually holds on to it and is only deleted on the next line.</p>
<p>Why one runs out of memory and not the other is due to fragmentation? Not sure what would be the reason here.</p>
Convert 0-dim torch::Tensor to torch::Scalar

username: helange23
cooked: <p>This is a stupid question. I am trying to do a very simple thing. How do I convert a 0-dim <code>torch::Tensor</code> into a <code>torch::Scalar</code>.</p>
<p>Basically, what I am trying to achieve is the following:</p>
<blockquote>
<p>auto center1 = torch::linspace(a[0], a[1], K+1);</p>
</blockquote>
<p>My problem is that <code>a[0]</code> is a 0-dim Tensor but <code>linspace</code> requires a <code>torch::Scalar</code>.</p>
<p>When I tried to simply cast to a scalar, I get the following error:</p>
<blockquote>
<p>error: no suitable user-defined conversion from “at::Tensor” to “c10::Scalar” exists</p>
</blockquote>

username: helange23
cooked: <p>I figured this finally out.</p>
<p>The correct way of doing it, is: <code>a[0].item&lt;float&gt;()</code></p>
Redistributing PyTorch

username: danieldk
cooked: <p>First of all, thanks a lot for PyTorch. I have been using PyTorch in my research and I love it!</p>
<p>I want to package precompiled <code>libtorch</code> in a Linux distribution. I know that PyTorch itself is licensed under the BSD license. However, the binary builds include parts of Intel MKL and also parts of the CUDA toolkit. I wanted to ask if the resulting binaries are redistributable. In other words, is it OK to take the libtorch [1] archive, repackage it in the distribution’s native package format and provide it for download?</p>
<p>[1] E.g. <a href="https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.6.0.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.6.0.zip</a></p>
Diag Matrix for images

username: g33sean
cooked: <p>Hey would you make a diagonal matrix with a size of 64x64x3?</p>

username: JuanFMontesinos
cooked: <pre><code class="lang-auto">torch.stack([torch.diagflat(numbers) for _ in range(3)],dim=2)
</code></pre>
<p><a href="https://pytorch.org/docs/stable/generated/torch.diagflat.html?highlight=diag#torch.diagflat" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.diagflat.html?highlight=diag#torch.diagflat</a></p>
Pytorch Onnx input size

username: Morray_AH
cooked: <p>Hi Everyone,<br>
I would like to know how to unfreeze the input size of an exported Onnx model?<br>
Example: <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html</a></p>
<p>Super-Resolution is an image-to-image translation, i can’t see how i would consume this model without resizing the input to 224 x 224 , and this mean the input quality will decrease<br>
thank you.</p>

username: Morray_AH
cooked: <p>Not to spam, but i’ll appreciate any help…</p>
Can I perform RandomizedSearchCV for multiple independent models in parallel?

username: fkempf92
cooked: <p>I was wondering if there is a way to train multiple independent neural networks in parallel on CPU only with random grid search?</p>
<p>Imagine a trivial example of a network like stated below (this is just an example):</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.Linear = nn.Linear(1, 1, bias=True)

    def forward(self, x):
        out = self.Linear(x)
        return out
</code></pre>
<p>I subsequently perform some form of grid search:</p>
<pre><code class="lang-auto">from skorch import NeuralNetRegressor
from skorch.callbacks import EarlyStopping
from sklearn.model_selection import RandomizedSearchCV

es = EarlyStopping(monitor='valid_loss', patience=5, threshold=1e-4, threshold_mode='abs', lower_is_better=True)

model = NeuralNetRegressor(
        module= NeuralNet,
        criterion=nn.MSELoss,
        optimizer=torch.optim.SGD,
        optimizer__momentum=0.95,
        optimizer__weight_decay=0.2,
        max_epochs=200,
        batch_size=32,
        lr=1e-4,
        callbacks=[es]
    )
model.verbose = 0
params_grid = {
        'lr': [1e-5, 1e-4],
        'optimizer__momentum': [0.9, 0.95, 0.99],
        'optimizer__weight_decay': [0.1, 0.9, 1.5]
}

gs = RandomizedSearchCV(model, params_grid, refit=True, cv=3,
n_jobs=5, n_iter=10, random_state=1234, scoring='neg_mean_squared_error')
gs.fit(x_train, y_train)

</code></pre>
<p>Can I perform the above simultaneously with let’s say three different models all at the same time, where all models are completely independent? i.e. can I perform three independent grid searches for three independent models at the same time if I have let’s say 80 cores or so and each grid search uses 20 cores.</p>
<p>My approach was to define a function that performs the grid search, and then execute the function in parallel but it did not seem to speed up any of my calculations? How would you approach a problem like this?</p>
<p>I hope this is not a stupid question!</p>
Autocast with normalization layers

username: Konpat_Ta_Preechakul
cooked: <p>Normalization layers, such as batch normalization and group normalization, comprise of reduce operations which, as Nvidia’s Apex <a href="https://nvidia.github.io/apex/amp.html" rel="nofollow noopener">suggests</a>, should be done in full FP32.</p>
<p>However, I have not seen these layers’ operations in the <a href="https://pytorch.org/docs/stable/amp.html#op-eligibility" rel="nofollow noopener">Ops eligibility</a> page. Hence, the behavior of these layers are non-changing which would result in FP16 outputs.</p>
<p>Is this a valid concern? It was not mentioned in the documentation.</p>

username: ptrblck
cooked: <p><code>BatchNorm</code> layers should keep their parameters in FP32 and an FP16 input will be transformed to FP32 before the operations are applied.</p>

username: Konpat_Ta_Preechakul
cooked: <p>Could you point out how to read the documentation and get this kind of wisdom?</p>

username: tom
cooked: <p>group_norm autocasting might be taken as a hint, but you are right that the page you link doesn’t mention batch norm (your chance to be a hero to others having the same concern! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>

username: Konpat_Ta_Preechakul
cooked: <p>I can verify that the output from batch normalization is not autocasted to float32, unlike that from layer normalization, which is right according to the documentation, only layer_norm is in the “autocast to float32” list.</p>
<p>I still cannot agree that this is the expected behavior.</p>
<p>Pytorch version: 1.6.0</p>
<pre><code class="lang-auto">with autocast():
    net = nn.BatchNorm1d(4).cuda()
    x = torch.randn(2, 4).half().cuda()
    y = net(x)
    print(y.dtype)  # torch.float16
</code></pre>
<pre><code class="lang-auto">with autocast():
    net = nn.LayerNorm(4).cuda()
    x = torch.randn(2, 4).half().cuda()
    y = net(x)
    print(y.dtype)  # torch.float32
</code></pre>
Pytorch, GRU memory leak

username: GoingMyWay
cooked: <p>I am struggling with using GRU with the memory leak, here is the demo code</p>
<pre><code class="lang-python">import tqdm
import torch as th
import torch.nn as nn
import torch.nn.functional as F

from torch.optim import Adam


class MyModel(nn.Module):
    def __init__(self, input_shape=348):
        super(MyModel, self).__init__()
        self.arrange = th.arange(55).int().cuda()

        self.fc1 = nn.Linear(input_shape, 32)
        self.rnn = nn.GRUCell(32, 32)
        self.fc2 = nn.Linear(32, 10*55)

    def forward(self, inputs, rnn_state):
        x = F.relu(self.fc1(inputs))
        rnn_state = rnn_state.reshape(-1, 32)
        h = self.rnn(x, rnn_state)
        logits = self.fc2(h)
        w = th.softmax(logits, dim=-1)
        alpha = th.div((th.argmax(w, dim=-1)+1).float(), float(w.shape[-1]))
        alpha_new = th.ceil(alpha*55-1).int()
        logits = logits.view(-1, 10, 55)
        values, indices = th.sort(logits, dim=2)
        masks = (self.arrange &lt;= alpha_new[..., None]).float()
        masks = masks[:, None, :]
        values = th.sum(values * masks, dim=2)
        return values, h

def main():
    model = MyModel(input_shape=348)
    model.cuda()

    hidden_in = th.rand(1, 32).cuda()
    hidden_in = hidden_in.unsqueeze(0).expand(32, 27, -1)
    optimiser = Adam(params=model.parameters())
    for i in range(10000):
        inputs, inputs2 = th.rand(27*32, 348).cuda(), th.rand(27*32, 348).cuda()
        print(th.cuda.memory_summary())
        outputs, hidden_in = model(inputs, hidden_in)
        print(th.cuda.memory_summary())
        outputs2, _ = model(inputs2, hidden_in)

        loss = th.mean((outputs - outputs2.detach()) ** 2)
        optimiser.zero_grad()
        loss.backward(retain_graph=True)
        optimiser.step()


if __name__ == '__main__':
    main()
</code></pre>
<p>While training, the GPU memory usage is increasing. However, after removing the RNN, GPU memory usage is stable.</p>

username: tom
cooked: <p>If you use</p>
<pre><code class="lang-auto">retain_graph=True
</code></pre>
<p>you in every loop, it should not surprising that nothing gets freed up (just from the literal meaning of “retain” – not to say you should have seen known before but maybe it is a good thing to remember).<br>
What happens is if you don’t have that is that the thing says “trying to backward twice blabla”, and that is true!<br>
In the end, you need to detach hidden_in at the end of your loop before starting the next to “forget what has happened”.</p>
<p>Best regards</p>
<p>Thomas</p>
What is the proper way to train regression network with certain output range?

username: 032june
cooked: <p>I want to make a simple regression network to estimate certain value.<br>
The value should be in the range [0, 2pi], since it is a angle.<br>
I tried to use sigmoid() at the end of the network, and multiply 2pi on it.</p>
<p>But backprop works awful at its head an tail, and gradients are ruined.<br>
Is there any proper way to restrict the prediction value of network, without harm its optimization?</p>

username: Henry_Chibueze
cooked: <p>In my opinion the output range has to be specified in your training data and after that you can then check the accuracy of the target with respect to the given features.</p>
<p>Also your cost function plays a role in this too<br>
Eg: their are cost functions that a suitable for regression models like MSEloss, RMSEloss, and basically other loss functions you’d normally use to estimate error back in high school when dealing with continuous data.</p>
<p>I also suggest that in this scenario instead of using sigmoid function, use softmax in your model output layer. If your training data target is set with the values ranging from (0 to 2) then I suggest that you also normalize them to be in the range of (0 to 1) too,  so that after your model is being trained and makes a prediction you can then apply a reverse of the training target normalization on that prediction, after that u can approximate the values to the nearest whatever u want and see how well it trained.</p>
<p>Hope this helped?</p>

username: ilkarman
cooked: <p>Perhaps keep the sigmoid output constraining to (0,1) but scale your target by 1/2.pi</p>
Logging gradients on each iteration

username: TinfoilHat0
cooked: <p>Hi,</p>
<p>I’d like to log gradients obtained during training to a file to analyze/replicate the training later.<br>
What’s a convenient way of doing this in PyTorch ?</p>

username: tom
cooked: <pre><code class="lang-python">grads = {n:p.grad.cpu() for n, p in model.named_parameters()}
</code></pre>
<p>gives you the grads of <code>model</code>'s parameters. You can now store them away, either directly on disk (<code>torch.save</code> or, if you feel fancy, hdf5) or keep a list of them (when moving to cpu probably is a good idea, so I threw that in above) or so.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ilkarman
cooked: <p>If you use tensorboardX you could also log the grades to tensorboard to visualise:</p>
<pre><code class="lang-auto">from tensorboardX import SummaryWriter
logger = SummaryWriter(LOG_DIR)
    
def log_gradients_in_model(model, logger, step):
    for tag, value in model.named_parameters():
        if value.grad is not None:
            self.logger.add_histogram(tag + "/grad", value.grad.cpu(), step)
</code></pre>
Can distributed training be used to solve the CUDA out of memory error

username: Aryaman_Sriram
cooked: <p>I am finetuning DialoGPT-2 and VGG-16 together for a particular task but dude to the heavy nature of both models, the training loop is crashing before the first iteration even completes. I was reading up about distributed training. Can it be used to solve this issue. If not what other solutions can I apply.</p>

username: ilkarman
cooked: <p><strong>Data-Parallel</strong>: Distributed training either Distributed Data Parallel or Data Parallel across multiple GPUs (or nodes with ddp) should let you use a bigger batch, which could help. For example if you are out-of-memory with a batch of 16 you could convert that to a batch of 4 per GPU (and have *4 GPUs). You could have a small issue if your batch is too small per-GPU and you are used regular batch-norm since the batch statistics are not broadcast unless you use syn-batch-norm which is quite a bit slower.</p>
<p><strong>Model-Parallel</strong>: However, if even with a batch size of 1 you are running out of memory then you could investigate whether the model can be split across multiple gpus</p>
<p><strong>Synthetic Batch</strong> -&gt; Get a larger batch without increasing GPUs</p>
<p>This will run slow but you could accumulate your loss-gradients across several batches and then only step your optimizer every n_forward passes. For example if you want a batch of 16 but can only fit a batch of 2:</p>
<pre><code class="lang-auto"># How many forward-passes to do before updating gradient
num_steps_per_grad = 8

iter = 0
for data in dataloader:
	iter += 1
	# ...
	loss.backward()
	# if batch_size == 2 &amp; num_steps_per_grad == 8 then ~batch_size of 16
	if iter == num_steps_per_grad:
		iter = 0
		optimizer.step()
		optimizer.zero_grad()
		lr_sched.step()
</code></pre>
How to make different EmbeddingBag share same embedding matrix?

username: Varg_Nord
cooked: <p>I would like to perform both the “mean” and “sum” method on one BoW feature. Docs show that <code>EmbeddingBag</code> could be initialized with the same weights using the <code>from_pretrained</code> method. But it seems the underlying <code>THPVariable_make_subclass</code> function would initialize different weight tensors into the computational graph.</p>

username: tom
cooked: <p>The most straightforward way to share weights might be to use the functional interface for the “second invocation”, where you replace the <code>self</code> in the call to <code>F.embedding_bag</code> taken from the forward:</p>
<pre><code class="lang-auto">    def forward(self, input, offsets=None, per_sample_weights=None):
        # type: (Tensor, Optional[Tensor], Optional[Tensor]) -&gt; Tensor
        return F.embedding_bag(input, self.weight, offsets,
                               self.max_norm, self.norm_type,
                               self.scale_grad_by_freq, self.mode, self.sparse,
                               per_sample_weights, self.include_last_offset)
</code></pre>
<p>That said, I would imagine that for the same <em>features</em>, it’s more efficient to use Embedding + sum/mean manually (or even just use EmbeddingBag with sum and scale yourself to go from sum to mean?)</p>
<p>Best regards</p>
<p>Thomas</p>
How to separate conv1d channels?

username: Topsoil
cooked: <p>If i have a tensor of shape (<em>batch_size</em>,<em>channels</em>,<em>features</em>), how do i separate it into a list of tensors of shape (batch_size,1,features) with length of <em>channels</em>?</p>

username: Topsoil
cooked: <p>Answer found: use torch.split(tensor,1,channel_dim_index)|</p>
<p>So if tensor is of shape (3,100,250), use torch.split(tensor,1,1) to get a tuple of 100 tensor of shape (3,1,250)</p>