akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Profiler: torch.cuda takes more time than expected

username: Deeply
cooked: <p>I am trying to optimize the performance using Spyder profiler. This line, which is part of the code, <code>z = torch.tensor(x).float().cuda()</code>, seems to take too much time, approximately 20 ms per image.</p>
<p>I’ve tried some other initialization ways;  <code>z =torch.cuda.FloatTensor(x)</code>, and<br>
<code> to.device(...)</code>; yet they both take slightly more time.</p>
<p><strong>Image Info:</strong><br>
Image size on average, (<code>x</code> as a numpy array):<br>
210,000 bytes (~70,000 bytes per channel).</p>
<p><strong>Note:</strong> No batching is used</p>
<p><strong>The profiler refers to this as the source of time depletion:</strong><br>
<code>&lt;method 'cuda' of 'torch._C._TensorBase' objects&gt;</code></p>
<p>and probably this:</p>
<pre><code class="lang-auto">..\Anaconda3\envs\PyTorchEight\lib\site-packages\torch\cuda\__init__.py : 144
Lazy init
</code></pre>
<p><strong>System info:</strong><br>
PyTorch 1.8.1; Python 3.8; CUDA Version: 11.2; Windows 10 pro. Everything works well and the GUP does what it is supposed to do. Installed via (Cu 11.2 vs 11.1):<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge</code></p>

username: cpeters
cooked: <p>I believe that method is both allocating space on the GPU for the tensor in question as well as copying it over to the device. It’s going to take longer than most other operations because it takes longer to copy from system memory to the GPU than to do operations entirely within memory.</p>

username: Deeply
cooked: <p>That’s correct. Might be better to load the images into torch tensors, and then copy them to the device.</p>
<p>I hope this worths the hassle, not sure how much speed up would that offer!</p>

username: cpeters
cooked: <p>Well, if you can use the DataLoader class, then that offers pinned memory, which allows for faster transfers and multithreaded loading?</p>

username: Deeply
cooked: <p>Yes. Just gave it a go and I noticed that <code>torchvision</code> image loading functions load the images to the CPU, and then one has to be copy them to the device (GPU). It would be nice if we can directly load to GPU. Not sure if this would be feasible.</p>

username: cpeters
cooked: <p>You mean the GPU loading directly from a drive? Nvidia has a separate technology for that, but it specifically requires hardware support for PCIe P2P and workstation or data centre accelerator cards, along with limited software support (Ubuntu only, for instance). I don’t know if directly loading into the GPU without going through the CPU is doable in CUDA alone, though, but communicating from storage to GPUs is also possible because of texture streaming.</p>
<p>One thing to think about is if the 20ms it takes to transfer to the GPU is really going to be a bottleneck when working on more than one file compared to processing time.</p>
Backward for composition loss

username: kavka
cooked: <p>I was trying to .backward() a loss like following.</p>
<pre><code class="lang-auto">model = model(2,2,1,2) #suppose model is just a linear model with 2 input, 2 neuron in 1 hidden layer, 1 output
x = torch.Tensor([[0.3,0.52],[-0.1,0.2]])

'this part try to calculate the second derivative with respect to g'
g = x.clone()
g.requires_grad = True
UF = model.forward(g) 
tmp   = torch.autograd.grad(outputs = UF, inputs = g, grad_outputs = torch.ones(UF.size()), retain_graph=True, create_graph=True)[0]
UFxx  = torch.autograd.grad(outputs = tmp, inputs = g, grad_outputs = torch.ones(tmp.size()))[0][:,[0]]

''
U = model.forward(x) 
F     =U*UFxx
lossF = (F**2).mean()
lossF.backward()
</code></pre>
<p>However, the following error comes out,</p>
<pre><code class="lang-auto">Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.
</code></pre>
<p>if I modified as following,</p>
<pre><code class="lang-auto">UFxx  = torch.autograd.grad(outputs = tmp, inputs = g, grad_outputs = torch.ones(tmp.size()), create_graph=True)[0][:,[0]]
</code></pre>
<p>then the code run fine.</p>
<p>My question is that, why I need to set the “create_graph=True”,  I already have the U part which will give me the graph and the UFxx should be just a number?</p>
<pre><code class="lang-auto">U = model.forward(x) 
F     =U*UFxx
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>There are two things:</p>
<ul>
<li>create_graph=True means that you want the graph to be created while doing the backward so that the gradients have requires_grad=True.</li>
<li>retain_graph=True means that you don’t want the graph to be deleted because you want to call backward on it again.</li>
</ul>
<p>Note that create_graph=True forces retain_graph=True.</p>
<p>So in this case, if you plan to do multiple backwards, you should use retain_graph=True. But if you plan to backward through the first backward, you should use create_graph=True.</p>
Type Checking: Types Tensor and TensorFloat incompatible

username: James_Trueb
cooked: <p>I am new to static type checking in python, therefore I am not sure if this is a PyTorch Problem at all.<br>
I am using PyRight (default for VSCode Pylance installation) for type checking. Unfortunately it keeps giving me the warning, that <code>FloatTensor</code> and <code>Tensor</code> are incompatible types. Minimal example:</p>
<pre><code class="lang-python">import torch
from torch import FloatTensor

def createFloatTensor() -&gt; FloatTensor:
    return torch.rand(1, 2, requires_grad=False)
</code></pre>
<p>Tested version:</p>
<p>Python 3.8.8<br>
PyTorch 1.8.1 (Cuda)<br>
VSCode: PyLance 2021.4.1</p>
<p>Is this expected behaviour and is there a quick fix for this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The short answer is that <code>FloatTensor</code> are not a thing anymore. They are still here only for backward compatibility reasons.<br>
All the Tensors are of type <code>torch.Tensor</code> (whatever their dtype, device, layout, etc).</p>
How to use BCELossWithLogits for foreground,background(binary) semantic classifaction

username: santhan
cooked: <p>Hi everyone, I am trying to use the BCE logit loss function for binary semantic segmentation.Output of my network is [batch size,2,h,w] and the label is  [batchsize,h,w]. I have read all the forums regarding this problem my doubt is not clarified. How to feed this input and target to the BCElogitloss without doing any one-hot encoding.<br>
thanks in advance</p>

username: JuanFMontesinos
cooked: <p>You just need a GT whose shape matches the output.<br>
You should make your network to predict a single channel, encoding 1 as foreground and 0 as background.</p>
How to move a tensor from one gpu to another?

username: seyeeet
cooked: <p>I have a tensor which is on gpu 0.<br>
how can I move (or make a copy of it one gpu 1)?</p>
<pre><code class="lang-auto">A = torch.rand(1)
cuda0 = torch.device('cuda:0')
cuda1 = torch.device('cuda:1')
A.to(cuda0)
# now I want to move A to gpu 1
A.to(cuda1)
</code></pre>
<p>gives me error:</p>
<p><code>RuntimeError: CUDA error: invalid device ordinal</code></p>

username: albanD
cooked: <p>Hi,</p>
<p>Are you sure the second GPU is properly available? What does device_count() returns?<br>
You might be hidding the second GPU with CUDA_VISIBLE_DEVICES=0 as well.</p>

username: seyeeet
cooked: <p>Hi,</p>
<p>yeah my bad. I missed to change the number of devices in CUDA_VISIBLE.<br>
Thanks for pointing that out</p>

username: coincheung
cooked: <p>Hi,</p>
<p>Would you tell me what happened when tensor.to is called? Is the tensor transfered directly from gpu 1 to gpu 2, or it is first transferred from gpu 1 to cpu memory and then transferred from cpu to gpu 2?</p>

username: albanD
cooked: <p>It depends on the hardware you have.<br>
If possible, it will be sent directly from one GPU to the other. But not all cards support that AFAIK and it might have to go through the CPU.</p>
got nan in tensor.reverse()

username: WBS_WHU
cooked: <p>When I run the following code, the second parameter of pixel2cam contains nan</p>
<pre><code class="lang-auto">cam_coords = pixel2cam(depth.squeeze(1), intrinsics.inverse()) # [B,3,H,W]
</code></pre>
<p>then, i added this code before pixel2cam</p>
<pre><code class="lang-auto">    if torch.any(torch.isnan(intrinsics.inverse())):
        print(intrinsics)
        print(intrinsics.inverse())
        print(torch.isnan(intrinsics.inverse()))
        print(torch.any(torch.isnan(intrinsics.inverse())))
</code></pre>
<p>but got output as follow image:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd96fb100fbf916b535081a10ba1455fc3c0147f.png" data-download-href="https://discuss.pytorch.org/uploads/default/dd96fb100fbf916b535081a10ba1455fc3c0147f" title="2021-04-20 21-04-59 的屏幕截图"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/d/dd96fb100fbf916b535081a10ba1455fc3c0147f.png" alt="2021-04-20 21-04-59 的屏幕截图" data-base62-sha1="vCgZJiSZ2L9s8EYKvBRtithB14X" width="512" height="500" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2021-04-20 21-04-59 的屏幕截图</span><span class="informations">730×712 40.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This result looks very contradictory. After entering the <code>if torch.any(torch.isnan(intrinsics.inverse()))</code> , why thre output of <code>torch.any(torch.isnan(intrinsics.inverse())</code> is still False?</p>
GPU speed decreases immensely on deployment

username: lblopes
cooked: <p>Hi,<br>
I developed a c++ software to make predictions on images by segmentation. Using libtorch 1.7, cuda toolkit 10.2 and cudnn 8. Everything works well on my computer (windows10, i7, GTX1070) and on my colleague’s. However, when deploying in a brand new computer (windows10, i7, RTX 3070) the calculation speed drops immensely.<br>
Using only the CPU it takes 2minutes to load the network and perform one prediction. However, when using the GPU, it takes about 5 minutes only to load the network (where it should take 10 sec to do it all).<br>
Libtorch detects the gpu and performs all operations, but really really slow.</p>
<p>This is the new computer’s benchmark:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://www.userbenchmark.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://www.userbenchmark.com/UserRun/42206824#GRAPHICS_CARD" target="_blank" rel="noopener nofollow ugc">userbenchmark.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://www.userbenchmark.com/resources/img/wri/bench-speed/bench-speed-150.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://www.userbenchmark.com/UserRun/42206824#GRAPHICS_CARD" target="_blank" rel="noopener nofollow ugc">Asus ASUS TUF Dash F15 FX516PR_FX516PR Performance Results - UserBenchmark</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
Everything else works well on the GPU. The Nvidia driver that we are using was released on 2021.4.14 (6 days ago)</p>
<p>Do you know where it might be the problem?</p>
Reset LSTM hidden states efficiently

username: kaixin
cooked: <p>Is there an efficient way for PyTorch to reset during unrolls? Something like <a href="https://dm-haiku.readthedocs.io/en/latest/api.html?highlight=resetcore#resetcore" rel="noopener nofollow ugc">haiku.ResetCore()</a>. It is useful in implementing a recurrent policy in RL.</p>
Pytorch model predicts same class for all images

username: siva_sankar
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210416-060051-RC00_368813563" class="site-icon" width="15" height="15">
      <a href="https://colab.research.google.com/drive/1CNPEMabRunNKPhCfiDmp_1xXqhZXUZQ1?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1CNPEMabRunNKPhCfiDmp_1xXqhZXUZQ1?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Above link is my colab notebook for cnn model for mask recognition.Model gives the validation accuracy of 89  % and test accuracy of 85 % but when I try to predict the image by loading separately ,it gives me the same output for all images(with_mask).</p>
<p><a href="https://drive.google.com/drive/folders/11EVQ15gH-0ODc6I0DR87sc9tVblkzEgg?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/11EVQ15gH-0ODc6I0DR87sc9tVblkzEgg?usp=sharing</a></p>
<p>Above is the dataset.</p>
How to constrain one half of a trainable Tensor to be equal to the flipped first half?

username: kof123
cooked: <p>Hello everybody,</p>
<p>I am looking for the following:</p>
<p>Assume that I have a trainable tensor T of shape torch.size([8]).<br>
Now I would like to train that tensor but also to have the following constraint:<br>
the tensor sould be symmetric in the sense that<br>
T[0] = T[7]<br>
T[1] = T[6],<br>
T[2] = T[5],<br>
T[3] = T[4].</p>
<p>i.e. the tensor T consists of actually two tensors, one of which is the flipped version of the other one.</p>
<p>I have tried to assign to the first half of T its flipped version, i.e.</p>
<p>self.T[4:] = torch.flip(self.T[:4],dims=[0])</p>
<p>but this fails as the optimizer tells me the it can’t optimize a non-leaf tensor.</p>
<p>Any ideas on how to make that work?</p>
<p>Thank you very much in advance!</p>

username: KFrank
cooked: <p>Hi Kof!</p>
<aside class="quote no-group" data-username="kof123" data-post="1" data-topic="118729" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a6a055/48.png" class="avatar"> kof123:</div>
<blockquote>
<p>I would like to train that tensor but also to have the following constraint:<br>
the tensor sould be symmetric</p>
</blockquote>
</aside>
<p>I would not use a tensor of length 8 and attempt to constrain its first<br>
half to mirror its second.  Just use a tensor of length 4, and use each<br>
of its 4 elements twice in your computations.</p>
<p>Here is an example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; T = torch.randn (4, requires_grad = True)
&gt;&gt;&gt; v = torch.arange (8).float()
&gt;&gt;&gt; torch.dot (torch.cat ((T, torch.flip (T, (0,)))), v).backward()
&gt;&gt;&gt; T.grad
tensor([7., 7., 7., 7.])
</code></pre>
<p>(The business with <code>v</code> and <code>torch.dot()</code> is just to give the example<br>
something to run <code>.backward()</code> on to show that autograd works.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: kof123
cooked: <p>Hi Frank!</p>
<p>Workd like a charm!<br>
Thanks a lot!</p>
How to compile pytorch in an executable?

username: blackonyyx
cooked: <p>Hi, Im trying to compile a game using a pytorch model for part of the game. I want it in an executable format. For some context, I am doing this using cx_freeze on windows</p>
<p>But im getting a compilation error as shown in the image below. (Sorry, was unable to extract the text efficiently)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a24298e958c4b56cd70536ca97a6afdaf8a464f1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a24298e958c4b56cd70536ca97a6afdaf8a464f1" title="photo_2020-07-15_00-22-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a24298e958c4b56cd70536ca97a6afdaf8a464f1_2_214x499.jpeg" alt="photo_2020-07-15_00-22-46" data-base62-sha1="n9q27OTrpB9Xv7gLeJi6dYOXFGp" width="214" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a24298e958c4b56cd70536ca97a6afdaf8a464f1_2_214x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a24298e958c4b56cd70536ca97a6afdaf8a464f1_2_321x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a24298e958c4b56cd70536ca97a6afdaf8a464f1.jpeg 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">photo_2020-07-15_00-22-46</span><span class="informations">414×965 91.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I dug around the errors a bit, and saw that the issue was related to how _VF is importing torch from within torch itself? I am unsure how to amend this though.</p>
<p>Any suggestions as to how I can troubleshoot this? The only thing i need the model to be able to do in compilation after training the weights before all this is to feedforward an image/stack of images and come to a predicted action.</p>
<p>Any advise or suggestion is much appreciated! Thanks!</p>

username: fingertap
cooked: <p>Same problem here.</p>
<p>Cannot freeze torch application appropriately, due to the import issue of ‘_VF’</p>
Padding zero size tensors

username: Vanjoy
cooked: <p>Hi,</p>
<p>I got the following error when trying to pad zero size tensors.</p>
<pre><code class="lang-auto">torch.nn.functional.pad(torch.ones((0, 7)), (0, 8))
Out[29]: tensor([], size=(0, 15))

In [30]: torch.nn.functional.pad(torch.ones((0, 7)), (0, 0, 0, 0))
Out[30]: tensor([], size=(0, 7))


In [31]: torch.nn.functional.pad(torch.ones((0, 7)), [0, 0, 0, 3])                                                                                                                                                                               
Out[31]: 
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]])

In [32]: torch.nn.functional.pad(torch.ones((0, 7)), (0, 8, 0, 0))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-31-b8dd95848884&gt; in &lt;module&gt;
----&gt; 1 torch.nn.functional.pad(torch.ones((0, 7)), (0, 8, 0, 0))

/home/hjj/.direnv/python-3.8.7/lib/python3.8/site-packages/torch/nn/functional.py in _pad(input, pad, mode, value)
   3998     assert len(pad) // 2 &lt;= input.dim(), "Padding length too large"
   3999     if mode == "constant":
-&gt; 4000         return _VF.constant_pad_nd(input, pad, value)
   4001     else:
   4002         assert value == 0, 'Padding mode "{}"" doesn\'t take in value argument'.format(mode)

RuntimeError: The input size 0, plus negative padding 0 and 0resulted in a negative output size, which is invalid. Check dimension 0of your input.
</code></pre>
<p>I expected the forth one will work since both the first three examples work. Any idea of why? Thanks.</p>
Optimizers: RProp vs SGD

username: eduardo4jesus
cooked: <p>Hello Fellows,</p>
<p>I was reading about the <a href="https://optimization.cbe.cornell.edu/index.php?title=RMSProp" rel="noopener nofollow ugc">RMSProp</a> when I got to know about an optimizer called RProp (check reference for RMSProp).</p>
<p>At first it seemed to be just like the SGD update. Then this statement came out</p>
<blockquote>
<p>RProp algorithm does not work for mini-batches is because it violates the central idea behind stochastic gradient descent, when we have a small enough learning rate, it averages the gradients over successive mini-batches.</p>
</blockquote>
<p>Hence, I concluded that SProp <strong>is different</strong> from SGD, other than a concept of mini-batch instead of batch - which introduces the solution to the stochastic realm, I didn’t see any other changes.</p>
<p>Then RProp is mentioned again by The Godfather <a href="https://youtu.be/defQQqkXEfE?list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&amp;t=258" rel="noopener nofollow ugc">in here</a> making it sound like very similar to the regular gradient descendent (no mini-batch).</p>
<p>However, I got really confused when seeming RProp on <a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">the list of optimizers</a>, given that we already have SGD there, and the mini-batch is not a parameters set at the optimizer.</p>
<h1><a name="question-1" class="anchor" href="#question-1"></a>Question:</h1>
<p>So, can anyone share the difference between them?</p>

username: googlebot
cooked: <p>That wiki just confuses readers by mentioning RProp without explaining it properly. Rprop is basically a heuristic to adjust learning rate based on gradient sign conformity. Newer methods use more sophisticated measures than sign (as a consequence - they work better with mini-batches).</p>
Https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113/33

username: leyla_khaleghi
cooked: <p>Hi,  I have a multi-input model with 3 parallel resnet and I want to just load one of them. I have no idea that how I could make a dictionary of the name of those layers that I want to use and use the code in this forum, <a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113/33" class="inline-onebox">How to load part of pre trained model? - #33 by MariosOreo</a>.</p>
<p>for key in model_dict:<br>
if  ‘resnet1’ in key:<br>
# print (key)<br>
pretrained_dict.append(key)</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit more?<br>
Are you using a custom module, which contains 3 separate resnets?<br>
If so, you should be able to access the desired submodel directly e.g. via: <code>model.resnet1.state_dict()</code>.</p>
Ensuring complete lazy evaluation with enumerate/iter and Dataloader

username: aknirala
cooked: <p>I was trying to generate some images using code and vary it by using randomness. I found that when I use DataLoader and then use enumerate or iter on it, then memory overshoots unexpectedly. On debugging I found that <code>__getitem__</code> function is getting called right when I create an enumerate or iter object.</p>
<p>Q: How can I ensure that it only gets called when I call next for iterator or extract next element from enumerate?</p>
<p>For reference here is my toy example (to expose that issue is with calling of <code>__getitem__</code> (tested on Python3.7 and 3.8)</p>
<pre><code class="lang-auto">#Testing enumeration here
from torch.utils.data import Dataset, DataLoader
import torch
class test_enum(Dataset):
    def __init__(self, root="/home/aknirala/data/clocks/", transform=None):
        self.transform = transform
        self.root = root
    def __len__(self):
        return 1000000 
    def __getitem__(self, idx):
        print(str(idx) + ", ")
        return idx


dataset = test_enum()
dataloader = torch.utils.data.DataLoader(dataset, batch_size=4,
                                            shuffle=True, num_workers=2)
print("Len of iterations per epochs: ",len(dataloader))
</code></pre>
<p>Now moment I do:</p>
<p><code>data_iter = iter(dataloader)</code></p>
<p>I get something like: (There are batch_size<em>num_workers</em>2 numbers printed below)</p>
<pre><code class="lang-auto">143666, 315805, 

357128, 851433, 

298989, 706179, 

432269, 11821, 

995092, 
911574, 
812109, 
598460, 
276834, 
563535, 
551306, 
343008,
</code></pre>
<p>Also instead if I do:<br>
<code>e = enumerate(dataloader)</code></p>
<p>I get something like:</p>
<pre><code class="lang-auto">640834, 336805, 

792848, 183726, 

727800, 322372, 
703700, 
870395, 
856172, 
370866, 
792898, 
415834, 
700454, 
516118, 
916330, 

733028,
</code></pre>
<p>Later when I call next(data_iter) or run a for loop on e, each time I see 4 numbers which is as expected (as batch_size is s4). However, why does getitem gets called while I am creating enumerate or iter object? And how can I stop it?</p>

username: ptrblck
cooked: <p>I cannot reproduce the print statements by calling <code>iter(dataloader)</code> or <code>enumerate(dataloader)</code>, however I’m also unsure if the prefetching behavior has changed recently (as I’m using a source build).<br>
In any case, you should be able to remove the prefetching by using <code>prefetch_factor = 0</code> in your <code>DataLoader</code>. This should make sure that no samples are pre-loaded.</p>
How to delete layer in pretrained Nvidia Nemo model?

username: benak
cooked: <p>I am using the pre-trained model of Biomegatron. I want to get word embeddings by dropping the last layer. But I couldn’t figure out how can I do that with Megatron models or Nemo models. I would be appreciated if you help me I’m digging that for 3 days tnx.</p>

username: ptrblck
cooked: <p>Usually you can “drop” layers by replacing them with <code>nn.Identity</code> modules, which would allow you to just reuse the <code>forward</code> method without overriding it.<br>
<code>print(model)</code> should give you all layer names (or alternatively check the model implementation in the corresponding repository).</p>
Converting keras model to pytorch

username: ananthceh19
cooked: <p>Hi,<br>
My Keras model looks something like this:<br>
def <strong>init</strong>(self, in_channels=1):</p>
<pre><code>    super(CNN, self).__init__()
    self.base_model = ResNet50(weights='imagenet', include_top=False)
    self.NUM_CLASSES = 22

    self.DENSE_LAYER_ACTIVATION = 'softmax'

    for layer in self.base_model.layers:
        layer.trainable = True
    #layer_name = 'conv1_relu'
    self.layer_name = 'conv3_block4_2_relu'
    #layer_name = 'conv2_block2_1_relu'
    self.intermediate_layer_model = Model(inputs=self.base_model.input,
                                    outputs=self.base_model.get_layer(self.layer_name).output)

    #w = self.intermediate_layer_model.output
    self.GAP = GlobalAveragePooling2D()
    self.Dropout = Dropout(rate=0.2)

    self.Dense1 = Dense(256, activation = 'relu')
    self.Dense2 = Dense(128, activation = 'relu')
    self.predictions = Dense(self.NUM_CLASSES,
            activation = self.DENSE_LAYER_ACTIVATION)

def forward(self, x):
    x=self.base_model.get_layer(self.layer_name)(x)
    print(x.shape)
    x=self.GAP(x)
    x=self.Dropout(x)
    x=self.Dense1(x)
    x=self.Dropout(x)
    x=self.Dense2(x)
    x=self.Dropout(x)
    x=self.predictions(x)

    return x
</code></pre>
<p>I want to convert this to pytorch. Any help would be greatly appreciated.<br>
Thanks.</p>

username: ptrblck
cooked: <p>Where are you currently stuck in the conversion?<br>
Based on the provided code I would assume that changing the layer names (e.g. <code>Dense</code> to <code>nn.Linear</code>) could solve the majority of the issues.</p>

username: ananthceh19
cooked: <p>Thanks for the response.<br>
Specifically:<br>
a) How do I specify conv3_block4_2_relu?<br>
b) How do i convert this line:<br>
self.intermediate_layer_model = Model(inputs=self.base_model.input,<br>
outputs=self.base_model.get_layer(self.layer_name).output)</p>

username: ptrblck
cooked: <p>I don’t know how <code>Model</code> is defined but assuming that <code>self.layer_name</code> gets an intermediate activation from <code>ResNet50</code>, you could either use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> or override the <code>forward</code> of the resnet implementation so that the desired activation will be returned.</p>
C++ Frontend: Dynamic runtime network definitions?

username: r2dliu
cooked: <p>It appears the typical way you’d create a network is to create a struct that extends torch::nn::module, with members of the various submodules that are part of it. Obviously you can’t dynamically generate a struct with these members, so that’s kind of off the table.</p>
<p>Is there any other way to dynamically generate a model at runtime? Imagine calling a function that adds another layer to your top level module somehow. It’s possible to create a list of modules with torch::nn::ModuleList, and then create individual modules to push_back into it at runtime, but again stuck at the creation of the struct / forward function.</p>
<p>I realize the answer to this is probably not but maybe there are some crazy C++ wizards out there who know things I don’t</p>
How to print/iterate through the network structure in PyTorch?

username: aman26kbm
cooked: <p>PyTorch doesn’t create a static graph representation of a neural network. So, we can not iterate through the network’s layers once it is created (and query or print them). Is it possible to get the information about the network structure at runtime? Alternatively, is there a static mode that we can run PyTorch in?</p>
<p>Here’s my usecase: I am taking a neural network from the pytorch hub and trying to perform some analysis on the network. So, I need to iterate over layers in the network and query for things like tensor sizes and next/previous layer names.</p>
<p>There are multiple ways to list out or iterate over the flattened list of layers in the network (including Keras style model.summary from sksq96’s pytorch-summary github). But the problem with these methods is that they don’t provide information about the edges of the neural network graph (eg. which layer was before a particular layer, or which layer does this layer feed into, etc). These lists also seems to miss any the elementwise addition layers created using the “+” operator in the network code.</p>
<p>I searched around and do see some posts that talk about it, but haven’t found a solution:</p>
<p><a href="https://stackoverflow.com/questions/56248645/how-do-i-get-the-precursor-nodes-of-each-layer-in-pytorch" rel="noopener nofollow ugc">python - How do I get the precursor nodes of each layer in Pytorch? - Stack Overflow</a></p>
<p><a href="https://discuss.pytorch.org/t/how-to-traverse-a-network/10477/6">How to traverse a network - PyTorch Forums</a></p>
<p>Note that the networks are not just defined using nn.Sequential. They can be complex like containing multiple nn.Sequential blocks and other modules.</p>
<p>Any help/advice is appreciated.</p>

username: ptrblck
cooked: <p>Scripted or traced model provide a representation of the model, which you could probably use for your experiments.<br>
Have you checked the created code from a scripted model and would it be useful in your use case?</p>

username: aman26kbm
cooked: <p>Thanks for your response, Piotr <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I haven’t heard of scripted or traced models. Lemme check it out.</p>
Parallel sum specific array values using torch.where()

username: cpsanders
cooked: <p>Hello, I need to manually accumulate the parameter.grad attributes of my network based on the values of another tensor. I’m working with the following variables:</p>
<ol>
<li>I have a bit-string array of sampled bits <code>m</code>, of size <code>(M,)</code> (where M is the number of samples)</li>
<li>I also have two tensors of gradients, <code>grad1</code> and <code>grad2</code>. <code>grad1[i]</code> and <code>grad2[i]</code> represent the gradients corresponding to sampled bits of -1 and 1 respectively. Each specific gradient tensor located at <code>i</code> is a different size based on the parameter being modified. Ex. If M=5 (5 sampled bits), I will have <code>grad1</code> and <code>grad2</code> will each be tensors of 5 parameter gradient matrices.</li>
<li>I have <code>total_grad</code>, which takes the same shape as <code>grad1</code> and <code>grad2</code>. <code>total_grad</code> is increased by either <code>grad1</code> or <code>grad2</code>.</li>
</ol>
<p>I’m trying to parallelize the process of accumulating <code>total_grad</code> based on <code>m</code>. If <code>m[i] &gt; 0</code>, I want to add <code>grad1[i]</code> to <code>total_grad[i]</code> for all samples. Otherwise, I want to add <code>grad2[i]</code>. However, I don’t want to do this with a for loop (for obvious efficiency reasons), so I tried to parallelize this using <code>torch.where()</code>:</p>
<pre><code class="lang-auto">for param in range(params)
    total_grads[param][:] += torch.where(m[:] &gt; 0, grad1[param][:], grad2[param][:])
</code></pre>
<p>This code appears to work for the first list of parameter gradients (of size <code>(M, 4, 2)</code>), but not for the second (of size <code>(M,4)</code>). I’m receiving the following Runtime error:</p>
<pre><code class="lang-auto">The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 1
</code></pre>
<p>Can someone help me understand whats going on here, or a better way to do this? Thanks!</p>

username: cpsanders
cooked: <p>Just figured it out! I had to re-shape <code>m</code> to be compatible with the shape of <code>grad1</code>.</p>
<pre><code class="lang-auto">reshaped_m = m.reshape(m.shape + (1,)*(grads_per_param[param].ndim-1))
</code></pre>
ValGrind shows potential memory leak

username: Albert_Christianto
cooked: <p>Hi, recently l noticed some potential memory leak in libtorch_cpu.<br>
I am using Libtorch_cpu 1.8.1 in linux. Here is the c++ code that I used for testing.</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;

#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;

bool f()
{
    torch::NoGradGuard no_grad;
    std::cout &lt;&lt; "--------------------------------------------------------------------" &lt;&lt; std::endl;
    bool det = true;
    try {
        std::cout &lt;&lt; "Engine test" &lt;&lt; std::endl;
        //torch::jit::script::Module model = torch::jit::load("/home/ion3/Downloads/retinaface-mnet-complete.pt");
        torch::jit::script::Module model = torch::jit::load("/home/ion3/Downloads/arcface-r101.pt");
        model.eval();
        model.to(torch::kCPU);

        for (int i = 0; i &lt; 100; ++i) {
            std::cout &lt;&lt; i &lt;&lt; std::endl;
            //torch::Tensor input_tensor = torch::rand({ 1, 3, 480, 640}, torch::kFloat32).to(torch::Device(torch::kCPU));
            torch::Tensor input_tensor = torch::rand({ 1, 3, 112, 112}, torch::kFloat32).to(torch::Device(torch::kCPU));
            //auto outputs = model.forward({ input_tensor }).toTuple();
            auto outputs = model.forward({ input_tensor });
        }
    }
    catch (std::exception&amp; e) {
        std::cout &lt;&lt; e.what() &lt;&lt; std::endl;
        std::cout &lt;&lt; "--------------------------------------------------------------------\n";
        return false;
    }
    std::cout &lt;&lt; "--------------------------------------------------------------------\n";
    return det;
}


int main(int, char**) {
    bool res = f();
    return res;
}
</code></pre>
<p>I am not sure if this is libtorch bug or not because valgrind shows</p>
<pre><code class="lang-auto">==39105== LEAK SUMMARY:
==39105==    definitely lost: 0 bytes in 0 blocks
==39105==    indirectly lost: 0 bytes in 0 blocks
==39105==      possibly lost: 70,976 bytes in 5 blocks
==39105==    still reachable: 776,523 bytes in 12,086 blocks
==39105==         suppressed: 0 bytes in 0 blocks
==39105== 
==39105== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)
</code></pre>
<p>Any opinion regarding this problem?</p>
<p>PS:<br>
here is the link for the full valgrind report. I have tested with 2 different model.<br>
<a href="https://drive.google.com/file/d/1qHvIgSFTMzPLZH9Zxbyi4PpIAS57mDWw/view?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/file/d/1qHvIgSFTMzPLZH9Zxbyi4PpIAS57mDWw/view?usp=sharing</a></p>
How can I apply L2/L1 loss with 3D voxels?

username: davidlee
cooked: <p>Hi,</p>
<p>I have my own data with the following dimensions for super resolution tasks.<br>
(Batch, channels, 128, 128, 128).<br>
ex) predict = (2, 1, 128, 128, 128) / output = (2, 1, 128, 128, 128)</p>
<p>torch.nn.MSELoss() or torch.nn.L1Loss() can be directly applied in this voxel case?<br>
for instance,</p>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss()
pred = torch.randn(2,1,128,128,128)
y = torch.randn(2,1,128,128,128)
loss = criterion(pred, y)
</code></pre>
<p>Or should I make other versions of losses for 3D voxels?</p>
<p>+) can anybody let me know if there is any strategy how to utilize SSIM loss with 3D voxels?</p>

username: ptrblck
cooked: <p>You can directly apply both mentioned losses, as they would expect the model output and target to have the same shape, which is the case for your use case.<br>
Unfortunately, I not sure how SSIM can be used for your use case, but if I’m not mistaken the original implementation uses 2D convs internally, so you might change it to 3D ones.</p>

username: davidlee
cooked: <p>Thank you so much:) it helped a lot!</p>
3D Pooling Layers in FP32 in Graph Mode

username: AfonsoSalgadoSousa
cooked: <p>Hello. I am trying to post quantize a 3D ResNet using the new graph mode quantization. As far as I understand, both AdaptiveAvgPool3d and MaxPool3d have no quantized kernel and should be marked as “non traceable”. From here (<a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_quant_guide.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Quantization User Guide — PyTorch Tutorials 1.8.1+cu102 documentation</a>), I thought the following should do the trick:</p>
<pre><code class="lang-auto">prep_config_dict = {
    "non_traceable_module_class": [nn.MaxPool3d, nn.AdaptiveAvgPool3d]
}
prepared_model = prepare_fx(
    model_to_quantize, qconfig_dict, prepare_custom_config_dict=prep_config_dict)
</code></pre>
<p>Yet, it does not. Neither did some experiments using “non_traceable_module_name”.</p>
<p>Appreciate any help I can get. Thank you in advance.</p>

username: Zafar
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> I thought the maxpool and avgpool was supported, can you comment on this, please?</p>

username: AfonsoSalgadoSousa
cooked: <p>Given I am using a 3D ResNet, I also need to use the 3D versions of every model. As far as I can tell, only the Average Pooling 3D is supported. Neither Adaptive Average Pooling 3D nor Max Pooling 3D are supported.</p>

username: jerryzh168
cooked: <p>you don’t need to mark them as non-traceable module I think, they are leaf modules and will not be traced by default, non traceable module is typically used to mark a submodule as untraceable (e.g. a submodule contains conv - linear - other_ops - etc.)</p>
<p>adaptive advg pol3d <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fx/quantization_patterns.py#L934" class="inline-onebox" rel="noopener nofollow ugc">pytorch/quantization_patterns.py at master · pytorch/pytorch · GitHub</a> and Maxpool3d: <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fx/quantization_patterns.py#L943" class="inline-onebox" rel="noopener nofollow ugc">pytorch/quantization_patterns.py at master · pytorch/pytorch · GitHub</a> are supported, these ops works for both float and quantized inputs.</p>
Discrepancy in output shape

username: circa
cooked: <pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self)
        super(Model, self).__init__()
        self.layer = resnext50_32x4d(pretrained = False)
        self.layer.fc = nn.Linear(2048,100)
        self.query = nn.Linear(100,100, bias = False)
        self.key = nn.Linear(100,100, bias = False)
        self.value = nn.Linear(100,100, bias = False)
        self.nl = nn.LogSoftmax(dim = 1)

    def forward(self, X):
        x = self.layer(X)
        q = self.query(x)
        k = self.key(x)
        v = self.value(x)
        kqdots = torch.matmul(q, torch.transpose(k, 0, 1))
        return kqdots
</code></pre>
<p>This is the Model I am using. The batch size is 64.<br>
I get the predictions using</p>
<pre><code class="lang-auto">model = Model()
model = torch.nn.DataParallel(model).cuda()
pred = model(data)
</code></pre>
<p>Shape of input  batch is</p>
<pre><code class="lang-auto">torch.Size(64,3,256,256)
</code></pre>
<p>However the shape of the output I am getting is</p>
<pre><code class="lang-auto">torch.Size(200, 32)
</code></pre>
<p>Shouldn’t it be</p>
<pre><code class="lang-auto">torch.Size(64, 64)
</code></pre>
<p>I am using Pytorch version 1.1.0</p>

username: ptrblck
cooked: <p>I cannot reproduce the output shape using 8 GPUs and get <code>[64, 8]</code>, which is expected due to the last line of code in your <code>forward</code>:</p>
<pre><code class="lang-python">kqdots = torch.matmul(q, torch.transpose(k, 0, 1))
</code></pre>
<p>This will perform a matrix multiplication using tensors with the shape <code>[batch_size/n_gpu, 100]</code> and <code>[100, batch_size/n_gpu]</code>, which creates an output of <code>[batch_size/n_gpu, batch_size/n_gpu]</code>.<br>
After concatenating the outputs from each GPU, the output will be <code>[batch_size, batch_size/n_gpu]</code>, which corresponds to <code>[64, 8]</code> in my case.</p>

username: circa
cooked: <p>I am using 2 GPUs. Could you try and check with two please ?</p>

username: ptrblck
cooked: <p>Sure! No difference and the shape is as expected:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=0,1 python tmp.py
torch.Size([64, 32]) # [batch_size, batch_size/n_gpu]
</code></pre>
<p>using:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torchvision.models as models

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer = models.resnext50_32x4d(pretrained = False)
        self.layer.fc = nn.Linear(2048,100)
        self.query = nn.Linear(100,100, bias = False)
        self.key = nn.Linear(100,100, bias = False)
        self.value = nn.Linear(100,100, bias = False)
        self.nl = nn.LogSoftmax(dim = 1)

    def forward(self, X):
        x = self.layer(X)
        q = self.query(x)
        k = self.key(x)
        v = self.value(x)
        kqdots = torch.matmul(q, torch.transpose(k, 0, 1))
        return kqdots

model = Model()
model = nn.DataParallel(model).cuda()
x = torch.randn(64, 3, 256 ,256).cuda()
out = model(x)
print(out.shape)
</code></pre>
Trying to do transfer learning with LSTM and add a layer to the front of the network

username: user12233
cooked: <p>I am able to train the following model and now I want to do some transfer learning to some other data.  The new data has 8 features.  Below is the model.</p>
<pre><code>LSTMModel(
  (lstm): LSTM(5000, 128, num_layers=2, batch_first=True)
  (fc): Linear(in_features=128, out_features=2, bias=True)
)
</code></pre>
<p>I’m trying to take inspiration from the <a href="https://discuss.pytorch.org/t/transfer-learning-usage-with-different-input-size/20744">here</a> but that’s a CNN and I can’t find anything similar for a LSTM.  Below is what I’ve tried with no luck.  The new model will also perform binary classification.</p>
<pre><code>first_lstm_layer = [nn.LSTM(8, hidden_dim, layer_dim, batch_first=True)]
first_lstm_layer.extend(list(model))
newModel = nn.LSTM(*first_lstm_layer)</code></pre>
How to broadcast 2D Tensor over 4D Tensor?

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I’m trying to broadcast a 2D Tensor over a 4D Tensor and I’m not 100% how to do it. Let’s say I have two tensors, <code>mat1</code> of size <code>[B, D]</code> and another Tensor <code>mat2</code> of size <code>[B, D, N, N]</code>. How could I broadcast <code>mat1</code> over dim 2 and 3 of <code>mat2</code>?</p>
<pre><code class="lang-auto">mat1 = torch.randn(1, 4)
mat2 = torch.randn(1,4,2,2) #B=1, D=4, N=2
mat1*mat2 #throws errror
</code></pre>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 3
</code></pre>
<p>The one thing I have done is use <code>torch.unsqueeze</code> to force a broadcasting but another problem arises during the backward pass as I’m changing the size of the Tensor!</p>
<p>Any help is appreciated!</p>

username: rad
cooked: <p>Broadcasting starts with the rightmost indices and can succeed if and only if the dimensions are equal, one of them is 1 or it does not exist. You could either permute the dims of the matrix like this</p>
<pre><code class="lang-auto">b = 1
d = 4
n = 2
mat1 = torch.randn((1, 4))
mat2 = torch.randn((1, 4, 2, 2)).permute(2, 3, 0 ,1) # swap dims 0 with 2 and 1 with 3

mat3=mat1*mat2
mat3=permute(2, 3, 0, 1) # swap back the dims

mat3.shape
torch.Size([1, 4, 2, 2])
</code></pre>
<p>Or you can multiply using the einsum notation, like this:</p>
<pre><code class="lang-auto">b = 1
d = 4
n = 2
mat1 = torch.randn((1, 4))
mat2 = torch.randn((1, 4, 2, 2))

mat3 = torch.einsum('bd, bdij -&gt; bdij', mat1, mat2).

mat3.shape
torch.Size([1, 4, 2, 2])
</code></pre>
Computing loss for two sequential modules together

username: Lancial
cooked: <p>Hello all, I have a network that contains a base and a refiner component. The base generates coarse image output and the refiner takes the coarse output and improves it. I want to train both modules end-to-end. I wonder how I can do loss calculation in this case?</p>
<p>Is it ok if I just do:</p>
<pre><code class="lang-auto">loss = loss_func(coarse, ground_truth) + loss_func(refined, ground_truth) 
loss.backward()
optimizer.step()
</code></pre>
<p>If not what will be an efficient alternative? Thank you!</p>
Reading Image from torchvision and cv2 is not same

username: Tejan_Mehndiratta
cooked: <pre><code class="lang-auto">import torchvision
import torch
img=torchvision.io.read_image(path='/home/tejanm/Downloads/subjective_img/subjective_img/sample1.jpg')
img.dtype, img.shape
</code></pre>
<p>The above snippet outputs:</p>
<pre><code class="lang-auto">(torch.uint8, torch.Size([3, 500, 375]))
</code></pre>
<pre><code class="lang-auto">import cv2
img1 = cv2.imread('/home/tejanm/Downloads/subjective_img/subjective_img/sample1.jpg')
img1 = torch.from_numpy(img1)
img1 = img1.permute(2, 0, 1)
img1.dtype, img1.shape
</code></pre>
<p>The above snippet also outputs:</p>
<pre><code class="lang-auto">(torch.uint8, torch.Size([3, 500, 375]))
</code></pre>
<p>The reading of the image from both torchvision and cv2 has the same dtype and shape.</p>
<p>But,</p>
<pre><code class="lang-auto">(img1 == img).all()
</code></pre>
<p>is:</p>
<pre><code class="lang-auto">tensor(False)
</code></pre>

username: ptrblck
cooked: <p><code>torchvision</code> uses <code>PIL</code> by default, so you could check the result arrays in <code>PIL</code> vs. <code>OpenCV</code>, which might use different decoders etc.<br>
It would also be interesting to see how large the <code>abs().max()</code> difference is between both arrays.</p>

username: Tejan_Mehndiratta
cooked: <p>No.<br>
The documentation says:</p>
<pre><code class="lang-auto">torchvision.io.read_image(path: str, mode: torchvision.io.image.ImageReadMode = &lt;ImageReadMode.UNCHANGED: 0&gt;) → torch.Tensor
</code></pre>
<p>where</p>
<pre><code class="lang-auto">path (str) – path of the JPEG or PNG image
</code></pre>
<p>So torchvision will simply read the image and output the tensor. There is no interference of PIL in between.</p>

username: Tejan_Mehndiratta
cooked: <p>I don’t understand. Could you please explain in detail?</p>