akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Mismatched dimensions when loading model on CPU

John_Munroe: <p>Hi</p>
<p>I’m having problems loading a GPU trained model on CPU. The best model was saved with:</p>
<pre><code class="lang-auto">        data = {'opt':self.opt.state_dict(),'d':self.state_dict()}
        t.save(data, path)
</code></pre>
<p>I tried loading the best model saved (on CPU) using the code below:</p>
<pre><code class="lang-auto">        checkpoint = torch.load('bestcheckpoint', map_location='cpu')
        model.load_state_dict(checkpoint['d'])
</code></pre>
<p>Unfortunately I keep getting an error saying:</p>
<pre><code class="lang-auto">    size mismatch for content_convs.3.4.bias: copying a param of torch.Size([200]) from checkpoint, where the shape is torch.Size([250]) in current model.
	size mismatch for content_convs.3.4.weight: copying a param of torch.Size([200]) from checkpoint, where the shape is torch.Size([250]) in current model.
	size mismatch for content_convs.3.4.running_mean: copying a param of torch.Size([200]) from checkpoint, where the shape is torch.Size([250]) in current model.
	size mismatch for fc.0.weight: copying a param of torch.Size([2000, 1200]) from checkpoint, where the shape is torch.Size([2000, 2000]) in current model.
</code></pre>
<p>However, the same model can be loaded on GPU perfectly. Does anyone know what the dimensions differ when it’s loaded on CPU?</p>
<p>Thanks in advance</p>

John_Munroe: <p>OK. Solved. It was just a mistake.</p>

zhangtingyu11: <p>I got the similar error.Can you tell me how you solve this problem?Thanks in advance.</p>
Model parallel get network name pytorch

ooodragon: <p>Hello!<br>
I’m using nn.Dataparallel and tried to get its name by model.<strong>class</strong>.<strong>base</strong><br>
but it does not work with nn.Dataparallel which only gives string (DataParallel)</p>
<p>is there any solution without changing the model’s status?</p>

ptrblck: <p>After wrapping a model in <code>nn.DataParallel</code>, you can access it as <code>model.module</code>.<br>
So you might need to add this additional <code>.module</code> keyword to your call.</p>
<p>Let me know, if that would work.</p>

ooodragon: <p>thanks!<br>
I totally forgot that!<br>
So dataparallel is a wrapper?<br>
May I ask you what it really does and HOW it distributes data to several gpus if you are available?</p>

ptrblck: <p><code>nn.DataParallel</code> will create model replica on all specified devices. During the forward pass, the data will be split in dim0 and each chunk will be sent to the corresponding device.<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html">This tutorial</a> gives you more information regarding the shapes and <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a> explains the overall work flow as well as some disadvantages.</p>

ooodragon: <p>Thanks! U r the best</p>
Poor performance of dynamic quantization

estrulyov: <p>Hi all,</p>
<p>I read about dynamic quantization and wanted to replicate it in our environment. I actually got pretty good results… when using a single thread. However, we have a multithreaded GRPC application and I did not see any improvement when using multiple threads. In fact with 10 threads, the original (non-quantized) model actually had better throughput, although it was also using 5x CPU.</p>
<p>So I started to wonder if locking is to blame, specifically python’s GIL. Profiling our code revealed that it spends by far the majority of its time in linear_dynamic function which is defined here:  <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qlinear_dynamic.cpp</a></p>
<p>I noticed that code in python_torch_functions.cpp makes an explicit call to release the GIL:</p>
<p>pybind11::gil_scoped_release no_gil;</p>
<p>But I don’t see the same call in qlinear_dynamic.cpp</p>
<p>I tried to add this myself but I am having difficulty compiling torch from source. It’s much more complicated than  the documentation makes it out to be. I also don’t have enough information to determine whether apply_dynamic_impl() method is thread-safe.  Can any pytorch engineers comment? There is potentially a massive performance bottleneck here.</p>
<p>thanks,</p>
<p>Eugene</p>
Equivalent of keras.Model in pytorch

rebeen: <p>I am trying to combine two models and I need something similar to Keras.Model in PyTorch could you please let me know<br>
<code>    combine_model = Model (inputs=inputs, outputs=[first_model, second_model])</code></p>

Nikronic: <p>Hi,</p>
<p>It is similar to <code>nn.Module</code> in PyTorch. You can combine your classes however you want to creating a class that derives <code>nn.Module</code>. Here is example:</p>
<pre><code class="lang-auto">class Combined(nn.Module):
    def __init__(self, model1, model2):
        super(Combined, self).__init__()
        self.first_model=model1
        self.second_model=model2

    def forward(self, inputs):
        outputs1 = self.first_model(inputs)
        outputs2 = self.second_model(inputs)
        # or anything you want can be done here
        return outputs1, outputs2
</code></pre>
<p>Bests</p>

rebeen: <p>Thank you very much,</p>
Return last non-zero element of each zero-padded row in a 2D array

Steven_Chan: <p>Say I have a 2D array in which each row is zero-padded to the maximum length:</p>
<blockquote>
<p>[[1, 2, 0],<br>
[3, 0, 0],<br>
[4, 5, 6]]</p>
</blockquote>
<p>And I also have the length of the rows before they were padded:</p>
<blockquote>
<p>[2, 1, 3]</p>
</blockquote>
<p>I want to return the last non-zero element of each row, i.e.,</p>
<blockquote>
<p>[2, 3, 6]</p>
</blockquote>
<p>Is there an elegant(no loops) to achieve this?</p>

Nikronic: <p>Hi,</p>
<p>I think this can help:</p>
<pre><code class="lang-auto">#inputs
x = torch.tensor([[1, 2, 0],
                  [3, 0, 0],
                  [4, 5, 6]])
y = torch.tensor([2, 1, 3]) - 1

# creating one-hot mask
# borrowed from https://discuss.pytorch.org/t/convert-int-into-one-hot-format/507/4?u=nikronic
mask = torch.LongTensor(len(y), len(y))
mask.zero_()
mask.scatter_(1, y.view(-1, 1), 1)

# applying mask to zero non desired values
res = x*mask

# extract values that are non zero in each row
res[tuple(res.nonzero().T)]
</code></pre>
<p>But I am not sure there is any better solution or not, it just came to my mind. I hope it helps!</p>
<p>Bests</p>
Instance Norm Batch Size

chinmay5: <p>I am using InstanceNorm3D in my model. Due to memory constraints I can use a batch size of only 1 and that is confusing me a bit. I know that batch size &lt; 8 is not good at all for BatchNorm and can lead to poor results. I am not sure if the same holds true for Instance Norm. Can I use InstanceNorm with batch size of 1 and still expect correct results?</p>
Torch is not found in jupyter notebook

Aditya_Agarwal: <p>I am using Ubuntu 18.04 and I have installed the anaconda and installed the pytorch using <code>conda install pytorch-cpu torchvision-cpu -c pytorch</code>. The import of pytorch works well in terminal but whenever I try to import it in jupyter notebook, it fails.</p>
<p>I have installed the whole anaconda two times but it is not working. I have tried with different environment but it is not working. I have also tried to update anaconda but it already up to date and also have tried to downgrad mkl to 2018</p>

albanD: <p>Hi,</p>
<p>This is most certainly because your notebook is not using the same python as anaconda. You need to make sure that the notebook’s python is the one where you installed pytorch.</p>

Aditya_Agarwal: <p>How can I know that which python my jupyter notebook is using and how can I change to use conda python?<br>
New to this jupyter stuff😅</p>

albanD: <p>I don’t use jupyter either so I’m not sure <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:">  You should be able to find info on google. And you can use <code>.__file__</code> on any python object to see where it comes from. You can do <code>import os; print(os.__file__)</code> to see where the os library is, that should give you a hint of where the current python is installing packages <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Aditya_Agarwal: <p>The problem has resolved. I had two versions of jupyter notebook , one of pip and one of anaconda. I simply removed the pip packages and now it is working like a gem.<br>
Thanks everyone for help.</p>

Wisdom: <p>It doesn’t work for me. I’ve got no pytorch of pip. Both jupyter and anaconda run on python 3.7.7. Yet… Can someone smart tell me what to do? I’ve tried heaven and earth for a couple of days now without any success. All info from the internet has definitely been failing me.</p>
Gradient accumulation causes worse generalization

seliad: <p>I notice that when I accumulate gradients there is consistently worse generalization.<br>
(by small number), e.g perplexity 12.24 vs 12.02</p>
<p>This happens especially when combining it with checkpointing, I did not notice checkpointing alone harms generalization for my models.</p>
<p>I checked this for several tasks:<br>
I tried it in deterministic and non-deterministic cudnn modes, I tried it with LayerNorm to avoid batch norm problems.</p>
<p>I wonder if it could be  about LayerNorm epsilon, maybe I should change it when I scale the loss for gradient accumulation? or maybe its an overflow?</p>
<p>Any suggestions would be appreciated</p>

albanD: <p>Hi,</p>
<p>One thing to check with checkpointing is that at least one input to the module needs to require gradients. So if it is the first module in your net, and your input does not require gradient, it can prevent the first layer from training. Setting the input to require gradient is a simple workaround.</p>
<blockquote>
<p>I notice that when I accumulate gradients there is consistently worse generalization</p>
</blockquote>
<p>What do you mean by accumulate gradients? You mean using <code>.backward()</code> vs <code>autograd.grad()</code> ?</p>

seliad: <p>But as long as there is some nn.Parameter the result will require grad so not sure why is that necessary?</p>
<p>By accomulating gradients I mean several<br>
forward and backward passes without calling zero_grad between them.<br>
The gradients are accomulated in .grad field.<br>
It’s used for example to simulate bigger batches (together with loss scaling).</p>

albanD: <blockquote>
<p>But as long as there is some nn.Parameter the result will require grad so not sure why is that necessary?</p>
</blockquote>
<p>The output of the layer yes. But for the first layer, the input does not. And it is a limitation of the checkpoint module that at least one input needs to require gradient if you want to be able to backprop through it.</p>
<blockquote>
<p>By accomulating gradients I mean several<br>
forward and backward passes without calling zero_grad between them.</p>
</blockquote>
<p>No it should not make any difference.</p>
Interop PyTorch with TensorFlow(Lite)

Peter_Peter: <p>Hi,</p>
<p>Are there some torch scripts, modules, converter options using PyTorch to import/export models of TensorFlow (pb, pbtx, h5) and TensroflowLite format (tflite)?<br>
Or any idea how to import models, do some processing (e.g PoststTrainingQuantization, QAT, other optimizations, etc…) in PyTorch and export model?</p>
<p>Thank you</p>
Pytorch with mixed precision

tyterry: <p>Hi all,</p>
<p>I am trying to implement mixed precision training in pytorch, which would involve scaling the loss (computed in half precision) with a scaling factor and compute the gradient, then update the master model (in one precision) with the computed gradient divided by the same scale factor.</p>
<p>I know that I can extract the gradient by:</p>
<p>print(model.layer.weight.grad)</p>
<p>However, are there any ways that I could manually manipulate such gradient (say division/multiplication) and apply it to update another model for mixed precision training?</p>
<p>Many thanks for your help!</p>

albanD: <p>Hi,</p>
<p>I guess you can do:</p>
<pre><code class="lang-auto">for orig_p, new_p in zip(model.parameters(), other_model.parameters()):
    new_val = your_scaling(orig_p.grad)
    new_p.grad = new_val
</code></pre>

ptrblck: <p>Additionally to <a class="mention" href="/u/alband">@albanD</a>’s suggestion, you could scale the loss before the <code>backward</code> call, which would automatically scale all gradients and unscale the gradients before the update.</p>
<p>Note that automatic mixed-precision is currently available in the <a href="https://pytorch.org/docs/master/amp.html">master and nightly binaries</a>, in case you don’t want to reimplement it manually.</p>

tyterry: <p>Much thanks for <a class="mention" href="/u/alband">@albanD</a>’s and <a class="mention" href="/u/ptrblck">@ptrblck</a>’s replies. I will try out the automatic mixed-precision.</p>
How to combine gradients

perryd37: <p>I’m trying to combine the gradients of different instances of the same model but not really sure how to do it. I’m pretty new to PyTorch so I thought iterating through lists of gradients would work but printing out the gradients they seem to be in a format I’m not familiar with. Any help or advice would be greatly appreciated, as I’m not sure where to go from here. Thank you!</p>

albanD: <p>Hi,</p>
<p>Does something like this works?</p>
<pre><code class="lang-auto">for p1, p2, p3 in zip(model1.parameters(), model2.parameters(), model3.parameters()):
    combined = your_fn(p1.grad, p2.grad, p3.grad)</code></pre>
Custom dataset normalization

Javier_Rodriguez_Pui: <p>Hi I am applying normalization to SceneNet Dataset <a href="https://robotvault.bitbucket.io/scenenet-rgbd.html" class="inline-onebox" rel="noopener nofollow ugc">SceneNet RGB-D: Photorealistic Rendering of 5M Images with Perfect Ground Truth</a></p>
<p>Like this:</p>
<pre><code class="lang-auto">   transform = transforms.Compose([transforms.Resize((192, 256)),
                                        transforms.ToTensor(),
                                        transforms.Normalize(mean=[0.4451, 0.4262, 0.3959],
                                                             std=[0.2411, 0.2403, 0.2466])])
</code></pre>
<p>and I am getting images like this one.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03cbf06037d863d0620cc80dafe768f94ba6ee0f.png" alt="image" data-base62-sha1="xAmUSuH5nj5OpGdwqFUeJfK4Uf" width="640" height="480"><br>
To calculate the mean and std per channel I used the function “online_mean_and_sd” from here: <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/18" class="inline-onebox">About Normalization using pre-trained vgg16 networks - #18 by amit_js</a></p>
<p>is the normalization process correct?</p>

Nikronic: <p>Hi,</p>
<p>Can you please share the way you visualize your images? Even if the std and mean are not correct, these values are near to ImageNet statistics so this kind of image cannot be produced (so strange).</p>
<p>Bests</p>

Javier_Rodriguez_Pui: <p>Here: =)</p>
<pre><code class="lang-auto">def imshow(img, title=""):
    transform = transforms.ToPILImage()
    img = transform(img)
    plt.title(title)
    plt.imshow(img.convert("RGB"))
    plt.show()
</code></pre>

Nikronic: <p>Ow, I see, you need to unnormalize images after normalizing them for visualization purposes.</p>
<p>Here is small snippet that does this:</p>
<pre><code class="lang-auto">img = img.numpy().transpose((1, 2, 0))
mean = np.array([0.4451, 0.4262, 0.3959])  
std = np.array([0.2411, 0.2403, 0.2466])])
img = std * img + mean
img = np.clip(img*255, 0, 255)
plt.imshow(img)
</code></pre>
How to deploy C++ LibTorch in Windows XP 32bit?

sakura: <p>I want to deploy a CNN model in Windows XP 32 bit, and here are my operations:</p>
<ol>
<li>compile LibTorch-1.4.0 32bit with VS2017;</li>
<li>finish the C++11 code and the .exe run successfully in win10 and win7 32bit;</li>
<li>The code fails in XP which reports “MSVCP140.dll is invalid”.</li>
</ol>
<p>I want to use VS2015_XP to compile and avoid the error in XP. However, the LibTorch uses C++11/14 which is not supported by VS2015. What should I do to run successfully in XP 32bit? Need help! TAT</p>

ptrblck: <p>I cannot be of much help regarding the current error, but you should note that Windows XP is not supported since April 2014 anymore, so I would recommend to abandon it completely.</p>

sakura: <p>Thanks for your reply~ Maybe XP is indeed too old… <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed_relieved.png?v=9" title=":disappointed_relieved:" class="emoji" alt=":disappointed_relieved:"></p>

ptrblck: <p>The error might be unrelated to XP, but even if you get it working, you would create a honeypot with your machine and I would be really concerned to use an unsupported (and thus potentially vulnerable OS). <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Force dropout to ignore specific index

Crisostomi: <p>Hello,</p>
<p>I am working on Semantic Role Labeling, I have tensors which are the result of concatenating the BERT embedding of a word and a bit which indicates where the predicate is in the sentence, so I have tensors of shape (batch_size, sequence_len, bert_embedding+1). I pass these to a dropout layer but the fact that the indicator bit will be zeroed is not desirable. Is there a way to tell dropout not to apply to that index?</p>
<p>Thanks in advance.</p>

ptrblck: <p>There is no option for the <code>nn.Dropout</code> layer or the functional <code>F.dropout</code> call.<br>
However, you could simply create a mask using <code>torch.bernoulli</code> and create a custom dropout layer.<br>
Here is a simple exmaple:</p>
<pre><code class="lang-python">x = torch.randn(10)
p = 0.5
probs = torch.cat((torch.ones(2,), torch.tensor([p]*8))) # don't zero out first 2 values
mask = torch.bernoulli(probs)

out = x * mask
</code></pre>
<p>Don’t forget to add the scaling during training or validation as is done for the vanilla dropout. Otherwise the expected values will differ when you disable your dropout and the model might perform bad.</p>

Crisostomi: <p>Thank you,</p>
<p>regarding the scaling, is that something that is done in the nn.Dropout layer?</p>

ptrblck: <p>Yes, this is done during training with a scale factor of <code>1/(1-p)</code> as seen here:</p>
<pre><code class="lang-python">drop = nn.Dropout(p=0.8)
x = torch.ones(10)
out = drop(x)
print(out)
&gt; tensor([0., 0., 0., 0., 5., 5., 0., 0., 0., 0.])
</code></pre>
<p>The <a href="https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf">dropout paper</a> explains the scaling in section 10.</p>
Equivelent or analogous pytorch version of `tf.train.Checkpoint`

Nathan_Wood: <p><code>tf.train.Checkpoint</code> and <code>tf.train.CheckpointManager</code> is incredibly useful for training and saving models.</p>
<p>The API works like this:</p>
<pre><code class="lang-python">ckpt = tf.train.Checkpoint(model=model, optimizer=optimizer, step=step)
manager = tf.train.CheckpointManager(ckpt, 'ckpt', max_to_keep=3, keep_checkpoint_every_n_hours=1)

# train loop
for x, y in dataset:
    ...
manager.save()
</code></pre>
<p>Is there anything or any library with similar functionality in PyTorch? I understand how to save things using a <code>state_dict</code> but would like to know if there’s a method that’s as simple as TF’s <code>Checkpoint</code>.</p>

ptrblck: <p>If you don’t want to manually store the <code>state_dicts</code>, then I would recommend to check out some higher-level APIs such as Ignite and Lightning, which both have their checkpointing utility functions.</p>
<p>CC <a class="mention" href="/u/vfdev-5">@vfdev-5</a> and <a class="mention" href="/u/williamfalcon">@williamFalcon</a> for more information.</p>

vfdev-5: <p><a class="mention" href="/u/nathan_wood">@Nathan_Wood</a> concerning <a href="https://github.com/pytorch/ignite" rel="nofollow noopener">Ignite</a>, we have <a href="https://pytorch.org/ignite/handlers.html#ignite.handlers.Checkpoint" rel="nofollow noopener">Checkpoint class</a> that can be used to store 1) training checkpoints with model, optimizer, lr scheduler etc and 2) best N models.</p>
<p>Complete example of usage can be found <a href="https://github.com/pytorch/ignite/blob/master/examples/contrib/cifar10/" rel="nofollow noopener">here</a>.</p>
<p>HTH</p>
<p>PS: Please, take a look and feel free to leave us a feedback <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Change input shape of pretrained model to process input like this [8, 30, 3, 256, 256]

Muhammad_Abubakar: <p>I am building a classifier using MRIs with pretrained alexnet so my batch size has become the number of MRI slices for example one MRI have 30 slices so the input shape becomes [30, 3 , 256, 256] but i want to parallelize the training by passing batches of MRIs, lets say batches of 8 MRIs and the input shape will be [8, 30, 3, 256, 256]. How can i alter Alexnet or any pretrained model from torchvision to accept this input size?</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/transfer-learning-usage-with-different-input-size/20744/12">here</a> with answer.</p>
Make (off-)diagonal matrix from a given tensor of shape `(batch_size, seq_len)`

seewoo5: <p>I have a tensor of shape <code>(batch_size, seq_len)</code>. From this, I want to make a tensor of shape <code>(bs, seq_len+1, seq_len+1)</code> where each tensor corresponds to a single batch is an off-diagonal matrix with entries from given tensor.</p>
<p>More precisely, from a given tensor<br>
x = [[1, 2, 3], [4, 5, 6]],<br>
I want to make a function that returns<br>
y = [[[0, 1, 0, 0], [0, 0, 2, 0], [0, 0, 0, 3], [0, 0, 0, 0]],<br>
[[0, 4, 0, 0], [0, 0, 5, 0], [0, 0, 0, 6], [0, 0, 0, 0]]].</p>
<p>I can’t directly apply <code>torch.diag</code> since it gives diagonals of given tensor when the tensor has dimension &gt; 1. I think it is possible to do with <code>torch.diagflat</code>, but it is highly memory-inefficient since it constructs a tensor of shape <code>(batch_size * seq_len - 1, batch_size * seq_len - 1)</code>.</p>
<p>EDIT: I just found that, for my purpose, it is enough to construct following function: transform<br>
x = [[1, 2, 3], [4, 5, 6]]<br>
to<br>
y = [[[1, 0, 0], [0, 2, 0], [0, 0, 3]],<br>
[[4, 0, 0], [0, 5, 0], [0, 0, 6]]]<br>
i.e. transform each row vectors into diagonal matrices, not off-diagonals.</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">x = torch.tensor([[1, 2, 3], [4, 5, 6]]).float()
idx = torch.arange(3).view(1, 1, 3).repeat(2, 1, 1)
res = torch.zeros(2, 3, 3).scatter_(1, idx , x.unsqueeze(1))
</code></pre>
Output features of conv layers inconsistant after pruning a empty filter

AlvinZheng: <p>Hi~<br>
I encountered a strange phenomenon: after pruning an empty filter in a conv layer, the output features of the conv is inconsistent with the previous one, (The shape change is as expected, what I said is the value inconsistency)<br>
To reproduce:</p>
<pre><code class="lang-auto">c1=torch.nn.Conv2d(3,10,7,bias=False)
c2=torch.nn.Conv2d(3,9,7,bias=False)
for i in range(9):
    c1.weight.data[i] = c2.weight.data[i]

c1.weight.data[9]=torch.zeros(3, 7, 7)
data = torch.rand(64,3, 224,224)
out1=c1(data)
out2=c2(data)
print(torch.sum(out1))
print(torch.sum(out2))

</code></pre>
<pre><code class="lang-auto">tensor(2090009.8750, grad_fn=&lt;SumBackward0&gt;)
tensor(2090010.1250, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>The sum of the output features of c1 and c2 are different. c1 only have one more empty filter than c2, and they convolve the same input data, so, the sum of their output features should be the same theoretically.  Why the sum of these two output features are different? Is this a bug? or pytorch has some optimization of the convolution operation that results in this inconsistency?</p>

ptrblck: <p>The difference is most likely created due to the limited floating point precision and a different order of operations as shown in this example:</p>
<pre><code class="lang-python">x = torch.empty(100, 100, 100).uniform_(0, 100)
print(x.sum())
&gt; tensor(49965816.)
print(x.sum(0).sum(0).sum(0))
&gt; tensor(49965808.)
</code></pre>
<p>Also, you shouldn’t use the <code>.data</code> attribute, as it might yield unwanted side effects.</p>
Out Of Memory error in GPU when performing random search

ire: <p>I am training an lstm model and currently I am performing random search.</p>
<p>Initially after each random search I was emptying the cache (<code>torch.cuda.empty_cache()</code>), however I was getting the OOM error after some number of random searches (usually around 3).</p>
<p>Then I read that in order for the memory to be freed I need to do <code>del variable</code> first. However, even after that I continued having the same issue.  I am tracing the allocated gpu memory ( <code>torch.cuda.</code> <code>memory_allocated()</code>) and I can see that after each random search the memory is being freed. Although, when a new random search starts the memory allocated is a bit higher than the previous random search. I don’t think that this is caused by some variable that is not erased. Is there something that I am missing?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8c35acb9f4e28ca809761a1b6431fdf4b202ec9.png" alt="image" data-base62-sha1="xd7jKFIbuvzi2hEnWo3dMrgWi93" width="379" height="167"></p>

ptrblck: <p>What is the random search doing? Could it change some hyperparameters and thus increase the model parameters, which could yield the OOM issue?</p>
Keyerror when for loop dataloader

Jinyoung: <p>When I use Dataloader, It occurs error below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dd5d6ed30a46579e8072bad27dc065ef74139b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/6dd5d6ed30a46579e8072bad27dc065ef74139b8" title="Screenshot from 2020-06-08 13-06-13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dd5d6ed30a46579e8072bad27dc065ef74139b8_2_690x330.png" alt="Screenshot from 2020-06-08 13-06-13" data-base62-sha1="fFEbZuw0ybjgIZWBbpkDontUH9K" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dd5d6ed30a46579e8072bad27dc065ef74139b8_2_690x330.png, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dd5d6ed30a46579e8072bad27dc065ef74139b8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dd5d6ed30a46579e8072bad27dc065ef74139b8.png 2x" data-dominant-color="451C38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-06-08 13-06-13</span><span class="informations">1016×487 89.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t know why the error occurs. Is there any problem raising this error?</p>
<p>Thank you</p>

ptrblck: <p>Could you use <code>num_workers=0</code> and rerun the script?<br>
If that’s working, could you iterate the <code>Dataset</code> directly without wrapping it into a <code>DataLoader</code>?</p>
<p>PS: It’s better to post code directly by wrapping it into three backticks ```, as it makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Jinyoung: <p>Thank ptrblc!<br>
When num_workers=0 is set, the problem is solved.<br>
And as you mentione, I iterated the Dataset directly with no errors</p>
<p>Thank you for your reply!!</p>

ptrblck: <p>That’s weird, as the <code>DataLoader</code> seems to raise this error even though you can successfully iterate the <code>Dataset</code>.<br>
Are you somehow manipulating the dataset inside your training loop?</p>
Assessing model accuracy using Tensor board

alameer: <p>Could anyone please provide an example of how i could assess model accuracy using Tensor board? Many Thanks</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html">TensorBoard tutorial</a> gives you some information on how to plot statistics etc., so you could treat your current accuracy as another statistics such as the loss.</p>
Reduce dimension by selecting along one domain inside another domain

weidler: <p>I have a tensor of order 5 and want to reduce it to order 4 by selectively picking along the third domain based on the position in the second domain.</p>
<p>Let’s say my original tensor’s shape is <code>(1, 16, 16, 8, 8)</code> and I want to get <code>(1, 16, 8, 8)</code>. So far, I am doing this as follows iteratively:</p>
<pre><code class="lang-auto">import torch

original_tensor = torch.randint(10, (1, 4, 4, 2, 2))
output_tensor = torch.empty((1, 4, 2, 2))
for i in range(output_tensor.shape[1]):
    output_tensor[:, i, :, :] = original_tensor[:, i, i, :, :].squeeze(1)
</code></pre>
<p>Putting the outputs in here would make the post very long, but you can execute the snippet on its own and print out both <code>original_tensor</code> and <code>output_tensor</code>. However for a second order into first order tensor consider this example:</p>
<pre><code class="lang-auto">ORIGINAL_TENSOR
tensor([[5, 8, 2, 5],
        [1, 4, 5, 8],
        [1, 5, 6, 0],
        [3, 2, 7, 2]])

OUTPUT_TENSOR
tensor([5., 4., 6., 2.])
</code></pre>
<p>Iteratively, as you can imagine, this is way to slow given that as part of a layer I need to execute this a lot. I do though beforehand know (that is, it is constant) the indices which i want to keep. So I could precalculate some sort of mask if that simplifies it.</p>
<p>Please consider the added complexity due to the order and size of the real example I gave in the first snippet. A solution that just helps with the particular second example would not help.</p>

ptrblck: <aside class="quote no-group" data-username="weidler" data-post="1" data-topic="84942">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weidler/48/26047_2.png" class="avatar"> weidler:</div>
<blockquote>
<p>by selectively picking along the third domain based on the position in the second domain.</p>
</blockquote>
</aside>
<p>I don’t quite understand this use case, but based on your loop approach, you could use <code>arange</code> to index the tensor via:</p>
<pre data-code-wrap="python"><code class="lang-python">original_tensor[:, torch.arange(original_tensor.size(1)), torch.arange(original_tensor.size(2))]
</code></pre>
<p>which would result in the same output as <code>output_tensor</code>.</p>

weidler: <p>Perfect! Thank you!</p>
<p>Am I correctly assuming that this works because the slicing associates the indices of the slices for the individual domains? Because intuitively it feels weird to just give a list of all available dimensions as a list, as I would have expected this to return the complete tensor.</p>

ptrblck: <p>Yes, this indexing will be equal to your loop, such that both dimensions are indexed with <code>[0, 0], [1, 1], [2, 2], ...</code>, if you use <code>torch.arange</code>.</p>
Dataloader for Librispeech based on .json

Stefan_Pushkov: <p>Hi! I am trying create dataloader in pytorch for asr model training. I have a .json (attached below).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/37220027/84420079-2baf5d80-ac22-11ea-8820-884476e4cda5.png" title="json example" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/37220027/84420079-2baf5d80-ac22-11ea-8820-884476e4cda5.png" alt="json example" width="690" height="388"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">json example</span><span class="informations">1920×1080</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div>. How can i make a dataloader? Thanks!</p>
"preserve_rng_state" in gradient checkpoint

Mainul: <p>I am trying to use <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">gradient checkpoint</a> so that I can fine-tune a huge transformer model in 12 GB GPU.<br>
I am confused about the argument  <code>preserve_rng_state</code>.<br>
Basically I don’t understand the following section from that official documentation.</p>
<blockquote>
<p>Checkpointing is implemented by rerunning a forward-pass segment for each checkpointed segment during backward. This can cause persistent states like the RNG state to be advanced than they would without checkpointing. By default, checkpointing includes logic to juggle the RNG state such that checkpointed passes making use of RNG (through dropout for example) have deterministic output as compared to non-checkpointed passes. The logic to stash and restore RNG states can incur a moderate performance hit depending on the runtime of checkpointed operations. If deterministic output compared to non-checkpointed passes is not required, supply  <code>preserve_rng_state=False</code>  to  <code>checkpoint</code>  or  <code>checkpoint_sequential</code>  to omit stashing and restoring the RNG state during each checkpoint.</p>
<p>The stashing logic saves and restores the RNG state for the current device and the device of all cuda Tensor arguments to the  <code>run_fn</code> . However, the logic has no way to anticipate if the user will move Tensors to a new device within the  <code>run_fn</code>  itself. Therefore, if you move Tensors to a new device (“new” meaning not belonging to the set of [current device + devices of Tensor arguments]) within  <code>run_fn</code> , deterministic output compared to non-checkpointed passes is never guaranteed.</p>
</blockquote>
<p>I have a basic question:</p>
<ul>
<li>What is the meaning of this sentence:</li>
</ul>
<blockquote>
<p>This can cause persistent states like the RNG state to be advanced than they would without checkpointing.</p>
</blockquote>
<p>What I understand is, random number generator only needed for weight initialization of the layers which happens in model initialization section not in the forward method of the model. Am I right?<br>
If so, what is RNG state has anything to do with checkpointing?<br>
A simple example would be very helpful.</p>

Ognjen_Kocic: <p>Random number generator is needed for layers such as Dropout as well. If you checkpoint module which contains Dropout you want you dropout mask to be the same when doing forward pass and when re-computing tensors in backward pass.</p>
How I can achieve this in side dataset get

Jaideep_Valani: <p>I have a python  list of some items   with in dataset get function . List has got just 3 to 5 elements.<br>
Is there a function that can select distinct elements from  list for every dataset item in a new epoch .<br>
Say get (i1)  epoch 1 ,select an element x<br>
get(i1) epoch 2 ,select element y from list<br>
Once all elements done then elements can be reselected in subsequent epochs .<br>
Random.sample and shuffle dint guarantee selection of different elements before all elements are exhausted.</p>

ptrblck: <p>You could store an internal element counter and increase it after each epoch in the dataset.<br>
Here is a simple example:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(5, 1)
        self.elements = torch.arange(3)
        self.element_idx = 0
        
    def __getitem__(self, index):
        x = self.data[index]
        element = self.elements[self.element_idx % len(self.elements)]
        return x, element
    
    def __len__(self):
        return len(self.data)
    
dataset = MyDataset()
loader = DataLoader(dataset, batch_size=2, num_workers=2)

for epoch in range(5):
    for data, element in loader:
        print('epoch {}, element {}'.format(epoch, element))
    
    # increase element_idx
    loader.dataset.element_idx += 1
</code></pre>
<p>Let me know, if this would work for you.</p>
Complexity of '=='

DDavid: <p>What is the complexity of the following when the length of <code>vector</code> is <code>n</code> :</p>
<blockquote>
<p>import torch<br>
vector = torch.tensor([5, 8, 5, 8, 5, 9, 1, 3, 2, 5, 3, 1])<br>
f1 = vector.unsqueeze(1)<br>
f2 = vector.unsqueeze(0)<br>
matches = (f1 == f2)<br>
matches<br>
tensor([[1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0],<br>
[0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0],<br>
[1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0],<br>
[0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0],<br>
[1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0],<br>
[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0],<br>
[0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1],<br>
[0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],<br>
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],<br>
[1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0],<br>
[0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0],<br>
[0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]], dtype=torch.uint8)</p>
</blockquote>
<p>I thought the complexity of this command is <code>O(n^2)</code> due to the <code>==</code>. But the speed of the code is way faster than I thought.</p>
Installing pytorch in anaconda env

Hyemin_Ju: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae7a1a15cc12c12c6e89825d4b1ae4dfc2cd9cb2.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae7a1a15cc12c12c6e89825d4b1ae4dfc2cd9cb2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae7a1a15cc12c12c6e89825d4b1ae4dfc2cd9cb2.png" alt="image" data-base62-sha1="oTuGhsBZ8WvusAnAH3csnYV2uqK" width="485" height="499" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">642×661 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My env’s name is “torch” and the version of cuda that I used is 10.1<br>
Although I installed pytorch in my env “torch” (I also checked), but ModuleNotFountError occured…<br>
"ModuleNotFoundError: No module named ‘pytorch.’</p>
<p>I tried to defuse the issue by updating anaconda or create new environment by another version cuda… but it did not work. I wanna install pytorch in “torch” env, and apply it to pycharm project.<br>
What should I do?</p>

harsha_g: <p>It should be <code>import torch</code> and <strong>not</strong> <code>import pytorch</code> as you may have typed.</p>
Fail to load parameters with Spectral norm?

Wei_Sun: <p>I tried to train a generator and save the state_dict. But got error <code>KeyError: 'conv1_1.weight'</code> when load the saved state_dict.</p>
<p>When I printed out keys of saved state_dict, only find <code>conv1_1.weight_orig</code>, <code>conv1_1.weight_u</code> and <code>conv1_1.weight_v</code>.</p>
<p>How can I solve this issue? Or how can I save the trained model to make it loadable? Thanks</p>

aluo-x: <p>In case you are still suffering from this issue, check that the ‘_metadata’ attribute is not being removed from the state_dict that you load.</p>

ljqcava: <p>How did you implement your SN layer, with <code>torch.nn.utils.spectralnom</code> or something like <a href="https://github.com/heykeetae/Self-Attention-GAN/blob/master/spectral.py" rel="nofollow noopener">spectralnorm</a>?</p>

aluo-x: <p>Yes, see <a href="https://github.com/NVlabs/SPADE/blob/6d3aa2b8390c8014939cdc529f810b2dd42d9777/models/networks/normalization.py#L27" rel="nofollow noopener">SPADE</a> for an example</p>

mazzzystar: <p>Great ! Fixed this issue by keeping “_metadata” attribute not deleted.</p>
Renaming Model Object

bendangnuksung: <p>Model:</p>
<pre><code class="lang-auto">EfficientDet(
  (backbone): EfficientNetFeatures(
    (conv_stem): Conv2d(4, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
...
...
</code></pre>
<p>Is there a way to rename <code>backbone </code> object to some other name?</p>

Nikronic: <p>Hi,</p>
<p>Actually, <code>nn.Module</code> is a <code>OrderedDict</code> and a python code can change it by traversing over keys/values in this dictionary. Here is an example:</p>
<pre><code class="lang-auto">from collections import OrderedDict
model.__dict__['_modules'] = OrderedDict([('new_backbone_name', v) if k == 'backbone' else (k, v) for k, v in model.__dict__['_modules'].items()])
</code></pre>
<p>Bests</p>
Pytorch-Lightning example – Semantic Segmentation for self-driving cars

lavanya: <p><a href="https://app.wandb.ai/borisd13/lightning-kitti/reports/Lightning-Kitti--Vmlldzo3MTcyMw" rel="nofollow noopener">Nice example</a> of using Pytorch-Lightning, and doing hyperparameter search on a semantic segmentation model on the Kitti dataset.</p>
'Undefined symbols for architecture x86_64' for libtorch in swift unit test

Bryan_Wang: <p>When trying to build my tests, I exclusively run into errors such as the following:</p>
<pre><code class="lang-auto">Undefined symbols for architecture x86_64:
  "vtable for c10::AutogradMetaInterface", referenced from:
      c10::AutogradMetaInterface::AutogradMetaInterface() in TorchModule.o
</code></pre>
<p>None of my fixes have worked so far. I have created an objective-c bridging header for my tests, and here is my Podfile:</p>
<pre><code class="lang-auto"># Uncomment the next line to define a global platform for your project
# platform :ios, '9.0'

target 'FinalDancer' do
  # Comment the next line if you don't want to use dynamic frameworks
  use_frameworks!

  # Pods for FinalDancer
  pod 'LibTorch', '~&gt; 1.4.0'

  target 'FinalDancerTests' do
    inherit! :search_paths
    # Pods for testing
    pod 'LibTorch', '~&gt; 1.4.0'
  end

end
</code></pre>
<p>Hope this helps, thanks!</p>

Bryan_Wang: <p>For some reason, my xcconfig for test didn’t have the following line:<br>
<code>LIBRARY_SEARCH_PATHS = $(inherited) "${PODS_ROOT}/LibTorch/install/lib"</code><br>
It was also missing some OTHER_LDFLAGS like pytorch_qnnpack.</p>
<p>Not really sure why or how this fixed it, but I pasted those lines in. There is some code duplication that my compiler is warning me of but my tests are running.</p>

xta0: <p><a class="mention" href="/u/bryan_wang">@Bryan_Wang</a> has your issue been resolved?</p>

Bryan_Wang: <p>Hi <a class="mention" href="/u/xta0">@xta0</a>,</p>
<p>The issue hasn’t been resolved, unfortunately. Instead, there is a runtime error</p>
<pre><code class="lang-auto">Unsupported TypeMeta in ATen: float (please report this error)
</code></pre>
<p>I suspect that it’s this issue: <a href="https://gitmemory.com/issue/pytorch/pytorch/25489/526862641" rel="nofollow noopener">https://gitmemory.com/issue/pytorch/pytorch/25489/526862641</a>, but I’m not sure how to fix it.</p>
<p>I tried looking for examples of unit tests in the Helloworld project, but there don’t seem to be any examples I can pull from. Any help would be appreciated!</p>
<p>Thanks,<br>
Bryan</p>

xta0: <p>Seems like it’s duplicated with <a href="https://github.com/pytorch/pytorch/issues/32040" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32040</a>. I can come up with a PR to disable NNPACK, but not sure if it’ll fix your issue. Anyway, I’ll give an update here once I have the PR.</p>

Bryan_Wang: <p><a class="mention" href="/u/xta0">@xta0</a> Seems like the exact issue I’m experiencing. Thank you!</p>

xta0: <p><a class="mention" href="/u/bryan_wang">@Bryan_Wang</a> can you try to apply this PR?  <a href="https://github.com/pytorch/pytorch/pull/39868" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/39868</a>? Let me know if the issue still exist. Thanks</p>

Bryan_Wang: <p>I’ll let you know what happens!</p>
<p>Edit: Actually, how do I apply this PR to my XCode project? I am quite new to this. Or if you would like me to test on a sample project, I can do that as well.</p>
How to calculate cross entropy loss along the last axis?

xiaoas: <p>I’m new to pytorch and is trying to train a model with cross entropy loss.<br>
the output of my model is of size [miniBatchSize, n, m]<br>
and label is of size [miniBatchSize, n]<br>
where M is the number of categories, label elements are ints between 0 and M - 1<br>
my current approach is</p>
<pre><code class="lang-python">labels = torch.eye(m)[labels]
outputs = pointnet(inputs)
outputs = F.log_softmax(outputs, -1)
loss = labels * outputS
loss = loss.sum()
loss.backward()
</code></pre>
<p>and I have absolutely no idea if it’s gonna work correctly or not. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"><br>
are there any functions that can do the job for me? thx in advance.</p>

Bjorn_Lindqvist: <p>That looks incorrect. The easiest way to solve it is to reshape your output and target to tensors to (bs * n, c) and (bs *n) where bs is the batch size, n the same as your n and c the number of classes:</p>
<pre><code>labels = labels.reshape(-1)
outputs = outputs.reshape(outputs.shape[0] * outputs.shape[1], -1)
</code></pre>
<p>Then you compute the normal cross entropy loss:</p>
<pre><code>loss_fn = CrossEntropyLoss()
loss = loss_fn(outputs, labels)
</code></pre>
<p>There is also a multi-dimensional version of CrossEntropyLoss, but unless your dimensions are in the order it expects, the ordinary one is easier to use.</p>

xiaoas: <p>That’s exactly what I’m looking for, thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Pytorch Equivalent to Keras Conv2d Layer

garthtrickett: <p>I Have a conv2d layer in keras with the input shape from input_1 (InputLayer) [(None, 100, 40, 1)]</p>
<pre><code class="lang-auto">   input_lmd = Input(shape=(T, NF, 1))

    # build the convolutional block
    conv_first1 = Conv2D(32, (1, 2), strides=(1, 2))(input_lmd)
</code></pre>
<p>I’m trying to do the equivalent in Pytorch with something like</p>
<pre><code class="lang-auto"> self.conv1 = nn.Conv2d(100, 32, (1, 2), stride=(1, 2))
</code></pre>
<p>But i’m getting the error</p>
<p>*** RuntimeError: Calculated padded input size per channel: (40 x 1). Kernel size: (1 x 2). Kernel size can’t be greater than actual input size</p>

weidler: <p>The width of your input is 1, but your kernel has width 2. That will not work. You probably want to make your kernel and stride (2, 1) or switch H and W in your input.</p>

garthtrickett: <p>Turns out keras/tensorflow use channel last so I needed to reshape the input tensor to account for this</p>
RuntimeError: The size of tensor a (128) must match the size of tensor b (2) at non-singleton dimension 4

jueqi_wang: <p>I am using <code>BCEWithLogitsLoss</code> as loss function, and the output size and targets size is the same as ·torch.Size([4, 1, 128, 128, 128])· but I run into that error.</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> should work, if the model output and target have the same shape.<br>
Could you print the shape of both inputs again and check, where the size of 2 is coming from?</p>