akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert pytorch model to Nvidia faster transformer?

username: wangxiang2713
cooked: <p>Hi, i want to detect Bert model structure in pytorch model and convert the structure by Nvidia faster transformer op automatically.<br>
Is there any existing project? If not, i want to develop one, so should i develop on origin pytorch or TorchScript? Should i develop a pass to detect Bert in TorchScript IR and replace it by faster transformer op?<br>
Thankyou very much!</p>
Transformer LM in Pytorch Documentation performs poorly

username: Hari_Krishnan
cooked: <p>I trained a Transformer based Language Model using code available in PyTorch documentation, the results are worse than a RNN based LM. The code I’m using is from this part of documentation <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a>.</p>
<p>Here are some output examples</p>
<pre><code class="lang-auto">&lt;eos&gt; @ settlement heavy of , , lined the she &lt;unk&gt; of . interception the dried . , would his 
= . , losses the and and the &lt;unk&gt; was , the the , &lt;unk&gt; &lt;unk&gt; the and be first 

= 1 was rains ireland and starting with hairy had found the &lt;unk&gt; to possibility heads other which receive gift 
= @ the , , the in the , been in &lt;unk&gt; , the of , , are the , 

valkyria rebounds rapid over . truely their lead &lt;unk&gt; recently at year sharif take of as symptoms usually an for 
chronicles , and the the , first to , been the , , a the the , &lt;unk&gt; average the 

chronicles , in the since and drive were leaves left the , , a carpenter ear may happen index seizing 
iii @ the storm the the , the the the &lt;unk&gt; and and touchdown , , be , , the 

iii 3 particular island the son at built are the site he the 56 becoming ornaments include every @-@ on 
, @ by , republic , the in the &lt;unk&gt; , was &lt;unk&gt; @-@ the , &lt;unk&gt; year &lt;unk&gt; the 

= @ , , 1960s &lt;unk&gt; their during crowded &lt;unk&gt; . was cambridge – a . &lt;unk&gt; few linked telling 
= . the the , , first the , , the a , 0 &lt;unk&gt; the , years to the 
</code></pre>
<p>You can see that text generated doesn’t even remotely make sense and is riddled with grammatical mistakes.</p>
<p>Someone else has posted the same issue in the forums and is currently unanswered. Here’s the link for that <a href="https://discuss.pytorch.org/t/transformer-model-of-language-model-example-performs-worse-much/87448" class="inline-onebox">Transformer model of Language Model Example performs worse much</a></p>
<p>Theoretically, wouldn’t transformers produce better results than RNN? What is the cause for this.</p>
Porting LSTM model from Pytorch to ONNX

username: nitya05
cooked: <p>I am trying to convert a very simple LSTM model from Pytorch to ONNX. Even after using a batch size of 1 and specifying h0, c0 inputs, I am getting the following warning:<br>
<strong>UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.<br>
"or define the initial states (h0/c0) as inputs of the model. ")</strong></p>
<p>Please suggest a valid way to successfully port LSTM from Pytorch to ONNX. Attaching my Pytorch code for reference.</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torch.optim as optim</p>
<p>import torch.onnx<br>
import torchvision</p>
<p>torch.manual_seed(1)</p>
<p>lstm = nn.LSTM(3, 3, num_layers=1)<br>
lstm.eval()</p>
<p>with torch.no_grad():<br>
inputs = [torch.randn(1,3) for _ in range(5)] <span class="hashtag">#make</span> a sequence of length 5<br>
inputs = torch.cat(inputs).view(len(inputs), 1, -1)<br>
h0 = torch.randn(1, 1, 3)<br>
c0 = torch.randn(1, 1, 3)<br>
out, (hn, cn) = lstm(inputs, (h0, c0))</p>
<p>input_names = [“input”, “h0”, “c0”]<br>
output_names = [“output”, “hn”, “cn”]</p>
<p>torch.onnx.export(lstm, (inputs, (h0, c0)), “SimpleLSTM_2.onnx”, input_names=input_names, output_names=output_names)</p>
Padding a tensor by replicating itself around it

username: Ykk_L
cooked: <p>I want to pad a tensor (B, C, H, W) by replicating itself around it. For example, the original tensor is</p>
<pre><code class="lang-auto">[[0, 1, 2], 
 [3, 4, 5],
 [6, 7, 8]].
</code></pre>
<p>When padding size=2, the expected results is</p>
<pre><code class="lang-auto">[[4, 5, 3, 4, 5, 3, 4],
 [7, 8, 6, 7, 8, 6, 7],
 [1, 2, 0, 1, 2, 0, 1],
 [4, 5, 3, 4, 5, 3, 4],
 [7, 8, 6, 7, 8, 6, 7],
 [1, 2, 0, 1, 2, 3, 4],
 [4, 5, 3, 4, 5, 6, 7]].
</code></pre>
<p>Furthermore, how can I start convolution at position (2,2) and end at (4,4)?</p>
How to pass a Tensor list to cuda or c++ kernel

username: Yanan
cooked: <p>I can successfully pass a Tensor from the python front end to a cuda kernel’s argument following the tutorial at <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension</a></p>
<p>However, I am wondering how can I pass a Tensor List from python to cuda, as there is no such c or cuda library for a “Tensor List”.</p>
<p>For example, if I have a Tensor List <strong>tensor_list = [tensor1, tensor2]</strong>, and a cuda kernel <strong>add_cu</strong>,<br>
if I want to be able to call add_cu by <strong>tensor3 = add_cu(tensor_list)</strong>, then how should I declare this cuda kernel?</p>
How to convert coordinate to image with tensor API

username: wxystudio
cooked: <p>Suppose I have n points, represented as tensor(n, 2), the second dimension is coordinate, such as (x, y).  I want to convert it to image, represented as (W, H, 1), the first and second dimension represent width and height of the image, and the third dimension is channel.<br>
Now I use opencv to imwrite the points and imread the image, but I want to realize it with <strong>only tensor operation</strong> , Can you help me ,thx</p>
Simple (generalizable) Animal Pose Estimation Baseline

username: Mona_Jalal
cooked: <p>I am looking for a animal pose estimation code that predicts the pose of an animal given 100-500 annotated frames from scratch say for 10-12 keypoints on each frame using deep learning on a frame-by-frame basis. Is there any such code? I am not looking for something like DeepLabCut or DeepPoseKit or LEAP/SLEAP.ai tools. Looking for a simple baseline written in PyTorch that could be easily modified to add my own loss functions or regularizers on top of that.</p>
Calculating the gradients with respect to the input

username: Abueidda
cooked: <p>Hi there,</p>
<p>I have this problem regarding gradient calculation. Any help is appreciated.</p>
<p>Given the input x, the output u is inferenced from a NN model. Then, u is used in calculating the tensor s=f(u). I am trying to find ds/dx.</p>
<p>I really appreciate any help you can provide.</p>

username: LRHammond
cooked: <p>I’m a relative newbie, so you should defer to more experienced people if they comment, but have you tried:</p>
<pre><code class="lang-auto">d = torch.autograd.grad(outputs=f(nn(x)), inputs=x)
</code></pre>
<p>where <code>nn</code> is your model? This should (I think) get you the gradient ds/dx. If you haven’t already read the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py" rel="noopener nofollow ugc">Autograd tutorial</a> I’d recommend it!</p>
Determinant Slow on GPU

username: Jack_Rolph
cooked: <p>I have a tensor, X:</p>
<pre><code class="lang-auto">X.shape = torch.Size([B,N,C,C])
</code></pre>
<p>I am trying to calculate the determinant of the elements along <code>B</code> and <code>N</code>, i.e for each<code> CxC</code> matrix:</p>
<pre><code class="lang-auto">X_det = X.view(-1, C, C)
X_det = torch.det(X_det)
X_det = X.view(B, C).unsqueeze(-1).repeat(1,1, C)
</code></pre>
<p>This function is exceptionally slow on GPU, which is not the case when running on CPU.</p>
<p>I am using Pytorch 1.6.0.</p>
<p>What can I do to implement this function quickly on the GPU?</p>

username: tom
cooked: <p>What’s the size of your tensor, i.e. what are B, N, C, C?<br>
How have you benchmarked?</p>
<p>Generally, the linear algebra algorithms have a harder time to exploit the GPU parallelism than many other tasks as the <a href="https://en.wikipedia.org/wiki/Direct_method_(computational_mathematics)" rel="noopener nofollow ugc">direct methods</a> are fairly sequential.</p>
<p>Best regards</p>
<p>Thomas</p>
PyTorch Quantization

username: samhithaaaa
cooked: <p>Hi all,<br>
I am trying the resnet50 model quantization with PyTorch and I tried these 3 lines of code :<br>
the import, model=qn.resnet50(pretrain=true), and model.state_dict()), and why the coefficients being shown are all float values if this is the quarantined version of the model?<br>
Noticed this while trying to figure out how to save/load the coefficients for a quantized model, and is there anything special you need to do convert model coefficients between float and int8.</p>
<p>Please let me know. Appreciate any help/suggestions , Thanks!</p>

username: raghuramank100
cooked: <p>You need to set quantized=True when you load the model. i.e do model =qn.resnet(pretrained=True, quantized=True)</p>
Use same gradient to maximize one part of the model and minimize another part of the same model

username: Mainul
cooked: <p>I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two optimizers responsible for two part of the network/model and one of the optimizers has a negative learning rate. But it seems that PyTorch does not allow negative learning rate.</p>
<p>In this case what I am doing is:</p>
<pre><code>loss.backward()
optimzer_for_one_part of the model.step() 
</code></pre>
<p>and then</p>
<pre><code>(-loss).backward()
</code></pre>
<p>Problem is, This time the again calculated gradient will not be the same(values are different but flopped of course) because some weights of the same network (same computation graph) have already been changed. But, Ideally, I want to use the flipped version of the previous gradient.<br>
How can I achieve this?</p>

username: tom
cooked: <p>As I wrote on the other thread:</p>
<p>Well, it seems the use case is rare enough to expect that it is an error to want negative learning rates. In the end, even the name loss suggests that you’re trying to minimize it.</p>
<p>But here is an easy workaround: What you could try is to set the learning rate to a negative value after initializing the optimizer ( <code>opt.param_groups[0]['lr'] *= -1</code> or loop over the <code>param_groups</code> if you have several / pick the one you want to ascend with), preferably with a comment explaining what you are up to. <strong>But you are up to your own as to if optimizers work well with it (but SGD and Adam probably should</strong> ).</p>
<p>If you then do wildly successful work with it, maybe one day the restriction will be lifted. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best regards</p>
<p>Thomas</p>
Negative learning rate for gradient ascent

username: cdjhz
cooked: <p>Why do the optimizers require positive learning rate by default?</p>
<p>In some cases, the loss is maximized for one network and minimized for another network. It seems ideal to set one optimizer with positive learning rate and another negative learning rate instead of forwarding the input once more to compute <code>(-loss).backward()</code>.</p>

username: Mainul
cooked: <p>This in not a solution. I am just asking again. I am also facing the same problem. If I do first</p>
<pre><code class="lang-auto">loss.backward()
optimzer_for_one_part of the model.step()
</code></pre>
<p>some weights are changed and then for maximization of the another part:<br>
<code>(-loss).backward()</code><br>
This time the again calculated gradient will not be the same because some weights of the same network has already been changed.<br>
Ideal case would be, same gradient to maximize one part and minimize another part</p>

username: tom
cooked: <p>Well, it seems the use case is rare enough to expect that it is an error to want negative learning rates. In the end, even the name loss suggests that you’re trying to minimize it.</p>
<p>But here is an easy workaround: What you could try is to set the learning rate to a negative value after initializing the optimizer (<code>opt.param_groups[0]['lr'] *= -1</code> or loop over the <code>param_groups</code> if you have several / pick the one you want to ascend with), preferably with a comment explaining what you are up to.  <strong>But you are up to your own as to if optimizers work well with it (but SGD and Adam probably should</strong>).</p>
<p>If you then do wildly successful work with it, maybe one day the restriction will be lifted. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best regards</p>
<p>Thomas</p>
TorchRayLib++: A CMake based integration of the RayLib GUI library and the PyTorch / Libtorch C++ Deep Learning Library

username: dambo
cooked: <p><strong>TorchRayLib++</strong>  is a CMake based  <strong>integration</strong>  of the well-known  <em><strong>raylib GUI</strong></em>  library with my favourite Deep Learning Library Libtorch: the  <em><strong>PyTorch</strong></em>  C++ frontend, mainly for working on the  <em><strong>GPU</strong></em> .</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eae69346ae09d465bc696d6d3ee2b29dcdfa7a59.gif" alt="neuralanim" data-base62-sha1="xw1Jt7fjLpVdr2xMeiE7lGmyMbv" width="600" height="493" class="animated"></p>
<p><strong>Why??++</strong>  Because its fun, for gaming, for deep learning researchers, for AI, for reinforcement learning, because the <strong>LibTorch C++ community does not have a dedicated UI</strong>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88ffc707bd5f2ca851e123c4d718b6cb4e028fa0.png" data-download-href="https://discuss.pytorch.org/uploads/default/88ffc707bd5f2ca851e123c4d718b6cb4e028fa0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ffc707bd5f2ca851e123c4d718b6cb4e028fa0_2_333x250.png" alt="image" data-base62-sha1="jxWW4xFymRsmdAAFs6DBmD6k3E4" width="333" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ffc707bd5f2ca851e123c4d718b6cb4e028fa0_2_333x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ffc707bd5f2ca851e123c4d718b6cb4e028fa0_2_499x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ffc707bd5f2ca851e123c4d718b6cb4e028fa0_2_666x500.png 2x" data-dominant-color="9B4FC4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×690 47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>All the details here:<br>
<a href="https://github.com/QuantScientist/TorchRayLib" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/QuantScientist/TorchRayLib</a></p>
<p>Just clone and open the CMake file in CLion, take a break and have a coffee while everything is downloaded for you.</p>
<p>Best,</p>
nn.Transformer for NMT

username: DonCano
cooked: <p>Im trying to implement the transformer for NMT</p>
<p>so far, my code looks like this</p>
<pre><code class="lang-auto">class transformer(nn.Module):
    def __init__(self,src_vocab_size, trg_vocab_size, emb_dim=512, m_dim = 512, d_model = 512):
        super(transf, self).__init__()
        self.transformer = nn.Transformer(d_model = d_model)
        self.Embedding = nn.Embedding(src_vocab_size, emb_dim)
        self.pos_encoder = PositionalEncoding(emb_dim)
        self.fc_out = nn.Linear(m_dim,trg_vocab_size)
        
    def forward(self, src,trg):
        src = self.Embedding(src)
        src = self.pos_encoder(src)
        trg = self.Embedding(trg)
        trg = self.pos_encoder(trg)
        self.trg_mask = self.transformer.generate_square_subsequent_mask(trg.shape[0]).to(device)
        out = self.transformer(src,trg, src_mask = None, tgt_mask = self.trg_mask)
        out = self.fc_out(out)
        return out


</code></pre>
<p>the input is<br>
src = SRC_LENGTH, BATCH_SIZE<br>
trg = TRG_LENGTH,BATCH_SIZE</p>
<p>but im getting a loss too small. I think is because of the attention mask in       self.trg_mask = self.transformer.generate_square_subsequent_mask(trg.shape[0]).to(device)<br>
but im not really sure.</p>
<p>What do you guys think?</p>
<p>Thanks</p>
<p>edit: Im thinking it’s because im creating just 1mask and i need BATCH_SIZE masks. Im trying with batch_size = 1</p>

username: zhangguanheng66
cooked: <p>Some example <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">here</a></p>

username: Johnas_Solomon
cooked: <p>Have you managed to implement the transformer for NMT? Please share you experience --Thanks</p>

username: vincentzlt
cooked: <p>This example is not for NMT.</p>
<p>I tried to follow the hyperparameters in  <code>attention is all you need</code> paper and can not reproduce the expected result.</p>
How to convert fully connected layer to fully convolutional layer

username: qiminchen
cooked: <p>Hi,</p>
<p>Suppose after feeding a 224x224x3 image into a backbone network, the output of last conv layer is 7x7x1280, and I have a well-trained two layers <a href="https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html" rel="noopener nofollow ugc">MLPClassifier</a> which has <code>1280x100</code> weights and <code>100x6</code> weights, how can I convert these two weights to do fully convolution on <code>7x7x1280</code> feature so I can get <code>7x7x100</code> and then <code>7x7x6</code>. I know I have to reshape these two weights into some <code>1x1</code> convolutional kernel but I don’t know how?</p>
<p>Should it be something like this?</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(1280, 100, kernel_size = 1, stride = 1, padding = 0)
self.conv1.weight.data = a 1x1x1280x100 array?
self.conv2 = nn.Conv2d(100, 6, kernel_size = 1, stride = 1, padding = 0)
self.conv2.weight.data = a 1x1x100x6 array?
</code></pre>

username: tom
cooked: <p>Linear layer weights are of shape <code>out_feat x in_feat</code>, conv weights are <code>out_chan x in_chan x kernel_height x kernel_width</code>, so all you need is to use channels as features and then add two dimensions to the weight:</p>
<pre><code class="lang-auto">with torch.no_grad():
  conv_layer.weight.copy_(lin_layer[:, :, None, None])
</code></pre>
<p>Indexing with <code>None</code> adds a dimension with size 1.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>Best regards</p>
<p>Thomas</p>
Computing mean of list of generator objects

username: Krishna_Garg
cooked: <pre><code class="lang-auto">// Initialize 2-D array, each entry is some neural network
phi = [[None] * n for _ in range(m)]
for i in range(m):
    for j in range(n):
        phi[i][j] = NeuralNetwork()

// Let k, i be arbitrary indices
p1 = torch.nn.utils.parameters_to_vector(phi[k][i - 1].parameters())
p2 = torch.nn.utils.parameters_to_vector(mean of phi[:][i-1])

</code></pre>
<p>I want to basically compute the mean squared error between the parameters <code>phi[k][i-1]</code> and average of the entire column <code>phi[:][i-1]</code> i.e. <code>((p1 - p2)**2).sum()</code> I tried in the following way:</p>
<pre><code class="lang-auto">tmp = [x.parameters() for x in self.phi[:][i - 1]]
mean_params = torch.mean(torch.stack(tmp), dim=0)
p2 = torch.nn.utils.parameters_to_vector(mean_params)
</code></pre>
<p>But this doesn’t work out because tmp is a list of generator objects. More specifically, I guess my problem is to compute the mean from that generator object.</p>

username: tom
cooked: <p>Three things I can think of:</p>
<ul>
<li>So there isn’t anything <em>per se</em> wrong with making a list of the parameters. In particular, it will loop over the parameters and create a new list, but you won’t be copying tensors a lot but only references to them. That might make it easier to just express what you want with for loops, even if it might not be the solution with maximal computational efficiency. For example, you could use <code>parameters_to_vector</code> inside the list comprehension (or <code>parameters_to_vector(...)</code>).</li>
<li>You can nest list comprehensions. For example you could use<pre><code class="lang-auto">tmp = [[something(p) for p in x[i - 1].parameters()] for x in self.phi]
</code></pre>
or so. I would recommend to exercise judgement whether or not this is very understandable and also add commentary what’s going on if you do.<br>
A way to get easier to understand things might be using named functions - similar to <code>parameters_to_vector</code> to achieve the things you want to do.<br>
You might need nested cat/stack if you want to build a “matrix of means”.</li>
<li>If you don’t need gradients, you likely want to detach the parameters before computing with them.</li>
</ul>
<p>As an aside, some care probably is to be taken with <code>parameters</code>. While I think they will usually come up in the same order for different instances of the same network, I’m not entirely sure that it is a hard guarantee (it might fail if people do really funny stuff in instantiation or use dict structures in older Python, probably not much risk with standard PyTorch modules these days). It certainly isn’t an obvious (i.e. explicit) guarantee to me. That said, it’s not clear to me that it matters at all for your use case (i.e. if you’re taking statistics that don’t depend on the order of the parameters, you should be fine anyways) and it could be that I’m overthinking this.</p>
<p>Best regards</p>
<p>Thomas</p>
Caffe2 for python = 2.7

username: Abhay_Garg
cooked: <p>I need to build caffe2 from source using Python =2.7 for using an old code.</p>
<p>It seems the current official Pytorch repo doesn’t support python2.7</p>
<p>Is there any way pytorch/caffe2 be built using python2.7 ??</p>
<p>Thank You</p>

username: BramVanroy
cooked: <p>You can go through the releases on Github and find ones that support 2.7. It seems that all original caffe2 releases support 2.7, sucha s this one: <a href="https://github.com/facebookarchive/caffe2/releases/tag/v0.8.1" rel="nofollow noopener">https://github.com/facebookarchive/caffe2/releases/tag/v0.8.1</a></p>

username: ptrblck
cooked: <p><a class="mention" href="/u/bramvanroy">@BramVanroy</a>’s suggestion might work, but I would generally advice against using a deprecated software stack, which won’t get any updates anymore. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Abhay_Garg
cooked: <p>Thank You <a class="mention" href="/u/bramvanroy">@BramVanroy</a> for the link. But the link seems to be missing modules in ~caffe2/third_party.</p>
<p>Thank You <a class="mention" href="/u/ptrblck">@ptrblck</a> for your suggestion. I am able to change the code to suit for Python 3.</p>
<p>However I want to install R3D - <a href="https://github.com/facebookresearch/video-nonlocal-net/blob/master/INSTALL.md" rel="noopener nofollow ugc">https://github.com/facebookresearch/video-nonlocal-net/blob/master/INSTALL.md</a></p>
<p>In this work they had modified the files in caffe2/video and then installed caffe2 from source.</p>
<p>While doing the same with the current official code of Pytorch/caffe2 it seems their modified modules are not compatible with the current Pytorch/caffe2.</p>
<p>What are your suggestions on how to install R3D??</p>
Multi-gpu via torch::nn::parallel::data_parallel

username: afshin67
cooked: <p>Hi,<br>
I tried to use multi-gpu capability in <code>C++</code>, but when I write <code>torch::nn::parallel::data_parallel</code>, I am getting <code>No member named 'parallel' in namespace 'torch::nn'</code> error. Is <code>torch::nn::parallel::data_parallel</code> functional now?<br>
Thanks,<br>
Afshin</p>

username: HwanUng_Lee
cooked: <p>you need<br>
<span class="hashtag">#include</span> “torch/nn/parallel/data_parallel.h”</p>
Torch.cat() massive bottleneck?!

username: Haxxardoux
cooked: <p>I was training something that looks like a transformer, and of course, this requires that you accumulate a tensor of previous tokens/other things in order to compute attention. This is facilitated by torch.cat() at each timestep,  which i have found slows down my code by a factor of 2, while also increasing memory usage by a factor of 2.</p>
<p>It makes sense that making new arrays is differentiable whereas inplace assignment to a preallocated array is not, but is this really the best we can do?! Since so many GPU hours are spent on training transformers, i find it very hard to believe that such a huge bottleneck like this is so widely adopted, and nobody is looking into alternatives.</p>

username: ptrblck
cooked: <p>If you are using <code>torch.cat</code> in a loop, you should instead append all tensors to a <code>list</code> and create the tensor out side of the loop. Also, you could try to preallocate the output tensor and fill it. If this operation would break your training, you should get an error and would have to fall back to the first approach.</p>

username: Haxxardoux
cooked: <p>This was my approach before incorporating attention - but the issue is that computing attention autoregressively requires that you use the output from all previous timesteps in the next timestep.</p>
<p>I could perhaps live with the .cat bottleneck, but i still cant help but wonder, what makes inplace assignment such a no-no? not just tensor[:, 1] = something, but tensor[:, 1] += something. It seems like as long as this operation is kept track of, this operation <em>could</em> be differentiable, and there would be no need to make duplicate tensors.</p>

username: ptrblck
cooked: <p>Inplace operations are not allowed where the original input is needed to calculate the gradients.<br>
E.g. in a vanilla CNN you could try to use the inplace <code>nn.ReLU</code> as long as PyTorch doesn’t raise an error.<br>
If it does, then the input needs to be unchanged for Autograd.</p>
Complex support

username: Anders_G
cooked: <p>What is the roadmap for complex number support looking like? I find that it works for some functions but not for some other. Where can I find out what the plans for this support look like?</p>

username: tom
cooked: <p>I’m not sure there is a roadmap document. I think the main source is the list of issues <a href="https://github.com/pytorch/pytorch/issues?q=is%3Aissue+is%3Aopen+label%3A%22module%3A+complex%22" rel="noopener nofollow ugc">tagged with module:complex</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Delta Encoding in Pytorch

username: gnadaf
cooked: <p>I am referring the paper “On Efficient Constructions of Checkpoints”</p>
<p>Design:<br>
Delta Encoding   =&gt; Quantization =&gt; Huffman Coding =&gt; Save Checkpoint</p>
<p>Do you have idea related Delta Encoding in Pytorch, how to use in it in Pytorch?</p>
How to disable operations recording for the computation graph and also save previous operations

username: Omer_B
cooked: <p>Hey, I need to apply a single operation on a tensor that I don’t want to be recorded for the computation graph. Also, I need that previous computation graph operations will be saved and won’t be deleted.</p>
<p>I tried to implement it with <code>torch.no_grad()</code> by this way:</p>
<pre><code class="lang-python">import torch

t = torch.rand(1).requires_grad_()

# some previous tensor operations
t = t.add(0)

# operations that I dont want to be recorded
with torch.no_grad():
    t = t.sub(0)
</code></pre>
<p>the result is that <code>t.grad_fn = None</code>, but I would want it to be <code>AddBackward0</code></p>
<p>is there any convenient way to do so? Thanks.</p>
Traning is slow

username: gnadaf
cooked: <p>I am running my code on the server (it has 7 GPUs), initially when I used to run my code. It takes around 40 sec to for total execution</p>
<p>python file.py</p>
<p>But, since from 2 days, same code is taking more time (similar to CPU), even I tried allocating different GPUs(cuda:0 and cuda:1). but still slow.</p>
<p>what would be the issue?</p>

username: ptrblck
cooked: <p>Did you change anything in the last 2 days, e.g. reinstalled drivers, libraries, etc.?<br>
If you haven’t changed anything and the code is suddenly slow, I would guess your setup is in a “bad” state. E.g. are you using a network drive and could the network have issues?<br>
To narrow down the root cause you could profile separate code snippets and check, which part of the training is slow.</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="97577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>rofile separate code snippets</p>
</blockquote>
</aside>
<p>No, I did not change code, I do not have a network issue, maybe a setup problem.<br>
Thanks for your detailed info</p>
Id function of tensor changes all the time

username: b224618
cooked: <p>I used function id() to check the reference of an element of a torch tensor and found it was changing all the time. But if I check the id() of an element in a list, it is stable. I just wonder why is this?</p>
<p>The code is below:</p>
<p>import torch<br>
…: a = torch.tensor([0,1,2,3])<br>
…: b = [0,1,2,3]<br>
…:<br>
id(a[0])<br>
Out[16]: 140023032068568<br>
id(a[0])<br>
Out[17]: 140023032095656<br>
id(a[0])<br>
Out[18]: 140023032069288<br>
id(b[0])<br>
Out[19]: 94672250583712<br>
id(b[0])<br>
Out[20]: 94672250583712<br>
id(b[0])<br>
Out[21]: 94672250583712<br>
id(a)<br>
Out[22]: 140024314842496<br>
id(a)<br>
Out[23]: 140024314842496<br>
id(a)<br>
Out[24]: 140024314842496</p>

username: googlebot
cooked: <p>You’re querying python objects, and these are basically created for everything. List indexing is a built-in operation that avoids object creation or copying, so it is a bit special. And tensors are wrappers, sliced with code in <code>__getitem__</code>.</p>
<p>The stable thing is the memory address of tensor’s storage (that’s technically a number exported from c++ part): a.storage().data_ptr()</p>

username: b224618
cooked: <p>Thanks a lot for your answer! The function storage().data_ptr() helps a lot for me. But I am still confused about the function id(). If it does not returns the memory address, does it return a number for referencing the corresponding object? Then why is it unstable for elements of a tensor a? Is the tensor a constantly changing its reference to its elements?<br>
Thanks!!</p>

username: googlebot
cooked: <p>You’re using id() on wrapper objects, that are created on the fly. E.g. when you do a[100000] you get a new object that shares storage with “a”, but also records storage offset. Repeated call a[100000] creates a new wrapper, as it is impossible to cache these.</p>

username: b224618
cooked: <p>I understant, Thank you so much!</p>
How to realize an arbitrary dimension batchnorm layer?

username: hnczwj2008
cooked: <p>Hi, I want an arbitrary batchnorm layer for my CNN-based progressor. could some one tell me how to realize it ?<br>
Here is an example of  an anbitrary conv layer: <a href="https://github.com/pvjosue/pytorch_convNd" rel="noopener nofollow ugc">https://github.com/pvjosue/pytorch_convNd</a>. I want to apply the batchnorm6d after conv6d.<br>
thank you.</p>

username: ptrblck
cooked: <p>Cross post from <a href="https://github.com/ptrblck/pytorch_misc/issues/7#issuecomment-699613184">here</a> with a potential approach.</p>
How to use pytorch docker?

username: alanzhai219
cooked: <p>There are some images in <a href="https://hub.docker.com/u/pytorch" rel="nofollow noopener">https://hub.docker.com/u/pytorch</a>. Is PyTorch installed in such image? If so, how to use it. If not, what is the purpose of such images.<br>
[<strong>It is <a href="https://hub.docker.com/u/pytorch" rel="nofollow noopener">https://hub.docker.com/u/pytorch</a> official docker instead of official nvidia-docker.</strong>]</p>
<p>BTW: what is pytorch/dockerfile designed for?<br>
That confuses me a lot.</p>

username: ptrblck
cooked: <p>Yes, PyTorch is installed in these containers.<br>
No, they are not maintained by NVIDIA.<br>
You can find more information on docker containers <a href="https://www.docker.com/resources/what-container">here</a>.</p>
<p>The <code>Dockerfile</code> is used to build the container.</p>

username: alanzhai219
cooked: <p>Thanks for your reply. It seems some are installed pytorch while the other are not.<br>
For example, <a href="https://hub.docker.com/r/pytorch/manylinux-cuda102" rel="noopener nofollow ugc">https://hub.docker.com/r/pytorch/manylinux-cuda102</a> is just installed cuda toolkits without pytorch.</p>

username: ptrblck
cooked: <p>Sorry for not being clear. Some of these containers are used as the base e.g. to build the binaries etc.<br>
The framework is installed in <code>pytorch/pytorch</code> using the latest stable version (<code>1.6.0</code>).</p>

username: alanzhai219
cooked: <p>Much thanks, I understand it.<br>
Summarize,<br>
1.something like <a href="https://hub.docker.com/r/pytorch/manylinux-cuda102" rel="noopener nofollow ugc">https://hub.docker.com/r/pytorch/manylinux-cuda102</a> is base docker and some kits in it.<br>
2.something like <a href="https://hub.docker.com/r/pytorch/pytorch" rel="noopener nofollow ugc">https://hub.docker.com/r/pytorch/pytorch</a> is docker with pytorch installed without source code.<br>
3.<a href="https://ngc.nvidia.com/catalog/containers/nvidia:pytorch" rel="noopener nofollow ugc">https://ngc.nvidia.com/catalog/containers/nvidia:pytorch</a> is maintained by nv and pytorch/cuda-kits installed. And pytorch code in it.</p>
‘tensor’ is not a class, namespace, or enumeration

username: Flock1
cooked: <p>Hi,</p>
<p>I am very new to C++ and I was following <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="noopener nofollow ugc">this</a> tutorial to run the first code.</p>
<p>When I ran <code>cmake --build . --config Release</code>, I got the following error:</p>
<pre><code class="lang-auto">Scanning dependencies of target dcgan
[ 50%] Building CXX object CMakeFiles/dcgan.dir/dcgan.cpp.o
/home/sarvagya/Desktop/cpp/dcgan/dcgan.cpp: In function ‘int main()’:
/home/sarvagya/Desktop/cpp/dcgan/dcgan.cpp:5:25: error: ‘tensor’ is not a class, namespace, or enumeration
    5 |  torch::Tensor tensor = tensor::eye(3);
      |                         ^~~~~~
make[2]: *** [CMakeFiles/dcgan.dir/build.make:63: CMakeFiles/dcgan.dir/dcgan.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/dcgan.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
</code></pre>
<p>What am I doing wrong?</p>

username: ptrblck
cooked: <p>The example is defined as</p>
<pre><code class="lang-auto">int main() {
  torch::Tensor tensor = torch::eye(3);
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}
</code></pre>
<p>so it seems you’ve changed the <code>torch::eye</code> call to <code>tensor::eye</code>.<br>
Could you revert this change and retry the build?</p>

username: Flock1
cooked: <p>No. This was so clumsy of me. I am sorry I created this topic. Should I delete it? It’s not going to help anyone.</p>
How to convert yolov3 model to libtorch model

username: yaoxingtian
cooked: <p>It’s  really a problem for me now. when I try <a href="https://github.com/eriklindernoren/PyTorch-YOLOv3" rel="noopener nofollow ugc">https://github.com/eriklindernoren/PyTorch-YOLOv3</a>. this project and convert yolov3.weights to yolov3.pt by torch.jit.trace(model.example), errors issued:<br>
magic_number = pickle_module.load(f, **pickle_load_args)<br>
_pickle.UnpicklingError: invalid load key, ‘\x00’.</p>
<p>my code is as below:</p>
<p>model = Darknet(’/home/yao/Documents/PyTorch-YOLOv3-master/config/yolov3.cfg’, 416)<br>
model.load_darknet_weights(torch.load(’/home/yao/Documents/PyTorch-YOLOv3-master/weights/yolov3.weights’))<br>
model.eval()</p>
<p>example_img = torch.rand(1,3,416,416)<br>
with torch.jit.optimized_execution(True):<br>
traced_script_module = torch.jit.trace(model, example_img)# save the converted model<br>
traced_script_module.save(“yolov3.pt”)<br>
print(‘ok’)</p>
<p>if any body can help me , really appreciate it!</p>
Efficiently slicing tensor like a convolution?

username: yonkshi
cooked: <p>I have a tensor T with shape [B, C, H, W] and I would like to use a “sliding window” to slice this tensor into S sub-tensors with shape [h, w]. So I should have an output tensor of  [S, B, C, h, w]. Similar to a 2d convolution without actually multiplying the weights.</p>
<p>Here is an intuitive example (ignoring the batch and channels)</p>
<pre><code class="lang-auto">T = [[ a, a, b, b],
     [ a, a, b, b],
     [ c, c, d, d],
     [ c, c, d, d]] 

kernel = (2,2)
stride = (2,2)

T_s = [[[a,a],[a,a]],[[b,b],[b,b]], [[c,c],[c,c]], [[d,d],[d,d]]
</code></pre>
<p>My solution right now is using a nested loop and slice each block using indexer <code>T[i:i+h, j:j+w]</code> , but this has proven to be quite inefficient. I’ve been digging through the docs and can’t seem to find an efficient way</p>

username: ptrblck
cooked: <p><code>tensor.unfold</code> should yield the desired output:</p>
<pre><code class="lang-python">B, C, H, W = 2, 3, 4, 4
x = torch.arange(B*C*H*W).view(B, C, H, W)

kernel_h, kernel_w = 2, 2
stride = 2

patches = x.unfold(2, kernel_h, stride).unfold(3, kernel_w, stride)
print(patches )
</code></pre>

username: urukhai
cooked: <p>Just to save others from making my mistake: I was confused at first because there is a function, a method, and a class all called “unfold”, and Google finds <span class="hashtag">#1</span> first.  However,  <a class="mention" href="/u/ptrblck">@ptrblck</a> used <span class="hashtag">#3</span> in the answer above,  I think (<span class="hashtag">#1</span> has different arguments).</p>
<ol>
<li>
<a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=unfold#torch.nn.functional.unfold" rel="noopener nofollow ugc">torch.nn.functional.unfold</a> (Python function, in torch.nn.functional)</li>
<li>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html?highlight=unfold#torch.nn.Unfold" rel="noopener nofollow ugc">torch.nn.Unfold</a> (Python class, in Unfold)</li>
<li>
<a href="https://pytorch.org/docs/stable/tensors.html?highlight=unfold#torch.Tensor.unfold" rel="noopener nofollow ugc">torch.Tensor.unfold</a> (Python method, in torch.Tensor)</li>
</ol>
How to load an older model in PyTorch 1.6?

username: blackberry
cooked: <p>I have some models trained in PyTorch 1.5 and saved by <code>torch.save(filename, model)</code>, instead of <code>torch.save(filename, model.state_dict())</code>.</p>
<p>Now, I am not able to load them in PyTorch 1.6.<br>
I get <code>torch.nn.modules.module.ModuleAttributeError: 'BatchNorm2d' object has no attribute '_non_persistent_buffers_set'</code></p>
<p>What is the correct way to load such models?</p>
<p>Following <strong><a href="https://github.com/ultralytics/yolov5/issues/58" rel="nofollow noopener">https://github.com/ultralytics/yolov5/issues/58</a></strong>, I tried this on the model and seemed to load without problem. But is this the right way?</p>
<pre><code class="lang-auto">model = ModelDenseNet()
model1 = torch.load(model_file)
for k, m in model1.named_modules():
   m._non_persistent_buffers_set = set()  # pytorch 1.6.0 compatability
model.load_state_dict(model1.state_dict())

</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>I would consider this a workaround and would try to make sure your model is still working as intended.<br>
E.g. you could create some tests using predefined data and compare the outputs, gradients, losses etc.<br>
The recommended way would be to store the <code>state_dict</code> instead of the model directly.</p>

username: blackberry
cooked: <p>The workaround seems to work (gives the same accuracy on the test set). I was wondering if there is a better way.</p>
<p>Would be great if PyTorch provided and option to <code>torch.save/torch.load</code> to save load either the whole model or the state_dict.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="blackberry" data-post="3" data-topic="97423">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ecd19e/48.png" class="avatar"> blackberry:</div>
<blockquote>
<p>Would be great if PyTorch provided and option to <code>torch.save/torch.load</code> to save load either the whole model or the state_dict.</p>
</blockquote>
</aside>
<p>The recommended way of storing the <code>state_dict</code> should be working fine or are you seeing any issues with it? As explained in the <a href="https://pytorch.org/docs/stable/notes/serialization.html">Serialization semantics</a>, storing and loading the model directly could be breaking in various ways due to the way Python pickles the model.</p>
UserWarning: Mixed memory format inputs

username: NearIt
cooked: <p>I have implemented Unet with custom loss function : Dice loss.</p>
<p>while training, I receive the following message  :<br>
<code>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:32: UserWarning: Mixed memory format inputs detected while calling the operator. The operator will output contiguous tensor even if some of the inputs are in channels_last format. (Triggered internally at  /pytorch/aten/src/ATen/native/TensorIterator.cpp:918.)</code></p>
<p>this warning appear only when calling the Dice loss function. this does not happens when I am calling pytorch function torch.nn.CrossEntropyLoss().</p>
<p>this is my Dice implementation (copied from some webpage):</p>
<pre><code class="lang-auto">def calculate_dice_loss(logits,true, eps = 1e-7):

    """Computes the Sørensen–Dice loss.
    Note that PyTorch optimizers minimize a loss. In this
    case, we would like to maximize the dice loss so we
    return the negated dice loss.
    Args:
        true: a tensor of shape [Batch size  x 512 x 512].
        logits: a tensor of shape [Batch size x numLabels x 512 x 512]. Corresponds to
            the raw output or logits of the model.
        eps: added to the denominator for numerical stability.
    Returns:
        dice_loss: the Sørensen–Dice loss.
    """
    num_classes = logits.shape[1]
    true = true.unsqueeze(1) # now true: a tensor of shape [Batch size x 1 x 512 x 512].
    true_1_hot = torch.eye(num_classes)[true.squeeze(1)]
    true_1_hot = true_1_hot.permute(0, 3, 1, 2).float()
    probas = F.softmax(logits, dim = 1)
    true_1_hot = true_1_hot.type(logits.type())
    dims = (0,) + tuple(range(2, true.ndimension()))
    intersection = torch.sum(probas * true_1_hot, dims)
    cardinality = torch.sum(probas + true_1_hot, dims)
    dice_loss = (2. * intersection / (cardinality + eps)).mean()
    return (1 - dice_loss)
</code></pre>
<p>and this is how I call the loss function while training:</p>
<pre><code class="lang-auto"># loss values
    CrossEntropy_Loss = CrossEntropy_criterion(logits, true_labels)
    dice_loss =  calculate_dice_loss(logits,true_labels)

    loss =  dice_loss 


    # Back propagation

    unet.zero_grad()
    loss.backward()
    optimizer.step()
    loss_list.append(loss.item())
</code></pre>

username: ptrblck
cooked: <p>Are you using the <code>channels_last</code> memory format in your model?<br>
If not, could you print the shapes of <code>logits</code> and <code>true_labels</code> before passing them into the loss function?</p>
Type conversion issue while porting from older pytorch to new version

username: mahmoodn
cooked: <p>I have used the dcgan <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">tutorial</a> with older versions of pytorch and python2.7. However, Now I get this error with the latest revision and python3</p>
<pre><code class="lang-auto">Starting Training Loop...
Traceback (most recent call last):
  File "main.py", line 245, in &lt;module&gt;
    errD_real = criterion(output, label)
  File "/home/mahmood/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/mahmood/.local/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 530, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "/home/mahmood/.local/lib/python3.7/site-packages/torch/nn/functional.py", line 2527, in binary_cross_entropy
    input, target, weight, reduction_enum)
RuntimeError: Found dtype Long but expected Float
</code></pre>
<p>Part of the code is</p>
<pre><code class="lang-auto">        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
</code></pre>
<p>So, I would like to know how can I fix that to be compatible with the latest pytorch?</p>

username: ptrblck
cooked: <p>I guess your <code>label</code> tensor might be a <code>LongTensor</code> while a <code>FloatTensor</code> is expected.<br>
If that’s the case, use <code>label = torch.full(..., dtype=torch.float32)</code> and rerun the code.</p>
Removing 2 layers from inception pretrained model is giving errors

username: Harshit_Pandey
cooked: <p>On running the code snippet below I get the following error: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [192, 768, 1, 1], but got 2-dimensional input of size [2, 1000] instead</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
from torchsummary import summary
inception = models.inception_v3(pretrained=True)
remove = list(inception.children())[:-2]
model= torch.nn.Sequential(*remove)
summary(model,(3,299,299))
</code></pre>
<p>I then tried to check if inception’s summary was working, and it was.</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
from torchsummary import summary
inception = models.inception_v3(pretrained=True)
summary(inception ,(3,299,299))
</code></pre>
<p>In the end , instead of [:-2], I allowed all layers to be present in the variable “remove”. This again gave me the same error : RuntimeError: Expected 4-dimensional input for 4-dimensional weight [192, 768, 1, 1], but got 2-dimensional input of size [2, 1000] instead</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
from torchsummary import summary
inception = models.inception_v3(pretrained=True)
remove = list(inception.children())[:]
model= torch.nn.Sequential(*remove)
summary(model,(3,299,299))
</code></pre>
<p>Is this a bug , or am I missing something here ? I tried this for Resnet and it worked perfectly</p>

username: ptrblck
cooked: <p>Wrapping all modules in an <code>nn.Sequential</code> container might work for a simple model definition.<br>
In your case the inception model is failing, since <code>inception.children()</code> will return the child modules in the order they were initialized. <code>model[15]</code> would thus contain the <code>InceptionAux</code> module (which is used in <a href="https://github.com/pytorch/vision/blob/662373f6057bb0d39eaf6e5fde3083639ed93af3/torchvision/models/inception.py#L168-L171">this side branch</a> of the model) and will thus apply a linear layer to your activations.<br>
The next layer will fail because of the shape mismatch.</p>
<p>If you want to change the model execution I would recommend to write a custom model by deriving from the inception model and by changing the <code>forward</code> definition.</p>