akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Inconsistency between the pretrained models and labels

seyeeet: <p>is there any code/tutorials to show how to use the pretrained pytorch models and test the evaluation performance for image classification on Imagenet and cifar10 100? (just testing not training, since I dont have the hardware for training)<br>
For imagenet I am facing an inconsistency issue and I cannot find the details for the labels that are used to train the pytorch models.<br>
In other words:<br>
I notice that for pretrain models that are provided the labels are not consistent.<br>
For example, vgg16 class 1 is different from Resnet50 class 1.<br>
Can you let us know where we can find the corresponding labels for each model?<br>
For vgg i notice that the one that looks like this:</p>
<pre><code class="lang-auto">  "0": [
    "n01440764",
    "tench"
  ],
  "1": [
    "n01443537",
    "goldfish"
  ],
  "2": [
    "n01484850",
    "great_white_shark"
  ],
  "3": [
    "n01491361",
    "tiger_shark"
  ],
  "4": [
    "n01494475",
    "hammerhead"
  ],
  "5": [
    "n01496331",
    "electric_ray"
  ],
  "6": [
    "n01498041",
    "stingray"
  ],
  "7": [
    "n01514668",
    "cock"
  ],
  "8": [
    "n01514859",
    "hen"
</code></pre>
<p>works, but this one is not the one that we should use for resnets, please let us know what we should do.<br>
Thanks</p>
Processing sequential data in PyTorch Geometric

Alpay_Tekin: <p>I have code which is about GRU and written in TensorFlow. I want to translate this code to PyTorch Geometric so that I can deal with the graph-level structure. Is there any model in PyTorch Geometrics for time series data or should I use PyTorch Geometric Temporal instead.</p>

benedekrozemberczki: <p>Hi there Alpay,</p>
<p>I am the lead on PyTorch Geometric Temporal. What is your exact use case?</p>
<p>Benedek</p>
Different types in if statement branches produce error in TorchScript

fermat97: <p>I try to <code>TorchScript</code> my model and it seems there is a problem while different branches possess different types. For instance in the following scenario once <code>x</code> is a <code>torch.Tensor</code>:</p>
<pre><code class="lang-auto">if a &gt; b:
    y.append(x)
else:
    y.append(None)
</code></pre>
<p>I got the following error:</p>
<blockquote>
<p>RuntimeError:<br>
aten::append.t(t<a></a> self, t(c → *) el) → (t<a></a>):<br>
Expected a value of type ‘t’ for argument ‘el’ but instead found type ‘None’.</p>
</blockquote>
<p>Is it the case? And does anybody know a workaround for such cases?</p>

eellison: <p>Hi <a class="mention" href="/u/fermat97">@fermat97</a>, You should try annotation your list as being <code>Optional[torch.Tensor]</code> when you initialize it:</p>
<p><code>x : List[Optional[torch.Tensor]] = []</code></p>

fermat97: <p>Thanks <a class="mention" href="/u/eellison">@eellison</a>, it works for <code>None</code>. But since <code>TorchScript</code> does not support <code>Union</code>, if we consider the alternatives as <code>int</code> or any other <code>typing</code>, is there any solution for that?</p>

eellison: <p>Union is a WIP but slated for 1.9 release. A not particularly ergonomic solution is to use <code>Any</code> until Union is supported.</p>
How to use CosineSimilarity function in right way?

chatterboy: <p>I’m confused the document of <a href="https://pytorch.org/docs/1.6.0/generated/torch.nn.CosineSimilarity.html?highlight=cosine#torch.nn.CosineSimilarity" rel="noopener nofollow ugc">CosineSimilarity</a>. A similarity can be computed between two objects. If we have a hundred of vectors then the total number of similarities is one hundred by one hundred. But, the size of the output in the document is one hundred. I don’t understand what it means…</p>
<p>Btw, in my case I have two matrices, the shape of first matrix is (N, C) and the shape of second matrix is (M, C) in row vectors. In this case, how can I compute a similarity matrix between two matrices of which size is (N, M) using CosineSimilarity function?</p>

kirk86: <p>To compute cosine similarity between two vectors <code>x, y</code> you need to have both vectors same size as you can see from the example in the same page you linked in your comment:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; input1 = torch.randn(100, 128)
&gt;&gt;&gt; input2 = torch.randn(100, 128)
&gt;&gt;&gt; cos = nn.CosineSimilarity(dim)
</code></pre>
<p>in your case you have <code>N =/= M</code> so that would be challenging if you use the built in cosine function in pytorch.</p>
<p>You could instead calculate the cosine as follows:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/6/5679ec81bccbf46230952ac3ad3f4f680465af4a.png" alt="image" data-base62-sha1="cl0hyhCyN02WiNBCJtMHkgVgA0G" width="234" height="102"></p>
What is gpu_0/data?

Marat: <p>I build GLOW and now want to test in GPU mode. Recently I work with image-classifier tool but I failed to understand what is</p>
<pre><code class="lang-auto">-model-input-name=gpu_0/data
</code></pre>
<p>???<br>
I pass ONNX model file and PNG picture what role model-input-name plays?</p>
<p>Fill line is</p>
<pre><code class="lang-auto">./bin/image-classifier tests/images/imagenet/*.png -image-mode=0to1 -m=resnet50 -model-input-name=gpu_0/data?
</code></pre>

Marat: <p>I found fix using lldb debugger if I pass</p>
<pre><code class="lang-auto">-model-input-name=gpu_0/data_0
</code></pre>
<p>instead it works in my case.</p>

Marat: <p>How to specify appropriate -model-input-name when using -input-image-list-file option of image-classifier tool?</p>
<p>If use gpu_0/data_0 I will get</p>
<pre><code class="lang-auto">Reshape
name : OC2_DUMMY_0
Input : float&lt;9 x 2048 x 1 x 1&gt;
Dims : [1, 2048]
users : 1
Result : float&lt;1 x 2048&gt;

Reshape into a different size
For comparison `LHS Equal RHS` with:
LHS: 2048
RHS: 18432
Encountered an error, exiting.
location: ../lib/Importer/ONNXModelLoader.cpp:1265 message: Function verification failed.
</code></pre>
<p>Which is not good</p>

jfix: <p>Please see <a href="https://discuss.pytorch.org/t/quantization-example-resnet50/39568/13">this comment here and the couple after it</a> about the model input name. It has nothing to do with the input image list – it’s the name of the input in the protobuf representing the model itself.</p>

bucky: <p>Hello, How do I run glow on a GPU?</p>
<p>./bin.mnsit -backend=OpenCL doesnt seems to work. I have a nvidia gpu in my computer.</p>

jfix: <p>When you say it doesn’t work, does that mean it executes but doesn’t appear to execute on the GPU, or it doesn’t execute at all?</p>
<p>If it’s the former, OpenCL allows you to target many different devices on your machine. You may need to specify which device you want to target via <code>--device=[integer&gt;=0]</code>, e.g. <code>--device=1</code>.</p>
How to manually define batches for each epoch?

kyy: <p>Hi, I am trying to manually define training batches for each epoch such that each batch contains only samples that satisfying specific condition (e.g. first batch are all from class 1 with image size smaller than 128*128, second batch are all from class 2 with image size greater than 256*256 and so on, and batches may have different number of samples).  That said, using <code>for batch in dataloader</code> would probably not give me correct grouping of batches.  I am wondering if anyone know any good way to manually “assign batches” before each training epoch starts?</p>
<p>Thank you!</p>

Manuel_Alejandro_Dia: <p>What you could do is to create the batches manually. Calling your dataset and then evaluating a conditional to see if the input should be added to the batch.</p>

kyy: <p>Thanks for your advice!  I implemented it with a batch sampler which defines the batches based on some conditionals (ref: <a href="https://www.scottcondron.com/jupyter/visualisation/audio/2020/12/02/dataloaders-samplers-collate.html#BatchSampler" class="inline-onebox" rel="noopener nofollow ugc">But what are PyTorch DataLoaders really? | Scott Condron’s Blog</a>) which works well.</p>
How to do weight normalization in every forward pass

jfyeh: <p>Hi all,<br>
I am trying to implement cosFace loss (Large Margin Cosine Loss), in this loss I need to normalize the weight of last Linear layer in the module every forward pass.</p>
<p>I have checked several related posts, like <a href="https://discuss.pytorch.org/t/how-to-do-weight-normalization-in-last-classification-layer/35193/3">here</a>.<br>
But in my case, the normalization operation should have gradients and backpropagation.</p>
<p>So I might use some code like:</p>
<pre><code class="lang-auto">fc.weight = nn.Parameter(F.normalize(fc.weight, p=2, dim=1))
</code></pre>
<p>I just wanna know those cast operation(cast Tensor to Parameter) would affect the gradient or not?<br>
Or there is another way to deal with it?</p>
<p>Thanks!</p>

grsilva: <p>Hey!<br>
Did you manage to solve this problem?</p>
<p>I am in a similar situation.</p>
Different weight decay for different layes

twtygqyy: <p>Hi, is it possible to set different values for different layers or disable the weight decay for some specified layers, thanks</p>

apaszke: <p>It is. Check out the <a href="http://pytorch.org/docs/optim.html#per-parameter-options">per-parameter options</a> section.</p>

twtygqyy: <p>Thanks for the reply, it works.</p>

matthias.l: <p><a class="mention" href="/u/apaszke">@apaszke</a> Is it also possible to make this dynamic? In my case some (mostly) CNN weights are imploding out and the bias parameters are exploding.</p>
Low training accuracy using pre-trained torchvision model

AndreiXYZ: <p>Hello,<br>
I am trying to evaluate a pre-trained mobilenetv2 model from torchvision on the ImageNet training dataset using the official example (<a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>).<br>
To do so, I modify lines 235-237 to perform validation on the train loader instead of the val loader:<br>
<code>    if args.evaluate:</code> <code>        validate(train_loader, model, criterion, args)</code> <code>        return</code><br>
Everything else is left untouched. The command I use to run is:<br>
<code>python imagenet_train_example.py -a mobilenet_v2 -j 16 -b 1024 -e --pretrained /data/ImageNet</code><br>
However, the results are much lower than expected:</p>
<blockquote>
<p>Acc@1 2.926 Acc@5 15.079 Loss 11.795791</p>
</blockquote>
<p>I was wondering if anyone knows why that might be? Am I doing something wrong?<br>
Cheers!</p>
Skip dimensions using index_select/index_copy

steven_morad: <p>I am trying to copy a small tensor into another tensor at a list of indices.</p>
<p>I have three tensors:</p>
<pre><code class="lang-auto">a # Shape (batch, k, feat)
b # Shape (batch, feat)
c # Shape (batch,1)
</code></pre>
<p>I want to plug <code>b</code> into <code>a</code> using <code>c</code>. Right now I am using a for loop</p>
<pre><code class="lang-auto">for batch in a.shape[0]:
  a[batch, c[batch]] = b[batch]
</code></pre>
<p>I have tried to speed this up using <code>index_copy</code>, <code>index_select</code>, and <code>gather</code>, but I can’t figure out how to do this using vectorized torch functions. I believe this is similar to this unanswered question: <a href="https://discuss.pytorch.org/t/index-copy-on-several-dimensions/19646" class="inline-onebox">Index_copy_ on several dimensions</a></p>

steven_morad: <p>There is probably a better way, but this seems to leverage pytorch vectorization</p>
<pre><code class="lang-auto">idxs_0 = torch.arange(a.shape[0])
idxs_1 = c.squeeze()
a[idxs_0, idxs_1] = b[idx_0]
</code></pre>
Numpy to tensor changes values dramatically?

Mortelli: <p>Hi everyone,</p>
<p>I’m struggling with the following issue, and can’t find a possible explanation (I’m quite the pytorch amateur, so please forgive me for not finding possible obvious solutions).<br>
I’m feeding MR images to a 3D Unet, and reshape the already normalized numpy array <code>image</code> like this:</p>
<blockquote>
<p>imagefinal=(image.reshape(1,1, image.shape[0], image.shape[1], image.shape[2]))</p>
</blockquote>
<p>after which I convert it to a tensor using</p>
<blockquote>
<p>tensor_x = torch.from_numpy(imagefinal).float()</p>
</blockquote>
<p>However, using</p>
<blockquote>
<p>print(imagefinal.max())</p>
</blockquote>
<blockquote>
<p>print(torch.max(tensor_x))</p>
</blockquote>
<p>returns</p>
<blockquote>
<p>1.0<br>
tensor(2652589.7500, device=‘cuda:0’)<br>
1.0<br>
tensor(4033200., device=‘cuda:0’)<br>
1.0<br>
tensor(3447660.5000, device=‘cuda:0’)<br>
1.0<br>
tensor(3224289., device=‘cuda:0’)<br>
1.0</p>
</blockquote>
<p>Do you have any idea why the maximum value of the array would change to very large values after converting it to a tensor? I’ve tried  forcing the tensor to be of different datatypes with no success. Is there something obvious I’m missing here?</p>
<p>Thanks in advance for your help!</p>

Alexey_Demyanchuk: <p>Consider this toy example:</p>
<pre><code class="lang-auto">np_array = np.random.randn(2,3,4)
print(np_array.shape, np_array.dtype, np_array.max())
&gt;&gt;((2, 3, 4), dtype('float64'), 2.2915268560525193)
tensor = torch.from_numpy(np_array)
print(tensor.shape, tensor.dtype, tensor.max())
&gt;&gt;torch.Size([2, 3, 4]) torch.float64 tensor(2.3412, dtype=torch.float64)
</code></pre>
<p>Everything is as expected. Notice, that default numpy float type is <code>float64</code>, torch is <code>float32</code>. So, you may convert float64 to float32 and have possible overflow (which I am a bit doubt). I would also suggest to carefully review if you may change <code>imagefinal</code> somewhere in the code before converting to tensor.</p>

Mortelli: <p>Hi Alexey,</p>
<p>Thank you very much for your reply. After some additional digging, I found the problem. I’m masking my MR image arrays with the <code>np.ma.masked_array</code> function, returning a <code>MaskedArray</code> datatype. I wasn’t able to find an explanation for this online, but <code>torch.from_numpy</code> doesn’t seem able to directly copy values from MaskedArray types. After first converting the MaskedArray to a normal numpy array using <code>x=np.asarray(MaskedArray)</code>, the <code>torch.from_numpy</code> function works as intended.</p>
<p>Best,</p>
<p>Laurens</p>

Alexey_Demyanchuk: <p>Good to hear, you figure it out.</p>
<p>Cheers,<br>
Alexey</p>
Bug with Wrapping DataLoader example

ams09: <p>I’m working through the tutorials here: <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">What is torch.nn really? — PyTorch Tutorials 1.7.1 documentation</a>, specifically on the Wrapping DataLoader example. When I run the code on my end, I get the following:</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in ()<br>
40 opt = optim.SGD(model.parameters(), lr=lr, momentum=0.9)<br>
41<br>
—&gt; 42 fit(epochs, model, loss_func, opt, train_dl, valid_dl)</p>
<p>TypeError: ‘tuple’ object is not callable</p>
<p>I even recopied the code from the page directly and continued to get this error. Could someone clarify this?</p>

ptrblck: <p>I just reran the tutorial on Colab without any errors.<br>
Are you redefining the <code>fit</code> function somewhere?</p>

ams09: <p>Yes, I think I did redefine the fit function, which caused the issue. Thanks!</p>
How to optimize multi model's parameter in one optimizer

Kyle: <p>I know we can use “optimizer = optim.Adam(model1.parameters())” to optimize a model, but how can I optimize multi model in one optimizer?</p>

smth: <pre><code class="lang-auto">optim.Adam(list(model1.parameters()) + list(model2.parameters())
</code></pre>

Neo_li: <p>Could I put model1, model2 in a nn.ModulList, and give the parameters() generator to optimizer？</p>

smth: <p>Yes you can do that.</p>

Andybert: <p>When I construct a optimizer like that, the torch.nn.utils.clip_grad_norm_ function is not available anymore. Is there any way to make both of them available?</p>

ptrblck: <p>Seems to be <a href="https://discuss.pytorch.org/t/optimizer-on-multi-neural-networks/20572/4?u=ptrblck">this issue</a>. Let’s keep the discussion in the other thread to keep this one clean. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

balbok0: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/smth">@smth</a>, just wanted to make sure that method with concatenating lists of paramaters from two models would work even if there is some highly non-linear transformation between them?<br>
The case I am thinking about is that model 1 will create some outputs y_1. Then a non-linear function f is applied applied, such that, the inputs to model 2: x_2 are x_2 = f(y_1).<br>
Or is there a way of forcing a optimizer to be multiplied by a gradient of that function in between?</p>
<p>Thanks for your time!</p>

ptrblck: <p>The optimizer will apply its updating scheme to the passed parameters, so it should work if you pass the parameters of both models to it.</p>

RobertYu: <p>You can use itertools.chain</p>
<pre><code class="lang-auto">from itertools import chain
chain(net1.parameters(), net2.parameters())
</code></pre>

rony2: <p>Is it possible to use a different learning rate for each of the models with the same optimizer?</p>

Alexey_Demyanchuk: <p>You can use different learning rate and other hyperparameters as well.<br>
Take a look on official example</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p><a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="noopener nofollow ugc">Reference</a> to official docs.</p>

Alexey_Demyanchuk: <p>I believe, all pytorch optimizers provide the same interface, so should work with Adam as well.</p>
How should I use template to write a class while using TORCH_MODULE

Allent_Dan: <p>Original declaration goes as follows:</p>
<pre><code class="lang-cpp">class UNetImpl : public torch::nn::Module
{
public:
    UNetImpl(int num_classes, std::string encoder_name = "resnet18", std::string pretrained_path = "", int encoder_depth = 5,
             std::vector&lt;int&gt; decoder_channels={256, 128, 64, 32, 16}, bool use_attention = false);
    torch::Tensor forward(torch::Tensor x);
private:
    ResNet encoder{nullptr};
    UNetDecoder decoder{nullptr};
    SegmentationHead segmentation_head{nullptr};
    int num_classes = 1;
    std::vector&lt;int&gt; BasicChannels = {3, 64, 64, 128, 256, 512};
    std::vector&lt;int&gt; BottleChannels = {3, 64, 256, 512, 1024, 2048};
    std::map&lt;std::string, std::vector&lt;int&gt;&gt; name2layers = getParams();
};TORCH_MODULE(UNet);
</code></pre>
<p>But when I tried to replace ResNet with a template class “Backbone”, there was an error “XML comment  contains invalid XML”.</p>

ptrblck: <p>Could you post the complete error message here, please?</p>

Allent_Dan: <p>|Error|C3203|‘UNetImpl’: unspecialized class template can’t be used as a template argument for template parameter ‘Contained’, expected a real type|VSLibtorch|d:\allentfiles\code\personal\c++\vslibtorch\vslibtorch\unet.h|35||</p>

ptrblck: <p>Thanks for the update. I don’t know why this C++ error is raised and don’t know what class implementations are supported.<br>
Based on the <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html">tutorial</a> <code>struct</code>s are used, but I don’t know if this is causing an error. Is this a custom code snippet or are you using any tutorial as the base?</p>

Allent_Dan: <p>Thanks for your kindly reply.<br>
Well, it is indeed a code snippet from my project <a href="https://github.com/AllentDan/SegmentationCpp" rel="noopener nofollow ugc">here</a>. I want to replace ResNet with different backbones like SeNet. To improve code quality, I don’t want to add a new class member or “if else”. So I tried to use template. The class implementations can be find <a href="https://github.com/AllentDan/SegmentationCpp/blob/main/src/UNet.cpp" rel="noopener nofollow ugc">here</a>.</p>
Torchvision.transforms.functional.normalize throws Error: img should be PIL Image. Got <class 'torch.Tensor'>

jenn: <p>Hi there,</p>
<p>I am trying to transform my input data. I have images of shape [6, height, width] and want to reshape them to [6, 112, 112]. Since I have 6 channels, I can’t use PILImage. Therefore I coose to use torchvision.transforms.functional.resize() because it is supposed to take torch.Tensors as input aswell.<br>
I tried:</p>
<pre><code class="lang-auto">            img = torch.tensor(img)
            img = transforms.functional.resize(img, size =(112,112))
            img = transforms.functional.normalize(img, (0.5,0.5,0.5,0.5,0.5,0.5), (0.25,0.25,0.25,0.25,0.25,0.25))
</code></pre>
<p>But I get the following error:</p>
<pre><code class="lang-auto">TypeError: img should be PIL Image. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>Can somebody tell me what I can do to transform my 6 channel images?</p>
<p>Thanks =)</p>

ptrblck: <p>You might need to update to the latest <code>torchvision</code> version, as the tensor input functionality was just recently added. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Maria_Culman: <p>Could you point out which version of <code>torchvision</code> do you refer to?</p>
<p>I am using <code>torch==1.7.1+cu101 torchvision==0.8.2+cu101</code>. According to PyTorch 1.7.1 documentation: “All transformations accept PIL Image, Tensor Image or batch of Tensor Images as input.”</p>
<p>Still, when trying to apply <code>transforms.RandomVerticalFlip()</code> and <code>transforms.RandomHorizontalFlip()</code> to a torch.tensor with <code>torch.Size([10, 8, 8])</code> → <code>(C, H, W)</code> shape, I am getting:</p>
<p><code>TypeError: img should be PIL Image. Got &lt;class 'numpy.ndarray'&gt;</code> from site-packages/torchvision/transforms/functional_pil.py", line 73, in vflip<br>
raise TypeError(‘img should be PIL Image. Got {}’.format(type(img))).</p>

ptrblck: <p>The error points towards a numpy array as the input, which isn’t supported.<br>
Try to convert it to a tensor and rerun it.</p>

Maria_Culman: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Yes, you are right, I was inputting a NumPy array. What I finally did was building Torchvision from the source, as the tensor use is only allowed in the master branch (not available in the releases).</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>

ptrblck: <p>Good to hear it’s working now!<br>
As a side note: you could also install the nightly binary, which would ship with the latest features (built from the master branch from “last night”) by selecting the “Preview (Nightly)” tab <a href="https://pytorch.org/get-started/locally/">here</a>.</p>
Convolution between >3d tensor and 1d kernel?

sibowi: <p>I have two tensors, <code>A</code> and <code>B</code>, with sizes</p>
<pre><code class="lang-auto">A.size()
&gt;&gt; torch.Size([10, 5, 5, 5])

B.size()
&gt;&gt; torch.Size([4])
</code></pre>
<p>In scipy it’s possible to perform a convolution of the two along a specific axis by</p>
<pre><code class="lang-auto">scipy.ndimage.convolve1d(A, B, axis=0)
</code></pre>
<p>Is this possible with pytorch? I’ve been trying to make it work with <code>torch.nn.functional.conv1d</code>, but haven’t succeded.</p>
<p>Thanks a lot for your time!</p>
<p>Best regards</p>

Eta_C: <p>I think <code>pad</code> + <code>unfold</code> + <code>matmul</code> + <code>slice</code> may work.</p>

sibowi: <p><a class="mention" href="/u/eta_c">@Eta_C</a>  I’m sorry, but I can’t follow. Would you elaborate a little?</p>
Loss.backward() on Huggingface Reformer model gives error

Hari_Krishnan: <p>I am using a <code>ReformerForQuestionAnswering</code>for training on a QA task.</p>
<p>Here’s a snippet of the code that can reproduce the error.</p>
<pre><code class="lang-auto">from transformers import ReformerTokenizer, ReformerForQuestionAnswering

tokenizer = ReformerTokenizer.from_pretrained('google/reformer-crime-and-punishment')
model = ReformerForQuestionAnswering.from_pretrained('google/reformer-crime-and-punishment')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
inputs = tokenizer(question, text, return_tensors='pt')
start_positions = torch.tensor([1])
end_positions = torch.tensor([3])

outputs = model(**inputs, start_positions=start_positions, end_positions=end_positions)
loss = outputs.loss
loss.backward()
</code></pre>
<p>The backward() call throws the following error</p>
<blockquote>
<p>TypeError: int() argument must be a string, a bytes-like object or a number, not ‘NoneType’</p>
</blockquote>
<p>Thank You</p>

ptrblck: <p>I don’t know which operation raises this error, so could you please post the complete stack trace?<br>
Also, you might get faster and better answers in the <a href="https://discuss.huggingface.co/">huggingface forum</a> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Hari_Krishnan: <p>I hadn’t put the model in <code>train()</code> mode.</p>
<p>After doing <code>model.train()</code>. It worked</p>
how do i load saved “state_dict” on 3rd and 4th layer of ResNet-152

prashant_bahuguna: <ol>
<li>
<p>i tried copying weight manually from the state dictionary however I am only able to load a single layer<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dbadb5cd7d89f784a0cc1452cfc89544838306f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8dbadb5cd7d89f784a0cc1452cfc89544838306f" title="WhatsApp Image 2021-03-03 at 5.57.52 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8dbadb5cd7d89f784a0cc1452cfc89544838306f_2_544x500.jpeg" alt="WhatsApp Image 2021-03-03 at 5.57.52 PM" data-base62-sha1="kdNF47SVj7giRxbUNvJgJ7rrt7h" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8dbadb5cd7d89f784a0cc1452cfc89544838306f_2_544x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dbadb5cd7d89f784a0cc1452cfc89544838306f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dbadb5cd7d89f784a0cc1452cfc89544838306f.jpeg 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2021-03-03 at 5.57.52 PM</span><span class="informations">626×575 56.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>then i thought of manually adding dictionary words but the process is impractical</p>
</li>
</ol>

ptrblck: <p>If some layer names are equal, you could load these parameters via <code>load_state_dict(..., strict=False)</code>.</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
        self.new_layer = nn.Conv2d(64, 64, 3, 1, 1)
    
    def forward(self, x):
        x = self.conv1(x)
        x = self.new_layer(x)
        return x

ref_model = models.resnet152()
model = MyModel()
model.load_state_dict(ref_model.state_dict(), strict=False)

# compare
print((model.conv1.weight == ref_model.conv1.weight).all())
&gt; tensor(True)
</code></pre>
<p>Note that <code>strict=False</code> will ignore all incompatible keys and you should double check, that the wanted parameters were indeed copied.</p>
How to save tensor value to bin file

sunyoung: <p>I know how I could save the tensor itself by torch.save.</p>
<p>I need to use the torch value not in pytorch but in other program. so I want to export the only “VALUE” of the tensor as bin file.</p>
<p>Right now, I don’t know how to do that so just print out all the tensor value and make some code to make it binary value. but I think it is not the best idea because it is not actual bin file.</p>
<p>Is there any way to save the VALUE of the tensor, like Neural networks’s weights, as bin file?<br>
please help me!</p>
SyncBatchNorm for libtorch, anyone?

botelho: <p>I’m looking for a way to implement the behavior of <code>nn.SyncBatchNorm</code> in libtorch. Does anybody have any ideas/suggestions on how to do this?</p>
<p>Without synchronizing batch statistics, running distributed training with libtorch is a no-go (assuming the network has BatchNorm layers), as I have learned the hard way! Others must have run through the same issue, and hopefully implemented a custom layer to solve the problem? I’m not even hoping for a ready-to-use solution, but maybe some ideas on how to put it together. Thanks!</p>
How to use multiprocessing and data parallelism together?

ysleer: <p>Hello, thank you for reading my writing.</p>
<p>I am now studying on multiprocessing.</p>
<p>What I want to do is load data from multiple data loaders and It supplies data to the queue using multiprocessing.</p>
<p>The reason for creating multiple data loaders is that each has a different image size and batch size.</p>
<p>The question here is, is it an error to see if dataparallel crashes after creating a process for the data loader?</p>
<p>Thank you.</p>
Different CIFAR10 performance on different version of torch

zhengxiawu: <h2>I tested the NASNet on cifar10 with the different torch versions. Surprisingly, 1.0.1.post2 has the highest performance, what is the difference of 1.0.1.post2 with other versions (PS: I also tested other architectures and have a similar result)</h2>
<p>The detailed experiments are listed as follows:</p>
<p>Version Acc. training time (hours)<br>
1.5.0 97.02 31.43<br>
1.0.1 96.79 32.7025<br>
1.0.1.post2 97.11 40.8297<br>
1.1.0 96.86 42.3611<br>
1.2.0 96.79 40.7019<br>
1.3.0 96.86 34.0188<br>
1.3.1 96.69 41.6177<br>
1.4.0 97.02 34.2972</p>

ptrblck: <p>How many runs did you try and how large are the standard deviation and mean values for these versions? Single runs between different versions might not be bitwise accurate due to changes in the libs (CUDA, cudnn) as well as the PyTorch backend.</p>

zhengxiawu: <p>To generate exactly with the same results, I manually set all random seed to 2, including torch/torch coda/ python random seed/ and numpy random seed, every network is run with 4 times, each run have the same results. Actually, I want to know what is the difference between 1.0.1.post2 with other versions? Is there any files that summarize the changes?</p>

ptrblck: <p>The deterministic results are good for the same version, but I don’t think we can guarantee bitwise accurate results between different versions, so you would have to rerun the scripts with different seeds and check the std dev of the final results.</p>
<aside class="quote no-group" data-username="zhengxiawu" data-post="3" data-topic="84265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhengxiawu/48/6819_2.png" class="avatar"> zhengxiawu:</div>
<blockquote>
<p>Actually, I want to know what is the difference between 1.0.1.post2 with other versions? Is there any files that summarize the changes?</p>
</blockquote>
</aside>
<p>You could check the <a href="https://github.com/pytorch/pytorch/releases">Release notes</a> for a high-level overview or use the <code>compare</code> feature of GitHub.<br>
<a href="https://github.com/pytorch/pytorch/compare/v1.5.0...v1.0.1">Compare for 1.5.0 vs 1.0.1</a></p>

Guangxin_Su: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="84265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The deterministic results are good for the same version, but I don’t think we can guarantee bitwise accurate results between different versions, so you would have to rerun the scripts with different seeds and check the std dev of the final results.</p>
</blockquote>
</aside>
<p>I use the same code on 10.2 and 0.4.1 the f1 value differents 0.25, and both of them are stable,</p>
How does Pytorch implement nn.functional.conv_transpose2d?

lnair: <p>Hello!</p>
<p>I reviewed “A guide to convolution arithmetic for deep learning” (Dumoulin and Visin 2016), which states that “it is always possible to emulate a transposed convolution with a direct convolution”. However, they mention that this is an inefficient implementation. So how does Pytorch implement the transposed convolution?</p>
<p>I tried (unsuccessfully) to trace the actual implementation of conv_transpose2d. Hence, any pointers would greatly help.</p>
<p>Thanks in advance!</p>

albanD: <p>Hey!</p>
<p>So the way to go is:</p>
<ul>
<li>
<code>function.conv_transpose2d</code> calls into <code>torch.conv_transpose2d</code>.</li>
<li>
<code>torch.conv_transpose2d</code> is nowhere in python files <a href="https://github.com/pytorch/pytorch/search?l=Python&amp;q=conv_transpose2d" class="inline-onebox">Search · conv_transpose2d · GitHub</a> meaning that it comes from c++</li>
<li>All C++ functions are defined in a nice yaml file: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml" class="inline-onebox">pytorch/native_functions.yaml at master · pytorch/pytorch · GitHub</a>
</li>
<li>You can find <code>conv_transpose2d</code> there: <a href="https://github.com/pytorch/pytorch/blob/12d63cc2f59e31c6472742babb6122eb1a1d7140/aten/src/ATen/native/native_functions.yaml#L1101-L1102" class="inline-onebox">pytorch/native_functions.yaml at 12d63cc2f59e31c6472742babb6122eb1a1d7140 · pytorch/pytorch · GitHub</a>
</li>
<li>Nothing special mentionned, meaning that it is implemented in the <code>native/</code> folder there (look for " conv_transpose2d(" )</li>
<li>You can find the function there: <a href="https://github.com/pytorch/pytorch/blob/d8730194e7a7c46bea66d43d75238bc4a4d80627/aten/src/ATen/native/Convolution.cpp#L582-L587" class="inline-onebox">pytorch/Convolution.cpp at d8730194e7a7c46bea66d43d75238bc4a4d80627 · pytorch/pytorch · GitHub</a>
</li>
<li>Calling back into <code>at::</code> namespace means that it goes through the <code>native_functions.yaml</code> file again.</li>
<li>No special dispatch for this one either: <a href="https://github.com/pytorch/pytorch/blob/d8730194e7a7c46bea66d43d75238bc4a4d80627/aten/src/ATen/native/native_functions.yaml#L1050-L1051" class="inline-onebox">pytorch/native_functions.yaml at d8730194e7a7c46bea66d43d75238bc4a4d80627 · pytorch/pytorch · GitHub</a> so looking for " convolution(" now.</li>
<li>You have it here: <a href="https://github.com/pytorch/pytorch/blob/d8730194e7a7c46bea66d43d75238bc4a4d80627/aten/src/ATen/native/Convolution.cpp#L596" class="inline-onebox">pytorch/Convolution.cpp at d8730194e7a7c46bea66d43d75238bc4a4d80627 · pytorch/pytorch · GitHub</a>
</li>
<li>Which then calls into the core conv implementation here: <a href="https://github.com/pytorch/pytorch/blob/d8730194e7a7c46bea66d43d75238bc4a4d80627/aten/src/ATen/native/Convolution.cpp#L615" class="inline-onebox">pytorch/Convolution.cpp at d8730194e7a7c46bea66d43d75238bc4a4d80627 · pytorch/pytorch · GitHub</a> where we choose the backend depending on what is available and what the user wants.</li>
</ul>
OSError: undefined symbol: free_gemm_select, version libcublasLt.so.11

Wilan: <p>When importing torch, I’m getting an OSError<br>
Does anyone know what may be the issue?</p>
<p>I’ve created an environment on Ubuntu 20.04 LTS and have re-created it on 20.04.2 LTS</p>
<p>Error:</p>
<pre><code class="lang-auto">OSError                                   Traceback (most recent call last)
&lt;ipython-input-1-fd2d69af7979&gt; in &lt;module&gt;
      1 import os
      2 import shutil
----&gt; 3 import torch
      4 import cv2
      5 import numpy as np

~/anaconda3/envs/cv/lib/python3.6/site-packages/torch/__init__.py in &lt;module&gt;
    187     # See Note [Global dependencies]
    188     if USE_GLOBAL_DEPS:
--&gt; 189         _load_global_deps()
    190     from torch._C import *
    191 

~/anaconda3/envs/cv/lib/python3.6/site-packages/torch/__init__.py in _load_global_deps()
    140     lib_path = os.path.join(os.path.dirname(here), 'lib', lib_name)
    141 
--&gt; 142     ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
    143 
    144 

~/anaconda3/envs/cv/lib/python3.6/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    346 
    347         if handle is None:
--&gt; 348             self._handle = _dlopen(self._name, mode)
    349         else:
    350             self._handle = handle

OSError: /home/wilson/anaconda3/envs/cv/lib/python3.6/site-packages/torch/lib/../../../../libcublas.so.11: undefined symbol: free_gemm_select, version libcublasLt.so.11
</code></pre>

ptrblck: <p>This should already be solved in <a href="https://github.com/pytorch/pytorch/issues/51080#issuecomment-787133939">this issue</a>.</p>

Wilan: <p>Thanks for the reply! Worked out perfectly</p>
Tensor/tensor.h missing from libtorch packages

josevalim: <p>Hi! <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"></p>
<p>We are writing libtorch bindings <a href="https://github.com/elixir-nx/nx" rel="noopener nofollow ugc">for Elixir’s Nx</a> - thank you for making it easily accessible - and we have noticed the precompiled libtorch archives, for both Linux and Mac, are missing the file “include/tensor/tensor.h”.</p>
<p>Am I missing some crucial step in setting this up? For now we are using aten.</p>
<p>EDIT: We are using a <a href="https://github.com/elixir-nx/nx/blob/sv-torchx/torchx/Makefile" rel="noopener nofollow ugc">Makefile</a> to build libtorch and perhaps that’s the root cause? We should be using CMake to build certain headers for us?</p>
<p>Thank you!</p>

josevalim: <p>Mystery cracked, sorry for the noise: there is a second include directory at <code>include/torch/csrc/api/include</code>. We fixed it by doing:</p>
<pre><code>LIBTORCH_INCLUDE_PATH = $(LIBTORCH_DIR)/include/ $(LIBTORCH_DIR)/include/torch/csrc/api/include
</code></pre>
<p>I will be glad to send a PR to the docs if you think it is worth talking about a non-CMake build. Thanks!</p>
Numerical Instability for Normalization

iordanis: <p>The numerical instability is very high when converting from long to float in range(-1,1) and then back to long. So much so that approximately 15% of the numbers are converted incorrectly.</p>
<p>What is more interesting is that for specific K, there are no numerical instability issues, and some of the K seem to be prime numbers as well.</p>
<p>The results are very consistent. Any ideas on numerically stable normalization?</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210302-060106-RC00_360395915" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1OWU_LQw_s7x1dLR2W8z5QPxQX1HIuWCE?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1OWU_LQw_s7x1dLR2W8z5QPxQX1HIuWCE?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Trace Backward Pass

cpp_for_torch: <p>Hey guys, I would like to trace/ Build a Pytorch IR for a backward pass/backprop using Torch’s JIT, but for some reason everytime I try trace, it goes through the program 3 times. The first time, it outputs that the loss tensor have grad_fn. But the steps after that it shows that the loss_fn does not have grad_fn.</p>
<p>Toy example:</p>
<pre><code class="lang-python">import torch

def simple(x):
    loss = x.sum()
    print(loss)
    loss.backward()
    print("hello")
    return loss

if __name__ == "__main__":
    example = torch.rand(1, 3, 224, 224, requires_grad=True)
    traced_script_loss = torch.jit.trace(simple,example)

</code></pre>
<p>error:<br>
in backward allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>Any guidance would be very appreicated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

alannnna: <p>I am having the same issue where tracing through backward() gives me the same error: <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>. The stack trace reports <code>loss.backward()</code> as being the issue, but loss requires a grad.</p>
<p>It <a href="https://discuss.pytorch.org/t/torchscript-for-the-backward-autograd-graph/81873/2">looks like</a> tracing backward() calls should be supported…</p>
Problem about WGAN

yichuan9527: <p>I run WGAN’s code<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/martinarjovsky/WassersteinGAN" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/5272722?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/martinarjovsky/WassersteinGAN" target="_blank" rel="nofollow noopener">martinarjovsky/WassersteinGAN</a></h3>

<p>Contribute to martinarjovsky/WassersteinGAN development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I get this bug:<br>
in the line 184<br>
input.resize_as_(real_cpu).copy_(real_cpu)<br>
TypeError: resize_as_ received an invalid combination of arguments - got (!torch.FloatTensor!), but expected (torch.cuda.FloatTensor template)<br>
Is any body have the same problem? How to solve this?</p>

smth: <p>this has been fixed in the latest version of the code.</p>

yichuan9527: <p>your meaning the fixed problem is WGAN code or pytorch?</p>

yichuan9527: <p>I solve this problem!<br>
Thanks very much!</p>
Create Dataloaders that works with dictionaries

Matheus_Sasso: <p>I am developping a QG (Question Generation) application. Where my data is on the following format:</p>
<pre><code class="lang-auto">[{"context":"Some Context",
"question":["question1", "question2", "question3"]},
{"context":"Some other  Context",
"question":["question1"]},
{"context":" Another Context",
"question":["question1", "question2", "question3","question4"]},

...
]
</code></pre>
<p>In a fist moment I have created dataframe replicating the context for every question. So I have created this Dataset that works well, but dont allows me to correctly track my metrics (BLEU,ROUGE,…).</p>
<pre><code class="lang-auto">from typing import List, Union,Optional
class CustomDataset(Dataset):
    def __init__(self,PREFIX,tokenizer,X_context:np.ndarray,y_question:Optional[np.ndarray]=[],
                 source_max_length: int = 32, target_max_length: int = 32):
        
        self.tokenizer = tokenizer
        self.X_context = X_context
        
        self.y_question = y_question
        self.source_max_length = source_max_length + len(PREFIX.split(' '))
        self.target_max_length = target_max_length
        self.PREFIX = PREFIX
        

    def __len__(self):
        return len(self.X_context)
    
    def __getitem__(self, idx):
        #Source
        original_source = self.X_context[idx]
        source = f"{PREFIX} {original_source}"
        source_encoder = self.encoder_plus(source,self.source_max_length)
        source_token_ids = source_encoder['input_ids']
        source_mask = source_encoder['attention_mask']
        source_token_ids = torch.tensor(source_token_ids).type(torch.long)
        source_mask = torch.tensor(source_mask).type(torch.long)
        
        # Target
        original_target = self.y_question[idx]
        target = f"{original_target}"
        target_encoder = self.encoder_plus(target,self.target_max_length)
        target_token_ids = target_encoder['input_ids']
        target_mask = target_encoder['attention_mask']
        target_token_ids = torch.tensor(target_token_ids).type(torch.long)
        target_mask = torch.tensor(target_mask).type(torch.long)

        return (source_token_ids, source_mask, target_token_ids, target_mask,
                original_source, original_target)
          
    
    def encoder_plus(self,text,L):
      return self.tokenizer.encode_plus(text,
                                        max_length = L,
                                        truncation=True,
                                        padding="max_length")
</code></pre>
<p>But now I  want to have all the questions that are correlated with the same context <strong>on the same batch</strong> to be able to track all metrics on my model. The problem is when I generate a question, I they dont are on the same batch I cant know to wich question compare. I image that possibily i could use a collate function, but cant figure out how.</p>
Optimization of patch location

imesery: <p>Hi guys,</p>
<p>Is there a simple way to use autograd to optimize a patch location over an image (“slide” the patch over the image)?<br>
for simplicity sake, let’s assume I only have a 3x3 black patch which I apply on MNIST images in order to do some task, for example - try to maximize the score of some target class I choose (simple targeted adversarial attack).</p>
<p>the standard phrasing of similar attacks is the following:<br>
let’s assume a trainable patch_mask tensor that has the same dims as the image (28x28 in MNIST), where all the elements are 0’s besides a 3x3 patch with 1’s. the modified input to some CNN network will be:<br>
clip(input_image+patch_mask*(-1), 0, 1).</p>
<p>with this phrasing I can’t see how it’s possible to enforce patch_mask to be all 0’s besides a 3x3 square of 1’s, but maybe I’m wrong… and maybe I should look at it in a totally different way?</p>
<p>Thanks in advance,<br>
Itai</p>