akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to debug a Seq2Seq model?

Shifu_Jain: <p>Hi Everyone,</p>
<p>I have a question on debugging a Sequence2Sequence neural network in Pytorch. Here is the scenario:<br>
We have 2 models, one is implemented in Tensorflow and the other in Pytorch. After running 1 epoch for each, we got training loss of 1.32 (Tensorflow) and 2.1 (Pytorch). The problem here is the predicted summaries for Tensorflow code are much closer to the gold summaries than Pytorch predicted summaries. Also, most of the PyTorch model predictions are PAD tokens.<br>
Can anyone help us in debugging the PyTorch seq2seq model to get comparable predictions to Tensorflow predictions?<br>
P.S. → The TensorFlow code was implemented using Tensorflow 1.0 and we are now trying to replicate it in the newest version of Pytorch.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07f08fa85b7d8b65b0ff2a3978e344043a65d5e4.png" alt="Rouge" data-base62-sha1="18eK12KLdRUxcLfuUSV1Ho0xKRu" width="358" height="113"></p>

vdw: <p>Well, one basic indicator is whether you can overfit your network on a (very) small training dataset. If you can’t get it to overfit, i.e., get a training accuracy of 100% or close to, then your network is not training. Most likely because of some issues in your code.</p>
<p>Do you have a Github link or something to have a look at your code. If it’s not to much, maybe you can post your code here. The important parts are usually the model (with the forward method) and the training loop.</p>
Help with Simple ANN for data prediction

RAVI_CHANDAN: <p>I am a mechanical engineering student and this is my first time venturing into coding. I wish to create a Neural network for predicting an experimental output.</p>
<p>I have three inputs<br>
Let’s say Power§, feed(F), speed(V) and<br>
Two outputs<br>
Let’s say X, Y</p>
<p>I hear that there different types of Neural networks like ANN, RNN and CNN etc. These are some which I know of, so I want to create a customised Neural network in which we can change the no. of neurons, layers and training functions and different aspects to compare different networks and decide the best one with higher prediction accuracy.</p>
<p>Can anyone please help.</p>

ptrblck: <p>To get familiar with PyTorch I would recommend to take a look at <a href="https://pytorch.org/tutorials/">the tutorials</a>.<br>
However, given that you just start working in machine learning, I think a course such as <a href="https://www.fast.ai/">fast.ai</a> might also be a good starter.</p>
<p>If you want to learn more about deep learning specifically, you could take a look at the <a href="https://www.deeplearningbook.org/">DeepLearning Book</a> or at <a href="https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf">Bishop’s Pattern Recognition and Machine Learning</a> for a general ML introduction. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
AttributeError: module 'torch.utils' has no attribute 'checkpoint'

hadaev8: <p>I have it like this</p>
<pre><code>import torch
bla bla bla
mels_self = torch.utils.checkpoint.checkpoint(self.decoder_self_run, mels, encoder_outputs, batch['text_lengths'], batch['mel_lengths'])
</code></pre>
<p>With</p>
<pre><code>from torch.utils.checkpoint import checkpoint
</code></pre>
<p>same code works fine, why so?</p>

ptrblck: <p>Which PyTorch version are you using (you can check it via <code>print(torch.__version__)</code>).<br>
<code>torch.utils.checkpoint</code> was introduced some time ago, so I’ wondering if you are working with a really old PyTorch build. If so, could you update to the latest stable release (<code>1.7.0</code>) and retry your code?</p>

hadaev8: <p>Pytorch 1.6<br>
So it works, but only if I import checkpoint separately.<br>
I tried to update the version to 1.7, but another library raises an error.</p>

hadaev8: <p>The problem still exists on 1.7</p>

ptrblck: <p>I still cannot reproduce it in <code>1.7.0</code>, so could you post an executable code snippet, which raises the issue in this PyTorch version, please?</p>

hadaev8: <p>Here is code</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201105-085602-RC00_340833107" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1K7XaUTADfaq4XGeyx7tNffPExuJdHfXr?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1K7XaUTADfaq4XGeyx7tNffPExuJdHfXr?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Thanks for the notebook.<br>
It works fine when I uncomment</p>
<pre><code class="lang-python">from torch.utils.checkpoint import checkpoint
</code></pre>

hadaev8: <p>Yes, but why I need this import?<br>
Shouldnt it be covered by whole pytorch import?</p>

ptrblck: <p>I’m not completely sure how the import mechanism works in different Python versions and my understanding is that newer Python versions are more “flexible”. You could thus try it with e.g. Python3.8 and if it’s still not working would need to import the method directly (unsure if it’s a PyTorch limitation or the desired workflow in Python).</p>
Can convolutions in pytorch be 3x1 rectangular or just square

miner_tom: <p>I am new to pytorch but not new to image processing. I am so old that when I started with image processing we used to build our own convolvers from discrete multipliers and accumulators (a 3x3 conv took 9 discrete multipliers).</p>
<p>Of course, in discrete code like C or python one can do a convolutional filter in any size or or shape. But I am new to pytorch and I see convolution functions but so far have not seen a way to adjust the “shape” of the convolution filter to be anything but square as opposed to rectangular.</p>
<p>Thank You<br>
Tom</p>

Usama_Hasan: <p>Hy Tom,<br>
Yes you can use rectangular convolution filter.</p>
<pre><code class="lang-auto"># consider input (3,128,128)
torch.nn.Conv2d(3,64,(1,3),stride=2,groups=1)
torch.nn.Conv2d(64,128,(3,1),stride=2,groups=1)
# Then output after these convolutions will be (128,31,32)
</code></pre>
Giving multiple parameters in optimizer

yunjey: <p>How can i give multiple parameters to the optimizer?</p>
<pre><code>fc1 = nn.Linear(784, 500)
fc2 = nn.Linear(500, 10)
optimizer = torch.optim.SGD([fc1.parameters(), fc2.parameters()], lr=0.01)  # This causes an error.
</code></pre>
<p>In this case, for simplicity, i don’t want to use a class with nn.Module.</p>

smth: <p>you have to concatenate python lists:</p>
<pre><code class="lang-auto">params = list(fc1.parameters()) + list(fc2.parameters())

torch.optim.SGD(params, lr=0.01)
</code></pre>

yunjey: <p>Thanks, it works well.</p>

Zichun_Zhang: <p>dear  smth<br>
you really know a lot<br>
thx for help all along</p>

MrPositron: <p>Dear Soumith,</p>
<p>While executing your approach, it says:</p>
<p>TypeError: add() received an invalid combination of arguments - got (list), but expected one of:</p>
<ul>
<li>(Tensor other, Number alpha)</li>
<li>(Number other, Number alpha)</li>
</ul>
<p>Can you help me?)</p>
<p>Is there something wrong?</p>

justusschock: <p>Probably you set a bracket to the wrong place. You have to convert the parameters to a list separately and add the lists afterwards.</p>

MrPositron: <p>[SOLVED]</p>
<pre><code class="lang-auto">params = self.net.state_dict()
pas = list(params['net.0.weight']) + list(params['net.0.bias']) + list(params['net.3.weight'] + list(params['net.3.bias'])  + list(params['net.6.weight']) + list(params['net.6.bias']))
self.optimizer1 = optim.Adam(pas, lr = 0.01)
</code></pre>
<p>Here is my code. I think everything is ok</p>

huangbiubiu: <p>Since <code>parameters()</code> actually returns a iteration, <code>itertools.chain()</code> looks a better approach:</p>
<pre><code class="lang-auto">import itertools

params = [fc1.parameters(), fc2.parameters()]

torch.optim.SGD(itertools.chain(*params), lr=0.01)
</code></pre>

riley: <p>How is this different from just putting all of the tensors in a list directly as OP did?</p>

NeverConvex: <p>If your models are in a list or tuple somewhere already, you can also use a nested list comprehension:</p>
<pre><code class="lang-auto">models = [nn.Linear(784, 500),
          nn.Linear(500, 10)
          ]
optimizer = torch.optim.SGD((par for model in models for par in model.parameters()),
                             lr=0.01)
</code></pre>

Shubajit_Saha: <p>OP is passing a list of lists.</p>

ncuxomun: <p>Thank you! This helped me a lot!</p>
Torch.save and AttributeError: Can't pickle local object

hosseinshn: <p>Hello,<br>
I have a network architecture like below that chooses different options based an input argument.<br>
When I try to save the model at a certain epoch during training using torch.save() I get:</p>
<pre><code class="lang-auto">AttributeError: Can't pickle local object 'main.&lt;locals&gt;.Net1'
</code></pre>
<p>Here is the part that I design the network in the main():</p>
<pre><code class="lang-auto">IE_dim = X_tr.shape[1]        
    if args.hd == 1:
        class Net1(nn.Module):
            def __init__(self, args):
                super(Net1, self).__init__()

                self.features = torch.nn.Sequential(
                    nn.Dropout(args.idr),
                    nn.Linear(IE_dim, 512),
                    nn.Tanh(),
                    nn.Dropout(args.ldr),
                    nn.Linear(512, 1))

            def forward(self, x):
                out = self.features(x)
                return out  
        Model = Net1(args)

    elif args.hd == 2: 
        class Net2(nn.Module):
            def __init__(self, args):
                super(Net2, self).__init__()

                self.features = torch.nn.Sequential(
                    nn.Dropout(args.idr),
                    nn.Linear(IE_dim, 256),
                    nn.Tanh(),
                    nn.Dropout(args.ldr),
                    nn.Linear(256, 256),
                    nn.Tanh(),
                    nn.Dropout(args.ldr),
                    nn.Linear(256, 1)) 

            def forward(self, x):
                out = self.features(x)
                return out            
        Model = Net2(args)

    elif args.hd == 3:
        class Net3(nn.Module):
            def __init__(self, args):
                super(Net3, self).__init__()    

                self.features = torch.nn.Sequential(
                    nn.Dropout(args.idr),
                    nn.Linear(IE_dim, 128),
                    nn.Tanh(),
                    nn.Dropout(args.ldr),
                    nn.Linear(128, 128),
                    nn.Tanh(),
                    nn.Dropout(args.ldr),
                    nn.Linear(128, 128),
                    nn.Tanh(),
                    nn.Dropout(args.ldr),
                    nn.Linear(128, 1))                        

            def forward(self, x):
                out = self.features(x)
                return out
        Model = Net3(args)        
</code></pre>
<p>and here is how I use torch.save():</p>
<pre><code class="lang-auto">torch.save(Model, os.path.join(SOME PATH, 'Best_Model.pt'))
</code></pre>
<p>I had that piece of code first in a separate .py file and used import * in the main() but got the above error and then, I moved the code to the main() function but got the same error.<br>
I appreciate any help!</p>
GPU memory consumption increases for LSTM models if not sorting samples by lengths

galactica147: <p>Hi, I’m training a LSTM model with variable-length samples. One thing I observe is that if I sort all training data by sample length and then prepare the data loader for that, I can afford to use bigger batch sizes than without sorting them first.</p>
<p>Since the training data stays the same and also the max length of a padded batch also stays the same, I’m wondering if pack_padded_sequence() in the unsorted case might actually cost more GPU memory.</p>
<p>Say we have a padded batch after sorting by length, <code>s*_t*</code> means <code>sample*_timestep*</code></p>
<pre><code class="lang-auto">[s0_t0, 0, 0, 0]
[s1_t0, s1_t1, 0, 0]
[s2_t0, s2_t1, s2_t2, 0]
[s3_t0, s3_t1, s3_t2, s3_t3]
</code></pre>
<p>In the unsorted case it might look arbitrary in terms of padding:</p>
<pre><code class="lang-auto">[s0_t0, s0_t1, s0_t2, 0]
[s1_t0, 0, 0, 0]
[s2_t0, s2_t1, 0, 0]
[s3_t0, 0, 0, 0]
</code></pre>
<p>My hypothesis is in the unsorted case we tend to pad more, so we may use more GPU memory.<br>
Is that the case here? I guess it depends on how pack_padded_sequence() is implemented in CUDA too.</p>
<p>Any other thoughts why GPU memory may increase?<br>
Thanks!</p>

galactica147: <p>bump, any thoughts guys?</p>

galactica147: <p>just want to update that it is not because of we have any GPU memory increase in LSTM layers, but the fact that it’s followed by FC layers where without sorting, the sequence length gets longer as input to FC, thus bigger FC and GPU RAM consumption.</p>
My libtorch build is much slower than prebuilt ones

Mohammadreza_Nazari: <p>I am trying to replicate building libtorch with dependencies. I use the instructions on <a href="https://github.com/pytorch/builder/" rel="noopener nofollow ugc">https://github.com/pytorch/builder/</a>, but still there are some performance degradation. The performance loss is even worse when I try a different GPU other than the one that I used for building the binraries.</p>
<p>I have two questions:</p>
<ol>
<li>
<p>Is the <a href="https://github.com/pytorch/builder" rel="noopener nofollow ugc">https://github.com/pytorch/builder</a> is the official pytorch project that is used for building libtorch? It seems that the instructions are a bit outdated.</p>
</li>
<li>
<p>Is there any place to see the build log of pre-built libtorch binaries. For instance, which version of cudnn is used, what are the dependencies, etc.</p>
</li>
</ol>
Set value of torch tensor up to some index

Faeze_Br: <p>I have a 2D torch tensor (X) and a 1D index torch tensor (ind). I want to set values of each row in x (until corresponding index) to 2. like following:</p>
<p>X = [[1,0,4,5,6],<br>
[3,6,7,10, 13],<br>
[1,4,2,8,21]]<br>
ind = [2,3, 1]</p>
<p>result = [[2,2,4,5,6],<br>
[2, 2, 2, 10, 13]<br>
[2, 4, 2, 8, 21]]</p>
<p>Is there an easy way to do so?<br>
Thanks</p>

Nikronic: <p>Hi,</p>
<p>You can do this by creating binary mask.<br>
First we create a matrix with same shape as <code>X</code> filled with zeros, then put <code>1</code>s where index matches with <code>ind</code> tensor and finally by using <code>cumsum</code>, set <code>1</code> after previously located points. Finally, we can mask <code>X</code>:</p>
<pre><code class="lang-auto">X = torch.tensor(
    [[1,0,4,5,6],
     [3,6,7,10, 13],
     [1,4,2,8,21]])


ind = torch.tensor([2,3, 1])


mask = torch.zeros_like(X)
mask[(torch.arange(X.shape[0]), ind)] = 1
mask = 1 - mask.cumsum(dim=-1)
mask
# mask
# tensor([[1, 1, 0, 0, 0],
#         [1, 1, 1, 0, 0],
#         [1, 0, 0, 0, 0]])

X[mask.bool()] = # your desired value
</code></pre>
<p>Bests</p>

Faeze_Br: <aside class="quote no-group" data-username="Nikronic" data-post="2" data-topic="102097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p><code>mask = 1 - mask.cumsum(dim=-1)</code></p>
</blockquote>
</aside>
<p>great! thanks <a class="mention" href="/u/nikronic">@Nikronic</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
TypeError: 'NoneType' object is not callable while use the model

Lame: <p>I’m using PyTorch for training GCN, I have a simple model that saved using this command:</p>
<pre><code class="lang-auto">torch.save(MyNet().state_dict(), PATH)
</code></pre>
<p>Then I load it by doing the following:</p>
<pre><code class="lang-auto">model = MyNet()
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>But then when I tried to input the data as follows:</p>
<pre><code class="lang-auto">output = model(dataset)
</code></pre>
<p>I got this error for the previous line:</p>
<pre><code class="lang-auto">TypeError: 'NoneType' object is not callable
</code></pre>
<p>I don’t know what’s the reason for that, could you please help me?</p>

ptrblck: <p>Could you add some debug statements to your code and check, if the <code>model</code> variable was replaced somewhere?</p>
<pre><code class="lang-python">model = MyNet()
print(model)
[...]
print(model)
output = model(dataset)
</code></pre>
<p>I guess that <code>model</code> was initially a valid object, but might have been replaced accidentally with <code>None</code>.</p>

Lame: <p>When I tried to print the model after <code>model = MyNet()</code> I got the model printed well, but then when I tried to print it before <code>output = model(dataset)</code> I got <code>None</code></p>
<p>What could be the problem?</p>

ptrblck: <p>It seems your code is overwriting the <code>model</code> variable at one point.<br>
Search for all usages of <code>model</code> and make sure it’s not replaced by a <code>None</code> value.</p>
Maintain aspect ratio during grid sample

pgp: <p>I am using affine grid and grid sample to rotate an image by 90 degrees. This is the original image<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5b484ee5dcd5f2a7735677741540e73d14c0e28.jpeg" alt="pic" data-base62-sha1="pVrbl12zjgFXz0bia2oEiYBlXHG" width="280" height="180"></p>
<p>This is the output of grid sample. The aspect ratio is not maintained.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/836d2a71d0b532ecdcdecb91c9671e2e4a114994.jpeg" alt="py_rotate" data-base62-sha1="iKEqZcccMhDl5VBEmfEQXIfSgZe" width="280" height="180"></p>
<p>This is the expected output with padding set to zeros.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22ccf70387121b29fb916af6f6889d4687496487.jpeg" alt="sitk_rotate" data-base62-sha1="4XRmTrJ01uUtI8G4057wlqovAeb" width="280" height="180"></p>
<p>Is there anything else that needs to be considered?</p>

pgp: <p>Code to reproduce output:</p>
<pre><code class="lang-auto">import torch
import numpy as np
import matplotlib.pyplot as plt
import imageio

angle = 90
affine_matrix = np.array(
        (
            (np.cos(np.deg2rad(angle)), -np.sin(np.deg2rad(angle)), 0),
            (np.sin(np.deg2rad(angle)),  np.cos(np.deg2rad(angle)), 0),
            (0, 0, 1)), dtype=np.double
    )
rotate_pyorch_2d = torch.from_numpy(affine_matrix[:2, :3]).unsqueeze(0)
grid_2d = torch.nn.functional.affine_grid(rotate_pyorch_2d, img_2d_tensor.shape)
pytorch_resampled_2d = torch.nn.functional.grid_sample(img_2d_tensor, grid_2d, mode="nearest", padding_mode="zeros")
plt.imshow(pytorch_resampled_2d.squeeze().permute(1,2,0).numpy().astype(np.uint8))
</code></pre>

ptrblck: <p>I think the result is expected, since your <code>grid_2d</code> tensor fills the all pixels of the output tensor, which is why it would stretch the input.<br>
I think you can achieve the last output by setting the <code>size</code> in <code>affine_grid</code> to the smaller desired size and place the transformed image in the larger empty output.</p>

pgp: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> the angle of rotation can be other than 90 degrees. Also, the white part of the last image is the part of the image itself which is not been displayed in the first two images. In the last image, the whole image is rotated but by maintaining the aspect ratio. Can we set the size to make the rotation work for any degree of rotation?</p>

ptrblck: <p>If I understand the <code>grid_sample</code> and <code>affine_grid</code> method correctly, you are defining a sampling grid, where each position gives you the pixel location of the input image.<br>
If your grid is thus containing values in all positions, it would just fill these using the values.<br>
Thus, if you don’t want to fill the complete output image (i.e. leave the borders empty to maintain the aspect ratio), you would have to change the grid itself.</p>
How Efficient Is Sparse Matrix Computation on GPU?

Kevin96: <p>Hi,</p>
<p>PyTorch supports a few a few sparse matrix computation such as spmm. In principle, sparsity can reduce the complexity of matrix computation. So they are faster than the dense implementation on CPU for sure. However, on GPUs, these sparse operations are difficult to implement in parallel. In particular, this post <a href="https://discuss.pytorch.org/t/backprop-through-sparse-tensor-is-not-memory-efficient/66409" class="inline-onebox">Backprop Through Sparse Tensor Is Not Memory Efficient?</a> already shows that their memory usage may be as large as dense implementation.</p>
<p>So I am wondering their performance on GPUs (both speed and memory) and whether I should use the sparse implementation when I encounter sparse matrices.  I can provide necessary information about my use case (matrix size, sparsity level etc) if necessary. Also, I would appreciate it if there are empirical studies comparing dense and sparse functions in pytorch. Thanks.</p>
Why `torch.gather` has not input shape check?

zestyzapus: <p>The document of <a href="https://pytorch.org/docs/stable/generated/torch.gather.html#" rel="noopener nofollow ugc"><code>torch.gather</code></a> says the <code>index</code> argument must be an <code>n</code>-dimensional tensor with some certain shape. I thought this means that it will check the input dimension but it turned out it didn’t check and did the unexpected thing silently.</p>
<p>The following code has a bug, but it runs without even a warning.</p>
<pre><code class="lang-python">import torch
torch.manual_seed(0)
input = torch.rand(4, 2)
index = torch.randint(2, size=(4,)).unsqueeze(0)  # intended to be unsqueeze(1)
dim = 1
output = torch.gather(input, dim, index)
print("input = ", input)
print("index = ", index)
print("output = ", output)
</code></pre>
<p>I thought it would be good if we check the <code>index</code> shape. Otherwise, the documentation should mention that input dimension is not checked.</p>

ptrblck: <p>It seems the check was dropped somewhere between PyTorch <code>1.5.1</code> and <code>1.6</code>.<br>
Would you mind creating an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that we can track it?</p>

zestyzapus: <p>Here is the Github issue: <a href="https://github.com/pytorch/pytorch/issues/47610" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/47610</a></p>
SWA Manual Update OOM

Mario_Parreno: <p>Hi guys!</p>
<p>I am tying to use SWA with my custom dataloader but I have a doubt. This is my code:</p>
<pre><code class="lang-auto">swa_model.train()
for indx, batch in enumerate(train_loader):
    image = batch["image"].type(torch.float).cuda()
    _ = swa_model(image)
</code></pre>
<p>But this runs out of memory fast. If I encapsulate in torch.no_grad() runs without problems… Maybe gradients are not cleaned or something but I don’t have clear how to do it properly for SWA model.</p>
<p>“If your dataloader has a different structure, you can update the batch normalization statistics of the swa_model by doing a forward pass with the swa_model on each element of the dataset.”</p>

Mario_Parreno: <p>Using torch.no_grad() updates statistics:</p>
<pre><code class="lang-auto">swa_model.train()
with torch.no_grad()
    for indx, batch in enumerate(train_loader):
        image = batch["image"].type(torch.float).cuda()
        _ = swa_model(image)
</code></pre>
<p>Check with:</p>
<p>for module in s</p>
<pre><code class="lang-auto">wa_model.modules():
    if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
        print(module.running_mean)
        print(module.running_var)
        print(module.momentum)
        break
</code></pre>
Using .index_put() with a 3D tensor

antoniogois: <p>if I have a 2D tensor like this one:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.ones((2, 4))
[[1, 1, 1, 1],
 [1, 1, 1, 1]]
</code></pre>
<p>and want to fill two positions per row with 0, to get:</p>
<pre><code class="lang-auto">[[1, 0, 1, 0],
 [0, 1, 1, 0]]
</code></pre>
<p>I can do:</p>
<pre><code class="lang-auto">torch.ones((2, 4)).index_put((torch.arange(2).unsqueeze(1), torch.LongTensor([[1,3], [0,3]])), torch.Tensor([0]))
</code></pre>
<p>What about a 3D tensor? Let’s say I want to fill in a torch.ones(2, 3, 4) tensor with some zeros, to get:</p>
<pre><code class="lang-auto">tensor([[[1., 0., 1., 0.],
         [0., 1., 1., 0.],
         [1., 0., 0., 1.]],

        [[0., 1., 0., 1.],
         [1., 0., 0., 1.],
         [1., 1., 0., 0.]]])
</code></pre>
<p>if I have the zero-indices stored as:</p>
<pre><code class="lang-auto">torch.LongTensor([[[1,3],
                   [0,3],
                   [1,2]],

                  [[0,2],
                   [1,2],
                   [2,3]]])
</code></pre>
<p>is there a way to use these indices, to tell .index_put() where to place the zeros?</p>

albanD: <p>Hi,</p>
<p>For any number of dimensions, you can use scatter to achieve this:</p>
<pre><code class="lang-python">import torch

ind= torch.tensor([[[1,3],
                   [0,3],
                   [1,2]],

                  [[0,2],
                   [1,2],
                   [2,3]]])

base = torch.ones(2, 3, 4)

base.scatter_(2, ind, 0)

print(base)
</code></pre>

antoniogois: <p>Hi Alban!</p>
<p>That’s exactly what I needed, thanks a lot</p>
What is the best way to run crf with DDP and be compatible with JIT

tengerye: <p>Hi, I am trying to implement a crf model like <a href="https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html" rel="noopener nofollow ugc">this tutorial</a>. However, when I wrap the model with DDP, it raises the error “AttributeError: ‘DistributedDataParallel’ object has no attribute ‘neg_log_likelihood’.” One possible way is to extract the crf loss calculation out, but that seems not a good approach since: (1) The crf contains viterbi which will also be extracted out from my model; (2) I will have problems when converting my model to JIT.</p>
<p>Could someone help? Thank you in advance.</p>

tom: <p>So what happens is that DDP wraps the model, so the original model is ddp_model.model.<br>
Then DDP has a forward method that essentially does the “distributed-admin” (getting the data to the various cards and collecting results and preparing for backward results to be treated).<br>
Now you could run <code>ddp_model.model.neg_log_likelihood</code>, but that would run on a single instance, defeating the purpose of using DDP.<br>
I think the easiest way to is probably to fold all entry points (that should be distributed) into the forward method with a flag “what” that you check for the various functions.</p>
<p>Best regards</p>
<p>Thomas</p>
Multiple transform outputs

stases: <p>Hello! I am having (conceptual) issues with writing my own custom transform. Imagine that I have a image classification case and I wish to apply e.g 10 random crops for the same image so I write a custom transform just for that. Let’s say that I wish to do this because images are extremely large and I wish to avoid loading them many times (less epochs and larger batches). The problem I am having is what should be the output of this transform, since now I have multiple crops? First thing that comes to my mind is a list of tuples as (image, class), but then how would I feed this into my network without changing the architecture (I still wish to feed one image and get class scores).</p>
<p>Thanks in advance!</p>

ptrblck: <p>You could use a similar approach as shown in the <a href="https://pytorch.org/docs/1.7.0/torchvision/transforms.html?highlight=tencrop#torchvision.transforms.TenCrop"><code>TenCrop</code></a> approach, where the tensors are stacked to a 5d tensor first and later flattened such that the batch size increases.</p>

stases: <p>Thank you for your reply. I have tried implementing this, but the problem arises as my dataset does not acquire batch dimension until it passes through data loader. Is there any way to flatten after the data loader and before passing data to my network? Thanks!</p>

ptrblck: <p>This would be what the code example is doing, wouldn’t it?<br>
The <code>Dataset</code> would return a tensor in the shape <code>[crops, channels, height, width]</code>, while the <code>DataLoader</code> would add the batch dimension to <code>[batch_size, crops, channels, height, width]</code>.<br>
Inside the <code>DataLoader</code> loop you would then flatten it to <code>[batch_size*crops, channels, height, width]</code> and pass it to the model.</p>

stases: <p>Yeah, you’re right. For some reason I thought I would need to edit how DataLoader loads files, instead of simply flattening the data while looping. Anyways, thank you for your answer!</p>
Custom loss function for Pixel Wise Cross Entropy Loss

Entropy: <p>I want to implement a custom loss function that looks like the one in the image below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3333a1b5e6ae156a2349ef69763fbfc794cb1df.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3333a1b5e6ae156a2349ef69763fbfc794cb1df" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3333a1b5e6ae156a2349ef69763fbfc794cb1df_2_690x142.png" alt="Capture" data-base62-sha1="yHrMh0qOlV1HFd9xqQGs4BCnzCn" width="690" height="142" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3333a1b5e6ae156a2349ef69763fbfc794cb1df_2_690x142.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3333a1b5e6ae156a2349ef69763fbfc794cb1df_2_1035x213.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3333a1b5e6ae156a2349ef69763fbfc794cb1df_2_1380x284.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1542×319 55.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have never implemented a custom loss before so I am kinda stuck.</p>

albanD: <p>Hi,</p>
<p>If this is just the cross entropy loss for each pixel independently, then you can use the existing cross entropy provided by pytorch.<br>
The pytorch function only accepts input of size <code>(batch_dim, n_classes)</code>.<br>
So if your output is of size <code>(batch, height, width, n_classes)</code>, you can use <code>.view(batch * height * width, n_classes)</code> before giving it to the cross entropy function (considering each pixel as a different batch element) to achieve what you want.</p>

Entropy: <p>Hey! Thank you for responding! My output is of the shape [100, 11, 80, 80] where batch = 100, n_classes = 11, height/width = 80. How do I convert this tensor to a tensor that will be accepted by cross entropy loss?</p>

albanD: <p>You can do <code>out = inp.permute(0, 2, 3, 1).contiguous().view(-1, inp.size(1))</code> to have a Tensor whose shape is [640000, 11]</p>
Torch.onnx.export alternate layers for SnapML importing?

vergevaliant19: <p>I’ve implemented the SINet model from <a href="https://paperswithcode.com/paper/sinet-extreme-lightweight-portrait" rel="noopener nofollow ugc">this paper</a> so that I can <a href="https://lensstudio.snapchat.com/guides/machine-learning/ml-frameworks/export-from-pytorch/" rel="noopener nofollow ugc">export it to a .onnx and then to SnapML</a>.</p>
<p>However SnapML will not import some onnx layers like “Gather” which are generated from <code>torch.onnx.export()</code> for common methods like <code>torch.tensor.view()</code>. Is there an equivalent to <code>torch.tensor.view()</code> that <code>torch.onnx</code> won’t export to a “Gather” layer?</p>
Class attributes are not recognized in forward()

AfonsoSalgadoSousa: <p>Hello. I am trying to run the work of DSQ (paper: <a href="https://arxiv.org/pdf/1908.05033.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1908.05033.pdf</a>, code: <a href="https://github.com/ricky40403/DSQ" rel="noopener nofollow ugc">https://github.com/ricky40403/DSQ</a>) and faced a strange issue. I get “torch.nn.modules.module.ModuleAttributeError: ‘DSQConv’ object has no attribute ‘running_lw’”. Inside forward(), almost every attribute from the DSQConv class seems to not be noticed inside forward().</p>
<p>Also, can someone tell me how should I store an encoded version of the models to analyse the storage size difference between different bit ranges?</p>
Implement tf.nn.in_top_k in PyTorch

RoseFun: <p>How to achieve this function in pytorch?</p>
<pre><code class="lang-auto">tf.math.in_top_k(
    targets, predictions, k, name=None
)
</code></pre>
<p><a href="https://www.tensorflow.org/api_docs/python/tf/math/in_top_k" rel="noopener nofollow ugc">tf.nn.in_top_k</a></p>

Abhilash_Srivastava: <p>Try something similar.</p>
<pre><code class="lang-auto">top_k = torch.topk(targets, k)
if any([(targets == c_).all() for c_ in predictions]):
      print('targets in top_k predictions')
</code></pre>
Use pytorch for arabic sentiment analysis

Moise_Sacko2020: <p>Hi here. I am working on a project in which I have to analyse sentiment on a arabic dataset. I have been using Camel_tools with the original dataset and Textblob, AllenNLP with the translated dataset. Both methods give me an unefficient result. That’s why I want to know if there is a pytorch module i could use with the original dataset. If not, We could work as a community to build it.<br>
Thanks.</p>

Abhilash_Srivastava: <p>There are two things I have in mind:</p>
<ol>
<li>If you have sufficient labeled train data, you can simply treat it as an English sentiment analysis task.</li>
<li>Try a multi-language pretrained model (eg: XLMR and M-BERT) and add your sentiment analysis layer ton top.</li>
</ol>
A bug in torch.unique with return_inverse=True and dim != None?

cruvadom: <p>Try this:</p>
<p>aa = torch.tensor([[1, 2, 2], [3, 3, 3]])<br>
unq, groups = aa.unique_consecutive(dim=1, return_inverse=True)</p>
<p>groups is:<br>
tensor([0, 1, 1])</p>
<p>which are the group assignments for the first row only! expected groups to be of shape [2, 3], with the group assignments for each row… Is this a bug? Any way of achieving what I intended in an efficient way?</p>

Nikronic: <p>Hi,</p>
<p>You are reducing your function over second dimension, so it has to be in shape of <code>[3, ]</code> in your case.</p>
<aside class="quote no-group" data-username="cruvadom" data-post="1" data-topic="102113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cruvadom/48/30786_2.png" class="avatar"> cruvadom:</div>
<blockquote>
<p>dim=1</p>
</blockquote>
</aside>
<p>Also, it was just a coincidence that the returned <code>groups</code> is same as <code>groups</code> for just the first row!<br>
When you set the <code>dim=1</code>, your function will be applied on given dimension rather than whole tensor as a flattened array. So, set <code>dim=None</code>.</p>
<p>You can run the following code in 4 cases,</p>
<ul>
<li>with <code>aa=aa1</code> and <code>dim=None</code></li>
<li>With <code>aa=aa1</code> and <code>dim=1</code></li>
<li>with <code>aa=aa2</code> and <code>dim=None</code></li>
<li>With <code>aa=aa2</code> and <code>dim=1</code></li>
</ul>
<pre><code class="lang-auto">aa1 = torch.tensor([[1, 0, 0],
                   [0, 0, 1]])
aa2 = torch.tensor([[1, 1, 1],
                   [0, 1, 1]])
aa = aa1  # change this to aa2 with dim=1 and dim=None and see the difference
print(aa)
unq, groups = aa.unique_consecutive(dim=1, return_inverse=True)
print(groups)
unq, groups = aa.unique_consecutive(dim=None, return_inverse=True)
print(groups)
</code></pre>
<p>Bests</p>
Show grid of how often a given situation occurs

ErikVats: <p>Hello,</p>
<p>I am trying to predict injury time in a football match, but my results are not too good.</p>
<p>I want to get a matrix/grid/plot where declared minutes of injury time is on the x-axis and predicted minutes of injury time is on the y-axis. In each position I want to know the precentage of this outcome. So, in (1,1) I want the precentage of how often my model predicts 1 minute if 1 minute is declared, in (1,2) I want to know how often 2 minutes is predicted, but only 1 minute is declared, and so on. There is a name for this, but I can’t remember.</p>
<p>Hopefully somebody can help me implement this in pytorch.</p>
<p>Thank you in advance!</p>

ptrblck: <p>It sounds as if you are looking to implement a <a href="https://en.wikipedia.org/wiki/Confusion_matrix">confusion matrix</a>?</p>

ErikVats: <p>Yes, I would like to implement a confusion matrix!</p>

ptrblck: <p>In that case you could use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html"><code>sklearn.metrics.confusion_matrix</code></a> or implement it directly in PyTorch (we have some implementations in this forum, e.g. <a href="https://discuss.pytorch.org/t/indexing-with-byte-tensor-into-another-tensor/38457/12">here</a>).</p>
Ask for simplified Implementation of tf.gather_nd in PyTorch

Hao_Meng: <p>Hi， everyone.<br>
I am now trying to implement tf.gather_nd using Pytorch.<br>
I have tried several methods discussed here, but none of them could meet my need.<br>
Here I give my code which has been proven to able to work,but I want someone could gvie me a more simplified code using pytorch.<br>
Code are showed below</p>
<pre><code class="lang-auto">import torch

uv = torch.ones(42).view([-1, 2]).long()
print("uv=", uv.shape)

dmap = torch.rand([1, 32, 32, 21, 3])[0]
print("dmap.shape", dmap.shape)



delta = torch.zeros((21, 3)).float().cuda()
for j in range(21):
    uv_j = uv[j].long()
    dmap_j = dmap[:, :, j, :].squeeze()
    # print("dmap_j.shape=", dmap_j.shape)
    # print(uv_j[0].item(), uv_j[1].item())
    delta[j] = dmap_j[uv_j[0].item(), uv_j[1].item()]

print("delta.shape=", delta.shape)
print("delta=", delta)
</code></pre>
<p>In tensorflow it is really simple:<br>
<code>delta = tf.gather_nd(     tf.transpose(dmap, [0, 3, 1, 2, 4]), uv, batch_dims=2 )[0]</code></p>
<p>So would developers could  solve this issue or did I miss something?</p>