akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there a good way to do this kind of label operation?

username: 111482
cooked: <p>Hello.</p>
<p>Suppose I have a tensor of class labels of arbitrary shape (suppose it’s 2-dimensional, i.e., H x W)</p>
<p>I want to get the masks (corresponding positions) of each class as below.</p>
<pre><code class="lang-auto"># C = 3; H = 4; W = 4;
L = torch.LongTensor([[1,1,0,2],
                      [2,0,1,0],
                      [0,0,0,1],
                      [2,1,1,2]])
m0 = L.eq(0)
m1 = L.eq(1)
m2 = L.eq(2)

print(m0)
&gt;&gt;&gt; tensor([[0, 0, 1, 0],
            [0, 1, 0, 1],
            [1, 1, 1, 0],
            [0, 0, 0, 0]])

print(m1)
&gt;&gt;&gt; tensor([[1, 1, 0, 0],
            [0, 0, 1, 0],
            [0, 0, 0, 1],
            [0, 1, 1, 0]])

print(m2)
&gt;&gt;&gt; tensor([[0, 0, 0, 1],
            [1, 0, 0, 0],
            [0, 0, 0, 0],
            [1, 0, 0, 1]])
</code></pre>
<p>Currently, there are two ways that I can do this:</p>
<pre><code class="lang-auto">#1. use a for loop : memory efficient, but slow
for c in range(C):
    m = L.eq(c)     # m.shape: [H,W]
    do_something(m)
</code></pre>
<pre><code class="lang-auto">#2. fast, but huge memory consumption
classes = torch.arange(C).view(C,1,1)
print(classes)
&gt;&gt;&gt; tensor([[[0]],

            [[1]],

            [[2]]])

sparse_mask = L.eq(classes)  # a sparse mask of shape [C,H,W]
do_something(sparse_mask)
</code></pre>
<p>The problem I am dealing with has : H=1,000, W=10,000, C=5,000.</p>
<p>Using <span class="hashtag">#1</span> is very slow since it has to loop the <strong>do_something</strong> 5,000 times.</p>
<p>On the other hand, <span class="hashtag">#2</span> is almost prohibitive due to the memory consumption.</p>
<p>Is there any good solution for this?</p>
<p><em>p.s.: While writing this, I noticed that there might be a way to exploit the sparsity of the mask in <span class="hashtag">#2</span>, but I do not have much knowledge about using sparse operations…</em></p>
Updating Dataset Class Variables from Multiple Process

username: tommyc
cooked: <p>I am writing my own torch.utils.data.Dataset object, which has a variable that would be updated as the Dataset keeps loading data entries. For example, considering an example that I have 10 entries in my entire dataset and I have a list of counters (int’s) which keeps track of how many times each of the data entry is loaded during the training process (this is just an example illustrating the issue of interest). The Dataset object would probably be something like this:</p>
<pre><code class="lang-auto">class MyDataset(Dataset) :
    def __init__(self, X, y) :
        # X: list of inputs, say, with shape (10, 3, H, W) if each entry is an RGB image
        # y: list of outputs, say, with shape (10, 1) if each label is just one integer class label
        
        # Counters to keep track of how many times each item is used.
        self.counters = np.zeros(10)

        #....

    def  __getitem__(self, i):
         # Mark entry "i" being used here once
         self.counters[i] = self.counters[i]

         # other steps to load and process the data
</code></pre>
<p>I believe this would work when this only run on one worker/process (sorry that I have not done a thorough experiment on this). Meanwhile, as one may suspect, this would cause some problems when this Dataset object is run with multiple workers/processes, as the same class variable would be updated from multiple processes at the same time.</p>
<p>So I am wondering if there would be some mechanism in PyTorch that would help modifying class variables of torch.utils.data.Dataset objects when multiple workers/processes are enabled?</p>
<p>Thank you.</p>
Pytorch - Combined derivative with respect two parameters

username: jamesboom
cooked: <p>I am interested in combined derivatives using Pytorch: <img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/baed57059aa42984cd680344241d60383e5a1712.png" alt="image" data-base62-sha1="qFDj5MJ62Zoeufwtm22H5kZcO6C" width="145" height="92"></p>
<p>In the implemented code below, I have tried, but the code compute two partial derivative (e.g. it computed firstly d’f/d’x and secondly d’f/d’y). Is it possible modify the code in some way that we can compute this derivative with respect two parameters?</p>
<pre><code class="lang-auto">import torch
def function(x,y):
    f = x**3+y**3
    return f

a =  torch.tensor([4., 5., 6.], requires_grad=True)
b =  torch.tensor([1., 2., 6.], requires_grad=True)
derivative = torch.autograd.functional.jacobian(function, (a,b))
print(derivative)
</code></pre>
How could I use nn.DataParallel to run with different sizes?

username: coincheung
cooked: <p>Hi,</p>
<p>I need to run inference of image recognition models. I have 8gpus, and I would like to run one image per gpu each iteration, because I do not want to crop the images.<br>
I hope to use nn.DataParallel because everything is in a same process rather than muthi-processing. ( I need to gather the result and save it to disk).<br>
The problem is that images have different sizes, and nn.DataParallel seems not to work with this. (because I use dataloader which cannot collate images with different sizes).</p>
<p>I do not use DistributedDataParallel, because my results are actually embedding vectors, and I have plenty of images whose embeddings would make gpu out of memory. If I have only one process, I can copy these results to host memory at each iteration. But the multi-processing does not support gather results on different processes if they lie in host memory.</p>
<p>How could I solve this problem please?</p>
Make_layer method in resnet

username: Mona_Jalal
cooked: <p>I’m having hard time to completely understand the make_layer method here. Could someone please help me with a bit more clarification?</p>
<pre><code class="lang-auto">def _make_layer(block,inplanes,planes, blocks, stride=1):
    downsample = None
    if stride != 1 or inplanes != planes:
        downsample = nn.Sequential(
            nn.Conv2d(inplanes, planes, 1, stride, bias=False),
            nn.BatchNorm2d(planes),
        )
    layers = []
    layers.append(block(inplanes, planes, stride, downsample))
    inplanes = planes
    for _ in range(1, blocks):
        layers.append(block(inplanes, planes))
    return nn.Sequential(*layers)
</code></pre>
<p><a href="https://jarvislabs.ai/blogs/resnet" class="onebox" target="_blank" rel="noopener nofollow ugc">https://jarvislabs.ai/blogs/resnet</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40ee233897376c3a855b897dffc72e4de4df6409.png" data-download-href="https://discuss.pytorch.org/uploads/default/40ee233897376c3a855b897dffc72e4de4df6409" title="Screen Shot 2021-08-27 at 12.15.08 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40ee233897376c3a855b897dffc72e4de4df6409_2_690x376.png" alt="Screen Shot 2021-08-27 at 12.15.08 AM" data-base62-sha1="9goLsGcwdpMS0efkcO23eZIS1n3" width="690" height="376" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40ee233897376c3a855b897dffc72e4de4df6409_2_690x376.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40ee233897376c3a855b897dffc72e4de4df6409_2_1035x564.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40ee233897376c3a855b897dffc72e4de4df6409_2_1380x752.png 2x" data-dominant-color="EFF0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-27 at 12.15.08 AM</span><span class="informations">1884×1028 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is this overfitting..?

username: jhp
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c62c835fecfc29dca1f95918376a43c17cfe2f0c.png" data-download-href="https://discuss.pytorch.org/uploads/default/c62c835fecfc29dca1f95918376a43c17cfe2f0c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c62c835fecfc29dca1f95918376a43c17cfe2f0c_2_690x257.png" alt="image" data-base62-sha1="sh7Uj85f6o63ka2axWZqY2uHa2w" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c62c835fecfc29dca1f95918376a43c17cfe2f0c_2_690x257.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c62c835fecfc29dca1f95918376a43c17cfe2f0c_2_1035x385.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c62c835fecfc29dca1f95918376a43c17cfe2f0c_2_1380x514.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2505×934 96.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi I have a quick question about overfitting.<br>
In the graph above, yellow is baseline, and what I changed after modifying the model is gray. If you look at it, my model(gray) is very good at first, but it catches up and gets bad results. Can we call this overfitting?<br>
I don’t think the gray one is learning wrong. (perhaps not <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> ) How can I change the training schedule? Is there any other things to consider?</p>

username: gphilip
cooked: <aside class="quote no-group" data-username="jhp" data-post="1" data-topic="130412">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jhp/48/31228_2.png" class="avatar"> jhp:</div>
<blockquote>
<p>Can we call this overfitting?</p>
</blockquote>
</aside>
<p>As per the first link that turned up when I asked Google about overfitting, overfitting “<a href="https://www.ibm.com/cloud/learn/overfitting" rel="noopener nofollow ugc">occurs when a statistical model fits exactly against its training data</a>”. If you think that this is happening in your picture, then you can call it overfitting.</p>
Rectangular Images for GAN

username: thecoderboy123
cooked: <p>Hi there. I am currently doing some Proof of Concepts for DCGAN with rectangular images. I am a completely new to DCGANs as well.</p>
<pre><code class="lang-auto">    class Discriminator(nn.Module):
        def __init__(self,ngpu):
            super(Discriminator, self).__init__()
            self.ngpu = ngpu
            self.main = nn.Sequential(
                
                nn.Conv2d(nc,ndf,2,2,1,bias=False),
                nn.LeakyReLU(0.2,inplace=True),
                
                nn.Conv2d(ndf,ndf*2,2,2,1,bias=False),
                nn.BatchNorm2d(ndf * 2),
                nn.LeakyReLU(0.2,inplace=True),
                
                nn.Conv2d(ndf*2,ndf*4,2,2,1,bias=False),
                nn.BatchNorm2d(ndf * 4),
                nn.LeakyReLU(0.2,inplace=True),
                
                nn.Conv2d(ndf*4,ndf*8,2,2,1,bias=False),
                nn.BatchNorm2d(ndf * 8),
                nn.LeakyReLU(0.2,inplace=True),
                
                nn.Conv2d(ndf*8,1,2,1,0,bias=False),
                nn.Sigmoid()
            )
        def forward(self,input):
            return self.main(input)
</code></pre>
<p>Currently, this is the code to my Discriminator, where nc=3 and ndf = 64. My images are in 72 (width) x 14 (height). I have also modified to only use 2x2 as the kernel size.</p>
<p>When I am training the Discriminator, I realised one problem that has appeared is there is an error being thrown when I am calculating the loss. The loss used is BCELoss().</p>
<pre><code class="lang-auto">real_cpu = data[0].to(device)
            b_size = real_cpu.size(0) #batch size
            #batch size 64, thus it prints out 64 labels. make sense
            label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
            output = netD(real_cpu).view(-1)
            print(real_cpu.size())
            print(netD(real_cpu).size())
            print(output)
            print(netD(netG(torch.randn(b_size,nz,1,1))).size())
            errD_real = criterion(output, label)
</code></pre>
<p>Based on the code above, the variable ‘output’ throws a size of 320, where it is supposed to show only 64. Is it possible to get some advice on this issue? Would appreciate if I could get some help. Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch 1.9 with CUDA 11.0?

username: turian
cooked: <p>I am using Google GCP GPUs, and it appears the only machine image they provide is CUDA 11.0 (!). Only pytorch &lt;= 1.7 supports CUDA 11.0.</p>
<p>I am creating a Dockerfile for my project. However, some of my library’s dependencies want pytorch 1.9, so they upgrade from pytorch 1.7 GPU version to pytorch 1.9 CPU version.</p>
<p>I think Pytorch 1.9 is a must. But then I am not sure what workaround is the least painful:</p>
<ol>
<li>Can I use CUDA 10.2 in the Docker, even though the bare metal system is CUDA 11.0? Or will that cause problems. I have seen conflicting advice on this.</li>
<li>I could try to build my own Google machine image. This seems very painful tho.</li>
<li>I could try to build pytorch 1.9 from scratch in my Docker and use CUDA 11.0 in my Docker. I haven’t found good Dockerfiles explaining how to do this.</li>
</ol>
<p>Any other suggestions?</p>

username: nbingo
cooked: <p>I don’t have an exact answer to your question, but I tried something similar as in your third option here: <a href="https://discuss.pytorch.org/t/torch-cuda-unknown-error-but-cuda-and-nvidia-smi-properly-installed-on-azure-k8s-service/130177">Torch CUDA unknown error, but CUDA and nvidia-smi properly installed on Azure K8s Service - PyTorch Forums</a>. I’m using Microsoft Azure instead of Google cloud, but same principle for building a docker image. The example I have there correctly installs CUDA and <code>nvidia-smi</code>, so that CUDA 11.4 is detected, but unfortunately pytorch can’t initialize CUDA properly for some reason, so I’d also be interested if you can find a solution using your third option.</p>

username: ptrblck
cooked: <ol>
<li>Yes, you should be able to use a CUDA10.2 docker container, as its driver requirement would be met by the newer CUDA11 driver.</li>
<li>I don’t know what a Google machine image is and how hard it would be to build it.</li>
<li>You could reuse the <a href="https://github.com/pytorch/pytorch/blob/0bd8d0951dcb4063c0f7552a7404bd7f0e7b6e6f/Dockerfile">Dockerfile</a> from the PyTorch repository.</li>
</ol>
<p>Also, did you try to install the CUDA11.1 Pytorch 1.9.0 binaries? I don’t know which driver is installed on the node, but the CUDA enhanced compatibility might be used.</p>
Flatten trainingsdata in batch

username: NPC
cooked: <p>Hi,<br>
I am sorry to bother you but hope you can help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .<br>
Yesterday I started working with libtorch (I want to increase the speed of a model a bit) and started with a simple toy model. Now I would like to get the MNIST data as a 1D not a 2D tensor. Unfortunatly, I do not understand how the libtorch transformation works (and I didnt find any good documentation) and thus I am not able to make the train data flat.<br>
I am currently using this:</p>
<pre><code class="lang-auto">auto data_loader = torch::data::make_data_loader(
            torch::data::datasets::MNIST(R"(...\MNIST\raw)").map(torch::data::transforms::Stack&lt;&gt;()), 64);
</code></pre>
<p>to get the training data as {64,1,28,28} but I want it as {64, 784} instead. How can I modify the above to get the result I want?</p>
<p>Best regards<br>
NPC</p>

username: NPC
cooked: <p>Hi,<br>
just in case anybody else is a little bit lost:</p>
<pre><code class="lang-auto">at::Tensor flatter(at::Tensor tensor_){
    return tensor_[0].flatten();
}
</code></pre>
<pre><code class="lang-auto">auto data_loader = torch::data::make_data_loader(
            torch::data::datasets::MNIST(PATH).map(
                    torch::data::transforms::TensorLambda&lt;&gt;(flatter)).map(
                    torch::data::transforms::Stack&lt;&gt;()), 64);
</code></pre>
Dataloader throws random error

username: Joel_fre
cooked: <p>File “C:\Users\joelf_000\anaconda3\envs\hdr_env\lib\site-packages\tqdm\std.py”, line 1185, in <strong>iter</strong><br>
for obj in iterable:<br>
File “C:\Users\joelf_000\anaconda3\envs\hdr_env\lib\site-packages\torch\utils\data\dataloader.py”, line 521, in <strong>next</strong><br>
data = self._next_data()<br>
File “C:\Users\joelf_000\anaconda3\envs\hdr_env\lib\site-packages\torch\utils\data\dataloader.py”, line 1183, in _next_data<br>
return self._process_data(data)<br>
File “C:\Users\joelf_000\anaconda3\envs\hdr_env\lib\site-packages\torch\utils\data\dataloader.py”, line 1229, in _process_data<br>
data.reraise()<br>
File “C:\Users\joelf_000\anaconda3\envs\hdr_env\lib\site-packages\torch_utils.py”, line 425, in reraise<br>
raise self.exc_type(msg)<br>
TypeError: <strong>init</strong>() missing 1 required positional argument: ‘dtype’</p>
<p>My dataloader is randomly throwing this error while training. I have no idea why. Im running kinda low on ram but it shouldnt be the problem since the error appears randomly (sometimes during the first epoch, other times after 10+ epochs). Any help would be appreciated.</p>
Debugging "CUDA out of memory"

username: Roman27
cooked: <p>Hello everyone.  I suspect I coded something wrong because I am using a Tesla P100-PCIE-16 B from colab and the tensors i am generating are not that big (10000x100x100). So I think somewhere something is accumulating. I trimmed the code so a lot of variables are not defined but I don’t think that matters much</p>
<pre><code class="lang-auto">    for j in range(100):
        
        f_old=f[...,-1]
        ...
        X=# This input is of dimension 10000x 100 x102
        sigma=net(X)

        alpha= torch.sum(sigma * torch.cumsum(sigma *dt,1),-1)
        W=torch.randn(10000,1).unsqueeze(1)*torch.sqrt(torch.tensor(dt))
        W=W.to(device)
        f_new=f_old[:,j:] +  (alpha *dt + torch.sum(sigma*W,-1))
        f_new=torch.cat([torch.zeros(MC,j,device=device),f_new],1)
        f[...,j+1]=f_new   
        del X , W, f_new,f_old, x_mat, x_input_t, x_input_2 , alpha, sigma
</code></pre>
<p>I get the error message around j=50. Any Idea where the problem with my code is?</p>

username: ptrblck
cooked: <p>Depending on the used model the input might be alright or huge. Given that you are able to use ~50 iterations before the OOM is raised, I would recommend to check, if the memory usage is increasing inside the loop. If that’s the case you might (accidentally) store tensors attached to the computation graph and could try to isolate it by calling <code>detach()</code> on tensors.</p>
The confict of multipy target box

username: sunshine1
cooked: <p>The code is as follows</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>tbox[ bi,gj,gi,ai]=s_target[:,2:6]</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>it means that the anchor located in (bi,gj,gi,ai) in prediction layer is responsible for predicting one target box. In some case where the (bi,gj,gi, ai) of at least two target box are equal, how to process the conflict?<br>
Does the next target box overlap the previous one and the previous one is ignored?</p>

username: gphilip
cooked: <p>Your post is hard to read because of the way you have formatted the code. Could you enclose code between two lines, where each line has just three consecutive back-ticks on it? The back-tick is this character: ` . It may be on the same key as the tilde, ~ . It is <strong>not</strong> the same as the single quote (’) or the double quote ("). (You can just copy-paste the above back-tick, if you can’t find it on your keyboard.)</p>
<p>So the formatting will look like this:</p>
<p>line1 with just three back-ticks one after the other</p>
<p>your code goes here</p>
<p>line2 with just three back-ticks one after the other</p>
AttributeError: 'BucketIterator' object has no attribute 'i'

username: muhammadfhadli1453
cooked: <p>I’m doing seq2seq machine translation on my own dataset. I have preproceed my dataset using this code. The problem comes when i tried to split <em>train_data</em> using BucketIterator.split()<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/460d21c7f7a155c54a2fea3721ce934c69fd4613.png" data-download-href="https://discuss.pytorch.org/uploads/default/460d21c7f7a155c54a2fea3721ce934c69fd4613" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/460d21c7f7a155c54a2fea3721ce934c69fd4613.png" alt="image" data-base62-sha1="9ZHyB1WshhECLYj73gw5a7aAYqT" width="503" height="500" data-dominant-color="F4F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">741×736 27 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am very confuse, because i don’t know what <em>key</em> i should use for train iterator. Thank you for your help</p>

username: gphilip
cooked: <p>Try changing <code>print(data.i)</code> to <code>print(data)</code>. That should give you some ideas on the names involved.</p>

username: muhammadfhadli1453
cooked: <p><a class="mention" href="/u/gphilip">@gphilip</a> Thank you for your response. I have tried, but it makes me more confuse<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ece07a3b54602784c66fc910e802d57a3990247.jpeg" alt="image" data-base62-sha1="mEQR9casLvYLCGtfPs888hBf1zh" width="465" height="316"></p>

username: gphilip
cooked: <p>Asking Google about “torchtext BucketIterator” threw up <a href="https://gmihaila.github.io/tutorial_notebooks/pytorchtext_bucketiterator/" rel="noopener nofollow ugc">this tutorial</a>. You may find it useful.</p>
Changing dataset during training, adapting len

username: Btorch
cooked: <p>Hi, we would like to  go easy with our model and show him easy samples for the first few epcohs and in every few epochs to increase the hard samples.<br>
We code the solution by</p>
<pre><code class="lang-auto">class ImageFolderNew(ImageFolder):
    def __len__(self) -&gt; int:
        return len(self.filtered_imgs)
   
    def filtered_images(self,p=0.1):
        self.filtered_imgs = {}
        for k in range(len(self.imgs)):
            path = self.imgs[k][0]
            if condition not fulfils:
                self.filtered_imgs[k] = self.imgs[k]
    def __getitem__(self, index):
           #Here we work on filtered_imgs not self.imgs!!
            path = self.filtered_imgs[index][0]#imgs[index][0]
</code></pre>
<p>This solution works well, the only issue is than <em>getitem</em>_ stil samles from (0,len(self.imgs)) and not from (0,len(self.filtered_imgs)), how can we make <strong>getitem</strong> understand that we have less samples in this epoch, but in the next epoch we will have more.</p>
<p>I’ll be happy for your help!<br>
Thanks!!</p>

username: PURUSHOTTAM_BHARDWAJ
cooked: <p>One solution can be <code>torch.utils.data.IterableDataset</code>   where the length of datasets and batch size can be controlled by custom user-defined function in dataset class.<br>
<a href="https://pytorch.org/docs/stable/data.html#iterable-style-datasets" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#iterable-style-datasets</a></p>
Error when exporting TorchScript model to ONNX - "RuntimeError: module contains attributes values that overlaps"

username: alegasalv
cooked: <aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/68320251/error-when-exporting-torchscript-model-to-onnx-runtimeerror-module-contains">
  <header class="source">

      <a href="https://stackoverflow.com/questions/68320251/error-when-exporting-torchscript-model-to-onnx-runtimeerror-module-contains" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2998665/alegasalv" target="_blank" rel="noopener nofollow ugc">
    <img alt="alegasalv" src="https://www.gravatar.com/avatar/e14d3460cd24d3a6b064f487b05d87c7?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/68320251/error-when-exporting-torchscript-model-to-onnx-runtimeerror-module-contains" target="_blank" rel="noopener nofollow ugc">Error when exporting TorchScript model to ONNX - "RuntimeError: module contains attributes values that overlaps"</a>
</h4>

<div class="tags">
  <strong>pytorch, onnx, torchscript</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2998665/alegasalv" target="_blank" rel="noopener nofollow ugc">
    alegasalv
  </a>
  on <a href="https://stackoverflow.com/questions/68320251/error-when-exporting-torchscript-model-to-onnx-runtimeerror-module-contains" target="_blank" rel="noopener nofollow ugc">04:44PM - 09 Jul 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>Could you edit the title and add more information to the post, as the StackOverflow link is dead?</p>

username: alegasalv
cooked: <p>Sorry, I missed copy it</p>

username: alegasalv
cooked: <p>Hello, can anyone help me with this?</p>
Index tensor with mask that have variable number of True values per batch

username: aleSuglia
cooked: <p>Hi there,</p>
<p>I would like to index a tensor using a mask that contains a variable number of True elements for each batch row. Particularly, imagine you are in the following situation:</p>
<pre><code class="lang-auto">x = torch.randn((2, 5, 768))
mask = torch.tensor([[False, True, False, False, False], [False, True, False, False, True]])
</code></pre>
<p>I would like to create a function that extracts the indexes with <code>True</code> from <code>x</code> and then makes sure that the output is stored in a tensor that preserves the batch dimension. Particularly, I would like an output <code>y</code> that has size <code>(batch_size, 2, 768)</code> (in this case is 2 because the maximum number of True values in a row is 2).</p>
<p>In the easiest case where the two batch elements have the same number of True values, I can simply do <code>x[mask]</code> and then reshape the tensor. But if they are not, it will require some sort of padding?</p>
How to convert torch::nn::Module to torch::jit::Module with forward

username: beichen2012
cooked: <p>how to convert torch::nn::Module to torch::jit::Module with forward?<br>
Using torch::nn::Module::save, not compile the forward to the scriptmodule, just the weights(buffers, parameters) and the net structure.</p>

username: beichen2012
cooked: <p>AttributeError: ‘RecursiveScriptModule’ object has no attribute ‘forward’</p>
DataLoader worker exited unexpectedly (pid(s) 49965, 49966, 49967)

username: aidan.goetzinger
cooked: <p>When running my code I recieved this error message “<em>RuntimeError: DataLoader worker (pid(s) 49965, 49966, 49967) exited unexpectedly</em>” I am completely unsure where to begin to solve this issue. Any guidance at all would be greatly appreciated. Code and traceback posted below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65806306040655e493ed0e5c76d7592f1e9082c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/65806306040655e493ed0e5c76d7592f1e9082c6" title="Screen Shot 2021-08-25 at 8.47.23 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65806306040655e493ed0e5c76d7592f1e9082c6_2_578x500.png" alt="Screen Shot 2021-08-25 at 8.47.23 PM" data-base62-sha1="etVi6oZSOLEmIEkFn6lbXBnuiP4" width="578" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65806306040655e493ed0e5c76d7592f1e9082c6_2_578x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65806306040655e493ed0e5c76d7592f1e9082c6_2_867x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65806306040655e493ed0e5c76d7592f1e9082c6_2_1156x1000.png 2x" data-dominant-color="232525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-25 at 8.47.23 PM</span><span class="informations">1326×1146 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23a05c7613d78c6ca4436b210d44796d55a3a228.png" data-download-href="https://discuss.pytorch.org/uploads/default/23a05c7613d78c6ca4436b210d44796d55a3a228" title="Screen Shot 2021-08-25 at 8.49.48 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23a05c7613d78c6ca4436b210d44796d55a3a228.png" alt="Screen Shot 2021-08-25 at 8.49.48 PM" data-base62-sha1="55ahAycwjge6BZj6MLUuN6n8nBC" width="690" height="162" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-25 at 8.49.48 PM</span><span class="informations">1761×416 46.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Rerun your code with <code>num_workers=0</code> and check, if an entire epoch can be loaded properly.<br>
If that’s the case then the usage of multiprocessing might be causing some issues, e.g. if not enough shared memory is available etc.</p>
Propagating loss for simultaneous neural nets?

username: rrm39
cooked: <p>I am trying to create a script with nested neural nets, and I’m wondering how to ensure that the losses for each net are propagated to the right place.</p>
<p>As an overall structure to my code: I define an autoencoder net A, then train it for 100 epochs. Every 10 epochs, I evaluate the partially trained autoencoder net A on a data sample, then train the encoded data on a binary classifer net B.</p>
<p>I am worried that there is some interference between nets A and B. When I plot the loss v. epochs curve for net A, the curve gets a lot noisier when I train net B for a larger number of epochs, so I do think there is some interference going on there.</p>
<p>For A, I have:</p>
<pre><code class="lang-auto">xdevice = torch.device( "cuda" if torch.cuda.is_available() else "cpu" )
A.to( device )
</code></pre>
<p>then all the data for A is also sent to the device. For B, I do not send the net or the data to the device. But, since loss is propagated only with <code>loss.backward()</code>, I’m worried that the loss I get when training classifier B is getting backwarded to net A.</p>
<p>How can I control where the loss is sent? Do I need to define 2 devices for training?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="rrm39" data-post="1" data-topic="130313">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rrm39/48/41644_2.png" class="avatar"> rrm39:</div>
<blockquote>
<p>I’m worried that the loss I get when training classifier B is getting backwarded to net A.</p>
</blockquote>
</aside>
<p>This would be the case if you pass the output of <code>netA</code> to <code>netB</code> directly since both models are treated as any other <code>nn.Module</code> (e.g. <code>nn.Conv2d</code>).<br>
If you want to train <code>netB</code> alone without calculating the gradients in <code>netA</code> and updating its parameters, you can detach <code>netA</code>’s output via:</p>
<pre data-code-wrap="python"><code class="lang-python">out = netA(input)
out = netB(out.detach())
loss = criterion(out, target)
loss.backward() # will calculate the gradients in netB only
</code></pre>
Propagating a vector backwards

username: a_d
cooked: <p>Hello  Everyone,</p>
<p>I have a model which has to predict 2 things, and therefore My output is of shape <code>[batch, 2]</code>. My Target too, is of shape <code>[batch, 2]</code>. I calculate the loss as follows -</p>
<pre><code class="lang-auto">    dCoordinates = torch.abs(output - coordinates)
    loss = torch.sum(dCoordinates, 1)
</code></pre>
<p>Therefore loss is of shape <code>[batch]</code>. How do I propagate this loss backward?</p>
<p>TIA</p>

username: gphilip
cooked: <p>Does</p>
<pre><code class="lang-auto">loss.backward(torch.ones_like(loss))
</code></pre>
<p>do what you want?</p>

username: a_d
cooked: <p>thank you for your reply <a class="mention" href="/u/gphilip">@gphilip</a></p>
<p>Suppose the batch size if 4. Then the loss would be something like [0,443, 0,54, 0,21, 0,534]. Each loss would correspond to the respective inputs in the batch. I want to propagate the loss backward to the network for each input.</p>
<p>I do not know how to do that, and I do not know whether what you suggested would do that.<br>
Does it perform the same thing ?</p>
Few shot learning

username: Arthur_Zakirov
cooked: <p>Hello everyone,</p>
<p>I’m trying to implement a training method, which trains the model on dataset A first and then continues the training on dataset B. Both datasets should be shuffled independently of each other. There is a class called torch.data.utils.ConcatDataset which enables the combination of 2 datasets however as far as I know it has no shuffling. The DataLoader class can perform the shuffle, but it will mix up the 2 datasets. So how do I implement the shuffling pre concatenation?</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Probably not the cleanest approach, but you could create the indices for both datasets first, shuffle them, wrap both datasets into a <code>ConcatDataset</code>, and use a <code>Subset</code> with the shuffled indices afterwards.<br>
Something like this might work:</p>
<pre><code class="lang-python"># create datasets
dataset1 = torch.utils.data.TensorDataset(torch.arange(10))
dataset2 = torch.utils.data.TensorDataset(torch.arange(10, 20))
dataset = torch.utils.data.ConcatDataset((dataset1, dataset2))

# get indices
idx1 = torch.arange(len(dataset1))
idx2 = len(idx1) + torch.arange(len(dataset2))

# shuffle
idx1 = idx1[torch.randperm(len(idx1))]
idx2 = idx2[torch.randperm(len(idx2))]
idx = torch.cat((idx1, idx2))

# use shuffled indices to create Subset
dataset_shuffled = torch.utils.data.Subset(dataset, idx)

loader = torch.utils.data.DataLoader(
    dataset_shuffled, shuffle=False, batch_size=2)

for data in loader:
    print(data)
</code></pre>
Using forward hooks vs Modifying the forward function

username: Chirag
cooked: <p>Hi,</p>
<p>When we need to modify input to and output from forward function of a layer, I can think of two ways:</p>
<ol>
<li>Add forward_pre and forward hooks</li>
<li>Modify the forward function of the layer itself.</li>
</ol>
<p>Below are simple illustration of each of these.</p>
<p>In my opinion the two are exactly equivalent. Anything that can be done using 1 can be done using 2 and vice versa.</p>
<p>My question: Is this understanding correct? Is there are any reason to prefer one over the other? Either for performance or as general good practice.</p>
<p>Demo:</p>
<ol>
<li>Using hooks</li>
</ol>
<pre><code class="lang-auto">from torch import nn

################### Using forward hooks #########################

## Setup
conv1 = nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2).to('cuda:0')
inp = torch.rand((1, 3, 299, 299)).to('cuda:0')

## pre-forward hook
def print_pre(module, input):
    print("I am about to start the forward run!")
# # post-forward hook  
def print_post(module ,input, output):
    print("I am done with the forward")

## Register the hooks
conv1.register_forward_pre_hook(print_pre)
conv1.register_forward_hook(print_post)

## forward run
out1 = conv1(inp)
</code></pre>
<ol start="2">
<li>Using wrapper</li>
</ol>
<pre><code class="lang-auto">############### Using forward Wrapper #################
from torch import nn

conv2 = nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2).to('cuda:0')
inp = torch.rand((1, 3, 299, 299)).to('cuda:0')

## save original forward function
original_forward = conv2.forward

## define a wrapper around the original forward function
def modified_forward(module, *input):
    print("I am about to start the forward run!")
    out = original_forward(module,*input)
    print("I am done with the forward")
    return out

## set wrapper as the new forward 
conv2.forward = modified_forward

## forward run
out2 = conv2(inp)
</code></pre>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Yes, you are correct that both approaches should yield the same results.<br>
Using hooks could be easier in case you want to reuse a predefined model and don’t want to override the <code>forward</code> method (e.g <code>torchvision.models.resnet50()</code>), but this also depends on your personal coding style. E.g. I would prefer to override the <code>forward</code> method in case I really want to manipulate the forward pass and not only use the hooks for debugging or a quick testing.</p>
<p>Additionally, I think hooks are not supported in scripted models (feature request tracked <a href="https://github.com/pytorch/pytorch/issues/34329">here</a>) so in case you would like to script the model in the future, I would also override the <code>forward</code> method.</p>

username: Chirag
cooked: <p>Thanks you <a class="mention" href="/u/ptrblck">@ptrblck</a> ! That makes sense.</p>
Normalized embeddings from LSTM

username: arijitthegame
cooked: <p>Hi all,</p>
<p>I am training a simple LSTM on Language Modeling task on the Penn Tree Bank. I have tied the weights of the encoder and the decoder.</p>
<p>However the caveat is during training I would like the embeddings to have unit L^2 norm. The following is my code but the losses do not decrease during training,</p>
<p>‘’’<br>
class RNNModel(nn.Module):<br>
“”“Container module with an encoder, a recurrent module, and a decoder.”""</p>
<pre><code>def __init__(self, ntoken, ninp, nhid, nlayers, dropout=0.2):
    super(RNNModel, self).__init__()
    self.drop = nn.Dropout(dropout)
    self.encoder = nn.Embedding(ntoken, ninp) # Token2Embeddings
   
    self.rnn = nn.LSTM(ninp, ninp, nlayers, dropout=dropout)
    
    self.decoder = nn.Linear(nhid, ntoken, bias=False)

    # Optionally tie weights as in:
    # "Using the Output Embedding to Improve Language Models" (Press &amp; Wolf 2016)
    # https://arxiv.org/abs/1608.05859
    # and
    # "Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling" (Inan et al. 2016)
    # https://arxiv.org/abs/1611.01462

    self.decoder.weight = self.encoder.weight

    self.init_weights()

    
    self.nhid = nhid
    self.nlayers = nlayers

def init_weights(self):
    initrange = 0.1
    nn.init.uniform_(self.encoder.weight, -initrange, initrange)
    nn.init.zeros_(self.decoder.weight)
    nn.init.uniform_(self.decoder.weight, -initrange, initrange)
   

def forward(self, input, hidden):
    self.encoder.weight.data = F.normalize(self.encoder.weight.data, p=2, dim=1)
    emb = self.drop(self.encoder(input))
    emb = F.normalize(emb, p=2, dim=1)
    output, hidden = self.rnn(emb, hidden)
    output = self.drop(output)
    output = F.normalize(output.view(output.size(0)*output.size(1), output.size(2)), p=2, dim=1)
    decoded = self.decoder(output) #want here the dot product of unit length vectors
    return decoded, hidden

def init_hidden(self, bsz):
    weight = next(self.parameters()).data
   
    return (Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()),
                Variable(weight.new(self.nlayers, bsz, self.nhid).zero_()))
</code></pre>
<p>‘’’<br>
To explain my point further: when the decoder is doing the matrix multiplication with it’s weights and the output, I would want the weights to have unit norm. With my implementation as above, the losses do not go down. Any help will be highly appreciated.</p>
Broadcast Errors with Multidimensional Masks

username: Zayd
cooked: <p>I have a 3D tensors of size <em>N</em> x <em>M</em> x <em>C</em> with <em>N</em> and <em>M</em> very large.  I want to increment various elements in the tensor.  For the first two dimensions, the indices to be updated are stored in two separate 1D tensors (potentially of different length).  As a simple example:</p>
<pre><code class="lang-auto">x = torch.zeros([5, 6, 2])
idx1 = torch.tensor([3, 2], dtype=torch.long)
idx2 = torch.tensor([0, 1, 2], dtype=torch.long)
x[idx1, idx2, 0] += 1
</code></pre>
<p>Ideally, this would update the elements in <code>x</code> at indices (3,0,0), (3,1,0), (3,2,0), (2,0,0), etc.  However the above code raises the error:</p>
<p><code>IndexError: shape mismatch: indexing tensors could not be broadcast together with shapes [2], [3]</code></p>
<p>A <a href="https://github.com/numpy/numpy/issues/13255" rel="noopener nofollow ugc">similar question</a> was previously asked and that solution works if <code>x</code> is only accessed and not modified (i.e., call <code>x[idx1][:,idx2,0]</code>).  N and M are large enough that calling <code>torch.cartesian_prod</code> to generate the indices yields out of memory errors.</p>
<p>In low-level CUDA, this feels reasonably straightforward and memory efficient to do, but I could not figure out the right Python syntax to make it work at all much less efficiently.</p>
Any plan for adding DDP support for complex tensor?

username: Hongkai_Zheng
cooked: <p>Hello folks!</p>
<p>Since automatic differentiation for complex number has been supported for a while and people are using it to train complex networks, it would be nice to add DDP support as well. Does Pytorch have plan to support DDP for complex numbers? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=10" title=":thinking:" class="emoji" alt=":thinking:"></p>
Recreating ResNet110(n=18)

username: abhinav_hari
cooked: <p>I was trying to replicate the same result of resent 110 in cifar 10 as in <a href="https://arxiv.org/abs/1512.03385" rel="noopener nofollow ugc">Deep Residual Learning for Image Recognition</a>. It says something like this for resnet having less than 110 layers</p>
<blockquote>
<p>We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations, which is determined on a 45k/5k train/val split.</p>
</blockquote>
<p>and for a resnet with 110 layers it says</p>
<blockquote>
<p>In this case, we find that the initial learning rate of 0.1 is slightly too large to start converging. So we use 0.01 to warm up the training until the training error is below 80% (about 400 iterations), and then go back to 0.1 and con- tinue training. The rest of the learning schedule is as done previously.</p>
</blockquote>
<p>How can I change learning rate according to the error ??</p>

username: yoshitomo-matsubara
cooked: <p>Hi <a class="mention" href="/u/abhinav_hari">@abhinav_hari</a></p>
<p>My guess is that they used a warming-up learning rate 0.01 only for the first 400 iterations, and then the training error was 80% as a result.<br>
For warm-up strategy, <a href="https://github.com/pytorch/vision/blob/master/references/detection/engine.py#L19-L24" rel="noopener nofollow ugc">torchvision’s example</a> code may help.</p>
<p>FYI, I reproduced the accuracy of ResNet-110 for CIFAR-10 even without the warm-up strategy, and the code + trained model weights are publicly available as part of torchdistill.<br>
See <a href="https://github.com/yoshitomo-matsubara/torchdistill/releases/tag/v0.1.1" rel="noopener nofollow ugc">this page</a> for the reproduced result. Google Colab examples to reproduce the trends are also available <a href="https://github.com/yoshitomo-matsubara/torchdistill#cifar-10-and-cifar-100" rel="noopener nofollow ugc">here</a>.</p>

username: abhinav_hari
cooked: <p><a class="mention" href="/u/yoshitomo-matsubara">@yoshitomo-matsubara</a> thanks very much. I have one more doubt, it would really help me if you can clarify it.</p>
<p>The results in the official <a href="https://arxiv.org/abs/1512.03385" rel="noopener nofollow ugc">ResNet paper</a> report the score of the model on cifar 10  in a column named <strong>error%</strong>. I am not sure how that score is calculated. My current guess is that it is the mean of of top1 accuracy += std.div of the top1 accuracy. if not so, can you guide me on how to calculate the value in the ‘error%’ column.</p>

username: yoshitomo-matsubara
cooked: <p>Hi <a class="mention" href="/u/abhinav_hari">@abhinav_hari</a> ,</p>
<p>I think the error you mentioned is standard deviation (over 5 runs) as explained in the table caption</p>
<blockquote>
<p>Table 6. Classification error on the CIFAR-10 test set. All methods are with data augmentation. For ResNet-110, we run it 5 times and show “best (mean±std)” as in [43].</p>
</blockquote>
Loss masking chamfer distance

username: lycaenidae
cooked: <p>Hi,</p>
<p>I calculate chamfer loss for different parts of the object and would like to mask out for some of the losses. I am confused a little bit. Could you help me?</p>
<p>loss_ch, _ = chamfer_distance(src_0, trg_0) * mask[0]</p>
<p>out_0: torch.Size([2, 1538, 3])<br>
trg_0: torch.Size([2, 1538, 3])<br>
mask[0] torch.Size([2]) # just a binary 0 or 1</p>
<p>batch_size = 2</p>
Memory leakage during realtime inference of pytorch_pretrained_bert's BertForSequenceClassification model

username: Ashbajawed
cooked: <p>0</p>
<p>I’m facing memory leakage in the realtime inference of pytorch_pretrained_bert’s BertForSequenceClassification model.</p>
<p>Although I’m using GPU but still CPU memory is exhausting</p>
<pre><code class="lang-auto">      with torch.no_grad():
         logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)
         logits = logits.detach().cpu().numpy()
         predictions.append(logits)
         gc.collect()
         torch.cuda.empty_cache()
         del logits
</code></pre>
<p>model, b_input_ids and b_input_mask are mapped on gpu using .to(device)</p>
<p>Can anyone suggest me what can I do… memory is increasing gradually</p>

username: basingse
cooked: <aside class="quote no-group" data-username="Ashbajawed" data-post="1" data-topic="130272">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ashbajawed/48/41621_2.png" class="avatar"> Ashbajawed:</div>
<blockquote>
<p>Although I’m using GPU but still CPU memory is exhausting</p>
</blockquote>
</aside>
<p>I see in the code you are appending the <code>logits</code> to <code>predictions</code> which will increase the cpu memory.</p>

username: Ashbajawed
cooked: <p>i have detach the graphs further predictions array is being empty after every few iterations</p>