akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Zero_grad in network

pastrana1909: <p>Imagine that I have a network class called myModel. I instantiated in the following fashion</p>
<pre><code class="lang-auto">net = mYModel()
</code></pre>
<p>In my training loop, I zero out the gradient before calling the forward pass.</p>
<pre><code class="lang-auto">for x, y in examples:
    net.zero_grad()
    loss = some_loss_function(net.forward(x), y)
    loss.backward()
    optimizer.step()
</code></pre>
<p>The previous code seems to work just fine. However, if instead of calling the forward pass, I invoke the call method, the gradients are not getting register. Look at the following code for reference:</p>
<pre><code class="lang-auto">for x, y in examples:
    net.zero_grad()
    loss = some_loss_function(net(x), y)
    loss.backward()
    optimizer.step()
</code></pre>
<p>The second code does not work and the backpropagation is happening with a zero gradient, so there is not a change in the network params. It is worth mentioning that I am training only one model and the optimizer is referenced to the network params.</p>
<p>Is this behavior intended?</p>

ptrblck: <p>That’s quite strange, as the second approach is the correct one.<br>
It’s usually not recommended to call the <code>forward</code> method directly, as it would skip all registered hooks.<br>
However, could it be the case that you are using forward hooks, which might somehow delete the gradients?<br>
If not, could you post the model definition, so that we could debug the issue?</p>
Custom layer with B-splines approximation and backpropagation

sdsgsh: <p>Hello, I am currently trying to implement a MLP with a custom layer.<br>
This custom layer takes as input a sequence of 2d points p and a parameterization t for these points<br>
and approximates the sequence of points p using B-Splines functions. The loss is then calculated as the<br>
distance between the sequence of points p and the approximation calculated by this approach.</p>
<p>Checking Scipy’s documentation I found</p>
<p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.BSpline.html#scipy.interpolate.BSpline" class="onebox" target="_blank" rel="nofollow noopener">https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.BSpline.html#scipy.interpolate.BSpline</a></p>
<p>which allows you to represent a B-spline and</p>
<p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.make_lsq_spline.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.make_lsq_spline.html</a></p>
<p>usable to approximate a sequence of points provided with parameterization using B-splines.</p>
<p>So I was wondering in what way I could implement this layer in PyTorch in order to correctly perform<br>
backpropagation. It is possible to somehow reuse the methods in the links above that use Numpy arrays<br>
or is it necessary to implement these operations using PyTorch Tensors?</p>

William_Ashbee: <p>did you ever find a solution to this</p>

FA_mn: <p>Hello,</p>
<p>I know there is <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/conv/spline_conv.html" rel="noopener nofollow ugc">The spline-based convolutional operator</a>.<br>
Related to the paper <a href="https://arxiv.org/abs/1711.08920" rel="noopener nofollow ugc">“SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels”</a><br>
I hope it works for you.</p>
Enable and disable pytorch profiler for different modules

nobodypeng: <p>I am trying to analyze operators’ performance using <a href="https://pytorch.org/docs/master/profiler.html#torch-profiler" rel="noopener nofollow ugc">torch.profiler</a>. I only want to record the operators used in the model (forward and backward) <strong>for several iteration</strong>, but I don’t want the operators used in generating data, putting data to device, etc…</p>
<pre><code class="lang-python">    # DEFINE A profiler here as prof and disable it initially
    for i, (images, target) in enumerate(train_loader):
      
        train_iters=train_iters+1

        # set data to device
        images = images.to(device)
        target = target.to(device)

        # START TO TRACE
        # compute output
        output = model(images)
        loss = criterion(output, target)

        # compute gradient and do SGD step
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        # PUASE THE TRACING USING PROFILER
   # After several iterations
   # 
</code></pre>
<p>What I have tried:</p>
<ol>
<li>The profiler record something I don’t want (operators used in data generating, to device)</li>
</ol>
<pre><code class="lang-python">    with profiler.profile(use_cuda=True,record_shapes=True) as prof:
        for i, (images, target) in enumerate(train_loader):
        
            train_iters=train_iters+1

            # ============PROFILE FROM HERE FOR SEVER ITER==========
            # set data to device
            images = images.to(device)
            target = target.to(device)

            # compute output
            output = model(images)
            loss = criterion(output, target)

            # compute gradient and do SGD step
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            # ============PROFILE END==========
    print(prof.key_averages(group_by_input_shape=True))
</code></pre>
<ol start="2">
<li>using <code>profiler.__enter__</code> and <code>profiler.__exit__</code>, but it can’t be used for recording multiple iterations.</li>
</ol>
<p>So how can I break the profiler for some lines and resume when I want? Hope for your reply. Thanks.</p>
How to apply a nn.Module (i.e. CNN) across an axis (i.e. Video input) in a parallelizable way

seunggs: <p>Hi, I’m trying to apply CNN to each image in a video. Currently, my implementation uses a for loop and torch.cat where I take each image and apply the CNN module in the loop. But clearly, this is sequential and I don’t see why it can’t be parallelized in theory since all images are independent from each other.</p>
<p>However, I’m not sure how this can be accomplished. I couldn’t find any built-in function for PyTorch. Is there a way to use matrix multiplication for something like this?</p>
<p>My video input shape looks like this: (batch_size, seq_len, channel, height, width) and CNN takes input shape of (batch_size, channel, height, width).</p>
<p>Thanks in advance for your help!</p>
MATLAB to Pytorch-Geometric

Anthony_Sirico: <p>I have a .mat file, with a struct. One of the columns in the struct is 43249 cells containing a different adjacency matrix of different sizes. The other struct columns are 43249 cells containing single digit int. My goal is to right a DL program that will train on the the ints and graphs, and predict graphs.</p>
<p>I am new to pytorch, so can someone point me in the right direction? For instance, I am able to add all the matrices to a dictionary or a list, have pytorch convert them all to tensors, and then i try to use loader on them, but i get the following:</p>
<pre><code class="lang-auto">
raceback (most recent call last):
  File "&lt;input&gt;", line 1, in &lt;module&gt;
  File "/home/anthony/anaconda3/envs/GeoDL/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/anthony/anaconda3/envs/GeoDL/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/anthony/anaconda3/envs/GeoDL/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/anthony/anaconda3/envs/GeoDL/lib/python3.8/site-packages/torch_geometric/data/dataloader.py", line 34, in __call__
    return self.collate(batch)
  File "/home/anthony/anaconda3/envs/GeoDL/lib/python3.8/site-packages/torch_geometric/data/dataloader.py", line 17, in collate
    return default_collate(batch)
  File "/home/anthony/anaconda3/envs/GeoDL/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [14, 14] at entry 0 and [16, 16] at entry 3
</code></pre>
<p>My other question is: is what i am asking possible? Thanks for the feedback.</p>
Win10+PyTorch+DataParallel got warning:"PyTorch is not compiled with NCCL support"

frozen86: <p>I’m using PyTorch under Win10, and when apply DataParallel to models, the following warning appears:</p>
<ul>
<li>C:\Anaconda3\lib\site-packages\torch\cuda\nccl.py:27: UserWarning: PyTorch is not compiled with NCCL support<br>
warnings.warn(‘PyTorch is not compiled with NCCL support’)</li>
</ul>
<ol>
<li>Is it because my pytorch was installed under Win10?</li>
<li>Does it slow down the program?</li>
</ol>

frozen86: <p>The cause is that Win10 have no NCCL support.</p>
Fast.ai SimpleSelfAttention implementation in pre-trained ResNet-50 or VGG-16

Vipul_Mehra: <p>Hi Team Pytorch,</p>
<p>I am pretty new to deep learning, and I am lost on how to implement SimpleSelfAttention layer to my existing pre-trained VGG-16 or ResNet-50 model.<br>
SimpleSelfAttention is based on fast.ai code - <a href="https://github.com/sdoria/SimpleSelfAttention" class="inline-onebox" rel="noopener nofollow ugc">GitHub - sdoria/SimpleSelfAttention: A simpler version of the self-attention layer from SAGAN, and some image classification results.</a>. I am unable to wrap my head around this. I tried applying this to avgpool, was able to add, however it did not work.<br>
Also, considering parameters are frozen, I have my own custom Fully-Connected layer, if I add this to let us say 40th layer will the network be trainable after 40th layer? Any help would be highly appreciated.</p>
AttributeError: cannot assign module before Module.__init__() call

wasiahmad: <p>I am getting the following error.</p>
<p><code>AttributeError: cannot assign module before Module.__init__() call</code></p>
<p>I have a class as follows.</p>
<blockquote>
<p>class QuestionClassifier(nn.Module):</p>
</blockquote>
<blockquote>
<pre><code>def __init__(self, dictionary, embeddings_index, max_seq_length, args):
    """"Constructor of the class"""
</code></pre>
</blockquote>
<blockquote>
<pre><code>    self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
    self.encoder = EncoderRNN(args.emsize, args.nhid, args.model, args.bidirection, 
</code></pre>
</blockquote>
<pre><code>                                                   args.nlayers, args.dropout)
</code></pre>
<blockquote>
<pre><code>    self.drop = nn.Dropout(args.dropout)
</code></pre>
</blockquote>
<p>So, when I run the following line:</p>
<p><code>question_classifier = QuestionClassifier(corpus.dictionary, embeddings_index, corpus.max_sent_length, args)</code></p>
<p>I get the above mentioned error. Here, <code>EmbeddingLayer</code> and <code>EncoderRNN</code> is a class written by me which inherits <code>nn.module</code> like the <code>QuestionClassifier</code> class.</p>
<p>What I am doing wrong here?</p>

miguelvr: <p>The first thing you should always do when you create a module is call its super constructor…</p>
<p>So, your class should look like this:</p>
<pre><code>class QuestionClassifier(nn.Module):

    def __init__(self, dictionary, embeddings_index, max_seq_length, args):
        """"Constructor of the class"""
        super(QuestionClassifier, self).__init__()

        self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
        self.encoder = EncoderRNN(args.emsize, args.nhid, args.model, args.bidirection, args.nlayers, args.dropout)
        
        self.drop = nn.Dropout(args.dropout)</code></pre>

AbdulsalamBande: <p>Add<br>
<code>super(QuestionClassifier, self).__init__()</code><br>
after <code>def __init__(self, dictionary, embeddings_index, max_seq_length, args):</code> line</p>
<p>Pytorch keeps track of your custom class submodules like <code>Embedding, nn.RNN</code> using an ordered dictionary in addition to other things. And this dictionary is first initialized in <code>nn.Module</code>.  If you inherit <code>nn.Module</code> as you did in in the first line <code>class QuestionClassifier(nn.Module):</code> , you still need to call the <code>__init__()</code> method of  the <code>nn.Module</code> to initialize the ordered dictionary which will hold your submodules like <code>nn.RNN</code>. Use the <code>super</code> keyword to call the <code>__init__()</code> of the parent class.</p>
Access epoch number from within the model's forward pass

dpetermann: <p>I apologize if this question has already been covered, I couldn’t find much resources on this topic. I am trying to execute some block of code in my model’s forward pass, but only after a certain number of epochs has been reached. From my understanding there is no straightforward way of achieving this (e.g. this <a href="https://github.com/pytorch/fairseq/issues/2396" rel="noopener nofollow ugc">GitHub issue</a> actually describes exactly what I’m trying to do, but led to nowhere, unfortunately).</p>
<p>Any suggestions on how to achieve this ?</p>
<p>Thank you<br>
Darius</p>

dazzle-me: <p>I guess it is doable.</p>
<p>Consider an example :</p>
<pre><code class="lang-auto">class MyModel(torch.nn.Module):
    def __init__(self):
         super(MyModel, self).__init__()
         self.current_epoch = 0

    def update_epoch(self, epoch):
         self.current_epoch = epoch

    def forward(self, x):
         if some_condition(self.current_epoch):
             ...
         else:
             ...
</code></pre>
<p>Before every epoch of your training, you can update epoch inside the model and that’s it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">for epoch in range(1, epochs + 1):
     model.update_epoch(epoch)
     for data, labels in dataloader:
         ...
</code></pre>

dpetermann: <p>Thanks, this did the trick! Fairly clear and elegant solution!</p>
Precision of network gradient

sajjad: <p>Hi dear all<br>
I am using pytorch to generate adversarial samples.<br>
When back propagating the error to input pattern elements in .eval() mode, I observe by changing the batch size from 1 to larger, the gradient values varies slightly.<br>
Does anyone have any idea that this is common or I made a mistake.<br>
two typical gradients are:</p>
<p>Batchsize&gt;1<br>
-0.012502569<br>
-0.06345608<br>
0.024056617<br>
0.00014444088<br>
0.030317063<br>
-0.049938474<br>
0.0059496835</p>
<p>Batchsize=1<br>
-0.012502571<br>
-0.06345608<br>
0.024056613<br>
0.00014443748<br>
0.030317055<br>
-0.049938496<br>
0.005949676</p>
<p>For reproducibility, I have used the following commands:</p>
<p>torch.backends.cudnn.deterministic = True<br>
torch.manual_seed(1)<br>
np.random.seed(1)</p>

tom: <p>This is expected. The typical origin of such fluctuations is that larger sums of floating point values are not commutative. Changing the batch size can typically change the layout of how the computational kernels access data, so these things happen and are outside of any “guarantees” of determinism.</p>

sajjad: <p>Thanks Thomas for your response.<br>
Up to my knowledge, the gradient of loss function w.r.t an input pattern in .eval() mode is independent of  other input patterns in the mini-batch. In such a way as you mentioned the number of summation would increase by larger batch size but these summations can be categorized into independent calculations each corresponding to an individual input pattern. Thus I expect the calculations result in the same gradient. Do you have any idea about it?</p>

tom: <p>It is crucial that this isn’t about the “theoretical maths” but about numerical precision.</p>
<p>So this:</p>
<pre><code class="lang-python">a = torch.randn(5,5)
torch.zeros(10,10)
b[:5,:5] = a
a.sum() - b.sum()
</code></pre>
<p>will typically give something like <code>tensor(-5.3644e-07)</code>.<br>
Even<br>
<code>a.sum() - b[:5,:5].sum()</code><br>
will usually not give exactly zero.</p>
<p>This is because the PyTorch functions make no guarantee in which order the 25 nonzero elements are summed and the sum actually depends on the order due to limited numerical precision.</p>
<p>Best regards</p>
<p>Thomas</p>

sajjad: <p>Thanks Thomas for the clear response. I understand.<br>
Best<br>
Sajjad</p>
Wrong accuracy reported in Torchvision site?

RicLaGra: <p>I used a Resnet-18 pretrained on Imagenet as you can see in torchvision library.<br>
The site report 69.7, but I obtained 68.9.<br>
I used the same preprocessing (test set) as reported in torchvision:</p>
<ul>
<li>normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], 0.229, 0.224, 0.225])</li>
<li>torchvision.transforms.Resize(256)</li>
<li>torchvision.transforms.CenterCrop(224)</li>
</ul>
<p>Can anyone help me? Is it possible an error on the table reported in torchvision?<br>
<a href="https://pytorch.org/vision/0.8/models.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/0.8/models.html</a></p>

MrPositron: <p>Did you use <code>model.eval()</code>?</p>

RicLaGra: <p>Sure.</p>
<pre><code class="lang-auto">def test(model, testloader):
    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for idx, (data, labels) in enumerate(testloader):
                data, labels = data.cuda(), labels.cuda()
                outputs = model(data)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().float().item()

    acc = correct * 100. / total
    err = 100. - acc
    return acc, err
</code></pre>

MrPositron: <p>Can you check this post.</p><aside class="quote quote-modified" data-post="1" data-topic="44995">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e79b87/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torchvision-pretrained-has-accuracy-gap-to-the-reported-value/44995">Torchvision pretrained has accuracy gap to the reported value</a> 
  </div>
  <blockquote>
    Hi everyone, I’m trying to train something using the pretrained densenet201 from torchvision as base. First I tested the pretrained densenet201 to see how its actual accuracy (the reported accuracy is 77 - 77.5) is: 


By using densenet201 from torchvision.models, and follow the example code from the website:<a href="https://github.com/pytorch/examples/tree/42e5b996718797e45c46a25c55b031e6768f8440/imagenet" rel="noopener nofollow ugc">https://github.com/pytorch/examples/tree/42e5b996718797e45c46a25c55b031e6768f8440/imagenet</a> 
I got the top-1 ImageNet accuracy of 76.896 


I also tried using the pretrained model from origin…
  </blockquote>
</aside>
C++ optimization using LibTorch

DineshRajan_T: <p>Hi all,<br>
I have implemented the C++ inference for PyTorch models referring to this tutorial: <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 1.7.1 documentation</a><br>
Is there any ways that I can optimize beyond this both in cpu and gpu<br>
Note: LIbTorch is available for both GPU and CPU and I would like to know …the optimizations after using LibTorch.</p>
PyTorch optimization for CPU

Yosi_K: <p>Hey guys, we are interested in deploying our models on cloud CPUs as they are significantly cheaper than GPUs/TPUs etc.</p>
<p>Intel used to support cpu-based deployment:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://software.intel.com/etc.clientlibs/settings/wcm/designs/intel/default/resources/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://software.intel.com/content/www/us/en/develop/articles/getting-started-with-intel-optimization-of-pytorch.html" target="_blank" rel="noopener nofollow ugc">Intel</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://www.intel.com/content/dam/develop/public/us/en/images/thumbnails/fallback-icons-articles.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://software.intel.com/content/www/us/en/develop/articles/getting-started-with-intel-optimization-of-pytorch.html" target="_blank" rel="noopener nofollow ugc">Getting Started with Intel® Optimization for PyTorch* on Second...</a></h3>

<p>Accelerate deep learning PyTorch* code on second generation Intel® Xeon® Scalable processor with Intel® Deep Learning Boost.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>What is the current status of CPU optimization for PyTorch models?</p>
<p>Is there a preferred distribution (conda etc.) to use?</p>
Model saving as a zip file

Muhammad4hmed: <p>Hi, I am trying to run this repo on my custom dataset: <a href="https://github.com/Lotayou/everybody_dance_now_pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Lotayou/everybody_dance_now_pytorch: A PyTorch Implementation of "Everybody Dance Now" from Berkeley AI lab.</a></p>
<p>the repo requires pytorch’s 0.4.1.post2 version but I’m running on latest pytorch version 1.7+ on google colab<br>
In models/spynet.py file they have used serialization of pytorch which is removed so I used <a href="https://github.com/bshillingford/python-torchfile" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bshillingford/python-torchfile: Deserialize Lua torch-serialized objects from Python</a> for that. except for that, code is almost same as the repository.</p>
<p>When I run training, everything goes fine and weights are saved in checkpoint folder (created during training)<br>
but the very strange thing is, the saved weights, <code>latest_net_G.pth</code> is actually a zip file(without zip extension) and on extracting it, I get this structure<br>
Archive</p>
<ul>
<li>Data Folder consisting of hundreds of pth files</li>
<li>some data.pkl file of weight 3KB</li>
</ul>
<p>Now thats totally strange, I have also checked models/base_models.py file and the model is saved normally there</p>
<p>I’m confused, kindly help, Thanks.</p>

tom: <p>PyTorch (but I’d recommend to use the .pt extension) uses a zip-based format since PyToch 1.6, see <em>Deprecated old <code>torch.save</code> serialization format</em> in the <a href="https://github.com/pytorch/pytorch/releases/tag/v1.6.0" rel="noopener nofollow ugc">1.6 release notes</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

Muhammad4hmed: <p>so should I simply load that zip file and use it? I’m sorry I’m a bit new and have confusion.<br>
thanks</p>

tom: <p>Yeah, torch.load will do the right thing (or torch.jit.load if it’s a scripted model).</p>

Muhammad4hmed: <p>thank you for the help.</p>
Training with custom layer using unfold + mm + fold - Weights optimization

Degs: <p>I am running into an issue where I have a custom convolutional layer using <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html" rel="noopener nofollow ugc">nn.Unfold</a>, which I get the same values comparing with a regular <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">nn.Conv2d</a> with the same weights initialized.<br>
However, if I replace the layer when training a network, the weights seem not to be optimized as the regular nn.Conv2d do, causing the network to be unable to learn.</p>
<p>Is there any additional step to implement in order to backpropagate the error the same way as the nn.Conv2d does?</p>
<p>Thank you for your attention!</p>

ptrblck: <p><code>torch.unfold</code> or <code>nn.Unfold</code> would propagate the gradients.<br>
Could you post your custom implementation of the conv module, so that we could have a look?</p>

Degs: <p>Thank you very much for your answer, you are correct!<br>
It was a small mistake of mine, concerning the use of the same <code>nn.BatchNorm2D</code>, before and after <code>nn.Conv2d</code> in the model’s forward pass… Apparently, there was a conflict I didn’t know about, my bad!<br>
Specifying each <code>nn.BatchNorm2D</code> solved the problem. Thanks!</p>
Ability to sort dataset batch to reduce exccessive padding?

ankitvad: <p>The current update to torchtext makes it a nightmare to understand and all the legacy code is not supported with newer versions of PyTorch and Pytorch-Lightning.</p>
<p>As a result, I’d like to switch to the PyTorch - DataLoader and Dataset. One of the features I find massively lacking is the need to provide a max_length metric to truncate or pad data items within a sampled batch.</p>
<p>Is there a way where we can select a batch from a descending list of data items so that the variability of lengths within each batch is minuscule and as a result, we can get away with minimal padding and no truncating?</p>
<p>One way I can think of is to rewrite the dataset in a particular-sort. Chose a batch without sorting but sort within the batch? Can we do it in an online streaming sort of way?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">This topic</a> by <a class="mention" href="/u/vdw">@vdw</a> might be helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ankitvad: <p>Thanks! I found some good ideas here - <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284/" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></p>
Intersection between tensors

Z_Mario: <p>Given two tensor:</p>
<pre><code class="lang-auto">A = torch.tensor([[0, 2],
                  [1, 2],
                  [2, 0],
                  [2, 1],
                  [2, 3],
                  [3, 2],
                  [1, 8],
                  [8, 1]])

B = torch.tensor([[0, 3],
                  [1, 8],
                  [8, 1]])

</code></pre>
<p>How two get the intersection of two tensors, so that I can get <code>tensor([[1,8],[8,1]])</code>?</p>

Eta_C: <p>Maybe</p>
<pre><code class="lang-python">import torch

A = torch.tensor([[0, 2], [1, 2], [2, 0], [2, 1],
                  [2, 8], [3, 2], [1, 8], [8, 2]])

B = torch.tensor([[0, 3], [1, 8], [8, 2]])

C = (A[:, None] == B[None, :])
C1 = (C[..., 0] &amp; C[..., 1]).nonzero()
interset = A[C1[:, 0], :]
print(interset)
</code></pre>
How does `AdaptiveAvgPool2d` work for increasing the size?

a_d: <p>Hello All,</p>
<p>I have a section in my network which needs constant size inputs and I have variable size data and therefore I am using <code>adaptive avg pool2D</code>. Mostly the image sized would get downscaled but there might be certain cases where image sizes go up let us say from <code>(120, 120)</code> to <code>(160, 160)</code>.  So how does the function takes care of such cases. I believe it would pad the image internally, so what is the kind of padding?. And Secondly, is upscaling like this advisable?</p>
<p>TIA</p>

Eta_C: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/690eaf9c43f3f71051d90e9f47f25b014ce84681/aten/src/ATen/native/cpu/AdaptiveAvgPoolKernel.cpp#L13-L64" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/690eaf9c43f3f71051d90e9f47f25b014ce84681/aten/src/ATen/native/cpu/AdaptiveAvgPoolKernel.cpp#L13-L64" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/690eaf9c43f3f71051d90e9f47f25b014ce84681/aten/src/ATen/native/cpu/AdaptiveAvgPoolKernel.cpp#L13-L64</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="13" style="counter-reset: li-counter 12 ;">
<li>template &lt;typename scalar_t&gt;</li><li>void cpu_adaptive_avg_pool(</li><li>    Tensor&amp; output_,</li><li>    const Tensor&amp; input_,</li><li>    IntArrayRef output_size) {</li><li>  auto input = input_.contiguous();</li><li>  auto output = output_.contiguous();</li><li></li><li>  auto input_data = input.data_ptr&lt;scalar_t&gt;();</li><li>  auto output_data = output.data_ptr&lt;scalar_t&gt;();</li><li></li><li>  int64_t ndim = input.ndimension();</li><li>  // treat batch size and channels as one dimension</li><li>  int64_t channels = ndim == 3 ? input.size(0) : input.size(0) * input.size(1);</li><li>  int64_t input_height = input.size(-2);</li><li>  int64_t input_width = input.size(-1);</li><li>  int64_t output_height = output_size[0];</li><li>  int64_t output_width = output_size[1];</li><li></li><li>  // parallel on dim of N, C</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/690eaf9c43f3f71051d90e9f47f25b014ce84681/aten/src/ATen/native/cpu/AdaptiveAvgPoolKernel.cpp#L13-L64" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How we can remove this specific inplace operator?

silvester: <p>Hello Experienced Ones:<br>
How I can avoid the following inplace operators?</p>
<pre><code class="lang-auto">H = torch.zeros(10, 10)
H[:5, :5] = R.t()
H[8:, 8:] = K.t()
</code></pre>
<p>where R and K are 5 x 5 and 2 x 2 tensors.</p>

ptrblck: <p>You could use a few <code>torch.cat</code> operations to create the result tensor through a few parts, if you need to avoid inplace ops.</p>
Using upper upper directories' name as labels

xu555: <p>I want to use torchvision ImgaeFolder class to create my dataset.</p>
<p>By default ImageFolder creates labels according to upper directories’ name. I want to change this behaviour to upper upper directories.</p>
<p>Here is my directories structure:</p>
<pre><code class="lang-auto">Image/
    -001/
        -frontal/
            -01.jpg
            -02.jpg
    -002/
        -frontal/
            -01.jpg
            -02.jpg
    -003/
        -frontal/
            -01.jpg
            -02.jpg
</code></pre>
<p>So as you can see, the upper directories’ name are all ‘frontal’. I want to set upper upper directories’ name(001,002,003) as label name.</p>
<p>How can I modify the torchvision ImgaeFolder class?</p>
<p>Thank you in advance!</p>

ptrblck: <p>You could use the <a href="https://github.com/pytorch/vision/blob/ced96a0ca66e35ae61b64267e083fc3e3a62172a/torchvision/datasets/folder.py#L83-L187"><code>DatasetFolder</code></a> implementation and directly modify it in your script.</p>
Will there be a PyTorch equivalent of Tensorflow's MCMC module?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I was wondering if PyTorch will ever make a Markov Chain Monte Carlo module like Tensorflow’s mcmc module?</p>
<p>I’ve made a basic version of a Metropolis-Hastings sampler myself, and want to have a looking at doing other (more complicated) samplers in the future. However, I was just curious about if this is something that will be implemented in a later version of PyTorch?</p>
<p>Thank you in advance!</p>

ptrblck: <p>I think <a href="https://github.com/pyro-ppl/pyro">Pyro</a> has an implementation of Markov Chain Monte Carlo.</p>
Imagenet pretrained models image dimensions

over9k: <p>Hi,</p>
<p>I see here (<a href="https://pytorch.org/docs/stable/torchvision/models.html#classification" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html#classification</a>) that the imagenet pretrained models expect image height and width dimensions to be “at least 224”. What does that mean? Can we pass in images smaller or larger?</p>
<p>Were the models trained on Imagenet images that were resized to 224x224 first? If that were the case, would we not force the images we pass in to also be 224x224? So</p>
<p>This poses another question - the Imagenet images are of different resolutions and it appears that 256x256 is a common size to use. Why then were the Pytorch models trained on 224x224?</p>
<p>Thank you for helping me understand this better.</p>

ptrblck: <p>The pretrained models are most likely sticking to the literature for the corresponding model, which often used input images of the shape <code>224 x 224</code> (often randomly cropped to this shape).</p>
<p>Since these <code>torchvision</code> models use adaptive pooling layers the strict size restriction was relaxed and you would be able to pass bigger images and (some) smaller images. Note that the min. size depends on the conv and pooling operations, which would create an empty output, if your input is too small.</p>
<p>That being said, I would not assume to see a good performance for largely differently shapes input images and would try to finetune the model for this use case.</p>

over9k: <p>Great, thank you for your answers. Will bug you again if more questions arise.</p>

over9k: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Coming back to this.</p>
<p>I am able to pass much larger images and as you said, the adaptive pool layer before the fully connected layer allows for this.</p>
<p>What do you mean by fine-tuning the model? In terms of fine-tuning in the transfer learning domain or something more structural. For example, I was thinking adding a convolutional layer at the beginning of the pretrained network, followed by an AdaptiveAvgPool2d layer to bring the output size to 224x224 and then pass it into the next conv layer, which is in fact the first of the pretrained layer. Does this make sense?</p>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="over9k" data-post="4" data-topic="94649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/a87d85/48.png" class="avatar"> over9k:</div>
<blockquote>
<p>For example, I was thinking adding a convolutional layer at the beginning of the pretrained network, followed by an AdaptiveAvgPool2d layer to bring the output size to 224x224 and then pass it into the next conv layer, which is in fact the first of the pretrained layer. Does this make sense?</p>
</blockquote>
</aside>
<p>It could make sense and I would definitely try it out and compare it to passing the larger images directly to the pretrained model.</p>
Change the input to grayscale in the pytorch program

ww5171351: <p>Change input from three-channel color to grayscale，use：self.turn = transforms.Grayscale(num_output_channels=1).<br>
The same program can run through on one computer, and if it is changed to another computer (with the same environment), it will prompt an error:img should be PIL Image. Got &lt;class ‘torch.Tensor’&gt;。Who can help me answer</p>

JamesDickens: <p>Are you sure the input is a PIL image either read from a file or from a numpy array?</p>

ww5171351: <p>I use this program：<a href="https://github.com/KaiyangZhou/deep-person-reid%E3%80%82" rel="noopener nofollow ugc">https://github.com/KaiyangZhou/deep-person-reid。</a><br>
Now the interesting problem is that the program I modified can run on a computer, but when it is put on the server (strong computing power), it prompts an error.</p>

Alexey_Demyanchuk: <p>Hi) Most likely it is the torchvision version is different on your PC versus server. The opportunity to use transforms on tensors was added only recently. So pytorch and torchvision update on the server should help.</p>

ww5171351: <p>Thanks for your reply, I will update the next version to try.</p>

ww5171351: <p>=&gt; Start training<br>
epoch: [1/250][50/202]  time 0.283 (0.297)</p>
<p>Thank you, the question has been resolved.</p>
Is the right way to use data loader in nlp?

Rohan_Kumar: <p>Here is my understanding of dataloaders -</p>
<ul>
<li>create dataset class with getitem and len attributes
<ul>
<li>build vocab</li>
<li>convert words to ids</li>
<li>pad the sentence</li>
<li>convert it into tensor?</li>
</ul>
</li>
<li>pass into data loader</li>
<li>pass it into the model<br>
is this right?</li>
</ul>
<p>i am working on imdb dataset rn, i have the review, the rating and sentiment, so i do this, and do i have to do more, can anyone link a code that does this, or a tutorial for dataloaders in nlp. or can someone give a detailed step by step (explain like i am 5) of dataloaders.</p>

vdw: <p>Yes, this is standard list of steps in that order. You might want to look online for <code>torchtext</code> examples.</p>

zhangguanheng66: <p>And more docs related to <code>DataLoader</code> and <code>Sampler</code> <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">here</a><br>
A tutorial <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">here</a></p>

Brando_Miranda: <p>those are not related to NLP. Why would you link that when the OP asked for nlp specific things</p>
<blockquote>
<p>a tutorial for dataloaders in nlp</p>
</blockquote>

Brando_Miranda: <p>did you find an example/tutorial for NLP data loaders?</p>
Torch detects 1 GPU inside Jupyter but multiple from the CLI?

EvanZ: <p>Not sure what could be the issue but when I do <code>torch.cuda.device_count()</code> inside a Python REPL it says 4, but from inside a Jupyter notebook it only reports 1. I’m running this on an Amazon EC2 instance (p3.16xlarge).</p>

ptrblck: <p>Could you check, if your Jupyter environment uses <code>CUDA_VISIBLE_DEVICES</code> somewhere, which would mask the devices and only make the passed ones visible?</p>

EvanZ: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I don’t see that variable using <code>%env</code>. Is there somewhere else I should check?</p>

ptrblck: <p>I don’t know, how Jupyter is launched and if specific env vars are passed it its launch.<br>
However, checking the environment via <code>export</code> could work.<br>
Also, I don’t know, if Jupyter itself uses some arguments to mask specific GPUs.</p>
The code is running faster on CPU

silvester: <p>Due to the iterations for the update steps, the following code is running faster on CPU.<br>
Also I realized for the update step and inference it is running faster with numpy than pytorch. Is there a way to optimize it?</p>
<pre><code class="lang-auto">import torch
device = torch.device('cpu')
import pdb


class MesNet(torch.nn.Module):
        def __init__(self):
            super(MesNet, self).__init__()

            self.cov_lin = torch.nn.Sequential(torch.nn.Linear(6, 5)).double()

        def forward(self, u):
            z_cov = self.cov_lin(u.transpose(0, 2).squeeze(-1))
            return z_cov 


class UpdateModel(torch.nn.Module):

    def __init__(self):
        torch.nn.Module.__init__(self)
        self.P_dim = 18
        self.Id3 = torch.eye(3).double()
        
    def run_KF(self):
        N = 10
        u = torch.randn(N, 6).double()
        v = [torch.zeros(3).double()]
        model = MesNet()
        measurements_covs_l = model(u.t().unsqueeze(0))
        # remember to remove this afterwards
        torch.autograd.set_detect_anomaly(True)
        for i in range(1, N):
            v_new = self.update_pos(v[i-1].detach(), measurements_covs_l[i-1])
            v.append(v_new.clone())

        criterion = torch.nn.MSELoss(reduction="sum")
        targ = torch.rand(10, 3).double()
        pdb.set_trace()
        loss = criterion(v, targ)
        loss = torch.mean(loss)
        loss.backward()
        return v, p


    def update_pos(self, v, measurement_cov):
        Omega = torch.eye(3).double() 
        H = torch.ones((5, self.P_dim)).double()
        R = torch.diag(measurement_cov)
        Kt = H.t().mm(torch.inverse(R))
        # it is indicating inplace error even with this: 
        # Kt = H.t().mm(R)
        dx = Kt.mv(torch.ones(5).double())
        dR = self.trans(dx[:9].clone())
        v_up = dR.mv(v)
        return v_up

    def trans(self, xi):
        phi = xi[:3].clone()
        angle = torch.norm(phi.clone())

        if angle.abs().lt(1e-10):

            skew_phi = torch.eye(3).double()
            J = self.Id3 + 0.5 * skew_phi
            Rot = self.Id3 + skew_phi
        else:
            axis = phi / angle
            skew_axis = torch.eye(3).double()
            s = torch.sin(angle)
            c = torch.cos(angle)

            Rot = c * self.Id3
        return Rot


net =  UpdateModel()
net.run_KF()
</code></pre>
My PyTorch playlist

Dimitris_Tosidis: <p>            <iframe width="480" height="360" src="https://www.youtube.com/embed/nkFnCNDXzG0?feature=oembed&amp;wmode=opaque&amp;list=PLArFyb0P44yd_U8AVqlzn3zybcS-Z-nl6" frameborder="0" allowfullscreen="" seamless="seamless" sandbox="allow-same-origin allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-presentation"></iframe>
</p>
How to create dynamic Dataset

ConstantSun: <p>Hi, I’m doing active learning for my segmentation project. I have an “IMG folder” containing about 3000 images. My task is: First, loading 500 images of that IMG folder for training a model X; Second, using some constraint ( which is defined by is_collected() function) to collect next data ( also in that IMG folder)  for the next training phase, and so on…<br>
What should I do with the loading Dataset?</p>

Reuben_Wenisch: <p>Do you mean batch size of 500? The fundamental idea is a generator wherein each loop you increment say “i” and then continue from there (next 500) when you use the next command.</p>

ConstantSun: <p>No, I mean:  I want to use 500 images for the first training time (batch size can be any ). After that, using some constraint ( e.g: entropy loss of the image &lt; some alpha ) to collect new data that is still in that original pool dataset =&gt; I have to create a new DataLoader? How can I do this?</p>

dazzle-me: <p>Well, I think you can maintain some kind of “don’t load this image anymore” list, which will contain img’s names on which your cross_entropy  &lt; alpha after training.</p>
<p>Consider an example :</p>
<pre><code class="lang-auto">class RestrictedDataset(torch.utils.data.Dataset):
    def __init__(self, img_directory, banned_images):
        self.img_pathes = [image_name for image_name in os.listdir(img_directory) \
                           if not image_name in banned_images]
    def __getitem__(self, index):
        image_name = self.img_pathes[index]
        # somehow load image
        # image = ...
        return image, image_name
    def __len__(self):
        return len(self.img_pathes)
</code></pre>
<p>And then during training you will have to discard image_name (because there no use of it during training)</p>
<p>But when when the training is done, and you want to remove some images that pass the threshold, - you’ll have this image_name to add it to banned_images list, hope you get the idea.</p>
<p>P.S. you will have to recreate dataset and dataloader after each training cycle, but I think it won’t bring any overhead in your solution</p>

ConstantSun: <p>Yea, thank you <a class="mention" href="/u/dazzle-me">@dazzle-me</a> ! I think the same as you, moreover, I think I should create a json file that contains all the banned images, and this json file will be updated every training phase.</p>
LSTMCell Example

Carlos_Jose_Gonzalez: <p>Hi, I think that there is an error in the example presented for a LSTMCell:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html:" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html:</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60d8291e7f31d596e83b8736b046f466f7dfbb85.png" data-download-href="https://discuss.pytorch.org/uploads/default/60d8291e7f31d596e83b8736b046f466f7dfbb85" title="Screenshot from 2021-02-04 16-32-03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d8291e7f31d596e83b8736b046f466f7dfbb85_2_690x228.png" alt="Screenshot from 2021-02-04 16-32-03" data-base62-sha1="dOIXt0rnoGpy54TBTCnkJXvuyl7" width="690" height="228" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d8291e7f31d596e83b8736b046f466f7dfbb85_2_690x228.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d8291e7f31d596e83b8736b046f466f7dfbb85_2_1035x342.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/0/60d8291e7f31d596e83b8736b046f466f7dfbb85.png 2x" data-dominant-color="F3F4F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-02-04 16-32-03</span><span class="informations">1044×345 21.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The loop iterates over 6 steps, but the input has only 3 steps.  I also think that there is an error with the shape of the initial hidden and cell states. Here is my suggestion:</p>
<p>rnn = nn.LSTMCell(10, 20)<br>
input = torch.randn(3, 10)<br>
hx = torch.randn(1, 20)<br>
cx = torch.randn(1, 20)<br>
output = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(input.size()[0]):<br>
hx, cx = rnn(input[i:i+1], (hx, cx))<br>
output.append(hx)</p>
<p>Please let me know if I misunderstood something <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Thanks for pointing out this issue. Could you create an issue on GitHub, so that we can track and fix it?<br>
Based on the current code snippet I assume the example should use <code>6</code> time steps, so <code>input</code> would have to be initialized as e.g. <code>torch.randn(6, 3, 10)</code>.</p>

Carlos_Jose_Gonzalez: <p>Ok, I did it ( LSTM Cell example <span class="hashtag-raw">#51801</span>).  I suggested the example of the GRUCell adapted to the gates of an LSTM:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50df893220ccf1dcf0dba2208851f63f93981a5b.png" alt="Screenshot from 2021-02-05 16-32-44" data-base62-sha1="bxr7vECOGZqjvwKj8yFhzzgsAjN" width="501" height="420"></p>
<p>Thanks for your answer!</p>
How to fuse layers of any convolutional neural network?

studML: <p>Hello,<br>
Is there any generalized way or code to fuse layers of any convolutional model ?<br>
ex. AlexNet, ResNet, VGG<br>
just one code which will work for all sort of model to fuse their conv + bn + relu layers.</p>

Thomas_Carstens: <p>Any update on this topic?<br>
Thank you.</p>

Eta_C: <p>Check <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/fuse_modules.py" rel="noopener nofollow ugc">here</a>.</p>
<p>Simply, for fusing conv + bn + relu, you can replace the convolution operators to <a href="https://pytorch.org/docs/master/torch.nn.intrinsic.html" rel="noopener nofollow ugc">these operators</a>, and then replace bn operator and relu operator to <code>nn.Identity()</code>.</p>
<p>PS: Please check the correctness by yourself.</p>

jerryzh168: <p>this can be achieved by FX Graph Mode Quantization, which will be released as a prototype feature in PyTorch 1.8</p>
Why are my saved models identical?

palloc: <p>Hi,</p>
<p>I’ve been training a model and saving it at each epoch using:</p>
<pre><code class="lang-auto">torch.save(cnn.state_dict(), filepath)
</code></pre>
<p>The saved models seem all the same. They are identical in size and the pth.tar contains the same files.<br>
What am I doing wrong?</p>
<p>Thanks in advance!<br>
A</p>

BramVanroy: <p>Why wouldn’t they be the same size? The model and number of parameters are the same so the size of the files should be the same.</p>