akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to correctly write type annotation for Subset of extended Dataset class?

bilzrd: <p>In this code, the static code analyzer show error with it says “targets not defined in dataset”.<br>
If <code>targets</code> are defined in extended class of Dataset, how to correctly type annotate about this?</p>
<pre><code class="lang-python">from typing import Sequence
from torch.utils.data import Dataset, Subset

def log_class_statistics(subset: Subset):
    train_classes = [subset.dataset.targets[i] for i in subset.indices]
    print(dict(Counter(train_classes)))

class ExtDataset(Dataset):
    targets: Sequence[int]

subset = Subset(ExtDatset(...))
log_class_statistics(subset)
</code></pre>
Does setting num_workers > 0 implement sharding for Map style datasets

prakharsharma: <p>Hi,</p>
<p>I have a question regarding the PyTorch DataLoaders. I know that when we set <code>num_workers &gt; 0</code>, in the <code>dataloader</code>, it creates multiple process and not threads, so no shared memory among them. Each process is passed an object of the <code>dataset</code> class, <code>collate_fn</code>, and <code>worker_init_fn</code> (According to the docs <a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" rel="noopener nofollow ugc">here</a>).</p>
<p>Now, if my dataset object has an attribute which contains all the data and say it takes <code>n</code> amount of memory on my RAM. Will my program take <code>n * num_workers</code> amount of memory or will the <code>dataloader</code> somehow shard the data such that each workers has access to only subset of data and sum of memory across all workers on RAM stays just <code>n</code>?</p>

ptrblck: <p>Your <code>DataLoader</code> would create copies of the data and use <code>n * num_workers</code> amount of memory.</p>
Cuda error on a NLP transformer

Armin_Dashti: <p>I created an NLP transformer. But I don’t know why I got this error randomly. It’s not related to a specific batch. It raise randomly.</p>
<pre><code class="lang-auto">C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1141: block: [131,0,0], thread: [123,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
</code></pre>
<p>I have an NVIDIA RTX 2060 OC 6GB</p>

ptrblck: <p>Based on the error message an indexing operation fails due to invalid values in the <code>index</code> tensor.<br>
Rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stacktrace to see which line of code fails.</p>
Diffusions for pix2pix translation

JungHwang: <p>Hi,<br>
I am wondering if diffusion models are suitable for pix2pix or img2img transformation without text prompts.</p>
<p>What I am looking for is transforming a color image with 3 channels to a UAV infrared image look alike image, with 1 channel, taken by a Infrared sensor. CycleGAN does the exact job that I need, but diffusion models seem to produce much better images. So I am planning on training a diffusion model, CycleDiffusioin might be a great point to start I guess, but all the diffusion models seem to have something to do with texts.</p>
how to calculate gradient manually like autograd function in backpropagation if neural network architecture consists of linear, batch norm, leaky relu, linear?diri dari linear, batch norm, leaky relu, linear?

octadion: <p>what are the steps to manually calculate the backpropagation gradient with the architecture that I mentioned? because I’m confused, the architecture on google regarding backprop is different from the neural network architecture that I use, I’m confused about the linear layer that doesn’t use the activation function and how to calculate the gradient on the batch norm with its derivative function. I’ve tried to calculate the gradient loss to output, here the loss I use is bcewithlogitsloss, then I try to calculate the linear layer by multiplying the output of the previous layer by the gradient loss to output, and starting here I feel wrong. the output I want is a gradient value that I can use to update the weights with the adam optimizer</p>
No pretrained weights exist for this model, using random initialization

wsouza: <p>Hello,</p>
<p>I have a question regarding transfer learning. Every time that I try to load the model’s weight from the model that I have trained, I get: “No pretrained weights exist for this model, using random initialization”. I am wondering if I am doing the transfer learning properly.</p>
<p>Here is my class definition</p>
<pre><code class="lang-auto">class TIMMModels(pl.LightningModule):

    def __init__(self):
        super().__init__()
        self.model = timm.create_model('resnet34', pretrained=True, in_chans = 1, drop_rate=0.2, num_classes = 2)
        for param in self.model.parameters():
            param.requires_grad = False
        self.model.eval() #in case the model has some normalization layer, behave like in the evaluation mode

        num_in_features = self.model.get_classifier().in_features
        self.model.fc = torch.nn.Sequential(#by default gradient is required
        torch.nn.BatchNorm1d(num_in_features),
        torch.nn.Linear(in_features=num_in_features, out_features=256, bias=False),
        torch.nn.ReLU(),
        torch.nn.Linear(in_features=256, out_features=2, bias=False))

    def forward(self, x):
        x = self.model(x)
        return x
</code></pre>
<p>After the training, I try to load the weights</p>
<pre><code class="lang-auto">model_trained = TIMMModels.load_from_checkpoint(checkpoint_path).to(device)
</code></pre>
<p>and then I get: “No pretrained weights exist for this model, using random initialization”. I double checked the path, and the file was there. Do I need to save the checkpoint in a different way? Or am I doing something wrong regarding the transfer learning procedure?</p>
<p>Thank you.</p>

wsouza: <p>I found the problem, the name of the model was wrong.</p>
Where is <ATen/ops/upsample_bilinear2d.h>?

MartinZhang: <p>where is &lt;ATen/ops/upsample_bilinear2d.h&gt;?</p>
<p>I can’t find it, who can help me.</p>
<p>I can’t find it in <a href="https://github.com/pytorch/pytorch/tree/652af5ec15b81c39ec7413519d0ce9938d87bcf1/aten/src/ATen/ops" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/ops at 652af5ec15b81c39ec7413519d0ce9938d87bcf1 · pytorch/pytorch · GitHub</a></p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/652af5ec15b81c39ec7413519d0ce9938d87bcf1/aten/src/ATen/native/UpSampleBilinear2d.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/652af5ec15b81c39ec7413519d0ce9938d87bcf1/aten/src/ATen/native/UpSampleBilinear2d.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/652af5ec15b81c39ec7413519d0ce9938d87bcf1/aten/src/ATen/native/UpSampleBilinear2d.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/652af5ec15b81c39ec7413519d0ce9938d87bcf1/aten/src/ATen/native/UpSampleBilinear2d.cpp</a></h4>


      <pre><code class="lang-cpp">// Adapted from interp.cpp from Caffe util by Pauline Luc
// Originally developed by George Papandreou
#define TORCH_ASSERT_ONLY_METHOD_OPERATORS

#include &lt;ATen/core/Tensor.h&gt;
#include &lt;ATen/TensorMeta.h&gt;
#include &lt;ATen/native/UpSample.h&gt;
#include &lt;c10/util/irange.h&gt;

#ifndef AT_PER_OPERATOR_HEADERS
#include &lt;ATen/Functions.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#else
#include &lt;ATen/ops/_upsample_bilinear2d_aa.h&gt;
#include &lt;ATen/ops/_upsample_bilinear2d_aa_backward.h&gt;
#include &lt;ATen/ops/_upsample_bilinear2d_aa_backward_native.h&gt;
#include &lt;ATen/ops/_upsample_bilinear2d_aa_native.h&gt;
#include &lt;ATen/ops/upsample_bilinear2d.h&gt;
#include &lt;ATen/ops/upsample_bilinear2d_backward.h&gt;
#include &lt;ATen/ops/upsample_bilinear2d_backward_native.h&gt;
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/652af5ec15b81c39ec7413519d0ce9938d87bcf1/aten/src/ATen/native/UpSampleBilinear2d.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>The file is generated by <a href="https://github.com/pytorch/pytorch/blob/1ec655565d215ef706d1178fe1d4d4ce84b5e6ce/torchgen/gen.py"><code>torchgen/gen.py</code></a> from <a href="https://github.com/pytorch/pytorch/blob/1ec655565d215ef706d1178fe1d4d4ce84b5e6ce/aten/src/ATen/native/native_functions.yaml#L11814-L11821"><code>native_functions.yaml</code></a> and is located in <code>torchh/include/ATen/ops/upsample_bilinear2d.h</code> after PyTorch is built.</p>

MartinZhang: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> ，I find it. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"></p>
Slow convergence of loss for Distributed Data Parallel in mulit-GPU environment

Mrutuynjay_Wanjerkhe: <p>Hi,</p>
<p>I’m training my model on 2-gpu system with CUDA 11.0 &amp; PyTorch 1.7.1. Using DDP, I see that on a single GPU model loss is converging faster. But when use both GPUs slow convergence.</p>
<p>What could be the problem? Tried initializing the random seed, same results.</p>
<p>1-GPU Epochs with DDP</p>
<pre><code class="lang-auto">===&gt; Epoch 0 Complete: Avg. Loss: 0.36975980444256995
===&gt; Epoch 1 Complete: Avg. Loss: 0.32454686222479784
===&gt; Epoch 2 Complete: Avg. Loss: 0.3071180762120964
===&gt; Epoch 3 Complete: Avg. Loss: 0.2750444378917671
===&gt; Epoch 4 Complete: Avg. Loss: 0.24473399923287129
===&gt; Epoch 5 Complete: Avg. Loss: 0.21892599486872508
===&gt; Epoch 6 Complete: Avg. Loss: 0.20032298285795483
===&gt; Epoch 7 Complete: Avg. Loss: 0.1875871649501547
===&gt; Epoch 8 Complete: Avg. Loss: 0.17785485737093265
===&gt; Epoch 9 Complete: Avg. Loss: 0.17077650971643155
</code></pre>
<p>2-GPU Epochs with DDP:</p>
<blockquote>
<p>===&gt; Epoch 0 Complete: Avg. Loss: 0.4675159709281232</p>
</blockquote>
<pre><code class="lang-auto">&gt; ===&gt; Epoch 1 Complete: Avg. Loss: 0.33464150040982715
&gt; ===&gt; Epoch 2 Complete: Avg. Loss: 0.330990740333695
&gt; ===&gt; Epoch 3 Complete: Avg. Loss: 0.3278805889997138
&gt; ===&gt; Epoch 4 Complete: Avg. Loss: 0.3254638583545225
&gt; ===&gt; Epoch 5 Complete: Avg. Loss: 0.3231941443609904
&gt; ===&gt; Epoch 6 Complete: Avg. Loss: 0.3184018903468029
&gt; ===&gt; Epoch 7 Complete: Avg. Loss: 0.3123437018997698
&gt; ===&gt; Epoch 8 Complete: Avg. Loss: 0.30351564180420104
&gt; ===&gt; Epoch 9 Complete: Avg. Loss: 0.29410275745104597
</code></pre>
<p>Regards,<br>
MJay</p>

Seikegn: <p>Hi, I’m confronted with a similiar problem, maybe even worth.</p>
<p>For a model training on ModelNet 40 dataset, with the same setting, when using a <strong>single GPU, the  error decreases to 5e-4 and accuracy achieves 80% in 88 epochs</strong>.</p>
<p>However, using <strong>2 GPUs</strong> wrapped in DDP,  <strong>at epoch 88 the error was around 15e-4 and accuracy 48%</strong>; using <strong>4 GPUs, at epoch 88 was around 14e-4 and accuracy around 53%</strong>. And in either case, the error and accuracy fail to be improved effectively given more epochs.</p>
<p>For the possibility of batch size of one, I didn’t use batchnorm, instead I applied <strong>layernorm</strong>. Therefore no SyncNorm was added. Is this choice of normalization the reason for such huge discrepancy? Also I used <strong>torch.cuda.amp</strong> throughout.</p>
<p>Hope to have some discussions.</p>
<p>The platform is Pytorch 1.7.1 with cuda 10.1.</p>

bdebowski: <p>Did you ever resolve this problem?  I am having a similar problem</p>

HrealcodeH: <p>Did you ever resolve this problem? I am having a similar problem <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"></p>
Is cuda.synchronize() needed to validate a model?

Miguel_Campos: <p>According to the documentation: “Waits for all kernels in all streams on a CUDA device to complete.”<br>
So my question is if this is needed (or good practice) before starting the validation phase (only one GPU is used).</p>
<p>Thanks.</p>

ptrblck: <p>No, you don’t need to manually synchronize your code unless e.g. you want to profile the code or if you are using custom CUDA streams and want to synchronize the entire device. PyTorch will use the default stream and thus no explicit synchronizations are needed.</p>
Problems with pytorch Mask R-CNN tutorial

W.Ottevanger: <p>Hi,</p>
<p>I was trying out the pytorch <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">Mask R-CNN/Faster R-CNN tutorial</a> and noticed a few problems.</p>
<p>Problem <span class="hashtag">#1:</span><br>
The first thing you see when you open the page is that there is a <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb" rel="noopener nofollow ugc">google colab you can open</a>, but this link doesn’t work anymore.</p>
<p>I have pytorch 10.0 installed and torchvision 0.11.1.</p>
<p>I ignored the not existing google colab notebook and tried just running the code in the tutorial, and experienced some more problems.</p>
<p>Problem <span class="hashtag">#2:</span></p>
<pre><code class="lang-auto">import transforms as T
</code></pre>
<p>Did not work, as there was no transform package, I tried pip install transforms, but that is another unrelated package. I eventually fixed this by changing it to:</p>
<pre><code class="lang-auto">import torchvision.transforms as T
</code></pre>
<p>Problem <span class="hashtag">#3:</span><br>
In the tutorial this line of code is given:</p>
<pre><code class="lang-auto">data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4, collate_fn=collate_fn)
</code></pre>
<p>This causes problems because collate_fn is not an existing function, or I can’t find it. I tried looking in torch and torchvision but couldn’t find any collate_fn.</p>
<p>The putting everything together paragraph the tutorial states:<br>
“In references/detection/, we have a number of helper functions to simplify training and evaluating detection models. Here, we will use references/detection/engine.py, references/detection/utils.py and references/detection/transforms.py. Just copy everything under references/detection to your folder and use them here.”</p>
<p>I have no idea where to find these folders to paste in my solution folder, I suspect the solution to my problem can be found there. Does anyone know where I can find them?</p>
<p>I downloaded the PennFudanDataset, I installed pycocotools because I am using a windows machine. I feel like I followed the appropriate steps.</p>
<p>Am I missing something? Or is this tutorial outdated and not possible to follow?</p>
<p>Thank you for your help in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why comparision of single value torch.Tensor with float - performance

bartholomew: <p>Why comparison of single value tensor with float is much slower than a comparison of tensors item method output with float?<br>
I am optimizing my code and have just seen this behavior. Is it expected?<br>
Example below:</p>
<pre><code class="lang-auto">In [7]: one = torch.tensor(1.)

In [8]: %timeit one &gt; 1.5
6.28 µs ± 9.48 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [9]: %timeit one.item() &gt; 1.5
562 ns ± 1.18 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [19]: %timeit torch.gt(one, 1.5)
6.42 µs ± 18.9 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
</code></pre>

ptrblck: <p>The tensor comparison would dispatch to internal PyTorch methods, might cast the <code>dtype</code> of the value you are comparing against and could even move it to the same <code>device</code> where the tensor is stored. The <code>item()</code> call creates a plain Python value to compare against the other float.</p>
Autograd calculate the analytical derivative of NN

ElmBh: <p>I wan’t to know if the torch.autograd.grad calculate the analytical gradient of NN output or just the eulerienne gradient</p>

111357: <p>It calculates the <strong>analytical gradient</strong> by chain rule.</p>
<p>You can look at <a href="https://pytorch.org/blog/overview-of-pytorch-autograd-engine/" rel="noopener nofollow ugc">Overview of PyTorch Autograd Engine</a> to get more details.</p>
ONNX export failed on an operator with unrecognized namespace 'torch_scatter::scatter_max'

Sajjad_Aemmi: <p>I have a pytorch network like this</p>
<pre><code class="lang-python">import torch.nn as nn
import torch_scatter.scatter_max

class DGCN(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        ...
        torch_scatter.scatter_max(x, index, dim=0)
        ...
</code></pre>
<p>But when i want export my model to onnx, i face this error:</p>
<pre><code class="lang-bash">  ...
  File "/usr/local/lib/python3.9/dist-packages/torch/onnx/utils.py", line 1115, in _model_to_graph
    graph = _optimize_graph(
  File "/usr/local/lib/python3.9/dist-packages/torch/onnx/utils.py", line 663, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/usr/local/lib/python3.9/dist-packages/torch/onnx/utils.py", line 1909, in _run_symbolic_function
    raise errors.UnsupportedOperatorError(
torch.onnx.errors.UnsupportedOperatorError: ONNX export failed on an operator with unrecognized namespace 'torch_scatter::scatter_max'. 
If you are trying to export a custom operator, make sure you registered it with the right domain and version.
</code></pre>
<p>So, How i can do this exactly?</p>

111357: <p>This issue was due to the <code>torch_scatter::scatter_max</code> not being supported by either stock PyTorch or ONNX.</p>
<p>Some users have the same <a href="https://github.com/pytorch/pytorch/issues/73388" rel="noopener nofollow ugc">issue</a>.</p>
<p>To solve this issue, I think you can implement the <code>torch_scatter::scatter_max</code> by <code>torch.nn.Module</code> or <code>TORCH.NN.FUNCTIONAL</code>.</p>
how to deal with this problem

hui_day: <p>from torchtext import data<br>
PAD_TOKEN = ‘’<br>
TEXT = data.Field(sequential = True,batch_first = True,lower = True,pad_token = PAD_TOKEN)<br>
LABEL = data.Field(sequential = False,batch_first = True,unk_tokn=None)<br>
datafields = [(“PhraseId”, None), # 不需要的filed设置为None<br>
(“SentenceId”, None),<br>
(‘Phrase’, TEXT),<br>
(‘Sentiment’, LABEL)]</p>
<p>train, val, test = data.TabularDataset.splits(<br>
path=‘./data/’, train=‘train.tsv’,<br>
validation=‘val.tsv’, test=‘test.tsv’, format=‘tsv’,<br>
fields=datafields)<br>
print(train.shape,val.shape,test.shape)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/098f2997e9173429e4deea63cd2719e14c337155.png" data-download-href="https://discuss.pytorch.org/uploads/default/098f2997e9173429e4deea63cd2719e14c337155" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/098f2997e9173429e4deea63cd2719e14c337155.png" alt="image" data-base62-sha1="1mz1kl5Wvc69ZkXxFUBE7f3YmfH" width="690" height="64" data-dominant-color="30353A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">870×81 10.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AbdulsalamBande: <p>You are most likely using a new version of torchtext. Latest version of <code>torchtext</code> does not have the Field attributes. You can use this <a href="https://colab.research.google.com/github/pytorch/text/blob/master/examples/legacy_tutorial/migration_tutorial.ipynb" rel="noopener nofollow ugc">tutorial</a> on how to handle text data in the version.</p>
Create build Release smaller size

DejNea: <p>Hello, I’m building LibTorch 1.13.1. (CUDA 11.6) with<br>
cmake --build . --config Release<br>
and I get size for all dlls around 3.7GB. Is it possible to generate smaller size dlls than this? I’m also using CUDA with gpu NVidia Quadro T2000. I’m wondering if LibTorch builds any unnecessary code that can be unused.</p>
Different gradient mechanism in place replacement

panos: <p>I would like to replace the autograd mechanism with my own custom gradient calculations.  Is there an easy way to do that in pytorch?  I am looking for some examples where the gradient calculations can be supplied by another mechanism.  My DNN is very simple-fully connected with RELU and I want to run Adam with my own gradient mechanism.</p>
<p>Thank you.</p>

111357: <p>Hi, to custom gradient calculation, you can define a new <code>layer</code> by inheriting the <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc"><code>Function</code></a>.<br>
Here is an example <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">PYTORCH: DEFINING NEW AUTOGRAD FUNCTIONS</a>,<br>
You can find more detail about the <code>Function</code> at <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">here</a>.</p>

panos: <p>Thanks I had considered that. I assume there is no drop in replacement for autograd and pytorch and autograd are tightly integrated on the python end.  Is there a way to do this on the C++ side of things?</p>

111357: <p>Of course, you can custom op in c++ and then build it into a shared library.<br>
Here is an example <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html#using-the-torchscript-custom-operator-in-c" rel="noopener nofollow ugc"> Using the TorchScript Custom Operator in C++</a></p>
How to automatically detect residual connections in a model and apply user-defined function?

eric4: <p>I’m currently working on a project where I have a model that includes residual connections, and I’d like to customize the behavior of these connections using a custom function. Specifically, I want to replace the usual <code>res = res + conv(x)</code> operation with <code>res = f(res) + conv(x)</code>, where <code>f</code> is a user-defined function.</p>
<p>Here’s the current implementation of my model:</p>
<pre><code class="lang-auto">class CONV(nn.Module):

    def __init__(self,input_channels,output_channels):

        super(CONV, self).__init__()
        self.conv1= nn.Conv2d(input_channels, 32, 3,bias=False)
        self.batch_norm1 = nn.BatchNorm2d(32)

        self.conv2= nn.Conv2d(32, 32, 3,bias=False)
        self.conv3= nn.Conv2d(32, 32, 3,bias=False,padding=2)
        self.maxpool = nn.MaxPool2d(3, stride=2)
        self.conv4= nn.Conv2d(32, output_channels,3,bias=False)

    def forward(self, x):

        x = self.conv1(x)
        x_res = x

        x=self.batch_norm1(x)

        x = nn.functional.relu(x)


        x = self.conv2(x)
        x = nn.functional.relu(x)

        x = x_res+self.conv3(x)
        x = nn.functional.relu(x)
 
        x=self.maxpool(x)
        x = self.conv4(x)

        return x
</code></pre>
<p>After instantiation :<br>
<code>model = CONV(3, 3)</code></p>
<p>I’d like to write a function <code>residual(model, f)</code> that modifies the residual connections in the model after its creation(by modifying named_modules …)  because the function <code>f</code> is evolving over the time after each epoch…. How can I do this?</p>
<p>Any help would be greatly appreciated. Thank you!</p>

absolved: <p>Why not just instantiate the model with x = f(x_res) + self.conv3(x) ? I’m not understanding the use case. You can even rewrite your model init to include a boolean flag ‘use_f’ then code<br>
if self.use_f:<br>
x = f(x_res) + self.conv3(x)<br>
else:<br>
x = x_res + self.conv3(x)</p>
<p>Then initialize two models for comparison.<br>
model = CONV(3, 3, False)<br>
model_f = CONV(3, 3, True)<br>
Is there some reason this simpler idea won’t work for you?</p>

eric4: <p><a class="mention" href="/u/absolved">@absolved</a> I want to apply the function dynamically after the creation of the model, because the function f is evolving over the time (after each epoch)…and I’m not the one who creates the model class, but I use the model given by a user.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> is it feasible?</p>
Get input and output on backward hook

gggg111: <p>My goal is to implement the gradient * input relevance algorithm manually on my model. I know I can get the input of any layer using forward hooks. The problem is when creating a backward hook to obtain the gradients, I can not obtain the input and output values of the layer in the backward hook. What is the best way to obtain the input values of the layer in the backward hooks?</p>
<pre><code class="lang-auto">  def __fw_hook(self, layer, input, output):
      self.input = input
      self.output = output
</code></pre>
<pre><code class="lang-auto">  def __bw_hook(self, layer, input_grads, output_grads):
      self.input_grads = input_grads
      self.output_grads = output_grads
      # How to get the actual input in the current layer?

</code></pre>

ptrblck: <p>You could try to use e.g. a global <code>dict</code> storing the forward activations and index it in the backward hook, but this approach sounds a bit hacky.<br>
A cleaner way might be to implement custom <code>autograd.Function</code>s which would allow you to access the forward activations in the <code>backward</code> function directly, but you might then also need to reimplement the actual backward pass which also sounds like unnecessary work.</p>

gggg111: <p>I’m weighing both options, but I think I’ll simply use the first one. Thank you for your response!</p>
Use jit.trace to convert the model to a mobile-deployable model

Axboexx: <p>My pytorch_lite version is 1.9, when I convert with 1.9 version of pytorch, I get the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d96a4918d3075350b93dd5ddd0827fff54352b21.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d96a4918d3075350b93dd5ddd0827fff54352b21" title="QQ截图20230314150252"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d96a4918d3075350b93dd5ddd0827fff54352b21_2_690x326.jpeg" alt="QQ截图20230314150252" data-base62-sha1="v1lkgcEgZSf7qmxHPsD2XOVJDVf" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d96a4918d3075350b93dd5ddd0827fff54352b21_2_690x326.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d96a4918d3075350b93dd5ddd0827fff54352b21_2_1035x489.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d96a4918d3075350b93dd5ddd0827fff54352b21_2_1380x652.jpeg 2x" data-dominant-color="27252C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20230314150252</span><span class="informations">1696×803 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>transforms.RandAugment</code> was added in <code>torchvision==0.12</code> so you might need to update your <code>torchvison</code> (and PyTorch) installation.</p>
Different results using jit models

absolved: <p>Hi all, I have a custom regression model implemented in pytorch and I am noticing some discrepancies in performance that seem larger than random drift.</p>
<p>First let me describe my data. I have a training set(generated ‘randomly’), a dummy test set(generated ‘randomly’ in the same way as the training set), and a real world test set(significantly smaller than the dummy test set, not generated in the same way)</p>
<p>There are 3 cases I am comparing:<br>
case 1: training model(custom nn.Module) with no jit at all<br>
case 2: training torch.jit.script(model) where modules are custom nn.Module<br>
case 3: training model where nn.Modules have been rewritten as torch.jit.ScriptModules with all methods decorated with <span class="mention">@torch.jit.script_method</span>.</p>
<p>In case 1 and 2 the results are similar enough in terms of test/predict error and run times. Case 3 is where things get interesting. In case 3 I obtain significantly worse results with respect to a subset of the output features( ~10% worse MAPE than case 1 and 2) on the dummy test set. This is fine, it probably means there’s a bug in my ScriptModule code. BUT, using this ScriptModule to predict on the real world test set yields a result ~10% better than case 1 or 2. The issue here is twofold. First, I don’t actually care about my results on the dummy test set. Second, in this case I have ground truth knowledge for my real world test data, but in reality I will not have access to this knowledge and can only compare models in terms of their performance on dummy test data. How do I proceed in situations like this? I feel like I’ve opened pandora’s box here because I only started using jit to obtain some performance gains and was not expecting large differences in model results.</p>

ptrblck: <p>You could try to narrow down which module might create the different outputs between model 1/2 and 3 to further isolate the issue as it seems your custom <code>jit ScriptModule</code> approach might cause the issues.<br>
Once you’ve isolated the layer, could you post it here so that we could take a look at it?</p>
"New Tensor Cores eligible operator found" when running tensorboard on output of pytorch profiler

divinho: <p>What does this message mean? It happens when I run tensorboard --logdir on the output of a pytorch profile.</p>
<p>In my case the full message is: <code>New Tensor Cores eligible operator found: 'aten::thnn_conv2d_backward'!</code></p>
<p>Is this saying tensor cores aren’t being used when they could be or something?</p>

ptrblck: <p>I’m not sure of you are using DLProf or which library raises the warning so also don’t know how to interpret it. However, in case you are using an Ampere+ GPU TensorCores <em>could</em> be used via <code>TF32</code> in cuDNN and should be used when mixed-precision training is used.</p>

divinho: <p>Yes I’m using A100 GPUs. I’m using torch.profiler and <code>prof.export_chrome_trace("trace.json")</code>.</p>
<p>I am using mixed precision training. I also notice many kernels being launched for a single conv layer forward call, which seems wrong to me (why can’t the conv operation be done in one kernel?), which is why I’m asking myself if things are somehow setup suboptimally by not the right kernels being used.</p>

ptrblck: <p>If you are using <code>torch.backends.cudnn.benchmark = True</code> each new workload will profile multiple kernels which could be shown in the profile.<br>
If that’s not the case, please share a minimal and executable code snippet showing the unexpected behavior.</p>
How to backpropagate a loss through time-series RNN?

andrei-rusu: <p>I am trying to implement an RNN-based model for time-series data and any help would be much appreciated! I have a reward signal I would like to utilize to backpropagate a loss through the RNN every n steps. I cannot seem to find a way to backpropagate anything without detaching the hidden state, but I don’t think that is a good approach in this case.<br>
Let’s start from a simple example, straight from the PyTorch documentation, but amended such that one can step through the optimizer within the time loop:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

rnn = nn.GRUCell(10, 20)
optimizer = torch.optim.SGD(rnn.parameters(), lr=1e-3)
input = torch.randn(6, 3, 10)
hx = torch.randn(3, 20)
output = []
for i in range(6):
    hx = rnn(input[i], hx)
    output.append(hx)
    hx.mean().backward(retain_graph=retain_graph)
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>In the above script, if <code>retain_graph</code> is False, I get a “RuntimeError: Trying to backward through the graph a second time”. When <code>retain_graph</code> is True, I get “RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [20, 60]]”.<br>
I’ve seen some time-series RNN implementations online that don’t seem to bother with detaching the hidden state, so I suppose this used to work well in previous PyTorch versions.</p>

Andre_Amaral_IST: <p>Hey,<br>
To keep the states (cell and hidden) you only need to initialize the weights before the training phase or in the init method (since it is only called once).<br>
If you need any further help I can help you since I am developing work in that field.<br>
Regards<br>
André</p>

andrei-rusu: <p>Hello,<br>
Thank you for your quick reply but I don’t quite understand what you mean, sorry? Just to clarify, I want to utilize the updated hidden states for subsequent RNN steps while also being able to backpropagate through the network at various intervals (i.e. in an online learning fashion). So, I do initialize the states before the training commences (as the example above illustrates), but I want to be able to use the updated states for subsequent forward passes without reinitializing or detaching the states from the graph at each forward pass, if that makes sense. If you reckon you have a solution for the above, a quick code snippet would be really helpful! Thank you for your time!<br>
Andrei</p>

Messiah: <p>I am wondering if you were able to find a solution to this problem.</p>

J_Johnson: <p>If trying to backpropagate based on a reward function on a time series problem, you should look at reinforcement learning with DQN or PPO. Such a setup should store the states, actions and rewards, and then recursively train the model with the data before repeating(getting new states, actions and rewards).</p>
<p>Here is an example of a DQN:<br>
<a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a></p>
TorchDynamo - how to fallback to the fine grained subgraph?

aeron: <p>Assuming that user’s custom backend only support compile a convolution operator, is it possible to config the torch dynamo / torch.compile to make it fallback to the fine grained subgraph automatically?</p>

GLZhu: <p>I assume you mean to partition the subgraph and only run supported ops on your custom backend. For  this you might checkout this <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=zaMCj87J_JLU" rel="noopener nofollow ugc">colab</a>.</p>
Cuda out of Memeory Errro when Loading ImageNet dataset

External_happy: <p>I have a problem with loading ImageNet data. I want to extract features from the images and save them in a matrix (# of images, feature size). My GPU is Nvidia Quadro RTX 6000 with 24 GB ram. I get a “cuda out of memory” error when I run my code.<br>
I tried to fix it by doing these things:</p>
<ol>
<li>lowering the batch size (from 256 to 4) and the number of workers (from 8 to 2)</li>
<li>setting max_split_size_mb =2</li>
<li>clearing the cache and deleting unused variables<br>
But none of them worked. How can I solve this problem? Did I make any mistakes?</li>
</ol>

ptrblck: <p>How many features do you want to save and how large is each feature as storing these features alone could already use a large portion of the GPU memory (besides the model parameters etc.).<br>
Also, did you <code>.detach()</code> each feature before storing it or ran the forward pass in a <code>with torch.no_grad()</code> guard to avoid storing the intermediates?</p>
Torchaudio.load is not able to find the backend soundfile

Rakshith_V: <p>I’m trying to read an audio file<br>
data,sr = torchaudio.load(sample_wav)<br>
But it is throwing an error<br>
“RuntimeError: torchaudio.backend.soundfile_backend.load requires soundfile”</p>
<p>But<br>
str(torchaudio.get_audio_backend()) → gives<br>
‘soundfile’</p>
<p>torch 1.10.2<br>
WIndows (So can’t use sox)</p>

nateanl: <p>Hi <a class="mention" href="/u/rakshith_v">@Rakshith_V</a>, have you pip installed <code>soundfile</code> in your environment?</p>

Rakshith_V: <p>ya I did<br>
I’m able to run<br>
import soundfile</p>
<p>but I have issues with reading from torchaudio</p>

ejkitchen: <p>I am having the exact same issue on an Ubuntu image created with Docker.<br>
DISTRIB_ID=Ubuntu<br>
DISTRIB_RELEASE=20.04<br>
DISTRIB_CODENAME=focal<br>
DISTRIB_DESCRIPTION=“Ubuntu 20.04.5 LTS”</p>
<p>python3 test_cuda.py<br>
/usr/local/lib/python3.8/dist-packages/torchaudio/_internal/module_utils.py:99: UserWarning: Failed to import soundfile. ‘soundfile’ backend is not available.</p>
<p>warnings.warn(“Failed to import soundfile. ‘soundfile’ backend is not available.”)</p>
<pre><code>      PyTorch version: 1.11.0+cu113
  PyTorch can use GPU: True
   Torch CUDA version: 11.3

   Torchaudio version: 0.11.0+cu113
   Torchaudio backend: sox_io
</code></pre>
<p>Trochaudio backends list: [‘soundfile’, ‘sox_io’]</p>
<pre><code>  Torchvision version: 0.12.0+cu113
</code></pre>
<p>Torchvision video backend: pyav</p>
<pre><code>   Torch CUDA available: True
      CUDA device count: 1
    Current CUDA device: 0
            Device name: NVIDIA GeForce RTX 4090
</code></pre>
<p>pip list | grep -i sound<br>
SoundFile                 0.10.2<br>
pip list | grep librosa<br>
librosa                   0.9.2</p>
<p>apt list | grep libsnd</p>
<p>libsndfile1-dev/focal-updates,focal-security 1.0.28-7ubuntu0.1 amd64<br>
libsndfile1/focal-updates,focal-security 1.0.28-7ubuntu0.1 amd64<br>
libsndifsdl2-dev/focal 0.8.3-2 amd64<br>
libsndio-dev/focal 1.5.0-3 amd64<br>
libsndio7.0/focal 1.5.0-3 amd64<br>
libsndobj-dev/focal 2.6.7+ds1-3 amd64<br>
libsndobj2v5/focal 2.6.7+ds1-3 amd64</p>
<p>I am also running apt update at the very top of my Dockerfile. I also started the container and just to be sure ran apt update and apt-get update and same problem.</p>
<p>python3 -c “import soundfile”<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/usr/local/lib/python3.8/dist-packages/soundfile.py”, line 142, in <br>
raise OSError(‘sndfile library not found’)<br>
OSError: sndfile library not found</p>
<p>Now from the reading I have done online, Linux uses sox and not soundfile so why is torchaudio trying to access this in the first place?</p>

Avishka_Perera: <p>The same issue occurred to me in windows 10 after installing soundfile. Simply <strong>restarting the computer</strong> fixed the issue. The environment variables seems to be missing after a fresh installation with <code>pip install soundfile</code>.</p>
Input size / Batch size

yunusi: <p>Hi,<br>
I am training a semantic segmentatşon model with 256<em>256 input. I was using 16 as my batch size and I was getting good results. For an unrelated reason I need to feed my Model with 32</em>32 images, thus I divided my input images to 32*32 tiles and I am using 1024 as my batch size. Model is 2x slower now and accuracy is much worse. What could be the reason?</p>

ptrblck: <p>You could profile your code via the native PyTorch profiler or Nsight Systems to get a clear answer, but based on your description it seems you have increased the data loading and processing workload by quite a bit, which could have created a bottleneck.</p>
How to run functional.conv2d with different weights for each sample in batch?

kksinghal: <p>I have different weights for each sample in batch, so, how can I run conv2d with custom weights for each sample on GPU simultaneously? According to functional.conv2d, it convolves the same weights on each input. The slower way I found is to run functional.conv2d in a for loop over each data sample.</p>
<p>Thanks</p>

ptrblck: <p>You could move the batch dimension of the input into the channels and use the <code>groups</code> argument of <code>F.conv2d</code>.</p>

moi90: <p>Can you elaborate? I can’t get it to work…</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn.functional as F

# input shape (N,C_in,H,W)
input = torch.zeros((4,8,9,16))

# weight shape (N,C_out,C_in,1,1)
weight = torch.zeros((4,5,8,1,1))

# new_shape (C_out, N*C_in, h_k, w_k)
new_shape = (weight.shape[1],weight.shape[0]*weight.shape[2]) + weight.shape[3:]
result = F.conv2d(input, weight.reshape(new_shape), bias=None, stride=1, groups=weight.shape[0])

# Both C_out AND C_in are divided by groups
"RuntimeError: Given groups=4, expected weight to be divisible by 4 at dimension 0, but got weight of size [[5, 32, 1, 1]] instead"

# new_shape (N*C_out, N*C_in, h_k, w_k)
new_shape = (weight.shape[0]*weight.shape[1],weight.shape[0]*weight.shape[2]) + weight.shape[3:]
result = F.conv2d(input, weight.reshape(new_shape), bias=None, stride=1, groups=weight.shape[0])

# Does not work, either (obviously)
"RuntimeError: shape '[20, 32, 1, 1]' is invalid for input of size 160"
</code></pre>
<p>What I want is this:</p>
<pre data-code-wrap="python"><code class="lang-python"># Expected result shape (N,C_out,W,H)
result = torch.cat([F.conv2d(input[i].unsqueeze(0), w, bias=None, stride=1) for i,w in enumerate(weight)], dim=0)
result.shape
</code></pre>
<p>But it seems wasteful to run N separate convolutions:</p>
<blockquote>
<p>The slower way I found is to run functional.conv2d in a for loop over each data sample.</p>
</blockquote>
<p>Is there another way?</p>

moi90: <p>I have figured it out:</p>
<pre><code class="lang-python">
%%timeit
# Expected result shape (N,C_out,W,H)
expected_result = torch.cat([F.conv2d(input[i].unsqueeze(0), w, bias=None, stride=1) for i,w in enumerate(weight)], dim=0)

"854 µs ± 80.8 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)"

%%timeit
s_in = (1,-1,) + input.shape[2:]
s_w = (-1,) + weight.shape[2:]
s_out = (input.shape[0], weight.shape[1]) + input.shape[2:]
result = F.conv2d(input.reshape(s_in), weight.reshape(s_w), bias=None, stride=1, groups=weight.shape[0]).reshape(s_out)

"305 µs ± 52.8 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)"

torch.testing.assert_close(result, expected_result)
</code></pre>
<p>(Unscientific test using CUDA tensors on an Intel(R) Core™ i7-6700K CPU @ 4.00GHz and NVIDIA GeForce GTX TITAN X system.)</p>
<p>The key lies in modifying the input shape as well.</p>
How to select a subset of a 4-D tensor based on 3-D indices

TommyGun: <p>Hello PyTorch community!</p>
<p>I want to select a subset of a 4-D tensor (attention matrix of a multi-head-attention-block) based on the indices specified by a 3-D tensor B. Tensor A has shape (a, b, c, d) and a tensor B of shape (a, b, g). B contains the indices I want to select from A. Here g &lt; c meaning that I want to select a subset (size g) from the axis c of A.  The respective entries from dimension d of A should be kept. The resulting subset of A should therefore have shape (a, b, g, d).</p>
<p>If you know how to do this, I would really appreciate your help. I’ve tried to solve this on my own for hours but I could not figure it out. Thank you for your reply!</p>

ptrblck: <p>You haven’t posted a (slow) reference code, so I cannot verify my approach, but this might work:</p>
<pre><code class="lang-python">a, b, c, d = 2, 3, 4, 5
g = 6

A = torch.randn(a, b, c, d)
B = torch.randint(0, c, (a, b, g))
out = A[torch.arange(a)[:, None, None], torch.arange(b)[:, None], B]
print(out.shape)
# torch.Size([2, 3, 6, 5]) = [a, b, g, d]
</code></pre>

TommyGun: <p>Thank you for your reply!</p>
<p>Since I messed up the description, this was not quite the solution I was looking for. Luckily I found a solution by now. I also updated the post and here is the solution for anyone in the future who is looking for it:</p>
<p>def select_subset(A, B):<br>
# A.shape = (a, b, c, d)  (float)<br>
# B.shape = (a, b, g)     (int 0 &lt;= g &lt; c)<br>
# C.shape = (a, b, g, d)  (parallely for a, b select vectors d of A at indices g in c.)</p>
<pre><code>a, b, c, d = A.shape
a_, b_, g = B.shape
assert a == a_ and b == b_ and g &lt;= c, "Invalid shapes of A and B!"

indices = B.unsqueeze(-1).expand(a, b, g, d)  # shape: (a, b, g, d)
C = torch.gather(A, dim=2, index=indices)  # shape: (a, b, g, d)
return C
</code></pre>
Can't pickle local object 'BirdDataset.__init__.<locals>.<lambda>

Rubiksel: <p>Hi everyone,<br>
I’m new to pytorch and the world of AI-coding. I have been working as part of a team on the BirdCLEF competition and have so far implemented several neural networks (that was how I got familiar with deep learning and image recognition). However, now that I’ve explored around, I must now work on the code my team-leader has written: <a href="https://github.com/Arcadia24/CuiCuiProject" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Arcadia24/CuiCuiProject</a>. I have two problems when running the .ipynb in this code:</p>
<p>examples = next(iter(train_loader))<br>
print(examples[0].shape)</p>
<p>this part of the code seems to be an infinite loop, and I don’t understand why because it works fine on GitHub (please note my team-leader made this on Linux and I am on Windows).</p>
<p>My second issue is the following:<br>
In the train.fit() cell, I get the following error message: Can’t pickle local object 'BirdDataset.<strong>init</strong>..</p>
<p>Can someone maybe help me figure out what’s wrong? I would greatly appreciate it as I’m still not familiar with all the requirements and system errors that may come when using pytorch, pytorch_lightning and wandb.</p>
<p>By the way the dataset used is the BirdCLEF 2021 train dataset, and the transform function in the project converts the audio files to .png files of size 128*313. These pictures are stored as follows: spectrogram\label of species\filename.png</p>
<p>Thanks in advance,</p>
<p>Aksel</p>

ptrblck: <p>You might need to add the <code>if-clause</code> protection on Windows as described <a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">here</a>.</p>

Rubiksel: <p>Thank you for the response. I updated the cell to :</p>
<p>if <strong>name</strong> == “<strong>main</strong>”:</p>
<pre><code>trainer.fit(model = model_pl, 
            train_dataloaders = train_loader, 
            val_dataloaders = valid_loader)
</code></pre>
<p>but I still get the can’t pickle local object error</p>

Rubiksel: <p>I also tried with</p>
<pre><code class="lang-auto">def main():
     trainer.fit(model=model_pl,
                    train_dataloaders=train_loader,
                    val_dataloaders=valid_loader)

if __name__ == '__main__':
     trainer.fit(model = model_pl, 
                    train_dataloaders = train_loader, 
                    val_dataloaders = valid_loader)
</code></pre>

ptrblck: <p>I’m no a Windows expert so don’t know why the code work in Linux, but maybe <a href="https://stackoverflow.com/a/52283968">this post</a> could help as it has a few suggestions to fix similar issues.</p>