akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.quantization.convert does not work if my model has QConfig activation is set to torch.nn.Identity

Rami_Ismael: <p>My goal is to quantize the weight of the model and not to quantize the activation function in QAT. I went through the  Pytorch GitHub. However, it gives me an error<br>
AttributeError: ‘Identity’ object has no attribute ‘calculate_qparams’</p>

HDCharles: <p>you can’t just use identity because the quantization flow uses information from the qconfig to determine a number of things and makes assumptions about the types of objects in the qconfig, you should use the actual weight only qconfig instead</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.quantization.qconfig.default_weight_only_qconfig.html?highlight=weight_only_qconfig#torch.quantization.qconfig.default_weight_only_qconfig" rel="noopener nofollow ugc">float_qparams_weight_only_qconfig</a></p>
Gradients or parameters exchange

yuri123: <p>Hi,</p>
<p>I’m studying distribted training by using pytorchDDP.<br>
I have a question about dataparallel. when GPUs communicate with each other(all_reduce), do they exchange gradients or parameters? I thought It was gradients.</p>
<p>But, I set the bucket size to 25mb when I use pytorchDDP, I can check the Avg size is 92mb, not 25mb. I think that 92mb is the transferred model parameters size that is divided by buckets. what is right?</p>
<p>(I used VGG16 model that is about 528mb, and I profiled 10 step. so Calls per step of all_reduce is 6)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f40fd0dfa84863068d621de217b8fb73bfeb89a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/f40fd0dfa84863068d621de217b8fb73bfeb89a5" title="25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f40fd0dfa84863068d621de217b8fb73bfeb89a5_2_690x66.png" alt="25" data-base62-sha1="yP4o4lcNMZ6GXTRmRIAUk3hxHhP" width="690" height="66" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f40fd0dfa84863068d621de217b8fb73bfeb89a5_2_690x66.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f40fd0dfa84863068d621de217b8fb73bfeb89a5_2_1035x99.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f40fd0dfa84863068d621de217b8fb73bfeb89a5_2_1380x132.png 2x" data-dominant-color="F2F3F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">25</span><span class="informations">1462×140 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and I want to know the relationship of model parameter size and gradient size.<br>
Thank you.</p>

agu: <p>I saw your <a href="https://discuss.pytorch.org/t/pytorch-ddp-bucket-size-does-not-match-model-size/162420/3">comment</a> in the other thread, but could you provide a minimal script to reproduce the behavior?</p>

yuri123: <p>Hello Andrew,</p>
<p>I’m sorry but I can’t give you the script. but I’m just wondering why the bucket size and Avg size(of transferred data) are different. I thought the bucket size and Avg size would be the same.  I want to know what point I thought wrong.<br>
Thank you.</p>

agu: <p>Unfortunately, it is hard for me to say for sure without seeing the code or at least the full model definition. Would it be possible to print out the model and provide the parameter sizes?</p>
<p>One possibility off the top of my head is that there are some parameters that individually are large, which may cause a bucket to exceed the 25 MB limit. For your model, do you think this could be happening?</p>
Unfold and fold to apply the overlap and add algorithm

Dalisson_Figueiredo: <p>Hi, I have an audio processing problem, first I need to chunk and put through and rnn, i start with a tensor of shape<br>
(64, 16308) then I apply fold:</p>
<p>tensor.fold(-1, 178, 89)</p>
<p>so I get the tensor with shape (64, 182, 178) with nice overlapping sections</p>
<p>now I can pass it through my rnns. Problem is, now I would like to apply the overlap and add method to get the original tensor back, I thought nn.Fold would be a good match to try and do that, but I dont understand its use well enough to know if I can even use it to do that.</p>

ptrblck: <p>I assume you were using <code>tensor.unfold</code> to create the patches.<br>
If that’s the case, <code>nn.Fold</code> should be working as it’s the reverting operation to <code>nn.Unfold</code>.<br>
Where are you currently stuck and what have you tried so far?</p>

Dalisson_Figueiredo: <p>Last night, after a lot of try and error I got nn.Unfold to work, what I had to do was now that I had a tensor with shape (64,182, 178) I transposed it to (64, 178, 182) and now i can use nn.Fold(my_tensor, output_size=(1 ,16198), kernel_size=(1, 178) stride=(1, 89)) to get back a tensor with the original size of (1, 16198) with the overlapping sections added, completing the overlap and add algirithm, sorry that i said my original tensor was shaped (1, 16308) when it is actually (1, 16198). Still not very sure of how nn.Fold works though.</p>

Ahmed_m: <p>Hi,<br>
I am using nn.fold to apply overlap-add on a sequence of frames generated by a GRU layer. However, the direct addition of the overlapped frames without weighting leads to unstable training behavior.</p>
<p>Is there anyway I can have weighted overlap-add algorithm through nn.fold? Or it least to enable nn.fold with the possibility of taking the mean of the overlapped frames as done in multi-channel convolutional layers?</p>

markusMM: <p>Hey <a class="mention" href="/u/dalisson_figueiredo">@Dalisson_Figueiredo</a>, for 1D signals, I’d avoid using <code>nn.Fold</code> and <code>nn.Unfold</code>, as possible, since they mostly are for some 2D function, I guess.<br>
<code>torch.Tensor.unfold</code> does exactly what is expected in this case!</p>
<p>I wonder, if they will add <code>torch.Tensor.fold</code> or it might already exist!?!</p>
<p>Cheers!</p>
Split loss.backward() into two parts

GeorgeQ-Q: <p>For multitask learning, ie a network has multiple heads, it is useful to lower done the peak gpu memory usage if we can backpropogate the loss into a common node for each indivisual heads before during the forward phase. In order to achieve that, we need to cut the backward into two stage. For example,</p>
<pre><code class="lang-auto">import torch
a = torch.rand(4,requires_grad=True)
b = torch.rand(4,requires_grad=True)
c = a + b
d = 2*c
e = d.sum()

#calculate e, d grad
e.backward() ?

#...

#calculate a,b and c grad
c.backward() ? 
</code></pre>
<p>For the above code, I want to calculate the grad for node d and e first, and in some latter stage calculate the grad for a,b and c. Is there a way to achieve that behaviour?</p>

srishti-git1110: <p>Just to clarify, could you please explicitly state in mathematical symbols maybe what values you want?</p>
<aside class="quote no-group" data-username="GeorgeQ-Q" data-post="1" data-topic="163185">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/georgeq-q/48/37477_2.png" class="avatar"> GeorgeQ-Q:</div>
<blockquote>
<p>I want to calculate the grad for node d and e first</p>
</blockquote>
</aside>
<p>Do you mean you want to call backward on d and e or you mean calculating the gradient of some tensor with respect to d and e?</p>
'checkpoint' object has no attribute 'dir'

PPB: <p>Why did this error occur considering that the directory path was already provided?</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “main.py”, line 12, in <br>
checkpoint = utility.checkpoint(args)<br>
File “F:\DASR-main\utility.py”, line 71, in <strong>init</strong><br>
_make_dir(self.dir)<br>
AttributeError: ‘checkpoint’ object has no attribute ‘dir’</p>
</blockquote>
<pre><code class="lang-auto">class checkpoint():
    def __init__(self, args):
        self.args = args
        self.ok = True
        self.log = torch.Tensor()
        now = datetime.datetime.now().strftime('%Y-%m-%d-%H:%M:%S')

        if args.blur_type == 'iso_gaussian':
            self.dir = './experiment/' + args.save + '_x' + str(int(args.scale[0])) + '_' + args.mode + '_iso'
        elif args.blur_type == 'aniso_gaussian':
            self.dir = './experiment/' + args.save + '_x' + str(int(args.scale[0])) + '_' + args.mode + '_aniso'

        def _make_dir(path):
            if not os.path.exists(path): os.makedirs(path)

        _make_dir(self.dir)
        _make_dir(self.dir + '/model')
        _make_dir(self.dir + '/results')
</code></pre>

agu: <p>It looks like <code>args.blur_type</code> is neither <code>"iso_gaussian"</code> nor <code>"aniso_gaussian"</code>, so <code>self.dir</code> is not defined when the program reaches <code>_make_dir(self.dir)</code>.</p>

PPB: <p>The blur type is already provided in the command:<br>
<code>python main.py --dir_data='F:/DASR-main/dataset'  --model='blindsr' --scale='2' --**blur_type='iso_gaussian**' --noise=0.0 --sig_min=0.2 --sig_max=4.0</code></p>

agu: <p>Before <code>if args.blur_type == 'iso_gaussian':</code>, could you print <code>args.blur_type</code>? (It could be good to check that the args are correctly being parsed and passed through to the <code>checkpoint()</code> object.)</p>
<p>Otherwise, I am not sure based on your provided code snippet why <code>self.dir</code> would not be defined as an attribute <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
Cross entropy loss for 3D tensor

Saswat: <p>I have an output tensor (both target and predicted) of dimension (32 x 8 x 5000). Here, the batch size is 32, the number of classes is 5000 and the number of points per batch is 8. I want to calculate CELoss on this in such a way that, the loss is computed for every point and then averaged across 8 of them. How can I do this? Let me know if my question isn’t clear or ambiguous</p>

srishti-git1110: <p>Hi,<br>
CrossEntropyLoss expects an input of dim = (N, C) where N=batch_size and C=no. of classes,<br>
and a target of dim = (N,).</p>
<p>Additional dimensions are used for “K-dimensional loss” as stated in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">docs</a>.</p>
<aside class="quote no-group" data-username="Saswat" data-post="1" data-topic="163182">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>the loss is computed for every point and then averaged across 8 of them.</p>
</blockquote>
</aside>
<p>It’s not quite clear to me what the second dimension (8) represents but potentially check out the <code>reduction</code> parameter, that, by default is “mean” so you shouldn’t need modification for your use.</p>
Obtaining a gradient of a metric obtained by the backward pass

Miki97: <p>For my research I’m trying to obtain a derivative of a metric obtained by using XAI method called LRP that was calculated using a backward pass on the network. Here is the <a href="https://github.com/fhvilshoj/TorchLRP" rel="noopener nofollow ugc">github to the LRP implementation</a> I am using.</p>
<p>I have gotten myself familiar with the Pytorch’s autograd engine but I am not sure how I would implement this because this implementation wraps around standard Pytorch layers (Conv2D and Linear) and uses custom autograd.Functions to propagate relevance through the backward pass instead of the gradient. I would somehow need to propagate backward through the graph that was created by the forward pass and the first backward pass.</p>
<p>Any ideas on how you would implement this?</p>
Visualise Network Architecture

yviiendaleee: <p>I am trying to visualise my NN architecture. I use this code : <a href="https://github.com/jayroxis/PINNs/blob/master/Burgers%20Equation/Burgers%20Inference%20(PyTorch).ipynb" rel="noopener nofollow ugc">PINNs/Burgers Inference (PyTorch).ipynb at master · jayroxis/PINNs (github.com)</a> which is available on Github.</p>
<p>This is my NN:</p>
<p>class DNN(torch.nn.Module):</p>
<pre><code>def __init__(self, layers):
    super(DNN, self).__init__()
    
    # parameters
    self.depth = len(layers) - 1
    
    # set up layer order dict
    self.activation = torch.nn.Tanh
    
    layer_list = list()
    for i in range(self.depth - 1): 
        layer_list.append(
            ('layer_%d' % i, torch.nn.Linear(layers[i], layers[i+1]))
        )
        layer_list.append(('activation_%d' % i, self.activation()))
        
    layer_list.append(
        ('layer_%d' % (self.depth - 1), torch.nn.Linear(layers[-2], layers[-1]))
    )
    layerDict = OrderedDict(layer_list)
    
    # deploy layers
    self.layers = torch.nn.Sequential(layerDict)
    
def forward(self, x):
    out = self.layers(x)
    return out
</code></pre>
<p>How can I visualise it, best as a PNG file? I would like to see the layers, neurons, weights biases etc.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Saswat: <p>Refer <a href="https://discuss.pytorch.org/t/understanding-deep-network-visualize-weights/2060/6">this</a></p>
Layers whose .grad exists but grad_sample is None

Anirban_Nath: <p>I have a few layers in my model whose<code> .grad</code> exists but <code>.grad_sample</code> is None. How is <code>.grad_sample</code> different from <code>.grad</code> and how can one be None and the other not?</p>

alexandresablayrolle: <p>Thanks for reaching out! Can you share a minimal reproducing example? The <code>grad_sample</code> is populated by Opacus during the backward and it is aggregated, added to Gaussian noise into <code>grad</code> during <code>optimizer.step()</code>.</p>
How to use data previously sent to device in DataParallel module

tsp_t: <p>I’m in a situation where the dataset is too big to load on a single GPU, but can be split across multiple GPUs on the machine.</p>
<p>In this case, I want to train on a (DataParalleled) model distributed across each GPU using data from each device.<br>
DataParallel splits the given data into batch dimensions and sends it to a model distributed across each device.<br>
A problem arises in this part.<br>
I can think of a way to send the data already in each device to the CPU or one GPU, and then to each GPU again, but this is inefficient.<br>
Running on small data, the model was able to process 100000 samples, equivalent to 200 GB per second. But with the added step of sending to the cpu, it’s 50x slower.</p>
<p>Are there any DataParallel tricks suitable for use in my situation? Or is it better to use multithreading?</p>
Partial Weight Initialization

Erica_Zheng: <p>Dear All,<br>
I want to initialize FC layers in a customized way.</p>
<p>For example,<br>
<code>self.last_layer = nn.Linear(100, 10)</code></p>
<p>For the first 50 units’ weights, I want it to be all zeros, and the last 50 to be random.</p>
<p>Any idea how to implement this?</p>
<p>Thank you.</p>

ptrblck: <p>You could create a custom <code>nn.Parameter</code> filled with your desired values and assign it directly to the <code>.weight</code> attribute of the layer in a <code>with torch.no_grad()</code> context or you could also <code>.copy_</code> the desired tensor to the <code>.weight</code> in the <code>no_grad</code> guard.</p>
PILLOW_VERSION error

Abdelkareem_Elkhateb: <p>when I execute</p>
<pre><code class="lang-auto">from torchvision import datasets, transforms

</code></pre>
<pre><code class="lang-auto">ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (/home/kemo/anaconda3/envs/python10/lib/python3.10/site-packages/PIL/__init__.py)
</code></pre>
<p>I tried to downgrade PILLow_VERSION with conda into 6.2.1 but in vain, nothing happened!<br>
any help?<br>
I think this error happens after updating conda env.</p>

Abdelkareem_Elkhateb: <p>I solved it with the following code</p>
<pre><code class="lang-auto">conda install pytorch torchvision -c pytorch
</code></pre>
<p>I don’t know what is the difference with this code and</p>
<pre><code class="lang-auto">conda update --all
</code></pre>
Very slow inference instead of OoM

RunDevelopment: <p>I have an ESRGAN model that takes a lot of VRAM depending on how large the input image is. I noticed that VRAM scales roughly linearly with the number of pixels in the image until my VRAM is maxed out. However, to my surprise, I found that pytorch is still able to upscale larger images that (judging from the number of pixels) it does not have enough VRAM for. While the inference succeeds, it also takes longer than expected, again judging from the number of pixels in the input image.</p>
<p>So my question is: Does pytorch implement clever tricks to make inference succeed with low VRAM, e.g. by using algorithms that make different time-space tradeoffs?</p>
<p>If so, is there a way for me to turn them off? In my use case, I actually want pytorch to OoM error if the fastest inference isn’t possible, instead of slowly making its way through.</p>

ptrblck: <p>PyTorch will clear the memory cache and rerun the allocation once it’s running into an OOM. If the second allocation also fails the OOM error will be reraised to the user. Based on your description it sounds as if you might indeed be clearing the cache in each iteration.</p>

joeyballentine: <p>Is there a way to force it to raise an OOM error when the first allocation fails? Or maybe even a way to beforehand get the amount of memory needed for inference?</p>

ptrblck: <p>No, I’m not aware of a way to change this behavior in user code without changing and rebuilding the backend. You could try to delete unneeded intermediates manually in your code and also try to avoid creating memory fragmentation (e.g. by using the largest inputs and decreasing it in case you are working with variable input data). However, make sure the cache clearing is indeed what slows down the code by profiling it via e.g. Nsight Systems which should show the <code>cudaFree</code> calls.</p>
Torchaudio and torchtext installed but failed to import

jiapei100: <p>I’m quite sure <code>/opt/intel/oneapi/mkl/latest/lib/intel64</code> is already included under variable <strong>LD_LIBRARY_PATH</strong>. Anything wrong?</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchaudio as ta
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "~/.local/lib/python3.10/site-packages/torchaudio/__init__.py", line 1, in &lt;module&gt;
    from torchaudio import (  # noqa: F401
  File "~/.local/lib/python3.10/site-packages/torchaudio/_extension.py", line 131, in &lt;module&gt;
    _init_extension()
  File "~/.local/lib/python3.10/site-packages/torchaudio/_extension.py", line 101, in _init_extension
    _load_lib("libtorchaudio")
  File "~/.local/lib/python3.10/site-packages/torchaudio/_extension.py", line 52, in _load_lib
    torch.ops.load_library(path)
  File "~/.local/lib/python3.10/site-packages/torch/_ops.py", line 568, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_intel_thread.so.2: undefined symbol: __kmpc_reduce_nowait
</code></pre>
<p>Actually, <strong>torchtext</strong> has similiar issue:</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchtext as tt
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "~/.local/lib/python3.10/site-packages/torchtext/__init__.py", line 6, in &lt;module&gt;
    from torchtext import _extension  # noqa: F401
  File "~/.local/lib/python3.10/site-packages/torchtext/_extension.py", line 64, in &lt;module&gt;
    _init_extension()
  File "~/.local/lib/python3.10/site-packages/torchtext/_extension.py", line 58, in _init_extension
    _load_lib("libtorchtext")
  File "~/.local/lib/python3.10/site-packages/torchtext/_extension.py", line 50, in _load_lib
    torch.ops.load_library(path)
  File "~/.local/lib/python3.10/site-packages/torch/_ops.py", line 568, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ~/.local/lib/python3.10/site-packages/torchtext/lib/libtorchtext.so: undefined symbol: _ZTVN5torch8autograd12AutogradMetaE
</code></pre>
<p>Cheers</p>
Sending Fully Connected Layer to CPU

tiramisuNcustard: <p>In CNN, can the FC (Fully Connected Layer) be sent to the CPU after convolution layer is done on the GPU? Can I use the following? Thank you.</p>
<pre><code class="lang-auto">cpu = torch.device("cpu")
self.fc1.to(cpu)
</code></pre>

agu: <p>Yes, the code snippet you provided will move <code>self.fc1</code> (i.e. its <code>weight</code> and <code>bias</code> parameters) to CPU because <code>.to(cpu)</code> will be an in-place operation.</p>
Transform Indices on Indexes

ryanbty: <p>Let</p>
<p>x =  tensor([[[[0, 1, 1, 1],<br>
[1, 1, 1, 1],<br>
[1, 1, 1, 1],<br>
[1, 1, 1, 1]]]])</p>
<p>if we do x.nonzero() we get</p>
<p>x.nonzero() =  tensor([[<br>
[0, 0, 0, 1],<br>
[0, 0, 0, 2],<br>
[0, 0, 0, 3],<br>
[0, 0, 1, 0],<br>
[0, 0, 1, 1],<br>
[0, 0, 1, 2],<br>
[0, 0, 1, 3],<br>
[0, 0, 2, 0],<br>
[0, 0, 2, 1],<br>
[0, 0, 2, 2],<br>
[0, 0, 2, 3],<br>
[0, 0, 3, 0],<br>
[0, 0, 3, 1],<br>
[0, 0, 3, 2],<br>
[0, 0, 3, 3]])</p>
<p>It is possible to have a 1-D Array where we return the number of the index of x :</p>
<p>tensor([[[[ 1,2,3],[4,5,6,7],<br>
[ 8,9,10,11],[12,13,14,15]]])</p>

ptrblck: <p>Your desired output format won’t work as it’s a nested tensor. However, you could flatten <code>x</code> to get at least the desired output values in a flattened shape:</p>
<pre><code class="lang-python">x.view(-1).nonzero()
# tensor([[ 1],
#         [ 2],
#         [ 3],
#         [ 4],
#         [ 5],
#         [ 6],
#         [ 7],
#         [ 8],
#         [ 9],
#         [10],
#         [11],
#         [12],
#         [13],
#         [14],
#         [15]])
</code></pre>
BackPropagation of custom gradients in pytorch

kgoyal40: <p>Hi, I’m working on a project where I need to calculate the gradient of the last layer of a network using a custom approach and then back propagate this custom gradient through the network. Is there a way this can be achieved in PyTorch?</p>

AlphaBetaGamma96: <p>You’ll want to use a <code>torch.autograd.Function</code> object and manually define the forward and backward formulae for your custom funcion.</p>
<p>You can read more in the docs, <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.12 documentation</a></p>
Multiple patches from a tensor

Max_Reynolds: <p>I am working with 3d images. If I have a single image tensor of size d x w x h, I would like to extract many (~100) same-sized (e.g. 32 x 32 x 32) 3d patches defined by a tensor of size (n_patches x 3) corresponding to the minimum value (in each of 3 dimensions) of each patch.</p>
<p>Ideally, the output would be a n_patches x 32 x 32 x 32 tensor containing all of the extracted patches.</p>
<p>Is there a more elegant/quick way to do this indexing without looping n_patches times and concatenating the patches?</p>

Max_Reynolds: <p>A toy example in 2D:</p>
<p>say image tensor X is</p>
<pre><code class="lang-auto">tensor([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
        [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
        [30, 31, 32, 33, 34, 35, 36, 37, 38, 39]])
</code></pre>
<p>and the patch array is</p>
<pre><code class="lang-auto">tensor([[1, 6],
        [2, 0],
        [2, 8]])
</code></pre>
<p>then with patches of size 2,<br>
the result would be</p>
<pre><code class="lang-auto">tensor([[[16, 17],
         [18, 19]],

        [[20, 21],
         [30, 31]],

        [[28, 29],
         [38, 39]]])
</code></pre>

kyjsj: <p><code>torch.nn.Unfold</code> may be useful.</p>
Low GPU utilization possibly limited by data loading?

usr0909: <p>I am pretty new to PyTorch and sorry if this question is too naive. My training data is saved in .npy format in the following way:<br>
– root directory:<br>
---- class 1:<br>
-------- *npy<br>
---- class 2:<br>
-------- *npy</p>
<p>So I used torchvision.datasets.DatasetFolder to load my training data. When I implement the training, the average GPU utilization is only ~30%. I suspect it is limited by data loading. Is there any way to verify this? And also, what would be the best practice to load data efficently?</p>
RuntimeError: bad allocation in LSTMAE

ran_ran1: <p>Hi,<br>
when I run <strong>LSTMAE</strong> I get this error ,what is the problem?</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\multivariate anomaly detection for evnt logs\LSTAME.py”, line 258, in <br>
train_loss = train(epoch, model, optimizer)<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\multivariate anomaly detection for evnt logs\LSTAME.py”, line 230, in train<br>
recon_data = model(batch_data)<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\multivariate anomaly detection for evnt logs\models.py”, line 302, in forward<br>
encoded_input, hidden = self.encoder(input)<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\multivariate anomaly detection for evnt logs\models.py”, line 267, in forward<br>
encoded_input, hidden = self.lstm(input, (h0, c0))<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\rnn.py”, line 769, in forward<br>
result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,<br>
RuntimeError: bad allocation</p>
</blockquote>
<p>this is forward function:</p>
<blockquote>
<p>def forward(self, input):<br>
encoded_input, hidden = self.encoder(input)<br>
decoded_output = self.decoder(encoded_input, hidden)<br>
return decoded_outpu</p>
</blockquote>
<p>and in class EncoderLSTM:</p>
<blockquote>
<p>def forward(self, input):<br>
tt = torch.cuda if self.isCuda else torch<br>
h0 = Variable(tt.FloatTensor(self.num_layers, input.size(0), self.hidden_size).zero_(), requires_grad=False)<br>
c0 = Variable(tt.FloatTensor(self.num_layers, input.size(0), self.hidden_size).zero_(), requires_grad=False)<br>
encoded_input, hidden = self.lstm(input, (h0, c0))<br>
return encoded_input, hidden</p>
</blockquote>
<p>can I get help?</p>
What's the difference between self attention and spatial attention?

ali_hejrati: <p>At a glance, self-attention is proposed for representing contextual information of a context. for example, if we consider the sentence below: “I swam across the river until I reach the next bank” here the word bank in d dimensional space is too far than the river and the goal of self-attention is to bring them close together in terms of geometrically, while the goal of spatial attention in a visual task can consider like to focusing on the area of interest and ignoring other irrelevant parts something like an image cropping around the area of interest.</p>
<p>Is this description correct?</p>
<p>I found <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">this</a> for self-attention but I can’t find any implementation for spatial attention. is there a function for spatial attention in pytorch?</p>
Strange Behavior of nn.Linear()

XHH: <p>Hi,<br>
My code is simple and as follows:<br>
import torch<br>
device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
m = torch.nn.Linear(20, 30).to(device)<br>
input = torch.randn(128, 40).to(device)<br>
output = m(input)<br>
print(output.shape)</p>
<p>In this code, it is expected to return an error due to shape mismatch on gpus. However, it could work properly and return the output.shape = (128,30). But when shifted to cpu, it returns the mismatch error. And if the input shape is shifted to (128, N), where N &lt; 20, then it returns an error on GPUs.<br>
My torch.<strong>version</strong> =  1.9.0+cu111  &amp;  torch.version.cuda = 11.1 , My GPUs are NVIDIA DGX A100 640GB</p>

ptrblck: <p>Could you update PyTorch to the latest stable or nightly version, please?<br>
This was a known issue which was fixed in a patch release a while ago if I’m not mistaken.</p>
Imagenet 2012 test set labels

binbbaz: <p>I have the imagenet train, validation and test set. I have been able to assign each image in the validation set in its respective class folders with the help of some online resources. However, for the test set, I could not find the labels.</p>
<p>All the model evaluation experiments on Imagenet dataset I’ve seen people do only use a fraction of the validation set. I have not been able to find one that used the test set or a fraction of it.</p>
<p>I really want to use the imagenet test set to evaluate my model. How do I go about it or where can I get the labels for the test set?</p>

srishti-git1110: <p>I haven’t worked with ImageNet but a possible approach (in case you aren’t able to access test set labels) could be to partition your train set into training and validation divisions, and use the validation set as the test set.</p>

binbbaz: <p>Thank you for your response.</p>
<p>I have actually done this by taking some samples from the validation set and making evaluation on it.</p>
<p>I am doing evaluation on ViT. It is mentioned in the paper that imagenet training set was <strong>part</strong> of the data used for the training. Though the usage of the validation set wasn’t mentioned but I want to believe that It might have been used for hyperparameter tuning.</p>
<p>So, I just want to make evaluation on samples I am very sure the model has not seen before.</p>
<p>Don’t know if what I am trying to do is right</p>
RuntimeError: Function 'AcosBackward0' returned nan values in its 0th output.

xian_zhang: <p>When I calculated the spectral angles of the two images in the program, I get a error:"RuntimeError: Function ‘AcosBackward0’ returned nan values in its 0th output."I tried the suggestions on the Internet, and used the torch.clamp function to limited the data range to (-1, 1) before the arccos operation. but the problem is not solved.Below is my code：</p>
<pre><code>def forward(self, lr_image, sr_image) -&gt; torch.Tensor:
    # 将 SR 下采样至与 LR 尺寸相同
    sr_image = F.interpolate(sr_image, scale_factor=1 / 3, mode='bicubic')

    assert lr_image.ndim == 4 and lr_image.shape == sr_image.shape

    dot_sum = torch.sum(lr_image * sr_image, dim=1)

    # 求范数
    norm_lr = torch.linalg.norm(lr_image, dim=1)
    norm_sr = torch.linalg.norm(sr_image, dim=1)

    res = torch.arccos(torch.clamp(input=dot_sum / norm_lr / norm_sr, min=-0.99999, max=0.99999))

    is_nan = torch.nonzero(torch.isnan(res), as_tuple=True)

    for c, x, y in zip(is_nan[0], is_nan[1], is_nan[2]):
        res[c, x, y] = 0

    return torch.mean(res)
</code></pre>
Model performance significantly drop when updating from pytorch==1.10.1+cu111 to pytorch==1.12.1+cu113 on a Linux machine with cuda==11.4 installed

pt_z: <p>I was running pytorch on a Ubuntu 20.04 machine that was equipped with a cudatoolkit==11.4. Previously I installed <code>pytorch==1.10.1+cu111</code> and everything goes fine. However, yesterday I reinstalled pytorch with <code>pytorch==1.12.1+cu113</code> and the model performance drops <code>10</code> percentage.</p>
<p>There are two things to note:</p>
<ol>
<li>not all the models’ performance dropped, only several of them</li>
<li>I’m not training any new model, I just tested the existing checkpoint with the same code and input.</li>
</ol>
<p>After reinstalling the <code>pytorch==1.10.1+cu111</code>, everything is fine again.</p>
<p>Anyone can tell the reason? Or providing some resources of the compatability of different cuda version?</p>

ptrblck: <p>Could you describe your use case in more detail and share the model definition if possible?<br>
It would also be interesting to see outputs of the model in <code>eval()</code> mode for a constant input (e.g. <code>torch.ones</code>) and see what multiple forward passes return in both PyTorch versions.</p>

pt_z: <p>Good point. However, I have removed the torch==1.12.1+cu113 env; I may reinstall it but it should be a couple of days later.</p>
Is there a way to make Pytorch return zero instead of None

alireSa_Ranjbar: <pre><code class="lang-auto">def f (x, u):
    return x.pow(2)+u.pow(2)
y = f(x,u)
yx = grad(y, x, create_graph=True)[0]
print(yx)
yxu = grad(yx,u,allow_unused=True)[0]
print(yxu)

output:
        tensor([4.], grad_fn=&lt;ThMulBackward&gt;)
        None
</code></pre>
<p>Mathematically the above second derivative should be zero, right?</p>

albanD: <p>A convention we use in pytorch to reduce memory usage and improve speed during backward is that <code>None</code> is equal to a Tensor full of zeros.<br>
That allows many optimizations during the backward pass.<br>
Is it a problem in your usecase?</p>

alireSa_Ranjbar: <p>There are I guess other reasons that a None is returned for such gradient, for example, because of a mistake in the code. But, apparently if the function’s gradient is independent of the<br>
target variable, a None is also returned (which should actually be a zero mathematically).</p>
<p>So the main problem is that one will not be sure if the None is because of a bug in the code or because of the independence of the function with respect to the variable.</p>

albanD: <p>I’m not sure what you mean by “a bug in the code”?</p>

alireSa_Ranjbar: <p>I mean for example I have a mistake in the code that I have written and although my function should return a valid derivative, it returns a None. But maybe it’s not my mistake in my code and is rather the fact that my function is not dependent on the variable I am calculating the derivative for.</p>
<p>So when I get a None, I wouldn’t know if the reason is because of a mistake in my code or the fact that my function is independent of the variable.</p>

albanD: <p>But from pytorch’s point of view, the two are exactly the same <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
If there is not link between two variables because of an error or because of the function you implement are indistinguishable.</p>

alireSa_Ranjbar: <p>From mathematical point of view, if the first derivative of a function is a constant number, the second derivative is zero, not None.</p>

albanD: <p>But if you make a mistake and implement the wrong function and ask for <code>d f(x) / d y</code> (use the wrong variable in your function), then mathematically, the derivative is also 0.<br>
I’m not sure to see where the difference is?</p>

Ilker_GURCAN: <p><a class="mention" href="/u/alband">@albanD</a> , I see your point and your concern about memory usage, etc.; but what about adding an optional named parameter default to “False”? e.g. “none_to_zero=False”. It would not hurt your framework, would it? By doing so, we could make good use of it within our own custom differentiation stuff.</p>
Logic check for multi-class Dice Loss

azhanmohammed: <p>I have a multi-class segmentation problem, and I want to use Dice Loss to solve the class imbalance. I found a simple implementation of dice loss given by:</p>
<pre><code class="lang-python">def criterionDice(prediction, groundTruth):
    diceScore = (2*(prediction * groundTruth).sum())/((prediction + groundTruth).sum() + 1e-8)
    diceLoss = 1 - diceScore
    return diceLoss
</code></pre>
<p>This works well when I have a simple binary segmentation problem, but when I move to multi-class problem, I face issues because my prediction and ground truth do not have the same dimensions, rather than using one hot encoding for labels, I left them as single dimension maps, so now my predictions and ground truth have the following shape:</p>
<pre><code class="lang-python">Prediction shape: Batch Size, Number of Class, 256, 256
Ground truth shape: Batch Size, 256, 256
</code></pre>
<p>To ensure my prediction and ground truth have the same size while calculating loss, I did the following:</p>
<pre><code class="lang-python">def criterionDice(prediction, groundTruth):
    probs = torch.nn.functional.softmax(prediction, dim=1)
    probs = torch.argmax(probs, dim=1)
    diceScore = (2*(probs * groundTruth).sum())/((probs + groundTruth).sum() + 1e-8)
    diceLoss = 1 - diceScore
    return diceLoss
</code></pre>
<p>But I am really not sure if this is the right way to solve the issue. Can someone please check if this is the correct logical implementation for dice loss. Any help would be really appreciated.</p>
I'm trying to get the gradient of a certain layer of my model, can somebody explain the difference between these 2 approachs?

Chedi_Morchdi: <p>I’m trying to get the gradient of a certain layer of my model, can somebody explain the difference between these 2 approachs ?</p>
<p>1st approach is:<br>
current_grads=[ ]<br>
for param in net1.conv_layer[0].parameters():<br>
\indent current_grads.append(param.grad.view(-1))<br>
current_grads=torch.cat(current_grads)</p>
<p>2nd approach is:<br>
current_grads=torch.flatten(net1.conv_layer[0].weight.grad)</p>
<p>it seems that the resultant tensor of the 2nd approach is contained in the resultant temsor of the 1st approach. can someone explain why ? thank you.</p>

ptrblck: <p>The first approach will iterate all parameters of the layer (which is often the <code>weight</code> and <code>bias</code> parameter), will flatten the corresponding gradients, and will concatenate them to a single tensor:</p>
<pre><code class="lang-python">layer = nn.Conv2d(1, 3, 3)
out = layer(torch.randn(1, 1, 24, 24)).mean()
out.backward()

current_grads = []
for param in layer.parameters():
    current_grads.append(param.grad.view(-1))
print(current_grads)
# [tensor([0.0149, 0.0038, 0.0107, 0.0230, 0.0130, 0.0209, 0.0298, 0.0194, 0.0287,
#         0.0149, 0.0038, 0.0107, 0.0230, 0.0130, 0.0209, 0.0298, 0.0194, 0.0287,
#         0.0149, 0.0038, 0.0107, 0.0230, 0.0130, 0.0209, 0.0298, 0.0194, 0.0287]), tensor([0.3333, 0.3333, 0.3333])]
current_grads=torch.cat(current_grads)
</code></pre>
<p>while the second approach will only store the flattened <code>weight</code> gradient.</p>
Tips for wrapping Conv layers in FSDP

rowhanm: <p>Hey thanks for putting together the <code>transformer_auto_wrap_policy</code> for FSDP. I wanted to check if there are any tips as to which layers we can combine when we’re wrapping Conv blocks, or if wrapping the whole blocks in an FSDP unit should be good.</p>
<p>Any insight would be great!</p>

agu: <p>Is it possible to provide a print out of your model definition? I may be able to provide more directed help in that case.</p>

rowhanm: <p>I was thinking of something like the CoAtNet models, the timm implementation for instance [1]</p>
<p>They are kinda tedious to print out so here’s the whole thing - <a href="https://gist.github.com/rowhanm/6380ccd79937c4f38c1d6e9fc255a95f" class="inline-onebox" rel="noopener nofollow ugc">timm coatnet definition as per https://arxiv.org/pdf/2106.04803v2.pdf · GitHub</a></p>
<p>The basic structure is that it has a few MbConv blocks, sequentially followed by Transformer blocks. I wrapped the transformer layers as described in the <code>transformer_auto_wrap_policy</code> and tried wrapping the Conv blocks with <code>size_based_auto_wrap_policy</code> but felt that was inefficient.</p>
<p>Looking forward to your suggestions!</p>
<p>[1] - <a href="https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/maxxvit.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-image-models/maxxvit.py at master · rwightman/pytorch-image-models · GitHub</a></p>

agu: <p>Is the gist representative of your scaled model? If not and you are planning to scale the model further, would it be by increasing individual <code>nn.Parameter</code> sizes, adding more layers, or both?</p>
<p>I was thinking that something like the following might be good for scaling:</p>
<pre><code class="lang-auto">import functools
auto_wrap_policy = functools.partial(
    transformer_auto_wrap_policy,
    transformer_layer_cls={
        Stem,
        MaxxVitStage,
        TransformerBlock2d,  # possibly
        ClassifierHead,
    },
)
</code></pre>
<p>If the parameter sizes are exactly as in that gist, then including <code>TransformerBlock2d</code> may not be worth it. Let me know if you encounter any issues!</p>

rowhanm: <p>This is interesting, so it should be safe to add non-transformer classes to <code>transformer_auto_wrap_policy</code>? Wasn’t sure if adding Conv layers to it would be safe so I was thinking of adding a wrapper class around <code>transformer_auto_wrap_policy</code> and <code>size_based</code>. This is insightful thanks!</p>

agu: <p>The name is a misnomer, and we need to address that. Apologies for the confusion.</p>
How to debug "RuntimeError: CUDA error: invalid configuration argument"?

Max_Unhold: <p>Hello, I get the following error (using <code>os.environ['CUDA_LAUNCH_BLOCKING'] = "1"</code>)</p>
<pre><code class="lang-python">RuntimeError                              Traceback (most recent call last)
notebook.ipynb Zelle 13 in &lt;cell line: 19&gt;()
     22 pred = model(input.to(device))
     23 loss = criterion(pred,label.to(device))
---&gt; 24 loss.backward()
     25 optimizer.step()
     26 optimizer.zero_grad()

File c:\ProgramData\Anaconda3\lib\site-packages\torch\_tensor.py:396, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    387 if has_torch_function_unary(self):
    388     return handle_torch_function(
    389         Tensor.backward,
    390         (self,),
   (...)
    394         create_graph=create_graph,
    395         inputs=inputs)
--&gt; 396 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

File c:\ProgramData\Anaconda3\lib\site-packages\torch\autograd\__init__.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    168     retain_graph = create_graph
    170 # The reason we repeat same the comment below is that
    171 # some Python versions print out the first line of a multi-line function
    172 # calls in the traceback and some print out the last line
--&gt; 173 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175     allow_unreachable=True, accumulate_grad=True)

RuntimeError: CUDA error: invalid configuration argument
</code></pre>
<p>How can I debug this error that appears when I calculate the backward of sparse tensors on cuda? The error does not appear when using the cpu.</p>

ptrblck: <p>Could you post a minimal, executable code snippet as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>
Using hugginface pipeline in pytorch mps device

asakal: <p>Hi<br>
i want to run pipeline abstract for zero-shot-classification task on the mps device. Here is my code</p>
<pre><code class="lang-auto">pipe = pipeline('zero-shot-classification', device = mps_device)
seq = "i love watching the office show"
labels = ['negative', 'positive']
pipe(seq, labels)
</code></pre>
<p>The error generated is</p>
<pre><code class="lang-auto">RuntimeError: Placeholder storage has not been allocated on MPS device!
</code></pre>
<p>Which my guess is because seq is on my cpu and not mps. How can i fix this ?<br>
Is there a way to send seq to the mps device so that i can pass it to the pipe for inference?</p>
<p>Thanks</p>

krispy: <p>Here is the simplest solution I could come up with to address the problem (I was getting the same error and happened across your question):</p>
<pre><code class="lang-auto">import os
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"

import torch
from transformers import pipeline

mps_device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")

pipe = pipeline('zero-shot-classification', device = mps_device)
if torch.backends.mps.is_available():
  print("Loading model into MPS device...")
  pipe.model = pipe.model.to(mps_device)

seq = "i love watching the office show"
labels = ['negative', 'positive']
pipe(seq, labels)
</code></pre>
Error pytorch install

Set-7: <p>Hi everyone, i’m trying to install SuperStableDiffusion. I completed all the steps (git, python and archives but when I run git, I have an error message that says “can’t install Pytorch”</p>
<p>This is the environment I am using <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> windows 11<br>
Python 3.10.7<br>
Git installer<br>
I did all the double check but I can’t find how it doesn’t run. That stops me to go ahead for stable diffusion</p>

ptrblck: <p>Could you explain your install steps a bit more, please?<br>
“Can’t install PyTorch” seems to be raised by a custom install script?<br>
Are you trying to build from source? If so, could you post the build log including the failure?<br>
In case you are trying to install the binaries, could you also post the install log including the errors?</p>
How to get the memory consumption of a sparse tensor in pytorch?

Max_Unhold: <p>I would like to know how one can measure the memory consumption of a sparse tensor in pytorch</p>
<pre><code class="lang-auto">i = [[0, 1, 1],
v =  [3, 4, 5]
s = torch.sparse_coo_tensor(i, v, (2, 3))
</code></pre>
CUDA_LAUNCH_BLOCKING in Jupyter Notebook

Max_Unhold: <p>I would like to debug the error</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
</code></pre>
<p><a href="https://discuss.pytorch.org/t/whats-the-meaning-of-this-error-how-can-i-debug-when-i-use-gpu/8052/3">In this answer</a> , it is suggested to use</p>
<pre><code class="lang-auto">os.environ['CUDA_LAUNCH_BLOCKING'] = 1
</code></pre>
<p>at the beginning of a jupyter notebook (I use VS Code) in order to debug but it throws the error</p>
<pre><code class="lang-auto">\my_notebook.ipynb Zelle 13 in &lt;cell line: 1&gt;()
----&gt; 1 os.environ['CUDA_LAUNCH_BLOCKING'] = 1
      3 # %load_ext autoreload
      4 # %autoreload 2

File c:\ProgramData\Anaconda3\lib\os.py:684, in _Environ.__setitem__(self, key, value)
    682 def __setitem__(self, key, value):
    683     key = self.encodekey(key)
--&gt; 684     value = self.encodevalue(value)
    685     putenv(key, value)
    686     self._data[key] = value

File c:\ProgramData\Anaconda3\lib\os.py:742, in _createenviron.&lt;locals&gt;.check_str(value)
    740 def check_str(value):
    741     if not isinstance(value, str):
--&gt; 742         raise TypeError("str expected, not %s" % type(value).__name__)
    743     return value

TypeError: str expected, not int
</code></pre>

ptrblck: <p>As the error message points out, pass a string if you want to set it in Python:</p>
<pre><code class="lang-python">os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>
<p>and make sure it’s set before the CUDA context is initialized as it won’t have any effect otherwise.</p>