akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cannot load .mp3 with torchaudio due to no appropriate backend found

username: brummelisa
cooked: <p>Hi,</p>
<p>I’m trying to load the metadata of an mp3 file in a list of mp3 files with torchaudio using the following code:</p>
<p><span class="hashtag-raw">#Load</span> the metadata of the audio files<br>
meta_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for audio_file in os.listdir(audiodir):<br>
if audio_file != ‘.DS_Store’:<br>
meta_list.append(torchaudio.info(audiodir + ‘/’ + audio_file, format=‘mp3’))</p>
<p>However, when I try to do this, I get the following error:</p>
<p>Exception has occurred: RuntimeError<br>
Couldn’t find appropriate backend to handle uri /mnt/data/duck.mp3 and format mp3.<br>
File “/workspaces/audio-generation-vst/train_generator.py”, line 18, in <br>
meta_list.append(torchaudio.info(audiodir + ‘/’ + audio_file, format=‘mp3’))<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: Couldn’t find appropriate backend to handle uri /mnt/data/duck.mp3 and format mp3.</p>
<p>I am running torchaudio in a Docker dev container with Python 3.11 on a Macbook with M1. I have tried installing the following packages in the Dockerfile:</p>
<p>RUN pip install soundfile<br>
RUN pip install sox<br>
RUN pip install ffmpeg</p>
<p>However, it’s not wokring. Is torchaudio supported on M1?</p>
<p>Best</p>

username: StillinBed
cooked: <p>I have the same problem but on a M2. Did you solve it?</p>
Why pointwise nn.Conv1d is not using matmul kernels?

username: vadimkantorov
cooked: <p>Example in <a href="https://github.com/pytorch/pytorch/issues/116369#issuecomment-1868543434" class="inline-onebox" rel="noopener nofollow ugc">CPU implementation of Conv1d seems to work non-deterministically · Issue #116369 · pytorch/pytorch · GitHub</a>. For both cases there, conv2d kernels are used</p>
<p>What would be used for the <code>torch.matmul</code> with transposed inputs?</p>
<p>I think in theory both should use the same gemm path accepting and producing transposed values (ideally, it should depend on “actual” memory contiguity format?)</p>

username: albanD
cooked: <p>Hey!</p>
<p>My expectation is that historically, the conv2d kernel was the most optimized of them all and so it was the simplest and overall fastest fallback.<br>
But if there are easy to identify cases where matmul is faster, then we can definitely add a branch there to call matmul instead!</p>

username: vadimkantorov
cooked: <p>If gemm is not faster for these stride patterns, then matmul should use this conv path <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: vadimkantorov
cooked: <p>opened <a href="https://github.com/pytorch/pytorch/issues/116506" class="inline-onebox" rel="noopener nofollow ugc">[discussion] Route pointwise Conv1d/Conv2d to matmul? · Issue #116506 · pytorch/pytorch · GitHub</a> for further discussion</p>
PyTorch's backward implementation of cuda min/max functions?

username: XenadonWang
cooked: <p>Hi,</p>
<p>I’m looking to understand PyTorch’s backward pass implementation for min(), max(), minimum(), and maximum() on CUDA tensors. The forward pass is straightforward, but the design of backwards seems tricky. I’ve searched aten/src/ATen/native/cuda but didn’t see any file related to the corresponding backwards. If anyone can point me toward where the backward logic for these CUDA ops is implemented, I’d appreciate it.</p>

username: janeyx99
cooked: <p>Hi! The backward implementations for <a href="https://github.com/pytorch/pytorch/blob/main/tools/autograd/derivatives.yaml#L1146" rel="noopener nofollow ugc">min</a>/<a href="https://github.com/pytorch/pytorch/blob/main/tools/autograd/derivatives.yaml#L1146" rel="noopener nofollow ugc">minimum</a> and <a href="https://github.com/pytorch/pytorch/blob/main/tools/autograd/derivatives.yaml#L1090" rel="noopener nofollow ugc">max</a>/<a href="https://github.com/pytorch/pytorch/blob/main/tools/autograd/derivatives.yaml#L1094" rel="noopener nofollow ugc">maximum</a> are found in <a href="https://github.com/pytorch/pytorch/blob/main/tools/autograd/derivatives.yaml#L1090" rel="noopener nofollow ugc">pytorch/tools/autograd/derivatives.yaml</a>, which is a file containing explicit backward functions.</p>
<p>The backward function for max and min call into <code>evenly_distribute_backward</code> which is defined in <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/autograd/FunctionsManual.cpp#L1779" rel="noopener nofollow ugc">FunctionsManual.cpp</a>. The backward for maximum and minimum are defined in the DSL explained in the big comment on the top of the the <a>derivatives.yaml file</a>. Note that both these methods end up using other ATen ops like <code>at::where</code>. <code>masked_fill</code>, <code>*</code>, <code>/</code>, and <code>sum</code>. You had specifically asked for the implementation for CUDA tensors–and so you’ll have to look into how those ATen ops are implemented in CUDA by checking in <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/native_functions.yaml" rel="noopener nofollow ugc">native_functions.yaml</a>. A lot of them end up using our TensorIterator (<a href="https://labs.quansight.org/blog/2021/04/pytorch-tensoriterator-internals-update" rel="noopener nofollow ugc">blog</a> for more details), which is a performant way of bunching pointwise compute on CUDA.</p>
<p>Additionally, you may find <a href="https://github.com/pytorch/pytorch/wiki/Autograd-Basics" rel="noopener nofollow ugc">our wiki about autograd basics</a> useful for gaining more background if desired.</p>
Does Libtorch installation require CUDA and cuDNN to be downloaded and installed?

username: potter_john
cooked: <p>I need to install libtorch on my win10 laptop, and I was wondering if I still need to manually install CUDA and cuDNN (and set up the required environment variables?) after downloading the pre-compiled binary packages from the official website now??</p>
<p>The last time I tried to install Libtorch I looked up a lot of bits and pieces of tutorials, and I tried installing Cuda and cuDNN (and seemingly other things), and it didn’t turn out very well.</p>
<p>I just checked this thread (<a href="https://discuss.pytorch.org/t/how-to-install-cudnn-for-pytorch-2-1-0-with-cuda-12-1/179864" class="inline-onebox">How to install cuDNN for PyTorch 2.1.0 with CUDA 12.1</a>).</p>
<p>Does it mean that now I just need to download the binary package and then configure the directory for the .lib and header files within Visual Studio?</p>
<p>Is there a clear document/tutorial on how to install it if it still needs to be done manually?</p>
<p>The official site seems to only have a Python tutorial. The VS plugin is of little use to me, as I need to use libtorch in my existing OpenGL project.</p>
How to normalize a complex matrix

username: jianyikele
cooked: <p>please help me how to normalize a complex matrix by python？</p>

username: J_Johnson
cooked: <p>We have to first define what you mean by normalize. Especially in terms of complex numbers, where there are multiple ways you can approach normalization.</p>
<p>For instance, we can approach it from a strictly cartesian method, treating real and imaginary values separately to apply a min-max feature scaling.</p>
<p>Or, the more natural approach, in my opinion, would be with a polar method, maintaining angles while scaling only the radius.</p>
<p>Here are both approaches in PyTorch, where we scale the values to -1 and 1 for the cartesian approach and a radius of 0 to 1 for the polar approach:</p>
<pre><code class="lang-auto">import torch
from copy import deepcopy

A = torch.randn((3,4), dtype = torch.cfloat)*10
B = deepcopy(A)
print(A)

def complex_norm_cart(x, lower, upper):
    max_r = torch.max(x.real)
    min_r = torch.min(x.real)
    max_i = torch.max(x.imag)
    min_i = torch.min(x.imag)
    x.real = lower +(x.real - min_r)*(upper-lower)/(max_r-min_r)
    x.imag = lower + (x.imag - min_i) * (upper - lower) / (max_i - min_i)
    return x

print(complex_norm_cart(A, -1, 1))

def complex_norm_polar(x, upper):
    rad = torch.abs(x)
    max_rad = torch.max(rad)
    norm_rad = rad*upper/max_rad
    x_angle = x.angle()
    return torch.view_as_complex(torch.stack([norm_rad*torch.cos(x_angle), norm_rad*torch.sin(x_angle)], dim=-1))

print(complex_norm_polar(B, 1))
</code></pre>
Trace CUDA context initialization

username: vadimkantorov
cooked: <p>Is it possible to somehow trace/log CUDA context initialization? (including a simple way of tracing CUDA memory allocations which would trigger context initialization too)</p>
<p>Basically, with large frameworks sometimes CUDA context gets unexpectedly initialized and it would be nice to see what it is triggered by (usually some usage somewhere of <code>torch.cuda.is_available()</code> or <code>torch.cuda.get_device_count()</code> or some sloppy tensor allocation on CUDA).</p>
<p>Can I use profiler for this?</p>

username: ptrblck
cooked: <p>I would probably run the workload in <code>gdb</code> and break at <code>cuInit</code>, which should then show which module wrongfully tries to initialize a new context.</p>

username: vadimkantorov
cooked: <p>I guess for this and to get stack traces, one would need a manually-compiled version of Python with debug symbols, and manually compiled version of PyTorch with debug symbols - both are quite nasty to get for a one-time tracing job <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>I wonder if we could get these <code>cuInit</code> calls from autograd profiler? from text-based output of nsys?</p>

username: ptrblck
cooked: <p>No, not necessarily. <code>gdb</code> will also work in release builds, but of course optimizations were performed and also you are right that some symbols might not be shown (e.g. if they were optimized away).<br>
I would first try this approach before starting a debug build.</p>

username: vadimkantorov
cooked: <p>I guess the same task is demonstrated in <a href="https://github.com/pytorch/pytorch/issues/116276" class="inline-onebox" rel="noopener nofollow ugc">`import torch` results in `cuInit` call · Issue #116276 · pytorch/pytorch · GitHub</a></p>

username: ptrblck
cooked: <p>Yes, the proposed breakpoint approach is a great way to detect unwanted initializations landing in <code>cuInit</code> as seen after the faulty PR landed.</p>

username: vadimkantorov
cooked: <p>maybe also <code>LD_PRELOAD</code> shim or <code>ltrace</code> lens could be used for similar tracing…</p>
Extremely high val loss in resnet18

username: mcfn
cooked: <p>when i train a resnet 18, i get extremely high (10^8) values for val loss. the train loss stays normal. even when i set the train dataset and val dataset to be identical.<br>
when i force model.train() in the validation stage (for the sake of isolating the problem), the values go back to normal, similar to the ones of the train loss - usually between going down from 3-5 and approaching zero.<br>
what could be the problem?</p>
Low precision backprop causes exploding gradients

username: mike3
cooked: <p>I am trying to run backprop on Mistral-7B. I have access to a 40GB GPU. With 32bit precision, I run out of space as soon as I call backward(). With 16bit precision, I can call backward(), but some of the earlier layer gradients become infinite. I tried running backprop with 32bit precision on a CPU and I can confirm that exploding gradients do not happen with 32bit precision. What can I do?</p>

username: ptrblck
cooked: <p>If you are using mixed precision training then note that overflowing gradients are expected in a few iterations and the gradient scaler will lower its scaling factor to avoid overflows in the next iteration. The docs explain this behavior in more details.<br>
On the other hand, if you are manually calling <code>.half()</code> on the model overflows can easily happen and you might not be able to recover from them. In this case use the mixed precision utils. with <code>float16</code> or <code>bfloat16</code>.</p>

username: mike3
cooked: <p>Thank you for your reply. I am not actually using mixed precision: I do everything in float16 (I set it as the default dtype for torch and I load the model and tokenizer in float16 precision to begin with). Does that rule out your suggested explanation?</p>

username: ptrblck
cooked: <p>Yes, since manually casting the model to <code>float16</code> will compute all intermediates in this <code>dtype</code>, which can easily under- and overflow due to the reduced precision. What’s the reason you are not using the mixed-precision utils.?</p>
Model(images, targets) is returning a list of dictionaries

username: Nicholas_O_Neal
cooked: <p>This is my first time using PyTorch (sorry if I picked the wrong topic listing), and I’ve ran into a strange issue when programming the validation phase of my model. Here’s the code snippet:</p>
<pre><code>with torch.no_grad():  
    for images, targets in validation_dataloader:
        # Forward pass

        loss_dicts = model(images, targets)  
        batch_losses = [sum(loss_dict.values()) for loss_dict in loss_dicts]
        total_loss = sum(batch_losses) / len(batch_losses)  

        total_val_loss += total_loss
</code></pre>
<p>model(images, targets)  is returning a list of dictionaries, each dictionary in the list corresponds to an image in the batch. This is not the understanding I had of the function, the understanding of the function I had is that it would return information about loss. Was what I read wrong, or is there anyway I can fix this?</p>
<p>If needed, I’d be more than happy to provide any other context.</p>

username: Adam_Prihoda
cooked: <p>You’re using the model function wrong, to get the loss you have to define a loss functions (aka criterion) like so:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
</code></pre>
<p>then you pass only the image throught the model and compare the output with the correct label using the criterion:</p>
<pre><code class="lang-auto">y_pred = model.forward(X_train)
loss = criterion(y_pred,y_train)
</code></pre>

username: Nicholas_O_Neal
cooked: <p>But it works how I hoped during the training phase:</p>
<pre><code class="lang-auto">    model.train()  # Set the model to training mode

    for images, targets in training_dataloader:  
        optimizer.zero_grad()  
        loss_dict = model(images, targets)  
        print(type(loss_dict))
        losses = sum(loss for loss in loss_dict.values())  

        losses.backward()  
        optimizer.step()  

    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {losses.item()}')
</code></pre>
Output of cnn is negative

username: Jake1
cooked: <p>tensor([[[-2.1100, -2.0569, -2.1977, -2.1016, -2.0881, -2.1053, -2.0627,<br>
-2.1315, -2.1637, -2.1518],<br>
[-2.1073, -2.0807, -2.1757, -2.1125, -2.1710, -2.1226, -2.1058,<br>
-2.1105, -2.1899, -2.0638],<br>
[-2.1084, -2.0598, -2.1171, -1.9930, -2.1466, -2.1421, -2.2110,<br>
-2.1488, -2.1378, -2.0872],<br>
[-2.1294, -2.1019, -2.0210, -2.0828, -2.1360, -2.1737, -2.1516,<br>
-2.1104, -2.2015, -2.1766]],</p>
<pre><code>    [[-2.2347, -2.1515, -2.2699, -2.1849, -2.1745, -2.1778, -2.1453,
      -2.2436, -2.2160, -2.2708],
     [-2.1920, -2.2320, -2.2267, -2.1805, -2.1464, -2.1578, -2.2395,
      -2.1214, -2.2896, -2.1644],
     [-2.1465, -2.1630, -2.2229, -2.1454, -2.2178, -2.1901, -2.2487,
      -2.2327, -2.2180, -2.1987],
     [-2.2261, -2.1394, -2.0773, -2.1365, -2.2153, -2.2789, -2.2637,
      -2.1818, -2.2304, -2.2258]],
</code></pre>
<p>this is output of cnn with loss function=nn.BCEWithLogitsLoss()</p>
<p>tensor([[[0.1709, 0.1671, 0.1541, 0.1607, 0.1622, 0.1673, 0.1629, 0.1589,<br>
0.1618, 0.1577],<br>
[0.1631, 0.1658, 0.1580, 0.1655, 0.1627, 0.1625, 0.1589, 0.1713,<br>
0.1580, 0.1706],<br>
[0.1670, 0.1695, 0.1612, 0.1720, 0.1635, 0.1591, 0.1608, 0.1624,<br>
0.1621, 0.1595],<br>
[0.1570, 0.1709, 0.1782, 0.1748, 0.1628, 0.1625, 0.1655, 0.1632,<br>
0.1527, 0.1606]],</p>
<pre><code>    [[0.0781, 0.0832, 0.0763, 0.0851, 0.0800, 0.0796, 0.0784, 0.0775,
      0.0751, 0.0807],
     [0.0810, 0.0764, 0.0735, 0.0796, 0.0740, 0.0773, 0.0757, 0.0826,
      0.0772, 0.0826],
     [0.0812, 0.0796, 0.0788, 0.0864, 0.0742, 0.0796, 0.0714, 0.0780,
      0.0784, 0.0829],
     [0.0777, 0.0786, 0.0870, 0.0794, 0.0759, 0.0753, 0.0749, 0.0793,
      0.0758, 0.0745]],
</code></pre>
<p>and this is what i transformed with using F.sigmoid()<br>
I want to classify multiclass image from using sigmoid function to binary, but all of elements are less than 0.5 so every binary is 0<br>
what is problem?</p>

username: ptrblck
cooked: <p>It’s unclear what might be causing the model to predict <code>class0</code> without any information about your use case, but I would probably start by checking if your dataset is imbalanced and the majority class would be <code>class0</code>.</p>

username: Julian_Lehrer
cooked: <p>For a multiclass (not multilabel) problem, you would use softmax. So <code>output = output.softmax(dim=-1)</code> should give you probabilities</p>

username: ptrblck
cooked: <p>NIT: you can use softmax to compute the probabilities given the returned logits, but you should not pass them to <code>nn.CrossEntropyLoss</code> as raw logits are expected for a multi-class use case.</p>
Confusion on Transformer src, tgt and loss calculation

username: JLenz
cooked: <p>Hello!</p>
<p>I’m training a standard(ish) transformer which has both the encoder and decoder. Ignoring padding for a moment, I’m confused about the general format of my data for the inputs to the encoder, decoder and loss calculation.</p>
<p>At the moment I’ve got it set up so that I have:<br>
<strong>src:</strong> the input, with an <code>[end_token]</code> at the end<br>
<strong>tgt:</strong> the input, with <code>[start_token]</code> appended at the beginning, thus ‘shifting’ it one to the right (also with an [end_token])</p>
<p>My model then produces an output of shape <code>[batch, seq_length, vocab_size]</code> and I calculate loss with:<br>
<code>prediction_loss = criterion(output, tgt)</code></p>
<p>However I am confused if it should be calculating loss against ‘<strong>tgt</strong>’ (which includes the [start_token]) or just ‘<strong>src</strong>’, or something else entirely? In other words, in a ‘perfect’ prediction, should there be a start/end token in the sequence?</p>
<p>As a follow up question, should the <code>[end_token]</code> be present in both sequences during training?</p>
<p>Many thanks in advance. Sorry if this is obvious, I have looked over many tutorials and codebases and am still quite confused on this topic.</p>
Pytorch-tpu runbook issue with dependencies

username: isaacr
cooked: <p>Hey folks,</p>
<p>I used to be able to do this, but something new must’ve integrated circa Dec 8th (that the setup scripts at the below link pull automagically) and broke things for me, resulting in errors.</p>
<p>I am following instructions: <a href="https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/SPMD_USER_GUIDE.md" rel="noopener nofollow ugc">https://github.com/pytorch-tpu/transformers/blob/llama2-google-next-training/SPMD_USER_GUIDE.md</a></p>
<p>During the setup phase,  I get a 3 iterations of this message:</p>
<p>ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.<br>
transformers 4.35.0.dev0 requires regex!=2019.12.17, which is not installed.<br>
transformers 4.35.0.dev0 requires tokenizers&lt;0.15,&gt;=0.14, which is not installed.</p>
<p>Is there a way to go back to an earlier release date (i.e. say December 7th) ?</p>
<p>Wondering what am I missing ?</p>
<p>Many thanks for any suggestions!</p>
MeanPool for rectangular data

username: tpress
cooked: <p>Hello, I have been trying to adapt the image model to work with rectangular data, say 5x1024 images instead of normal images. I have found that the only non-compatible section is a layer called <code>ConvMeanPool</code> which involves the mean pooling summation the forward pass:</p>
<pre><code class="lang-auto">        output = self.conv(inputs)
        output = sum([output[:, :, ::2, ::2], output[:, :, 1::2, ::2],
                      output[:, :, ::2, 1::2], output[:, :, 1::2, 1::2]]) / 4.
        return output
</code></pre>
<p>Which is the only layer that explicitly requires square data with 2 even dimensions. Using rectangular sizes with odd dimensions results in a size mismatch.  Are there any recommendations for some way around this?</p>

username: Julian_Lehrer
cooked: <p>What image model is this? Where is that sum called?</p>
Creating fixed weight nn.linear

username: hibama
cooked: <p>Hi.<br>
I would like to set up an nn.Linear layer in such a way that the data passing through the layer remains unchanged. I understand that I can achieve this by configuring the weight as an identity matrix. Is there a way to directly set the weight of the nn.Linear layer like this??</p>
<p>thanks always!!</p>

username: Kore_ana
cooked: <p>Maybe you can use F.Linear instead of nn.Linear and specify the weight?</p>

username: Julian_Lehrer
cooked: <p>Sure, you can use <a href="https://pytorch.org/docs/stable/generated/torch.nn.Identity.html" class="inline-onebox" rel="noopener nofollow ugc">Identity — PyTorch 2.1 documentation</a></p>
<pre data-code-wrap="python"><code class="lang-python">import torch.nn as nn
layer = nn.Identity()
</code></pre>
How to calculate F1 Score with PyTorch Lightning - T5 Model

username: Lio_Herb
cooked: <p>Hello,</p>
<p>I am fine-tuning a T5 model for QA task.<br>
Everything works fine, but the last thing that I want to implement is calculating F1 Score.<br>
After training the model by:</p>
<pre><code class="lang-auto">trainer.fit(model, data_module)
</code></pre>
<p>I tested it with:</p>
<pre><code class="lang-auto">trainer.test(dataloaders=test_dl)
</code></pre>
<p>And got only the test_loss.</p>
<p>I also tried to manually calculate it, but I don;t know what to do with the outputs from the model, here is an example of the outputs I have:</p>
<pre><code class="lang-auto">outputs = trained_model.model.generate(input_ids=input_ids, attention_mask=attention_mask)
print(outputs)
print(true_labels)
</code></pre>
<p>And it prints:</p>
<pre><code class="lang-auto">tensor([[    0,   532, 64177,   726,     1]], device='cuda:0')
tensor([[   259,  85679,  30010,   1393,  29887,    532,  64177,  11316,  94418,
         218480,   7815,  16736,    855,   6959,      1,   -100,   -100,   -100,
           -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,   -100,
           -100,   -100,   -100,   -100,   -100]], device='cuda:0')
</code></pre>
<p>So I don’t know how to compare them, even after removing the padding the sizes are different.</p>
<p>Thanks a lot!</p>

username: CFXTGJD
cooked: <p><a href="https://stackoverflow.com/questions/70452777/how-to-get-generated-tokens-in-t5-training-step-for-using-user-defined-metrics?rq=3" rel="noopener nofollow ugc">python - How to get generated tokens in T5 training_step for using user-defined metrics? - Stack Overflow</a><br>
This solution works well. I met the same problem and I spent hours searching for solutions until I saw this.</p>
After calling torch.nn.Module.cuda(), model doesn't seem to be freed from RAM

username: noob_programmer
cooked: <p>As per documentation <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.cuda" rel="noopener nofollow ugc">here</a> for <code>cuda</code> method in <code>torch.nn.Module</code>, the model parameters and buffers should have moved to GPU. SO I assume that the model in RAM will get wiped out, but when checking the Resident Set Size of the program, it was increasing 3x than the initial model in cpu.<br>
Here is the code  I’ve used to test,</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import psutil
import gc
import time
import sys

class TestModel(torch.nn.Module):
    def __init__(self) -&gt; None:
        super(TestModel, self).__init__()
        self.ip_l = torch.nn.Linear(1, 512)
        self.hidden = torch.nn.Sequential(*([torch.nn.Linear(512, 512)] * 1000))
        self.op_l = torch.nn.Linear(512, 1)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        self.op_l(self.hidden(self.ip_l(x)))
        return x


if __name__ == "__main__":
    ovr_ram = psutil.virtual_memory().total / (1024 * 1024)

    proc = psutil.Process()
    time.sleep(1)
    print(f"Before model init, \n\tRSS(mb) - {(proc.memory_percent() / 100) * ovr_ram}")

    model = TestModel()
    time.sleep(1)
    print(f"Model of size {sys.getsizeof(model) / 1024} kb at {set([i.device for i in model.parameters()])}, \n\tRSS(mb) - {(proc.memory_percent() / 100) * ovr_ram}") 

    model.cuda(0)
    time.sleep(1)
    print(f"Model of size {sys.getsizeof(model) / 1024} kb at {set([i.device for i in model.parameters()])}, \n\tRSS(mb) - {(proc.memory_percent() / 100) * ovr_ram}")

    gc.collect()
    time.sleep(1)
    print(f"After garbage collect model of size {sys.getsizeof(model) / 1024} kb, \n\tRSS(mb) - {(proc.memory_percent() / 100) * ovr_ram}")
</code></pre>
<p>Is it the right way to profile the memory usage of the program and also, is the <code>torch.nn.Module</code> making a copy of model parameters to GPU.<br>
Tried this in Ubuntu 22.04 x86_64 machine with torch:2.0.0+cu117</p>

username: ptrblck
cooked: <p>The model is tiny and I assume the lazily loaded driver uses the host RAM. Update to the most recent stable or nightly release with CUDA 11.8 or 12.1 and the host RAM usage should decrease.</p>

username: noob_programmer
cooked: <p>That worked, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Questions about updating weight

username: dukhl89
cooked: <p>Hi, I have some question about backward() function.<br>
As I understand, the backward() function only updates weight values that affect a lot to the loss.<br>
In this case, many weight values that do not affect to the loss are only do minor update or skip.<br>
I want to see the change in loss by updating all weight values, not just updating the weight values that have a large impact on the loss.<br>
Please tell me how to update all weight values.</p>
<p>Thank you so much.</p>

username: J_Johnson
cooked: <p>Gradient descent updates all of the weights. But some get more updated than others because they contributed most to the loss on <em>the data given during that batch</em>.</p>
<p>Various optimizers approach the problem of gradient descent updates differently, but all that I’m aware of evaluate and update all of the weights during each backward pass.</p>

username: dukhl89
cooked: <p>Thanks to you, it is easy to understand for me.<br>
Then I have another question. Can I adjust the degree of updates for weights?<br>
I want to reduce the update degree of weights that have a large impact on loss, and increase the update degree of other weights.</p>

username: J_Johnson
cooked: <p>You could just use clamp on the gradients. For example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(100, 128), nn.ReLU(), nn.Linear(128, 10))

data = torch.rand(128, 100)

labels = torch.rand(128, 10)

criterion = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

outputs = model(data)

loss = criterion(outputs, labels)
loss.backward()

# run this after loss.backward() and before optimizer.step()
for param in model.parameters():
    print(param.grad)
    param.grad = torch.clamp(param.grad, min = -0.001, max=0.001)
    print(param.grad)

optimizer.step()</code></pre>

username: dukhl89
cooked: <p>Thank you so much!<br>
It is very helpful to me.</p>
Questions about torch.onnx.dynamo_export

username: daniellevi
cooked: <p>I’m trying to move from torch.onnx.export to torch.onnx.dynamo_export</p>
<p>By using torch.onnx.export I can determine the input and output names, activate constant folding on the model and decide to activate the verbose mode during the export operation.<br>
Now, when I’m trying to use torch.onnx.dynamo_export, I don’t find such way to do the same things. These things are pretty basic so I think that maybe I miss something.</p>
<p>My second question is if there is a way to iterate the operators in the nn.Module and predict if the operators are supported by torch.onnx.dynamo_export. Is it possible to do so? I want to print all the unsupported operators in the nn.Module</p>
<p>Thanks</p>
Gdb python very slow when debug pytorch C++ extensions?

username: tuanhm2302
cooked: <p>I debugged pytorch C++ extension in source code by gdb python debgger. But when I run “import torch”, this command is very slow in gdb python. So how to make gdb faster when debugging source code?</p>

username: albanD
cooked: <p>Hey!</p>
<p>There are a few possible reasons for this:</p>
<ul>
<li>It is downloading pytorch’s dependencies’ symbols which can be slow. You can skip these or make sure they get properly cached</li>
<li>You using a debug build of pytorch which is known to be very big and slow. You can use <a href="https://github.com/pytorch/pytorch/blob/e14026bc2a6cd80bedffead77a5d7b75a37f8e67/setup.py#L11-L12C53">https://github.com/pytorch/pytorch/blob/e14026bc2a6cd80bedffead77a5d7b75a37f8e67/setup.py#L11-L12C53</a> to only enable symbols for the few files you care about</li>
<li>pytorch itself takes a couple seconds to import. That is expected even outside of gdb</li>
</ul>
<p>Hope this helps!</p>

username: tuanhm2302
cooked: <p>Thank you so much, i will have a try.</p>

username: tuanhm2302
cooked: <p>How to check dependencies’ symbols is cached properly or not? If not, how to cache or skip these symbols in pytorch build?</p>

username: albanD
cooked: <p>I think you would have to check with your version of gdb.<br>
For me, this often happens when trying to load libc and the like which can take a bit to download. If you don’t see a downloading message from gdb though, it is most likely something else.</p>
Use a different state_dict for inference

username: migperfer
cooked: <p>Hi everyone!</p>
<p>I remember that some time ago, there was an announcement (I can’t find the source now) of a new feature. This feature would allow for a given model to realize inference using a previously saved state_dict, rather than using the weights of the model being trained at the moment.</p>
<p>The example that was shown in the feature announcement was something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

from .models import MyModel
## ...

model_instance = MyModel()

# do some model training

previously_saved_state_dict = torch.load("path/to/pretrained/model.ckpt")

pseudolabels = model_instance.inference_with_dict_state(previously_saved_state_dict, input)

predictions = model_instance(input)

loss = my_loss_function(predictions, pseudolabels)
# ...
</code></pre>
<p>I can not find anything like this in the documentation, and Googling has not returned anything useful so far.<br>
Was this a beta feature that was finally not supported? Is it somewhere hidden in the documentation? Or did I just dream this feature?</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: albanD
cooked: <p>Hey,</p>
<p>This doesn’t ring a bell I’m afraid.<br>
The two things I would say that have nothing to do with inference are:</p>
<ul>
<li>You can just load_state_dict this state dict to use it as usual</li>
<li>There is `torch.func.functional_call() that can be used to evaluate the model using a given set of params/buffers instead of the ones contained in the model itself.</li>
</ul>
Encountering an Issue while Fine-Tuning BERT for Text Comparasion on Colab

username: Aikoin1
cooked: <p><strong>Problem:</strong> During my fine-tuning process with BERT on the GLUE mrpc dataset, I encountered the following issue:</p>
<blockquote>
<p>RuntimeError: stack expects each tensor to be equal size, but got [100] at entry 0 and [89] at entry 1</p>
</blockquote>
<p><strong>This is my code:</strong></p>
<pre><code class="lang-auto">raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def tokenize_function(example):
    return tokenizer(example["sentence1"],
                     example["sentence2"], 
                     padding=True,
                     truncation=True,
                     return_tensors='pt')

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

tokenized_datasets = tokenized_datasets.remove_columns(["sentence1", "sentence2", "idx"])
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")
tokenized_datasets["train"].column_names

train_dataloader = DataLoader(
    tokenized_datasets["train"], shuffle=True, batch_size=8, 
)
eval_dataloader = DataLoader(
    tokenized_datasets["validation"], batch_size=8, 
)

# the code snippet that resulted in an error
for batch in train_dataloader: 
    break
{k: v.shape for k, v in batch.items()}
</code></pre>
<p>I’m wondering why this error is occurring and how I can enhance my code to rectify it. Any guidance or insights would be greatly appreciated!</p>

username: Aikoin1
cooked: <p>my previous code was:</p>
<pre><code class="lang-auto">def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

tokenized_datasets = tokenized_datasets.remove_columns(["sentence1", "sentence2", "idx"])
tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
tokenized_datasets.set_format("torch")
tokenized_datasets["train"].column_names

train_dataloader = DataLoader(
    tokenized_datasets["train"], shuffle=True, batch_size=8, collate_fn=data_collator
)
eval_dataloader = DataLoader(
    tokenized_datasets["validation"], batch_size=8, collate_fn=data_collator
)

for batch in train_dataloader:
    break
{k: v.shape for k, v in batch.items()}
</code></pre>
<p>The code snippet doesn’t throw an error but triggers a warning:</p>
<blockquote>
<p>You’re using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the <strong>call</strong> method is faster than using a method to encode the text followed by a call to the pad method to get a padded encoding</p>
</blockquote>
<p>so I modified it to the aforementioned code</p>
Help with pytorch "memory leak" on CPU

username: nchen9191
cooked: <p>Hi,</p>
<p>I’m currently developing a differentiable physics engine using pytorch (2.1.0) that combines physics equations and machine learning. I am however seeing a memory leak (running on cpu, haven’t tried on gpu) where the memory continues to increase epoch after epoch. This is not a python memory, but instead a computational graph / gradient leak where tensors aren’t being released after I call backward(). While I’m searching for where the memory leak is occurring exactly, I’m tried this naive thing:</p>
<pre><code class="lang-auto">for _ in range(10):
  trainer = TrainingEngine()
  trainer.train(epoch=1)
  del trainer
  gc.collect()
</code></pre>
<p>But this code still results in memory that increases continually. Can someone explain to me why even after I delete the top level module that contains all tensors that the memory held by the computational graph is not released? I know ideally, I should find the exact place where tensors are created but not used, but is there a simple way to just tell pytorch to clear the computational graph at the end of each epoch? Thanks!</p>

username: ptrblck
cooked: <p>Unfortunately not if you have created references that keep the computation graph alive. Check if you are appending any loss or model output to e.g. a <code>list</code> somewhere.</p>

username: nchen9191
cooked: <p>I also noticed a weird behavior.</p>
<p>Opening up the train() method a little bit:</p>
<pre><code class="lang-auto">def train():
    curr_states = torch.zeros(n, 3)
    curr_state = ...
    for _ in range(n):
        curr_state = self.step(curr_state)
        curr_states[i] = curr_state
    loss = loss_fn(curr_states)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>I noticed that when n=1, memory doesn’t increase no matter how long I run it, but when n &gt; 1, thats when memory leaks. Does this info help?</p>

username: ptrblck
cooked: <p>Is <code>curr_states</code> used outside of this method or stored somewhere else? Does the memory decrease again if you delete it?</p>
Does SyncBatchNorm work normally without DDP?

username: pepper8362
cooked: <p>I have a network with SyncBatchNorm for multi-GPU training. If I train the network on a single GPU in a normal non-parallel way (i.e. without using DDP), do I need to change SyncBatchNorm to BatchNorm? Does SyncBatchNorm still work normally in this case? Thanks for your help!</p>

username: ptrblck
cooked: <p>It should fall back to the vanilla batchnorm operation if no distributed setup is detected as seen in <a href="https://github.com/pytorch/pytorch/blob/9c3ae37fc453505f5e437d1edadefdb278c2c39c/torch/nn/modules/batchnorm.py#L754">this line of code</a>.</p>
Unevenly distributed IterableDataset of batch size 1

username: MyungHa_Kwon
cooked: <p>I’m training LLM with multiple machines.<br>
Due to the memory limit, I set batch size as 1.<br>
Training data is distributed through <code>split_dataset_by_node</code>.<br>
The problem is that some process is finished 1 step earlier than the others and the other process can’t complete training as they cannot calculate gradient.<br>
For example, when the size of data is 5, and the data is distributed into 4 GPUs,<br>
they will have <code>[[0],[4]]</code>, <code>[[1], []]</code>, <code>[[2],[]]</code>, <code>[[3],[]]</code> data.<br>
So the first process waits for the others on second step but they are actually completed and escaped training loop.<br>
Is there any kind of solution?</p>

username: ptrblck
cooked: <p>Use a <code>DistributedSampler</code> which solves this issue by repeating samples or slicing the dataset on each rank to create the same lengths.</p>

username: MyungHa_Kwon
cooked: <p>Thanks for suggestion.<br>
Does it work on <code>IterableDataset</code>?</p>
<p>I got <code>TypeError: object of type 'IterableDataset' has no len()</code> because <code>IterableDataset</code> doesn’t have <code>len()</code>.</p>

username: ptrblck
cooked: <p>No, since a defined length is needed to split the datasets. If you are streaming data you might need to add a custom logic making sure each rank receives the same amount of samples.</p>

username: MyungHa_Kwon
cooked: <p>Thanks. I think <code>IterableDatasetShard</code> from <code>transformers</code> library works in this setting.</p>

username: ptrblck
cooked: <p>That’s interesting as I’m not familiar with this class. Share an update if you’ve found a working solution if possible.</p>
How to do Q learning delayed reward

username: zezeon1
cooked: <p>I am making a game. In this game all agensts interacts with each other and next state and reward<br>
can be calculated after all agents’ action is determined.</p>
<p>I want to train agent’s action based on Q learning.</p>
<p>But the problem is that reward can only be calculated after all agents’ action is determined.<br>
And if I accumulate the reward for all agent, it will be meaningless because it will be close to 0.</p>
<p>Basically Q learning needs reward and next state for each step(In this case, for every units).<br>
But in my game model, it can be only calculated after all actions of agents are determined.</p>
<p>How to deal with this situation?</p>
<p>Is it really impossible to do backward many times after forward many times in pytorch?</p>
<p>I tried this way. I get action from my model with no grad option. and I store the state and action. I do this for all the units.(battle result can only be calculated after all units’ action is determined)</p>
<p>Then after I calculated result with actions from neural net, I get reward for each units.</p>
<p>Then I use stored state and action, I update my model weights with backward() for every units.</p>
<p>But here is the problem. Because I updated my model for each units. I will get different action from the stored one except the first unit.</p>
<p>If I use gradient accumulation, the reward will be meaningless. Because sum of all reward will be close to 0.</p>
<p>How to deal with this situation?</p>
Version issues when building pytorch from source

username: Andy_P
cooked: <p>Dear community,<br>
I build pytorch from the source with v1.12.1 in conda env.<br>
git branch shown below.</p>
<pre><code class="lang-auto">(pytorch_compile_2) .../mypath/.../pytorch$ git branch
* (HEAD detached at v1.12.1)
  main
</code></pre>
<p>When I build successfully, the version looks like this：</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
1.12.0a0+git664058f
</code></pre>
<p>How do I successfully build v1.12.1 version of pytorch.</p>

username: ptrblck
cooked: <p>The commit is the right one as it’s pointing to the <code>HEAD</code> of <code>v1.12.1</code>. If you want to show the release version alone you might want to set the <code>PYTORCH_VERSION</code> environment variable.</p>

username: Andy_P
cooked: <p>Thank you for the quick answer.</p>
<pre><code class="lang-auto">(pytorch_compile_2) .../mypath/.../pytorch$ git branch
* (HEAD detached at v1.12.1)
  main
(pytorch_compile_2) .../mypath/.../pytorch$ export PYTORCH_VERSION=v1.12.1
(pytorch_compile_2) .../mypath/.../pytorch$ python setup.py install
Building wheel torch-1.12.0a0+git664058f
-- Building version 1.12.0a0+git664058f
cmake --build . --target install --config Release
[0/1] Install the project...
-- Install configuration: "Release"
running install
...
...
</code></pre>
<p>How to set the PYTORCH_VERSION environment variable.<br>
Even after setting it using <code>export PYTORCH_VERSION=v1.12.1</code>, the installation is not successful. When trying to install in the v1.12.1 folder, it shows <code>Building wheel torch-1.12.0a0+git664058f.</code></p>
<p>Thanks.</p>
How to run the pte model on GPU

username: Earth-M
cooked: <p>Hello,</p>
<p>I would like to konw if ExecuTorch supports GPU.<br>
Now I could export model into pte format and execute runtime for xnnpack backend in Intel device.<br>
The device has GPU.</p>
<p>But when I check GPU usage while running the application, GPU wasn’t utilized.<br>
If ExecuTorch supports GPU, can you please share me how to use GPU?</p>
<h3><a name="environment-1" class="anchor" href="#environment-1"></a>Environment</h3>
<pre><code class="lang-auto">$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    CPU family:          6
    Model:               142
    Thread(s) per core:  2
    Core(s) per socket:  2
    Socket(s):           1
    Stepping:            9
    CPU max MHz:         3600.0000
    CPU min MHz:         400.0000
    BogoMIPS:            4599.93
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon
                          pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe p
                         opcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad 
                         fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window 
                         hwp_epp md_clear flush_l1d arch_capabilities
</code></pre>
<p>Thanks,</p>

username: dbort
cooked: <p>ExecuTorch isn’t primarily aimed at desktops/servers, so we don’t have general GPU support today. Also, XNNPACK specifically optimizes for CPUs, and doesn’t know about GPUs.</p>
How to use argument fill in Pad

username: eve_22
cooked: <p><a href="https://pytorch.org/vision/main/generated/torchvision.transforms.Pad.html#pad" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.transforms.Pad.html#pad</a></p>
<p>I want to use this function and specify a color. The doc says <code>fill</code> accepts a tuple, but the following code does not work:</p>
<pre><code class="lang-auto">    image = read_image(img_path)
    # pad image
    _, height, width = image.shape
    pad_left = (self.max_width - width) // 2
    pad_right = self.max_width - width - pad_left
    pad_top = (self.max_height - height) // 2
    pad_bottom = self.max_height - height - pad_top
    # fill = 0 if img_path.startswith("sndn") else 255  
    fill = tuple(image[:,0,-1].tolist()) &lt;----- here
    print(fill)
    image = F.pad(image, (pad_left, pad_top, pad_right, pad_bottom), fill=fill)
</code></pre>
<p>I am passing a tuple but the code outputs “Got inappropriate fill arg”. How am I supposed to specify a color?</p>
Bus error finetuning whisper model in multi GPU instances

username: hitesh_agarwal
cooked: <p>Hi, I am trying to finetune Whisper according to the blog post <a href="https://huggingface.co/blog/fine-tune-whisper" rel="noopener nofollow ugc">here</a>. The finetuning works great in a single GPU scenario, however, fails with multi GPU instances. While executing <code>trainer.train()</code>, multi GPU instances return <code>Bus error (core dumped).</code></p>
<p>I am working on g5.12xlarge instance for multi GPU on AWS with AMI ID: ami-071323fe2bf59945b on Ubuntu. I would appreciate any guidance or suggestions to resolve this issue.|</p>

username: fduwjj
cooked: <p>Do you have a code snippet to share?</p>
Pytorch dont use GPU

username: mustafa_emre_dos
cooked: <p>Even though I have a GPU I get torch.<strong>version</strong>=“2.1.0” instead of torch.<strong>version</strong>=“2.1.0+cu118”<br>
What is the reason?</p>

username: ptrblck
cooked: <p>Install the PyTorch binaries with CUDA support and check <code>torch.version.cuda</code> additionally.</p>

username: mustafa_emre_dos
cooked: <p>I don’t understand your solution exactly. Please give me more details</p>

username: mustafa_emre_dos
cooked: <p>i give this image<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3befd4fa8046d891be12087412b626f4c362a73.png" alt="Ekran görüntüsü 2023-12-26 193421" data-base62-sha1="pE6EWgIIWld3rKsSY5B3CT9i2pZ" width="260" height="153"></p>