akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
OutOfMemoryError help

BoB372: <p>I get this error message</p>
<p>torch.OutOfMemoryError: Allocation on device</p>
<p>Can someone explain what this error means? Out of RAM or VRAM ?</p>
<p>comfyui.log</p>
<p>Total VRAM 12282 MB, total RAM 32605 MB<br>
pytorch version: 2.4.1+cu124<br>
Set vram state to: NORMAL_VRAM<br>
Device: cuda:0 NVIDIA GeForce RTX 4070 : cudaMallocAsync</p>
<p>Thanks</p>

soulitzer: <aside class="quote no-group" data-username="BoB372" data-post="1" data-topic="214918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/8baadc/48.png" class="avatar"> BoB372:</div>
<blockquote>
<p>Can someone explain what this error means? Out of RAM or VRAM ?</p>
</blockquote>
</aside>
<p>VRAM - It means that you’re trying to make another allocation but you’re out of GPU memory</p>
Pytorch cuda missing

sarathi: <p>Hello Team ,<br>
After the installation of pytorch i can see cuda is missing</p>
<p>[saratpa3@home ~]$ module load PyTorch/2.1.2-foss-2023a-CUDA-12.1.1<br>
[saratpa3@home ~]$ python<br>
Python 3.11.3 (main, Nov 27 2024, 09:42:36) [GCC 12.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()<br>
False</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p>Could you describe what <code>PyTorch/2.1.2-foss-2023a-CUDA-12.1.1</code> is and where you got this binary/build from?</p>
Modify Resnet File structure and how to import it

Izan_C_G: <p>Hi, I would like to modify the structure of the model <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py" rel="noopener nofollow ugc">Resnet50 </a>. My goal is neither to add nor to remove layers, only to replace the convolutions that in the code are made by the pytorch nn.Conv function by convolutions made by the Nvidia CUTLASS library (<a href="https://github.com/NVIDIA/cutlass/blob/main/examples/python/02_pytorch_extension_grouped_gemm.ipynb" class="inline-onebox" rel="noopener nofollow ugc">cutlass/examples/python/02_pytorch_extension_grouped_gemm.ipynb at main · NVIDIA/cutlass · GitHub</a>).</p>
<p>I don’t intend either to retrain or to modify weights, only to substitute the call to the convolutions with the call to a convolution of cutlass in a similar way to how I describe it in the pytorch forum: <a href="https://discuss.pytorch.org/t/using-diffetent-conv2d-ops-with-pre-trained-models/214367" class="inline-onebox">Using diffetent conv2d ops with pre trained models</a></p>
<p>My question is if it is possible, within the guidelines of the repository and then how can I import the file <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py" rel="noopener nofollow ugc">Resnet50 </a> from another file or pytorch as it would be done with <a href="https://pytorch.org/vision/main/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision main documentation</a>.</p>
<p>Thanks.</p>
How does scale=True work in new v2 transforms?

pytorchexplorer: <p>Mostly title, but, say in torchvision.transforms <code>v2.ToDtype(torch.float32, scale=True)</code> how exactly does <strong>scale=True</strong> scale the values? <strong>Min-max scaling</strong>? or something else.</p>

Abdiaziz_Aden: <p>It scales the values based on the range of the data type. For example <code>torch.float32</code> data type tensor, the values are scaled to the range <code>[0.0, 1.0]</code></p>
How to deal with when some of the input is None?

moondeokjong: <p>Hi, basically I’m trying to generate an image based on, let’s say, 10 conditions which are wrapped in a tensor.</p>
<p>However, among those 10 conditions, fifth and sixth conditions are sometimes missing, which means those conditions do not apply in those observations and should be considered as something like None.</p>
<p>In this case, how could you deal with the situation or how would you make a neural network based on the description above?</p>

ptrblck: <p>Could you describe how these conditions are used? Are you passing them into the model directly?</p>

moondeokjong: <p>Yes, they are meant to be passed into the model directly.</p>
<p>For example, I’m generating an image of a car and the conditions I’m using are like <code>wheel type</code>, <code>car colour</code>, <code>car number plate</code> etc,. However, among them, my dataset has some observations with no <code>car number plate</code> and in that case I need to pass <code>None</code> to the model</p>
<p>The tricky thing is the model input size seems to be fixed, so I have pass in any value for that missing <code>car number plate</code></p>

ptrblck: <p>The data handling would depend on your use case and eg. if you are using categorical inputs, you could try to encode the empty inputs with placeholder values.<br>
However, unsure if this would work for you.</p>

moondeokjong: <p>Yes that’s what I thought at first as a solution.</p>
<p>But the thing is those concerned data contain a categorical value and a continuous value as well. In the latter case, I’m not sure how the model would manage to interpret the embedding of the placeholder value… I’m thinking of using <code>-100</code> or that kind of irrelevant and extremely distant value from <code>[-1, 1]</code> to make it distinguishable. Would you suggest any other value if my option seems problematic?</p>

m_rf: <p>I’m not sure if this translates to variable-length input models, if so, try a structure like transformer. If you can’t change your model, I think you can try transformer as an encoder, a bit like VAE</p>

moondeokjong: <p>Thanks <a class="mention" href="/u/m_rf">@m_rf</a> . It might be a bit like a variable-length input model. But how could a transformer deal with that?</p>

moondeokjong: <p>For the future reference, if someone is in the same situation and needs to try with placeholder values as pbrlbck mentioned, I tried with a very extreme value such as <code>-100</code>. However, it seems like because of its great numerical magnitude compared with other normal values ranging <code>[-1, 1]</code> the trained model seems to produce something like a grey canvas without any object in it. But interestingly, when I put a normal value for that input, it seemed working.</p>
<p>I’ll try to fix the placeholder value not like before and something near the normal range, but slightly out of it and see the result.</p>
Why the gradient values seems to be reversed in Tensor.backward()

nadeeer: <p>I am trying to print the gradient values of 3 tensors but I find the printed gradients do not match my manual calculation, specifically the gradient of a is <strong>swapped</strong> with the gradient of c. Check the code below.</p>
<blockquote>
<p>import torch<br>
a = torch.tensor(3.0,requires_grad=True)<br>
b = a*2<br>
c = b ** 2</p>
<p>b.retain_grad()<br>
c.retain_grad()<br>
c.backward() # Computes the gradient of current tensor wrt graph leaves.<br>
print(a.grad)<br>
print(b.grad)<br>
print(c.grad)</p>
</blockquote>
<p>Here is the output:</p>
<blockquote>
<p>tensor(24.)<br>
tensor(12.)<br>
tensor(1.)</p>
</blockquote>

soulitzer: <p>These results seem expected to me.</p>
<pre><code class="lang-auto">c = (a*2)**2 = 4*a**2
dc/da = 8a = 8(3) = 24
</code></pre>

nadeeer: <p>So you’ve calculated dc/da which is the gradient value for c, and I agree that 24 is the correct answer. Though,  when I print c.grad the output is 1 and confusingly the print of a.grad is 24. Do you see where I am confused?</p>

soulitzer: <p>Yes, that would be strange.<br>
I see though from your original post that the prints are correct?</p>

nadeeer: <p>They don’t seem to be correct as in they’re not matching the manual calculation me and you did. Here is what I am getting:<br>
print(a.grad) → tensor(24.)<br>
print(b.grad) → tensor(12.)<br>
print(c.grad) → tensor(1.)</p>

soulitzer: <blockquote>
<p>So you’ve calculated dc/da which is the gradient value for c,</p>
</blockquote>
<p>a.grad actually means dc/da (and c.grad means dc/dc, which is 1)</p>
<p>When using .backward() we are doing backprop not forwardprop (which PyTorch also supports <a href="https://pytorch.org/tutorials/intermediate/forward_ad_usage.html" class="inline-onebox" rel="noopener nofollow ugc">Forward-mode Automatic Differentiation (Beta) — PyTorch Tutorials 2.5.0+cu124 documentation</a>)</p>

nadeeer: <p>That makes sense and resolves the confusion!</p>
<blockquote>
<p>a.grad actually means dc/da (and c.grad means dc/dc, which is 1)</p>
</blockquote>
<p>Any where in the documentation that mentions this information? I never came across it before.</p>

soulitzer: <p>Not sure it is mentioned explicitly, possibly because c.retains_grad(); c.backward() is relatively rare to do (e.g. loss.grad is never populated) so usually it is harder to confuse.</p>

soulitzer: <p>The common case is more like: I have a loss and many parameter, and its obvious that the gradient of the loss wrt each of params is stored in each of param’s grad fields.</p>

nadeeer: <p>Probably. I’ve wrote the above code just to understand how the gradients are calculated in .</p>
<p>This was a helpful discussion, I appreciate your insights!</p>
What is the best approach to save a tensor with the "actual size"?

Fei_Liu: <p>Say I have a original large tensor as:</p>
<pre><code class="lang-auto">x_large = torch.tensor(100_000, 100_000)
</code></pre>
<p>Now I do:</p>
<pre><code class="lang-auto">x_small = x[::1000, ::1000]
</code></pre>
<p>and I only want to save this small tensor to disk. If I directly do <code>torch_save(x_small, &lt;file&gt;)</code> , it will save the full big data chunk along with <code>x_small</code>’s super wide stride (as <a href="https://pytorch.org/docs/stable/notes/serialization.html#preserve-storage-sharing" rel="noopener nofollow ugc">how torch.save is designed with storage sharing</a>). This costs huge disk space that I don’t need in this particular case. The way to get around this is to do save <code>x_small.contiguous()</code> instead – I get the contiguous format that is often desired, besides the size on disk is compact. In addition, if a tensor is already contiguous then this additional <code>.contiguous()</code> call barely has any cost. So, for most “usual” use cases, I just do <code>.contiguous()</code> before saving as a good habit.</p>
<p>However, this does not solve all the cases. Consider again I have the <code>x_large</code> above, and now my <code>x_small2</code> is just:</p>
<pre><code class="lang-auto">x_small2 = x_large[0]
</code></pre>
<p>In this case, <code>x_small2</code> is already contiguous, but the underlying storage still points to the big chunk. If I want to save the “actual” <code>x_small2</code>, I’ll have to do:</p>
<pre><code class="lang-auto">torch.save(x_small2.contiguous().clone(), &lt;file&gt;)
</code></pre>
<p>Now to generalize this “good practice”, I’ll have <code>clone</code> all the time before I save the tensor. Are there any less tedious / more recommended ways to accomplish this save task?</p>
<p>Thank you!</p>

aoot: <p>A more general solution could be to detach the tensor before cloning it.</p>
<p>In the example you provided, there are not computation graph linked to the tensor. But in case there is, this <code>x_small2.detach().clone()</code><sup class="footnote-ref"><a href="#footnote-462443-1" id="footnote-ref-462443-1">[1]</a></sup> solution would address it.</p>
<hr class="footnotes-sep">

<ol class="footnotes-list">
<li id="footnote-462443-1" class="footnote-item"><p><a href="https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor" rel="noopener nofollow ugc">Stackoverflow preferred way to copy a tensor</a> <a href="#footnote-ref-462443-1" class="footnote-backref">↩︎</a></p>
</li>
</ol>

soulitzer: <p>If the issue is that torch_save is saving the base in the case where the saved tensor is a view, we can conditionally clone depending on whether <code>t._is_view() is True and t._base.numel() &gt; t.numel()</code>.</p>
Gradient of dot product for every index pair

sherlock.h: <p>Hello!</p>
<p>I am interested in computing <code>\nabla_{x_i}(s(x_i)^t s(x_j))</code>.</p>
<p>The dot products are computed as <code>product_pairs = s.matmul(s.transpose(-2, -1))</code>.</p>
<p>However, I am unsure about how to proceed for the grads. <code>grad(product_pairs.sum(), x)</code> is not the way to go given that it sums over <code>j</code> as well, and the output has only two free indices (<code>i</code> and dimension <code>d</code>) whereas I expect three (<code>i</code>, <code>j</code>, and <code>d</code>).</p>
<p>Any solutions for this? Thanks!</p>

soulitzer: <p>Are you looking for something like <a href="https://pytorch.org/docs/stable/generated/torch.vmap.html" class="inline-onebox" rel="noopener nofollow ugc">torch.vmap — PyTorch 2.5 documentation</a></p>
Log file generation

R_JENILA: <p>I have used the below code save the print statements as log.txt file.</p>
<p>‘’'import logging</p>
<h1><a name="p-462419-set-up-logging-1" class="anchor" href="#p-462419-set-up-logging-1"></a>Set up logging</h1>
<p>log_file = ‘/content/drive/MyDrive/macro_rudy_pin_congestion/training_log.txt’<br>
log_dir = os.path.dirname(log_file)<br>
os.makedirs(log_dir, exist_ok=True)</p>
<p>logging.basicConfig(filename=log_file, level=logging.INFO, format=‘%(asctime)s - %(message)s’)</p>
<h1><a name="p-462419-define-a-function-to-log-messages-2" class="anchor" href="#p-462419-define-a-function-to-log-messages-2"></a>Define a function to log messages</h1>
<p>def log_message(message):<br>
logging.info(message)‘’’</p>
<p><span class="hashtag-raw">#Training</span> loop<br>
epochs = 10</p>
<p>for epoch in range(epochs):<br>
model.train()<br>
train_loss = 0.0<br>
iters = len(train_loader) <span class="hashtag-raw">#added</span> this line to define iters</p>
<p>for i, (features, labels) in enumerate(train_loader):<br>
features = features.permute(0, 3, 1, 2).to(device)<br>
labels = labels.permute(0,3,1,2).to(device)</p>
<pre><code>optimizer.zero_grad()

#forward pass
outputs = model(features)

#calculate loss
loss = loss_fn(outputs, labels)

loss.backward()

#update weights
optimizer.step()
scheduler.step(epoch + i / iters)

train_loss += loss.item() * features.size(0)
</code></pre>
<p>train_loss /= len(train_loader.dataset)<br>
print(f’Epoch {epoch+1}/{epochs}, Train Loss: {train_loss:.4f}‘)<br>
log_message(f’Epoch {epoch+1}/{epochs}, Train Loss: {train_loss:.4f}’)</p>
<p>if (epoch + 1) % 5 == 0:<br>
log_message(‘saving checkpoint…’)<br>
save_cp(model=model, optimizer=optimizer, epoch=epoch+1, cp_path=‘/content/drive/MyDrive/macro_rudy_pin_congestion/files’)<br>
But the training_log.txt is not getting created in the drive. Could you please help me to resolve this issue</p>

ptrblck: <p>Is this issue specific to the <code>logging</code> module or PyTorch or is the used file path inaccessible or wrong?</p>

R_JENILA: <p>It’s the code mistake. Now I got it. Thanks.<br>
import logging</p>
<h1><a name="p-462500-set-up-logging-1" class="anchor" href="#p-462500-set-up-logging-1"></a>Set up logging</h1>
<p>log_file = ‘/content/drive/MyDrive/macro_rudy_pin_congestion/training_log.txt’<br>
log_dir = os.path.dirname(log_file)<br>
os.makedirs(log_dir, exist_ok=True)</p>
<p>if not os.path.exists(log_file):<br>
open(log_file, ‘w’).close()<br>
print(f"Log file created: {log_file}")</p>
<p>logger = logging.getLogger()<br>
logger.setLevel(logging.INFO)<br>
formatter = logging.Formatter(‘%(asctime)s - %(message)s’)<br>
file_handler = logging.FileHandler(log_file)<br>
file_handler.setFormatter(formatter)<br>
logger.addHandler(file_handler)</p>
<h1><a name="p-462500-define-a-function-to-log-messages-2" class="anchor" href="#p-462500-define-a-function-to-log-messages-2"></a>Define a function to log messages</h1>
<p>def log_message(message):<br>
logger.info(message)</p>
if pytorch wheel package support avx512? #144032

risemeup1: <p>“My CPU system supports AVX512, and I want to use a PyTorch package that supports AVX512. Which one should I choose, or do I have to build from source?”</p>

ptrblck: <p>From your cross-post:</p>
<blockquote>
<p>But to answer your question, yes PyTorch wheels support AVX512 on Linux x86 (and probably on Windows as well)</p>
</blockquote>
Realtime Raw Video Denoiser for Mobile phone Cameras

pranavmicro7: <p>I’m working on a personal project called <strong>Realtime Low-Light Raw Video Denoiser</strong>. The main aim is to remove noise and artifacts from videos taken in low-light conditions, specifically for <strong>mobile cameras</strong>. I’ve noticed there aren’t many papers or resources that provide strong, practical solutions for this.</p>
<p>I’d love to get some suggestions on how to achieve real-time raw image denoising for mobile cameras. Any advice or insights would be really helpful!</p>
Stack a vector and another tensor of size 0 or 2

AviZ: <p>I have <code>a</code> which is a tensor object with shape <code>(3, )</code>.<br>
I have <code>b</code> which is a tensor object with shape either <code>[]</code> (scalar) or <code>(2, )</code>.</p>
<p>I want to concatenate <code>a</code> and <code>b</code> into a single vector of length <code>(4, )</code> or <code>(5, )</code>.<br>
It seems that <code>torch.cat</code> or <code>torch.stack</code> will fail doing so.</p>
<p>Is there other one liner option to do without <code>if</code> to handle the case of the scalar?</p>

AviZ: <p>I found something to work:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

a = torch.rand(3)
b1 = torch.rand(2)
b2 = b1[0]

torch.cat((a, torch.atleast_1d(b1)), dim = 0)
torch.cat((a, torch.atleast_1d(b2)), dim = 0)
</code></pre>
<p>If there a better way, I am all ears.</p>
Slow training of rCNN and torch.jit.script() function not compatible

Mattis1337: <p>Hello,<br>
in the following gist you can find some example code of my rCNN which I use for analyzing and training on chess games:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/Mattis1337/f75b62b0b42bbc273e769fbc8f3b7664">
  <header class="source">

      <a href="https://gist.github.com/Mattis1337/f75b62b0b42bbc273e769fbc8f3b7664" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/Mattis1337/f75b62b0b42bbc273e769fbc8f3b7664" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/Mattis1337/f75b62b0b42bbc273e769fbc8f3b7664</a></h4>

  <h5>train_model.py</h5>
  <pre><code class="Python">import numpy as np
import torch
from torch import nn
from random import shuffle


def train_rnn(dataset, model, criterion, optimizer, device):
    """
    Training a recurrent neural network
    :param dataset: the dataset containing the white/black moves which shall be used for training</code></pre>
   This file has been truncated. <a href="https://gist.github.com/Mattis1337/f75b62b0b42bbc273e769fbc8f3b7664" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The data is structured as follows:<br>
input: [sequence length, 12, 8, 8]<br>
target: [sequence length, number of possible outputs]</p>
<p>The output tensor is a hot encoded tensor using all possible chess moves in my chess dataset.</p>
<p>The issue that I am facing is that despite my several attempts of optimizing the model (channel_last memory format, disabling debug APIs, etc.) I am getting very long training times and also the torch.jit.script() module is handing me an error whenever I try to parse my model like this:<br>
<code>model = torch.jit.script(model)</code><br>
Any help would be greatly appreciated!</p>
Adding a TensorOption to Torch

jackbondpreston: <p>Hi,</p>
<p>I’m currently trying to work on a prototype change to address a previously mentioned issue: <a href="https://dev-discuss.pytorch.org/t/how-to-support-heterogeneous-memories/1352" class="inline-onebox" rel="noopener nofollow ugc">How to support heterogeneous memories - hardware-backends - PyTorch Developer Mailing List</a></p>
<p>To do this, I am taking the approach mentioned in this post - trying to add an additional default argument to tensor constructors (e.g. zeros(), etc.). I can make the changes to the APIs, native_functions.yaml, etc. but I do end up hitting several issues.</p>
<p>I looked for some previous work, and found the commit adding the <a href="https://github.com/pytorch/pytorch/commit/c484cf43a02863efd2f4a76aad43246fb0191ab5" rel="noopener nofollow ugc">commit adding pin_memory to the TensorOptions</a>, but it seems a fair amount has changed since then.</p>
<p>My approach so far has been to update all TensorFactories methods, update native_functions.yaml, update TensorOptions and torchgen code relating to parsing/building tensoroptions, update derivatives.yaml, and various other small changes to glue all changes together.</p>
<p>Some of the issues I have hit are:</p>
<ol>
<li>Some of the methods I would want to add this new argument to are listed in torchgen/aoti/fallback_ops.py. This results in torchgen/gen.py#L2424 being thrown, which is a bit confusing to me wrt. how to address it. Indeed, it seems by case would be (2), but I’m not too clear on what the listed steps actually entail (e.g. adding the shim, what/where is the version number to bump?, etc.)</li>
<li>Additionally, if I bypass issue (1) by just omitting these methods for now, I still hit other issues at the code generation stage of build. For example, the gen_python_functions.py hitting an error where it finds the empty.out overload, but can’t find a matching empty function. This seems to be because it doesn’t expect empty.out to have my extra argument, and can only find empty <em>with</em> the extra argument. But I’m not sure why this is occurring.</li>
</ol>
<p>Is there any advice or info on how to go about changing these fairly fundamental APIs? The code generation etc. seems to complicate such things a lot, even if it’s adding just a new default argument.</p>
<p>Please feel free to ask for any extra useful info, I wanted to try not to include too much extraneous detail for readability.</p>

jackbondpreston: <p>Made some progress on this. For future references:</p>
<ol>
<li>Ensure all TensorOptions are grouped and in the same order in native_functions.yaml definitions</li>
<li>New TensorOptions need to be added to quite a few places in the codebase. I had missed torchgen/api/native.py. There’s also torchgen/api/python.py, torchgen/api/translate.py, torchgen/api/dispatcher.py, torchgen/api/cpp.py</li>
</ol>
Running Llama-3.3-70b-instruct: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

Charlie_G: <p>I am trying to run a quantised version of Llama-3.3-70b-instruct on my NVIDIA 6000 Ada GPU. I get the following error:</p>
<pre><code class="lang-auto">  File "/mnt/c/Users/user1/myenv/lib/python3.12/site-packages/bitsandbytes/autograd/_functions.py", line 462, in forward
    output = torch.nn.functional.linear(A, F.dequantize_4bit(B, quant_state).to(A.dtype).t(), bias)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<p>nvcc --version shows CUDA 12.0. nvidia-smi shows the driver version as 553.50 and CUDA as 12.4. My understanding is that the 12.0/12.4 difference is not impactful. The VRAM is not being exceeded when the model is loaded.<br>
Do you know what might be causing this? I have put the script below.</p>
<pre><code class="lang-auto">import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "meta-llama/Llama-3.3-70B-Instruct"
token_content = "hf_XXXXXXXXXXXXXXXXXXXXX"
# 1. Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    token=token_content,
    cache_dir="./huggingface_cache",
    use_fast=False  # Some Llama-based models do not have fast tokenizers yet
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    token=token_content,
    cache_dir="./huggingface_cache",
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.float16
)

prompt = "Explain the concept of Reinforcement Learning in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        top_p=0.95,
        temperature=0.7
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

</code></pre>
Autograd failing for sparse matmul at half precision

Tanay_Arora: <p>Hello. I am currently attempting to run a full autocast-ed forward and backward pass over a mostly sparse model (the sparse weights are utilized each in only torch.sparse.mm() calls). However, if I store the weights in coo format, matrix multiplication simply isn’t implemented (at half precision), and for csr, the backward fails without a proper error message:</p>
<pre><code class="lang-auto">RuntimeError: sampled_addmm: Expected mat1 and mat2 to have the same dtype, but got Half and Float
</code></pre>
<p>Is there anything I can do to utilize autocast with this setup? Note that full-precision sparse training and half-precision dense training both work.</p>
<p>A simple reproduction of the printed error is below.</p>
<pre><code class="lang-auto">weight = torch.rand(5, 4, device = "cuda", dtype = torch.float32)
weight = (weight * (weight &lt; 0.2)).to_sparse_csr().requires_grad_(True)

inp = torch.rand(4, 3, device = "cuda", dtype = torch.float32, requires_grad = False)

with torch.autocast("cuda", dtype = torch.float16, enabled = True):
	loss = torch.sparse.mm(weight, inp).sum()

loss.backward()
</code></pre>
<p>When autocast’s enabled=False, it works fine. When it is formatted to coo instead, it fails on the matmul.</p>
<p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Sparse CSR support is experimental so this issue might be expected. Could you create a GitHub issue for this error assuming you are still seeing it in the latest nightly release, please?</p>

Tanay_Arora: <p>It does remain in the latest nightly release - sure thing.</p>
Custom layer-specific gradient calculation

DawidL: <p>Hi,</p>
<p>I would like to change the gradient calculation for one of the layers, depending on where the gradient is coming from. Is there an easy way to do so?</p>
<p>Here is a concrete example of what I want to achieve:</p>
<p>I have a layer (A) connected to two other layers (B, C). The gradient of the initial layer (A) will be equal to the sum of the gradients coming from the two other layers (B, C), however I would like the gradient coming from one of the other layers (B) to have less of an impact on the first layer. To do so I want to multiply it by 0.9 before the addition.</p>
<p>So the code would look something like this:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 1)
        self.fc3 = nn.Linear(2, 1)
        
    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        y = self.fc2(x)
        y = F.relu(y)
        z = self.fc3(x)
        z = F.relu(z)
        x = y+z
        return x
</code></pre>
<p>I know that I could use a hook, but this seems to only change the whole gradient after the summation, so I don’t think that would work. I also obviously want the gradient of other layers to remain unaffected when taking training step, so changing the gradient of one of the other layers completely is not a good solution.</p>

soulitzer: <p>You right in that hooks registered via t.register_hook(…) will apply changes after all the gradients from B and C are both accumulated.</p>
<p>What you can do instead is register a different type of hook that interposes at a different point. Specifically here, you’d want to use a post-hook to modify the gradient after backward has been computed for B via t.grad_fn.register_hook(…).</p>
<p>For more details you can see:<br>
<a href="https://pytorch.org/docs/stable/notes/autograd.html#backward-hooks-execution" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/autograd.html#backward-hooks-execution</a></p>
OutOfMemoryError: CUDA out of memory with a gan 

Freeziey: <p>Hello, I have been trying to setup this gan but i cannot escape this error: <a href="https://pastebin.com/rp0kNXiH" class="inline-onebox" rel="noopener nofollow ugc">error - Pastebin.com</a> (sometimes it’s not the same line)</p>
<p>I have tried gc.collect but maybe i misplaced it. I also tried to have as little as possible things on the gpu in my training loop but nothing seems to work.</p>
<p>I code on a kaggle notebook.</p>
<p>here’s my code: <a href="https://paste.pythondiscord.com/IVMQ" class="inline-onebox" rel="noopener nofollow ugc">View paste IVMQ</a></p>

soulitzer: <p>Is the suspicion that there is a leak?</p>
<p>Taking a memory trace (<a href="https://pytorch.org/blog/understanding-gpu-memory-1/" class="inline-onebox" rel="noopener nofollow ugc">Understanding GPU Memory 1: Visualizing All Allocations over Time | PyTorch</a>) can help confirm this + identify which tensor is being kept alive.</p>
`num_features` parameter of `nn.InstanceNorm2d` does not change results

aoot: <p>I was wondering what would happen if the <code>num_features</code> specified for <code>nn.InstanceNorm2d</code> mismatch that of the number of features in the input and thus decided to test it. I learned that the <code>num_features</code> parameter does not really matter as the results remain the same.</p>
<p>Why doesn’t <code>num_features</code> matter? If it doesn’t matter, why is the parameter not optional? (I know there is a <code>nn.LazyInstanceNorm2d</code> which dynamically detect the input feature size.)</p>
<p>Thanks for your insight,</p>
<h3><a name="p-462049-minimal-working-code-1" class="anchor" href="#p-462049-minimal-working-code-1"></a>Minimal working code:</h3>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
import sys

print(f"Python version: {sys.version}; Torch version: {torch.__version__}")

input = torch.randint(low=0, high=255, size=(5, 3, 256, 256)).double()
output_base = nn.InstanceNorm2d(num_features=1)(input)

for i in range(2, 20):
    IN_layer = nn.InstanceNorm2d(num_features=i)
    
    output = IN_layer(input)
    result_check_same = (output_base == output).all()
    
    print(i, result_check_same)

    if not result_check_same:
        break
</code></pre>
<h3><a name="p-462049-output-2" class="anchor" href="#p-462049-output-2"></a>Output:</h3>
<pre data-code-wrap="bash"><code class="lang-bash">Python version: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:25:13) [Clang 14.0.6 ]; Torch version: 2.3.0.post100
2 tensor(True)
3 tensor(True)
4 tensor(True)
5 tensor(True)
6 tensor(True)
7 tensor(True)
8 tensor(True)
9 tensor(True)
10 tensor(True)
11 tensor(True)
12 tensor(True)
13 tensor(True)
14 tensor(True)
15 tensor(True)
16 tensor(True)
17 tensor(True)
18 tensor(True)
19 tensor(True)
</code></pre>

ptrblck: <p>A warning will be raised from <a href="https://github.com/pytorch/pytorch/blob/a8ac3a6b20b154e661e80ef930c6b0c563f3bb2d/torch/nn/modules/instancenorm.py#L108-L119">here</a> since you are using the default <code>affine=False</code> argument. If you use affine parameters, the code will fail with a shape mismatch.</p>

aoot: <p>Thank you, I realized that I was using an older version of PyTorch that did not have the warning. The warning is very informative.</p>
Is there a vectorize map function for tensor with different shape

shadowshadow: <p>I have a list of tensor. Each of them have a different shape，I want to apply same function on each tensor。Since each tensor have different shape，I can’t use <code>torch.vmap</code>. So is there any way to do this instead of using a for-loop or padding each tensor to same shape. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You could use the <code>_foreach_*</code> methods for your use case, which are used in e.g. optimizers internally:</p>
<pre data-code-wrap="python"><code class="lang-python">device = "cuda"
x = [torch.randn(i, device=device) for i in range(1, 10)]
print(x)
# [tensor([1.8445], device='cuda:0'), tensor([0.3656, 1.9296], device='cuda:0'), tensor([ 0.0386, -0.3014, -2.0722], device='cuda:0'), tensor([-2.3759,  1.0400, -0.1989, -1.2025], device='cuda:0'), tensor([-0.5833,  0.6642, -0.4505,  1.3151,  0.2400], device='cuda:0'), tensor([ 0.4314, -0.3071, -1.4707,  0.4154, -1.9433, -0.9677], device='cuda:0'), tensor([-0.5343,  0.5911,  0.7805, -0.2972,  1.7352,  0.1670,  0.4759],
#        device='cuda:0'), tensor([ 0.8740, -0.8012,  1.8532, -0.7882, -1.6124,  2.2543,  0.4694, -2.3202],
#        device='cuda:0'), tensor([-0.2593, -0.3326,  0.4430, -0.0338,  1.1399,  0.1994,  1.6222,  0.8456,
#         -0.1709], device='cuda:0')]

y = torch._foreach_abs(x)
print(y)
# (tensor([1.8445], device='cuda:0'), tensor([0.3656, 1.9296], device='cuda:0'), tensor([0.0386, 0.3014, 2.0722], device='cuda:0'), tensor([2.3759, 1.0400, 0.1989, 1.2025], device='cuda:0'), tensor([0.5833, 0.6642, 0.4505, 1.3151, 0.2400], device='cuda:0'), tensor([0.4314, 0.3071, 1.4707, 0.4154, 1.9433, 0.9677], device='cuda:0'), tensor([0.5343, 0.5911, 0.7805, 0.2972, 1.7352, 0.1670, 0.4759],
#        device='cuda:0'), tensor([0.8740, 0.8012, 1.8532, 0.7882, 1.6124, 2.2543, 0.4694, 2.3202],
#        device='cuda:0'), tensor([0.2593, 0.3326, 0.4430, 0.0338, 1.1399, 0.1994, 1.6222, 0.8456, 0.1709],
#        device='cuda:0'))
</code></pre>
<p>Note that these methods are internal and their API could change.</p>

soulitzer: <p>To add to piotr’s answer, if your tensors on differ in shape only on a single dimension you can try NestedTensors. <a href="https://discuss.pytorch.org/t/how-to-apply-vmap-on-a-heterogeneous-tensor/214109/2" class="inline-onebox">How to apply vmap on a heterogeneous tensor - #2 by soulitzer</a></p>
<p>Internally nested tensors represent ragged data using a packed representation.</p>
<p>You might be able to get more coverage on certain operators compared to foreach.</p>
How Does model.eval() Affect Gradient Descent in PyTorch and How to Handle Frequent Evaluations?

5o1: <p>I used the Adam optimizer. If I perform an evaluation (i.e., <code>model.train(), ...train steps... grad steps... model.eval()... model.train()</code> in a loop) after every iteration (instead of every epoch) or after every training step when <code>epoch length == 1</code>, the gradients fail to descend properly. However, if I perform evaluation after every &gt;100 iterations or increase the epoch length to a comparable scale, it works fine. It seems that <code>model.eval()</code> somehow affects the optimizer’s functionality.</p>
<p>My first question is: by what mechanism does <code>model.eval()</code> cause the gradients to fail to descend properly?</p>
<p>My second question is: if I must perform evaluation after every iteration—for instance, because I am using PyTorch for compressed sensing reconstruction and want to print the results after each iteration—what is the best way to handle this?</p>

soulitzer: <p>There’s a bit about model.eval() you can read about here <a href="https://pytorch.org/docs/stable/notes/autograd.html#evaluation-mode-nn-module-eval" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 2.5 documentation</a></p>
<p>I would expect .just enabling eval() to not affect training, e.g. when you aren’t calling .backward() and .step() when you performed forward with .eval().</p>

5o1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9eb9ed74bf11ccb3a431b8ae5985d1baf1facefa.png" data-download-href="https://discuss.pytorch.org/uploads/default/9eb9ed74bf11ccb3a431b8ae5985d1baf1facefa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9eb9ed74bf11ccb3a431b8ae5985d1baf1facefa_2_307x499.png" alt="image" data-base62-sha1="mE9MSdc9vVHcCOTsi4TIls3AurE" width="307" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9eb9ed74bf11ccb3a431b8ae5985d1baf1facefa_2_307x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9eb9ed74bf11ccb3a431b8ae5985d1baf1facefa_2_460x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9eb9ed74bf11ccb3a431b8ae5985d1baf1facefa_2_614x998.png 2x" data-dominant-color="262828"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">731×1187 76.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is my code. “accumulation_steps” is always value 1. And if I remove “if not model.training”, it is also easy to get hard to autograd.<br>
While every time the model switching to eval mode, it switches back to train mode at the next training step, maybe train() did something?</p>

soulitzer: <p>What happens if you change validation_step to still accept a batch, but do a no-op?</p>
Questions about built-in method torch.xxx(ops)

Truthuleave: <p>Hi there.<br>
Is there any specification about torch.xxx(ops), such as torch.max_pool2d,torch.conv2d? I cannot found it in <code>https://pytorch.org/docs</code>.</p>
<p>How do they differ from their <code>torch.nn.functional.xxx</code> counterpart?</p>

QLYYLQ: <p>You can see the different in <code>nn.Linear()</code> and <code>nn.functional.linear()</code>. In general speaking, one is class and another is base function interface. I remember that if you use <code>nn.functional.linear()</code>, you should set bias manually.</p>

soulitzer: <p>torch.conv2d is the exact same as torch.nn.functional.conv2d. You can check where they are defined in torch/nn/functional.py.</p>
Resize_ seems to do nothing in convert_conv3d_weight_memory_format

GodHforever: <p>I found a functon in <a href="https://convert_conv3d_weight_memory_format" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/2ed4d65af0a1993c0df7b081f4088d0f3614283e/torch/nn/utils/memory_format.py#L80C6-L80C7</a></p>
<p>I do the following test</p>
<pre><code class="lang-auto">def convert():
    a = torch.randn(1,2,3,4)
    print(a.is_contiguous()) # True
    print(a.is_contiguous(memory_format = torch.contiguous_format)) # True
    print(a.is_contiguous(memory_format = torch.channels_last)) # False
    print(a.shape) # torch.Size([1, 2, 3, 4])
    print(a.stride()) # (24, 12, 4, 1)

    b = (
        a.clone().contiguous(memory_format=torch.channels_last)
    )
    print(b.is_contiguous()) # False
    print(b.is_contiguous(memory_format = torch.contiguous_format)) # False
    print(b.is_contiguous(memory_format = torch.channels_last)) # True
    print(b.shape) # torch.Size([1, 2, 3, 4])
    print(b.stride()) # (24, 1, 8, 2)
    print(b.size())
    c = b.resize_(
        b.size(), memory_format=torch.channels_last
    ) # ! no use?????
    print(c.is_contiguous()) # False
    print(c.is_contiguous(memory_format = torch.contiguous_format)) # False
    print(c.is_contiguous(memory_format = torch.channels_last)) # True
    print(c.shape) # torch.Size([1, 2, 3, 4])
    print(c.stride()) # (24, 1, 8, 2)
</code></pre>
<p>so what is the use of resize_?</p>
How to access the computational graph?

Capangas: <p>I have seen thousands of people asking for this in the pytorch forum, and getting only nonsensical responses. If I have a loss function, and I call loss.backward, I want to know which tensors are going to receive gradients, etc. I want to access the computational graph.</p>

albanD: <p>Hi,</p>
<p>Which posts in particular?<br>
There are ways to traverse the graph but since we rebuild it at every iteration (to be able to handle all the dynamic controls user wants), it is not as structured as it is in other frameworks.</p>
<p>But as you might have seen in other posts, you can traverse the Nodes it using <code>.next_functions</code> after accessing the Tensor’s parent Node via <code>.grad_fn</code>.<br>
The AccumulateGrad Nodes are the ones responsible to accumulate gradients in the .grad fields on Tensors and have a <code>.variable</code> attribute that will tell you which Tensor they will accumulate the gradient into.</p>

Capangas: <p>Thanks, that’s (the “next_functions” method) is exactly what I was looking for. I noticed the next_functions method is not present in the documentation though. Why is that? next_functions returns a tuple, so it is unclear what each element of the tuple represents. Also, how can I access the AccumulateGrad nodes?</p>

albanD: <p>It is not present in the doc as it is more of an implementation detail than something the user should rely on. We provide it for convenience and debugging purposes.</p>
<p>The tuple contains the next Nodes for each input. Each element there is a pair of net Node and the index in the output of the Tensor that was used as input to this Node.<br>
the <code>AccumulateGrad</code> Nodes are just one type of Node. You can check the string representation and check that it matches I guess.<br>
To collect these, you will need to traverse the whole graph (it is guaranteed that there is no cycle there) and save all the AccumulateGrad Nodes you encounter.</p>

ado_sar: <p>Why the index always is 0? Is it due to that the operations (e.g. addition, multiplication) result in a single output?</p>

albanD: <p>That is indeed the case !</p>