akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Triton_ kernel after torch.compile

wjh: <p>Dear all,</p>
<p>I noticed lots of <em>triton_</em> kernel executions in nsight systems after I wrap my custom triton kernel and other functions with <em>torch.compile</em>.</p>
<p>Can anyone please let me know what <em>triton_</em> kernel is?</p>

ptrblck: <aside class="quote no-group" data-username="wjh" data-post="1" data-topic="204466">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wjh/48/70666_2.png" class="avatar"> wjh:</div>
<blockquote>
<p>after I wrap my custom triton kernel and other functions with <em>torch.compile</em>.</p>
</blockquote>
</aside>
<p>I’m not sure I understand the question as it seems you are already using custom Triton kernels, so I assume you know what Triton is. In any case, these are generated kernels via TorchInductor (used in <code>torch.compile</code>) using <a href="https://github.com/triton-lang/triton"><code>triton-lang/triton</code></a>.</p>

wjh: <p>Thank you for your kind reply!</p>

ezyang: <p>Please see also “How do I map a kernel back to Inductor code and graph?” in <a href="https://docs.google.com/document/d/1y5CRfMLdwEoF1nTk9q8qEu1mgMUuUtvhklPKJ2emLU8/edit#heading=h.vsritviyl28z" class="inline-onebox">torch.compile, the missing manual - Google Docs</a></p>

ultrasounder: <p>Wow, you should totally publish this on gumroad.</p>
Frequencies and time arays for spectrogram

sammlapp: <p>When we use the torchaudio.transforms.spectrogram transform we get a 2d array with dimensions corresponding to frequency and time. The exact frequencies and times of rows and columns depend on the spectrogram parameters.</p>
<p>The APIs for librosa and scipy.signal either directly return these values along with the spectrogram or provide other means of getting these values. Without them, interpretation of the output is difficult, and users are likely to try to manually create the time and frequency series - but are likely to be guessing at edge cases like how the window functions behave at the beginning and end of the audio signal - which will result in off-by-one and similar errors.</p>
<p>I was going to open this as a feature request on the torchaudio issues page, but there is a note saying that the issues are no longer monitored. Please let me know if torchaudio is deprecated or if there is a better place to post this question. Thanks</p>

ptrblck: <aside class="quote no-group" data-username="sammlapp" data-post="1" data-topic="209438">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sammlapp/48/36795_2.png" class="avatar"> sammlapp:</div>
<blockquote>
<p>but there is a note saying that the issues are no longer monitored</p>
</blockquote>
</aside>
<p>Could you link to this note here, please?</p>

sammlapp: <p>oops sorry for delay. For example if I go to create an issue:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/audio/issues/new?assignees=&amp;labels=&amp;projects=&amp;template=bug-report.yml">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/audio/issues/new?assignees=&amp;labels=&amp;projects=&amp;template=bug-report.yml" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://github.githubassets.com/assets/github-logo-55c5b9a1fe52.png" class="thumbnail onebox-avatar">

<h3><a href="https://github.com/pytorch/audio/issues/new?assignees=&amp;labels=&amp;projects=&amp;template=bug-report.yml" target="_blank" rel="noopener nofollow ugc">Build software better, together</a></h3>

  <p>GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 420 million projects.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>There is all-caps bold text below the Add a Title field:</p>
<p><strong>PLEASE NOTE THAT THE TORCHAUDIO REPOSITORY IS NO LONGER ACTIVELY MONITORED.</strong> You will not likely get a response. For open discussions, visit <a href="https://discuss.pytorch.org/">https://discuss.pytorch.org/</a>.</p>

ptrblck: <p>Thanks for sharing this message!<br>
Let me add <a class="mention" href="/u/moto">@moto</a> to help with this question if possible.</p>
Wav2Vec2FABundle emission frame size

pytorque: <p>The example waveform in the <a href="https://pytorch.org/audio/stable/tutorials/ctc_forced_alignment_api_tutorial.html" rel="noopener nofollow ugc">forced alignment tutorial</a> has 54k frames, and is sampled in 16 kHz. It renders an emission tensor of size <code>[1, 169, 28]</code> if I remember correctly. There are 28 labels, and seemingly 169 binned (?) frames, as in the Time label on the x axis on the plots.</p>
<p>I need to convert the transformed frames to “actual” sampled frames in my pipeline. Asking Copilot returns suggestions about window size and stride. Does anybody know if these frame parameters are fixed, and the actual values?</p>

pytorque: <p>Answering myself. After looking closer at the tutorial and the <code>display_segments</code> function in the <a href="https://pytorch.org/audio/stable/tutorials/forced_alignment_tutorial.html#audio-samples" rel="noopener nofollow ugc">Audio samples section</a>, I realize that the frame ratio is <code>size of waveform / number of rows in emission matrix</code>.</p>
Why do we need to detach() the output while doing eval?

rinkujadhav2013: <p>Here’s the function I have for eval:</p>
<pre><code class="lang-auto">@torch.no_grad()
def amp_eval(dataloader, loss_fn, model, optimiser, scaler):
    model.eval()
    y_true, y_pred = [], []
    for inputs, labels, filenames, num_tumour_pixels in dataloader:
        with torch.autocast(device_type='cuda', dtype=torch.float16):
            outputs, attention_weights = model(inputs)
            y_pred.append(outputs.cpu().detach().item())
            y_true.append(labels.item())
            # del outputs

        print(inputs.requires_grad, f'Memory usage: {get_process_memory()} GB', flush=True)
        print(f"Objects in memory: {len(gc.get_objects())}", flush=True)

</code></pre>
<p>If I don’t use .item(), the memory blows up giving OOM. Why do I need when I’m using <code>@torch.no_grad()</code>?</p>

ptrblck: <p>You don’t need it as no computation graph will be created as seen in this small example:</p>
<pre data-code-wrap="python"><code class="lang-python">def fun1(x):
    y = x ** 2
    return y

@torch.no_grad()
def fun2(x):
    y = x ** 2
    return y

x = torch.randn(1, requires_grad=True)

y1 = fun1(x)
print(y1.grad_fn)
# &lt;PowBackward0 object at 0x7f514be26f80&gt;

y2 = fun2(x)
print(y2.grad_fn)
# None
</code></pre>

rinkujadhav2013: <p>So, where could the problem be in my code? Weirdly enough, it works just fine while training but memory blows up during eval. I am using some custom layers but I’m not doing anything special for eval mode in them.<br>
Regardless, how should I debug? Check if any of the tensors require grad during the forward pass?</p>

ptrblck: <p>Yes, you should narrow down where exactly the unexpected memory usage comes from.</p>
How to disable 0/1 specialization for certain inputs?

chengmengli06-01: <p>0/1 specialization is done when do torch.export, how to avoid doing this for certain inputs? Otherwise, I have to construct inputs which has dimensions greater than 1, such as sequences, which is ugly. Is there anyway to specify that certain inputs will not do 0/1 specialization? Where is 0/1 specialization necessary? Which guards are affected by this specialization?</p>
LinuxHomeDesktopPC with local AI-PersonalAssistant

carrotti: <p>Project Description:<br>
i want to build an innovative local AI-powered HomePC personalAssistant that integrates advanced machine learning capabilities with home automation and security features. The linux with OpenSourceSoftware system utilizes a custom-built desktop PC with integrated microcomputers (Raspberry Pi 5? ) as modules for specialized tasks: <span class="hashtag-raw">#1:</span> Network management and security monitoring , <span class="hashtag-raw">#2:</span> xLSTM-based AI assistant <span class="hashtag-raw">#3:</span> IoT and home automation hub; aiming to create a powerful, versatile, and privacy-focused AI assistant for home use.<br>
The computer must have an intelligent, learning and empathetic Personal AI Assistant that supports the user in various tasks and provides an interface to authorities, social life and knowledge resources.</p>
<p>Project Goals:</p>
<ol>
<li>Create a high-performance, locally-run AI assistant that respects user privacy.</li>
<li>Integrate advanced machine learning capabilities for natural language processing and task automation.</li>
<li>Develop a modular system that combines the power of a desktop PC with the versatility of microcomputers</li>
<li>Implement robust home automation and security features.</li>
<li>Establish an open-source platform that encourages community contributions and innovations.</li>
</ol>
<p>Areas Needing Support:</p>
<ol>
<li>Optimizing machine learning models for local execution on custom hardware.</li>
<li>Developing efficient integration methods for multiple AI frameworks.</li>
<li>Creating scalable architecture for modular AI and IoT components.</li>
<li>Implementing secure data handling and privacy protection measures.</li>
<li>Designing user-friendly interfaces for complex AI interactions.</li>
</ol>
Pytorch installation on raspberry pi fails

Nicole_Merkle: <p>Hi all,</p>
<p>I have the follwoing problem. I would like to install pytorch on my raspberry pi. My installed Python version is 3.9.2 which should be compatible with Pytorch. I use the following command to install Pytorch:</p>
<blockquote>
<p>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cpu" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cpu</a></p>
</blockquote>
<p>Unfortunately I get the following error:</p>
<blockquote>
<p>ERROR: Could not find a version that satisfies the requirement torch<br>
ERROR: No matching distribution found for torch</p>
</blockquote>
<p>I grasped the internet, however did not find a solution to my problem. It seems also that my Python Version should be supported. Any ideas what might went wrong?</p>

nicewarmcoffee: <p>Hello,</p>
<p>I was googling with the same error.<br>
My issue was that I have 32bit Raspbian OS installed, not 64bit.</p>
<pre><code class="lang-auto">$ getconf LONG_BIT
32
</code></pre>
<p>you can see my gcc target is arm-linux-gnueabihf, not aarch64-linux-gnu</p>
<pre><code class="lang-auto"> $ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/12/lto-wrapper
Target: arm-linux-gnueabihf
</code></pre>
<p>Can you check if your Pi is running 64bit OS?</p>
<p>Kind regards,</p>
<p>Jae</p>
Is Export IR incompatible with fx.Interpreter?

kmaeng: <p>Dear Experts:</p>
<p>I am having trouble running Export IR through fx.Interpreter. I thought the two should be compatible because I read that export IR is based on FX IR and also code like torch.export.unflatten.InterpreterModule internally uses it. However, it seems like it is not the case (torch.export.unflatten.InterpreterModule also does not run correctly on my setup). Below is a simple test code I tried. The top runs correctly (FX IR + FX interpreter), but the bottom fails (Export IR + FX interpreter):</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.export import export

class MyModule(nn.Module):
    def __init__(self):
    super().__init__()
    self.layer0 = nn.Conv2d(4, 10, 1, 1)
    self.layer1 = nn.Conv2d(10, 5, 1, 1)

    def forward(self, x):
        return self.layer1(self.layer0(x))

x = torch.rand([7, 4, 32, 32])
net = MyModule()

# This runs correctly
gm = torch.fx.symbolic_trace(net)
ip = torch.fx.Interpreter(net, graph=gm.graph)
ip.run(x)

# This doesn't
exported_program = export(net, args=(x,))
ip = torch.fx.Interpreter(net, graph=exported_program.graph)
ip.run(x)
</code></pre>
<p>The error message is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/.../test.py", line 46, in &lt;module&gt;
    ip.run(x)
  File "/.../lib/python3.11/site-packages/torch/fx/interpreter.py", line 146, in run
    self.env[node] = self.run_node(node)
                     ^^^^^^^^^^^^^^^^^^^
  File "/.../lib/python3.11/site-packages/torch/fx/interpreter.py", line 203, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../lib/python3.11/site-packages/torch/fx/interpreter.py", line 236, in placeholder
    raise RuntimeError(f'Expected positional argument for parameter {target}, but one was not passed in!') from si
RuntimeError: Expected positional argument for parameter p_layer0_bias, but one was not passed in!
</code></pre>
<p>If I am doing anything wrong, it would be great if anyone can help me understand what I am doing incorrectly.</p>

kmaeng: <p>I figured out the solution. I need to first call torch.export.unflatten() to unflatten the exported_program before feeding it to the interpreter!</p>
Accuracy Decrease：PyTorch 2.1 Precision Issues Compared to 1.7/1.8

Kitto: <p>Hi everyone,</p>
<p>First off, I want to say that I’m really impressed with the improvements in PyTorch 2.1. The speed increase is noticeable and has been a great help in my experiments. Big thanks to the team for the continuous development!<br>
That being said, I’ve run into an issue with precision when using PyTorch 2.1. In my experiments, I’ve noticed a significant drop in accuracy compared to when I used versions 1.7 and 1.8. The same models and setups that performed well in the older versions now show a noticeable decline in precision with the new version. I’ve tried troubleshooting, but I can’t seem to figure out the root cause.<br>
Has anyone else experienced this issue? Is there something specific that changed between these versions that could explain the difference in accuracy? I’d really appreciate any insights or advice on how to resolve this. Thanks in advance!</p>
My linear layer's output does not have gradients

jkdev: <p>I have the following toy code:<br>
testx = torch.tensor([[1.0, 2.0, 3.0]], requires_grad=True)<br>
testw = torch.tensor([[0.1, 0.2, 0.3]], requires_grad=True)<br>
testb = torch.tensor([0.5], requires_grad=True)<br>
testy = F.linear(testx, testw, bias=testb)<br>
print([a.requires_grad for a in [testx, testw, testb, testy]])</p>
<p>When I run it standalone in a new python file  it correctly gives me [True, True, True, True] . But when I run exactly these lines in a larger program I get the output [True, True, True, False]</p>
<p>What could cause requires_grad to be set to false for testy in this case and how can I debug it? Is there some context that I need to set for this to work correctly?</p>
<p>Thanks in advance for your help.</p>

ptrblck: <p>You might have disabled gradient computation globally, e.g. via:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.set_grad_enabled(False)
testx = torch.tensor([[1.0, 2.0, 3.0]], requires_grad=True)
testw = torch.tensor([[0.1, 0.2, 0.3]], requires_grad=True)
testb = torch.tensor([0.5], requires_grad=True)
testy = F.linear(testx, testw, bias=testb)
print([a.requires_grad for a in [testx, testw, testb, testy]])
# [True, True, True, False]
</code></pre>
Registering a buffers the content of a dictionary?

dherrera1911: <p>I have some variables that I want to have as attributes in my model that are registered as buffers. These variables have a hierarchical structure such that it would make sense to have them grouped in dictionaries, i.e. <code>model.group1['var1']</code> <code>model.group2['var2']</code>.</p>
<p>Is there a way to register the contents of these dictionaries as buffers? Or just have them track the dtype and device of my model?</p>

ptrblck: <p>The feature request is tracked <a href="https://github.com/pytorch/pytorch/issues/37386">here</a> and after <code>nn.Buffer</code> landed should be actionable now.</p>
What is accumulating in /tmp/torchinductor_{USER}/triton?

danbochman: <p>Hi,</p>
<p>My training crashed because I ran out of disk space. After further inspection it seems like there are hundreds of GBs stored in my /tmp/torchinductor_azureuser/triton</p>
<p>For context — I am training my model with torch.compile and DDP</p>
<p>What is being stored there? Is there any way for me to prevent running out of storage in long (weeks) training runs?</p>
<p>TIA</p>

ptrblck: <p>This folder is the default cache location according to <a href="https://github.com/pytorch/pytorch/blob/01bf350967e27283376c4ce39af0770a69cf3780/torch/_inductor/runtime/runtime_utils.py#L97-L102">this code snippet</a>.</p>
Is all_gather supposed to take more time/memory than gather?

f_g: <p>From the definition of the two functions, I would imagine that all_gather would take more time and memory, because it has more communication to do (to non-root-rank machines), and the non-root-rank machines will have to store the <code>all_gather</code>ed result, so if we’re on a 4-gpu machine and we’re doing <code>all_gather</code> on cpu, the CPU memory of the machine should be 4 times more than doing a <code>gather</code>.</p>
<p>However experimentally that seems to not be the case, the memory and runtime usage of the two are similar, does anyone know why?</p>

yf225: <p>This article might be helpful: <a href="https://mpitutorial.com/tutorials/mpi-scatter-gather-and-allgather/" class="inline-onebox" rel="noopener nofollow ugc">MPI Scatter, Gather, and Allgather · MPI Tutorial</a>.</p>
<blockquote>
<p>so if we’re on a 4-gpu machine and we’re doing <code>all_gather</code> on cpu, the CPU memory of the machine should be 4 times more than doing a <code>gather</code> .</p>
</blockquote>
<p>I think the premise of “we’re on a 4-gpu machine and we’re doing <code>all_gather</code> on cpu” is confusing. Should it be “doing all_gather on gpu”?</p>
<p>My mental model is that we can look at all_gather as parallelized gathers happening for all ranks, so the memory and runtime usage is probably similar.</p>

f_g: <p>We run training on GPU, but the data we’re gathering is very big and would quickly go OOM on cuda, so we gather them on cpu instead before metrics computation.</p>
<p>If all_gather is parallelized gathers, I can understand that runtime would be similar. However wouldn’t memory still be different, because only root rank needs to store the gathered result, instead of all ranks storing the gathered result?<br>
Or is it because, since we’re calling all_gather on cpu, PyTorch under the hood knows it’s happening on the same hardware, so it’s actually still doing just <code>gather</code>?<br>
Therefore if we run <code>all_gather</code> on 2gpu vs 4gpu, there would be memory difference instead?</p>
I want to eliminate the accumulation of memory usage during the learning loop

kato0209: <p>Memory will be overwhelmed during the learning loop of the following code. Each loop is opening variables and opening the computed graph by loss.backward, and I don’t think there is anything to accumulate. (My understanding is that the gradient information is updated at each loop, but since the gradient is one-to-one for the parameters, it is not something that is accumulated.)</p>
<p>Also, the model uses hf models like meta-llama/Llama-2-7b-hf, but the memory increase in the 7B model is 30MB compared from the beginning to the end of the loop, whereas in the 70B it increases enough to cause memory leaks (about 1000MB per loop? )</p>
<p>Does anyone know the cause of this?</p>
<p>Translated with <a href="http://DeepL.com" rel="noopener nofollow ugc">DeepL.com</a> (free version)</p>
<pre><code class="lang-auto">import torch

def forward(model, state, y, normailized_similarity_scores, args, eval=False):
    loss_value = 0.0
    for n in range(args["batch_size"]):
        token_length = len(y[n])
        for t in range(token_length):
            if y[n][t] == 0:
                continue
            outputs = model(input_ids=state[n][t].detach())
            logits = outputs.logits

            last_token_logits = logits[0, -1, :]

            log_probs = torch.nn.functional.log_softmax(last_token_logits, dim=-1)
            target_token_log_prob = log_probs[y[n][t].detach().item()]

            baseline = torch.mean(normailized_similarity_scores)
            reward = normailized_similarity_scores[n].item()
            R = reward / baseline

            loss = target_token_log_prob * R / args["batch_size"] / token_length

            if eval == False:
                loss.backward()
            loss_value += loss.detach().item()
            del outputs, logits, last_token_logits, log_probs, target_token_log_prob, baseline, reward, R, loss
            torch.cuda.empty_cache()

    return loss_value
</code></pre>
LBFGS in multi gpu machine

Joeyoung: <p>Hi,</p>
<p>Does LBFGS work in a multi gpu(cuda) machine?<br>
PyTorch manual says that all parameters have to be on a single device.<br>
Does DDP + LBFGS work?<br>
I often get nan values during LBFGS optimization.</p>
<p>Thanks,<br>
Joe</p>

agu: <p>Are you getting nan values with using DDP + LBFGS or just LBFGS by itself?</p>
<p>For DDP, you can have each data parallel replica be its own process running on its own GPU, in which case LBFGS should work per replica.</p>

Joeyoung: <p>Nan happens only LBFGS in DDP. Works well in a single GPU.</p>
Data.to() overlap with the compute in eager mode

whatdhack: <p>In the eager mode, is it possible to overlap data.to() with the compute ?</p>
<p>for data in enumerate( dataloader):<br>
data.to(device)<br>
output= model(data)</p>

ptrblck: <p>Yes, assuming the data transfer is using a non-blocking call and actual computation can overlap the transfer.<br>
In your code snippet you are directly using <code>data</code> in the next call so nothing can overlap its transfer.</p>

whatdhack: <p>Thanks.  Any example code to illustrate the non-blocking call and compute overlap ?</p>

whatdhack: <p>The following code was spit out by an AI. Not sure if this would actually work. Although, this is specific for cuda - if its works,  then could be adapted for pure PyTorch I think.</p>
<pre><code class="lang-auto">import torch

# Create CUDA streams
data_stream = torch.cuda.Stream()
compute_stream = torch.cuda.Stream()

# Training loop
for data, target in dataloader:
    # Copy data to GPU using data_stream
    with torch.cuda.stream(data_stream):
        data, target = data.cuda(), target.cuda()

    # Perform model training on compute_stream
    with torch.cuda.stream(compute_stream):
        # ...
</code></pre>
How to set partial gradient to zero

Sam_Xu: <p>For simplicity, say that output = net(input) and output is of dimension 100 x 1. During training, maybe I want to zero out the gradient for index 2 and 4 positions of output. I wonder how I could do this and still backprop the rest of the gradients through the model? I was thinking of a two-step approach where the gradient w.r.t output is computed first, then some partial gradient is zero-out, and then compute the gradients for the rest of the model parameters using the new output gradient.</p>

albanD: <p>Hi,</p>
<p>You can register a <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.register_hook">hook</a> on output that will zero-out some part of the gradients.</p>

little_dragon: <p>If I have two kinds of loss both related to the output , such as loss = loss1(output) + loss2(output), I just want to zero the grad when I use loss1 to adjust weights, how can I make hook not to influece loss2 , I am a newer , I don’t know if the situation is reasonable</p>

albanD: <p>Hi,</p>
<p>You will need to clone the output:</p>
<pre><code class="lang-auto">output = your_code_here()

output1 = output.clone()
output1.register_hook(zero_stuff_for_loss1_only)

loss = loss1(output1) + loss2(output)
</code></pre>

abrahamowos: <aside class="quote no-group" data-username="albanD" data-post="5" data-topic="22135">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>output1.register_hook(zero_stuff_for_loss1_only)</code></p>
</blockquote>
</aside>
<p>I tried following this but I did not understand.  How would you define this <code>zero_stuff_for_loss1_only</code> ?<br>
I intended on doing this in HuggingFace Transformer trainers.</p>
Ask for help on this mismatch error

Raminguyen12: <p>Hello <span class="mention">@ptrblck_de</span>,</p>
<p>I was trying to infer my fine-tuned llama 3.2 vision model, but the thing is that I have this error “RuntimeError: mat1 and mat2 shapes cannot be multiplied (17x4096 and 1024x4096)”</p>
<p>I also included my <a href="https://github.com/raminguyen/LLMP/blob/main/LLMP/llama%20inference.ipynb" rel="noopener nofollow ugc">GitHub</a> link.</p>
<p>I appreciate your help <span class="mention">@ptrblck_de</span>.</p>
CUDA 11.4 on AGX Xavier

Hsnn: <p>Hi, I have AGX Xavier with ubuntu 20.04. I have also cuda 11.4, and installed torch 2.0.1 using pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a><br>
but when I write torch.cuda.is_available() it returns false. Is there any version mismatch? What should I do ?<br>
Thx</p>

ptrblck: <p>You can download binaries for JP6 from <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>
PyTorch and Nvidia Xavier NX

Halil_Huseyin_Calisk: <p>Hello, I have an Nvidia Jetson Xavier NX that comes with the JetPack system pre-installed and has CUDA 11.4.19. Which versions of torch, torchvision, and torchaudio should I download to perform GPU-based image processing on this board?</p>

ptrblck: <p>You can download binaries from <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>
Detach() for this specific case

abe7: <p>I know that <code>detach()</code> is used for detaching a variable from the computational graph. In that context, are the following expressions <code>x = x - torch.mean(x, dim=0).detach()</code> and <code>x = x - torch.mean(x, dim=0)</code> equivalent? I just want to subtract the mean out, don’t want to pass gradients through the average calculation.</p>

googlebot: <p>No, in second case you’ll have additional trainable shared term.</p>
<p>E.g. for two element vector, you’ll have xout[0] = x[0] - (x[0]+x[1])/2, so xout[0] gradient will affect x[1].</p>

a2jinhee: <p>Hello, I want to expand on this question a little bit.</p>
<p>I understand that the two cases will differ, but how exactly does adding <code>.detach()</code> to the second term affect training? I assume it depends on the gradient of <code>xout</code>, but I’m curious about how the parameter adjusts in each scenario:</p>
<ol>
<li>When subtracting the mean without <code>.detach()</code></li>
<li>When subtracting the mean with <code>.detach()</code></li>
</ol>
<p>Generally, subtracting the mean of a tensor centers it around zero, but how does using <code>.detach()</code> influence this behavior?</p>

abe7: <p>The difference is that, if you don’t detach before the mean operation, the upcoming gradients will backpropagate through the mean operation and x itself, during the training process. It is up to you whether that’s something you want.</p>

a2jinhee: <p>Thank you for your response.</p>
<p>This is what I think:<br>
When we <em>don’t</em> use detach, the network will likely “learn” to adjust the specified tensor toward a zero mean. On the other hand, when we <em>do</em> use detach, the tensor should learn its parameters independently of its mean.</p>
<p>I’d appreciate any further insights, as I’m still not entirely sure how the behavior will play out in either case. I’m diving into this problem a bit deeply, because my network is senstive to the statistical properties of the specified tensors.</p>
Empty columns in confusion matrix

tanishka: <p>I am training on pre-processed APTOS 2019 dataset for disease grading and the last two columns of my confusion matrix are constantly zero every time. Data is distributed as class0-1805 images, class1-370 images, class3-999 images, class4-193 images, class5-295 images. Here are results on 70 epochs:<br>
Test acc=77%<br>
Confusion matrix<br>
[[258   3   1   0   0]<br>
[  6  38  16   0   0]<br>
[ 13  13 130   0   0]<br>
[  0   7  22   0   0]<br>
[  4   9  30   0   0]]<br>
Classwise accuracy on test set of model [98.47328244 63.33333333 83.33333333  0.          0.        ]<br>
Sensitivity (Recall) for each class: [0.98473282 0.63333333 0.83333333 0.         0.        ]<br>
Specificity for each class: [0.98513011 0.95416667 0.92592593 0.94727273 0.92181818]<br>
using class weights didn’t help.<br>
Please help!</p>
Why first forward pass is significantly faster than the rest

Jitindra: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2ddc1553973f8c2969624127aa23d3619bdaea0.png" data-download-href="https://discuss.pytorch.org/uploads/default/b2ddc1553973f8c2969624127aa23d3619bdaea0" title="Screenshot 2024-10-05 at 17.43.27"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2ddc1553973f8c2969624127aa23d3619bdaea0_2_349x500.png" alt="Screenshot 2024-10-05 at 17.43.27" data-base62-sha1="pwk650YYWRWeasFzWt9FI9brHvW" width="349" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2ddc1553973f8c2969624127aa23d3619bdaea0_2_349x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2ddc1553973f8c2969624127aa23d3619bdaea0_2_523x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b2ddc1553973f8c2969624127aa23d3619bdaea0_2_698x1000.png 2x" data-dominant-color="2B2C30"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-10-05 at 17.43.27</span><span class="informations">1052×1505 152 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why the first forward pass is faster than the rest of the forward pass.<br>
Device: Mac (mps)<br>
Batch size: 8<br>
num_workers: 0<br>
pin_memory: True<br>
torch.backends.cudnn.benchmark = True<br>
Model Architecture: 2D UNET model<br>
Input Image Size: 256</p>
<p>Code:</p>
<pre><code class="lang-auto">def train_one_epoch(device, model, dl, optim, loss):
    start_time = time()
    total_loss = 0.
    model.train()

    for i, batch in enumerate(dl):
        optim.zero_grad(set_to_none=True)

        input, target = batch[0].to(device, non_blocking=True), batch[1].to(device, non_blocking=True)

        output_start_time = time()
        output = model(input)
        print(f'Output time: {time()-output_start_time}')
        torch.mps.empty_cache()
        batch_loss = loss(output, target)
        total_loss += batch_loss.item()
        backward_start_time = time()
        batch_loss.backward()
        print(f'Backward pass time: {time() - backward_start_time}')
        optim_start_time = time()
        optim.step()
        print(f'Optim time: {time() - optim_start_time} \n')
        if i &gt;=5:
            break

    return total_loss/len(dl), time()-start_time
</code></pre>
Cannot allocate memory for thread-local data: ABORT

nomisto: <p>Very unspecific error, however when I try to allocate a tensor (after some preceding memory intensive computations using pybind11 and C++)</p>
<pre><code class="lang-auto">scores = torch.zeros(5000, 15000, device="cpu")
</code></pre>
<p>I get the following error (presumably from C++ backend):<br>
<code>cannot allocate memory for thread-local data: ABORT</code></p>
<p>Any ideas?</p>
Using masking during training

korunosk: <p>Hello everyone,</p>
<p>I have data batches with dimensions (100,20,768) that I pass through a two-layer neural network. I also have masks with dimensions (100,20,1) for each batch that tell which of the samples are relevant for training. How can I stop the model from updating its weights for the samples where the mask is 0?</p>
<p>The reason why I am doing this is that the original samples are with varying number of rows. So I stacked zero tensors underneath to increase the number of rows so I can support batching, because, evaluation took a lot of time when doing it sample by sample.</p>
<p>Or, maybe there is a better approach for handling inputs with variable number of rows?</p>
<p>Thanks in advance.</p>

sathvik_udupa: <p>Quick fix, why don’t you ignore inputs with mask 0 if you aren’t training with it?</p>
<p>Are you using ANN or anything else? With CNN, you can use adaptive average pooling which converts varying input into a fixed length output.<br>
You can also create chunks of your input, for eg: with chunk length 20, slide length 10. This works well for CNN, LSTM, not sure about ANN.</p>

korunosk: <p>It is a simple two layer NN that tries to learn a transformation.</p>
<p>Is multiplying the input with the mask sufficient? It will nullify the samples but I think It will still calculate the gradient for those.</p>

sathvik_udupa: <p>I meant to say, why don’t you ignore thee data points with masks 0 if you don’t want to update weights? edited my reply appropriately</p>

korunosk: <p>That way the samples will have varying number of rows, and I cannot create batches. I would like to support batching since it increases the speed by 2 orders of magnitude.</p>

yanzhou01: <p>I am facing similar issue. Did you find a way to solve it? Best,</p>

abrahamowos: <p>is it possible to set the gradients of some specific rows in the layer matrix to be = 0. say the gradient then becomes [0,0,0,0, 0.5,0.04] for a  7x1 layer. I’m trying to use this to mask the embedding layers of a language model.</p>
_, pred = output.topk(maxk, 1, True, True) RuntimeError: selected index k out of range

Mai_Magdy: <p><strong>" i have 2 classes "</strong><br>
prec1, prec5 = accuracy(output.data, target, topk=(1,5))</p>
<p>def accuracy(output, target, topk=(1,)):</p>
<pre><code>maxk = max(topk)

batch_size = target.size(0)

print(maxk)

_, pred = output.topk(maxk, 1, True, True)

pred = pred.t()

correct = pred.eq(target.view(1, -1).expand_as(pred))

res = []

for k in topk:

    correct_k = correct[:k].view(-1).float().sum(0)

    res.append(correct_k.mul_(100.0 / batch_size))

return res</code></pre>

ptrblck: <p>The error is raised, if the <code>k</code> value is larger than the size of the specified dimension as seen here:</p>
<pre><code class="lang-python">output = torch.randn(3, 2)

maxk = 1
_, pred = output.topk(maxk, 1, True, True) # works

maxk = 2
_, pred = output.topk(maxk, 1, True, True) # works

maxk = 3
_, pred = output.topk(maxk, 1, True, True) # fails
&gt; RuntimeError: selected index k out of range
</code></pre>
<p>so you would have to check <code>output.shape</code> and make sure <code>dim1</code> is larger or equal to <code>maxk</code>.</p>

Mai_Magdy: <p>output.shape is [64,2]</p>

ptrblck: <p>The passed <code>topk</code> argument as <code>topk=(1,5)</code> would yield this error, since <code>maxk</code> would be <code>5</code>, which is larger than the size in dim1, which is 2.</p>

idil: <p>Since <code>maxk == dim1</code> fails, I believe this would work only for the larger case but not equal.</p>
Fault-tolerance for large scale training

zxiu: <p>Hello community!</p>
<p>I’m hoping to get some recommendations regarding fault-tolerance in large scale training.</p>
<p>From what I’ve read, one practice is to restart the job when a failure is detected and resuming from the last saved checkpoint. Then the problem becomes that how frequent the checkpoint can be saved and how fast the checkpoint can be loaded and I’ve seen efforts from the pytorch community on that like <a href="https://pytorch.org/tutorials/recipes/distributed_async_checkpoint_recipe.html" rel="noopener nofollow ugc">async checkpointing</a>.</p>
<p>Besides that, wondering if there’re other methods that can be leveraged. For example, can torchelastic be used for large scale training when there’s more than data parallelism but also TP and PP? From torchelastic’s documentation, it does not seem like the case but I was hoping to get more clarifications from the experts here.</p>
<p>Any advice is appreciated!</p>
Different results, and -sometimes- all nans

apytorch: <p>I noticed when doing inference on my custom model (that was always trained and evaluated using cuda) produces slightly different inference results on cpu. (The percent of instances with different classes is small, but still of interest).</p>
<p>However, I then tried it on mps as was blown away. Not only are the results different than cuda and cpu, but some of the examples will produce all nans when running on mps. Furthermore, it isn’t deterministic–which examples are all nans will change randomly.</p>
<p>I’ll add that I’m running this before everything (although, I understand most or all of these only apply to cuda, which is what the code has always been run on)</p>
<pre><code>os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" # must set variable beforehand
torch.use_deterministic_algorithms(True)
# When running on the CuDNN backend, two further options must be set
torch.backends.cudnn.benchmark = False # type: ignore
torch.backends.cudnn.deterministic = True # type: ignore
</code></pre>

AlexP4499: <p>What is your code for inference?</p>
<p>I’m assuming you’ve already set the model to inference mode / eval mode?</p>

apytorch: <p>Yes, the model is always set with .eval() before inference (and .train() before training).</p>
<blockquote>
<p>What is your code for inference?</p>
</blockquote>
<p>I’m curious how much code you are looking for here? I’ll add, I’ve -never- gotten nans on CUDA (and I’ve been running this same code, more or less, for several years), and when I recently tested on CPU, I also did not get any nans.</p>

apytorch: <p>Okay, I think I traced where the issue first crops up when using mps (and only mps, not cuda or cpu). The problem is because something is corrupting the input tensors before they even get to the forward pass.</p>
<p>I’ve verified that the issue is occurring with <code>.to(device, non_blocking=True)</code>. Only when non_blocking is <code>True</code>. Strangely, if I access the tensor (just store a reference to it, but never use it, before the <code>to</code> operation, then the corruption doesn’t happen. I can certainly add a conditional to check for ‘cuda’ (I guess, same with <code>pin_memory</code> for the dataloader), but the other discussions I can find online suggest this shouldn’t be an issue for mps (or was, but not with pytorch 2.4.1).</p>
<pre><code class="lang-auto"># this is a very simplified snippet of the code

for _, batch_data in enumerate(loader):
    # if I uncomment this next line, then no issue
    # dummy_variable = batch_data["input_tensor"]

    # if non_blocking is False, then no issue
    batch_data["input_tensor"] = batch_data["input_tensor"].to(device, non_blocking=True)

</code></pre>

apytorch: <p>I’ve since added <code>torch.mps.synchronize()</code> for the condition that mps is being used (which I didn’t think about, even though I had <code>torch.cuda.synchronize()</code> already there). I still find it strange that accessing the variable before the copy/move to mps was also a solution.</p>
[Flex Attention] Slicing block mask into micro-batches?

Avelina9X: <p>Computing a block mask is expensive, and as I understand it, it would make sense to perform this computation as few times as possible. As such, I feel like it would make sense to compute this mask for an entire batch, and then slice the block mask up into micro-batches when performing gradient accumulation. However I don’t know if or how this would be possible with the <code>BlockMask</code> object, especially since it appears that operations such as ANDing or ORing block masks is done by mutating the mask mod functions, rather than operating on actual BlockMask objects.</p>