akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to judge a tensor is view?

tgsmdww: <p>Like</p>
<pre><code class="lang-auto">import torch
x = torch.rand(2,3,4,5)
y = x.view(4,6,5)
</code></pre>
<p>If I want to know y is a view of x, which method can I use?</p>

pascal_notsawo: <p>According to the <a href="https://pytorch.org/docs/stable/tensor_view.html" rel="noopener nofollow ugc">documentation</a> you can know this by doing :<br>
<code>x.storage().data_ptr() == y.storage().data_ptr()</code>.</p>

tgsmdww: <p>In my models it is used like <code>x = torch.rand(2,3,4,5).view(4,6,5)</code> so I can’t ger origin tensor’s storage data_ptr</p>

pascal_notsawo: <p>I understand, but in this case you have no choice but to do the following.</p>
<pre><code class="lang-auto">y = torch.rand(2,3,4,5)
x = y.view(4,6,5)
</code></pre>
<p><code>torch.rand(2,3,4,5)</code> is not useful here if it is not carried by a known variable.</p>

googlebot: <p>Moreover, x and y here are conceptually indistinguishable; strides differ, but there are no “primary” strides.</p>

shadowshadow: <p>you can check tensor._base, if tensor is a view, _base will point to base tensor</p>
Cannot find NCCL libnccl-net.so file

furkan_kestane: <p>NCCL cannot find libnccl-net.so and running nccl --version does not give any output, what could be the reason.</p>
<p>CUDA VERSİON = 12.0<br>
PYTHON VERSİON = 3.8<br>
TORCH VERSİON = 2.0.1</p>

ptrblck: <aside class="quote no-group" data-username="furkan_kestane" data-post="1" data-topic="191551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/furkan_kestane/48/65349_2.png" class="avatar"> furkan_kestane:</div>
<blockquote>
<p>NCCL cannot find libnccl-net.so</p>
</blockquote>
</aside>
<p>You might want to install it in your setup then. More information about net plugins can be found <a href="https://github.com/NVIDIA/nccl/tree/0e35f5d3902f4e30c4c97b9700a6a64b25ba7f7f/ext-net">here</a>.</p>
<aside class="quote no-group" data-username="furkan_kestane" data-post="1" data-topic="191551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/furkan_kestane/48/65349_2.png" class="avatar"> furkan_kestane:</div>
<blockquote>
<p>running nccl --version does not give any output,</p>
</blockquote>
</aside>
<p><code>nccl</code> is not a binary command so unsure what exactly you are trying to run. NCCL is a library used inside PyTorch and you cannot execute <code>nccl --version</code> as a command in your terminal.</p>

lroberts: <p>you can use something like:</p>
<blockquote>
<p>import torch<br>
torch.cuda.nccl.version() # for me gives → (2, 14, 3)</p>
</blockquote>
<p>to get the nccl version being used by torch.</p>

furkan_kestane: <p><span class="mention">@prt</span> I don’t run nccl in the terminal, it just says that a certain file is not used and no matter how much I install pytorch, it can’t find this file in the desired path. Can I fix the problem if I manually install pytorch from githup?</p>

furkan_kestane: <p>Tried and tested.The problem is that some files are missing.</p>

Karl_Lok: <p>you can compile by yourself from <a href="https://github.com/Mellanox/nccl-rdma-sharp-plugins" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Mellanox/nccl-rdma-sharp-plugins: RDMA and SHARP plugins for nccl library</a></p>
Convert a tuple to torch tensor

Oussama_Bouldjedri: <p>I working on capsule networks and I get this error once I go through the loss function :</p>
<p>AttributeError: ‘tuple’ object has no attribute ‘size’ the error comes form this line:</p>
<p>t = torch.zeros(lengths.size()).long()</p>
<p>I tired multiple ways to convert it to tuples but always has an issue with detach from the GPU</p>
<p>this is the loss function:</p>
<p>class MarginLoss(nn.Module):<br>
def <strong>init</strong>(self, m_pos, m_neg, lambda_):<br>
super(MarginLoss, self).<strong>init</strong>()<br>
self.m_pos = m_pos<br>
self.m_neg = m_neg<br>
self.lambda_ = lambda_</p>
<pre><code>def forward(self, lengths, targets, size_average=True):
    
    t = torch.zeros(lengths.size()).long()
    if targets.is_cuda:
        t = t.cuda()
    t = t.scatter_(1, targets.data.view(-1, 1), 1)
    targets = Variable(t)
    losses = targets.float() * F.relu(self.m_pos - lengths).pow(2) + \
             self.lambda_ * (1. - targets.float()) * F.relu(lengths - self.m_neg).pow(2)
    return losses.mean() if size_average else losses.sum()
</code></pre>
<p>Thanks in advance</p>

ptrblck: <p>You can directly transform a <code>tuple</code> to a <code>tensor</code> if the entries are suitable:</p>
<pre data-code-wrap="python"><code class="lang-python">a = (1., 2.)
print(type(a))
# &lt;class 'tuple'&gt;

b = torch.tensor(a)
print(b)
# tensor([1., 2.])
</code></pre>

Oussama_Bouldjedri: <p>I just tried that it throws back this error :</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>

ptrblck: <p>What does the <code>tuple</code> contain?</p>

Oussama_Bouldjedri: <p>it contains the out put of the model : some thing like this<br>
log_probs=model(x)<br>
loss=margin_loss(log_probs,labels)</p>
<p>ones it fall in parameters log_probs will be lenghts and labels will be target</p>
1D convolutions in parallel

Andrew_Ulmer: <p>Hi, I have a set of K 1-dimensional convolutional filters. I want to convolve them temporally with a matrix Z, which has a shape (batches, time, K). The output should be (batches, time - (filter_length / 2), K), where each output dimension is simply the corresponding input dimension convolved with its respective filter. I want to avoid looping over each of the K dimensions using conv1d - how can I perform the convolution of each input dimension in parallel to save some time? Would conv2d accomplish this?</p>

ptrblck: <p>Based on your description it sounds like you are searching for depthwise convolutions where <code>groups==in_channels</code>.<br>
From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html">docs</a>:</p>
<blockquote>
<p>groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,</p>
</blockquote>
<pre><code>    At groups=1, all inputs are convolved to all outputs.

    At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels and producing half the output channels, and both subsequently concatenated.

    At groups= in_channels, each input channel is convolved with its own set of filters (of size out_channelsin_channelsin_channelsout_channels​).
</code></pre>
Will Convolution layers without activation collapse into one layer?

CDhere: <p>In plain neural networks, if we don’t apply activations (e.g., ReLU) after the weights, the matrices of multiple layers will mathematically collapse into one matrix due to simple matrix multiplications. I wonder if that’s also true for convolutional layers? I.e.:</p>
<p>If we don’t use nonlinearities after the convolution and stack multiple layers, will they effectively degenerate to fewer layers too? Can we mathematically prove it either they will/will not?</p>

googlebot: <p>Yes, because conv can be expressed as a masked matrix multiplication.</p>

Bjorn_Lindqvist: <p>Suppose that you stack two 3x3 convolutions. Now to compute the value of every pixel you have to look at the neighbors of the neighbors too. Effectively, the kernel size grows to 5x5. A 5x5 convolution requires 25 multiplies per pixel so it is not necessarily faster than two 3x3 convolutions which only require 18 multiplies.</p>
[CUDA] Does/Why `masked_select` Internally Use `aten::nonzero`?

HyperHyper: <p>I am micro-benchmarking <code>masked_select</code>, direct indexing and <code>index_select</code>  on a <strong>1D tensor</strong> to compare their performance. After running my benchmarks, I noticed that <strong>all operations exhibit very similar execution times</strong>.</p>
<p>I also examined the <strong>TorchScript graph IR</strong>, but it closely resembles my original Python code—except for the <strong>ATen function prefixes</strong>. To investigate further, I used the <strong>PyTorch Profiler</strong>, and in the profiling results, I observed that <strong>both <code>masked_select</code> and indexing include <code>aten::nonzero</code></strong>, with a <strong>similar overhead portion</strong>.</p>
<pre><code class="lang-auto">def is(tensor, threshold: int):
    mask = torch.lt(tensor, threshold)
    vector = torch.nonzero(mask).squeeze()
    return torch.index_select(tensor, 0, vector)
def ms(tensor, threshold: int):
    mask = torch.lt(tensor, threshold)
    return torch.masked_select(tensor, mask)
def indexing(tensor, threshold):
    return tensor[tensor &lt; threshold]
</code></pre>
<p>This raises a few questions:</p>
<ol>
<li><strong>Do <code>masked_select</code> and direct tensor indexing (<code>tensor[mask]</code>) all rely on <code>aten::nonzero</code> after graph IR transformation?</strong></li>
<li><strong>If they indeed share the same implementation under the hood, does that mean they are ultimately performing an <code>index_select</code> operation?</strong></li>
<li><strong>Is there a way to avoid the overhead of <code>aten::nonzero</code> when using <code>masked_select</code> or indexing?</strong></li>
</ol>
<p>Any insights on how PyTorch handles these operations internally and possible workarounds to avoid <code>aten::nonzero</code> would be greatly appreciated.</p>
<p>Thanks!</p>
Activation Checkpointing wrapper failed with torch.jit.save

jie_liu1: <p>I found that the current activation checkpointing wrapper <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.py at main · pytorch/pytorch · GitHub</a> can make the model fail at the torchscripting stage, especially in torch.jit.save.</p>
<pre><code class="lang-auto"> RuntimeError:
  Could not export Python function call '_NoopSaveInputs'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
  /usr/local/lib/python3.8/site-packages/torch/autograd/function.py(539): apply
  /usr/local/lib/python3.8/site-packages/torch/utils/checkpoint.py(1203): _checkpoint_without_reentrant_generator
  /usr/local/lib/python3.8/site-packages/torch/utils/checkpoint.py(457): checkpoint
  /usr/local/lib/python3.8/site-packages/torch/_dynamo/external_utils.py(17): inner
  /usr/local/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py(328): _fn
  /usr/local/lib/python3.8/site-packages/torch/_compile.py(24): inner
</code></pre>
<p>But if I just use the raw api <a href="https://pytorch.org/docs/stable/checkpoint.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.checkpoint — PyTorch 2.6 documentation</a>, then it is fine.<br>
Wonder the checkpointing wrapper essentially does not support the torchscripting?</p>

ptrblck: <p>TorchScript is in maintenance mode and I would thus not expect it to be compatible with all recent utils.</p>

jie_liu1: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> curious wonder what is the future plan for torchscript?</p>

ptrblck: <p>TorchScript is not in active development anymore and won’t receive major updates or fixes anymore and the recommended approach is to use <code>torch.compile</code> or <code>torch.export</code>.</p>
Torch.onnx.export with FakeTensor

temikgo: <p>I use <code>torch.onnx.dynamo_export</code> with FakeTensors, like this:</p>
<pre><code class="lang-auto">with torch.onnx.enable_fake_mode() as fake_context:
        x = torch.empty((2, 128, 1), dtype=torch.float16)
        model = Transformer(args)
export_options = torch.onnx.ExportOptions(
        fake_context=fake_context
)
export_output = torch.onnx.dynamo_export(
        torch_model, input_data, export_options=export_options
)
</code></pre>
<p><code>torch.onnx.dynamo_export</code> was recently deprecated<br>
but how can I use <code>torch.onnx.export(... dynamo=True ...)</code> with FakeTensors inside?</p>
Future of realtime inference in pytorch

victor-shepardson: <p>There is substantial interest in using pytorch for realtime deployments in e.g. audio, robotics. To my knowledge the standard way to do this is using torchscript / libtorch C++ API. However, the jit interpreter causes realtime violations with memory allocation and thread synchronization (see <a href="https://ieeexplore.ieee.org/document/10704099/?arnumber=10704099" class="inline-onebox" rel="noopener nofollow ugc">ANIRA: An Architecture for Neural Network Inference in Real-Time Audio Applications | IEEE Conference Publication | IEEE Xplore</a>).</p>
<p>I recall there was work on a “StaticRuntime” backend which would avoid most memory allocation when running the same module repeatedly, but it appears to have ceased, and now torch.jit itself seems to have been abandoned in favor of torch.compile, torch.export, AOTInductor and/or executorch (?)</p>
<p>At the moment it’s very hard to understand which APIs to use and what the intended tradeoffs are re:low latency applications, since most examples discuss vision or language modeling.</p>
<p>In particular, which technologies can be real-time friendly re: memory allocation? And which support internal mutable state the way torch.jit.ScriptModule does? Can someone knowledgeable comment on the near-future outlook for low-latency realtime inference?</p>

smth: <p>Hey Victor.</p>
<blockquote>
<p>now torch.jit itself seems to have been abandoned in favor of torch.compile, torch.export, AOTInductor and/or executorch (?)</p>
</blockquote>
<p>This is correct. <code>torch.export</code>ed programs can be targeted to run on AOTInductor or executorch.</p>
<p>If you are looking for a static runtime, Executorch interpreter qualifies.</p>
<p>for low-latency realtime inference for edge devices, executorch keeps up that discipline.<br>
AOTInductor also avoids any runtime dynamism as such, but does not avoid dynamic memory allocations.</p>
nn.Upsample bilinear mode algorithm

DNievesAcaron: <p>Greetings. I’ve been trying to find more resources about how exactly the output of nn.Upsample is determined when the mode is set to bilinear, but I have yet to find any as I got lost trying to find it in the rabbit hole that is the Pytorch repoistory.</p>
<p>Could anyone refer me to some document or the exact line of code where the bilinear mode for nn.Upsample gets called?</p>

ritwick_jha1: <p><a href="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/upsampling.py#L14" rel="noopener nofollow ugc">pytorch/torch/nn/modules/upsampling.py at v2.6.0 · pytorch/pytorch · GitHub</a></p>
<p>looks like it just calls F.interpolate</p>
<p>which has different implementations for bilinear mode depending upon the options used.<br>
The source implementation is probably coming from the C++ backend.</p>
<p>doing a quick search, maybe here.</p>
<p><a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cpu/UpSampleKernel.cpp" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/cpu/UpSampleKernel.cpp at main · pytorch/pytorch</a></p>
Need Language Model Advice

TheVictor_777: <p>Hi everybody,<br>
I’ve recently been training a small language model of my own from scratch - no pre-trained models whatsoever. I know that language models predict the next character based on a text input, but I figure that I need to give my language model custom instructions (behind the scenes) to behave like an actual conversational chat. Otherwise, it just continues the encoded text sequence which I have started. Assuming I’m on the right track with custom instructions, what should they look like?<br>
Currently, I have something like this:</p>
<pre><code class="lang-auto">input_string = f"You are MODEL_NAME, a conversational language model developed by DEVELOPER. The user said: 'USER_INPUT'. \nYou respond: "
</code></pre>
<p>Thanks for any advice!</p>
About torch.autograd.set_detect_anomaly(True):

Yangmin: <p>Hello.</p>
<p>I am training a CNN network with cross_entropy loss.<br>
When I train the network with debugging tool wrapped up</p>
<p>“with torch.autograd.set_detect_anomaly(True):”</p>
<p>I get runtime error like this,</p>
<p>[W python_anomaly_mode.cpp:60] Warning: Error detected in CudnnConvolutionBackward. Traceback of forward call that caused the error<br>
self.scaler.scale(self.losses).backward()<br>
File “/root/anaconda3/envs/gcl/lib/python3.7/site-packages/torch/tensor.py”, line 185, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/root/anaconda3/envs/gcl/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 127, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
<strong>RuntimeError: Function ‘CudnnConvolutionBackward’ returned nan values in its 0th output.</strong></p>
<p>I can see that the gradients are exploding? and getting an error like this but what I feel<br>
weird is that when I don’t use “with torch.autograd.set_detect_anomaly(True):”</p>
<p>I do not get any error…</p>
<ol>
<li>
<p>Why does only error come up when I use “with torch.autograd.set_detect_anomaly(True):”??</p>
</li>
<li>
<p>The final loss value is calculated properly I think. Is this NAN value occur because I did something wrong during forward pass like cutting the computational graph??</p>
</li>
</ol>
<p>Thanks in advance.!</p>

ptrblck: <ol>
<li>
<p><code>set_detect_anomaly(True)</code> is used to explicitly raise an error with a stack trace to easier debug which operation might have created the invalid values. Without setting this global flag, the invalid values would just be created and the training might be broken (e.g. if you update any parameter to NaN).</p>
</li>
<li>
<p>NaN values can be raised through multiple operations and often it’s used when e.g. you are dividing by zero in the model (or a very small value close to zero) and forget to add a small <code>eps</code> value to the divisor.</p>
</li>
</ol>

Yangmin: <p>I understand that “set_detect_anomaly(True)” finds an error and shows up in more detail.</p>
<p>When I don’t use it, the invalid values are just created and sometimes do not occur error in the forward pass?</p>

ptrblck: <p><code>set_detect_anomaly</code> won’t change the behavior of your code, just the error reporting. If you are not using it, the invalid values would also be created, but you would just “use” them, i.e. you might update your model parameters to NaNs etc.</p>

Yangmin: <p>I understand.<br>
Thank you so much.!</p>

yassineAlouini: <p>One additional thing to note is that the <code>set_detect_anomaly</code> context adds a big overhead on the execution speed so don’t use it in production once you have debugged your model (i.e. identified the source of the <code>NaN</code>).</p>
Torch IR support for torch.exported graph

Sung_Hyuck_Hong: <p>Hello,<br>
Is there any plan in PyTorch’s roadmap to enable the retrieval of a <code>GraphModule</code> with Torch IR directly in the future? To the best of my knowledge, tracing the model with <code>torch.export.export</code> directly will result in <code>GraphModule</code> with Aten IR. Therefore, I am currently using <code>torch._dynamo.export(model,...,aten_graph = False)(inputs)</code> to trace <code>GraphModule</code> in which basic torch.nn.Modules (e.g., Linear layer, matmul) remain as leaf nodes, as I am trying graph optimization on Torch IR level. I’m wondering if using <code>torch._dynamo.export</code> will remain the recommended approach for my purpose of interest moving forward, or if there would be any upcoming changes that might make it easier or more efficient to obtain the graph module with Torch IR. The only concern I have is that the underbar in “_dynamo” might implicate that it can be discarded in future torch release <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
ImportError : cannot import name ‘DiagnosticOptions’ from ‘torch.onnx._internal.exporter’

cyy: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3a542821235a5d5307f1577a9ede9d33da98334.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3a542821235a5d5307f1577a9ede9d33da98334" title="屏幕截图 2025-01-28 101341"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a542821235a5d5307f1577a9ede9d33da98334_2_690x190.png" alt="屏幕截图 2025-01-28 101341" data-base62-sha1="nlFTv14R1kEhRHvvvw6BTEjThtO" width="690" height="190" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a542821235a5d5307f1577a9ede9d33da98334_2_690x190.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a542821235a5d5307f1577a9ede9d33da98334_2_1035x285.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a542821235a5d5307f1577a9ede9d33da98334_2_1380x380.png 2x" data-dominant-color="F4F1F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2025-01-28 101341</span><span class="informations">2338×646 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ashok_arora: <p>Hey, were you able to solve this error?</p>
Best way to save a model including its parameters

Bjorn_Lindqvist: <p>What is the best way to save a model <em>including</em> parameters? A few different ways are discussed in <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 2.6.0+cu124 documentation</a>, but they all have drawbacks.</p>
<ul>
<li><code>torch.save(model.state_dict(), PATH)</code>: Doesn’t save the architecture, only the parameters.</li>
<li>" Save/Load Entire Model": Not recommended because “pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.”</li>
<li>" Export/Load Model in TorchScript Format": TorchScript is deprecated.</li>
<li>ONNX: Clumsy an doesn’t work for all models.</li>
</ul>

ptrblck: <p>The recommended approach would be the first one of saving the <code>state_dict</code> of the model as it would avoid common pitfalls causing issues during model loading.</p>

Bjorn_Lindqvist: <p>Sure, but then you need to have the source code of the model available in a different file. I want to be able to call my script like “python runmodel.py model-file” and run whatever kind of model model-file contains.</p>

ptrblck: <aside class="quote no-group" data-username="Bjorn_Lindqvist" data-post="4" data-topic="216331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bjorn_lindqvist/48/25326_2.png" class="avatar"> Bjorn_Lindqvist:</div>
<blockquote>
<p>Sure, but then you need to have the source code of the model available in a different file.</p>
</blockquote>
</aside>
<p>Yes, which is a softer requirement than the need to keep source files in the same structure without breaking changes if you store the “entire” model via pickle.</p>

Bjorn_Lindqvist: <p>Not sure I understand? Cause there <em>are</em> serialization formats that can store the weights <em>and</em> the model structure without causing fuss.</p>

Shmes1955: <p>Do you think that It<a href="https://palmbeachpropertyappraisers.com" rel="noopener nofollow ugc">’</a>s a best option??</p>

ptrblck: <aside class="quote no-group" data-username="Bjorn_Lindqvist" data-post="6" data-topic="216331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bjorn_lindqvist/48/25326_2.png" class="avatar"> Bjorn_Lindqvist:</div>
<blockquote>
<p>Cause there <em>are</em> serialization formats that can store the weights <em>and</em> the model structure without causing fuss.</p>
</blockquote>
</aside>
<p>Which ones are these?</p>

Bjorn_Lindqvist: <p>ONNX. But it’s painful and annoying to use.</p>
Confused between CUDA versions

Sourabh_Yadav: <p>in nvidia-smi I have cuda 12.8.<br>
From Pytorch, I have downloaded 12.6 One and I have the latest Nvidia drivers also. I am on Win 11 PC , intel chip v100 2x-32Gb<br>
 → Also if somewhere in some env I install torch version 12.4 / 11.8 will my CUDA 12.8 mess things up?<br>
 → Is cuDNN also required if I am working with LLMs?</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used as PyTorch binaries ship with their own CUDA runtime. The driver requirements are explained <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility">here</a>.</p>
What is the difference between the CUDA API and CUDA HW lines in the Nsight Systems GUI?

hello_e: <p>If I have the following code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">kernel1(xxx)
....other cpu code...
torch.cuda.nvtx.push_range("marker")
....other cpu code...
</code></pre>
<p>In the Nsight Systems GUI, does the time from the end of kernel1 in CUDA API to the start of the “marker” in the NVTX represent the execution time of the “other cpu code” even if I do not set CUDA_LAUNCH_BLOCKING=1 ?</p>
<p>My guess is that since the CUDA API line represents the CPU-side activity of kernel launches and the execution on the CPU is synchronous, the answer to the above question is “yes.”</p>
<p>If the graph displayed in the Nsight Systems GUI looks like the following image (note that CUDA_LAUNCH_BLOCKING was not set during the nsys profiling), and the code being run is:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">add_kernel(xxx)
print("hello")
print("hello")
torch.cuda.nvtx.range_push("after add_kernel")
</code></pre>
<p>Then the 7.474us shown in the graph is the time required for the two print statements, and it is also the time from when the CPU has launched the <code>add_kernel</code> to when it has executed the <code>range_push</code>, correct?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27cbe5e88c4ed57d27e04ca70978791ed7307f48.png" data-download-href="https://discuss.pytorch.org/uploads/default/27cbe5e88c4ed57d27e04ca70978791ed7307f48" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27cbe5e88c4ed57d27e04ca70978791ed7307f48_2_690x248.png" alt="image" data-base62-sha1="5G3tcoXTBHHmJa1YNDuVOUsVccU" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27cbe5e88c4ed57d27e04ca70978791ed7307f48_2_690x248.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27cbe5e88c4ed57d27e04ca70978791ed7307f48_2_1035x372.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27cbe5e88c4ed57d27e04ca70978791ed7307f48_2_1380x496.png 2x" data-dominant-color="E5E8E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1685×607 59.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to save checkpoint when using FSDP2?

Wongboo: <p>For FSDP+checkpoint, we have an awesome <a href="https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html" rel="noopener nofollow ugc">doc</a>.<br>
For FSDP2+checkpoint, the doc simply says</p>
<pre><code class="lang-auto">FSDP2 does not directly support full state dicts. Instead, users can reshard the sharded state dicts containing DTensor s to full state dicts themselves using DTensor APIs like DTensor.full_tensor() or by using higher-level APIs like PyTorch Distributed Checkpoint‘s distributed state dict APIs.
</code></pre>
<p>However, it’s too obscure and I have no clue on it.<br>
BTW: is it possible to use FSDP2 API to mimic DDP?</p>
How to deal with multiple loss backward when using apply_optimizer_in_backward?

biggerfish: <p>Sometime we need to use apply_optimizer_in_backward for performance. However, in some cases when some parts are not differentiable, we have to manually calculate part of the gradients and manually apply the gradients.</p>
<p>For example, in the code snippet below, there are two losses, one can be directly calculated using <code>nn.MSELoss</code>, another has non-differentiable part and must be calculated manually.</p>
<ul>
<li>Without fusion, there are 2 gradients accumulation and 1 optimization step</li>
<li>With fusion, there are 2 (gradient+optimize) fused step.</li>
</ul>
<p>My question is: <strong>now I perform 2 optimize, it might affect the convergence, what is the best practice for such situation?</strong></p>
<pre data-code-wrap="python"><code class="lang-python"># linear layer and loss function
mse_loss = nn.MSELoss()

# input data and labels
x = torch.rand(3 ,5)
labels = torch.randint(low=0, high=2, size=(3, 1))

# logits
logits = model(x)

# loss
loss1 = mse_loss(logits, labels)
loss1.backward() # this will perform a backward&amp;optimize fusion, and parameters will be updated?

# another loss with non-differentiable part, must be calculated manually
custom_gradient = NonDifferentiableLogic(...)
torch.autograd.backward(logits, grad_tensors=custom_gradient) # this will perform another backward&amp;optimize fusion, and parameters will be updated again?
</code></pre>

soulitzer: <p>Could you have folded the second backward into the first by doing the non-differentiable logic as part of a custom autograd Function?</p>

biggerfish: <p>Would be good if can merge loss, however, that part is legacy code, and I wonder if there is alternatives. Thanks anyway.</p>
Issues with CTCLoss

Butzti: <p>Hey,<br>
So I’m trying to implement the DeepSpeech2 architecture using PyTorch but I’m currently having trouble with the CTCLoss. The documentation says, that the input to the CTCLoss should be logarithmic probabilites for each timestep, a tensor of the token sequences, the lengths of the spectrograms (which are the inputs to the model) and the lengths of the token sequences.</p>
<p>The DeepSpeech2 architecture has 2 convolutional layers with a stride &gt; 1. This means the timesteps of my spectrograms will be reduced by the convolutional layers. I’m pretty sure this is the reason im getting the following error.</p>
<p>RuntimeError: Expected input_lengths to have value at most 317, but got value 633 (while checking arguments for ctc_loss_allocate_outputs)</p>
<p>So my spectrogram originally has 633 timesteps but after the forward pass, there are only 317 timesteps. I’m pretty sure the documentation says, I have to use the lenghts of the input tensors and not the lenghts of the output of my model and since my model reduces the amount of timesteps im  getting the error. Do you know how I can fix this issue?</p>