akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The order of graph export and optimization of aot_autograd

guo_delta: <p>Below is my custom torch.compiler backend. I hope aot_autograd could export the backward graph after the forward graph passes. I may replace the op in forward graph, which has been registered through torch.library.custom_op()，and bind it to the backward op. But I found that after I modify the forward graph, the exported backward graph does not change with my modification. Is there any way to achieve my expectation?</p>
<pre><code class="lang-auto">import torch
import functools
from torch._dynamo import register_backend
from torch._dynamo.backends.common import aot_autograd, device_from_inputs
from functorch.compile import make_boxed_func, min_cut_rematerialization_partition

# torch._dynamo.config.suppress_errors = True

from .passes import *

@register_backend
def my_compiler(gm: torch.fx.GraphModule, example_inputs):

    def my_compiler_forward(gm, example_inputs):
        torch_pass(gm)
        def exec_fw(*i_args):
            return gm.forward(*i_args)

        return make_boxed_func(exec_fw)

    def my_compiler_backward(gm, example_inputs):
        def exec_bw(*i_args):
            return gm.forward(*i_args)

        return make_boxed_func(exec_bw)

    return aot_autograd(
        fw_compiler=my_compiler_forward,
        bw_compiler=my_compiler_backward,
    )(
        gm, example_inputs
    )
</code></pre>
OOM problem when using Multiple GPU with CUDA_VISIBLE_DEVICES

andy_sherlock: <p>I have a Weriod OOM problem when I use 2 gpu like  CUDA_VISIBLE_DEVICES=4,5.</p>
<p>But, when I use  CUDA_VISIBLE_DEVICES =4,5,6,7, it seems to work well. And I check the memory usage like this below, which showed that the total usage of gpu 4,5,6,7 is less than 40G*2=80G.</p>
<p>Does this mean that I can use 2 gpu like CUDA_VISIBLE_DEVICES=4,5 to work well ?</p>
<p>By the way, I also don’t understand why the memory allocated is not in an average for each gpu in the picture. Maybe it is the machanism of pytorch?</p>
<p>Thanks for your help in advance!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/7383ae8face215236b3014ef2e7d31913b58b66f.png" data-download-href="https://discuss.pytorch.org/uploads/default/7383ae8face215236b3014ef2e7d31913b58b66f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/7383ae8face215236b3014ef2e7d31913b58b66f.png" alt="image" data-base62-sha1="gtT2fg8OFdZFuGkSSdjTNEN54fl" width="690" height="315" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">972×444 19.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It’s unclear what your use case is and what’s causing the OOM.<br>
PyTorch’s <code>DDP</code> will create a balanced memory usage assuming you are using the same local batch size while the legacy <code>DP</code> approach uses an imbalanced memory usage and is thus not recommended (besides also increasing the communication need).</p>

andy_sherlock: <p>Thanks for your prompt reply, and sorry for missing some necessary details!<br>
I did not use either <code>DDP</code> or <code>DP</code>. And my case is only doing LLM inference and concretely, I load 3 models to do inference on each sample(for my special case, I can’t sequentially load each model and do it sequentially).</p>
<p>The purpose of using “CUDA_VISIBLE_DEVICES=4,5” to include multiple GPUs is to avoid OOM when loading 3 models in the same time.</p>
<p>Did I have a correct usage of “CUDA_VISIBLE_DEVICES=4,5”? Or what is the best practice for achieving the purpose of loading multiple models without OOM when I have a few 40g gpus but no 80g gpus?</p>
<p>Thanks for your advice in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">    model1 = AutoModelForCausalLM.from_pretrained(
        model_path,
        load_in_8bit=True,
        torch_dtype=torch.float16,
        device_map="auto"
    )
    model2 = ...
    model3 = ...
</code></pre>

andy_sherlock: <p>Thanks anyway, I have solved the problem. We should set <code>device_map</code> to <code>auto</code> instead of <code>gpu</code>.</p>
Why torch.onnx.export() generate so many files?

triloon: <p>I used <code>torch.onnx.export()</code> to convert my torchscript to onnx. But the result files can have so many look like weight / bias files:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63fb4fe5d78b584742f792b67485d7a56bfc3e49.png" alt="image" data-base62-sha1="egtHBmid6b8yPuzRt8340Vu48Y1" width="240" height="402"></p>

ptrblck: <p>Could you post the code which is creating these files, please?</p>

jing_xu: <p>I find that if the exported onnx file size exceed a range, eg. 2g, it will generate too much files more 1 onnx file. Is it a limition or something else? Have you got some update for it.</p>

triloon: <p>thanks for the info, I will look this very later <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>

JosephRocca: <p>I’m also having this problem, and I think Jing Xu might be right, because the model I’m trying to export is more than 2GB too (protobuf limit?)</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> here’s some code that replicates this - just click “Runtime &gt; Run all”, but I think you’ll need a to use a high-RAM runtime: <a href="https://colab.research.google.com/gist/josephrocca/2d367775455b4f0d72b40a274d7b05e0/copy-of-blip2_image_text_matching.ipynb" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>I’ve made an <a href="https://github.com/pytorch/pytorch/issues/94280" rel="noopener nofollow ugc">issue here</a> which includes some more details and speculations.</p>

krasnoteh: <p>It seems it is normal behaviour. Each file is a separate weight. The set of files you got is still a valid onnx model. You can load it with</p>
<pre><code class="lang-auto">onnx_model = onnx.load("model.onnx")
</code></pre>
<p>the model.onnx would be your main model file, onnx loader will automatically find all other files.</p>
<p>Now, when the model is under onnx control, you can save it again in more common format, where weighs are stored in one separate .data file.</p>
<pre><code class="lang-auto">from onnx.external_data_helper import convert_model_to_external_data

convert_model_to_external_data(
    onnx_model,
    all_tensors_to_one_file=True,  
    location="model.data",          
    size_threshold=0,              
    convert_attribute=False        
)

onnx.save_model(
    onnx_model,
    "model_consolidated.onnx",
    save_as_external_data=True,
    all_tensors_to_one_file=True,
    location="model.data",
    size_threshold=0,
)
</code></pre>
<p>Moreover this two files now can be used to create tensorrt engine, its common onnx parser reads it correct. Tested it with 5Gb SDXL, works for me.</p>
Nvcc fatal: Could not set up the environment for Microsoft Visual Studio using vcvars64.bat during JIT compilation of bias_act.cu (CUDA 12.6, VS2022 BuildTools)

cltexe: <p>I’m encountering an error while a PyTorch-based project tries to JIT compile <code>bias_act.cu</code>. The build fails with the following message from <code>nvcc</code>:</p>
<pre><code class="lang-auto">nvcc fatal   : Could not set up the environment for Microsoft Visual Studio using 'C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.43.34808/bin/HostX86/x64/../../../../../../../VC/Auxiliary/Build/vcvars64.bat'
ninja: build stopped: subcommand failed.
</code></pre>
<p>The path to <code>vcvars64.bat</code> seems correct, and running it manually in a terminal sets up the environment successfully (e.g., <code>cl.exe</code> becomes available). However, <code>nvcc</code> fails when attempting to use it automatically.</p>
<p><strong>Details:</strong></p>
<ul>
<li>This happens during JIT compilation of <code>bias_act.cu</code></li>
<li>CUDA version: <strong>12.6</strong></li>
<li>Visual Studio Build Tools 2022 are installed (no full IDE, just Build Tools)</li>
<li>The following is included in my system <code>PATH</code>:<br>
<code>C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.43.34808\bin\Hostx86\x64</code></li>
</ul>
MNIST Download Error

Ra_Concilio: <p>Using this code, I am getting the following error. Any advice on how to fix it?</p>
<p><strong>Code:</strong></p>
<blockquote>
<p>train_data =datasets.MNIST(<br>
root = ‘data’,<br>
train = True,<br>
transform = ToTensor(),<br>
download = True<br>
)</p>
</blockquote>
<p><strong>Error</strong></p>
<blockquote>
<p>File c:\Users\Name\AppData\Local\Programs\Python\Python313\Lib\site-packages\torchvision\datasets\mnist.py:100, in MNIST.<strong>init</strong>(self, root, train, transform, target_transform, download)<br>
97     return<br>
99 if download:<br>
 → 100     self.download()<br>
102 if not self._check_exists():<br>
103     raise RuntimeError(“Dataset not found. You can use download=True to download it”)</p>
</blockquote>
Mu not getting gradient

Andrew4: <p>the mu.grad.item() consisteny gets zero, is this normal? here is the code:</p>
<pre><code class="lang-auto">import torch
mu = torch.zeros(2, requires_grad=True)
sigma = 1.0
eps = torch.randn_like(mu)
sampled = mu + sigma * eps
logp = -((sampled - mu) ** 2) / (2 * sigma ** 2) - torch.log(torch.tensor(sigma)) - 0.5 * torch.log(torch.tensor(2 * torch.pi))
loss = -logp.sum()
loss.backward()
print("loss:", loss)
print("eps:", eps)
print("mu.grad:", mu.grad)
</code></pre>

Andrew4: <p>mu + sigma * eps - mu will cancel mu</p>
Quantization method diff between fake quant and true quant

CapJohn: <p>according to the <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fx/_decomposed.py#L598" rel="noopener nofollow ugc">code</a>. The quantization formula for true quant is</p>
<p>int32_t  res = round_half_even(value / scale) + zero_point;</p>
<p>and according to the document, the formula for fake quant is</p>
<p>int32_t  res = round_half_even(value / scale + zero_point);</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c30bb8b6eabef6d4e6042b60f63bd2f40521665.png" data-download-href="https://discuss.pytorch.org/uploads/default/8c30bb8b6eabef6d4e6042b60f63bd2f40521665" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c30bb8b6eabef6d4e6042b60f63bd2f40521665_2_690x294.png" alt="image" data-base62-sha1="k0bfO6eRPKfxecify49RyYBqQHH" width="690" height="294" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c30bb8b6eabef6d4e6042b60f63bd2f40521665_2_690x294.png, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c30bb8b6eabef6d4e6042b60f63bd2f40521665.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c30bb8b6eabef6d4e6042b60f63bd2f40521665.png 2x" data-dominant-color="F5F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">950×405 25.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why is there a difference between these two?</p>

jerryzh168: <p>probably the document is a bit outdated, we are unifying the implementation in <a href="https://github.com/pytorch/ao/blob/04259ebd96a8f1d29db16a7fe73a50d1f21bddef/torchao/quantization/quant_primitives.py#L426" class="inline-onebox" rel="noopener nofollow ugc">ao/torchao/quantization/quant_primitives.py at 04259ebd96a8f1d29db16a7fe73a50d1f21bddef · pytorch/ao · GitHub</a></p>
How to give muxer options to torio.io.StreamingMediaEncoder

RK777: <p>Hi,<br>
I am trying to start the StreamingMediaEncoder with format=hls and also give the following muxer options- “hls_playlist_type” and “hls_time” (<a href="https://ffmpeg.org/ffmpeg-formats.html#hls-2" rel="noopener nofollow ugc">ffmpeg reference</a>).<br>
But it seems that there is no argument for muxer options (<a href="https://pytorch.org/audio/2.3.0/generated/torio.io.StreamingMediaEncoder.html#torio.io.StreamingMediaEncoder" rel="noopener nofollow ugc">documentation link</a>). I tried setting the options under the encoder_option argument of the add_video_stream method, but this does not work and results in “RuntimeError: Unexpected options: hls_playlist_type, hls_time”.<br>
Is there any way to do this with torio?</p>
<pre><code class="lang-auto">import torch
from torio.io import StreamingMediaEncoder

writer = StreamingMediaEncoder(
    hls_output_path,
    format="hls",
    # No options
    # options={"hls_time": "2", "hls_playlist_type": "event"},
)

width = 640  # Frame width
height = 360  # Frame height
writer.add_video_stream(
    frame_rate=25,
    encoder="libx264",
    encoder_option={
        "g": "10",
    },
    width=width,
    height=height,
)

with writer.open():
    for i in range(100):
        print("Generating random frames")
        frame = torch.randint(0, 256, (10, 3, height, width), dtype=torch.uint8)
        print("Writing frame to stream...")
        writer.write_video_chunk(0, frame)

</code></pre>

phillies: <p>I have the same issue and would also be interested in a solution/workaround to pass <code>hls_time</code> etc. to ffmpeg.</p>
TypeError: stack(): argument 'tensors' (position 1) must be tuple of Tensors, not generator

Roshni_Pathak: <p>def forward(self,x2):<br>
image, clinical_data = x2<br>
clinical_data = torch.stack(clinical_data,dim=1).squeeze(0)  #(torch.arange(clinical_data), dim=1)<br>
x1 = self.modelA(image) <span class="hashtag">#imaging_data_pred</span><br>
x3 = torch.cat((x1, clinical_data), dim=2)<br>
x2 = self.modelB(F.relu(x3)) #‘combined_pred’<br>
<span class="hashtag">#x</span> = torch.cat((x1, x2), dim=1)<br>
return x2</p>
<pre><code>def training_step(self, batch):
    x = (batch[self.hparams["weighting"]],(batch[key] for key in self.keys_of_clinical))
    y = batch["Class"]
    y_hat = self.predict(x)
    loss = F.cross_entropy(y_hat, y)
    with torch.no_grad():
        y_hat = F.softmax(y_hat, dim=1)
    return {'loss': loss, "train_y_hat": y_hat, "train_y": y}
</code></pre>
<p>How do I resolve this error?</p>

Alexey_Demyanchuk: <aside class="quote no-group" data-username="Roshni_Pathak" data-post="1" data-topic="106744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roshni_pathak/48/32567_2.png" class="avatar"> Roshni_Pathak:</div>
<blockquote>
<p>clinical_data = torch.stack(clinical_data,dim=1).squeeze(0) #(torch.arange(clinical_data), dim=1)</p>
</blockquote>
</aside>
<p>It seems like in this line you are providing a generator <code>clinical_data</code> to the <code>stack</code> function. Make sure your input to this function is <code>tuple</code> or <code>list</code></p>

Roshni_Pathak: <p>Hey Alexey,<br>
Thanks! I passed it as a tuple and it worked</p>

z_mrtz: <p>Hey Pathak,<br>
Could you explain to me how you did it? I’m a beginner in PyTorch, and I don’t know how to do it.</p>
ANN implementation using c++ for a text classification model

Hits: <p>I am trying to build a text classification(dataset=17 classess and total 127 texts) model using ANN in c++ but after several approaches i am not able to reach that accuracy , any suggestion would be a great help:<br>
below is the model and the respective code for the reference:</p>
<pre><code>int64_t input_size = all_words.size();
int64_t output_size = all_categories.size();
int64_t hidden_size = 250;
double learning_rate = 0.001;

ANN model(input_size, hidden_size, output_size);
</code></pre>
<p>struct ANN : torch::nn::Module {<br>
ANN(int64_t input_size, int64_t hidden_size, int64_t output_size) {<br>
i2h = register_module(“i2h”, torch::nn::Linear(input_size, hidden_size));<br>
h2o = register_module(“h2o”, torch::nn::Linear(hidden_size, output_size));<br>
softmax = register_module(“softmax”, torch::nn::LogSoftmax(1));<br>
}</p>
<pre><code>torch::Tensor forward(torch::Tensor x) {
    x = i2h-&gt;forward(x);
    x = h2o-&gt;forward(x);
    x = softmax-&gt;forward(x);
    return x;
}

torch::nn::Linear i2h{nullptr}, h2o{nullptr};
torch::nn::LogSoftmax softmax{nullptr};
</code></pre>
<p>};</p>
Bug on running TorchScript on H100

mikeybydun1: <p>Hello,<br>
I have a torch script that using torch 1.13.0 (cuda version),<br>
I am compiling a pytorch code into .pt file and then run the model.<br>
On every gpu its working well (a100 for example),<br>
but when i run the same code on NVIDIA H100 the results just became nan.<br>
Do you have any idea why?<br>
Pytorch version? what i need to configure?<br>
Thanks!</p>

ptrblck: <p>PyTorch 1.13.0 was released with CUDA 11.6 and 11.7 while the Hopper architecture was introduced in CUDA 11.8 so you would need to update your PyTorch binary.</p>

mikeybydun1: <p>Thanks you very much my friend,<br>
I have another question. I have a PyTorch torch script that do:<br>
torch.prod of tensor in this shape: (1000,400,400,144).<br>
This takes so much time (10 seconds on A100),<br>
The only effective optimization i found is using bfloat16.<br>
You have other suggestions?<br>
Thanks!</p>

ptrblck: <p>Using a lower <code>dtype</code> sounds like a good idea.</p>
Is the use of CUDA Graph Python API discouraged?

joshkim: <p>Hi,</p>
<p>I’m currently using both the <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" rel="noopener nofollow ugc">CUDA Graph API</a> and torch.compile() in my source.</p>
<p>I’m aware that torch.compile also provides an option to automatically use CUDA graph based on this <a href="https://discuss.pytorch.org/t/how-to-use-torch-compile-with-cuda-graphs-when-using-gradient-activation-checkpointing/179466/2">post</a>.</p>
<p><strong>Question</strong>: It is fair to say that the manual construction of CUDA Graph using the aforementioned API is discouraged then <a href="https://github.com/pytorch/pytorch/issues/71896#issuecomment-1169241054" rel="noopener nofollow ugc">*based on this post</a>? I couldn’t find any real world example (other than the ones in  <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" rel="noopener nofollow ugc">CUDA Graph API</a>) I find it quite challenging to debug as the only debug option is to use CUDAGraph.debug_dump() and view the graph.</p>

joshkim: <p>Sorry I should’ve posted this under <code>torch.compile</code> category. I can’t edit it anymore for some reason, though.</p>

ptrblck: <p>Let me move it for you.</p>
Inductor codegen debugging

maheshattarde: <p>I am debugging cpp backend code generated for gcc. I took help from page <a href="https://pytorch.org/tutorials/intermediate/inductor_debug_cpu.html" class="inline-onebox" rel="noopener nofollow ugc">Inductor CPU backend debugging and profiling — PyTorch Tutorials 2.6.0+cu124 documentation</a> and i was able to generate output.py<br>
I need to peek into compilation process so i have TORCH_LOG=all set which provided me with compilation command. However I face issues for header files , also mentioned in documents mentioned above.</p>
<pre><code class="lang-auto">#include "/tmp/torchinductor_root/lr/clrlgu27q4ggd472umdzwsu6qcpqxcuusjxqvx2hwitjbujiiz7z.h"
</code></pre>
<p>How do get access to this files in compilation.<br>
It would be really helpful if you can tell more about compilation flow using intermediate files. e.g. python3 output.py  // to generate kernel binary files directly from g++</p>
<p>Thanks<br>
Mahesh</p>
PyTorch algorithm optimization

mikeybydun1: <p>Hello,<br>
I have a PyTorch algorithm model (compiled into .pt file) that do torch.prod of tensor in shape (1000,400,400,144)<br>
The algorithm takes 10 seconds on strong nvidia GPU (for exmaple A100).<br>
I am trying to find way to make it run faster.<br>
For now, the only effective optimization was using BFloat16.<br>
You have any suggestion for other optimization?<br>
Thanks!</p>
Libtorch feature request/delay loading

chacha21: <p>I have the exact same question as <a href="https://discuss.pytorch.org/t/libtorch-dlls-delayed-loading/142386">https://discuss.pytorch.org/t/libtorch-dlls-delayed-loading/142386</a> : the symbols of c10.lib and c10_cuda.lib will prevent a <a href="https://learn.microsoft.com/en-us/cpp/build/reference/delayload-delay-load-import?view=msvc-170" rel="noopener nofollow ugc">delay load</a> of c10.dll and c10_cuda.dll.<br>
(we could discuss the reasons why I would like to achieve that, but that’s not the point).</p>
<p>Is the present forum the right place to ask for a fix (even doomed to a “wont fix” answer) ? I could not find a dedicated libtorch section on the pytorch github.</p>
Model input shape doesnt match

lukasnol: <p>Hello i am new to the topic of training a NN on my own so this might be a easy error, but i can’t fix it.</p>
<p>I have the following code</p>
<pre><code class="lang-auto">class ClassConditionedUned(nn.Module):
    def __init__(self, num_ela=8, class_emb_size=4):
        super().__init__()
        self.class_emb = nn.Sequential(
            nn.Linear(num_ela, 32),
            nn.ReLU(),
            nn.Linear(32, class_emb_size)
        )
        self.model = UNet1DModel(
            sample_size=512,
            in_channels=1+class_emb_size,
            out_channels=1,
            layers_per_block=1,  
            block_out_channels = (32, 32, 64),   
            down_block_types = ("DownBlock1DNoSkip", "DownBlock1D", "AttnDownBlock1D"),
            up_block_types = ("AttnUpBlock1D", "UpBlock1D", "UpBlock1DNoSkip"),   
        )
        
    def forward(self, x, t, ela_vec):
        bs, ch, h = x.shape
        class_cond = self.class_emb(ela_vec) # Map to embedding dimension
        class_cond = class_cond.view(bs, -1, 1).expand(-1, -1, h)
        net_input = torch.cat((x, class_cond), 1)
        print(net_input.shape)
        return self.model(net_input, t).sample

model = ClassConditionedUned()
x = torch.randn(1, 1, 512)
t = torch.randint(0, 1000, (1,))
ela_vec = torch.rand(1, 8)  # normalisierte ELA-Vektoren

with torch.no_grad():
    out = model(x, t, ela_vec)
</code></pre>
<p>and i get the error:</p>
<pre><code>out = model(x, t, ela_vec)
      ^^^^^^^^^^^^^^^^^^^^
</code></pre>
<p>RuntimeError: Given groups=1, weight of size [32, 5, 1], expected input[1, 21, 512] to have 5 channels, but got 21 channels instead</p>
<p>what am i doing wrong?</p>
<p>Thank you in advance!</p>

lukasnol: <p>These are the import if you want to test it yourself:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from diffusers import UNet1DModel

</code></pre>

lukasnol: <p>I fixed the error by changing in_channels</p>
<pre><code class="lang-auto">in_channels=1+class_emb_size+16,
</code></pre>
<p>according to<br>
<a href="https://github.com/huggingface/diffusers/issues/2967#issuecomment-1500800012" rel="noopener nofollow ugc">https://github.com/huggingface/diffusers/issues/2967#issuecomment-1500800012</a><br>
the timestep embedding gets concatenated on top of the 5 input channels.</p>
Memory usage tips for a custom implementation of Conv2d using unfolds

qq-me: <p>Convolution multiplies each patch of the image by kernel (<code>patch * kernel</code>), and I wanted to try different version of it, for example <code>(patch - kernel).abs()</code>, or weighted median filter, and so on.</p>
<p>So I made a version of convolution where those things can be implemented using unfolds and confirmed that it works the same as scipy.signal.convolve:</p>
<pre data-code-wrap="py"><code class="lang-py">input = torch.randn(3, 32, 32)
kernel = torch.randn(1, 5, 5)
patched = input.unfold(1, 5, 1).unfold(2, 5, 1) # patched.shape = 3, 28, 28, 5, 5
convolved = (patched * kernel).sum((-1, -2)) # convolved.shape = 3, 28, 28
</code></pre>
<p>And I made a convolutional layer:</p>
<pre data-code-wrap="py"><code class="lang-py">def conv2d_layer(input, kernel):
    ksize = kernel.shape[-2:]
    input = input.unsqueeze(-1).expand(*input.size(), out_channels).movedim(1, -1)
    # input.shape = (batch_size, H, W, out_channels, in_channels)
    patched = input.unfold(-4, ksize[0], 1).unfold(-4, ksize[1], 1)
    # patched.shape = (batch_size, 28, 28, out_channels, in_channels, K1, K2)
    return (patched * kernel).sum((-1, -2, -3)).movedim(-1, 1)
    # return.shape = (batch_size, out_channels, 28, 28)
</code></pre>
<p>I’ve tested, and output matches torch.nn.functional.conv2d. However, my function uses 300 times more VRAM (for example one layer from medium size CNN, pytorch one uses 0.1 GB, while mine tries to allocate 32GB and fails).</p>
<p>Also, in terms of performance, it seems to be only slightly slower than PyTorch version on GPU, however for some reason it is extremely slow on CPU (like about a 1000 times slower than pytorch), but that isn’t an issue because I use GPU, only maybe it can help diagnose where the issue is.</p>
<p>I also tried to squish all of that into one line but that didn’t seem to help</p>
<pre data-code-wrap="py"><code class="lang-py">def conv2d_layer2(input, kernel):
    ksize = kernel.shape[-2:]
    return (input.unsqueeze(-1).expand(*input.size(), out_channels).movedim(1, -1).unfold(-4, ksize[0], 1).unfold(-4, ksize[1], 1) * kernel).sum((-1, -2, -3)).movedim(-1, 1)
</code></pre>
<p>Are there any ways I could make this require less memory? I know you can implement convolution with unfolds and matrix multiplication but then I won’t be able to do my custom versions of convolution layer like <code>(patched - kernel).abs()</code> and other things like that`</p>
Appropriate Beginner Topics

Murdock022X: <p>Hi all! I’m graduating from undergrad in a few weeks and have been eyeing contributing to PyTorch for awhile. Through work I’ve gained exposure to a pretty decent chunk of the Python frontend including distributed as well as writing a few custom operators in C++/CUDA (haven’t dived into Triton yet). I have a pretty good background regarding computer performance and the low level details in general and was wondering if anybody could point me towards issues or features that might be appropriate for my skill set. Happy to provide any other information as requested!</p>

ptrblck: <p>The <a href="https://github.com/pytorch/pytorch/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22">“good first issue”</a> category could be a good starter to pick some tasks.<br>
Let me know if you can find something interesting.</p>

Murdock022X: <p>Found a few that I think would be of interest for me, I’m going to hold off on grabbing for now as I have outside stuff to finish up over the next few weeks. <a href="https://github.com/pytorch/pytorch/issues/142306" rel="noopener nofollow ugc">Param spec typing</a>, or <a href="https://github.com/pytorch/pytorch/issues/135521" rel="noopener nofollow ugc">Reduce torch.compile default logs</a> all sound great, especially as a way to get a good feel for the codebase. I appreciate the kind reply, looking forward to contributing!</p>

ptrblck: <p>That sounds great! Let me know how it goes once you find the time.</p>
Behavior of Autocontrast on Normalized Images and Default Value for `contrast_factor`

Fatemeh_A: <p>If we apply <a href="https://pytorch.org/vision/main/_modules/torchvision/transforms/functional.html#autocontrast" rel="noopener nofollow ugc">autocontrast</a> or <a href="https://pytorch.org/vision/main/_modules/torchvision/transforms/transforms.html#RandomAutocontrast" rel="noopener nofollow ugc">RandomAutocontrast</a> on an image with pixel values between 0 and 1, will the resulting image still have pixel values within the range [0,1]? or Should the Image Be Rescaled Again?</p>
<p>What is the default value for <code>contrast_factor</code> in this context? And What is the exact mathematical formula used by the <code>autocontrast</code> function? Is it something like <code>(pixel - min) / (2 * (max - min)) - 1</code>, or is there another formula used?"</p>
<p>Thanks very much!</p>
GDB debug "torch::add" op, why breakpoint 'at::native::Add' not hit

huangrt01: <p>Steps:</p>
<ol>
<li>I use Pytorch’s DEBUG build</li>
<li>I debug this python program in clion IDE, and breakpoint at ‘b = torch.rand([10,20])’</li>
</ol>
<pre><code class="lang-auto">import torch

if __name__ == "__main__":
  a = torch.rand([10,20])
  b = torch.rand([10,20])
  c = a + b
  print(c) 
</code></pre>
<ol start="3">
<li>I attach GDB to this program</li>
<li>in gdb, <code>b at::native::Add</code></li>
<li>continue gdb debugger</li>
<li>continue python debugger</li>
</ol>
<p>but at::native::Add cannot be hit, I want to know why and how to solve it (letting me debug the actual cpu kernel)</p>
<p>only at::Add can be hit, but after hitting it, it will dispatch the op, and I cannot step into some op impl like at::native::Add</p>
Uninstalling torch package

mahmoodn: <p>Hi,<br>
Any idea on why I can not remove the torch package via conda?</p>
<pre><code class="lang-auto">[mn@login02 llama2-70b]$ conda activate pt26-cuda126-py312-gcc13
(pt26-cuda126-py312-gcc13) [mn@login02 llama2-70b]$ conda list | grep torch
torch                     2.6.0+cu126              pypi_0    pypi
torchaudio                2.6.0+cu126              pypi_0    pypi
torchvision               0.21.0+cu126             pypi_0    pypi
(pt26-cuda126-py312-gcc13) [mn@login02 llama2-70b]$ conda remove torch
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are missing from the target environment:
  - torch

</code></pre>

ptrblck: <p><code>pip uninstall torch -y</code> should work.</p>
Feedback on a New Library to Replace Long, Repetitive Training Loops in PyTorch

amirhosseinghanipour: <p>Hi everyone,</p>
<p>I’m excited to share that I’m working on a new library to simplify PyTorch model training and validation by eliminating the need for long, repetitive training loops. Tools like Lightning and Ignite are awesome, but they can sometimes feel too opinionated, overly complex, or not flexible enough for certain use cases. What’s your take? My goal is to create something that’s simple yet customizable, without unnecessary abstractions. I’d love to hear your thoughts. What frustrates you about current approaches? Are there features you’ve always wanted in a training library? Let me know what you think!</p>
<p>Thanks!</p>
How to test the implementation of NAS search spaces

Introvert_Loner: <p>I am trying to implement my own search space (mixture of Conv-based like ResNet, UNet combined with Transformer and RNN-based like the one right here<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/0803365007d23f1103c8391fbdbebb4fa9ccdb6d.png" data-download-href="https://discuss.pytorch.org/uploads/default/0803365007d23f1103c8391fbdbebb4fa9ccdb6d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/0803365007d23f1103c8391fbdbebb4fa9ccdb6d_2_690x276.png" alt="image" data-base62-sha1="18SHvEfHBXikDFhhUDh8skehiX3" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/0803365007d23f1103c8391fbdbebb4fa9ccdb6d_2_690x276.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/0803365007d23f1103c8391fbdbebb4fa9ccdb6d_2_1035x414.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/8/0803365007d23f1103c8391fbdbebb4fa9ccdb6d.png 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1164×466 26.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
). Say, every is integrated within class <code>Model</code> with hyperparameters-to-be-inputted in the <code>__init__(self, n_layers, n_neurons, rnn_type,....)</code><br>
But the thing is that I don’t really know if I might mis-typed some part (such as * made into + or the skip connections are misplaced). In case the errors don’t pop up, how can I best test the implemented code?</p>
Exporting pth file to onnx with torch

PedroC: <p>Hi,</p>
<p>I’m trying to export a .pth model created by detectron2 into a onnx model. After some research I ended up with the following code. In fact it exports an onnx model, but the outputs are weird. When I trained, I put a maximum of 100 detections, which are coincident with the outputs (boxes, scores and classes).<br>
I noticed that the output of demo.py (with pth file) of detectron gives better results with scores above 90% but when I try to inference with the onnx model it gives me hardly 50% with all the 100 boxes having a similar value, even if they are placed in different places.</p>
<p>The code below works/exports, but I’m concerned that some parameters are being discarded and the onnx file is not the ideal one.</p>
<p>Is there anything wrong with this code?</p>
<p>`cfg = get_cfg()<br>
cfg.merge_from_file(model_zoo.get_config_file(‘COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml’))</p>
<p>cfg.OUTPUT_DIR=“onnx_export”<br>
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, “model_final.pth”)<br>
cfg.MODEL.ROI_HEADS.NUM_CLASSES=2</p>
<p>model=build_model(cfg)<br>
model.eval()</p>
<p>checkpoint=torch.load(“onnx_export/model_final.pth”)<br>
model.load_state_dict(checkpoint[“model”])<br>
in_features = model.roi_heads.box_predictor.cls_score.in_features</p>
<p>box2box_transform = Box2BoxTransform(<br>
weights=(10.0, 10.0, 5.0, 5.0),  # Standard weights for bounding box regression<br>
scale_clamp=math.log(1000.0)  # Standard clamp value to limit scaling<br>
)<br>
model.roi_heads.box_predictor = FastRCNNOutputLayers(model.roi_heads.box_head.output_shape, num_classes=2, box2box_transform=box2box_transform)</p>
<p>cpu_device = torch.device(‘cpu’)<br>
x = torch.randn(3, 768, 1344)<br>
model.to(cpu_device)</p>
<p>inputs = [{“image”: x}]<br>
print(inputs)<br>
model.to(cpu_device)<br>
traceable_model = TracingAdapter(model, inputs)</p>
<p>torch.onnx.export(<br>
traceable_model, (inputs[0][“image”],),<br>
“model.onnx”,<br>
opset_version=17,<br>
do_constant_folding=True,<br>
verbose=True,<br>
input_names=[“input”],<br>
output_names=[“output”]<br>
)`</p>

PedroC: <p>Hi there,</p>
<p>It has been a while since I created the topic, can anyone help me?</p>
<p>Thanks!</p>

Divya932: <p>I’ve got the same issue while trying to export Mask2Former trained with Detectron2 and saved as a .pth file.<br>
Accuracy or the performance does not even closely match across ONNX and PyTorch when converted using torch.onnx.export</p>
<p>Are we supposed to convert using Detectron2’s export_onnx.py script?<br>
If yes, then does it take .pth models as input? I can see options of conversion with tracing and scripting but for caffe and torchscript models.</p>
Extracting data from cudnnBackendDescriptor_t

chanheenam: <p>Hello!<br>
I’m recently studying about cudnn. To be more specific, I’m trying to extract data from (cudnnBackendDescriptor_t variantPack and cudnnBackendDescriptor_t executionPlan). However,  I extracted doubtful data from it. Below is my code and result.</p>
<p>The code<br>
void printVariantPackPointers(cudnnBackendDescriptor_t variantPack, cudnnBackendDescriptor_t executionPlan) {<br>
void* ptrValue = nullptr;<br>
int64_t count1 = 0;<br>
int64_t count2 = 0;<br>
int64_t requested_count = 10;<br>
int64_t workspaceSize = 32;</p>
<pre><code>cudnnStatus_t status = cudnnBackendGetAttribute(
    executionPlan,
    CUDNN_ATTR_EXECUTION_PLAN_WORKSPACE_SIZE,
    CUDNN_TYPE_INT64,
    1,
    &amp;count1,
    &amp;workspaceSize
);
if (status == CUDNN_STATUS_SUCCESS) {
    std::cout &lt;&lt; "Workspace Size: " &lt;&lt; workspaceSize &lt;&lt; " bytes" &lt;&lt; std::endl;
} else {
    std::cout &lt;&lt; "Failed to get workspace size: " &lt;&lt; cudnnGetErrorString(status) &lt;&lt; std::endl;
}

status = cudnnBackendGetAttribute(
    variantPack,
    CUDNN_ATTR_VARIANT_PACK_DATA_POINTERS,
    CUDNN_TYPE_VOID_PTR,
    requested_count,
    &amp;count2,
    ptrValue
);
if (status == CUDNN_STATUS_SUCCESS) {
    std::cout &lt;&lt; "Attribute (VOID_PTR): " &lt;&lt; ptrValue &lt;&lt; std::endl;
} else {
    std::cout &lt;&lt; "Failed to get attribute: " &lt;&lt; cudnnGetErrorString(status) &lt;&lt; std::endl;
}
</code></pre>
<p>}</p>
<p>The result<br>
Workspace Size: 0 bytes<br>
Failed to get attribute: CUDNN_STATUS_NOT_SUPPORTED</p>
<p>I’m pretty sure that the execution plan is from pytorch cudnn run_conv_plan function. Is there any chance that conv plan actually does not have any workspace?<br>
I’m also curious that why CUDNN_STATUS_NOT_SUPPORTED error happens. I will appreciate if someone help me here</p>
<p>Thank you in advance!</p>
Apply linalg.solve if matrix is not singular, return arbitrary vector otherwise

Stingpie_52: <p>I’m working on a way to reverse my neural network given some output, and to do that I’m using T.linalg.solve. I have a pre-bias (Bv) and a post-bias (Bw), so the output of any layer is:<br>
<code>W = (V-Bv)M + Bw</code><br>
The thing is, M can be completely zeros in some cases, and in that case I want to return Bv instead of the solution of M. The best solution I’ve come up with is:<br>
<code>V = T.where( is_matrix_singular(M), Bv, T.linalg.solve(M,W))</code><br>
The issue is that solve throws an error even if the result is never used. All of my other options suck, so I’m hoping someone knows more about this.<br>
The first sucky solution is to loop through each batch, and then conditionally call linalg.solve.<br>
The second sucky solution is to use T.where to replace the zero matrices with identity matrices, subtract W, and then add Bv if the matrix is singular.<br>
The third sucky solution is to concatenate Bv to each matrix, and concatenate a one to the end of each W, then solve that.</p>
<p>Any better ideas?</p>

KFrank: <p>Hi Stingpie!</p>
<aside class="quote no-group" data-username="Stingpie_52" data-post="1" data-topic="218914">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stingpie_52/48/62527_2.png" class="avatar"> Stingpie_52:</div>
<blockquote>
<p>The best solution I’ve come up with is:<br>
<code>V = T.where( is_matrix_singular(M), Bv, T.linalg.solve(M,W))</code><br>
The issue is that solve throws an error even if the result is never used.</p>
</blockquote>
</aside>
<p>Consider using <a href="https://pytorch.org/docs/2.6/generated/torch.linalg.lstsq.html" rel="noopener nofollow ugc">lstsq().</a>  It will give you a result for singular systems.  (Note, the documentation<br>
suggests that the cuda version might not handle singular systems.  If you need cuda, you<br>
might have to use <a href="https://pytorch.org/docs/2.6/generated/torch.linalg.pinv.html#torch.linalg.pinv" rel="noopener nofollow ugc">pinv()</a> – which I believe does work with cuda – and use the pseudoinverse<br>
to “solve” your singular system.)</p>
<p>Best.</p>
<p>K. Frank</p>
Is grad_fn for a non-differentiable function that function's inverse?

Geremia: <p>What is <code>grad_fn</code> for a non-differentiable function like <code>grad_fn=&lt;SliceBackward0&gt;</code>, <code>grad_fn=&lt;ViewBackward0&gt;</code>, etc.? Is it simply the function’s inverse operation?</p>
<p>Where in the source code can I see the implementation of <code>SliceBackward0</code>, <code>ViewBackward0</code>, etc.? I assume it’s in their <code>backward()</code> static methods somewhere.</p>

soulitzer: <p>Usually how they are implemented is materialize a full zeros tensor, replaying the view, and in-place copying the gradient output into that view. This effectively scatters the gradients into where they need to be.</p>
<p>You can find some impls in torch/csrc/autograd/FunctionsManual.cpp.</p>
Does tensor.register_post_accumulate_grad_hook() always fire once, or multiple times?

apaz-cli: <p>I’m looking at <a href="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/_tensor.py#L683-L686" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/_tensor.py at v2.6.0 · pytorch/pytorch · GitHub</a></p>
<p>Is this a guarantee that the hook will only be called once per parameter per backward pass? Even if I’m training something like an RNN where the grad is going to be accumulated multiple times? Or is that not what it’s saying? I’m not able to understand it from the docs or blog post.</p>
<p>My specific use case is an optimizer that calls register_post_accumulate_grad_hook on the parameters you pass into it. I’m using fsdp2’s CPUOffloadPolicy, so they’re already on the CPU. Then I get the tensor data, do the optimizer step in C++,</p>
<p>This is all implemented here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/PrimeIntellect-ai/CPUOptimizer/blob/main/tests/example.py#L28-L29">
  <header class="source">

      <a href="https://github.com/PrimeIntellect-ai/CPUOptimizer/blob/main/tests/example.py#L28-L29" target="_blank" rel="noopener nofollow ugc">github.com/PrimeIntellect-ai/CPUOptimizer</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/PrimeIntellect-ai/CPUOptimizer/blob/main/tests/example.py#L28-L29" target="_blank" rel="noopener nofollow ugc">tests/example.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/PrimeIntellect-ai/CPUOptimizer/blob/main/tests/example.py#L28-L29" rel="noopener nofollow ugc"><code>main</code></a>
</div>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="28" style="counter-reset: li-counter 27 ;">
          <li>def pipeline_hook(param):</li>
          <li>    optimizer.step_param(param)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Emperically it works, but optimizers are probably tolerant to this sort of thing, and I’d like to gain a better understanding of what the autograd engine does before it fires the hooks. If it’s “Ah, I have no more references to this tensor in my graph, let’s fire the hook” or if it’s more naive.</p>

soulitzer: <p>(Already answered elsewhere, but for posterity)<br>
Yes, the post accumulate grad hook should only fire once if a single backward call is made. If a tensor is used multiple times during forward, accumulation happens to a temporary buffer before it is applied to the .grad field, which may involve another accumulation if the .grad field is already populated.</p>