akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
UBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedEx" During Grounding DINO Training

username: sameerafzal
cooked: <p>I am encountering an issue while using the NVIDIA TAO Toolkit version 5.5. I followed the instructions in the Grounding DINO sample notebook available at the official NVIDIA</p>
<p>GitHub repository: <a href="https://github.com/NVIDIA/tao_tutorials/blob/main/notebooks/tao_launcher_starter_kit/grounding_dino/finetune_grounding_dino.ipynb" rel="noopener nofollow ugc">Grounding DINO Notebook</a></p>
<p>Steps to Reproduce:</p>
<ol>
<li>
<p>Installed NVIDIA TAO Toolkit 5.5.</p>
</li>
<li>
<p>Configured the train.yaml file for training Grounding DINO.</p>
</li>
<li>
<p>Ran the following training command:<br>
!tao model grounding_dino train<br>
-e $SPECS_DIR/train.yaml<br>
train.num_gpus=$NUM_TRAIN_GPUS<br>
results_dir=$RESULTS_DIR</p>
</li>
<li>
<p>During training, the following error was raised:<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedEx(…)</p>
</li>
</ol>
<p>System Details:</p>
<ul>
<li>GPU: NVIDIA GeForce GTX TITAN X</li>
<li>Driver Version: 555.42.06</li>
<li>CUDA Version: 12.5 (nvidia-smi)</li>
<li>CUDA Version: 12.4 (miniconda-environment)</li>
<li>PyTorch Version: 2.5.1+cu124 (miniconda-environment)</li>
</ul>
<p>Additional Notes:</p>
<ol>
<li>
<pre><code>Using Automatic Mixed Precision (AMP) with bfloat16.
</code></pre>
</li>
<li>
<pre><code>Reducing the batch size and disabling AMP partially mitigated the issue but training still fails.
</code></pre>
</li>
<li>
<pre><code>Input size and dataset configurations follow the default settings in the sample notebook.
</code></pre>
</li>
</ol>
<p>I suspect the issue may be related to:</p>
<ul>
<li>
<pre><code>Incompatibility of bfloat16 precision with my GPU.
</code></pre>
</li>
<li>
<pre><code>A mismatch in CUDA/PyTorch versions.
</code></pre>
</li>
</ul>
<p>Any guidance or suggestions would be greatly appreciated!</p>

username: ptrblck
cooked: <p>Could you isolate which call fails and post a minimal and executable code snippet reproducing the error, please?</p>
Accessing secrets variables preset in the repository in CI

username: swgu98
cooked: <p>I’ve recently been learning how to use github actions. I’ve set up secrets variables in my repository, and created a workflow triggered by <strong>pull_request</strong> by submitting a pull request from an external developer. At this point, the repository has a secrets variable set up and a workflow. When an external developer submits a pull request to the repository, this workflow is triggered, but the secrets variable cannot be accessed. Why?<br>
I’ve also observed that the</p>
<blockquote>
<p>pull.yaml</p>
</blockquote>
<p>workflow of pytorch is triggered by <strong>pull_request</strong>, but can access the <em>HUGGING_FACE_HUB_TOKEN</em> variable, which should be a secrets variable pre-set in the repository. In my repository, only <strong>pull_request_target</strong> is set to trigger the access to the secrets variable.</p>
torch.OutOfMemoryError Needing Help

username: meditrust
cooked: <p>Hello everyone,</p>
<p>I need some help.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/7177b82c5c3fa982bd5bcb76e82413cb1a9895ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/7177b82c5c3fa982bd5bcb76e82413cb1a9895ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/1/7177b82c5c3fa982bd5bcb76e82413cb1a9895ee.png" alt="image" data-base62-sha1="gbMs0UtNbe79V8auwBTPnPmx4vA" width="278" height="168"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">278×168 16.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
nvidia geforce 4060<br>
64 GB RAM</p>
<p>I have nothing open on my computer.</p>
<p>I installed comfyiu and fluxgym  and try to make lora on my own computer.</p>
<p>[2025-01-14 16:57:43] [INFO] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 7.23 GiB is allocated by PyTorch, and 14.36 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (<a href="https://pytorch.org/docs/stable/notes/cuda.html#environment-variables" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.5 documentation</a>)</p>
<p>When trying to launch one, i got this error.</p>
<p>I have no idea what to do.</p>
<p>Thx a lot</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="meditrust" data-post="1" data-topic="215383">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f1d935/48.png" class="avatar"> meditrust:</div>
<blockquote>
<p>Tried to allocate 80.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free.</p>
</blockquote>
</aside>
<p>Your GPU is running out of memory, so reduce the memory usage e.g. by decreasing the batch size or by using a smaller model.</p>
Multithreading in dataloader workers

username: Tristan_Brugere
cooked: <p>I’m doing training on data where the <code>collate()</code> function needs relatively heavy computation (some sequence packing).</p>
<p>Right now I am training with around 40 dataloader workers, but still experiencing locks as the main thread waits for data.</p>
<p>I noticed the workers each call <code>torch.set_num_threads(1)</code>. Is there a reason for that (apart from limiting the number of threads). Is it ok to raise the number of threads each worker can use? (by eg calling <code>torch.set_num_threads</code> in the <code>worker_init_fn</code>)?</p>
Best Practices: Using a buffer for the latent space?

username: jsully83
cooked: <p>Hi Everyone!<br>
I’m learning polar coordinates so I have parameters r and theta.  In the latent space I use r and theta to update a dictionary which requires some computation on r and theta with 4 other static tensors I create when the model is instantiated . Then I use the dictionary during the forward pass.   I want to keep the dictionary update on the GPU.  The 4 static tensors have no use outside of updating the dictionary and don’t need to persist in state_dict.</p>
<p>My question is what is the best practice in terms of registering the dictionary and the other tensors used to create it as buffers so i can use model.to(device) as opposed to .to(device) on every tensor to keep computation on the GPU?</p>
<p>Is it better to update a buffer or register_buffer with the new dictionary on every epoch?</p>
<p>Does registering buffers or constantly updating them have a significant effect on overall performance?</p>
<p>Thanks!</p>
Torch.autograd.grad and masking issue

username: robofar123
cooked: <p>Hello I have following issue, which I cannot solve, but I understand main idea behind issue.<br>
So, I have following tensors:<br>
<code>point: shape(N,3), require_grad=True</code><br>
<code>output: shape(N,) require_grad=True</code><br>
<code>mask: shape(N,) require_grad=False, dtype=bool</code></p>
<p>Output is function of point.</p>
<p>This works:<br>
<code>grad = torch.autograd.grad(output, point)</code></p>
<p>This does not work:<br>
<code>grad = torch.autograd.grad(output[mask], point[mask])</code></p>
<p>So it seems this masking breaks computation graph - i.e. <code>output[mask]</code> and <code>point[mask]</code> are not connected in computation graph.So I have following questions:</p>
<ol>
<li>Does masking creates copy of tensor, which does not have any connection to original tensor? And how is masking related to computation graph</li>
<li>How can I achieve this code works with masking? I am trying but not able to get it</li>
</ol>
<p>Thank you!</p>
A question about _RequiredParameter()

username: ToxicDoubleH
cooked: <p>I have read torch.optim.optimizer this night.<br>
here is the address<br>
<a href="https://pytorch.org/docs/master/_modules/torch/optim/optimizer.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/_modules/torch/optim/optimizer.html</a></p>
<pre><code class="lang-auto">class _RequiredParameter(object):
    """Singleton class representing a required parameter for an Optimizer."""
    def __repr__(self):
        return "&lt;required parameter&gt;"

required = _RequiredParameter()
</code></pre>
<p>I don’t know how this function work in the add_param_group</p>
<pre><code class="lang-auto">for name, default in self.defaults.items():
            if default is required and name not in param_group:
                raise ValueError("parameter group didn't specify a value of required optimization parameter " +
                                 name)
            else:
                param_group.setdefault(name, default)
</code></pre>
<p>Can somebody a nice guy tell me how the function _RequiredParameter works and if I don’t write deault lr in param_group, name is not in is True, if there is no error, default is required should be False, I am confused</p>

username: ptrblck
cooked: <p>This class was introduces in <a href="https://github.com/pytorch/pytorch/pull/13202">this PR</a> and based on its description it seems to improve the docs slightly. However, I don’t know if it’s still used or dead code by now.</p>

username: ToxicDoubleH
cooked: <p>Thanks for your respond. It’s still used, as you can see, it’s used in "def add_param_group() ", and as a judge part of a sentence. I think defaults is such as lr=0.3, so the name=‘lr’ and default=0.3.<br>
It is so hard for me to understand why default equal required(_RequiredParameter()) can work.<br>
Maybe I should give it up. Just remember, don’t delve into it</p>

username: addisonklinke
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> FYI this PR causes some <a href="https://github.com/mit-ll-responsible-ai/hydra-zen/issues/257#issuecomment-1287375024" rel="noopener nofollow ugc">issues</a> with Hydra configs for <code>torch.optim.SGD</code></p>

username: dmpiergiacomo
cooked: <pre><code class="lang-auto">_RequiredParameter
</code></pre>
<p>Is no longer needed. I documented everything in <a href="https://github.com/pytorch/pytorch/pull/144771" rel="noopener nofollow ugc">this PR</a>.</p>
Clarification on NVIDIA License Compliance for Using PyTorch (torch)

username: Garvit_Mathur
cooked: <p>Hi PyTorch Team,</p>
<p>In our organization, we are trying to install PyTorch (<code>torch</code>) via PyPI. However, our compliance tools are flagging a potential vulnerability due to the NVIDIA license included with the package. We understand that PyTorch is distributed under the BSD-3 License, but the NVIDIA license raises concerns regarding whether compliance with it is also mandatory when using <code>torch</code>.</p>
<p>We cannot use <code>torch-cpu</code> due to our organizational policy of only utilizing packages available on PyPI. Could you please clarify:</p>
<ol>
<li>Does installing and using PyTorch (<code>torch</code>) via PyPI necessitate compliance with the NVIDIA license?</li>
<li>If so, could you provide guidance on how this aligns with PyTorch’s BSD-3 License?</li>
</ol>
<p>I appreciate your support, and I look forward to your clarification!</p>
<p>Best regards,<br>
Garvit</p>
How to (efficiently) apply a function without a "dim" argument to each row of a 2D tensor?

username: Matt_T1
cooked: <p>Hello everyone.</p>
<p>Long story short, I have a 2D matrix of ones and zeros and I need to retrieve, for each row, the indexes of the elements set to one. The “standard” way to do so would be torch.nonzero, but that function is well known for being 1) a real bottleneck, since it does not know in advance the size of the final vector, and 2) it cannot be applied to each row of a 2D tensor in one shot since different rows may have different amounts of ones.</p>
<p>Recently, at::nonzero_static has been introduced, which solves the first point by giving the function the maximum number of nonzero elements (which is fine for my application). However, it does not feature a “dim” argument, meaning that it cannot be applied to each row/column individually, which in my opinion makes no sense since setting the size of the output guarantees that each row would feature the same amount of items, thus making the output a tensor.</p>
<p>Using a for loop would obviously solve my issue, but that would mean calling the function several times which is not GPU efficient. Does anyone know a way to apply nonzero_static efficiently to each row, and returning a tensor where each row is the result of its application to each slice of the tensor?</p>
<p>I will appreciate any help you can provide.<br>
Regards</p>
Object detection: torchmetrics mAP calculator question

username: matiasV-TYN
cooked: <p>Hi,<br>
I am using the torchmetrics mAP calculator for object detection.<br>
Documentation: <a href="https://lightning.ai/docs/torchmetrics/stable/detection/mean_average_precision.html" rel="noopener nofollow ugc">Mean-Average-Precision (mAP) — PyTorch-Metrics 1.6.1 documentation</a></p>
<p>My question is the following:<br>
Lets say I have 20 classes. I know these are required to be 0-indexed. I need a class for background (for images were no objects detected). Should my background class be included? So my background class would be index 0, last class would be index 20.<br>
When model doesn’t detect any classes in a given image, should the predictions dictionary contain a background prediction (label 0, score 0, bbox [0, 0, 0, 0])? Or should it just be empty?<br>
I’ve noticed that if I add a background class and enable per class metrics, I get mAP results for the background class too of course. Obviously the mAP for that class is -1 since it is all wrong detections, but is this correct?<br>
I have read the documentation but cant seem to find this. Maybe its a common knowledge thing so it is just taken for granted.</p>
<p>Thanks.</p>
Understanding Encoder-Decoder Transformer Architecture in Image Captioning

username: sriramgs
cooked: <p>Hello all,</p>
<p>I am a beginner in using transformer architecture and am trying to implement image captioning model. For this, I am using Encoder-Decoder architecture using transformer architecture. I am using 6 heads in both encoder and decoder blocks. I have a doubt in this regard.</p>
<p>Please tell whether my understanding is correct or not: "We need to use the encoder’s self-attention output from the 6th head for processing in decoder’s cross-attention? So that the same key, value will be used 6 times with different query (query generated by decoder’s self-attention) in decoder’s cross-attention.</p>
Need Help Troubleshooting Stack-GAN Implementation

username: Srijan_Kundu
cooked: <p>I am trying different GAN models, and recently implemented the original stack-GAN model. I have followed the paper and the official codebase.<br>
The stage 1 gan does not work and keeps producing random noise, whereas the stage 2 gan works fine. I need help to fix the stage 1 gan, so I need someone willing to go through my codebase.<br>
Please comment. Thank you in advance.</p>
Xcode got error "Error loading model: Following ops cannot be found: [aten::scaled_dot_product_attention]. Please check if the operator library is included in the build. If built with selected ops, check if these ops are in the list."

username: Danny_Vu
cooked: <p>I’m trying to run <a href="https://github.com/pytorch/ios-demo-app/tree/master/SpeechRecognition" class="inline-onebox" rel="noopener nofollow ugc">ios-demo-app/SpeechRecognition at master · pytorch/ios-demo-app · GitHub</a> and got errors in Xcode, my device: iphone 15 pro, iOS 18, Xcode 16.1,  LibTorch-Lite 1.13.0.1. → “Error loading model: Following ops cannot be found: [aten::scaled_dot_product_attention]. Please check if the operator library is included in the build. If built with selected ops, check if these ops are in the list. If you are a Meta employee, please see <a href="http://fburl.com/missing_ops" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for a fix.” please help me!!</p>

username: ptrblck
cooked: <p><code>scaled_dot_product_attention</code> was added in <code>torch==2.0.0</code> so you would need to update your PyTorch version.</p>

username: tarun292
cooked: <p><a class="mention" href="/u/danny_vu">@Danny_Vu</a> the code you’re looking at is the legacy solution offered by PyTorch to run models on edge devices using TorchScript. We have a newer framework called ExecuTorch that lets you run models based on phones. We support various transformer based models such as LLM’s. Take a look at <a href="https://pytorch.org/executorch/stable/index.html" class="inline-onebox" rel="noopener nofollow ugc">Welcome to the ExecuTorch Documentation — ExecuTorch 0.4 documentation</a></p>
<p>Would recommend using this as you’ll get the best support for this currently and in the future. Let us know if you have any questions.</p>

username: Danny_Vu
cooked: <p>The error gone when I pip torch==2.0.0 and torchaudio==2.0.0. Thank you very much Mr Ptrblck.</p>

username: Danny_Vu
cooked: <p>Thanks Mr Tarun Karuturi</p>
Questions about TorchVision MobileNet V2 GFLOPS in Pytorch website

username: Wenlong_Shi
cooked: <p>I recently checked the PyTorch official website(<a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.18 documentation</a>)  and noticed that the recorded GFLOPS for MobileNet V2 in torchvision is 0.3. However, according to the original MobileNet V2 paper, the reported MAdds is 300M (million multiply-add operations). As I understand it, FLOPS should typically be twice the number of MAdds, which would mean it should be 0.6GFLOPS.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc24f626fbfc292077c2028fb5b22f713ee3d9d2.png" alt="c68200ad71707adbdda9945cf6e819e" data-base62-sha1="qQoX9NAiKVWnoF8xQ2pdk38cQ7g" width="320" height="281"></p>
<p>Could anyone explain how the 0.3 GFLOPS is calculated? Is there a specific interpretation or context that relates this value to the MAdds reported in the original paper? I would appreciate any insights or clarifications on this discrepancy.</p>

username: Ayanokoji
cooked: <p>I think PyTorch official website reported GMAC</p>
Pytorch not compatible with sm_86 CUDA Capability

username: the-ml-bulldog
cooked: <p>I was training my model when I ran into this error:</p>
<pre><code class="lang-auto">/home/user/miniconda3/envs/wei-pip/lib/python3.8/site-packages/torch/cuda/__init__.py:143: UserWarning: 
NVIDIA GeForce RTX 3070 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_35 sm_50 sm_60 sm_61 sm_70 sm_75 compute_50.
If you want to use the NVIDIA GeForce RTX 3070 Laptop GPU GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
</code></pre>
<p>I checked online and saw <a href="https://discuss.pytorch.org/t/need-help-trouble-with-cuda-capability-sm-86/120235/3">this</a> post. I tried the fix suggested by <a href="https://discuss.pytorch.org/u/Ilias_Giannakopoulos">Ilias_Giannakopoulos</a> but it did not work. Any other ideas what I can do?</p>

username: ptrblck
cooked: <p>All of our current binaries support sm_86 and based on the error message it seems you might have installed an old PyTorch binary with CUDA&lt;=10.2 support.</p>

username: the-ml-bulldog
cooked: <p>I installed the latest version of Pytorch though (CUDA 12.4).</p>

username: ptrblck
cooked: <p>The latest release works fine, ships with compute capability 8.6, and supports the 30XX series:</p>
<pre data-code-wrap="python"><code class="lang-python">python -c "import torch; print(torch.__version__); print(torch.cuda.get_arch_list()); print(torch.cuda.get_device_properties(0)); print(torch.randn(1).cuda())"
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">2.5.1+cu124
['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
_CudaDeviceProperties(name='NVIDIA GeForce RTX 3090', major=8, minor=6, total_memory=24249MB, multi_processor_count=82, uuid=7d7b1de4-b35c-b01c-8b9a-e3623cbfecb0, L2_cache_size=6MB)
tensor([-1.2947], device='cuda:0')
</code></pre>

username: the-ml-bulldog
cooked: <p>It turns out the conda environment I was using was using an older version of PyTorch. Upgrading the version fixed the issue.</p>
FSDP2 evaluate during training

username: XYG_123
cooked: <p>Hi team, I am using the <a href="https://github.com/pytorch/torchtitan/blob/main/docs/fsdp.md" rel="noopener nofollow ugc">FSDP2</a> to shard my model.<br>
What is the best practice to evaluate (like calling <code>eval_forward</code> for the module) during training here?</p>
<p>I know we can register_forward hooks for <code>eval_forward</code> but it might call some unused hooks.<br>
Can I make it back to non-FSDP module and shard again after I finish the evaluation?</p>

username: XYG_123
cooked: <p>I have to do some hacky way:</p>
<pre><code class="lang-auto">def hack_unwrap_the_fsdp(model):
    fs_storage_writer = FileSystemWriter(SAVED_PATH)
    save(
        state_dict=model.state_dict(),
        storage_writer=fs_storage_writer,
    )
    unwrapped_model = ToyModel().cuda()
    fs_reader = FileSystemReader(SAVED_PATH)
    state_dict = unwrapped_model.state_dict()
    load(state_dict, storage_reader=fs_reader)
    unwrapped_model.load_state_dict(state_dict) 
    return unwrapped_model
</code></pre>
<p>And <code>unwrapped_model</code> is not sharded. Not sure if there is better way.</p>
Educational Framework Inspired by PyTorch

username: joseortizm
cooked: <p>Hello everyone, my name is Jose Ortiz, and I’ve created an educational framework called <a href="https://github.com/joseortizm/edugrad" rel="noopener nofollow ugc">Edugrad</a>, inspired by PyTorch and other projects, aimed at making AI and machine learning concepts easier to learn. The project is still under development, and I’m adding new features every week. My goal is to provide an accessible tool for students, educators, and anyone interested in the field.</p>
<p>If you’d like to collaborate, test new features, or offer suggestions, feel free to check out the repository. All help is welcome!</p>
How to download legacy nightlies stored in the S3 bucket

username: akavalar
cooked: <p>Hi,</p>
<p>Is it possible to download legacy versions of nightlies from the <code>pytorch</code> S3 bucket, e.g. <code>nightly-archive/whl/nightly/cu118/torch-2.0.0.dev20221221%2Bcu118-cp310-cp310-linux_x86_64.whl</code>? I can see them but am not allowed to download them.</p>
<p>I’m debugging something on our end and would like to use the nightlies to try to bisect the issue.</p>
<p>Thanks!</p>
Hardware Asus rog xg 2025- compatiblity pytorch

username: bruno_menon
cooked: <p>hello team</p>
<p>Please<br>
is this device is compatible with pytorch for trainning</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://rog.asus.com/external-graphic-docks/rog-xg-mobile-2025/">
  <header class="source">
      <img src="https://rog.asus.com/rog/nuxtStatic/img/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://rog.asus.com/external-graphic-docks/rog-xg-mobile-2025/" target="_blank" rel="noopener nofollow ugc">@ROG</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://dlcdnwebimgs.asus.com/gain/F26E974A-523A-4B3B-8954-CD0BAD788558" class="thumbnail onebox-avatar">

<h3><a href="https://rog.asus.com/external-graphic-docks/rog-xg-mobile-2025/" target="_blank" rel="noopener nofollow ugc">ROG XG Mobile (2025) | Gaming external-graphic-docks｜ROG - Republic of...</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thanks<br>
bruno</p>
Do I need to use nn.Parameter() again when changing a tensor wrapped in nn.Parameter()?

username: DawidL
cooked: <p>I have the following code (removed some parts for clarity):</p>
<pre><code class="lang-auto">class Cell(nn.Module):
    def __init__(self, core_size, conn_size, conn_num):
        super().__init__()
        [...]
        core = torch.Tensor(core_size)
        self.core = nn.Parameter(core)
        nn.init.ones_(self.core)

    def forward(self, x):
        [...]
        self.core = torch.mul(self.core, x)
        [...]
</code></pre>
<p>Then I try to run it I get an error related to the operation in forward:</p>
<blockquote>
<p>cannot assign ‘torch.FloatTensor’ as parameter ‘core’ (torch.nn.Parameter or None expected)</p>
</blockquote>
<p>I’m trying to change the self.core parameter based on the input and keep it stored within the model. Do I need to use torch.nn.Parameter again when adjusting the self.core parameter (e.q. self.core =  torch.nn.Parameter(torch.mul(self.core, x)))? Would doing so reset  the grad of self.core?</p>

username: ptrblck
cooked: <p>The <code>torch.mul</code> operation is differentiable and is thus creating a non-leaf tensor. If you explicitly want to set this output as the new value of the trainable parameter <code>self.core</code>, instead of computing its gradient and updating it as is the common use case, you could do so in a <code>no_grad</code> context. Something like this should work:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, x):
    [...]
    out = torch.mul(self.core, x)
    with torch.no_grad():
        self.core.copy_(out)
    [...]
</code></pre>
<p>However, make sure this is really what you want since you are creating a trainable parameter but are not using its gradients and would be updating it manually.</p>

username: DawidL
cooked: <p>Thank you for your response! I do believe that is what I want. Just to check, am I correctly assuming that I will still be able to backprop through this to any operations before torch.mul(), but with the old value of self.core being used in gradient calculations?</p>
Intuition behind index_select

username: tuero
cooked: <p>I’m trying to write my own <code>index_select</code> to understand how this is done under the hood efficiently. Suppose we have access to <code>index(t, indices)</code> where passing a list of raw integer indices will return a flattened tensor indexing into the passed tensor <code>t</code>.</p>
<p>What I have is the following: create a mask of all indices (i.e. <code>full(t.shape)</code>), loop over each passed <code>idx</code> in <code>indices</code>, slice into the mask to get the corresponding indices, then add to a master list of all indices we will index:</p>
<pre><code class="lang-auto">t # tensor of shape (3, 4, 2, 3)
dim = 2 # The dim to index_select over
selected_indices = []
full_indices = full(t.shape)
for idx in indices:
    mask = [Slice() for _ in t.dim()]
    mask[dim] = idx   # i.e. [:, :, idx, :]
    for i in full_indices[mask].flatten()
         selected_indices.append(i)
</code></pre>
<p>We can then get the raw values by using this list of indices into the tensor: <code>result = t.index(selected_indices)</code></p>
<p>This gives us the correct underlying data, but its in the wrong order. We can correct this by performing a permute, where we rotate to the left by one the sub-shape [0, dim]: <code>result = result.permute(1, 2, 0, 3)</code>.</p>
<p>I’m trying to wrap my head around how a nice rotate to the left everything before and including the selected dim works here. I can sort see that it works on paper, but I’m lacking a nice intuitive understanding. Is there an easier, more intuitive algorithm instead of having a lot of overhead through concatenations, etc.?</p>
<p>Here is the working example, the output is large so I don’t include that here.</p>
<pre data-code-wrap="python"><code class="lang-python"># input tensor and dim
_t = torch.arange(3*4*2*3).reshape(3, 4, 2, 3)
dim = 2
print(_t)
print(_t.shape)
print("index_select expected result:")
_t = torch.index_select(_t, dim, torch.tensor([0, 1, 0]))
print(_t)
# My algorithm returns the result of this, here I just apply the inverse required permute from index_select so I dont have to type out my algorithm
print("Result from my algorithm:")
_t = torch.permute(_t, (2, 0, 1, 3))
print(_t)
print(_t.shape)
print("Recovered correct result by permutting (rotation to left):")
_t = torch.permute(_t, (1, 2, 0, 3))
print(_t)
</code></pre>
Documents as parametric memory

username: Sai1
cooked: <p>Hello everyone,<br>
I have a set of docs and I want to perform retrieval. So far, it’s not going great as I have to do unsupervised retrieval. I was working on a RAG model and kind of felt that is there a way I could incorporate all my docs as parametric memory like gpt without explicitly having a QnA dataset ? So I can ask a question about the docs and it can generate relevant answers.</p>
<p>Thanks</p>
Merging all tensor files into one large tensor file

username: vr25
cooked: <p>Hi,</p>
<p>I have 10 PyTorch tensors saved as 1.pt, 2.pt, …, 10.pt (each file is ~ 3GB). I want to create a training dataset from all of them.</p>
<ol>
<li>
<p>One way to do this (inefficient way though) is to create a list of tensors and then use torch.stack on the list to create a large tensor. I run into the CUDA memory issue because of the large file size.</p>
</li>
<li>
<p>Other way is to load each file into separate batches but I am not sure how to actually implement this.</p>
</li>
</ol>
<p>Thanks!</p>

username: g_k
cooked: <p>Hey, were you able to find a solution for this?</p>
Load_model from MLflow model registry can't find model class - how best to make this available?

username: minamesephy
cooked: <p>I am working to support some model developers by prototyping some functionality with MLflow model registry.</p>
<p>We successfully register versions of the model fine (it’s a Pytorch Lightning model), and I recently also added code_paths to the log_model call so that theoretically all of the original model code ends up registered / in the artifacts (I’m not sure if this is best practice though? But either way the model code is super complex and I’m not sure how I can extract out just the stuff needed by the serializer on the consuming side atm).</p>
<pre><code class="lang-auto">        mlflow.pytorch.log_model(
            pytorch_model=cli.model,
            artifact_path="model",
            registered_model_name="MyModelName",
            code_paths=["src"],
        )
</code></pre>
<p>In any case, I can see the model registered and the artifacts tree looks like this now when I view them from the UI:</p>
<pre><code class="lang-auto">model
|--checkpoints
|--code
    |--src
        |--&lt;mymodelname&gt;
             |-- a bunch of other modules and nested stuff under here
        |--a bunch of other stuff here
|--data
     |--model.pth
     |--pickle_module_info.txt (this is a small file with just the text "mlflow.pytorch.pickle_module" in it)
|--MLmodel  (small file w/ metadata about the model)
|--conda.yaml
|--python_env.yaml
|--requirements.txt
</code></pre>
<p>I have a totally separate package outside of the model package and I’m testing consuming the model as an “external” client who wants to use the registered pre-trained model for further fine-tuning or inference. I confirmed that the code is getting retrieved when I do something like this:</p>
<pre><code class="lang-auto">model_uri = f"models:/{model_name}/{model_version}"
local_path = mlflow.artifacts.download_artifacts(model_uri)
</code></pre>
<p>the retrieved artifacts there include everything I listed above.</p>
<p>However, when I try to load the model:</p>
<pre><code class="lang-auto">    loaded_model = mlflow.pytorch.load_model(model_uri)
</code></pre>
<p>It fails with the error: <code>ModuleNotFoundError: No module named 'src.&lt;mymodelname&gt;'</code></p>
<p>Am I going about this totally wrong or is there just something small I’m missing here (like getting the picker to recognize the original model code under code/ somehow since it’s all there?)?</p>
<p>Any help would be much appreciated!</p>
Can I rotate two dimensional images (e.g. MNIST) without having to unsqueeze each one?

username: DawidL
cooked: <p>I’m trying to create a dataset of rotated MNIST images as inputs and the angles by which each one was rotated as outputs. Is there a way to do this without unsqueezing and squeezing each one?</p>
<p>When I try to do the following:</p>
<pre><code class="lang-auto">test = transforms.functional.rotate(dataset1.data[0], 90)
</code></pre>
<p>I get an error:</p>
<blockquote>
<p>grid_sampler(): expected grid to have size 1 in last dimension, but got grid with sizes [1, 28, 28, 2]</p>
</blockquote>
<p>This can be fixed by unsqueezing and squeezing the image</p>
<pre><code class="lang-auto">test = torch.unsqueeze(dataset1.data[0], 0)
test = transforms.functional.rotate(test, 90)
test = torch.squeeze(test, 0)
</code></pre>
<p>however this seems quite unnecessary and inefficient. Is there a way to avoid this?</p>
<p>I’m also not sure if this type of iterative approach is correct when it comes to creating this type of dataset, but is seems to me that there is no easy way to apply random rotation to all images, while keeping all of the angles saved somewhere (which is where torchvision.transforms.RandomRotation fails for me).</p>

username: ptrblck
cooked: <p>It’ll work without unsqueezing the raw data if you iterate the tensors:</p>
<pre data-code-wrap="python"><code class="lang-python">dataset = datasets.MNIST(root=root, download=False, transform=transforms.ToTensor())
data, target = next(iter(dataset))

test = transforms.functional.rotate(data, 90)
</code></pre>
Call to backward function for linear layer

username: flauron
cooked: <p>Hello,</p>
<p>I am trying to create my own custom linear layer.<br>
In my custom backward function I perform some calculation and then I need to call the function used in the backward pass</p>
<p>when I use the pytorch implementation I am able to do this:</p>
<pre data-code-wrap="python"><code class="lang-python">device = "cuda"
x = torch.randn(16, 16, requires_grad=True, device=device)
t = torch.randn(16, 16, requires_grad=True, device=device)
m = torch.nn.Linear(16, 16, device=device)

o = m(x)
loss = MSELoss()
l = loss(o,t)

l.backward()

print(o.grad_fn)
#==&gt;&lt;AddmmBackward0 at 0x7ee7ebf5e8c0&gt;
</code></pre>
<p>From what I understand, the function used to compute the backward pass in a linear layer is called <strong>addmm_backward</strong> (or something close to that).</p>
<p>I perform the same kind of research for convolution layer and batch norm layer and I have been able to find that the function called in the backward are (respectively) <strong>convolution_backward</strong> and <strong>batch_norm_backward</strong> .<br>
They are listed inside the file native_functions.yaml and are accessible through python/c++ API.<br>
But there is no function called <strong>addmm_backward</strong> (or something close to that).</p>
<p>So , my question is there another functions listed in native_functions.yaml to perform the backward pass for a linear layer or another way to call the function <strong>addmm_backward</strong> ?</p>
<p>Thanks in advance for your answers.</p>
Creating one giant C++ code covering code from all contexts

username: ksj20
cooked: <p>Hi, is there a way to create a giant consolidated c++ code without each AOT context having their own code (starting with # AOT ID: )? This is for running on only CPU. My code is as below.</p>
<pre data-code-wrap="python"><code class="lang-python">        from torch._inductor.utils import run_and_get_cpp_code
        t = torch.compile(model, backend="inductor")
        result, code = run_and_get_cpp_code(t, input_data)
        logger.debug(f"result: {result}")
        logger.debug(f"code: {code}")
</code></pre>
<p>And some version information:</p>
<pre><code class="lang-auto">Python 3.9.21 (main, Dec 11 2024, 16:24:11) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.5.1+cu124'
</code></pre>
<p>If I am doing something wrong, please fix me. Any guidance would be appreciated.<br>
Thank you</p>
How do I run torch.distributed between Docker containers on separate instances using the bridge network?

username: jaydsolanki20
cooked: <p>I am trying to run a simple torch.distributed script between two Docker containers running on separate instances. Below is the code I am using:</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist

def init_distributed():
    os.environ['MASTER_ADDR'] = "10.12.27.241"
    os.environ['MASTER_PORT'] = '29500'
    node_rank = int(os.environ.get('RANK', 0)) # 1 for worker
    world_size = 2

    dist.init_process_group(
        backend='gloo',
        rank=node_rank,
        world_size=world_size
    )
    print(f"Initialized process group: rank {node_rank} of {world_size}")
    return node_rank, world_size

def send_receive_message(rank, world_size):
    if rank == 0:
        # Node 0 sends a message
        message = torch.tensor([42, 43, 44], dtype=torch.int64)
        dist.send(message, dst=1)
        print(f"Rank {rank} sent message: {message}")
    else:
        # Node 1 receives the message
        message = torch.zeros(3, dtype=torch.int64)
        dist.recv(message, src=0)
        print(f"Rank {rank} received message: {message}")

if __name__ == "__main__":
    rank, world_size = init_distributed()
    send_receive_message(rank, world_size)
    # Barrier to ensure all processes have completed
    dist.barrier()
    # Clean up
    dist.destroy_process_group()
</code></pre>
<p>I am able to run this script successfully when using the --network=host option for docker run. However, due to organizational restrictions, I am required to use the --network=bridge option. When I use --network=bridge, I encounter the following error:</p>
<pre><code class="lang-auto">[E110 05:59:45.095859745 ProcessGroupGloo.cpp:143] Gloo connectFullMesh failed with [../third_party/gloo/gloo/transport/tcp/pair.cc:144] no error
Traceback (most recent call last):
  File "/data/exp/com.py", line 36, in &lt;module&gt;
    rank, world_size = init_distributed()
  File "/data/exp/com.py", line 12, in init_distributed
    dist.init_process_group(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
    func_return = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1527, in init_process_group
    default_pg, _ = _new_process_group_helper(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1744, in _new_process_group_helper
    backend_class = ProcessGroupGloo(
RuntimeError: Gloo connectFullMesh failed with [../third_party/gloo/gloo/transport/tcp/pair.cc:144] no error
</code></pre>
<p>How can I configure torch.distributed to work with the bridge network when running containers on separate instances? What additional steps or configurations are required to make Gloo backend communication succeed in this setup?</p>
<p>Any guidance or pointers would be greatly appreciated!</p>
Deploying multiple models that invoke each other

username: A_J
cooked: <p>Had a paddle ocr model, converted it into pytorch model. Trying to deploy it so that I get high throughput and low latency.</p>
<p>Bottom line of what I have is ModelA → compute on CPU → ModelB. And I want to deploy it in the most optimal manner.</p>
<p>I know we can trace one model graph with aws neuron and “split” it so that its partitions run on different cores.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9d63e932edcf31861868f81964acf0013344d73.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b9d63e932edcf31861868f81964acf0013344d73" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9d63e932edcf31861868f81964acf0013344d73.jpeg" alt="image" data-base62-sha1="qvZlC4Bu7EAByUpDJtUrTmAqVrl" width="690" height="282" data-dominant-color="475559"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">700×287 45.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This got me thinking if I can hack the modelA, compute and modelB together into one graph, Does this sounds like a bad idea ?</p>
<p>Also if you had to deploy something like this how would you do it ? (just to constrain the problem lets assume we have to do all this in one container)</p>

username: A_J
cooked: <p>to be more specific here’s the code I am working with</p>
<pre><code class="lang-auto">        dt_boxes, elapse = self.text_detector(img)            # model 1
        img_crop_list = []                                    # compute
        for bno in range(len(dt_boxes)):                      # compute
            do something                                      # compute
            img_crop_list.append(something)                   # compute
        rec_res, elapse = self.text_recognizer(img_crop_list) # model 2
</code></pre>

username: natalia_meira
cooked: <p>Hello,</p>
<p>I noticed that you successfully converted the PaddleOCR model to PyTorch. I am currently working on this as well but am encountering some difficulties.</p>
<p>You can find my questions and details about the issue in this discussion: <a href="https://discuss.pytorch.org/t/convert-onnx-to-pytorch-typeerror-conv2d-init-missing-2-required-positional-arguments-in-channels-and-out-channels/215169">ISSUE LINK</a>.</p>
<p>Would you be able to assist me with this?</p>
<p>Thank you in advance for your help!</p>
How should we use Single GPU for validation while doing multigpu training using DDP

username: Mukil
cooked: <p>I am currently training the model using DDP but while entering the validation loop within the same epoch i enter using the GPU 0 like this,</p>
<pre><code class="lang-auto">        if dist.get_rank() == 0:
            print("Started Evaluating")
            model.eval()
</code></pre>
<p>I get the validation accuracy and other metrics properly without any error. But<br>
when i plotted the confusion matrix for each epoch, I noticed the sum of number of instances along each row (total Ground truth labels per class) across different epochs are different.</p>
<p>Please let me know how to validate on Single GPU when the model is wrapped with DDP.</p>

username: agu
cooked: <p>I think it depends on how your eval data is setup. Is only rank 0 loading the eval data? Otherwise, you could still do eval as data parallel and make sure to aggregate metrics across data parallel workers.</p>

username: Mukil
cooked: <p>Yes youre right. The validation loop only runs on rank 0. Eval data loads on a single gpu and performs validation while other gpus would waits. Right now, it works totally fine with the same setup, just that the training time and validation takes up lot of time. Thats why I was confused whether if this is the standard way to perform validation on single gpu.</p>
Slow costum cost function

username: Alexander_DE
cooked: <p>I implemented a new cost function. It is working, but overall full batch loss time exploded (vs. nn.NLLLoss() from 10 ms to 900 ms on MNIST10 dataset).<br>
system: torch 2.5.1, cuda 12, python 3.12.3 on ubuntu 24.04<br>
algo idea (for each x,y entry)</p>
<ol>
<li>make x-tensor mean-free by offset m (m = mean within non target entries only),</li>
<li>calculate norm(x-m)</li>
<li>a final cosine-sim-like value<br>
( I marked the critical lines by !!! )</li>
</ol>
<pre><code class="lang-auto">def cos_loss(x: tt.Tensor, y: tt.Tensor) -&gt; tt.Tensor:  # x.dim=2, y.dim=1
    "custom loss : shifted cosine (by A. Kleinsorge + A. Fauck)"
    # 1 - cos(x-m, yy), |yy|=1, yy=(1,0,0), cos = nn.CosineSimilarity(dim=)
    bs: int = y.numel() # batch_size,  !classes==10!
    # for i in range(bs): xy[i] = x[i][y[i]] # x[y] entries
    xy: tt.Tensor = x[tt.arange(bs), y.int()]  # x.index_select(?)  !!!!
    m: tt.Tensor = (x.sum(dim=1) - xy) * (1.0 / (10 - 1))  # avg all but target
    xmn: tt.Tensor = tt.zeros(bs, device=y.device)  # create out for next line
    for i,(x1,m1) in enumerate(zip(x, m)): xmn[i] = (x1-m1).norm() # (x-m).norm(dim=0)  !!!!
    return 1.0 - ((xy-m) / (xmn + 1e-6)).mean()
</code></pre>
<p>Any idea how to speed up the code?<br>
This algo has some nice mathematical properties, but this is another story.<br>
Thanks for reading.<br>
Alex</p>
Porting self-written CUDA kernels to Intel GPUs

username: Matthias_Moller
cooked: <p>I am using the C++ API of LibTorch 2.5.1 and have a few self-written CUDA/HIP kernels. I managed to get my code running on Intel GPUs using the XPU device.</p>
<p>I would like to know how to port the CUDA kernels to kernels for Intel GPUs. It’s really not much</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">/**
   @brief Compute Greville abscissae
*/
template &lt;typename real_t&gt;
__global__ void
greville_kernel(torch::PackedTensorAccessor64&lt;real_t, 1&gt; greville,
                const torch::PackedTensorAccessor64&lt;real_t, 1&gt; knots,
                int64_t ncoeffs, short_t degree, bool interior) {
  for (int64_t k = blockIdx.x * blockDim.x + threadIdx.x;
       k &lt; ncoeffs - (interior ? 2 : 0); k += blockDim.x * gridDim.x) {
    for (short_t l = 1; l &lt;= degree; ++l)
      greville[k] += knots[k + (interior ? 1 : 0) + l];
    greville[k] /= real_t(degree);
  }
}

/**
   @brief Compute knot vector
*/
template &lt;typename real_t&gt;
__global__ void knots_kernel(torch::PackedTensorAccessor64&lt;real_t, 1&gt; knots,
                             int64_t ncoeffs, short_t degree) {
  for (int64_t k = blockIdx.x * blockDim.x + threadIdx.x;
       k &lt; ncoeffs + degree + 1; k += blockDim.x * gridDim.x) {
    knots[k] = (k &lt; degree        ? static_cast&lt;real_t&gt;(0)
                : k &lt; ncoeffs + 1 ? static_cast&lt;real_t&gt;(k - degree) /
                                        static_cast&lt;real_t&gt;(ncoeffs - degree)
                                  : static_cast&lt;real_t&gt;(1));
  }
}
</code></pre>

username: guangyey
cooked: <p>Hi <a class="mention" href="/u/matthias_moller">@Matthias_Moller</a> Thank you for your question. I noticed that the same question has already been addressed <a href="https://dev-discuss.pytorch.org/t/custom-kernels-intel-xpu-backend-in-libtorch-c-api/2730/3" rel="noopener nofollow ugc">here</a>. Feel free to share any additional thoughts or continue the discussion there.</p>