akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Specifying GPU version of pytorch for python package in pyproject.toml

yrkim98: <p>Hi all,<br>
We’re working on a ML app that uses pytorch, but we’re having trouble specifying the gpu version of pytorch as a dependency for the build. Our project uses <code>pyproject.toml</code> to specify all dependancies and <code>setuptools</code> for the build. Our goal is to allow both CPU and GPU (if available) runs of pytorch after a user <code>pip install</code>’s our app without any further configuration needed.</p>
<p>we want to specify <code>torch==2.0.1+cu118</code> for windows and ubuntu users- so that if they have a GPU we will be able to use gpu pytorch. (<code>+cu118</code> will just default to cpu if there is no gpu available). We also want to specify the cpu-only <code>torch==2.0.1</code> for osx users since there is not a CUDA build for OSX.</p>
<p>I dont believe there is a good way to do this solely using <code>pyproject.toml</code> and <code>setuptools</code>, since we have to specify the <code>index-url</code> for the <code>+cu118</code> version of pytorch which is not supported by PEP and recent versions of python.</p>
<p>Ideally we dont have to bring in other dependency managers like <code>poetry</code> or <code>pdm</code>- but if there is a good solution with those I will consider it. The perfect solution would work with <code>pyproject.toml</code> and <code>setuptools</code>.</p>
<p>Thanks for the help</p>

julfried: <p>I am facing the same problem. As far as I know, poetry also does not work for this. I am curious to know if you found a solution to this problem or if anyone else has an idea, how this can be achieved.</p>
<p>Thanks for any help!</p>

AbdullahHendy: <p>I am trying to do the exact same thing. Wondering if there is a clean solution to this.</p>
Most efficient way to re-use grad computations in a layer which is a linear combination of linear layers

nikitaved: <p>Hi,</p>
<p>suppose I have a layer like<br>
<code>layer(x) = MyLinearLayer(x) + nn.Linear1(x) + ... + nn.Lineark(x)</code> with no biases.<br>
Suppose all <code>nn.Linear{i}.weight</code>s have the same shape, and <code>MyLinearLayer</code> represents a proxy of a weight of the very same shape. Because of the linear structure, all the weights in these modules should receive the very same gradient. This gradient I compute in a custom <code>MyLinearLayer.backward</code>. What would be the most efficient way to re-use this gradient for all the modules <code>nn.Linear{i}</code>? Of course, I could write a custom <code>layer.backward</code> to do that, but are there other, simpler ways?</p>
<p>Thank you!</p>

ptrblck: <p>I don’t understand the question completely. The incoming <code>dgrad</code> will be the same as you already explained and will be passed to the <code>backward</code> function of each module. The <code>wgrad</code> computation is specific to each module and you won’t be able to reuse anything since the computation depends on the forward activation. The outgoing <code>dgrad</code> computation uses the parameters so also unsure what you want to reuse.<br>
Could you clarify your use case a bit more?</p>

nikitaved: <p>Thank you for your reply!</p>
<p>Let me clarify. Let’s assume that Linear: (x, W) → (Wx) := y, so y = Wx, then<br>
the JVP is dy = dWx + Wdx. Let gy be the in-flowing grad, then for the VJP<br>
Tr(gy^T dy) = Tr(gy^T dW x) + Tr(gy^T W dx) = Tr(x gy^T dW) + [sensitivity for x which is irrelevant here], so the VJP for W, gW, is equal to gy x^T, and is independent of the forward compute. Since the input, x, is shared among the linear layers, and so is gy, the grad for each weight should be the same.</p>
<p>And the question is, what would be the easiest way to tell PyTorch that I have that structure? One solution could be just writing a custom Module that represents a linear combination of other Linear models. That, however, might be problematic in my use case because I have a very limited capability to modify the model’s graph due to quite rigid structure of subsequent non-trivial external optimizers with all sorts of backward hooks… Basically, I can only replace nn.Linear with nn.Linear. And I am not aware whether direct manipulations with the <code>grad</code> attribute are safe. Probably, custom nn.Linear{i} might work as well, with some API for grad swap and custom backward. But maybe there are other ways which are not subclass-based?</p>
Freezing Backbone and training the detection head using torchvision detection models

Richie_1980: <p>I’m working with a COCO-style custom dataset and would like to <strong>freeze the backbone and train only the detection head</strong> for the following <code>torchvision</code> models pretrained on COCO:</p>
<ul>
<li><code>fasterrcnn_resnet50_fpn</code></li>
<li><code>ssd300_vgg16</code></li>
<li><code>retinanet_resnet50_fpn</code></li>
<li><code>fcos_resnet50_fpn</code></li>
</ul>
<p>My goal is to:</p>
<ol>
<li>Load each model with pretrained backbone weights,</li>
<li>Replace or reinitialize the detection head to match my custom number of classes,</li>
<li>Freeze the backbone (i.e., no gradient updates),</li>
<li>Train only the detection head.</li>
<li>Do inference and compare metrics like mAP etc</li>
</ol>
<p>Can someone provide a reference implementation or point me in the right direction on how to do this cleanly?</p>
<p>Thanks!</p>
Does we save a random parameters if no train for model

vfdff: <p>I try to create a simple <strong>Mlp model</strong> without train, and then use <strong>torch.save</strong> to save the model parameters （they are random values which can be checked by print(param_tensor,‘\t’,model.state_dict()[param_tensor])）, so in fact, I only save a lots of random values, and it is not a tuned parameters?</p>
<p>PS: I also try to open the saved data <strong>mlp-simple1.pth</strong> by vim, but it only shows a bunch of garbled characters.</p>
<pre><code class="lang-auto">z00443407@llvm:~/source/test/simple_mlp$ cat save_model1.py 
import torch
from torch import nn
import torch_mlir

name : str
class MlpModel(torch.nn.Module):
    def __init__(self):
        super(MlpModel, self).__init__()
        self.net = nn.Sequential(nn.Flatten(),
                    nn.Linear(784, 256),
                    nn.ReLU(),
                    nn.Linear(256, 10))

    def forward(self, x):
        return self.net(x)


# 保存模型
model = MlpModel()
def save_model():
    path = "./mlp-simple1.pth"
    for param_tensor in model.state_dict():
        print(param_tensor,'\t',model.state_dict()[param_tensor].size())
        # print(param_tensor,'\t',model.state_dict()[param_tensor]) # Random value
    torch.save(model.state_dict(), path)

save_model()
</code></pre>

ptrblck: <p>Yes, saving the <code>state_dict</code> directly after initializing the model will save it’s initial random values, which were not trained yet.</p>
How to quantize my torchscript to fp8

mikeybydun1: <p>Hey,<br>
I have a torch script that do torch.prod on tensor in shape: (1000, 400, 400, 144).<br>
I am running this torchscript on Nvidia H100 GPU.<br>
I am trying to find a way to use fp8 quantization.<br>
How to do it while creating the model?<br>
Thanks!</p>

jerryzh168: <p>if you don’t need to use torchscript, you can try this: <a href="https://github.com/pytorch/ao/tree/main/torchao/quantization#a8w8-float8-dynamic-quantization-with-rowwise-scaling" class="inline-onebox" rel="noopener nofollow ugc">ao/torchao/quantization at main · pytorch/ao · GitHub</a></p>
Question about quantized model save & load

cokespace2: <p>I have a model like:</p>
<pre><code class="lang-auto">model = MyModel()
</code></pre>
<p>and I quantized part of it:</p>
<pre><code class="lang-auto">model.head = quantize(model.head)
</code></pre>
<p>and it runs fine:</p>
<pre><code class="lang-auto">output = model(input)
</code></pre>
<p>but when I save and load it like:</p>
<pre><code class="lang-auto">torch.save(model, 'model.pt')
new_m = torch.load('model.pt')
</code></pre>
<p>an error throwed:<br>
<strong>AttributeError: ‘ConvReLU2d’ object has no attribute ‘_modules’</strong></p>
<p>How can I fix this?</p>

jerryzh168: <p>this might mean ConvReLU2d module does not call <code>super().__init__()</code> in its constructor</p>
<p>can you show the complete quantization code?</p>

cokespace2: <p>Thanks for reply.</p>
<p>Well I cant’ show you all the quantization code since we built a toolkit that based torch fx graph, and it’s kind complicated.</p>
<p>The key point is that we use symbolic_trace to retrieve the model graph from eager model, then use <code>torch.ao.quantization.fx.fuse</code> to fuse batch norm and other operators, then lower it to GPU.</p>
<p>The weird thing is that the <code>ConvReLU2d</code> we are using is imported from <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/intrinsic/quantized/modules/conv_relu.py#L102" rel="noopener nofollow ugc">here</a> , and it does called <code>super().__init__()</code>, so I don’t think that’s the root cause.</p>

jerryzh168: <p>I see, I did see this error before while I was working on fx/eager, but can’t recall how these are resolved anymore..</p>
<p>If it’s possible to update your tools, I’d recommend to migrate to the pt2e quant toolchain: <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a> it also uses fx graph and the graph capture is better than symbolic_trace, more details can be found in <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.6.0+cu124 documentation</a></p>

cokespace2: <p>Great!<br>
But we are still working on PyTorch v2.2.0, does the PT2E needs higher version of PyTorch, like v2.4.0, to have full functionality?</p>

jerryzh168: <p>yeah pt2e probably needs a higher version like 2.4 I think</p>
How to pre-train ResNet using my own dataset

2U3_1967: <p>Could you let me know any paper or URL?</p>
<p>TIA</p>

Eduardo_Lawson: <p>Medium post: <a href="https://medium.com/@imabhi1216/fine-tuning-a-pre-trained-resnet-18-model-for-image-classification-on-custom-dataset-with-pytorch-02df12e83c2c" class="inline-onebox" rel="noopener nofollow ugc">Fine-Tuning a Pre-Trained ResNet-18 Model for Image Classification on Custom Dataset with PyTorch | by Abhishek Selokar | Medium</a><br>
Video tutorial from a different person: <a href="https://medium.com/@imabhi1216/fine-tuning-a-pre-trained-resnet-18-model-for-image-classification-on-custom-dataset-with-pytorch-02df12e83c2c" class="inline-onebox" rel="noopener nofollow ugc">Fine-Tuning a Pre-Trained ResNet-18 Model for Image Classification on Custom Dataset with PyTorch | by Abhishek Selokar | Medium</a><br>
Original resnet paper: <a href="https://arxiv.org/pdf/1512.03385" rel="noopener nofollow ugc">https://arxiv.org/pdf/1512.03385</a></p>
Negative "warps per SM"

JB13: <p>So i was profiling inference of a model , and got this data in the trace file. I wanna know why exactly the value for warps per SM is negative</p>
<p>{<br>
“ph”: “X”, “cat”: “Kernel”,<br>
“name”: “void at::native::unrolled_elementwise_kernel&lt;at::native::copy_device_to_device(at::TensorIterator&amp;, bool)::{lambda()<span class="hashtag-raw">#2</span>}::operator()() const::{lambda()<span class="hashtag-raw">#8</span>}::operator()() const::{lambda(float)<span class="hashtag-raw">#1</span>}, at::detail::Array&lt;char*, 2&gt;, TrivialOffsetCalculator&lt;1, unsigned int&gt;, char*, at::native::memory::LoadWithCast&lt;1&gt;, at::detail::Array&lt;char*, 2&gt;::StoreWithCast&gt;(int, at::native::copy_device_to_device(at::TensorIterator&amp;, bool)::{lambda()<span class="hashtag-raw">#2</span>}::operator()() const::{lambda()<span class="hashtag-raw">#8</span>}::operator()() const::{lambda(float)<span class="hashtag-raw">#1</span>}, at::detail::Array&lt;char*, 2&gt;, TrivialOffsetCalculator&lt;1, unsigned int&gt;, char*, at::native::memory::LoadWithCast&lt;1&gt;, at::detail::Array&lt;char*, 2&gt;::StoreWithCast)”, “pid”: 0, “tid”: “stream 7”,<br>
“ts”: 1744798720334022, “dur”: 7,<br>
“args”: {<br>
“queued”: 0, “device”: 0, “context”: 1,<br>
“stream”: 7, “correlation”: 3997, “external id”: 26,<br>
“registers per thread”: 32,<br>
“shared memory”: 0,<br>
“warps per SM”: -4.0,<br>
“grid”: [2, 1, 1],<br>
“block”: [64, 1, 1]<br>
}</p>
Pytorch to onnx problem

Mio: <p>ONNX does not support include_self=False for scatter_reduce,I need help</p>

justinchuby: <p>This is fixed in <a href="https://github.com/microsoft/onnxscript/pull/2090" rel="noopener nofollow ugc">Fix include_self for scatter_reduce by xadupre · Pull Request #2090 · microsoft/onnxscript</a>. Please try installing onnxscript from the main branch.</p>
Syntax to add desktop MNIST Dataset?

Ra_Concilio: <p>How do direct to the local root and/or online root for MNIST dataset to train and transform it? I downloaded MNIST onto my harddrive here: <a href="https://www.kaggle.com/datasets/novinnouri/mnist-dataset?resource=download" rel="noopener nofollow ugc">MNIST_dataset | Kaggle</a> . I understand how to change the folder directory but I am not sure if the data is supposed to be in a specific structure.</p>
<p>I tried this but it is getting an error?<br>
mnist_root = your/mnist/path<br>
train_data =datasets.MNIST(<br>
root = mnist_root ,<br>
train = True,<br>
transform = ToTensor(),<br>
download = True<br>
)<br>
<a href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/308be214fe516e34c8046196978cbe20235ed951.png">https://discuss.pytorch.org/uploads/default/original/3X/3/0/308be214fe516e34c8046196978cbe20235ed951.png</a></p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/how-to-call-and-train-mnist-dataset/219117">here</a>.</p>

Ra_Concilio: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - Can you help on the other post?</p>
Implement cuda kernels for useprivate1 device

ishan_gambhir: <p>can custom device be created in such a away that it uses all operations as per cuda device, except the the one which we override,… or is there any way to create a new custom device on top of CUDA device… which will perform all the operations similar to default except the one which i override</p>

ptrblck: <p>Would <a href="https://pytorch.org/docs/main/notes/extending.html#extending-torch-python-api">extending PyTorch</a> work for your use case of overloading specific functions?</p>
Torch.nn.functional.grid_sample with align_corners=False

Pepper9283: <p>I think I might find a small problem of this function. Please can you let me know if I am right on this.</p>
<p>When <code>align_corners=False</code>, <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/GridSampler.h#L33" rel="noopener nofollow ugc">this line</a> maps normalized coordinate in [-1,1] to [-0.5, <code>size</code> - 0.5]. Then, it is possible to have coordinates larger than <code>size-1</code> after this mapping. Let’s say we have a coordinate mapped to (x, y) with x &gt; <code>size-1</code>. Then the bilinear interpolation for (x, y) will use x0 = floor(x), x1 = x0+1, y0 = floor(y), y1 = y0 + 1, as shown <a href="https://github.com/pytorch/pytorch/blob/f29fe78cf2a6c99498b15a3e776a0eeeda4c1ea0/aten/src/ATen/native/GridSampler.cpp#L613" rel="noopener nofollow ugc">here</a>. Since x &gt; <code>size-1</code>, we actually have x0 = <code>size-1</code> and x1=<code>size</code>. As a result, although we have four points (x0, y0), (x1, y0), (x0, y1), (x1, y1) for interpolation, (x1, y0) and (x1, y1) will not be involved in calculation as indicated by codes <a href="https://github.com/pytorch/pytorch/blob/f29fe78cf2a6c99498b15a3e776a0eeeda4c1ea0/aten/src/ATen/native/GridSampler.cpp#L637" rel="noopener nofollow ugc">here</a> and <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/GridSampler.h#L205" rel="noopener nofollow ugc">here</a>. Now, the result of bilinear interpolation becomes a weighted sum of features at (x0, y0) and (x0, y1), with their original bilinear interpolation weights. As we only have two points, the weights do not sum to 1. This contradicts a little bit to the idea of bilinear interpolation where the sum of weights for features at all coordinates is 1 (i.e. some kind of average despite of varying weights).</p>
<p>Is my understanding for the code correct? Could this be a problem for marginal cases? Thank you for your help!</p>
A good first issue for beginners

kandgesid: <p>Hi,</p>
<p>I’m a Computer Science graduate student eager to start contributing to open source, especially PyTorch. With a solid machine learning background, I’m looking for a beginner‑friendly issue to tackle as my first contribution. Could you recommend one?</p>
<p>Thank you!</p>

ptrblck: <p>Check the <a href="https://github.com/pytorch/pytorch/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22">“good first issue” label</a> on GitHub and let us know if you find something interesting.</p>
Expected Tuple but got String Eroor

daewoung: <p>Hello. I’ve been trying continuously to convert the Hugging Face CLIPText model to TorchScript and deploy it on the Triton server. However, I keep encountering an issue where, even when sending the correct input to the torch.jit model, I get an “Expected Tuple but got String” error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de88bb80bc1fbf98ae411e6510b281ca2c690305.png" data-download-href="https://discuss.pytorch.org/uploads/default/de88bb80bc1fbf98ae411e6510b281ca2c690305" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de88bb80bc1fbf98ae411e6510b281ca2c690305_2_690x232.png" alt="image" data-base62-sha1="vKCWFG6CxKs2YMhMustBGHb4u1v" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de88bb80bc1fbf98ae411e6510b281ca2c690305_2_690x232.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de88bb80bc1fbf98ae411e6510b281ca2c690305_2_1035x348.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de88bb80bc1fbf98ae411e6510b281ca2c690305_2_1380x464.png 2x" data-dominant-color="789AAA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2168×732 249 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>torch==2.0.1 cu117, tritonserver:22.05-py3, diffusers==0.22.3, transformers==4.34.0</p>
<p>Plz help me</p>

mckay_wang: <p><a href="https://github.com/pytorch/pytorch/issues/53895" rel="noopener nofollow ugc">terminate called after throwing an instance of ‘c10::Error’ what(): isTuple() INTERNAL ASSERT FAILED at “/home/wenda/libtorch/include/ATen/core/ivalue_inl.h”:927, please report a bug to PyTorch. Expected Tuple but got GenericList · Issue #53895 · pytorch/pytorch</a></p>
How to enable libuv with PyTorch on windows

benHeid: <p>Currently, in a CI of PyTorch forecasting I am getting the error: “RuntimeError: use_libuv was requested but PyTorch was build without libuv support”</p>
<p>So I wonder if there is a way to install PyTorch with libuv support for windows…</p>
<blockquote>
<pre><code>      return TCPStore(
           hostname, port, world_size, start_daemon, timeout, multi_tenant=True, &gt; use_libuv=use_libuv
        )
</code></pre>
<p>E           RuntimeError: use_libuv was requested but PyTorch was build without libuv support</p>
</blockquote>
<p>Link: <a href="https://github.com/jdb78/pytorch-forecasting/actions/runs/10614322317/job/29419980235?pr=1622" class="inline-onebox" rel="noopener nofollow ugc">[MNT] CI matrix extended to `windows-latest` · jdb78/pytorch-forecasting@f50f946 · GitHub</a></p>
<p>Additional information:</p>
<ul>
<li>Unfortunately, I have not really the control to set use_libuv or set the init_method since this is done via a third party library. The only option would be to set the environment variable. Thus, I wonder if there is an easy way to install pytorch with libuv support</li>
<li></li>
</ul>

smlum: <p>hei  bro,have you solved this problem yet?  save me please</p>
Now the torch nightly corresponding to cuda 12.8 is well supported on my machine, but some programs require me to install a lower version of torch, such as torch1.10. Is there a way to solve this problem now?

lukeven: <p>My graphics card is 5070ti  Windows 11</p>

ptrblck: <p>No, PyTorch 1.10 was released in Oct. 2021 and won’t be compatible with the latest CUDA toolkit.<br>
I would also not recommend trying to backport all needed changes into such an old release but to make your repository compatible with the latest PyTorch version instead.</p>
CUDA stream sync issue in custom activation offloader

Vatsal_Joshi: <p>Hi everyone!</p>
<p>I am trying to build an activation offloader, taking inspiration from <code>autograd's</code> <code>save_on_cpu</code> function: <a href="https://github.com/pytorch/pytorch/blob/main/torch/autograd/graph.py#L317" rel="noopener nofollow ugc">save_on_cpu</a>.</p>
<p>To add more asynchrony, I’m trying to offload activations using a separate CUDA stream like this:</p>
<pre><code class="lang-auto">class OffloadActivations(saved_tensors_hooks):
    def __init__(self, pin_memory=True):
        self.transfer_stream = torch.cuda.Stream()
        def pack_tensor(activation) -&gt; tuple[torch.device, torch.Tensor]:
            if not pin_memory:
                return (activation.device, activation.cpu(), None)
            packed = torch.empty(
                activation.size(),
                dtype=activation.dtype,
                layout=activation.layout,
                pin_memory=(not activation.is_sparse),
            )
            event = torch.cuda.Event(enable_timing=False)
            with torch.cuda.stream(self.transfer_stream):
                packed.copy_(activation, non_blocking=True)
                event.record()
            return (activation.device, packed, event)

        def unpack_tensor(packed: tuple[torch.device, torch.Tensor]) -&gt; torch.Tensor:
            device, tensor, event = packed
            if event is not None:
                event.synchronize()
            return tensor.to(device, non_blocking=pin_memory)

        super().__init__(pack_tensor, unpack_tensor)
</code></pre>
<p>However, when testing this setup against a <code>nullcontext()</code> in a model training iteration, I’m running into a race condition — I’m seeing tensor mismatches between this and the baseline.</p>
<p>Am I missing something in terms of how CUDA streams and events should be used for synchronization across streams? Should I be explicitly syncing with the default stream?</p>
<p>Any insight would be appreciated, thanks a lot!</p>
cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [381,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.  Cou

chenGlin: <p>Multiple models of multiprocessing pool instances are used to reason about concurrent requests in a multithreaded environment. At the beginning, everything is fine, and after the program runs for about 10 or 20 minutes (the error occurs at a different time when the program restarts after throwing an exception), an exception is thrown:</p>
<p>/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [180,0,0], thread: [92,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [180,0,0], thread: [93,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [210,0,0], thread: [92,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [210,0,0], thread: [93,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [210,0,0], thread: [94,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [210,0,0], thread: [95,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [381,0,0], thread: [64,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [381,0,0], thread: [65,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1422: indexSelectLargeIndex: block: [381,0,0], thread: [66,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.</p>
<p>RuntimeError: CUDA error: device-side assert triggered<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Could you please tell me how to solve this problem?<br>
thanks!</p>

ptrblck: <p>Based on the error message you are running into an indexing error and would need to narrow down which operation fails. Using <code>CUDA_LAUNCH_BLOCKING=1</code> usually helps in isolating with call triggers this error, but I’m not sure how it would behave in your custom multi-processing/mult-threading env.</p>
Question about chain LR scheduler (CosineAnnealingLR + ExponentiaLR)

jesus-333: <p>Hi everyone. I’m trying to create an LR scheduler that combines cosine annealing and exponential decrease. My objective is to have something similar to the left image (see bottom of the post)</p>
<p>To do that, I combine the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html" rel="noopener nofollow ugc">CosineAnnealingLR</a> and the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ExponentialLR.html" rel="noopener nofollow ugc">ExponentiaLR</a> using the ChainedScheduler.<br>
The code is something similar</p>
<pre><code class="lang-auto">scheduler_list = [
        torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max = 9, eta_min = 1e-5),
        torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma = 0.92)
]
lr_scheduler = torch.optim.lr_scheduler.ChainedScheduler(schedulers_list, optimizer)
</code></pre>
<p>But the final result (for 80 epochs) is the right image (see bottom of the post)</p>
<p>I have the initial decrease, but after that, it always rebounds at the same value.<br>
How can I obtain my initial goal? Am I doing something wrong with the parameter settings? Or is it directly the choice of schedulers?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afa0661593930b408a3818112e5b04b7664f390b.png" data-download-href="https://discuss.pytorch.org/uploads/default/afa0661593930b408a3818112e5b04b7664f390b" title="Screenshot 2025-04-16 at 10.17.49-imageonline.co-merged"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afa0661593930b408a3818112e5b04b7664f390b_2_690x129.png" alt="Screenshot 2025-04-16 at 10.17.49-imageonline.co-merged" data-base62-sha1="p3Fd1D2Oan0GftR1wGsvsTMQvTZ" width="690" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afa0661593930b408a3818112e5b04b7664f390b_2_690x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afa0661593930b408a3818112e5b04b7664f390b_2_1035x193.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afa0661593930b408a3818112e5b04b7664f390b_2_1380x258.png 2x" data-dominant-color="535455"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-04-16 at 10.17.49-imageonline.co-merged</span><span class="informations">1670×314 56.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch.jit._get_trace_graph

QLYYLQ: <h1><a name="p-469193-problem-1" class="anchor" href="#p-469193-problem-1"></a>Problem</h1>
<p><strong>Using segment_anything and fvcore but get an error in torch.jit._get_trace_graph().</strong></p>
<p>When i used fvcore.nn.FlopCountAnalysis(sam,input_batch).by_module_and_operator(), which all produced by meta, i met the error below.</p>
<p>The function: <code>graph, _ = _get_trace_graph(module, inputs)</code> meets: <code>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)</code></p>
<p>I add module and inputs into cuda before this function. Also i met segfault when using python debug tools</p>
<p>I changed some of sam’s code (making its input just a list of images, and setting prompt words (dot, mask, bbox) to None).</p>
Is gradient flow lost when using Numpy?

marcos_tulio: <p>I’m having difficulty training a deep learning model where some data is obtained from a library that uses Numpy. I’ve noticed that, although the necessary derivative calculations for training an optimization model are being performed, the model is not managing to learn. Is it possible that the tensor graph is being lost? My example uses the EPANET library (a library for reading hydraulic simulators), and there are several functions written in Numpy. These functions are called within the <code>grad_steps</code> function.</p>
<p>(…)<br>
solver_net = NNSolver(data, args)<br>
solver_net.to(DEVICE)<br>
solver_opt = optim.Adam(solver_net.parameters(), lr=solver_step)</p>
<pre><code>stats = {}
for i in range(nepochs):

    solver_net.train()
    for Xtrain in train_loader:
        Xtrain = Xtrain[0].to(DEVICE)
        solver_opt.zero_grad()
        Yhat_train = solver_net(Xtrain)
        Ynew_train = grad_steps(data, Xtrain, Yhat_train, args)
        train_loss = total_loss(data, Xtrain, Ynew_train, args)
        train_loss.sum().backward()
        solver_opt.step()
</code></pre>
<p>(…)</p>
<p>Thank you for any help</p>

ptrblck: <p>Yes, using 3rd party libs, such as <code>numpy</code>, will detach the tensor from the computation graph and you would need to implement custom <code>autograd.Function</code>s and define the <code>backward</code> method explicitly. Alternatively, try to use PyTorch operations instead of <code>numpy</code> (if possible).</p>

marcos_tulio: <p>Thank you for the response. Unfortunately, implementing the third-party package in PyTorch is not possible. I saw a possible implementation at <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch: Defining New autograd Functions — PyTorch Tutorials 2.6.0+cu124 documentation</a>, however, I have four different functions that use NumPy. Does that mean I need to create a separate class inheriting from autograd.Function for each function?</p>

ptrblck: <p>Yes, you would need to implement all custom functions and implement the backward method.</p>
Memory error on ONE GPU destribution on the CPU befor moving the data

Maria_Djeblahi: <p>I’m trying to train a complex models on very large molecular  dataset, each row on this dataset is tuple constructed of (drug graph, protein graph, label, Demb, Aemb), the type of the graphs are a PyG data and the embeds are a dictionaries, to accelerate the training of the models I tried to use the DDP on one GPU (so I’m not using the spawn), but when doing that I get a Memory Error / OOM killed / or too many open files  the code is organized as the followings:</p>
<pre><code class="lang-auto">def main () : 
# Dist. init_process_group ()
    # if rank = 0 
    # load a metadata file containing the ids 
    # split the data into train test validation 
    # broadcast_object_list of the train and validation set 
    # ALL ranks execute 
    # load the file  that contains the graphs and the Embedding information (more than 13G), retrieved using the    identifiers to the CPU 
   # for Each pair in the metadata retrieves the information from the files to construct the tuple (drug graph, protein graph, label, demb, aemb)
   #******* some time the code comes into this point and failed ***# 
   # initialise the model and move it to the GPU 
   #  model = DDP (model, device_ids=[current_device], broadcast_buffers=False)
  #  initialise the optimiser + criterion 
  # customed Dataset class initialization 
  # DistributedSampler 
  # Dataloader 
  # training 
  # validation 
</code></pre>
<p>I would now if I’m using the DDP in wrong ways that it causes the problem, and it seems that each rank duplicate the dataset which is already large and it causes the problem, and secondly if the problem comes from this point what is the solution:</p>
<ol>
<li>To load the data only on rank = 0 and then distributed it, how can we do that without stoking copies into the memory</li>
<li>If we used a shared Memory, we can get a problem with the access<br>
I hope you can help me with these, I can provide any additional problem</li>
</ol>

ptrblck: <p>Would it be possible to lazily load the data on each rank (which is the common approach using DDP)?</p>

Maria_Djeblahi: <p>In this case, is there any common solution to this problem in PyTorch?</p>

ptrblck: <p>I’m not sure I understand your question. I asked if you have considered to lazily load the data (which is possible in PyTorch) and if this would be a possible approach.<br>
If you are looking for an example check the <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files">Dataset tutorial</a>.</p>
Deterministic problem for torch.multinomial on different device

Stark_Ivan: <p>I observed that <code>torch.multinomial</code> behaves deterministically on the CPU device. However, I am unable to reproduce the same results on the GPU. The following script demonstrates the issue I encountered:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import random
import numpy as np
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

seed = 2147483647
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

w1 = w2 = torch.rand(3065713)

device='cuda'
g = torch.Generator(device=device).manual_seed(2147483647)           #############
random_selection1 = torch.multinomial(w1.to(device), 150, replacement=True, generator=g)          ##############
print(random_selection1)

g = torch.Generator(device=device).manual_seed(2147483647)           #############
random_selection2 = torch.multinomial(w2.to(device), 150, replacement=True, generator=g)          ##############
print(random_selection2)

print((random_selection1 != random_selection2).sum())
</code></pre>

ptrblck: <p>Not reproducible after setting <code>torch.use_deterministic_algorithms(True)</code> as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>.</p>

Stark_Ivan: <p>When I enable <code>torch.use_deterministic_algorithms(True)</code>, I encounter the following error: <code>RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'</code>.</p>

ptrblck: <p>I don’t see the error using <code>2.8.0.dev20250404+cu128</code> so you might need to update your PyTorch binary. Which release are you using?</p>

Stark_Ivan: <p>I am using PyTorch version 2.5.0+cu118.</p>

KFrank: <p>Hi Ivan!</p>
<aside class="quote no-group" data-username="Stark_Ivan" data-post="5" data-topic="219069" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark_ivan/48/74584_2.png" class="avatar"> Stark_Ivan:</div>
<blockquote>
<p>I am using PyTorch version 2.5.0+cu118.</p>
</blockquote>
</aside>
<p>I can reproduce your issue using pytorch 2.5.1 (with cuda 12.4) when I add<br>
<code>torch.use_deterministic_algorithms (True)</code> to the code you posted, but the error goes<br>
away – that is, <code>torch.multinomial()</code> becomes deterministic with no errors raised – when I<br>
use version 2.6.0+cu126 (the current stable release and that choice of cuda).</p>
<p>Try upgrading to the current stable version (or higher).</p>
<p>Best.</p>
<p>K. Frank</p>
CppExtension() with MPI

ettu: <p>Hi, I’m trying to use the CppExtension to import my C++ MPI code as a library. It works when I take out the MPI components, but with it, I get the “undefined symbol: ompi_mpi_int” error. Here’s what my CppExtension object looks like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61b2e2025161c76895060e2674dcff3f6f8bc730.png" data-download-href="https://discuss.pytorch.org/uploads/default/61b2e2025161c76895060e2674dcff3f6f8bc730" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61b2e2025161c76895060e2674dcff3f6f8bc730_2_690x145.png" alt="image" data-base62-sha1="dWhziEeqZ0hh3tOH9JfHN14lEhW" width="690" height="145" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61b2e2025161c76895060e2674dcff3f6f8bc730_2_690x145.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61b2e2025161c76895060e2674dcff3f6f8bc730_2_1035x217.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61b2e2025161c76895060e2674dcff3f6f8bc730_2_1380x290.png 2x" data-dominant-color="222426"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1429×302 37.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anything seem blatantly set up incorrectly? I’ve also tried setting the environment variables of my openmpi install manually:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/981eafbeefe990c4eba4e6c173292caa4f41b8ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/981eafbeefe990c4eba4e6c173292caa4f41b8ef" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/981eafbeefe990c4eba4e6c173292caa4f41b8ef.png" alt="image" data-base62-sha1="lHIjU6qMOzdHnIo4Q1PzvMIHjwX" width="669" height="181"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">669×181 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How do I remove torch_dtype=torch.float16 on Windows 11?

Taurus1983: <h1><a name="p-468266-pipelines-loaded-with-torch_dtypetorchfloat16-cannot-run-with-cpu-device-it-is-not-recommended-to-move-them-to-cpu-as-running-them-will-fail-please-make-sure-to-use-an-accelerator-to-run-the-pipeline-in-inference-due-to-the-lack-of-support-forfloat16-operations-on-this-device-in-pytorch-please-remove-the-torch_dtypetorchfloat16-argument-or-use-another-device-for-inference-1" class="anchor" href="#p-468266-pipelines-loaded-with-torch_dtypetorchfloat16-cannot-run-with-cpu-device-it-is-not-recommended-to-move-them-to-cpu-as-running-them-will-fail-please-make-sure-to-use-an-accelerator-to-run-the-pipeline-in-inference-due-to-the-lack-of-support-forfloat16-operations-on-this-device-in-pytorch-please-remove-the-torch_dtypetorchfloat16-argument-or-use-another-device-for-inference-1"></a>Pipelines loaded with <code>torch_dtype=torch.float16</code> cannot run with <code>cpu</code> device. It is not recommended to move them to <code>cpu</code> as running them will fail. Please make sure to use an accelerator to run the pipeline in inference, due to the lack of support for<code>float16</code> operations on this device in PyTorch. Please, remove the <code>torch_dtype=torch.float16</code> argument, or use another device for inference.</h1>

ptrblck: <p>Did you try removing <code>torch_dtype=torch.float16</code> in your model creation as explained in the error message?</p>

rxrue: <p>where would i find this file so i can remove it having the same problem</p>
Need Advice on Training a Model for Blind Navigation App (iOS)

albert2: <p>Hey everyone!</p>
<p>I’m currently working on a project for my master’s degree, and I’d love to get some feedback, guidance, or even collaboration if anyone’s interested!</p>
<p>The idea is to develop an iOS navigation app for blind users, which uses the phone’s camera and machine learning to analyze the environment in real time, and provide audio guidance to help users navigate safely.</p>
<p>How it works (in theory):<br>
The app will use the phone camera to detect important objects and features in the environment — things like:</p>
<ul>
<li><strong>Pedestrian paths</strong></li>
<li><strong>Traffic lights</strong></li>
<li><strong>Dangerous areas</strong> (e.g., stairs, escalators)</li>
<li><strong>Braille signs/text</strong> (if possible)</li>
</ul>
<p>It will also combine this visual input with the user’s GPS location to help guide them toward their destination.</p>
<p>I’m still defining what the ML model should be able to recognize and how to structure everything. Right now, I’m trying to figure out:</p>
<ul>
<li>Should I use object detection, image segmentation, or a combination of both?</li>
<li>How can I find or build datasets for training (e.g., traffic lights, stairs, Braille)?</li>
<li>Is this realistic as a single project, or should I drop some features for now?</li>
</ul>
<p>If anyone has experience with similar projects, or knows good resources, tutorials, or datasets — I’d be super grateful for any tips or advice!</p>
<p>Thanks a lot in advance!</p>
Return type of the closure in optimizer.step method

arampatzis: <p>In the <a href="https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.step.html#torch.optim.Optimizer.step" rel="noopener nofollow ugc">PyTorch documentation</a>, the closure function passed to <code>Optimizer.step()</code> is expected to return a <code>float</code>. However, in the <a href="https://pytorch.org/docs/stable/optim.html#optimizer-step-closure" rel="noopener nofollow ugc">official example</a>, the closure actually  returns a <code>torch.Tensor</code>.</p>
<p>In practice — especially when using optimizers like <code>LBFGS</code> — the closure <strong>must</strong> return a <code>torch.Tensor</code> so that <code>backward()</code> can be called properly and the optimizer has access to the computation graph.</p>
<p>This inconsistency leads to issues with static type checkers like <code>mypy</code>, which raise a type error if the closure returns a <code>torch.Tensor</code> instead of a <code>float</code>.</p>
<p>It would be helpful if the documentation and type annotations were updated to reflect the actual usage pattern. Or at least clarified that returning a <code>Tensor</code> is not only acceptable, but often required.</p>
Storing loss each iteration when using a closure (e.g. LBFGS)

ccook: <p>When you estimate using LBFGS, you have to wrap the optimization steps in a closure, so each pass through has something like the following:</p>
<pre><code class="lang-auto">def closure(): 
    optimizer.zero_grad()
    prediction = model(data)
    loss = criterion(prediction, target) 
    loss.backward()
    return loss

optimizer.step(closure)
</code></pre>
<p>Having the loss within the closure makes it difficult to stash what the current loss is each time (i.e. can’t just do <code>losses += [loss.detach().item()]</code>).</p>
<p>One option is to re-evaluate the model and criterion each time outside of the closure (with <code>torch.no_grad()</code>), but this is a waste of compute.</p>
<p>Is there a better way?</p>

arampatzis: <p>I have the same problem. As far as I understand, there is no way to know the loss value of the current iteration.</p>
<p>I end up re-computing the loss right after the <code>optimizer.step(closure)</code>.</p>
<p>Are there any update on this?</p>
TorchRuntimeError: Failed running call_module L__self___position_embedding_table(*(FakeTensor(..., device='mps', ...), **{}): Unhandled FakeTensor Device Propagation for aten.index_select.default, found two different devices mps:0, mps

Siladittya_Manna: <p>I am trying to run a transformer model on a Macbook M2. I have installed PyTorch with MPS support.</p>
<p>So, I have set <code>device = torch.device("mps")</code></p>
<p>I am following this tutorial notebook</p><aside class="onebox githubblob" data-onebox-src="https://github.com/ImadSaddik/Train_Your_Language_Model_Course/blob/main/notebooks/4_1_ModelTrainingAllBatches.ipynb">
  <header class="source">

      <a href="https://github.com/ImadSaddik/Train_Your_Language_Model_Course/blob/main/notebooks/4_1_ModelTrainingAllBatches.ipynb" target="_blank" rel="noopener nofollow ugc">github.com/ImadSaddik/Train_Your_Language_Model_Course</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/ImadSaddik/Train_Your_Language_Model_Course/blob/main/notebooks/4_1_ModelTrainingAllBatches.ipynb" target="_blank" rel="noopener nofollow ugc">notebooks/4_1_ModelTrainingAllBatches.ipynb</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/ImadSaddik/Train_Your_Language_Model_Course/blob/main/notebooks/4_1_ModelTrainingAllBatches.ipynb" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('..')"
   ]
  },
  {
</code></pre>



  This file has been truncated. <a href="https://github.com/ImadSaddik/Train_Your_Language_Model_Course/blob/main/notebooks/4_1_ModelTrainingAllBatches.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>When I am starting to run the training cell in the notebook, I face this error in this line</p>
<pre><code class="lang-auto">self.positional_embedding = self.position_embedding_table(torch.arange(T, device=self.device))  # (T,C)
</code></pre>
<p>where positional_embedding is declared as</p>
<pre><code class="lang-auto">self.position_embedding_table = nn.Embedding(block_size, n_embd)
</code></pre>
<p>Another piece of information: I am <code>compiling</code> the model after instantiating it.</p>
<p>UPDATE 1: The error does not occur when not using <code>torch.compile</code></p>
<p>UPDATE 2: This error does not occur when using <code>torch.compile</code> if I run the code on CUDA device.</p>
<p>Why is it showing two different “mps” devices? and how to solve this issue?</p>