akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to multiply two tensors along selected dimensions

nil_tdr: <p>Dear altruists,</p>
<p>I have two tensors with dim (2, 4, 4)) and (4,4) respectively.</p>
<blockquote>
<p>A = tensor([[[0, 0, 1, 0],<br>
[0, 0, 1, 0],<br>
[1, 1, 0, 1],<br>
[0, 0, 1, 0]],<br>
[[0, 0, 1, 0],<br>
[0, 0, 1, 0],<br>
[1, 1, 0, 1],<br>
[0, 0, 1, 0]]])</p>
</blockquote>
<blockquote>
<p>B = tensor([[0, 1, 1, 0],<br>
[1, 0, 1, 0],<br>
[1, 1, 0, 1],<br>
[0, 0, 1, 0]])</p>
</blockquote>
<p>I want to do multiplication along second and third axes. My desired output is a vector with dim 2.</p>
<p>I am trying to take advantage of the speed of torch.cuda() for matrix multiplication. Is there an elegant way to solve this problem?</p>
<p>Thanks  in advance!</p>

KFrank: <p>Hi Nil!</p>
<aside class="quote no-group" data-username="nil_tdr" data-post="1" data-topic="197470" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nil_tdr/48/65323_2.png" class="avatar"> nil_tdr:</div>
<blockquote>
<p>I want to do multiplication along second and third axes. My desired output is a vector with dim 2.</p>
</blockquote>
</aside>
<p>It’s not clear to me what you mean by this.</p>
<p>One guess is that you would like to perform a so-called <em>contraction</em> on<br>
the last two dimensions of your two tensors.  If so, <code>torch.einsum()</code><br>
might be the simplest approach:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.2.0'
&gt;&gt;&gt; A = torch.tensor([[[0, 0, 1, 0],
... [0, 0, 1, 0],
... [1, 1, 0, 1],
... [0, 0, 1, 0]],
... [[0, 0, 1, 0],
... [0, 0, 1, 0],
... [1, 1, 0, 1],
... [0, 0, 1, 0]]])
&gt;&gt;&gt; B = torch.tensor([[0, 1, 1, 0],
... [1, 0, 1, 0],
... [1, 1, 0, 1],
... [0, 0, 1, 0]])
&gt;&gt;&gt; torch.einsum ('ijk, jk -&gt; i', A, B)
tensor([6, 6])
</code></pre>
<p><code>einsum()</code> is relatively mature and well optimized so I would expect it to take<br>
nearly full advantage of the performance of a gpu.</p>
<p>(If this is not what you are trying to do, please post a script that implements<br>
your desired computation – using for loops, if that’s what works for you – to<br>
make clear specifically what you want.)</p>
<p>Best.</p>
<p>K. Frank</p>

nil_tdr: <p>Thanks K. Frank. I apologize for my late response. I solved the problem.</p>
<p>I added a dimension at the beginning of B to make it (1, 4, 4). Then I multiplied A and B. My output had dimension (2, 4, 4). I summed along second and third axis and I got final output vector of dim 2.</p>
RuntimeError: Unrecognized CachingAllocator option: 0

zhangwonderful: <p>Hi ,<br>
I ran my program in ubuntu 22, nvdidia infomation as followings:<br>
NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4<br>
torch version :2.2.1+cu121</p>
<p>File “/home/ubuntu/ChatGLM3/openai_api_demo/openai_api.py”, line 476, in <br>
model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).to(DEVICE).eval()<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling_utils.py”, line 2556, in to<br>
return super().to(*args, **kwargs)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1152, in to<br>
return self._apply(convert)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 802, in _apply<br>
module._apply(fn)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 802, in _apply<br>
module._apply(fn)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 802, in _apply<br>
module._apply(fn)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 825, in _apply<br>
param_applied = fn(param)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1150, in convert<br>
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)<br>
File “/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 302, in _lazy_init<br>
torch._C._cuda_init()<br>
RuntimeError: Unrecognized CachingAllocator option: 0</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue, please?</p>
Scaled Dot Product Attention Algorithm Selection—results don't match documentation/errors

chocolate_fireball: <p>In <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html</a>, I see there are three scaled dot product attention algorithms.</p>
<p>I am facing an interesting scenario where the kernels for flash and memory-efficient attention aren’t installed, but I get different results when I use</p>
<pre><code class="lang-auto">with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_math=True, enable_mem_efficient=False):
    ...
</code></pre>
<p>and</p>
<pre><code class="lang-auto">with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=True, enable_mem_efficient=True):
    ...
</code></pre>
<p>Specifically, the difference in results between running with and without <code>torch.use_deterministic_algorithms()</code> set and all sources of randomness accounted for is zero in the first instance (flash and memory-efficient disabled) and the second (all attention algorithms enabled). But as far as I know, there is only one possible implementation on my system. Is there any explanation for this?</p>
<p>Would massively appreciate help with this; it might end a five month debugging saga!</p>

ptrblck: <aside class="quote no-group" data-username="chocolate_fireball" data-post="1" data-topic="198333">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chocolate_fireball/48/66869_2.png" class="avatar"> chocolate_fireball:</div>
<blockquote>
<p>I am facing an interesting scenario where the kernels for flash and memory-efficient attention aren’t installed,</p>
</blockquote>
</aside>
<p>I don’t fully understand this statement since PyTorch itself ships with the needed kernels.</p>

chocolate_fireball: <p>Thanks for your reply! It appears they are all installed; I hadn’t fully read the error message, eg. when running memory-efficient attention with determinism.</p>
Does PyTorch have more "inplace" ops?

npz7yyk: <p>I am currently building memory efficient PyTorch systems. I find a really annoying and pitiful thing - for many ops in PyTorch, I dtk how to ensure that the output is stored at a certain place.</p>
<p>For example:<br>
a = b * c, say b.shape = (1024, 16), c.shape = (16,), then what I want to do is:</p>
<ol>
<li>a = torch.empty(1024, 16)</li>
<li>let the b * c result be written to the just allocated memory.</li>
</ol>
<p>A solution might be bc = b * c, a.copy_(bc), but this has extra costs. The demand is beneficial because sometimes I am aware of all ops in the program and I want to make sure that all valuable results are stored one by one in memory while all activations are allocated elsewhere.<br>
There might be certain ways for solving each op. And my following question is, is there a uniform way to solve this? i.e. a method independent from op. For example, a wrapper, used like wrap(mem, func, *arg), to ensure that the final result is written into mem.</p>
<p>Many thanks! And I dtk how to category this post, advice is helpful!</p>
Timing model modules

AidanGoldfarb: <p>I am attempting to time each layer of a network by registering hooks. I then run four inference benchmarks.</p>
<ul>
<li>A control, running the model as is</li>
<li>Compiling for the CPU with torch.compile()</li>
<li>Running the model on the GPU</li>
<li>Comping the model to Triton (GPU) using torch.compile()</li>
</ul>
<p>However, the introduction of hooks seems to cause the compiler a lot of problems. I get several cache limit hits, meaning torch attempted to recompile functions several times and failed. This leads to the runtime of the torch.compile() benches to be slower than their device counterpart. However, running these same benches without hooks, and with the <a href="https://pytorch.org/docs/stable/profiler.html" rel="noopener nofollow ugc">torch profiler</a>, I get the expected result that the compiled version is normally faster.</p>
<p>My hooks are setup like this:</p>
<pre><code class="lang-auto">    def register_hooks(self,model):
        def pre_hook_fn(module, input):
            module.start_time = time.time()

        def forward_hook_fn(module, input, output):
            end_time = time.time()
            rt = end_time - module.start_time
            self.layer_times.append( (str(module), float(rt)) ) #append each layer time to a dict

        for module in model.modules():
            module.register_forward_pre_hook(pre_hook_fn)
            module.register_forward_hook(forward_hook_fn)
</code></pre>
<p>I am confused as to why these hooks are causing so many issues with the compiler and its efficiency. The torch profiler is fairly undocumented and cumbersome to use, and does not provide me with the granularity (per layer) that I would like. Any advice would be appreciated, thanks!</p>
FSDP with size_based_auto_wrap_policy freezes training

CasellaJr: <p>I am doing multi-node and multi-GPU training. Right now, I am optimizing my multi-GPU code for faster training.</p>
<pre><code class="lang-auto">torch.cuda.set_device(self.rank)
model = FSDP(model, use_orig_params=True, device_id=torch.cuda.current_device())
</code></pre>
<p>Basically, everything works fine, but If I modify the model using the <code>size_based_auto_wrap_policy</code> what happens is that training starts, but exactly after 4 epochs it stops and it does not continue. Both the GPUs (2 A40) appears to be used 100% with <code>nvidia-smi</code>.</p>
<pre><code class="lang-auto">my_auto_wrap_policy = functools.partial(
        size_based_auto_wrap_policy, min_num_params=100
        )
model = FSDP(model, use_orig_params=True, device_id=torch.cuda.current_device(), auto_wrap_policy=my_auto_wrap_policy)
</code></pre>
<p>Moreover, another problem that happens when using this wrap policy is that when killing the script with CTRL+C, the process remains alive in background, indeed If I train to run again the script I have this error:</p>
<pre><code class="lang-auto">[W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use).
[W socket.cpp:401] [c10d] The server socket has failed to bind to ?UNKNOWN? (errno: 98 - Address already in use).
[E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
...
RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:12345 (errno: 98 - Address already in use). The server socket has failed to bind to 12345 (errno: 98 - Address already in use).
</code></pre>
DistributedSampler seed on spot instances

bhack: <p>I have a question about correctly handle DDP with spot instance.</p>
<p>Using <code>torch.utils.data.distributed.DistributedSampler(my_dataset)</code> it is going to set <code>seed=0</code> (its default for every GPU process).</p>
<p>From the doc:</p>
<blockquote>
<p>This number should be identical across all processes in the distributed group.</p>
</blockquote>
<p>The problem with spot instance is that the training could be restarted multiple times for preemption so with the default seed value are we going to repeat the initial dataset sequence  when the training job is going to restart from the last available checkpoint?</p>

bhack: <p>I don’t know if we want to merge this with <a href="https://discuss.pytorch.org/t/how-to-save-distributedsampler-state-for-resuming-later/121996" class="inline-onebox">How to save distributedSampler state for resuming later?</a> at it could be similar.<br>
But the reply there was not so clear about the internals.</p>
LoadLibrary error while using libtorch with ROS2

Ars_ML: <p>I am trying to build and run a ROS2 package which uses libtorch.</p>
<p>In CMakeLists.txt I wrote</p>
<pre><code class="lang-auto">set(Torch_DIR ~/libtorch/share/cmake/Torch/)
find_package(Torch)

  ament_auto_add_library(my_lib SHARED  
    src/node.cpp)
  
  target_link_libraries(my_lib 
    ${TORCH_LIBRARIES}
  )
</code></pre>
<p>Package can be built without errors, but there is an error while running it:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'class_loader::LibraryLoadException'
  what():  Could not load library LoadLibrary error: libtorch_cuda.so: cannot open shared object file: No such file or directory, at /tmp/binarydeb/ros-foxy-rcutils-1.1.5/src/shared_library.c:84
</code></pre>
<p>If I add path to libtorch to LD_LIBRARY_PATH:<br>
<code>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/libtorch/lib/</code><br>
there is another error while running package:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'class_loader::LibraryLoadException'
  what():  Could not load library LoadLibrary error: /home/arseniy.marin@nami.local/libtorch/lib/libgomp-75eea7e8.so.1: version `GOMP_4.5' not found (required by /lib/x86_64-linux-gnu/libpcl_common.so.1.10), at /tmp/binarydeb/ros-foxy-rcutils-1.1.5/src/shared_library.c:84
</code></pre>
<p>Libtorch 1.8.1<br>
ROS2 Foxy<br>
Ubuntu 20.0.4</p>
How to make models that contains `log_prob` and needs to create local tensors in `forward` parallelly trainable?

jack_S: <p>I created a toy model like following and I wanted to wrap it with <code>DistributedDataParallel</code> but I failed due to <code>tensor.device</code> errors (especially those in <code>Distribution</code>):</p>
<pre><code class="lang-auto">from torch import nn
from torch.distributions import Distribution, Uniform, Normal
from torch import Tensor, diag, zeros, randn

class Opt(nn.Module):
    def __init__(self):
        super().__init__()
        self.I = diag(Tensor([1., 1., 1.]))
    
    def forward(self, x:Tensor, p:Tensor):
        x = self._some_transforms_1(x, p)
        x = self._some_transforms_2(x, p)
        return x

    def _some_transforms_1(self, x:Tensor, p:Tensor):
        trans_matrix = zeros(3, 3)
        trans_matrix[0, 0:2] = p[0:2]
        trans_matrix += self.I
        return x @ trans_matrix

    def _some_transforms_2(self, x:Tensor, p:Tensor):
        trans_matrix = zeros(3, 3)
        trans_matrix[1, 0:2] = p[2:4]
        trans_matrix -= self.I
        return x @ trans_matrix
    
class Model(nn.Module):
    def __init__(self, opt:Opt, dist_1:Distribution, dist_2:Distribution):
        super().__init__()
        self.opt = opt
        self.dist_1 = dist_1
        self.dist_2 = dist_2

    def forward(self, x:Tensor, p:Tensor, w:float):
        x = self.opt(x, p)
        lp_1 = self.dist_1.log_prob(p)
        lp_2 = self.dist_2.log_prob(p)
        return x.sum() * lp_1 * lp_2
        
        
data = randn(100, 3, 3)
opt = Opt()
dist_1 = Normal(Tensor([0] * 4), Tensor([1] * 4))
dist_2 = Uniform(Tensor([-1] * 4), Tensor([1] * 4))
model = Model(opt, dist_1, dist_2)
for x in data:
    p = dist_2.sample((1,)).squeeze()
    loss = model(x, p, 1.)
</code></pre>
<p>I tried to do some operations like adding <code>self.I = nn.Parameter(self.I)</code> and <code>trans_matrix = trans_matrix.to(x.device)</code> to make the model parallelly trainable but I have some problem with the distributions:</p>
<pre><code class="lang-auto">(self.lower_bound &lt;= value) &amp; (value &lt;= self.upper_bound)
</code></pre>
<p>the <code>lower_bound</code> and <code>upper_bound</code> in <code>distributions.constraint._Interval</code> are not automatically moved to cuda devices. Could anyone tell me how to make this model parallelly trainable?</p>
Need to add background class or other labels should be increase

Deneme13: <p>I work on kitti dataset from torchvision.datasets.Kitti than tohse labels does not include background class. So i have to increase my labels coding 1 or add background class i think. but i got error<br>
<code>AttributeError: type object 'Kitti' has no attribute 'target'</code><br>
so how can i change my iinteger labels</p>
<pre><code class="lang-auto">transform = transforms.Compose(
    [transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,)),
    transforms.Resize((380,1500))
    ])
training_set = torchvision.datasets.Kitti(root, train=True, download=False,transform=transform)

validation_set = torchvision.datasets.Kitti(root, train=False, download=False, transform=transform)

training_set = datasets.wrap_dataset_for_transforms_v2(training_set)
validation_set = datasets.wrap_dataset_for_transforms_v2(validation_set)
sample = training_set[0]
img, target = sample
print(f"{type(img) = }\n{type(target) = }\n{target.keys() = }")
</code></pre>
<p>after all of this i get those output for those cells:</p>
<pre><code class="lang-auto">type(img) = &lt;class 'torch.Tensor'&gt;
type(target) = &lt;class 'dict'&gt;
target.keys() = dict_keys(['boxes', 'labels'])
</code></pre>
<p>in short<br>
how can i make my labels 0-8  to 1-9<br>
thank you in advance.</p>
Sparse AllReduce Performance With Large GPU Procesors

Eiji911: <p>Sparse all-reduce has been implemented in <a href="https://github.com/pytorch/pytorch/pull/22036" class="inline-onebox" rel="noopener nofollow ugc">Add sparse tensor allreduce by pietern · Pull Request #22036 · pytorch/pytorch · GitHub</a>.<br>
However in our case, when the gpu cluster scaled up to several hundred workers,  high sparsification ratios still generate significant communication overheads, which even worst than DenseAllReduce。</p>
<p>Is it possible to optimize the communication volume of sparse all-reduce with a large number of workers？</p>
Wav2vec2 ParametrizedConv1d code and dimention

mi_gupta: <p>for input size [1, 1, 768] , how can we calculate the o/p of the following layers?<br>
and can anyone help to code this block in pytorch (this one belongs to wav2vec2) ?</p>
<pre><code class="lang-auto">      (pos_conv_embed): ConvolutionalPositionalEmbedding(
        (conv): ParametrizedConv1d(
          768, 768, kernel_size=(128,), stride=(1,), padding=(64,), groups=16
          (parametrizations): ModuleDict(
            (weight): ParametrizationList(
              (0): _WeightNorm()
            )
          )
        )
      )

</code></pre>
Key Truncation Issue in Checkpoint Save/Load

provebison: <p>Hi everyone,</p>
<p>I am currently facing an issue when trying to load a checkpoint after training. The problem arises from a mismatch between the keys in the <code>state_dict</code>, resulting in the following error message:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for MyLightningModule:
Missing key(s) in state_dict: "model.llm.embeddings.word_embeddings.weight",
Unexpected key(s) in state_dict: "ings.word_embeddings.weight",
</code></pre>
<p>It seems that during the checkpoint save process, only “ings” from <code>model.llm.embeddings</code> is being preserved, while the rest of the key is getting cut off.</p>
<p>Could someone help me understand why this key truncation is happening during the save operation? I would greatly appreciate any insights or guidance you could provide. Thank you very much!</p>

ptrblck: <p>I haven’t seen such an issue before pointing to missing parts of the keys.<br>
Is this the only key that shows the mismatch?</p>

provebison: <p>Thank you so much for your reply! The mismatch happens to all keys. For instance, “model.pre_classifier.weight” and “model.pre_classifier.bias” were truncated to “fier.weight” and “fier.bias,” respectively. It seems that the first 16 characters were consistently removed from all keys in the saved checkpoint.</p>
<p>This is quite strange, and I greatly appreciate your insights into why this issue is occurring. Thanks again!</p>

provebison: <p>I have identified that the issue is attributed to the <code>Ray</code> package, and I will reach out to them to seek a solution. Thank you for your help!</p>

ptrblck: <p>That’s interesting! If you have a chance, I would be really interested in the root cause as it’s indeed a weird but interesting issue.</p>

provebison: <p>It turns out that <code>RayFSDPStrategy</code> from <code>Ray</code> truncates keys in <code>state_dict</code> using the following code:</p>
<pre><code class="lang-auto">state_dict = self.model.state_dict()
prefix_len = len("_forward_module.")
return {k[prefix_len:]: v for k, v in state_dict.items()}
</code></pre>
Dataloader implementation question

JamesDickens: <p>Are the Dataloader and Dataset classes written in pure python or do they use any c++ bindings?</p>

ptrblck: <p>They should use pure Python: <a href="https://github.com/pytorch/pytorch/blob/099ff51d459462cccf83361f33b9a59ecf93842e/torch/utils/data/dataset.py"><code>dataset.py</code></a>, <a href="https://github.com/pytorch/pytorch/blob/main/torch/utils/data/dataloader.py"><code>dataloader.py</code></a>.</p>
How to use Method `nccl_use_nonblocking` From 'torch/csrc/distributed/c10d/NCCLUtils.hpp'

jyxl: <p>Hello, I want to reform send&amp;recv method to adapter my net env. When I try to call <code>nccl_use_nonblocking</code> and get error with ‘undefined reference to `c10d::nccl_use_nonblocking()’‘’.<br>
Codes as follow:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/csrc/distributed/c10d/NCCLUtils.hpp&gt;
int main()
{
    auto x = c10d::nccl_use_nonblocking();
    std::cout &lt;&lt; "x: " &lt;&lt; x &lt;&lt; "\n";
    return 0;
}
</code></pre>
<p>Results as follow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15ac370460ecb4a388eadaea708fdf5f1e6eff47.png" data-download-href="https://discuss.pytorch.org/uploads/default/15ac370460ecb4a388eadaea708fdf5f1e6eff47" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15ac370460ecb4a388eadaea708fdf5f1e6eff47_2_690x124.png" alt="image" data-base62-sha1="35IZJaZuQeU374ZtdHmZxLoN29V" width="690" height="124" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15ac370460ecb4a388eadaea708fdf5f1e6eff47_2_690x124.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15ac370460ecb4a388eadaea708fdf5f1e6eff47_2_1035x186.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15ac370460ecb4a388eadaea708fdf5f1e6eff47_2_1380x248.png 2x" data-dominant-color="292726"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2231×403 50.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Sys info:<br>
sys: 4.15.0-213-generic <span class="hashtag-raw">#224-Ubuntu</span><br>
CUDA Version: 12.1<br>
python: 3.11.4<br>
torch: 2.1.2</p>
AttributeError: module 'torch' has no attribute '_C'

Viktor_Koling: <p>File ~/Downloads/FinRL-master/newenv/lib/python3.10/site-packages/torch/_VF.py:24, in VFModule.<strong>init</strong>(self, name)<br>
22 def <strong>init</strong>(self, name):<br>
23     super().<strong>init</strong>(name)<br>
—&gt; 24     self.vf = torch._C._VariableFunctions</p>
<p>AttributeError: module ‘torch’ has no attribute ‘_C’</p>
<p>how do i solve this?</p>

ptrblck: <p>You might have multiple PyTorch installations in your current environment, so either uninstall them all and reinstall a single binary, or create a new virtual environment and install PyTorch there.</p>
How to synchronize CUDA kernels from other package in Pytorch?

B0WEN-HU: <p>Dear Community,</p>
<p>I’m working on a project and pytorch works fine, but it’s just too slow. So I rewrote the slower parts of the algorithm with numba.cuda, a jit cuda programming package. In unit tests, each part works fine. But combined, it got weird. And in Python’s debug mode, everything miraculously works fine again.</p>
<p>I think the reason for this problem is due to that, pytorch launches a new kernel before numba.cuda’s kernel has finished working. There is no way for pytorch to know when numba.cuda’s kernel has finished. In my expriment, behind numba.cuda’s kernel lunch code, I immediately clone the results of numba.cuda to tensors twice and found that the two tensors cloned from the same results are not equal, which means that the content of the address has changed between the two clones. In debug mode, everything runs slower, giving numba.cuda time to finish its calculations before pytorch starts the new kernel.</p>
<p>I tried adding numba.cuda.synchronize() to the code, but it didn’t help. I would like to know if there is any possible solution.</p>

ptrblck: <p>You might want to use <code>record_stream</code> as explained in the <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-streams">CUDA Streams section</a> of the docs.</p>

B0WEN-HU: <p>Thank you very much, that does solve the problem. I use pytorch to generate a stream and pass it to the numba in a way that the pytorch stream and numba stream can wait for synchronize.</p>
<p>This brings me to a question, previously I was using default streams in both pytorch and numba (stream address 0), why wouldn’t they execute sequentially like in one stream?</p>

ptrblck: <p>I’m unsure if both libraries are using the same stream and you should check if with a visual profiler, e.g. Nsight Systems to see which streams are used for the execution.</p>
Increase in RAM when using torchvision's FasterRCNN model possibly due to Memory leak

pkn01: <p>Hi Everyone!<br>
I have been using torchvision detection model and i see that RAM keeps increasing over a period of time and end up getting below error</p>
<p>CUDA error: an illegal memory access was encountered\nCompile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions</p>
<p>i have model hosted behind an endpoint and load it on to GPU only during init() of the fastapi service with number of workers set to 5 and during prediction i am using below code:</p>
<pre><code>def load_model(cls):
    if torch.cuda.is_available():
        logger.info("CUDA Available, Using GPU")
        device = torch.device("cuda:0")
    else:
        logger.info("CUDA NOT Available, Using CPU")
        device = torch.device("cpu")

    try:
        # Load the JSON file and get the model type
        with open(os.path.join(MODEL_DIRECTORY_PATH, MODEL_CONFIG)) as f:
            data = json.load(f)
            model_type = data.get('model_type')

        if model_type not in MODEL_TYPE_DICT:
            logger.error(f"Model type '{model_type}' is not supported.")
            raise ValueError(f"Model type '{model_type}' is not supported.")

        model_ctor = MODEL_TYPE_DICT[model_type]
        labels = data['labels']

        logger.info(f"Loading Faster R-CNN Model, type: {model_type}")
        model = model_ctor(pretrained=False, num_classes=len(labels))

        model.load_state_dict(torch.load(os.path.join(MODEL_DIRECTORY_PATH, MODEL_NAME), map_location=device))

        model = model.to(device)
        model.eval()
        logger.info("Loaded Faster R-CNN Model")
</code></pre>
<p>def perform_prediction(img: Image, model, device, input_labels):<br>
logger.info(“Started performing prediction”)<br>
width, height = img.size<br>
img_t = transform_image(img, device)<br>
with torch.no_grad():<br>
prediction = model(img_t)[0]<br>
boxes, labels, scores = process_prediction(prediction)<br>
return boxes, labels, scores</p>
<p>Pretty much the standart way of doing things<br>
i’m using pytorch version: torch = “2.0.1”<br>
torchvision = “0.15.2”</p>
<p>i have seen people reporting it earlier as well, but i didn’t get a clearer view on how to go about solving the problem. can someone who has faced this earlier help me out with their approaches</p>
Launching only a rendezvous server without local workers

Greg1: <p>Is is possible with torchrun to only start a rendezvous server with no local workers?  I have a pool of potential GPU workers, all of which can come and go, and a different, more reliable machine (without a GPU) that I’d like to use to run only the rendezvous server.</p>
<p>I tried running <code>torchrun --nproc-per-node 0</code>, to try to launch just the c10d rendezvous server on a machine, but this yielded an error.</p>
When will `torch.jit.fork` create new cuda stream?

nahso: <p>I expect there will be two cuda streams to execute these two <code>interaction</code>, but nsys shows this two <code>interactoin</code> execute serially. I wonder how <code>torch.jit.fork</code> is implemented, and when will it create two cuda stream to execute kernels?</p>
<pre><code class="lang-auto">totally_local_future = torch.jit.fork(self.interaction, data_local)
data_ghost = self.interaction(data_ghost)
data_local = torch.jit.wait(totally_local_future)
</code></pre>
Pytorch Github org grooming

bhack: <p>I see that some very popular GitHub projects have a quite nice grooming program like in the <a href="https://github.com/microsoft/vscode/wiki/Issue-Grooming" rel="noopener nofollow ugc">Vscode ecosystem</a></p>
<p>As we have accumulated a lot of old open issues (<a href="https://cauldron.io/project/7233?from_date=2023-02-15&amp;to_date=2024-02-15&amp;tab=performance-issues" rel="noopener nofollow ugc">See the graphs/stats</a>) but also some relatively old PRs (<a href="https://cauldron.io/project/7233?from_date=2023-02-15&amp;to_date=2024-02-15&amp;tab=performance-issues" rel="noopener nofollow ugc">See graphs/stats</a>)</p>
<p>I think that we could start to adopt our grooming policy for the Pytorch org repositories.</p>
<p>As an extension to this policy we could occasionally organize some “grooming hackathon” to support these activities.</p>
<p>I think a regular cleanup could also help to improve the  <a href="https://chaoss.community/kb/metric-issue-label-inclusivity/" rel="noopener nofollow ugc">issue label inclusivity (CHAOSS/LinuxFoundsation) metrics</a> as it will help to clearly define and communicate to candidate contributors on what issues we really will want to still allocate the currently limited PR review resources.</p>

albanD: <p>Hi,</p>
<p>Thanks for bringing up this topic.<br>
We indeed have been struggling to keep up with the number of issues.</p>
<p>While I do think that there are many reasons for some old issues to still exist, I do agree that we most likely have a lot of them that are not relevant anymore.</p>
<p>I agree that we would need to first align and adopt such a policy before organizing such events.<br>
The VSCode description is a good start but we would need to adapt it.</p>

bhack: <p>How generally these things, like a grooming policy, are going to be introduced to the community?<br>
Just to understand where we could inline comment a policy draft.</p>

albanD: <p>Hey!</p>
<p>I think the main steps would be:</p>
<ul>
<li>Prepare a proposal for how we want to update the general grooming guidelines for the repo. Gather feedback from maintainers on these.</li>
<li>Present the new guidelines to a Core Maintainer meeting (<a href="https://dev-discuss.pytorch.org/t/2023-q4-pytorch-core-maintainer-meeting-dec-1st-2023/1650" class="inline-onebox">2023 Q4 PyTorch Core Maintainer Meeting - Dec 1st 2023 - PyTorch Dev Discussions</a>) and get them validated</li>
<li>Socialize the general practice with all maintainers</li>
<li>(optional) Organize hackathon-like event to do more directed push on this for specific topics</li>
</ul>

bhack: <p>Are you going to kick-start point 1 internally?</p>

albanD: <p>I’m not sure I will have much time to drive this. cc <a class="mention" href="/u/suraj.pt">@suraj.pt</a> <a class="mention" href="/u/svekars">@svekars</a> <a class="mention" href="/u/kit1980">@kit1980</a> might have some ideas on who can take a look at this?</p>

bhack: <p>Is there anyone else we can get involved?</p>
Matrix ill-conditioning

imranhcrot: <p>I have derived a gradient for backward propagation which requires matrix inversion. I am using moore-penrose inverse in torch for matrix inversion.  The gradient decreases the loss to some value after that it gives an error “matrix is ill-conditioned or has too many repeated singular values”. I changed the matrix precision from float 32 to float 64 which improved the results but still after decreasing the loss to some extent it gives the matrix ill-conditioning error.</p>
<p>Can anyone help?<br>
Thanks</p>
MPS backend out of memory in MAC

Malik_Ahsan: <p>please provide me solution anyone.</p>
<p>MPS backend out of memory (MPS allocated: 5.65 GB, other allocations: 492.00 KB, max allowed: 5.10 GB). Tried to allocate 500.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system f</p>

Malik_Ahsan: <p>firstly run this command in notebook cell</p>
<p>torch.mps.set_per_process_memory_fraction(0.0)</p>
BatchNorm returning NaN values

orbitalhybridization: <p>I’m having an issue with the BatchNorm2d layer of my CNN, where the output ends up being all NaNs. I’ve narrowed this down to the fact that the variance of my previous layer (Conv2d) is 0, which causes a NaN in the norm calculation. I’ve directly calculated the variance of the Conv2d layer output so I don’t think it’s due to precision issues.</p>
<p>Things I’ve tried:</p>
<ul>
<li><strong>Changing the epsilon value (eps=1e-5).</strong> This caused the loss to explode and vary greatly over each batch.</li>
<li><strong>Removing padding from Conv2d</strong> Tried this per ChatGPT advice. My mistake for even asking the thing.</li>
<li><strong>Changing the filter size of Conv2d</strong> NaNs still output from BatchNorm layer.</li>
</ul>
<p>All of this has not fixed the issue. I’ve seen this has been brought up before, but none of the fixes are working for me.</p>
<p>Any advice would be greatly appreciated!</p>
<p>Thanks,<br>
-Orbital</p>

JamesDickens: <p>Are you using 16 bit precision?</p>

JamesDickens: <p>I would argue that the fact the variance is 0 is a problem by itself.</p>
CNN with multiple images as input

ed3513: <p>I want to have a network that has multiple images as inputs and for each image, a CNN with the same architecture but different parameters is used.<br>
Lets say that each batch has 10 images of shape 3<em>224</em>224 and i want each image to go through a CNN, with the same architecture for all images but with different parameters</p>

J_Johnson: <p>You can do two things to accomplish this:</p>
<ol>
<li>Concat the images along the channels dim.</li>
<li>Set the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">Conv2d</a> <code>groups</code> variable to 10(in this case), and the <code>in_channels</code> to 30(3 channels and 10 images).</li>
</ol>
<p>Alternatively, you could just do something like this:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, device = "cpu"):
        super().__init__()
        self.layer1 = nn.ModuleList()
        self.dropout = nn.Dropout(0.1)
        for i in range(10):
            self.layer1.append(nn.Sequential(nn.Conv2d(3, 64, 3, padding=1), self.dropout))
        self.device = device
    def forward(self, x):
        output = torch.empty((0,64,224,224), device=self.device)
        for i in range(10):
            output = torch.cat([output, self.layer1[i](x[i,...]).unsqueeze(0)])
        return output

dummy_data = torch.rand((10, 3, 224, 224))
model = CNN()
print(model(dummy_data).size())
</code></pre>

ed3513: <p>Thank you very much for your answer <a class="mention" href="/u/j_johnson">@J_Johnson</a> . What if i want to do parameter sharing between the CNN of each image?<br>
Additionally, would it be possible to have a variable number of images. Lets say that some batch could have 10 images, but some others could have 13 images</p>

J_Johnson: <p>What do you mean by “parameter sharing”?</p>
<p>If you have a variable number of images, for the “groups” option, you’ll need to pass in zeros where there are no images and set the group to be the maximum you would expect.</p>
<p>For the second option, in the init, you’ll need to define the maximum number of images coming in. In the forward pass, you’ll need to determine which of the models you want activated for each image, possibly by sending in an array into the forward pass and using something like <code>for range(len(image_index)): </code> and replace <code>i</code> with <code>image_index[i]</code>.</p>
Can't find the template while installing vsixtorch

MocoX: <p>HELP!<br>
I’ve installed the vsixtorch through the installer but meet troubles creating projects.<br>
It says that the wizard class “VSIXTorch. WizardImplementation” referenced by the vstemplate file does not exist in the assembly “VSIXTorch, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null”.</p>
Why I got bad performance when I used DDP?

_yang_kr: <p>Here is my code:</p>
<pre><code class="lang-auto">    if opt.dist_gpu:
        from torch.distributed import distributed_c10d as dist
        rank = int(os.getenv('RANK', 0))
        world_size = int(os.getenv('WORLD_SIZE', 1))
        local_rank = int(os.getenv('LOCAL_RANK', 0))

        print('rank is %d, world_size is %d'%(rank, world_size))

        torch.cuda.set_device(local_rank)
        dist.init_process_group(
            backend="nccl",
            init_method=opt.dist_url,
            world_size=world_size,
            rank=rank,
        )
        dist.barrier()
        opt.gpu_ids = range(world_size)
        print("cfg.gpu_ids:", opt.gpu_ids)
        device = torch.device(f'cuda:{local_rank}' if torch.cuda.is_available() else 'cpu')
</code></pre>
<pre><code class="lang-auto">    if opt.dist_gpu:
        train_sampler = DistributedSampler(dataset=opencood_train_dataset, shuffle=True)
        train_batch_sampler = torch.utils.data.BatchSampler(
            train_sampler, hypes['train_params']['batch_size'], drop_last=True)

        train_loader = DataLoader(opencood_train_dataset,
                                  num_workers=4,
                                  collate_fn=opencood_train_dataset.collate_batch_train,
                                  pin_memory=True,
                                  prefetch_factor=2,
                                  batch_sampler=train_batch_sampler)  

        val_sampler = DistributedSampler(dataset=opencood_validate_dataset, shuffle=False)  
        val_loader = DataLoader(opencood_validate_dataset,
                                batch_size=hypes['train_params']['batch_size'],
                                num_workers=4,
                                collate_fn=opencood_train_dataset.collate_batch_train,
                                pin_memory=True,
                                drop_last=False,
                                prefetch_factor=2,
                                sampler=val_sampler) 
</code></pre>
<pre><code class="lang-auto">    if opt.dist_gpu:
        print('distributed training')
        model = model.cuda(local_rank)
        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank], find_unused_parameters=True)
</code></pre>
<p>It’s so strange, because when I trained in a single GPU, performance is normal, but in DDP training, model is very terrible.  I don’t know why.</p>
Indexing 2D tensor using 3D tensor

Maxpim: <p>Hi,</p>
<p>I want to index a 2D tensor [N, S] with a 3D tensor [N, P, K] with indices in range [0, S-1], the output should be [N, P, K]. Somehow I am stuck and I don’t know how to do it, here is a minimal example:</p>
<pre><code class="lang-auto">index = torch.randint(0, 5, size = [100, 20, 15])
X = torch.randn(size = [100, 5])
</code></pre>

J_Johnson: <p>What is the point of your K dimension on your index? Doesn’t seem you’re using it.</p>

Maxpim: <p>sorry my bad, the output should be [N, P, K] and the indices are in the range of [0,S-1]</p>

KFrank: <p>Hi Max!</p>
<aside class="quote no-group quote-modified" data-username="Maxpim" data-post="1" data-topic="198188" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ed8c4c/48.png" class="avatar"> Maxpim:</div>
<blockquote>
<p>I want to index a 2D tensor [N, S] with a 3D tensor [N, P, K] with indices in range [0, S-1], the output should be [N, P, K].<br>
…</p>
<pre><code class="lang-auto">index = torch.randint(0, 5, size = [100, 20, 15])
X = torch.randn(size = [100, 5])
</code></pre>
</blockquote>
</aside>
<p>Based on my simplest interpretation of what you mean by “indexing,” I expect<br>
you want <code>gather()</code>.</p>
<p>Consider (with smaller tensors for convenience):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.2.1
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2024)
&gt;&gt;&gt;
&gt;&gt;&gt; index = torch.randint (0, 3, size = [4, 2, 6])
&gt;&gt;&gt; X = torch.randn (size = [4, 3])
&gt;&gt;&gt;
&gt;&gt;&gt; output = X.unsqueeze (1).expand (4, 2, 3).gather (2, index)
&gt;&gt;&gt;
&gt;&gt;&gt; index
tensor([[[2, 2, 1, 2, 0, 2],
         [1, 2, 2, 0, 2, 1]],

        [[0, 1, 0, 2, 0, 2],
         [0, 1, 2, 1, 0, 1]],

        [[0, 2, 2, 2, 0, 2],
         [2, 1, 0, 2, 2, 2]],

        [[1, 2, 1, 0, 0, 0],
         [0, 0, 2, 1, 0, 2]]])
&gt;&gt;&gt; X
tensor([[ 0.6925, -0.6998,  0.3889],
        [ 0.5154,  0.3058, -1.3593],
        [-0.2669,  0.2686, -1.0352],
        [ 0.3725, -1.0863,  0.0871]])
&gt;&gt;&gt; output
tensor([[[ 0.3889,  0.3889, -0.6998,  0.3889,  0.6925,  0.3889],
         [-0.6998,  0.3889,  0.3889,  0.6925,  0.3889, -0.6998]],

        [[ 0.5154,  0.3058,  0.5154, -1.3593,  0.5154, -1.3593],
         [ 0.5154,  0.3058, -1.3593,  0.3058,  0.5154,  0.3058]],

        [[-0.2669, -1.0352, -1.0352, -1.0352, -0.2669, -1.0352],
         [-1.0352,  0.2686, -0.2669, -1.0352, -1.0352, -1.0352]],

        [[-1.0863,  0.0871, -1.0863,  0.3725,  0.3725,  0.3725],
         [ 0.3725,  0.3725,  0.0871, -1.0863,  0.3725,  0.0871]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Compiling error with tools/build_libtorch.py

ZailiWang: <p>Hi, I’m trying to build libtorch but met with compiling errors. My steps were like:</p>
<blockquote>
<p>git clone <a href="https://github.com/pytorch/pytorch.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
cd pytorch</p>
<h1>git checkout v2.2.1 (tried with latest main and rel 2.2.1, got same error)</h1>
<p>git submodule init<br>
git submodule update --init --recursive<br>
python tools/build_libtorch.py</p>
</blockquote>
<p>and the error message is</p>
<blockquote>
<p>/home/f/anaconda3/envs/libt-build/x86_64-conda-linux-gnu/include/c++/12.3.0/bits/stl_algobase.h:431:30: error: argument 1 null where non-null expected [-Werror=nonnull]<br>
431 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);<br>
|             ~







^







~~~~~~~~~~~~~~~~~~~~~~</p>
</blockquote>
<p>The GCC is v12.3.</p>
<p>Could someone help to diagnose how to use the script to build libtorch? Many thanks.</p>

ptrblck: <p>The issue seems to be related to <a href="https://github.com/pytorch/pytorch/issues/99278">this one</a> and I guess it might be caused by GCC 12.x. You could update the issue with your reproducer.</p>
Change underlying gpu memory address of Tensor

cheng_chen2: <p>Hi, here’s the thing I want to do:<br>
When I have a tensor_1 on GPU memory of device_1, I want to manually move the underlying GPU memory (ptr_1) of tensor_1 to another GPU device, with new device pointer (ptr_2) by using cuda-python package. Then I can get a valid GPU memory pointer (ptr_2). My question is, how can I change the underlying device ptr of tensor_1? Is it possible that I detach from tensor_1, so that I can get a seperate new tensor_2, then somehow I change the memory pointer of tensor_2 from ptr_1 to ptr_2, but everything else keep unchanged? I work on Python level.</p>
Avoid loop unrolling with TorchDynamo for memory savings

michael-swan: <p>I added a loop that breaks operations into sliced steps with the hope of reducing peak memory usage but TorchDynamo is unrolling the loop such that it no longer bounds memory allocation in the way I desire. Here is a somewhat simplified example of what I’m describing:</p>
<pre data-code-wrap="python"><code class="lang-python"># Generate slices of input X and perform two linear transformations upon that slice (matmul with Wa and Wb),
# then produce an outer-product of A and B which is then cumulatively summed to the result.
# The output is the equivalent of (X @ Wa).transpose(-1, -2) @ (X @ Wb)
# however it should consume 1/num_slices as much transient memory.
@torch.compile
def sliced_matmul_chain(X, Wa, Wb, num_slices):
    Xrows, Xcols = X.shape
    Xstep = Xrows / num_slices
    _, Acols = Wa.shape
    _, Bcols = Wb.shape
    C = torch.zeros(Acols, Bcols)
    for Xoff in range(0, Xrows, Xstep):
        Xi = X[Xoff:Xoff + Xstep, :]
        A = Xi @ Wa
        B = Xi @ Wb
        C += torch.einsum("ij,ik-&gt;jk", A, B) # i =&gt; Xstep size, j =&gt; Acols size, k =&gt; Bcols size
    return C
</code></pre>
<p>This loop ends up being unrolled such that it allocates memory equivalent to the full <code>X @ Wa</code> and <code>X @ Wb</code> intermediate. Are there ways to get TorchDynamo to not do this?</p>
How can I do static-quantization by Pytroch 2.1.0

RonaldYuren: <p>Please Help me to doing static-quantization with pytorch, I didin’t see my model to be quantized.<br>
Maybe i have some problem that I can’t figure out.  Thanks a lot !!!</p>
<p>Below are the code I use:<br>
#########<br>
import torch</p>
<p>import torch.nn as nn</p>
<p>from torchvision import models</p>
<p>from torch.ao.quantization import QuantStub, DeQuantStub, prepare, convert<br>
pruned_model.load_state_dict(torch.load(weight_path),False)   <span class="hashtag-raw">#load</span> pruned_model.pt<br>
pruned_model = QuantStub()(pruned_model)<br>
pruned_model = DeQuantStub()(pruned_model)</p>
<pre><code># 
pruned_model = prepare(pruned_model, qconfig_mapping)
quantized_model = convert(pruned_model, inplace=False)

# save model's weights
torch.save(pruned_model.state_dict(), "static_quantized_original.pt") 
</code></pre>
<p>#########</p>
Gumbel-Softmax like function

Woosung: <p>Hello everyone,</p>
<p>I’m unsure if it’s appropriate to ask a mathematical question here, but I don’t have anywhere else to turn.</p>
<p>Let me explain my goal. Consider an input tensor like [1,2,3,4,5]. I aim to obtain a one-hot encoded vector of the argmax of this input tensor, which would result in [0,0,0,0,1].</p>
<p>The issue is that this process needs to be differentiable. In my research, I’ve looked into the Gumbel-Softmax and softargmax functions. From my understanding, Gumbel-Softmax does not guarantee an output of [0,0,0,0,1] as it follows the categorical distribution of the input tensor.</p>
<p>With the softargmax function, I obtained an approximate argmax index. However, after applying one-hot encoding, I ended up with NaN loss values. Is one-hot encoding not differentiable, or did I make a mistake somewhere?</p>
<p>To summarize:</p>
<ol>
<li>Are there any alternatives to Gumbel-Softmax for achieving a differentiable one-hot(argmax(input_tensor))?</li>
<li>Is one-hot encoding non-differentiable?</li>
</ol>
Assertion `srcIndex < srcSelectDimSize` failed. i am not able to resolve this

kingsman: <p>please help here i am not able to find the file where the indexing should be corrected<br>
C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1237: block: [1,0,0], thread: [96,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed. C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\at</p>

ptrblck: <p>Rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stacktrace to see which operation fails.</p>