akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
MPS not built or available on Apple M2

username: grumbles
cooked: <p>I’ve got the following function to check whether MPS is enabled in Pytorch on my MacBook Pro Apple M2 Max. I get the response:</p>
<blockquote>
<p>MPS is not available<br>
MPS is not built</p>
</blockquote>
<pre><code class="lang-auto">def check_mps():
    if torch.backends.mps.is_available():
        print("MPS is available")
    else:
        print("MPS is not available")

    if torch.backends.mps.is_built():
        print("MPS is built")
    else:
        print("MPS is not built")

    if torch.backends.mps.is_macos_or_newer(13, 0):
        print("MPS is macOS 13 or newer")
    else:
        print("MPS is not macOS 13 or newer")
</code></pre>
<p>I’ve already verified my OS is up-to-date, and I’m not seeing many other steps to get Pytorch running on Metal. What am I missing?</p>
RuntimeError: HalfTensor is not supported

username: HassanBinHaroon
cooked: <p>I am getting the following error in the FX Graph Mode Quantization QAT</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/fx/graph_module.py”, line 738, in call_wrapped<br>
return self._wrapped_call(self, *args, **kwargs)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/fx/graph_module.py”, line 316, in <strong>call</strong><br>
raise e<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/fx/graph_module.py”, line 303, in <strong>call</strong><br>
return super(self.cls, obj).<strong>call</strong>(*args, **kwargs)  # type: ignore[misc]<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “&lt;eval_with_key&gt;.2”, line 197, in forward<br>
activation_post_process_84 = self.activation_post_process_84(matmul);  matmul = None<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/ao/quantization/fake_quantize.py”, line 199, in forward<br>
self.activation_post_process(X.detach())<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/ao/quantization/observer.py”, line 1230, in forward<br>
self.reset_histogram(x, min_val, max_val)<br>
File “/home/HassanBinHaroon/anaconda3/envs/FX_Graph_Mode_Env/lib/python3.9/site-packages/torch/ao/quantization/observer.py”, line 1203, in reset_histogram<br>
torch.histc(<br>
RuntimeError: HalfTensor is not supported</p>
Error when implementing custom autograd.Function

username: Elling_Svee
cooked: <p>I am currently learning how to implement custom <code>autograd.Functions</code>, and for the sake of practice I try to re-write the backwards pass of the <code>torch.linalg.cholesky</code>-function. The code in the implemented backwards pass is intended to be the same as the <code>cholesky_backward</code> function at line 1939 in <a href="https://github.com/pytorch/pytorch/blob/96b30dcb25c80513769dae2a8688aec080b00117/torch/csrc/autograd/FunctionsManual.cpp" rel="noopener nofollow ugc">FunctionsManual.cpp</a>. The only difference is that i have written it in Python, and attempted to avoid in-place operations.</p>
<p>However, the code does not pass the <code>gradcheck</code>, which raises as error stating that there is a “Jacobian mismatch”. Can anyone tell me what I am doing wrong?</p>
<pre><code class="lang-auto">import torch
class CustomCholFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx, A: torch.Tensor) -&gt; torch.Tensor:
        A_cpu = A.cpu().double()

        # Perform Cholesky decomposition
        L = torch.linalg.cholesky(A_cpu, upper=False)

        # Save the lower triangular matrix for backward pass
        L_cpu = L.cpu()
        ctx.L_cpu = L_cpu

        return L_cpu.to(A.device, dtype=A.dtype)

    @staticmethod
    def backward(ctx, gL: torch.Tensor) -&gt; torch.Tensor:
        gL_cpu = gL.cpu()
        L_cpu = ctx.L_cpu.clone()

        # This code is meant to be the same as the backwards pass in the
        # autograd implementation of torch.linalg.cholesky
        upper = False  # Currently assume lower triangular
        L_ = L_cpu.adjoint() if upper else L_cpu
        gL_ = gL_cpu.adjoint() if upper else gL_cpu

        # Compute gradient of A using out-of-place operations
        gA = torch.matmul(L_.adjoint(), gL_).tril()

        gA = 0.5 * (gA + gA.tril(-1).adjoint())
        gA = torch.linalg.solve_triangular(L_.adjoint(), gA, upper=True, left=True)
        gA = torch.linalg.solve_triangular(L_, gA, upper=False, left=False)

        return gA.to(gL.device)

# Make SPD matrix
A = torch.randn(5, 5, dtype=torch.double)
A = A @ A.t() + 0.1 * torch.eye(5, dtype=torch.double)
A.requires_grad = True

# Works as intended
CustomCholFunc.apply(A)

# Raises an error
torch.autograd.gradcheck(CustomCholFunc.apply, A)
</code></pre>
How to get torchrun to run on k8s kubenetes cluster?

username: CChung
cooked: <p>I am new to this forum and also need to torchrun.  So question is: how to get  torchrun to run on k8s</p>
<p>When I try to invoke torchrun on a training script, I got an error:</p>
<p>[W socket.cpp:464] [c10d] The server socket cannot be initialized on [::]:29500 (errno: 97 - Address family not supported by protocol).<br>
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:29500 (errno: 97 - Address family not supported by protocol).</p>
<p>I tried setting the redv socket IP without any luck…<br>
So does torchrun meant to work on k8s?<br>
(BTW, I was trying to run the mistral finetune example… It uses torch run in the tutori<br>
Thx!</p>

username: ptrblck
cooked: <p>I haven’t used it, but maybe <a href="https://pytorch.org/torchx/latest/quickstart.html#Quickstart"><code>torchx</code></a> could help as it’s supporting different schedulers.</p>
Dealing with converting _native_batch_norm_legit_functional to custom language

username: Subash
cooked: <p>Hi everyone,</p>
<p>I’m currently trying to build a custom converter to translate PyTorch models into a custom language. For this, I’m utilizing the PyTorch export functionality and <code>torch.fx</code> for model tracing. But, I’m having issue with converting the function <code>_native_batch_norm_legit_functional</code> during the conversion process.</p>
<p>I’m struggling with how to map this function’s logic into my target language. I’m looking for suggestions on how to handle this conversion. Has anyone faced a similar issue or could provide any guidance on how to approach converting these types of functions?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3becacc89df78aee4422974f5831c63841b3ae6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3becacc89df78aee4422974f5831c63841b3ae6c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3becacc89df78aee4422974f5831c63841b3ae6c_2_690x64.png" alt="image" data-base62-sha1="8y7f9JIREojCrI0xqCUBS2cHFvu" width="690" height="64" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3becacc89df78aee4422974f5831c63841b3ae6c_2_690x64.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3becacc89df78aee4422974f5831c63841b3ae6c_2_1035x96.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3becacc89df78aee4422974f5831c63841b3ae6c_2_1380x128.png 2x" data-dominant-color="202124"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2970×276 87.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Need advice for solving partially observed maze environment

username: Vefery
cooked: <p>I made a custom environment in unity with simple grid-based mazes like this. The agent can only see (through vector observations, not image) around itself. It shoots rays (red in the picture) in 8 directions and gets info about what it hit (wall, exit or nothing) and the distance to the hit point. And the discovered rooms count is also fed to the observations. As for rewards, it gets -1 every step, and positive rewards upon discovering new rooms and finding exit. The goal is to explore the maze and find the exit<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45f1b0d6c1d6b78f19254da679462ce8f0376665.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/45f1b0d6c1d6b78f19254da679462ce8f0376665" title="1000493657"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45f1b0d6c1d6b78f19254da679462ce8f0376665_2_310x250.jpeg" alt="1000493657" data-base62-sha1="9YKLsOXliCZoZiIMpLNQbB9aRlr" width="310" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45f1b0d6c1d6b78f19254da679462ce8f0376665_2_310x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45f1b0d6c1d6b78f19254da679462ce8f0376665_2_465x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45f1b0d6c1d6b78f19254da679462ce8f0376665_2_620x500.jpeg 2x" data-dominant-color="E8E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1000493657</span><span class="informations">1080×869 91 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The problem is that the agent keeps sticking to walls, circling around and acting randomly essentially. I’ve tried doubling penalty for staying in one room, positive reward for high velocity and other things. What else can I try? I have full control over the environment and I’m not binded to this exact design of the agent. I’m using SAC with automatic entropy</p>
Formula in RMSNorm documentation

username: Jadon_Chan
cooked: <p>In this page, <a href="https://pytorch.org/docs/stable/generated/torch.nn.RMSNorm.html" rel="noopener nofollow ugc">torch.nn.RMSNorm</a>.<br>
There’s a formula<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47d30f8d293561345b91e402b201cbac22c4cbad.png" data-download-href="https://discuss.pytorch.org/uploads/default/47d30f8d293561345b91e402b201cbac22c4cbad" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47d30f8d293561345b91e402b201cbac22c4cbad_2_690x172.png" alt="image" data-base62-sha1="afo5WTW5NsVV0mmPiOLBb2cHp3D" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47d30f8d293561345b91e402b201cbac22c4cbad_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47d30f8d293561345b91e402b201cbac22c4cbad_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/7/47d30f8d293561345b91e402b201cbac22c4cbad.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1162×290 6.35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems that this root sign should be removed? Because there’s already a root in RMS function.</p>

username: ptrblck
cooked: <p>Thanks for pointing this out. The issue is already fixed via <a href="https://github.com/pytorch/pytorch/pull/136727">PR #136727</a> and the current <code>master</code> docs also show the updated formula as seen <a href="https://pytorch.org/docs/main/generated/torch.nn.modules.normalization.RMSNorm.html">here</a>.</p>
Questions about build customized quantizer

username: cokespace2
cooked: <p>I’m trying to build a customized quantizer, which takes a FP32 model as input and then output a quantized model, basically I just need to quantize all convolution layers, and I got few questions.</p>
<ol>
<li>Should I replace all <code>nn.Conv2d</code> modules to <code>nn.quantized.Conv2d</code> modules?</li>
<li>If I set  a <code>nn.Conv2d</code> module’s weight to dtype <code>torch.int8</code> (not <code>torch.qint8</code>) and input a tensor of dtype <code>torch.int8</code> and remove bias, does it run like a <code>nn.quantized.Conv2d</code>?</li>
</ol>

username: jerryzh168
cooked: <p>are you talking about pt2e quantization? we have a guide here: <a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2 Export Quantization — PyTorch Tutorials 2.5.0+cu124 documentation</a></p>
<ol>
<li>you don’t need to replace nn.Conv2d modules in pt2e</li>
<li>nn.quantized.Conv2d also has output quantization I think so just replacing input is not enough</li>
</ol>
Is there something like "keras.utils.to_categorical" in pytorch

username: EsamOthman
cooked: <p>Is there something like “keras.utils.to_categorical” in pytorch</p>

username: kevinzakka
cooked: <pre><code>def to_categorical(y, num_classes):
    """ 1-hot encodes a tensor """
    return np.eye(num_classes, dtype='uint8')[y]</code></pre>

username: EsamOthman
cooked: <aside class="quote no-group" data-username="kevinzakka" data-post="2" data-topic="5960">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinzakka/48/3426_2.png" class="avatar"> kevinzakka:</div>
<blockquote>
<p>classes</p>
</blockquote>
</aside>
<p>Thnx, it works great.</p>

username: MarX
cooked: <p>Is this the expected solution in pytorch to build categorical tensors?</p>

username: chrischute
cooked: <p>Better yet, you can use the code directly from Keras: <a href="https://github.com/keras-team/keras/blob/master/keras/utils/np_utils.py#L9-L37" rel="nofollow noopener">https://github.com/keras-team/keras/blob/master/keras/utils/np_utils.py#L9-L37</a></p>

username: AlexanderZhou
cooked: <p>Dude, your code is awesome! How do you think up that. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

username: Zhong
cooked: <p>amazing! I am still learning the advanced indexing.</p>

username: Purusharth_Verma
cooked: <p>How did that work? What kind of sorcery is that? How is that kind of indexing possible. Could you link me to a useful resource?</p>

username: ahmedgad
cooked: <p>This code works! y is a 1D NumPy array holding the class number of the samples.</p>
<pre><code class="lang-python">temp_outs = numpy.zeros((y.shape[0], numpy.unique(y).size), dtype=numpy.uint8)
temp_outs[numpy.arange(y.shape[0]), numpy.uint8(y)] = 1

y = temp_outs
</code></pre>

username: kuangmeng
cooked: <p>How about</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
x = torch.tensor([1, 0])
F.one_hot(x, num_classes=2)
</code></pre>
<p>?</p>

username: tiol
cooked: <p>Did you refer to any resources to get this code snippet ?</p>
What's the best way to resize tensors for alignment purpose?

username: JimW
cooked: <p>In my project, I need to feed a tensor of shape [10, 17] to a transformer layer, which means there are 10 time stamps and each time stamp is a [1x17] vector.</p>
<p>Now, I have two tensors X1=[5, 64] and X2=[5, 17], and I hope to concatenate them together into a [10, 17] which will be consumed by the transformer layer.</p>
<p>Currently, I am thinking if I can convert X1 to [5, 17].  Maybe it can be achieved by applying a torch.nn.Linear(64,17) to each column of X1, but I am not sure how to implement it nicely with torch. Could someone give me some help?</p>

username: ptrblck
cooked: <p>Using an <code>nn.Linear</code> layer could be a valid approach. You could add it as a trainable layer into your model and apply it in the forward function via:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, X1, X2):
    X1 = self.linear(X1)
    # X1.shape = [5, 17]
    x = torch.cat((X1, X2), dim=0)
    # x.shape = [10, 17]
    x = self.layer(x)
    ...
</code></pre>

username: JimW
cooked: <p>Thanks a lot. But I was actually struggling with the self.linear() layer here.<br>
I know if X1 is a 1D vector I can do this:</p>
<p>X1 = torch.rand(1, 64)<br>
self.linear = torch.nn.Linear(64,17)<br>
X1 = self.linear(X1)</p>
<p>But I am not sure how to implement it when X1 = torch.rand(5, 64)</p>

username: ptrblck
cooked: <p>In the same way as <code>dim0</code> of the input tensor is treated as the batch dimension:</p>
<pre data-code-wrap="python"><code class="lang-python">X1 = torch.rand(5, 64)
linear = torch.nn.Linear(64,17)
X1 = linear(X1)
print(X1.shape)
# torch.Size([5, 17])
</code></pre>
<p>I don’t know if you really want to stack the batch dimensions afterwards and double the batch size, but this part should work.</p>
SAC doesn't converge in gym Mountain Car environment

username: Vefery
cooked: <p>I implemented SAC accodring to this <a href="https://arxiv.org/abs/1812.05905v2" rel="noopener nofollow ugc">paper</a>. I tested it on some gym environments (Half Cheetah and Inverted Pendulum) and got very decent results. So the algorithm <em>should</em> be working. However when I try it on the Mountain Car env, I can’t get it to learn. Sometimes I get better results with average reward above 0, but no matter what the model keeps getting stuck on a plateau after ~100 gradient steps:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5af8c62aa632dd670710a15c9061f927168cad20.png" data-download-href="https://discuss.pytorch.org/uploads/default/5af8c62aa632dd670710a15c9061f927168cad20" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af8c62aa632dd670710a15c9061f927168cad20_2_345x158.png" alt="image" data-base62-sha1="cYLY2oPcpumA92PN0xasndvtzmo" width="345" height="158" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af8c62aa632dd670710a15c9061f927168cad20_2_345x158.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af8c62aa632dd670710a15c9061f927168cad20_2_517x237.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5af8c62aa632dd670710a15c9061f927168cad20_2_690x316.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">953×438 19.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My params:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb6cdc0aea8e5ea3ccbddc32f66b2e7db0923668.png" alt="image" data-base62-sha1="qK2w58FuBwHdc4mSkfE7iYVkIu4" width="112" height="119"><br>
What can be the cause? Since this is a sparse-reward environment I thought I’d need a bigger batch size. I’ve tried from 256 to 8000 but got similar results so in must be something different</p>

username: wilhelm
cooked: <p>Check, if you can get <em>that</em> environment working with another SAC implementation like stable baselines3 or <a href="https://github.com/vwxyzjn/cleanrl" rel="noopener nofollow ugc">ClearRL</a>.<br>
It one of them works, then the problem is your code.<br>
The reparametrization trick is a very important concept, that must be implemented well.<br>
Ah! I discovered that the entropy term matters A LOT. So try to implement the auto entropy as in the CleanRL code above. Setting that value manually is a pain in the a**.</p>
Quantized model and Tensorrt deployment problem

username: urania880519_Li
cooked: <p>I have trained a model using QAT and I’ve successfully converted the model into onnx format. However, it always fails because of the data type of torch.quint8 when I tried to deploy the model using tensorrt engine. Is there a way to save my model in torch.qint8?<br>
I have tried several methods like torch2trt as well. It does not work for quantized model. Did anyone know the solution to run a quantized model with tensorrt? Thanks a lot.</p>

username: jerryzh168
cooked: <p>I’d recommend to start with our new flow: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.5.0+cu124 documentation</a> that allows more flexibility than the older flows, this will produce a representation of quantized model, with q/dq ops, that could be further lowered to tensorRT, but will need some support for TensorRTQuantizer and also lowering</p>
Shared Pin Memory

username: xiaomutt
cooked: <p>I have 512GB data that can be loaded into the pin memory. The pin memory works great if I only  use 1 GPU: the pin memory is fast enough to transfer data to cuda so that the GPU is 100% busy (which is not the case when data are not pin memed) . However, when I tried to launch 4 training processes using torchrun to fully use the 4 GPUs I have on one node, this is a problem: I needed 4 copies of the 512GB data in the pin memory which I don’t have.</p>
<p>Currently, my work around is to turn off the shuffle in the dataloader, so that the dataloader of each torchrun-lauched process only loads their own share of data on-demand (so in total still 512 GB).</p>
<p>The question is: is there a way to preload the dataset into pin memory and share the pin memory among all the torchrun-launched processes?</p>
<p>Thank you!</p>

username: weifengpy
cooked: <aside class="quote no-group" data-username="xiaomutt" data-post="1" data-topic="211888">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xiaomutt/48/73370_2.png" class="avatar"> xiaomutt:</div>
<blockquote>
<p>pin memory among all the torchrun-launched processes</p>
</blockquote>
</aside>
<p>I can imagine this to be a popular usecase in dataloader. I am quite noob in data loader. In principle, each process has its own memory space and is hard to share pinned memory</p>
<p>the closest solution might be shared_memory_ in <a href="https://pytorch.org/docs/stable/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing package - torch.multiprocessing — PyTorch 2.5 documentation</a></p>
Use just now updated weights in chain rule instead of old weights

username: AlexanderShpilevoj
cooked: <p>Hi! Is it possible to use just now updated weights to forward gradients to the previous layers? I see PyTorch firstly computes all gradients and only after updates weights. Is it possible to calculate gradient for the last layer, update the weights for the last layer and use new, updated weight to forward gradient to the previous layer? Can I change something in source code and use loss.backward() and optimizer.step()? I guess no. So is the only way - doing manual backpropagation with other way to forward gradients?</p>

username: ptrblck
cooked: <p>I don’t think this is directly possible as it would be mathematically wrong and you would need to add the <code>optimizer.step()</code> weight updates directly into the backward pass and gradient computation. Could you try to write custom <code>autograd.Function</code>s with a custom <code>backward</code> definition.</p>
Reseting loss value

username: orben
cooked: <p>Hey everyone, I’m experiencing some problems training a module and I’m not sure what’s causing it. It looks like whenever an epoch ends, the average loss resets to when the training began, almost like the module doesn’t retain the training that it has done on each epoch.</p>
<p>id really appriciate any help on this.</p>
<pre><code>def run_train(self, epochs: int, cycles: int, input_size: int=0):
    self.train()
    for epoch in range(epochs):
        for cycle in range(cycles + 1):
            self.cycle_count += 1
            random_index = random.randint(0, len(self.train_data) - input_size)

            tensor_input = torch.tensor(self.train_data.iloc[random_index:random_index + input_size].values, device=self.device).type(torch.float)

            result_compare = torch.tensor(self.train_data[self.prediction_list].iloc[random_index + input_size - 1:random_index + input_size].values, device=self.device).type(torch.float)

            pred = self.forward(tensor_input)

            loss = self.loss_function(pred, result_compare)
            self.running_loss += loss

            self.optimizer.zero_grad()

            loss.backward()
            self.optimizer.step()


            if cycle % 2000 == 0:
                self.print_trainrun_stats()
                self.running_loss = 0

        self.epoch_count += 1
        self.cycle_count = 0
        torch.save(self.state_dict(), fr'D:\code\playing-with-stocks\data\created data\modules\E{epoch}_model_state.pth')

    self.epoch_count = 0
    self.cycle_count = 0
</code></pre>
<p>here is the loss prtinted out during a run as well.</p>
<p>avarge_loss:  tensor(188016.0781, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  1<br>
epoch_count:  0<br>
avarge_loss:  tensor(20398.9414, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  2001<br>
epoch_count:  0<br>
avarge_loss:  tensor(12104.1943, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  4001<br>
epoch_count:  0<br>
avarge_loss:  tensor(7121.3501, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  6001<br>
epoch_count:  0<br>
avarge_loss:  tensor(5201.2925, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  8001<br>
epoch_count:  0<br>
avarge_loss:  tensor(4557.2524, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  10001<br>
epoch_count:  0<br>
avarge_loss:  tensor(24384.3477, device=‘cuda:0’, grad_fn=)<br>
cycle_count:  1<br>
epoch_count:  1</p>
Contribution Proposal: ScatterND Implementation for PyTorch

username: aboubez1999
cooked: <p>Hey everyone, I’m Ali, a software engineer from Nuro, and we’ve been working on migrating a lot of vision models from tensorflow to pytorch. One particularly critical operation for these vision models is tf.tensor_scatter_nd_update , and its reduction counterparts (e.g. tf.tensor_scatter_nd_min, tf.tensor_scatter_nd_max , etc).</p>
<p>We’ve implemented this operation in-house and have been using it successfully for a while now. There have been a few requests for an official pytorch implementation for this operation for a while now (e.g. <a href="https://discuss.pytorch.org/t/how-to-implement-tf-scatter-nd-function-of-tensorflow-in-pytorch/18358">here</a>, <a href="https://discuss.pytorch.org/t/how-to-achieve-tf-scatter-nd-in-pytorch-project-for-copynet/4688">here</a>, etc), so we’d like to offer to set this up.</p>
<p>Ask: Any strong objections to adding this new operation? And if not, who are the best reviewers to ping on the change?</p>
<p>The implementation uses only torch native layers, supports torch.compile and torch-xla, and is highly extensible. Happy to provide more details as needed!</p>

username: ptrblck
cooked: <p>Thanks for suggesting to implement this operation!<br>
Could you create an RFE on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> describing the use cases as well as your implementation so the corresponding code owners can take a look at it, please?</p>

username: aboubez1999
cooked: <p>I’ve opened an issue here: <a href="https://github.com/pytorch/pytorch/issues/138502" class="inline-onebox" rel="noopener nofollow ugc">Contribution Proposal: ScatterND Implementation for PyTorch · Issue #138502 · pytorch/pytorch · GitHub</a></p>

username: aboubez1999
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, any suggestions on potential code owners to provide feedback? Would appreciate if you tagged them on the issue. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Issues with Training UNet Architecture Model

username: anega_maheshwari
cooked: <p>Hi all,</p>
<p>While training a UNet architecture model on a dataset, have encountered challenges with the model’s loss, which saturates at 0.15 after about 220800 iterations and remain constant till 1.5 million iterations.</p>
<p>A batch size of 10 is being used and images are being padded before training. The dataset is sufficiently large, containing hundreds of thousands of images. However, the loss saturates around 0.15 and does not improve further.</p>
<p>Interestingly, when zero-padding is avoided the images are used in their original, variable shapes, the loss decreases to around 0.10 after the same number of iterations.</p>
<p>Also experimented with different learning rates, but higher rates tend to cause the loss to explode. The learning rate that has worked best so far is 0.0001. The loss functions being used are MSE Loss and L1 Loss.</p>
<p>Any suggestions or insights would be greatly appreciated.</p>

username: amaboh
cooked: <p>The issue you’re facing is likely due to the way zero padding affects the learning process of your U-Net model. By modifying how the loss is computed and how padding is handled, you can mitigate these effects. Try to implement masking in your loss function is a practical first step that directly addresses the problem of the model learning from irrelevant padded regions.</p>

username: Soumya_Kundu
cooked: <p>Dice + CE is generally a better loss to be used for segmentation (assuming) tasks. <a href="https://github.com/JunMa11/SegLossOdyssey" class="inline-onebox" rel="noopener nofollow ugc">GitHub - JunMa11/SegLossOdyssey: A collection of loss functions for medical image segmentation</a></p>
<p>You could also use this library to train – it generally does well as a baseline. <a href="https://github.com/MIC-DKFZ/nnUNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - MIC-DKFZ/nnUNet</a></p>
Multiple digit classification loss function

username: Saket_Vempaty
cooked: <p>As title suggests I have a modified mnist dataset with multiple digits per image 0-3 and each digit can be 0-9.<br>
I have modelled it as a multi label classification problem and am using cross entropy on 9*3 labels for each sample.<br>
Is this the right way to go about?<br>
My training accuracy is going down but validation accuracy is monotonically increasing</p>
<p>I think I am supposed to do something like cross entropy among the 10 labels for each digit seperately and average them out or something<br>
But I am not able to formulate it.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> is used for multi-class classification use cases and I’m unsure how your model output and target looks exactly. For multi-label classification use cases you would commonly use <code>nn.BCEWithLogitsLoss</code> adding another “no-class” class so that your output and target would have the shape <code>[batch_size, nb_total_classes=11*3]</code> or with another dimension as <code>[batch_size, num_digits, num_classes=11]</code> where <code>num_classes = 10 digit classes + 1 "background/no-class"</code> class. Let me know if this works.</p>

username: Saket_Vempaty
cooked: <p>Since there were 3 labels to be predicted and each could be 0-9 independent of the other<br>
I used average of the cross entropy per label as the loss function and it worked.<br>
I don’t think binary cross entropy is suitable for this case.</p>
Issue using PaddleDetection's YOLOv5 model in val.py after converting to ONNX - Error with scale_factor

username: gauri_collab
cooked: <p>Hi all,</p>
<p>I’m working with a YOLOE plus model from PaddleDetection <a href="https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.7" class="inline-onebox" rel="noopener nofollow ugc">GitHub - PaddlePaddle/PaddleDetection at release/2.7</a> and trying to use its .pdparams weights in the val.py script <a href="https://github.com/ultralytics/yolov5/blob/master/val.py" class="inline-onebox" rel="noopener nofollow ugc">yolov5/val.py at master · ultralytics/yolov5 · GitHub</a>. My process involves converting the Paddle model to ONNX and then using .onnx model for validation. (I wanted to use val.py from this repo as I want to compare some of my previous results of yolov5 with the yoloe plus). However, I am encountering an issue related to scale_factor during this process.</p>
<p>Issue Details:<br>
Model: YOLOv5 from PaddleDetection (weights in .pdparams format).<br>
Error: The script throws an error related to scale_factor.</p>
<p>Environment:<br>
PaddleDetection version: 3.0.0-betal<br>
PaddlePaddle version: release/2.5<br>
Pytorch version : 1.13.1</p>
<p>Question:<br>
Is there any known compatibility issue or additional steps required when using a PaddleDetection YOLO PP E model converted to ONNX with scale_factor in the val.py script? How should I adjust the scale_factor parameter properly in the val.py scenario to ensure correct evaluation? I have been trying something making the scale factor as 1.33 for onnx models and 1 for all other models. But it seems I am running into subsequent errors while implementing this. Is there an example I can refer to?<br>
Any guidance or suggestions would be greatly appreciated!</p>
<p>Thanks in advance.</p>
What is the relationship of Distributed Key-Value Store, Point-to-point communication, Synchronous and asynchronous collective operations

username: Guo_Danding
cooked: <p>It seems that <a href="https://pytorch.org/docs/stable/distributed.html#distributed-key-value-store:~:text=that%E2%80%99s%20blocking%20you.-,Distributed%20Key%2DValue%20Store,-The%20distributed%20package" rel="noopener nofollow ugc">Distributed Key-Value Store</a>,  <a href="https://pytorch.org/docs/stable/distributed.html#point-to-point-communication:~:text=%2C%207%5D%5D" rel="noopener nofollow ugc">Point-to-point communication</a>,  <a href="https://pytorch.org/docs/stable/distributed.html#collective-functions:~:text=(output)-,Collective%20functions,-torch.distributed." rel="noopener nofollow ugc">Collective functions</a> of <strong>torch.distributed</strong> can achieve the some goal(like send a tensor from device A to device B). So what is the relationship of them. Is there some relationship like the implemention of <strong>Collective functions</strong> is based on <strong>Point-to-point communication</strong> or <strong>Point-to-point communication</strong> is based on <strong>Distributed Key-Value Store</strong>? If they are unrelated, which one can gain a better performance on <strong>GPUs distributed environment</strong>?</p>

username: fduwjj
cooked: <p>distributed key-value store is totally different from another two. The rest two are provided by comm library like NCCL, MPI or gloo. They are mostly used for tensors communication (data) while the store is mostly used for dist initialization and exchanging of some control information. For GPU, collectives and NCCL are always the to-go choice.</p>

username: Guo_Danding
cooked: <p>Thanks, I will try it!</p>
Deploying libtorch project in unity has significant CPU utilization

username: minami
cooked: <p>Hi, I have deployed my libtorch project in Unity. I noticed that the CPU utilization becomes extremely high (reaching 100%). In contrast, when running the project alone, the CPU utilization is only about 20%.</p>
<p>Then, I added the following code to my project:</p>
<pre data-code-wrap="c++"><code class="lang-c++">torch::set_num_threads(1);
</code></pre>
<p>After that, everything works fine. So, I suspect there is some kind of conflict between torch multithreading and Unity. Can someone explain the reason for this, and maybe fix the bug in torch multithreading library?</p>
How can I track the nightly version of PyTorch

username: shaoyu_young
cooked: <p>Hi, guys!</p>
<p>I know I can track every official release version of pytorch here (<a href="https://github.com/pytorch/pytorch/releases" class="inline-onebox" rel="noopener nofollow ugc">Releases · pytorch/pytorch · GitHub</a>)</p>
<p>But if I want to know the nightly version is doing (e.g., what new features are added, what bugs are fixed). I mean is there any doc of these things?</p>
<p>Thx in advance!</p>

username: ptrblck
cooked: <p>No, release notes are not created daily, but you can use <code>torch.version.git_version</code> to check the commit the nightly is based on and check the commit history on <a href="https://github.com/pytorch/pytorch/commits/main/">GitHub</a>.</p>
How to use GPU virtual memory / share memory

username: mi_zhang
cooked: <p>when I ran llama3 on rtx 3080,  the 3080 card has only 10GB memory, the model need 16GB,  it fails<br>
self.weight = Parameter(torch.Tensor(self.output_size_per_partition, self.in_features))<br>
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. …</p>
<p>However , when I ran some others like stable diffusion image generation or video generation, of which the underlying component is also pytorch.  It will use the virtual memory and can alloc much more memory than dedicate memory on the GPU card.</p>
<p>All on the windows,  why?</p>
<p>Is there any configuration?</p>
Torch.compile seems not support torch.frombuffer(binary,dtype=torch.uint8))

username: Wung_Lou
cooked: <p>File “/root/butterfly/ComfyUI/env_wl/lib/python3.11/site-packages/torch/fx/proxy.py”, line 298, in create_arg<br>
raise NotImplementedError(f"argument of type: {type(a)}")<br>
torch._dynamo.exc.InternalTorchDynamoError: argument of type: &lt;class ‘bytes’&gt;</p>
<p>from user code:<br>
File “/root/butterfly/ComfyUI/./custom_nodes/ComfyUI-Optimus/compilable/op_compilable.py”, line 73, in custom_rgb2tensor<br>
t_image = torch.frombuffer(binary,dtype=torch.uint8)</p>
With TMA v.s. Without TMA

username: yehangyang
cooked: <p>Background:<br>
I’m just curious about the performance brought by TMA in this blog: <a href="https://pytorch.org/blog/hopper-tma-unit/" class="inline-onebox" rel="noopener nofollow ugc">Deep Dive on the Hopper TMA Unit for FP8 GEMMs | PyTorch</a></p>
<p>My Problem:</p>
<ol>
<li>I’m wondering if we access GMEM data in the same granularity and block sequence as TMA does but through the thread, can we improve performance?</li>
<li>But Thread can not reach TMA’s performance according to:</li>
</ol>
<ul>
<li>the thread may generate more addresses than TMA</li>
<li>TMA accesses memory in an asynchronous way</li>
</ul>
<p>Thank you all!</p>
Extension cannot find libc10.so/libtorch_cpu.so/libtorch_python.so even after importing torch

username: 71M073J
cooked: <p>Hi, in solving my issues in <a href="https://discuss.pytorch.org/t/undefined-symbol-cpublas4gemm-when-loading-extension/203763/1">importing/compiling an extension</a> i’ve been searching around for any clue possible.<br>
I came across ldd checking the compiled .so file as maybe some libraries were missing, and lo and behold:<br>
Pytorch somehow does not agree with me…</p>
<pre><code class="lang-auto">ldd /usr/local/lib/python3.10/dist-packages/install_test1-0.0.0-py3.10-linux-x86_64.egg/install_test1.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffe012bd000)
        libc10.so =&gt; not found
        libtorch_cpu.so =&gt; not found
        libtorch_python.so =&gt; not found
        libstdc++.so.6 =&gt; /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe6cbfc1000)
        libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe6cbfa1000)
        libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe6cbd76000)
        libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe6cbc8f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe6cc22a000)
</code></pre>
<p>this was done with both JIT and setuptools methods of importing the extension, files shown below:<br>
setup.py</p>
<pre><code class="lang-auto">from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name='install_test1',
      ext_modules=[cpp_extension.CppExtension('install_test1', ['ConvolutionMM2d_.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
</code></pre>
<p>and test.py</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.functional as F
from torch.utils.cpp_extension import load

perf = load(name="install_test1", sources=["ConvolutionMM2d_.cpp"], verbose=True)
</code></pre>
<p>and the setuptools version:</p>
<pre><code class="lang-auto">import torch
import install_test1
</code></pre>
<p>both of which result in an exception of the missing symbol as per the linked post.</p>
<p>This was done both with pip-install and building pytorch from source - the same version.<br>
(yes i tried a virtual environment as well, same thing happened)<br>
I can only ask how can i convince the extension files to recognise the pytorch c libraries?</p>

username: jwillette
cooked: <p>In the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">guide</a> it says to import torch first which should take care of the missing linked libraries</p>
<pre><code class="lang-auto">In [1]: import torch
In [2]: import lltm_cpp
In [3]: lltm_cpp.forward
Out[3]: &lt;function lltm.PyCapsule.forward&gt;</code></pre>
[Question] How to perform batched autograd?

username: vasdfsa
cooked: <p>Hi all,</p>
<p>New to this forum and really look forward to your help!</p>
<p>My question is about how to perform <em>for-looped backward</em> more efficiently</p>
<p>say I have a input <code>x</code> in shape of <code>(b, d)</code> where <code>b</code> is <code>batch_size</code> and <code>d</code> is <code>dimension</code> and <code>Q</code> is a trained neural network that converts <code>d</code> to scalar real value to provide gradient for <code>x</code>, i.e. <code>-Q(z, x).backward()</code>  where <code>z</code> is another conditional value</p>
<p>Now I wanna compute gradient of each <code>x</code> in <code>b</code> independently, how could I do this in a efficient way?</p>
<p>An naive method is like:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn
import numpy as np

torch.manual_seed(11111)
np.random.seed(11111)


class Net(nn.Module):
    def __init__(self, in_feature) -&gt; None:
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_feature, 5),
            nn.ReLU(),
            nn.Linear(5, 5),
            nn.ReLU(),
            nn.Linear(5, 1),
        )

    def forward(self, x):
        return self.net(x)


net = Net(515)

z = torch.randn((10, 3, 512))
a = torch.randn((10, 3, 3))
a.grad = None
a.requires_grad = True

s = torch.cat([z, a], dim=-1)
for i in range(s.shape[0]):
    net.zero_grad()
    net(s[i]).sum((-2, -1)).backward()
_grad2 = a.grad.flatten().clone()
print(_grad2)
</code></pre>
<p>But this is of course inefficient which I hope to optimize away</p>
<p>A current method I try now is</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn
import numpy as np

torch.manual_seed(11111)
np.random.seed(11111)


class Net(nn.Module):
    def __init__(self, in_feature) -&gt; None:
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_feature, 5),
            nn.ReLU(),
            nn.Linear(5, 5),
            nn.ReLU(),
            nn.Linear(5, 1),
        )

    def forward(self, x):
        return self.net(x)


net = Net(515)

z = torch.randn((10, 3, 512))
a = torch.randn((10, 3, 3))
a.grad = None
a.requires_grad = True

s = torch.cat([z, a], dim=-1)
for i in range(s.shape[0]):
    net.zero_grad()
    net(s[i]).sum((-2, -1)).backward()
_grad2 = a.grad.flatten().clone()
print(_grad2)

net = Net(515)

z = torch.randn((10, 3, 512))
a = torch.randn((10, 3, 3))
a.grad = None
a.requires_grad = True
s = torch.cat([z, a], dim=-1)
_out = net(s).sum((-2, -1))

# Flaw method, doesn't give correct results
_grad3 = torch.autograd.grad(
    outputs=_out,
    inputs=a,
    grad_outputs=torch.ones_like(_out),
    create_graph=True,
)

_grad3 = _grad3[0].clone()

print(_grad3)
_grad3 = _grad3.flatten()
assert torch.allclose(_grad2, _grad3)
print("end")

</code></pre>
<p>But the assert fails  and <code>_grad2</code> is not close to <code>_grad3</code></p>
<p>Big thanks for any help and advice!</p>
Concatenating data samples and labels using numpy

username: Oussama_Bouldjedri
cooked: <p>Hi , I am working toward creating Trin data loader in pytorch but before I need to merge the train data and the train labels, I am working on IMU data set for which the data has this shape:<br>
Train data  numpy nd array(33104,6,128,1) and the train labels numpy nd array (33104,1) how I am supposed to concatenate the labels at the last part ? since every sample of data has the shape (6,128) ?</p>

username: Tony-Y
cooked: <p>May the IMU dataset to be used be modeled as the following?</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np
data = np.random.rand(33104,6,128,1)
labels = np.random.randint(0,2,(33104,1))
</code></pre>
<p>Do you have multiple datasets that should be concatenated?</p>

username: Oussama_Bouldjedri
cooked: <p>1/Yes the IMU data set can be modeled as you mentioned<br>
2/only the train and test<br>
Thanks</p>

username: Tony-Y
cooked: <p>You can create a dataloader as follows:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; data = np.random.rand(33104,6,128,1)
&gt;&gt;&gt; labels = np.random.randint(0,2,(33104,1))
&gt;&gt;&gt; dataset = torch.utils.data.TensorDataset(torch.from_numpy(data), torch.from_numpy(labels))
&gt;&gt;&gt; loader = torch.utils.data.DataLoader(dataset, batch_size=32)
&gt;&gt;&gt; for input, target in loader:
...     print(input.shape, target.shape)
...     break
... 
torch.Size([32, 6, 128, 1]) torch.Size([32, 1])
&gt;&gt;&gt; input.squeeze(3).shape
torch.Size([32, 6, 128])
&gt;&gt;&gt; target.squeeze(1).shape
torch.Size([32])
</code></pre>
<p>You can drop the last dimension using <a href="https://pytorch.org/docs/stable/generated/torch.squeeze.html#torch.squeeze" rel="noopener nofollow ugc">torch.squeeze</a>.</p>