akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
GradScaler: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead

username: Mauro_Sciancalepore
cooked: <p>Hello,</p>
<p>I am running pytorch 2.5.0 on macos and I was training a simple VAE, attempting to run it with autocast.</p>
<p>This is the error I get:</p>
<pre><code class="lang-auto">scaler.unscale_(optimizer)
  File "..../lib/python3.10/site-packages/torch/amp/grad_scaler.py", line 335, in unscale_
    inv_scale = self._scale.double().reciprocal().float()
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
</code></pre>
<p>This is the code:</p>
<pre><code class="lang-auto">from torch.amp.autocast_mode import autocast
from torch.amp.grad_scaler import GradScaler
....
optimizer = optim.AdamW(vae.parameters(), lr=1e-3, fused=device == "cuda")
scaler = GradScaler(device)
with autocast(device):
    recon, mu, log_var = vae(data)
    recon_loss = F.mse_loss(recon, data, reduction="sum")
    # KL divergence
    kl_loss = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
    # Total loss
    loss = recon_loss + kl_loss

scaler.scale(loss).backward()
scaler.unscale_(optimizer) # &lt;- breaks here
tot_norm = torch.nn.utils.clip_grad_norm_(vae.parameters(), 8.0).item()
scaler.step(optimizer)
scaler.update()
</code></pre>
<p>I assume gradscaler is not yet supported on MPS, any workarounds?</p>

username: starksm64
cooked: <p>The workaround is to patch the torch/amp/grad_scaler.py code to avoid the conversion to double. This line for inv_scale:</p>
<pre data-code-wrap="python"><code class="lang-python">        # FP32 division can be imprecise for certain compile options, so we carry out the reciprocal in FP64.
        assert self._scale is not None
        inv_scale = self._scale.double().reciprocal().float() # No good on mps
</code></pre>
<p>needs to change to avoid the scaling to double:</p>
<pre data-code-wrap="python"><code class="lang-python">        # FP32 division can be imprecise for certain compile options, so we carry out the reciprocal in FP64.
        assert self._scale is not None
        if self._scale.device.type == "mps":
            inv_scale = self._scale.reciprocal() # Leave as float32 on mps!
        else:
            inv_scale = self._scale.double().reciprocal().float()
</code></pre>
3dcnn neeed 4 value for stride?

username: m_rf
cooked: <p>there is some error when i use pyro and pytorch:</p>
<pre><code class="lang-auto">RuntimeError: 
Cell In[9], line 54
     51     # 直接使用self.conv.stride，保持3维stride
     52     stride = self.conv.stride if isinstance(self.conv.stride, tuple) else (self.conv.stride,) * 3
---&gt; 54     return F.conv3d(x, weight, bias, stride, self.conv.padding, self.conv.dilation, self.conv.groups)
     55 else:
     56     return None

RuntimeError: expected stride to be a single integer value or a list of 4 values to match the convolution dimensions, but got stride=[1, 1, 1]
</code></pre>
<p>i do not want to show the detail ,it is too long</p>

username: ptrblck
cooked: <p>Try to pass a list of <code>4</code> values as the error message points our or please post a minimal and executable code snippet to reproduce the error in case you are stuck.</p>

username: m_rf
cooked: <p>Thank you very much for your reply. I posted this question to determine if this problem is common (maybe not). If it’s less common and I don’t find another way to solve my problem, I’ll consider showing my code.</p>
Pow return 'nan' instead of expected value for second and higher order deriviative orders

username: JosueCom
cooked: <p>When defining a polynomial <code>y</code></p>
<pre data-code-wrap="python"><code class="lang-python">n = 3
t = th.zeros(10, 1).requires_grad_() 
poly = th.rand(n)
orders = th.arange(n)
y = th.sum(poly*(t**orders), dim=1, keepdim=True)
</code></pre>
<p>and calculating gradients the following way</p>
<pre data-code-wrap="python"><code class="lang-python">y2 = th.autograd.grad(y.sum(), t, create_graph=True)[0]
y3 = th.autograd.grad(y2.sum(), t, create_graph=True)[0]
</code></pre>
<p>, any gradient y3 or higher (no matter what n is) results in <code>nan</code> when t = zeros but the right value when t is any other way. For t = zeros, y and y2 are correct which are poly[0] and poly[1].</p>
<p>Does anyone know what is wrong or a work around?</p>

username: soulitzer
cooked: <p>This is kind of a known issue unfortunately due to the way autograd handles exponents + masked semantics of gradients <a href="https://github.com/pytorch/pytorch/issues/89757" class="inline-onebox" rel="noopener nofollow ugc">third-order gradient of torch.pow with tensor args and certain input returns NaN · Issue #89757 · pytorch/pytorch · GitHub</a>.</p>

username: JosueCom
cooked: <p>Any expected solution to it in the future? Do you know?</p>

username: soulitzer
cooked: <p>I don’t think anyone is actively working it unfortunately</p>
Autograd error on VIT

username: Nikhil_Dugar
cooked: <p>I am running <a href="https://github.com/msed-Ebrahimi/ARoFace/tree/main" class="inline-onebox" rel="noopener nofollow ugc">GitHub - msed-Ebrahimi/ARoFace: Official repository for "ARoFace: Alignment Robustness to Improve Low-Quality Face Recognition" ECCV24</a> repository and trying to train a VIT model and getting the following error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/workspace/ARoFace/train_v2.py", line 283, in &lt;module&gt;
    main(parser.parse_args())
  File "/workspace/ARoFace/train_v2.py", line 188, in main
    img, local_labels = adversarial_img_warping(backbone=backbone,
  File "/workspace/ARoFace/AdvWarp.py", line 66, in adversarial_img_warping
    grad_scale, grad_theta, grad_t = torch.autograd.grad(loss, [scale, theta, t])
  File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 412, in grad
    result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/opt/conda/lib/python3.10/site-packages/torch/autograd/function.py", line 289, in apply
    return user_fn(self, *args)
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 267, in backward
    raise RuntimeError(
RuntimeError: Checkpointing is not compatible with .grad() or when an `inputs` parameter is passed to .backward(). Please use .backward() and do not pass its `inputs` argument.

</code></pre>
<p>I can run the same code for resnet100 so my guess is that vit might be having some layers that is not compatible with this function</p>

username: soulitzer
cooked: <p>The error occurs because using torch.utils.checkpoint with use_reentrant=True does not work with .grad(). Can you try use_reentrant=False instead?</p>

username: Nikhil_Dugar
cooked: <p>it works but can you explain why it works?</p>

username: soulitzer
cooked: <p>Sure, the two methods of doing activation checkpoint use different mechanism underneath. The use_reentrant=True approach, as the name suggests, secretly performs another reentrant .backward() during the backward pass. If the user does something special like specify inputs= arguments in the outer backward pass, the inner backward pass today is not able to emulate that, as it always does a plain .backward().</p>
Tensors cannot 'to' other devices right

username: yuexu0312
cooked: <p>I created a tensor on ‘cuda:1’, and sent it to ‘cuda:2’, but the data changed, why?<br>
if the tensor was created on ‘cuda:0’ and was sent to ‘cuda:2’, it would not be changed.<br>
see the screenshot, can help me?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5ce697bebf0ebdb4c0a38b272cc77f43604167e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/5ce697bebf0ebdb4c0a38b272cc77f43604167e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5ce697bebf0ebdb4c0a38b272cc77f43604167e0.png" alt="image" data-base62-sha1="dfPY8B8o9Vp7Jwc42ShFKrbL6Rq" width="502" height="500" data-dominant-color="152028"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">535×532 13.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Check if IO virtualization is enabled and if disabling it helps as described <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">here</a>.</p>
Why did my customized KL divergence cause NAN?

username: JimW
cooked: <p>According to some constraint within our project, I need to rewrite KL divergence computation with basic PyTorch operations. Here is my implementation of KL divergence.<br>
<code>torch.mean(q*torch.log(q/p)</code><br>
I compared the output values of this implement with the output values from the following:<br>
<code>nn.KLDivLoss()(p.log(), q)</code><br>
And they generate the same values for my limited test set.</p>
<p>When I kept using</p>
<blockquote>
<p>nn.KLDivLoss()(p.log(), q)</p>
</blockquote>
<p>,  everything was fine.  However, when I replaced it with</p>
<blockquote>
<p>torch.mean(q*torch.log(q/p)</p>
</blockquote>
<p>Some part of my model started generating NAN after several training epochs.</p>
<p>Could someone provide some guidance how this could happen and how to fix it?</p>

username: KFrank
cooked: <p>Hi Jim!</p>
<aside class="quote no-group quote-modified" data-username="JimW" data-post="1" data-topic="214339" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimw/48/38000_2.png" class="avatar"> JimW:</div>
<blockquote>
<blockquote>
<p>torch.mean(q*torch.log(q/p)</p>
</blockquote>
<p>Some part of my model started generating NAN after several training epochs.</p>
</blockquote>
</aside>
<p>Depending on your use case, computing <code>log (q) - log (p)</code> can be more stable<br>
numerically than computing <code>log (q / p)</code>.  (They are the same mathematically,<br>
but can differ numerically.)</p>
<p>Try using <code>torch.mean (q * (q.log() - p.loq()))</code>as your KL divergence<br>
implementation.</p>
<p>As an aside, <code>p</code> would typically be a prediction, that is, the output of your model.<br>
If so – for example, if the last layer of your model is a <code>Linear</code> – the output of<br>
your model – let’s call it <code>y</code> – might naturally already be in log-space.  That is,<br>
your model might naturally be outputting <code>y = p.log()</code>.  If so, just use <code>y</code> directly<br>
(without ever explicitly converting it to <code>p = y.exp()</code>).  In this case, your KL<br>
divergence would become <code>torch.mean (q * (q.log() - y))</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
Best way to find threshold of multilabel action recognition after using MultilabelAveragePrecision

username: Edoardo_Fazzari
cooked: <p>I trained a multilabel action recognition model using pytorch and I used for <code>BCEWithLogitsLoss</code> as criterion and during evaluation I evaluated using <code>MultilabelAveragePrecision(num_labels=num_classes, average='micro'</code> ). However, evaluating with <code>MultilabelAveragePrecision</code> works only when I have the the target output available, but since I would like to test the model on new video without any label I believe I need a thresholding system to do that. How can I obtain the best threshold, such as those used by <code>MultilabelAveragePrecision</code> for giving its results? Or are there any other way to fire only some on the actions that are most likely to be present in the clip?</p>
LayerNorm with variable shapes

username: wasabi_linguist
cooked: <p>Hi, I have a CNN that accepts inputs of shape <code>(4,H,W)</code> where <code>H</code> and <code>W</code> can vary. I would like to add a LayerNorm to normalize across the first dimension with a shape of <code>4</code>. But as I don’t know what <code>H</code> and <code>W</code>, I can’t create a <code>nn.LayerNorm</code> object. Is there any way to use LayerNorms with variable input shapes?</p>

username: soulitzer
cooked: <p>Using LayerNorm seems tricky for your case because LayerNorm learns two parameters, weight and bias that depend on the normalized shape.</p>

username: wasabi_linguist
cooked: <p>Hmm yeah, it seems like I can only use layernorm without weights using <code>torch.nn.functional.layer_norm</code>. Are there any other normalizations that I can use in this usecase?</p>
Restored optimizer always report: [rank0]: AssertionError: No inf checks were recorded for this optimizer

username: AlexLuya
cooked: <p>Hello,I have restored optimizer like this:</p>
<pre><code class="lang-auto">self.optimizer.load_state_dict(checkpoint[OPTIMIZER_STATE_DICT])
self.optimizer.param_groups = checkpoint[OPTIMIZER_PARAM_GROUP]
</code></pre>
<p>and keep got error like this:</p>
<pre><code class="lang-auto">[rank0]:   self.grad_scaler.step(self.optimizer)
[rank0]:   File "/data/py_envs/lib/python3.12/site-packages/torch/amp/grad_scaler.py", line 451, in step
[rank0]:   len(optimizer_state["found_inf_per_device"]) &gt; 0
[rank0]:   AssertionError: No inf checks were recorded for this optimizer.
</code></pre>
<p>And tried all solutions by googling,but still,can’t fix this problem.</p>
<p>please help to fix this</p>
Cannot access data pointer of Tensor that doesn't have storage in DistributedDataParallel + jit

username: GeophyAI
cooked: <p>The following codes cannot work and raise an error &lt;RuntimeError: Cannot access data pointer of Tensor that doesn’t have storage&gt;. But if I delete the line <code>rnn = DistributedDataParallel(rnn, device_ids=[0], output_device=0) </code> or <code>@torch.jit.script</code> it will work. Any ideas?</p>
<pre><code class="lang-auto">import torch
from torch.nn.parallel import DistributedDataParallel
import torch.distributed as dist

dist.init_process_group(backend='nccl')

@torch.jit.script
def jit_f(x1, x2, c):
    x = x2 - x1 + c
    return x

class MyFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, *args):
        y, ctx.vjp_fn = torch.func.vjp(jit_f, *args)
        return y

    @staticmethod
    def backward(ctx, grad_output):
        grad = ctx.vjp_fn(grad_output)
        return grad

class test(torch.nn.Module):
    
    def __init__(self, ):

        super(test, self).__init__()

    def forward(self, *args):
        return MyFunction.apply(*args)

x1= torch.zeros((1, 1, 128,128), dtype=torch.float32)
x2= torch.zeros((1, 1, 128,128), dtype=torch.float32)
c = torch.tensor([1.0]).float().requires_grad_().cuda()

rnn = test().cuda()
rnn.c = torch.nn.Parameter(c)
rnn = DistributedDataParallel(rnn, device_ids=[0], output_device=0)
for i in range(5):
    x_new = rnn(x1, x2, c)
    x1, x2 = x2, x_new

x1.sum().backward()
</code></pre>
Implementing custom .to function

username: rbenke
cooked: <p>Hi, I created my own backend ( based on privateuse1 ) and as of now Im using CPU to test some functionalities.</p>
<p>Currently I stuggle to implement _to_copy for that device so that it does not make a copy of the data ( underlaying storage ) but only change device property ( create a new tensor with the same underlayng storage but different device ).</p>
<p>So far I realized I need to ‘change’ the device in three places: key_set, device_opt_ and storage_.storage_impl_.data_ptr_.device_. Is there a way I can do it? Did I missed something?</p>
<p>I was able to change the last one using unsafeGetStorageImpl()-&gt;data()-&gt;unsafe_set_device(new_device). Is there somwhere similar setter for device_opt_ as well?</p>

username: rbenke
cooked: <p>I wrote something like this:</p>
<pre><code>auto storage = self.storage(); 
auto new_data_pt = at::DataPtr(
    storage.data_ptr().get(),
    storage.data_ptr().get_context(),
    storage.data_ptr().get_deleter(),
    device.value()
);

storage.mutable_data_ptr().release_context();
storage.set_data_ptr(std::move(new_data_pt));
c10::intrusive_ptr&lt;at::TensorImpl&gt; result_impl = 
c10::make_intrusive&lt;at::TensorImpl&gt;( 
    c10::Storage(storage),
    key_set,
    self.dtype()
); 
result_impl-&gt;set_sizes_and_strides(self.sizes(), self.strides()); 
auto result = at::Tensor(result_impl); 
</code></pre>
<p>But I would rather prefer to have a unique pointer to the data instead of changing the ownership ( by doing that this code will crash when the old tensor outlive the new one).</p>

username: ups
cooked: <p>Take a look at: <a href="https://github.com/pytorch/FBGEMM/blob/fe980ab54a6e28818d81c8694b6564e7f804418b/fbgemm_gpu/src/memory_utils/memory_utils.cu#L98" class="inline-onebox" rel="noopener nofollow ugc">FBGEMM/fbgemm_gpu/src/memory_utils/memory_utils.cu at fe980ab54a6e28818d81c8694b6564e7f804418b · pytorch/FBGEMM · GitHub</a><br>
Cuda UVM memory can also be accessed from both cpu and cuda device and adding a storage indirection solved the reference counting issue.</p>
Regularization of weights without using error term question

username: Nyakov
cooked: <p>So, I am doing strange stuff. I have nn tanh layer. And problem is - magnitude of weights rising to high in training and quickly outputs start clumping between 1 and -1</p>
<p>I am reading about different regularization techniques, for example L1\L2 norm. But they implying to add weights norm to error term.</p>
<p>But I do not really have loss function…</p>
<p>I only have weights gradient (G). So my learning’s looks like W = W+G*lr</p>
<p>Question is, what can I do to prevent weights grow to much when I have only access to weights and gradient?</p>
<p>I tried different stuff with mixed results.<br>
For example divide weights in each learning step be square of L2 norm:</p>
<pre><code class="lang-auto">W_norm = torch.linalg.vector_norm(W, dim=1, keepdim=True)
W.div_(w_norm.pow(2))
</code></pre>
<p>And it working somehow (I see that weights magnitude stop constantly growing) but some outputs still got clumped.</p>
Using BatchNorm1d for standardization

username: ado_sar
cooked: <p>I want to use <code>BatchNorm1d</code> layer as standardization, i.e. simply remove the mean and divide by standard deviation from embeddings of shape <code>(N, C)</code> where <code>N</code> is the batch size and <code>C</code> is the embedding dimension. I think that setting <code>affine</code> and <code>track_running_stats</code> both to <code>False</code> should do the trick, i.e.:</p>
<pre data-code-wrap="python"><code class="lang-python">standardization = nn.BatchNorm(num_features=C, affine=False, track_running_stats=False)
</code></pre>
<p>Is this correct or I should also set momentum to 1?</p>

username: soulitzer
cooked: <p>Momentum shouldn’t matter if you aren’t tracking running stats.</p>
Turning large matrix into a lower triangular matrix

username: alexj
cooked: <p>Hi Experts,</p>
<p>I need to turn a large square matrix into a lower triangual matrix (actually, I want to zero the diagonal as well). Is there a cleaner way of doing that than keeping a mask for Hadamard product?</p>
<p>Thank you</p>

username: soulitzer
cooked: <p>Is <a href="https://pytorch.org/docs/stable/generated/torch.tril.html" class="inline-onebox" rel="noopener nofollow ugc">torch.tril — PyTorch 2.5 documentation</a> what you are looking for?</p>
PyTorch scatter max for sparse tensors?

username: MartensCedric
cooked: <p>I have the following PyTorch code</p>
<pre><code class="lang-auto">value_tensor = torch.sparse_coo_tensor(indices=query_indices.t(), values=values, size=(num_lines, img_size, img_size)).to(device=device)
value_tensor = value_tensor.to_dense()
indices = torch.arange(0, img_size * img_size).repeat(len(lines)).to(device=device)
line_tensor_flat = value_tensor.flatten()
img, _ = scatter_max(line_tensor_flat, indices, dim=0)
img = torch.reshape(img, (img_size, img_size))
</code></pre>
<p>Note the line: <code>value_tensor = value_tensor.to_dense()</code>, this is unsurprisingly slow.</p>
<p>However, I cannot figure out how to obtain the same results with a sparse tensor. The function in question calls <code>reshape</code> which is not available on sparse tensors. I’m using <a href="https://pytorch-scatter.readthedocs.io/en/1.3.0/functions/max.html" rel="noopener nofollow ugc">Scatter Max</a> but opened to using anything that works.</p>
TorchScript compiling the model to run on OpenWrt

username: Rajesh_Singh
cooked: <p>I an having Ubuntu 22.04:</p>
<pre><code class="lang-auto">$  cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
</code></pre>
<p>I compiled a C++ program on this OS as follows:</p>
<pre><code class="lang-auto">$ g++ -std=c++17 -o test_model_2 test_model_2.cpp -I/home/user-M/workspace/my-project/libtorch/include -I/home/user-M/workspace/my-project/libtorch/include/torch/csrc/api/include -L/home/user-M/workspace/my-project/libtorch/lib -Wl,-rpath,/home/user-M/workspace/my-project/libtorch/lib -ltorch -ltorch_cpu -lc10 -lstdc++ -D_GLIBCXX_USE_CXX11_ABI=0 
</code></pre>
<p>I also have OpenWrt docker container (<a href="https://hub.docker.com/layers/openwrt/rootfs/x86-64-openwrt-19.07/images/sha256-63e78fa23261324a11c81ad4d357944085760bf57da9469305592ce7924a48e9" rel="noopener nofollow ugc"><code>openwrt/rootfs:x86-64-openwrt-19.07</code></a>) running:</p>
<pre><code class="lang-auto">#  cat /etc/os-release
NAME="OpenWrt"
VERSION="19.07.10"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 19.07.10"
VERSION_ID="19.07.10"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r11427-9ce6aa9d8d"
OPENWRT_BOARD="x86/64"
OPENWRT_ARCH="x86_64"
OPENWRT_TAINTS=""
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 19.07.10 r11427-9ce6aa9d8d"
</code></pre>
<p>But it does not run in this OS:</p>
<pre><code class="lang-auto"># ./test_model_2
sh: ./test_model_2: not found
</code></pre>
<p>Both are x86 64 bit. <strong>Q1.</strong> Still why am getting this error?</p>
<p>I have another question:</p>
<p><strong>Q2.</strong> My target device has 32-bit MIPS RISC processor running OpenWrt 19.07. Does torchscript support this target platform?</p>
Feature extraction blocks

username: jawi289p
cooked: <p>How can I see which blocks have what kind of layers? like mobilenetv3 has 5 downsampling blocks (5 stages) if one want to use as a backbone. But how can I know what is in stage 1, stage 2, stage 3, stage 4, and stage 5?</p>
<p>I have extracted the features of it using the below code:</p>
<pre><code class="lang-auto">from torchvision import models

mobilenet = models.mobilenet_v3_large(pretrained=True)
features = mobilenet.features
</code></pre>
<p>Can anyone guide? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>You can <code>print</code> the corresponding layers in your script, check the implementation in its source code, or could also try to visualize the model in e.g. <code>TensorBoard</code>.<br>
I usually start by <code>print</code>ing the interesting modules and then dig into the code to see how these layers are used.</p>
Reset optimizer stats

username: AreTor
cooked: <p>What’s the easiest way to reset an optimizer stats, such as Adam’s moving averages, while keeping the same weights?</p>
<p>To make an example, suppose I have a model and I have pretrained it on a dataset using Adam. Now, I want to reset Adam’s stats and train the model on another dataset, while keeping the same parameters to be optimized. What’s the best way to reset the optimizers state, except for <code>param_groups</code>?</p>

username: ptrblck
cooked: <p>You could simply recreate the optimizer via:</p>
<pre><code class="lang-python">optimizer = torch.optim.Adam(model.parameters(), lr=lr)
</code></pre>
<p>Would that work or do I misunderstand your question?</p>

username: AreTor
cooked: <p>Unfortunately, I don’t have access to the parameters such as the learning rate. The only data I have are a the model and the optimizer, that’s why I would like to “reset” the optimizer itself. Currently what I do is the following:</p>
<pre><code class="lang-auto">self.optimizer.__setstate__({'state': defaultdict(dict)})
</code></pre>
<p>and then reuse the same optimizer. Could it be a solution?</p>

username: ptrblck
cooked: <p>If you have the <code>optimizer</code> instance, you should be able to get its attributes via:</p>
<pre><code class="lang-python">optimizer.param_groups[0]['lr']
</code></pre>
<p>Your approach might work, but I would rather like to avoid manipulating the internal states.</p>

username: AreTor
cooked: <p>Thanks for the answer, I considered also that way, however I was confused by <code>param_groups</code> and <code>defaults</code> and decided to tweak directly the state. All in all, I would consider to add a method <code>reset</code>, to be able to reuse the same optimizer. Would it make sense?</p>

username: ptrblck
cooked: <p>That sounds like a good idea.<br>
It would just reset the internal states for some optimizers or do you have anything else in mind?</p>
<p>Would you like to create this feature request on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and explain your use case a bit?<br>
Also, would you be interested on implementing this feature in case the proposal gets some positive feedback?</p>

username: AreTor
cooked: <p>Yep, I’ll create the feature request, let’s see how it evolves <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

username: Atcold
cooked: <p>Any link to this feature? I just need the exact same thing.</p>

username: ptrblck
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/37410">This</a> should be the corresponding issue, so could you please comment on it with your use case, please?</p>

username: Roy_Wang
cooked: <p>I have a similar use case, and would like to know if something like this works (almost always):</p>
<pre data-code-wrap="python"><code class="lang-python">new_optimizer = type(optimizer)(params=optimizer.param_groups, defaults=optimizer.defaults)
</code></pre>
Construct a sparse tensor while propagating gradient?

username: MartensCedric
cooked: <p>I have code similar to this I would like to make faster:</p>
<pre data-code-wrap="python"><code class="lang-python"># indices: indices of a 3d tensor
# values associated to the indices


result = torch.zeros((L, N, N))
for idx, (i,j,k) in enumerate(indices):
        mask = torch.zeros_like(result)
        mask[i][j][k] = 1.0
        img = img + mask * values[idx]
</code></pre>
<p>Now, even if I chose to make a sparse mask, I notice that each iteration runs slower. Is there a simple solution to a function that will propagate the gradient of the form <code>img = func(indices, values) </code></p>
Broadcasting vs repeating

username: sherlock.h
cooked: <pre data-code-wrap="python"><code class="lang-python"># approach 1
x = torch.randn(n, d).requires_grad_()
y = torch.randn(d)
z = x * y

# approach 2
x = torch.randn(n, d).requires_grad_()
y = torch.randn(d).repeat(n, 1)
z = x * y
</code></pre>
<p>Hey guys! In the first approach, does torch internally repeat <code>y</code> or does it efficiently implement the multiplication without extra memory (especially  in the computational graph)? are there trade-offs between the two approaches?</p>

username: soulitzer
cooked: <p>You’ll want to prefer approach 1, as it avoids materializing the broadcasted tensor.</p>
User warning about `enable_nested_tensor`

username: kylebgorman
cooked: <p><a href="https://github.com/CUNY-CL/yoyodyne" rel="noopener nofollow ugc">My project</a> has an <a href="https://github.com/CUNY-CL/yoyodyne/blob/master/yoyodyne/models/transformer.py" rel="noopener nofollow ugc">implementation of a transformer sequence-to-sequence model</a>. When loading it for the first time, we get this <code>UserWarning</code>: <code>enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True</code>. This is not a very helpful warning to me because 1) I don’t immediately see why that’s a problem and 2) there’s no call to action: what am I supposed to do about it?</p>

username: wilhelm
cooked: <p>Any news on this?<br>
I get the same warning</p>

username: intexcor
cooked: <p>set batch_first=True in nn.TransformerEncoderLayer(**kwargs)</p>

username: nleroy917
cooked: <p>Oddly, this didnt resolve my problem even though everyone online claims it does. Must be a skill issue on my part…</p>

username: soulitzer
cooked: <p>It definitely sounds like that warning can be improved… feel free to file an issue about this.</p>
Getting "Could not initialize NNPACK! Reason: Unsupported hardware." warning even though NNPACK is enabled

username: KirilloCirillo
cooked: <p>Hi everyone,</p>
<p>I am trying to deploy EasyOCR locally on a VM and when executing the <code>output = reader.readtext(image_array)</code> command I get the following warning: “Could not initialize NNPACK! Reason: Unsupported hardware.”. I am deploying in a CPU only environment, on CPUs with the AVX512 instructions enabled. When the warning is displayed the model takes a lot more time to process and triggers a Timeout. I executed the following command <code>print(torch.__config__.show())</code> to see if NNPACK is available at runtime and indeed it is. This is the output right before the inference is processed:</p>
<pre><code class="lang-auto">PyTorch built with:
    - GCC 9.3
    - C++ Version: 201703
    - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
    - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
    - OpenMP 201511 (a.k.a. OpenMP 4.5)
    - LAPACK is enabled (usually provided by MKL)
    - NNPACK is enabled
    - CPU capability usage: AVX512
    - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
</code></pre>
<p>I don’t know why it does not detect NNPACK, when pytorch is built with this capability. Any help would be greatly appreciated.</p>
<p>My environment is:</p>
<pre><code class="lang-auto">easyocr==1.7.1
torch==2.4.0
torchvision==0.19.0
</code></pre>
<p>I have looked at other posts on this forum but I have not yet found a solution to this problem.</p>
[Bug] Memory leak in C++ libtorch

username: Theophile_Champion
cooked: <p>It seems there is a bug leading to a memory leak in the (C++) libtorch.</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;
#include &lt;thread&gt;


class FrameVector {

private:
    std::vector&lt;int&gt; data;

public:
    FrameVector() {
        std::vector&lt;int&gt; data(7056);
        this-&gt;data = data;
    }
};

class FrameTensor {

private:
    torch::Tensor data;

public:
    FrameTensor() {
        this-&gt;data = torch::zeros({1, 84, 84});
    }
};

template&lt;class T&gt;
void f() {

    int capacity = 1000000;
    std::vector&lt;std::vector&lt;T&gt;&gt; frames(capacity);

    for (auto i = 0; i &lt; capacity + 1000000; i++) {
        if (i == capacity) {
            std::cout &lt;&lt; "buffer is full!" &lt;&lt; std::endl;
            std::this_thread::sleep_for(std::chrono::seconds(2));
            std::cout &lt;&lt; "restart!" &lt;&lt; std::endl;
        }
        frames[i % capacity].push_back(T());
        if (i &gt;= capacity) {
            frames[i % capacity].erase(frames[i % capacity].begin());
        }
    }

}

int main(int argc, char *argv[])
{
    f&lt;FrameTensor&gt;();  // needs 34G to fill the replay buffer, then memory increases to 60G
    f&lt;FrameVector&gt;();  // needs 34G to fill the replay buffer, then memory stay constant (as it should)
}
</code></pre>
<p>The bug only seems to occur when the <code>torch::Tensor</code> is stored in nested containers for examples:</p>
<ul>
<li><code>std::vector&lt;std::vector&lt;T&gt;&gt;</code></li>
<li><code>std::vector&lt;std::deque&lt;T&gt;&gt;</code></li>
</ul>
<p>I believe the internal counter that keep track of the number of references to the <code>torch::Tensor</code> fail to count the correct number of references. This leads the tensors memory to never be released.</p>

username: Theophile_Champion
cooked: <p>Am I correct to believe this is a libtorch bug? If so, how do I report it? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: soulitzer
cooked: <p>Just to understand the numbers better, I’m confused why:</p>
<ul>
<li>7056 (elements) x 4 (Bytes) x 1000000 (tensors) is only 28G not 34G</li>
<li>60G isn’t perfectly double 34G</li>
</ul>
<p>Let me know if I’m missing something obvious.</p>

username: Theophile_Champion
cooked: <p>Hello, thanks for your fast answer, I used htop to monitor the memory usage. Some of the memory was already used before the program started (34 - 28 = 6G of already used memory). Then, 6 + 2 * 28 = 62G total. I allocated large enough chunks for memory so that I could ignored small variation in memory usage, i.e., if the memory increases significantly over 34G, I know there is a leak.</p>
SIGILL from the precompiled old LibTorch

username: Jean_Kim
cooked: <p>I’m trying to link the precompiled libtorch_cpu v1.9.0 to an existing application which is linked to its own MKL dependency.<br>
it is built with no problem but when I try to run forward from a script inside the app, I receive a SIGILL with following trace:</p>
<pre><code class="lang-auto">#0  0x00007f6fcb0d9550 mkl_blas_def_sgemm_kernel_0_bdz (libtorch_cpu.so)
</code></pre>
<p>Two puzzling leads to the problem:</p>
<ul>
<li>I ran this app under two different machines with/without AVX512 support. And it runs ok with AVX512 support. So it appears to be related to the AVX512.</li>
<li>I tested an independent example solely with libtorch on non-AVX512 machine and it works fine.</li>
</ul>
<p>So it seems something in my original application confuses libtorch to choose a wrong AVX… but I have no clue what could cause this kind of confusion.<br>
I wonder if anyone has an idea on this</p>
<p>Thanks</p>
Pytorch 2 Export QAT is training

username: PavanMV
cooked: <p>I am working on a Yolo class object detection model and was able to do QAT in Static eager mode. With Pytorch 2.5 I am trying Pytorch 2 Export dynamic QAT, the code runs without any error, but loss and eval mAP don’t change with training epochs.</p>
<p>Here is my QAT preparation code. (Code is running on CPU). Once I receive the PT2E model from the below code training is done as usual.</p>
<pre><code class="lang-auto">    from torch.ao.quantization.quantizer.xnnpack_quantizer import XNNPACKQuantizer, get_symmetric_quantization_config
    from torch.ao.quantization.quantize_pt2e import (
        prepare_qat_pt2e,
        convert_pt2e,
    )
    from torch.export import Dim

    batch_size = Dim("batch", min=1, max=512)  # Batch size between 1 and 512
    _height = Dim("_height", min=2)  # Base height
    _width = Dim("_width", min=2)  # Base width
    gt = Dim("gt", min=1, max=120)  # Ground truth dimension

    # Adjusted dynamic dimensions (height and width are multiples of 32)
    height = 32 * _height
    width = 32 * _width

    example_inputs = (
        torch.rand(2, 3, test_size[0], test_size[1]),
        # torch.rand(2, 120, 5)
    )

    dynamic_shapes=(
        {0: batch_size, 2: height, 3: width}, 
        # {0: batch_size, 1:gt }
    )

    float_model = deepcopy(model)
    exported_model = torch.export.export_for_training(
        float_model.to('cpu'), 
        example_inputs,
        dynamic_shapes=dynamic_shapes
    ).module()
    quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())
    prepared_model = prepare_qat_pt2e(exported_model, quantizer)

    original_weights = torch.load(ckpt_file)['model']
    prepared_model.load_state_dict(original_weights, strict=False)
    return prepared_model
</code></pre>
<p>Am I missing something?</p>
Reduce time to first kernel when using CUDA graphs

username: SzymonOzog
cooked: <p>I’ve been profiling the inference stack that I’m using against <a href="https://github.com/vllm-project/vllm" rel="noopener nofollow ugc">vLLM</a> and I found that in their case after calling graph replay, the first kernel gets executed almost instantly(left), whereas in my code(right) it is towards the end of <code>graph.replay</code>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08d201b851ec9db2bfa411e7306bea547f7c42aa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/08d201b851ec9db2bfa411e7306bea547f7c42aa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08d201b851ec9db2bfa411e7306bea547f7c42aa_2_606x500.jpeg" alt="image" data-base62-sha1="1g1KWeb5iC2vd4GlVg67zOCIPsm" width="606" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08d201b851ec9db2bfa411e7306bea547f7c42aa_2_606x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08d201b851ec9db2bfa411e7306bea547f7c42aa_2_909x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08d201b851ec9db2bfa411e7306bea547f7c42aa_2_1212x1000.jpeg 2x" data-dominant-color="CDBFCB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1592×1312 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone have any insight on what might cause this big delay? Graph launch takes 1-2ms so it’s a big slowdown for me</p>

username: soulitzer
cooked: <p>The delay should just be the cuda graph kernel launch, which seems expected, e.g. the kernel executes after it has launched. One way you can get overlap (like what you see in vLLM) is to launch a different kernel before.</p>
The differences between torch.mul(A,B) and A*B

username: Zichun_Zhang
cooked: <p>Hello!<br>
Is there any possibility that when use A*B in pytorch, the operator * automatically produce dot product if A,B are not suitable size for matrix product, and produce matrix product otherwise???<br>
I am  so confuse since this change when I use sometimes</p>
<p>So!   is torch.mul(A,B) the really matrix product operation and   A*B  is the dot product operation???</p>

username: albanD
cooked: <p>Hi,</p>
<p>None of them is dot product. They both are element-wise multiplication.<br>
Dot product/matrix multiplication is done with torch.mm or torch.mv or the @ symbol in python3.</p>

username: Zichun_Zhang
cooked: <p>Wow, I really did not know such difference. Here it is, torch.mm<br>
Thx a lot!</p>
Is it safe to assume labels will be the same for two different directories using datasets.ImageFolder

username: abdul_alotaibi
cooked: <p>I have images dataset in two different directories <code>Train_images</code> and <code>Test_images</code>. Both directories are organized the same way where images are inside another directory with their associated label.</p>
<p>I am loading the images into <code>train_dataset = datasets.ImageFolder(root='path/to/dataset/Train_images')</code> and <code>test_dataset = datasets.ImageFolder(root='path/to/dataset/Test_images')</code>. I wanted to know if ImageFolder sort the directories before loading them for consistency or does some smart way for making sure that labels will be the same from the two different folders. If this is not the case, would you help me understand how the function <code>ImageFolder</code> work.</p>

username: ptrblck
cooked: <p><code>ImageFolder</code> will call into <a href="https://github.com/pytorch/vision/blob/f7b1cfa8f7e10e0c157da6e55dc6f0237397faec/torchvision/datasets/folder.py#L36-L46"><code>find_classes</code></a> to create the class labels and will sort the folders. On the same system and OS you should see the same classes. However, I think I’ve seen differences in sorting between e.g. Linux and Windows, but don’t know if this was an artifact of older Python versions.</p>

username: abdul_alotaibi
cooked: <p>Thanks for your reply!</p>
How can I prevent infinity value after switching from float32 to float16?

username: JimW
cooked: <p>I have a PyTorch script involving division ops which worked earlier for torch.float32.</p>
<pre><code class="lang-auto">A = model(X[0])
B = model(X[1])
C = A/B
</code></pre>
<p>However, when I switched to torch.float16 by doing:</p>
<pre><code class="lang-auto">model = model.half()
A = model(X[0].to(torch.float16))
B = model(X[1].to(torch.float16))
C = A/B
</code></pre>
<p>I got infinity value inside C, because one element in B is</p>
<blockquote>
<p>[5.9605e-08] dtype=torch.float16)]</p>
</blockquote>
<p>What should I do to fix this infinity issue after switch to torch.float16 ?</p>

username: soulitzer
cooked: <p>If you need more range with the same amount of bits, bfloat16 can help you here.</p>
<p>You can also use higher-precision temporarily in parts of the model that require more precision, and continue using lower-precision in parts of your  model that would benefit, e.g. in terms of speed/memory from lower precision.</p>
<p><a href="https://pytorch.org/docs/stable/amp.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.amp — PyTorch 2.5 documentation</a> is a way to automate this.</p>
Unable to import torch after update no module torch._utils

username: SpaceBearEngee
cooked: <p>Hello, I recently attempted to update my torch setup and have encountered a strange issue.</p>
<p>I have been using torch through WSL to access ROCm on my AMD graphics card. I have torch installed in a Python 3.10 virtual environment on my Ubuntu 22.04 distribution. Recently, AMD released a new driver which I decided to update to and this required me to update to torch 2.3.0. After following their update procedure I was unable to import torch in either script or interactive python sessions. The import would fail with the following error message:</p>
<pre><code class="lang-auto">...sdenv310/lib/python3.10/site-packages/torch/__init__.py", line 27, in &lt;module&gt;
    from ._utils import _import_dotted_name, classproperty
ModuleNotFoundError: No module named 'torch._utils'
</code></pre>
<p>To test if this was an issue with the torch wheel on AMD’s site I tried installing directly from PyTorch instead but I get the same result and the same thing happens when I try 2.5</p>
<p>Any idea what could be interfering with it and stopping it from finding _utils?</p>

username: SpaceBearEngee
cooked: <p>To add to this: I uninstalled from the virtual environment and repeated the install procedure found here: <a href="https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/wsl/install-pytorch.html" rel="noopener nofollow ugc">Install PyTorch for ROCm — Use ROCm on Radeon GPUs</a> for the WSL-specific distribution of PyTorch+ROCm in the global environment instead.</p>
<p>It now works <em>but only in the global environment</em> . My virtual environment is configured to load packages from the global environment, I can see torch in the pip list in the virtual environment, but trying to import torch in the virtual environment still results in the same “ModuleNotFoundError”, while working perfectly in the global environment. I don’t know what to make of this.</p>

username: SpaceBearEngee
cooked: <p>Turned out to be an issue with pip uninstall not cleaning things up correctly for some reason. After calling pip uninstall for all torch-related packages <em>again</em> , both inside and outside of the virtual environment, I was able to verify that nothing was left behind. Reinstalling in the virtual environment worked after that.</p>