akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
A document question regarding batch inference example with TorchServe

DutchBro: <p>Hey I am looking into how to use TorchServe to do batch inference.</p>
<p>The example in this document is supposed to demonstrate how to do batch inference, but it seems to have only passed one image.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56aa997b49426d295e8d63a47f12c2240b4f4d81.png" data-download-href="https://discuss.pytorch.org/uploads/default/56aa997b49426d295e8d63a47f12c2240b4f4d81" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56aa997b49426d295e8d63a47f12c2240b4f4d81_2_690x291.png" alt="image" data-base62-sha1="cmGzma8Uzzc7MU8YGOA187stkjv" width="690" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56aa997b49426d295e8d63a47f12c2240b4f4d81_2_690x291.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56aa997b49426d295e8d63a47f12c2240b4f4d81_2_1035x436.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56aa997b49426d295e8d63a47f12c2240b4f4d81_2_1380x582.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3680×1556 357 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is it an unintentional example or expected? Thank you!</p>

ptrblck: <p>I believe the example shows that single images are still passed to <code>TorchServe</code>, but a batched inference will be used if multiple requests would be received in the specified <code>max_batch_delay</code> time window. At least this is how I understand the example, but others can of course correct me.</p>
Triplet/ArcFace Validation embedding loss and accuracy plateau during training midway

Sujit_Nalawade: <p>Hi folks, I am training model for PersonReid task. Following is my training setup.</p>
<ol>
<li>Backbone - ViT s16</li>
<li>Loss function - Triplet Loss with hard mining from OpenMetric learning. Margin used is 0.5. Also tried ArcFace loss. Additionally using CrossEntropyLoss for identities classification.</li>
<li>Dataset - Custom dataset with almost 1500 identities.</li>
<li>Start LR - 3e-4. Using CosineAnnealing LR scheduler with multi gpu training setup.</li>
</ol>
<p>Following are graphs with ArcFace loss.</p>
<ol>
<li><a href="https://ibb.co/HtrF8w9" rel="noopener nofollow ugc">https://ibb.co/HtrF8w9</a></li>
<li><a href="https://ibb.co/c1S2ccK" rel="noopener nofollow ugc">https://ibb.co/c1S2ccK</a></li>
</ol>
<p>Any suggestions/feedback for improving model metrics would be highly appreciated. Thanks</p>

Sujit_Nalawade: <p>Additionally following are metrics from Triplet Loss training.</p>
<ol>
<li><a href="https://ibb.co/KKR3J6Y" class="inline-onebox" rel="noopener nofollow ugc">vits-16-training-graph1 hosted at ImgBB — ImgBB</a></li>
<li><a href="https://ibb.co/jWwDyVj" class="inline-onebox" rel="noopener nofollow ugc">vits-16-training-graph2 hosted at ImgBB — ImgBB</a></li>
</ol>
Use integrated gradient without edge attribute

dongwei: <p>Hi! I’m facing a problem: We want to use Integrated gradient(IG) to explain a GCN model. We want to generate edge_mask for each edge in the network. But this GCN model doesn’t make use of edge attribute during both training and testing.<br>
So my question is: Can we use IG to explain this model without using edge attribute? Or I can only switch to other explainers?</p>
<p>Thanks so much!</p>
How to install Cuda installer for 12.1 in windows 10

Bhishan: <p>I am trying to install cuda binary in my Windows 10 computer (Dell computer).<br>
In the past month I was able to install cuda11.7 and torch was compiled with that and it was working.</p>
<p>Now, I would like to use cuda12.1 with newer version of torch. But, when I try to install cuda 12.1, the installer file does not work.</p>
<p>Binary file: <a href="https://developer.nvidia.com/cuda-12-1-1-download-archive?target_os=Windows&amp;target_arch=x86_64&amp;target_version=10&amp;target_type=exe_local" class="inline-onebox" rel="noopener nofollow ugc">CUDA Toolkit 12.1 Update 1 Downloads | NVIDIA Developer</a></p>
<p>attempt 01:  right click and install as admin ==&gt; gives can not create folder error<br>
attempt 02: double click and run ==&gt; gives can not create folder error<br>
attempt 03: rename cudaxx.exe to cudaxxx.7zip, unzip that, run ./setup.exe opens and runs some time, but it will fail installing many components and installation can not success.</p>
<p>How to install cuda 12.1 binary file in windows computer?</p>

ptrblck: <p>Your issue seems to be unrelated to PyTorch so you might want to post it into an NVIDIA-specific discussion board.<br>
With that being said, the PyTorch binaries ship with their own CUDA dependencies and your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension. You would need to properly install an NVIDIA driver to be able to use your GPU.</p>
Can you apply transformations in Custom Loss functions

Pavan_AAKASH: <p>I have an idea for a Custom IoU Loss function. Which I implemented like shown below. I get</p>
<blockquote>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>So my question is can I actually perform these transformation on my output and compute a loss value, or am I breaking fundamental chain rules. Explanation with reasoning or sources would be lovely. Thanks</p>
<pre><code class="lang-auto">class SobelIoULoss(nn.Module):
    def __init__(self, sobel_threshold):
        super(SobelIoULoss, self).__init__()
        self.sobel_threshold = sobel_threshold
        self.sobel_kernel = torch.tensor([[[[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]]],
                                          dtype=torch.float32).to('cuda')

    def forward(self, true, pred):

        # take the cube of inputs
        true = true**3
        pred = pred**3
        true.requires_grad_()
        pred.requires_grad_()

        true_sobel = F.conv2d(true, self.sobel_kernel, padding=1)
        pred_sobel = F.conv2d(pred, self.sobel_kernel, padding=1)

        # Create binary masks for true and pred
        true_mask = (true_sobel &gt; self.sobel_threshold).float()
        pred_mask = (pred_sobel &gt; self.sobel_threshold).float()

        # Compute intersection and union
        intersection = torch.sum(true_mask * pred_mask, dim=(1, 2, 3))
        union = torch.sum(true_mask + pred_mask - true_mask * pred_mask, dim=(1, 2, 3))

        # Compute IoU
        iou = intersection / union

        # Compute loss as 1 - IoU
        loss = 1 - iou

        # Return average loss of the batch
        return torch.mean(loss)

</code></pre>

ptrblck: <p>The comparison operation:</p>
<pre data-code-wrap="python"><code class="lang-python">        # Create binary masks for true and pred
        true_mask = (true_sobel &gt; self.sobel_threshold).float()
        pred_mask = (pred_sobel &gt; self.sobel_threshold).float()
</code></pre>
<p>is detaching the tensor from the computation graph as it’s not usefully differentiable. The gradient would be zero almost everywhere and undefined or <code>Inf</code> at the rounding point.</p>

Pavan_AAKASH: <p>Is there a workaround to achieve my idea? Basically I want to use IoU loss when not dealing with rectangles where you can estimate</p>
<blockquote>
<p>x1, x2, y1, y2.</p>
</blockquote>

ptrblck: <p>You could try to use a “soft” threshold as described <a href="https://discuss.pytorch.org/t/training-with-threshold-in-pytorch/145962/3">here</a>.</p>
ONNX dynamo exporter uses custom domains

Jose_Perez-Cano: <p>I have exported the usrnet (<a href="https://github.com/cszn/USRNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - cszn/USRNet: Deep Unfolding Network for Image Super-Resolution (CVPR, 2020) (PyTorch)</a>) model to onnx using the dynamo exporter because the legacy one does not support FFT. However, the exported onnx model contains nodes with custom domains like “pkg.onnxscript.torch_lib”. When converting the onnx model to tensorrt those custom domains are not supported and throw errors. Is there a way to export a pytorch model to onnx using only the ai.onnx domain?</p>
TrOCR model running slow on Jetson Nano 

parantapa: <p>Hi, I’m using TrOCR on a Nvidia Jetson Nano, and I’m noticing that it’s taking a considerable amount of time for inference, causing the system to perform slowly. Is there a way to improve this?</p>
<pre><code class="lang-auto">import torch
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
from PIL import Image

processor = TrOCRProcessor.from_pretrained('microsoft/trocr-large-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-large-printed')

def perform_ocr(image_path):
    image = Image.open(image_path).convert("RGB")

    pixel_values = processor(images=image, return_tensors="pt").pixel_values.to(device)
    generated_ids = model.generate(pixel_values)
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

    return generated_text
</code></pre>
<p>Thanks</p>
RuntimeError: The size of tensor a (8) must match the size of tensor b (2) at non-singleton dimension 0 with BCEWithLogitsLoss or BCELoss

robsver: <p>Hi all,<br>
I have a problem with the bce loss calculation.<br>
Here is a small snippet of the code:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
input = torch.tensor([[ 0.2363,  0.1395],[-0.0900, -0.0216],[ 0.4581,  0.1376],[ 0.0994,  0.1605],[-0.0092, -0.1196],[ 0.0835, -0.0903],[-0.0234, -0.1030],[ 0.0370, -0.1433]], device='cuda', requires_grad=True)
target = torch.tensor([0, 0, 1, 1, 0, 0, 0, 1], device='cuda')
weights = [1,1]
loss_function = nn.BCEWithLogitsLoss(weight=torch.tensor(tuple(weights), dtype=torch.float)).to('cuda')
loss = loss_function(input.argmax(1).float(), target.float())
</code></pre>
<p>and then I got this error message:</p>
<blockquote>
<p><strong>RuntimeError: The size of tensor a (8) must match the size of tensor b (2) at non-singleton dimension 0</strong></p>
</blockquote>
<p>I have tried to solve it, but I keep getting different errors every time.<br>
Do you know how to work this out?</p>
<p>Thanks,<br>
Roberto</p>

soulitzer: <p>You need to pass in the raw logits (don’t take argmax) and apply one_hot encoding to your targets, e.g. the shape for both the input and the target should be [batch_size, num_classes].</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
input = torch.tensor([[ 0.2363,  0.1395],[-0.0900, -0.0216],[ 0.4581,  0.1376],[ 0.0994,  0.1605],[-0.0092, -0.1196],[ 0.0835, -0.0903],[-0.0234, -0.1030],[ 0.0370, -0.1433]], device='cpu', requires_grad=True)
target = torch.tensor([0, 0, 1, 1, 0, 0, 0, 1], device='cpu')
target = nn.functional.one_hot(target)
weights = [1,1]
loss_function = nn.BCEWithLogitsLoss(weight=torch.tensor(tuple(weights), dtype=torch.float))
loss = loss_function(input, target.float())
</code></pre>

robsver: <p>Ok it works, thank you so much <a class="mention" href="/u/soulitzer">@soulitzer</a> !</p>
Anyway to get mean and std together just like tf.nn.moments()?

cbats: <p>It’s known that torch.mean() and torch.std() can return mean and standard deviation respectively, and also the computation of standard deviation depends on its mean value.</p>
<p>In some large scale tensor, the cost of calculating mean and std is high, so my question is, if I want both mean and std, is there anyway to get them together avoiding repeatedly calculate mean value again just like tf.nn.moments()?</p>
<p>Thanks</p>

Michael_Carlon: <p>You could write your own using tensor primitives:<br>
<a href="https://www.mathsisfun.com/data/standard-deviation-formulas.html" class="onebox" target="_blank" rel="nofollow noopener">https://www.mathsisfun.com/data/standard-deviation-formulas.html</a></p>
<p>Or perhaps delve into the code and make a modified version that returns std and mean in a single pass?</p>

AlbertZeyer: <p>It’s an old topic, but it was my first Google search result when searching for “torch moments”. There is such a function now:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.std_mean.html" rel="noopener nofollow ugc"><code>torch.std_mean</code></a> returns the tuple (std, mean) containing the standard deviation and mean.</p>
<p>Similarly, there is also <a href="https://pytorch.org/docs/stable/generated/torch.var_mean.html" rel="noopener nofollow ugc"><code>torch.var_mean</code></a>.</p>
The running will become slower and slower with epoch continuing, but without error when the mlpmixer model is used

minghuisvn: <p>Recently, i trained a mlp-based model in Geforce RTX 4090 with pytorch 2.3.1 and cuda11.8. When i run this script several times simultaneously, the first runed script will strangely become more and more slower with the epoch continuing, and the others will become paused, all without error reported. However, the other models, e.g., CNN, MLP, will work well. I paste the reproduction code in gist “<a href="https://gist.github.com/minghuisvn/a266bad9525166af25e66ddc0d991ab3" class="inline-onebox" rel="noopener nofollow ugc">reproduction code for pytorch issue: "The running is paused but without error when the specific model is used" · GitHub</a>”. I am stuck in this problem two days, If anyone could tell me the reason, I would be very grateful. Thanks!</p>
<h3><a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>pytorch: 2.3.1<br>
cuda:11.8<br>
gpu: RTX 4090.<br>
Environment info collected is as followed:<br>
Collecting environment information…<br>
PyTorch version: 2.3.1+cu118<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.8<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 professional<br>
GCC version: (MinGW-W64 x86_64-msvcrt-posix-seh, built by Brecht Sanders) 13.1.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: N/A</p>
<p>Python version: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.19045-SP0<br>
Is CUDA available: True<br>
CUDA runtime version: 12.1.66<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090<br>
Nvidia driver version: 535.98<br>
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\cudnn_ops_train64_8.dll<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture=9<br>
CurrentClockSpeed=3200<br>
DeviceID=CPU0<br>
Family=207<br>
L2CacheSize=16384<br>
L2CacheSpeed=<br>
Manufacturer=GenuineIntel<br>
MaxClockSpeed=3200<br>
Name=13th Gen Intel(R) Core™ i9-13900KS<br>
ProcessorType=3<br>
Revision=</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.26.2<br>
[pip3] torch==2.3.1+cu118<br>
[pip3] torchaudio==2.3.1+cu118<br>
[pip3] torchvision==0.18.1+cu118<br>
[conda] Could not collect</p>
Floor doesn't return integers

trappmartin: <p>Hi there,<br>
I am trying to get the <code>torch.floor</code> function return the “largest integer less than or equal to each elements of <code>input</code>” as described in the documentation. However, in practice the floor function seems to return the largest integer valued float that is less than or equal to each element of the input tensor.</p>
<p>See:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn((100,100)) * 10;
&gt;&gt;&gt; torch.floor(a).dtype
torch.float32
</code></pre>
<p>A quick search showed that there has been an open issue <a href="https://github.com/pytorch/pytorch/issues/36309" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/36309</a> about this which seems to have been “resolved” by introducing an optional <code>out</code> argument.</p>
<p>However, this also doesn’t seem to work:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn((100,100)) * 10;
&gt;&gt;&gt; b = torch.zeros((100,100), dtype=torch.int);
&gt;&gt;&gt; torch.floor(a, out = b)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: Found dtype Int but expected Float
</code></pre>
<p>The only way I found to get the “largest integer less than or equal to the input” is by explicitly casting the returned float to an int. This is of course rather stupid and introduces heavy overhead that I try to prevent.</p>
<p>Alternatively, one can simply directly cast <code>a</code> to a tensor of <code>int</code> values, which works if all elements in <code>a</code> are larger than zero. But fails in general.</p>
<p>Is there a better way to do that?</p>
Geometric kernels

Sparvieroo: <p>I would like to define some sort of geometric kernels for a convolutional layer and I am wondering which is the most efficient way to do it.</p>
<p>For example, I would like to define a binary kernel whose lit pixels are all in a straight line passing for the center of the kernel. The trainable parameters are therefore the angular coefficient and the width of the line.</p>
<p>The only way I found was to use cv2.line, but I feel it’s very convoluted and not efficient. Do you know some better way?</p>
How do know if the GPU memory size applied for exceeds the total GPU memory?

w1005444804: <p>using libtorch. When exceeded total gpu memory, the program does not crash, the reason is that the CPU memory is used as gpu memory, which causes the program to be slow, how to solve it?</p>
<p>…<br>
nvmlMemory_t memoryInfo;<br>
result = nvmlDeviceGetMemoryInfo(device, &amp;memoryInfo);<br>
…<br>
When exceeded total gpu memory. The value of memoryInfo looks like everything is fine…</p>
TorchScript: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 0: unexpected end of data

kabyanil: <p>Hi, I have the following Tokenizer class which I’m trying to jit to use in c++:</p>
<pre><code class="lang-auto">class Tokenizer(jit.ScriptModule):
  def __init__(self):
    super().__init__()
    self.tokens_to_idx : Dict[str, int] = {...}
    self.idx_to_tokens : Dict[int, str] = {...}

  @jit.script_method
  def encode(self, word : str):
    word_idx : List[int] = []

    for char in word.lower():
        word_idx.append(self.tokens_to_idx[char])

    return list(word_idx)
</code></pre>
<p>I am passing unicode strings to the encode() method with the following:</p>
<pre><code class="lang-auto">tokenizer_to_jit = Tokenizer()
tokenizer_jitted = torch.jit.script(tokenizer_to_jit)
tokenizer_jitted.encode("নমস্কাৰ")
</code></pre>
<p>This produces the following output:</p>
<pre><code class="lang-auto">UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 0: unexpected end of data
</code></pre>
<p>The same code works when I pass English strings. What could be the issue and how to resolve it?</p>
How does PyTorch update each child node when using Backward?

yinjun1131: <p>This question has been confusing me for a long time. How does pytorch confirm whether each child node should increase or decrease, how does it determine the initial value of the current node, and how does it search for it using Backward</p>

ptrblck: <aside class="quote no-group" data-username="yinjun1131" data-post="1" data-topic="204468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/439d5e/48.png" class="avatar"> yinjun1131:</div>
<blockquote>
<p>How does pytorch confirm whether each child node should increase or decrease</p>
</blockquote>
</aside>
<p>The backward pass will use the chain rule to compute the gradient of the loss w.r.t. all trainable parameters. This gradient will be used to update the parameters reducing the loss.</p>
<aside class="quote no-group" data-username="yinjun1131" data-post="1" data-topic="204468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/439d5e/48.png" class="avatar"> yinjun1131:</div>
<blockquote>
<p>how does it determine the initial value of the current node</p>
</blockquote>
</aside>
<p>Trainable parameters are initialized randomly (often using specific random distributions, scaling etc.) and re updated afterwards during the training.</p>
<aside class="quote no-group" data-username="yinjun1131" data-post="1" data-topic="204468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/439d5e/48.png" class="avatar"> yinjun1131:</div>
<blockquote>
<p>and how does it search for it using Backward</p>
</blockquote>
</aside>
<p>The computation graph is created in the forward pass via Autograd and the <code>backward</code> pass uses it for its backpropagation.</p>

yinjun1131: <p>Thank you very much for your guidance!</p>
Customize backprop process

mt-clemente: <p>Hi. I am toying with network optimizers and would like to modify the backprop process in Pytorch.</p>
<p>For example, I would like to only save int8 versions of the context, and only calculate int8 gradients, whatever the nn layers or operations.</p>
<p>Is that something that can be done?</p>
<p>( Some things seem to be available to come on top of the backprop but I want to override it, as speed is important and hooking over the habitual calculations would just make no sense.)</p>

soulitzer: <p>There is a context manager that allows you to interpose on the saving of the context needed for backward.</p>
<p><a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.graph.saved_tensors_hooks" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html#torch.autograd.graph.saved_tensors_hooks</a></p>
<p>Calculating the gradients only in int8 is a little trickier, however since autograd require the output tensors during forward and gradients computed wrt those outputs to have the same dtype.</p>
For the derivation of recursive functions, how to avoid in situ modification of the tensor leading to failure of derivation

gengxingze: <pre><code class="lang-auto">import torch
import sys
from typing import Dict, Optional, List, Tuple
import time
torch.autograd.set_detect_anomaly(True)
t1 = time.time()

def p_fun(n: int, rij: torch.Tensor, r_inner: float, r_outer: float) -&gt; torch.Tensor:
    fx = torch.zeros(1,2,10,8, n+1, dtype=rij.dtype, device=rij.device)
    mask = (rij &gt; r_inner) &amp; (rij &lt; r_outer)
    x = 2 * (rij[mask] - r_inner) / (r_outer - r_inner) - 1
    fx[..., 0:1][mask] = 1
    for i in range(1, n + 1):
        if i == 1:
            fx[..., 1:2][mask] = x
        if i &gt;= 2:
            fx[..., i] = 2 * fx[..., 1] * fx[..., i-1] - fx[..., i-2]
    return fx

rij = torch.rand(1,2,10,8,1)
rij.requires_grad_(True)
Tk = p_fun(6,rij, 0,6)
mask: List[Optional[torch.Tensor]] = [torch.ones_like(Tk)]
dgn = torch.autograd.grad([Tk], [rij], grad_outputs=mask, retain_graph=True, create_graph=True)[0]
t3 = time.time()
print('t3',t3-t1)
</code></pre>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 2, 10, 8]], which is output 0 of AsStridedBackward0, is at version 7; expected version 6 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>

soulitzer: <p>There is a context manager that automatically clones the saved variable when needed to avoid precisely this error:</p>
<p><a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors</a></p>
<p>Does this work for your case?</p>
Why is ReLU a nn.Module but scaled_dot_product is not

DeerFreak: <p>Hi, I was wondering what the difference between scaled_dot_product and ReLU is that makes one callable as a module while the other is only available in the functional sub-module.<br>
Is it that even though ReLU is stateless it only gets one input, facilitating using it in sequential models? Or that the scaled_dot_product is almost always used in the MultiheadAttention module, removing the need of wrapping it inside a module.<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

soulitzer: <p>IMO there’s no strong reason ReLU needs to be a module either, though as you mention, having more things as modules means that you are able to act on things in a more uniform fashion.</p>
Extend autograd.Function to support vmap and autograd

nuull: <p>Hi,</p>
<p>I am trying to extend some <code>pytorch3d</code> classes (that are <code>autograd.Function</code> ) to support <code>jacrev</code>. I made the adjustment on C++ side and can confirm it is correct. The example below where <code>_C.point_face_dist_backward</code> is its <code>backward</code> method from <code>pytorch3d</code>. It can now take <code>grad_dists</code> as a 2D vector that supports how vmap/jacrev would call it.</p>
<pre><code class="lang-auto"># simulating torch.autograd.functional.jacobian
# to compute the jacobian row by row
grad_dists = torch.eye(points.shape[0], dtype=torch.float32, device=device)
grad_point_list = []
for grad_d in grad_dists:
    grad_points, _ = _C.point_face_dist_backward(
        points, tris, p2f_inds, grad_d, min_triangle_area
    )
    grad_point_list.append(grad_points)
grad_points = torch.stack(grad_point_list)

# simulating torch.func.jacrev
# under vmap, it will be called with an identity grad_dists
vmap_grad_dists = torch.eye(points.shape[0], dtype=torch.float32, device=device)
vmap_grad_points, _ = _C.point_face_dist_backward(
    points, tris, p2f_inds, vmap_grad_dists, min_triangle_area
)

np.testing.assert_equal(grad_points.numpy(), vmap_grad_points.numpy())
</code></pre>
<p>The problem:</p>
<ol>
<li>
<p>when this <code>backward</code> method is called with <code> torch.autograd.functional.jacobian</code> (computing the jacobian row-by-row), it returns <code>grad_points</code> with shape of N x 3. This works fine to build the final full jacobian.</p>
</li>
<li>
<p>when it is called with <code>jacrev</code>, it returns <code>grad_points</code> with shape of M x N x 3. The final jacobian is a sparse 2D vector with a shape of M x (3N). I found this <a href="https://github.com/pytorch/pytorch/blob/main/torch/_functorch/vmap.py#L221-L226" rel="noopener nofollow ugc">line</a> in <code>vmap.py</code> reshapes the gradients (from M x N x 3 to M x (3N). While the shape is correct, it for some reason duplicates all the numbers row-by-row, so the resulting jacobian is a dense 2D vector with duplications.</p>
</li>
</ol>
<p>The question:</p>
<ol>
<li>
<p>Does anyone have experience on extend autograd.Function to support vmap? The most useful sentence from the <a href="https://pytorch.org/docs/stable/notes/extending.func.html" rel="noopener nofollow ugc">doc</a> is “For example, <code>torch.func.jacrev()</code>  performs <code>vmap()</code>  over the backward pass. So if you’re only interested in using <code>torch.func.jacrev()</code>, only the <code>backward()</code> staticmethod needs to be vmappable.” but it does not have an example how to exactly do that when <code>backward</code> method is calling C++/CUDA code.</p>
</li>
<li>
<p>If there is an example or a pointer, please also let me know!</p>
</li>
</ol>
<p>Thanks!</p>

soulitzer: <p>For future reference, this is xposted and answered here <a href="https://github.com/pytorch/pytorch/issues/128020" class="inline-onebox" rel="noopener nofollow ugc">Make `torch.autograd.Function` support `vmap` · Issue #128020 · pytorch/pytorch · GitHub</a>.</p>
Multi Processing Parquet IterDataPipe

lidorb: <p>Hey,</p>
<p>I created an IterDataPipe of .PARQUET files.<br>
My train data consists from a few parquet files with more than 100k rows each.<br>
Since I can’t read all the data at once, I iterate over each file and load chunks.</p>
<p>The datapipe is used with Lightning DataModule. The code I wrote based on the documentation:</p>
<pre><code class="lang-auto">@functional_datapipe("load_parquet_batch")
class ParquetLoaderIterDataPipe(IterDataPipe):
    def __init__(self, source_dp: IterDataPipe, columns: List[str], batch_size: int):
        self.source_dp = source_dp
        self.columns = columns
        self.batch_size = batch_size
        self.n_samples = self._get_num_samples()

    def _get_num_samples(self) -&gt; int:
        return sum([parquet.ParquetFile(path).metadata.num_rows for path in self.source_dp])

    def __iter__(self):
        for path in self.source_dp:
            parquet_file = parquet.ParquetFile(path)
            for batch in parquet_file.iter_batches(batch_size=self.batch_size, columns=self.columns):
                for _, sample in batch.to_pandas().iterrows():
                    yield sample

    def __len__(self):
        if self.n_samples == 0:
            raise TypeError # This error will be caught and the training will continue without a progress bar
        return self.n_samples


datapipe = dp.iter.FileLister("/path/to/parquet/files/", masks="*.parquet")
datapipe = datapipe.shuffle()  # shuffling the files
datapipe = datapipe.load_parquet_batch(COLS_LIST, batch_size=BATCH_SIZE)
datapipe = datapipe.shuffle(buffer_size=BUFFER_SIZE)  # shuffling the samples
datapipe = datapipe.sharding_filter()
</code></pre>
<p>I still have a few problems when trying to use multi processing (num_workers &gt; 1):</p>
<ol>
<li>
<p>Using <code>sharding_filter</code> means if there are N workers, then each of them first loads a copy of the entire data. Then it drops most of the data, and keeps a portion of size 1/N. This process is pretty inefficient in both time &amp; memory.<br>
Is there a way to prevent that overhead? Maybe allocating each file to a different worker?</p>
</li>
<li>
<p>I’m getting the following warning:<br>
<code>UserWarning: Your IterableDataset has __len__ defined. In combination with multi-process data loading (when num_workers &gt; 1), __len__ could be inaccurate if each worker is not configured independently to avoid having duplicate data.</code><br>
In case of inaccurate <code>__len__</code> the training will be crashed because of my early stopping which expects to have the validation metrics - The lightning trainer won’t start the validation as he should.<br>
Is it related to the first issue? How can I compute the exact actual length (excluding the dropped incomplete batches)?</p>
</li>
</ol>
<p>Thanks in advance.</p>

shahuzi: <p>Hi there. I have the same problem too, do you have any solutions?</p>
<p>For problem 1, I chose to load different parquet files for different workers, it works well in traning stage. But in valdation stage, the GPU will block after a while, I suspect this is because different files have different numbers of rows.</p>
How to consider multiple losses with priority ranking?

JosueCom: <p>I am training an PINN where I have initial, final, and transient conditions (turned into losses). I am training the model with a net <code>loss = loss_initial + loss_final + loss_transient</code>. How do I give priority to a loss for backpropagation? For example, scaling higher the contribution of one loss in the gradient with respect to the weights. Scaling the individual loss seems to have negligible effects.</p>

soulitzer: <p>Scaling an individual loss by some factor, e.g. loss = a * loss1 + b * loss2 should scale the gradient  backpropagated for that branch proportionally.</p>
Concurrent P2P operation (i.e., send and recv) fail

zizhao.mo: <p>Hi,</p>
<p>I have a four-GPU group where all GPUs form a process group in torch. GPU0 and GPU1 and simultaneously send data to GPU2 and GPU3 respectively. However, only the communication between GPU0 and GPU2 is ok, the communication between GPU1 and GPU3 always fails.</p>
<p>The traceback is as follows:</p>
<p>File “/tmp/ray/sesion 2024-86-1 20-30-85 58847 7961/runtime resourceshorking dir files/ raypkg 72b6f248418639b8/profiling/profilingpy”, line 129, in nehork profile secondar<br>
dist.recv(recv data, src=src rank)<br>
File “/home/mzz/miniconda3/envs/hetis/lib/python3.8/site-packages/torch/distributed/c10d logger.py”, line 47, in wrappe<br>
return func(*args,**kwargs)<br>
File “/home/mzz/miniconda3/envs/hetis/lib/python3.8/site-packages/torch/distributed/distributed c1ed.py”, line 1648, in recu<br>
pg.recv([tensorl,src,tag).wait()<br>
torch.distributed.Distbackenderon: NcL eror in: …/torch/csnc/distributed/cdled/procesGroupcl.cp:13, intemnal eror - please report this isue to the icl dewvelopers, Ncl version 2.18.1<br>
ncclInternalError:Internal check failed.<br>
Last error:<br>
Socket recv failed while polling for opId=0x7fa840084e30</p>
<p>I would like to know if it is because torch/nccl restricts that only one p2p communication is allowed at any time?</p>
<p>BR</p>

ptrblck: <p>Could you rerun the code via <code>NCCL_DEBUG=INFO</code> and post the logs here?</p>

zizhao.mo: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3ded45564da5b83243a92b868bd862e92fff9c5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ded45564da5b83243a92b868bd862e92fff9c5e" title="0fb7662c402785df150ab48f0529d5c"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3ded45564da5b83243a92b868bd862e92fff9c5e.png" alt="0fb7662c402785df150ab48f0529d5c" data-base62-sha1="8PPtwiGYObr0ToXaN70VO6GZq46" width="690" height="277" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0fb7662c402785df150ab48f0529d5c</span><span class="informations">1587×639 35.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi,</p>
<p>thanks for your reply. Here are the screenshot of the traceback. Could you give me some hint?</p>
<p>BR</p>

ptrblck: <p>I don’t see any NCCL logs in your screenshot so don’t know what’s causing the issue. It’s also always better to post formatted code instead of screenshots.</p>

zizhao.mo: <p>Hi,</p>
<p>I have solved this problem. The problem is related to the unknown NIC information of P2P peer. Since I use ray to init several processes, each newly created process cannot have the environment variables I set in the main process. Manually setting the NCCL_SOCKET_IFNAME in each process solve my problem.</p>
<p>Thanks.</p>
Undefined symbol torch8autograd4Node4name for custom C++ module

soulslicer: <p>I wrote a custom module extension in C++, and compiled it with Pybind. However, when I load it in Python i get a missing symbol error:<br>
<code>_ZNK5torch8autograd4Node4nameB5cxx11Ev</code></p>
<pre><code class="lang-auto">// Inherit from Function
class LinearFunction : public Function&lt;LinearFunction&gt; {
 public:
  // Note that both forward and backward are static functions

  // bias is an optional argument
  static torch::Tensor forward(
      AutogradContext *ctx, torch::Tensor input, torch::Tensor weight, torch::Tensor bias = torch::Tensor()) {
    ctx-&gt;save_for_backward({input, weight, bias});
    auto output = input.mm(weight.t());
    if (bias.defined()) {
      output += bias.unsqueeze(0).expand_as(output); 
    }
    return output;
  }

  static tensor_list backward(AutogradContext *ctx, tensor_list grad_outputs) {
    auto saved = ctx-&gt;get_saved_variables();
    auto input = saved[0];
    auto weight = saved[1];
    auto bias = saved[2];

    auto grad_output = grad_outputs[0];
    auto grad_input = grad_output.mm(weight);
    auto grad_weight = grad_output.t().mm(input);
    auto grad_bias = torch::Tensor();
    if (bias.defined()) {
      grad_bias = grad_output.sum(0);
    }

    return {grad_input, grad_weight, grad_bias};
  }
};


void testing(torch::Tensor x, torch::Tensor weight, torch::Tensor bias)
{
  auto y = LinearFunction::apply(x, weight);
  y.sum().backward();
}


PYBIND11_MODULE(libModule, m)
{
  m.def("testing", &amp;testing, "testing");
}
</code></pre>
<p>The error only occurs if I have <code>LinearFunction::apply(x, weight);</code> not commented out.</p>
<p>I have the following modules I am linking against:</p>
<pre><code class="lang-auto">        "lib/libtorch.so",
        "lib/libtorch_cpu.so",
        "lib/libc10.so",
        "lib/libgomp-75eea7e8.so.1",
        "lib/libtorch_python.so",
        "lib/libshm.so",
</code></pre>
<p>Everything works fine if i comment out <code>LinearFunction::apply</code> but if I enable it I get that missing symbol error <code>_ZNK5torch8autograd4Node4nameB5cxx11Ev</code></p>

Willyzw: <p>I encountered the same issue, have you found a fix or workaround? Thanks a lot in advance!</p>
Using FSDP to infer an image that doesn't fit on a single GPU

cporrasn: <p>I have an already trained Unet model, which has not been trained in a distributed environment. I would like to wrap that model in an FSDP so that it can perform an inference on a 4k image that does not fit on a single GPU.</p>
<p>I have doubts if FSDP actually fragments the model on the GPUs so that it can partition this image and perform full inference.</p>
<p>I ran the example at <a href="https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Fully Sharded Data Parallel(FSDP) — PyTorch Tutorials 2.3.0+cu121 documentation</a> but it gives me the impression that it performs the inference twice and not just once.</p>

sally2: <p>Hey there,<br>
It seems like you want to use FSDP to perform inference on a large 4k image using your Unet model. FSDP distributes model parameters across GPUs, but you’re not sure if it can split the image for full inference.<br>
The example you ran may not be doing what you need. FSDP is good for training but might need customization for inference on large images.<br>
You might want to explore other methods for distributed inference or ask for advice from the PyTorch community.</p>

ptrblck: <aside class="quote no-group" data-username="sally2" data-post="2" data-topic="204418">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sally2/48/70596_2.png" class="avatar"> sally2:</div>
<blockquote>
<p>FSDP is good for training but might need customization for inference on large images.</p>
</blockquote>
</aside>
<p>Could you describe which customizations are needed for inference?</p>

cporrasn: <p>Yes, please!! I need an example. When I use FSDP with 2 process, the model was executed two times.</p>
How to send tensor including grad_fn to different process

randomuserT: <p>When sending and receiving a Tensor via the send and recv functions, I find the grad_fn to be lost upon receival. Is there any way to retain the grad_fn when sending Tensors back and forth?</p>
<p>Example code that I’m using would be</p>
<pre><code class="lang-auto">import os

import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank):
    device = torch.device("cpu")

    tensor = torch.zeros(1, requires_grad=True)

    if rank == 0:
        tensor = tensor.mean()
        tensor = tensor.to(device)
        dist.send(tensor=tensor, dst=1)
        print("sent")
    else:
        # Receive tensor from process 0
        tensor = tensor.to(device)
        dist.recv(tensor=tensor, src=0)
        print("received tensor with grad_fn {}".format(tensor.grad_fn))

def init_process(rank, size, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    run(rank)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>I have a use case for which I need to split a model into 2 chunks, such that a forward pass starts at a client-side and is continued on a server-side. The predictions are then sent to the client, which will have to compute the loss function, which should subsequently be sent back to the server, performing backward propagation on the server-side using this received loss.</p>
<p>Now the issue is that because the grad_fn is lost during activations, I am unable to obtain the gradients during both the forward and backward passes.</p>
Controller Emulator: train chained pytorch models

dxx255: <p>Hello,<br>
I am trying to train a controller network on a pretrained emulator network<br>
(controller → emulator → output). The process is similar to <a href="https://discuss.pytorch.org/t/how-to-train-only-one-model-in-multi-models/68880" class="inline-onebox">How to train only one model in multi models?</a>, however, i do not want to update the emulator during backpropagation. (But i do need the gradients through the emulator to train the controller)</p>
<p>Since the emulator contains an lstm I cannot train the controller with emulator.eval() because of the following error: cudnn RNN backward can only be called in training mode</p>
<p>What can I do about that? Is the network altered if its parameters are not specified in optim.Adam(…)?</p>

ptrblck: <p>You could disable cuDNN globally via <code>torch.backends.cudnn.enabled = False</code> or only for the RNN module raising this error via the context manager:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.backends.cudnn.flags(enabled=False):
    ...
</code></pre>
Morphological Distance Transform on Pytorch

ZombaSY: <p>I want to define function “scipy.ndimage.morphology.distance_transform_edt” in tensor computation.<br>
It can be replaced by using “torch.from_numpy()” and above function, but it’s too slow for traning state.<br>
Is there any implementation or replacement for this computation?</p>
<p>The visualization of function process<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fd1cee98831719434d6c235eaade3d6eeb60eec.png" alt="image" data-base62-sha1="ieK9lS8hDXtWgBbBDXHXwKrrwZe" width="517" height="211"></p>
<p>shortcut for Scipy document “distance_transform_edt”<br>
<a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.distance_transform_edt.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.distance_transform_edt.html</a></p>

viveksthanam: <p>it has been long time, were you able to find the solution ? if so can you share it please.<br>
Thank you.</p>

Gustavo_Ulloa_Poblet: <p>Hello, I managed to use this function on GPU with the cupy library, it is very fast!<br>
<a href="https://docs.cupy.dev/en/v13.0.0rc1/reference/generated/cupyx.scipy.ndimage.distance_transform_edt.html#cupyx.scipy.ndimage.distance_transform_edt" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.cupy.dev/en/v13.0.0rc1/reference/generated/cupyx.scipy.ndimage.distance_transform_edt.html#cupyx.scipy.ndimage.distance_transform_edt</a></p>
<p>The extra thing is that you have to move the array to the vram using cupy.array()<br>
greetings!</p>
Torch rounding error on long cast

roach: <p>Hey everyone, i was doing some preprocessing on my data and realized that converting an array to a long torch tensor was consuming one of my classes. I’m pretty confused as to why this is happening. To give some context, my masks were created via VTK in C++ from cardiac contours.<br>
Anyone might have any idea why this is happening?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; q = nib.load(os.path.join(trainseg_dir, os.listdir(trainseg_dir)[0])).get_fdata()
&gt;&gt;&gt; print(np.unique(q))
[0.         1.00001526 2.        ]
&gt;&gt;&gt; print(np.unique(torch.tensor(q).long()))
[0 1]
&gt;&gt;&gt; print(np.unique(torch.tensor(np.round(q), dtype=torch.long)))
[0. 1. 2.]
&gt;&gt;&gt; print(torch.nn.functional.one_hot(torch.tensor(q).long()).shape)
torch.Size([256, 256, 2])
&gt;&gt;&gt; print(torch.nn.functional.one_hot(torch.tensor(np.round(q)).long()).shape)
torch.Size([256, 256, 3])
</code></pre>

KFrank: <p>Hi Roach!</p>
<aside class="quote no-group" data-username="roach" data-post="1" data-topic="204424" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roach/48/61570_2.png" class="avatar"> roach:</div>
<blockquote>
<p>converting an array to a long torch tensor was consuming one of my classes.</p>
</blockquote>
</aside>
<p>Your print precision is such that a number less that <code>2.</code> is being displayed<br>
as <code>2.</code>.  Your pytorch <code>.long()</code> conversion truncates this value down to <code>1</code><br>
(as expected).</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.3.1'
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; np.__version__
'1.26.4'
&gt;&gt;&gt; q = np.array ([0.0, 1.00001526, 1.999999999])
&gt;&gt;&gt; q
array([0.        , 1.00001526, 2.        ])
&gt;&gt;&gt; q &lt; 2.0
array([ True,  True,  True])
&gt;&gt;&gt; np.unique (q)
array([0.        , 1.00001526, 2.        ])
&gt;&gt;&gt; np.unique (torch.tensor (q).long())
array([0, 1], dtype=int64)
</code></pre>
<p>As an aside, please don’t post screenshots of textual information.  Doing<br>
so breaks accessibility, searchability, and copy-paste.</p>
<p>Best.</p>
<p>K. Frank</p>

roach: <p>Oh my, you’re absolutely right Frank!<br>
I wasn’t even aware that could happen, turns out my “2.” was in reality 1,9999999995716.<br>
Thanks, apologies for the screenshot I’ll edit it just in case.</p>
Autograd when the NN weights are sampled via NN sampler

toobayes: <p>Hi,</p>
<p>I have two NNs. The first NN generates random weights which are used in a second NN to compute the loss. How can I get the derivative w.r.t. the parameters of the first NN?</p>
<p>I tried using <code>vector_to_parameters</code> which allows me to compute the loss, but the derivative of interest is <code>none</code>.</p>

KFrank: <p>Hi Toobayes!</p>
<aside class="quote no-group" data-username="toobayes" data-post="1" data-topic="204422" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toobayes/48/70653_2.png" class="avatar"> toobayes:</div>
<blockquote>
<p>The first NN generates random weights which are used in a second NN to compute the loss. How can I get the derivative w.r.t. the parameters of the first NN?</p>
</blockquote>
</aside>
<p>Pass your second NN, the weights generated by your first NN, and the<br>
input you wish to pass to your second NN into <a href="https://pytorch.org/docs/2.3/generated/torch.func.functional_call.html" rel="noopener nofollow ugc">func.functional_call().</a></p>
<p>This will return the output of your second NN, but with it using the weights<br>
generated by your first NN.  You may then compute your loss from this<br>
output and backpropagate through <code>functional_call()</code> to get the gradient<br>
of your loss with respect to the parameters of your first NN.</p>
<p>Best.</p>
<p>K. Frank</p>