akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
No ouput by running the pytorch DDP example

Yihua_Xu: <p>I am a freshman on using DDP, so I am trying to run an example supported by Pytorch. Its link is <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu.py" rel="noopener nofollow ugc">multigpu.py</a>. There are 4 GPUs on my machine. PyTorch version: ‘2.0.1+cu117’; OS: Ubuntu 20.04.4 LTS<br>
When I run the code with the following command format <code>python multigpu.py 10 5 &gt; output.tx &amp;</code>, it will spend several hours without output, even without errors. I know it did run because it created a new file named “checkpoint.pt” It should finish in a minute and print out something like <code>[GPU 0 ...</code> <code>[GPU 1 ...</code> <code>[GPU 2 ...</code> <code>[GPU 3 ...</code><br>
Could someone help me?</p>
Does iteratively executing a fuction on training tensor affect the backward?

suihe_dolood: <p>I wrote following code.</p>
<pre><code class="lang-auto">for i in range(5000): 
    optimizer.zero_grad()
    tensor = tanh(tensor)
    activation = []
    model2(tensor)
    loss = loss_function()
    loss.backward(retain_graph=True)
    optimizer.step()
</code></pre>
<p>model2 is a trained model and it’s parameters are freezed. tensor is input of model2 and set requires_grad=True, which means I want to train tensor. The input of model2 should be in [-1,1]. So before executing model2, I uses tanh to transform tensor. But Does it stacks tanh many layers. e.g. tanh(tanh(tanh(…(tensor)…)) ?</p>

ptrblck: <p>No, in each iteration only a single <code>tanh</code> will be applied to tensor.<br>
However, since you are using <code>retain_graph=True</code>, the <code>backward</code> call will backpropagate through <em>all</em> iterations, so make sure this is the desired workflow.</p>

suihe_dolood: <p>Thanks a lot. I don’t want the backward to backpropagate through all iterations. But when I execute backward without retain_graph=False, I got following error.</p>
<pre><code class="lang-auto">trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). specify retain_graph=true if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>How do I avoid this error?</p>

ptrblck: <p>Your code has a recursion since you are reusing <code>tensor</code>, so the error is expected.<br>
If you don’t want to backpropagate through all previous iterations you would need to <code>.detach()</code> the <code>tensor</code> before passing it to the <code>tanh</code> in the new iteration.</p>
DDP no_sync causes increased memory usage

goof: <p>My application uses AMP, DDP, and gradient accumulation. I’m trying to use <code>no_sync</code> to optimize the training loop. However, when I used <code>no_sync</code> it causes an extra ~2GB of VRAM usage and ultimately OOM after the first optimizer step.  Without <code>no_sync</code> it runs fine (albeit inefficiently).  Here is a code snippet:</p>
<pre><code class="lang-auto">#with nullcontext():
with self.unet.no_sync() if not is_last_device_step else nullcontext():
	# Forward pass
	loss = self.get_model_pred(batch)
	loss = loss / self.gradient_accumulation_steps

	# Backward pass
	self.scaler.scale(loss).backward()
</code></pre>
<p>When I just use <code>nullcontext</code>, memory usage before the optimizer step is 18186MB as reported by <code>nvidia-smi</code>.  When I use <code>no_sync()</code> memory use is 20262MB.</p>
<p>Based on my understanding of <code>no_sync</code>, the only difference should be disabling reducing the gradients.  So I don’t see why it would cause an increase of memory use?  Is it keeping a second copy of the gradients for some reason?</p>
<p>For testing purposes I’m currently distributed across only a single GPU.</p>
<p>PyTorch 2.0.1, nVidia 3090, driver 525.105.17, CUDA 12.0.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x1536 and 512x4)

promach: <p>Could anyone help with the following error in <a href="https://colab.research.google.com/drive/1mEU5TPDOwZztqh2BySqHP1ZU1SwsM62D#scrollTo=iNV_S7NOa8O0&amp;line=38&amp;uniqifier=1" rel="noopener nofollow ugc">this code</a> ?</p>
<pre><code class="lang-auto">len(x) =  2
x[0].shape =  torch.Size([1024, 1, 512])
x[1].shape =  torch.Size([1, 1024, 1024])

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

&lt;ipython-input-47-1eb7eeebb607&gt; in &lt;cell line: 51&gt;()
     55     optimizer.zero_grad()
     56     inputs = torch.randint(low=0, high=1, size=(1024,1)).to(torch.long)
---&gt; 57     logits = model(inputs)
     58     labels = torch.randint(low=0, high=1, size=(1,))
     59     loss = loss_fn(logits, labels)

3 frames

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

&lt;ipython-input-47-1eb7eeebb607&gt; in forward(self, x)
     38         x = torch.cat((x[0], x[1]), dim=-1)
     39 
---&gt; 40         x = self.fc(x)
     41 
     42         return x

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1499                 or _global_backward_pre_hooks or _global_backward_hooks
   1500                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1501             return forward_call(*args, **kwargs)
   1502         # Do not call functions when jit is used
   1503         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py in forward(self, input)
    112 
    113     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 114         return F.linear(input, self.weight, self.bias)
    115 
    116     def extra_repr(self) -&gt; str:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x1536 and 512x4)
</code></pre>

ptrblck: <p>The issue is raised in a linear layer as the expected number of input features doesn’t match the feature size of the input activation.<br>
Based on the stacktrace it seems <code>self.fc</code> raises the error so you might need to check its <code>in_features</code> argument and adapt it.</p>
Improve LSTM classification network

juacamgo: <p>Hi, I am training a network to detect the shorelines in aereal images. The idea of my project is not use the typical convolutional network, and instead of this, I am using an LSTM to process column of the images and then classify them between land (0) and sea (1), to train something similar to a segmentation network.</p>
<p>Right now my network is like this:</p>
<pre><code class="lang-auto">class BiLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, num_layers):
        super(BiLSTM, self).__init__()
        self.hidden_size = hidden_size
        
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, bidirectional=True)
        self.hidden2line = nn.Linear(603, 603)
        self.relu = nn.ReLU()
        self.upsample = nn.Upsample(size=603)

    def forward(self, input_col):
        lstm_out, (hn, cn) = self.lstm(input_col.float())
        out = self.upsample(lstm_out)
        out = self.hidden2line(out[:, -1, :])
        out = self.relu(out)
        
        return out

    def backward(self, loss):
        loss.backward()
</code></pre>
<p>I feed columns of the image to the LSTM network, then I upsample them to be equal size to the column size, then I use the linear layer, and then a ReLu.</p>
<p>The output of the network is a vector of equal size of the column length, containing 0 or 1, depending of if it is land or sea.</p>
<p>After a few tests, I am getting “pretty good” results after 100 epochs, with a learning rate of 0.001, getting a MSE loss of 0.005 on test dataset.</p>
<p>But I am very newbie to deep learning, and I am seeking ways to improve the network, how can I try to improve the network?</p>
<p>This is an image passed to the network, and the result:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/475117dccadb5d314177e92aa2a9a6a7de33523d.jpeg" alt="image" data-base62-sha1="aaTDN72VD4lsb0dljzGCq85VY5f" width="544" height="439"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/463b9c90728e96c7c1e41eed8c83e1b19bd76558.jpeg" alt="image" data-base62-sha1="a1j8GJOBgCXoQKMD9sX7wWFzA8w" width="549" height="438"></p>

juacamgo: <p>Up, has anyone any tips so I can improve my neural network?</p>
<p>I insist I am very newbie to this, and I would like to learn as much as I can, thank you!</p>
How to perform `index_put_` in func of vmap?

tiantt: <p>I want to complete the assignment of the lower triangular matrix in the function and extend it to a batch of data using <code>vmap</code>:</p>
<pre><code class="lang-auto">import torch
from torch.func import vmap


def fn(l):
    L = torch.zeros(*l.shape[:-1], 2, 2)
    L[..., [0, 1, 1], [0, 0, 1]] = l
    return L


if __name__ == '__main__':
    L1 = fn(torch.tensor([1., 2., 3.]))
    print(L1)

    Ls1 = fn(torch.tensor([[1., 2., 3.], [4., 5., 6.]]))
    print(Ls1)

    Ls2 = vmap(fn)(torch.tensor([[1., 2., 3.], [4., 5., 6.]]))
    print(Ls2)
</code></pre>
<p>When called separately, it performs well. However, when using <code>vmap</code>, an error is reported</p>
<pre><code class="lang-auto">L1 tensor([[1., 0.],
        [2., 3.]])
Ls1 tensor([[[1., 0.],
         [2., 3.]],

        [[4., 0.],
         [5., 6.]]])
Traceback (most recent call last):
  File "/Users/frank/Projects/neural_lagrangian/neural_lagrangian/models/t.py", line 18, in &lt;module&gt;
    Ls2 = vmap(fn)(torch.tensor([[1., 2., 3.], [4., 5., 6.]]))
  File "/Users/frank/opt/anaconda3/envs/neural_lagrangian/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 434, in wrapped
    return _flat_vmap(
  File "/Users/frank/opt/anaconda3/envs/neural_lagrangian/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 39, in fn
    return f(*args, **kwargs)
  File "/Users/frank/opt/anaconda3/envs/neural_lagrangian/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 619, in _flat_vmap
    batched_outputs = func(*batched_inputs, **kwargs)
  File "/Users/frank/Projects/neural_lagrangian/neural_lagrangian/models/t.py", line 7, in fn
    L[..., [0, 1, 1], [0, 0, 1]] = l
RuntimeError: vmap: index_put_(self, *extra_args) is not possible because there exists a Tensor `other` in extra_args that has more elements than `self`. This happened due to `other` being vmapped over but `self` not being vmapped over in a vmap. Please try to use out-of-place operators instead of index_put_. If said operator is being called inside the PyTorch framework, please file a bug report instead.
</code></pre>
<p>Due to the need to use <code>jacrev</code> to calculate the Jacobian matrix in subsequent tasks, <code>vmap</code> is necessary. I tried using the out-of-place version of the <code>index_put</code> to solves this problem, but <code>index_put</code> does not support ellipses.</p>
<p>May I ask if you have any good suggestions? Thank you very much!</p>
Force_reload but still uses cache?

nicolas_s: <p>Hello,<br>
With my team we are currently facing a problem. We are working on tuning the last layers of a slowfast network to recognize some actions in videos.<br>
After some time, we are getting some “nan” outputs and we are looking for causes and solutions to solve it. So my question is not regarding that, but more the fact that even after</p>
<pre><code class="lang-auto">slowfast = torch.hub.load('facebookresearch/pytorchvideo', 'slowfast_r50', pretrained=True, force_reload=True)
</code></pre>
<p>I get nan again. It seems like even if I use force_reload=True, pytorch uses the cached model, which seems to have been modified <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p>What can I do to force to use a fresh new model?</p>
<p>What also brings me thinking the cache is used is that the first time the container is launched, those two lines appear:</p>
<pre><code class="lang-auto">Downloading: "https://github.com/facebookresearch/pytorchvideo/zipball/main" to /root/.cache/torch/hub/main.zip
Downloading: "https://dl.fbaipublicfiles.com/pytorchvideo/model_zoo/kinetics/SLOWFAST_8x8_R50.pyth" to /root/.cache/torch/hub/checkpoints/SLOWFAST_8x8_R50.pyth
</code></pre>
<p>After that, only the first line appears …<br>
Thank you in advance</p>
Pre-processing text for transformer model for text classification

amjass: <p>Hi all,</p>
<p>I have a corpus of text that is 80 words long per ‘paragraph’ - each ‘paragraph’ is itself a training example and contains a label for classification. I am using the pytorch transformer documentation and am confused about the preprocessing in the example.</p>
<p>I am using - <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and torchtext — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
<p>I have done tokenization before and so this part is trivial - however, when it comes to converting the corpus of text (with each paragraph being a training example)</p>
<pre><code class="lang-auto">def data_process(raw_text_iter: dataset.IterableDataset) -&gt; Tensor:
    """Converts raw text into a flat Tensor."""
    data = [torch.tensor(vocab(tokenizer(item)), dtype=torch.long) for item in raw_text_iter]
    return torch.cat(tuple(filter(lambda t: t.numel() &gt; 0, data)))
</code></pre>
<p>I am confused as to why a flat tensor is created? Maybe someone can shed light on this. the example that is used is the WikiText2 data, which when previewing appears still to be split by paragraphs. The ‘data_process’  function however creates a flat tensor of length 80 * 500,000 paragraphs for my data - so its simply one very long tensor of numbers, suggesting this is one text? but this is not the case. I feel like now, context information as well as training example is completely lost! also, it is import to note that, word 80 and 81 are NOT necessarily found the same context and must be treated as individual training samples. Can someone shed light on what is going on here? and if i have misunderstood, how it is that this flat tensor still retains information about the individual texts?</p>
<p>many thanks!</p>
Torch.jit.script is failing due to Sigmoid?

JosephJ: <p>I’m following <a href="https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html" class="inline-onebox" rel="noopener nofollow ugc">(Beta) Convert MobileNetV2 to NNAPI — PyTorch Tutorials 2.0.1+cu117 documentation</a> to get my model to work on android gpu device but I’m using</p>
<pre><code class="lang-auto">        traced = torch.jit.script(model, input_tensor)
</code></pre>
<p>Instead of trace, as it’s supposed to be more suitable for my model architecture (YoloX),</p>
<p>The trace is failing on the following function:</p>
<pre><code class="lang-auto">Arguments for call are not valid.
The following variants are available:
  
  aten::sigmoid(Tensor self) -&gt; (Tensor):
  Expected a value of type 'Tensor' for argument 'self' but instead found type '__torch__.yolox.models.network_blocks.SiLU (of Python compilation unit at: 0x3516fd0)'.
  
  aten::sigmoid.out(Tensor self, *, Tensor(a!) out) -&gt; (Tensor(a!)):
  Expected a value of type 'Tensor' for argument 'self' but instead found type '__torch__.yolox.models.network_blocks.SiLU (of Python compilation unit at: 0x3516fd0)'.

The original call is:
  File "/home/josephj/srcs/cv_training_code/pytorch/yolox/yolox/models/network_blocks.py", line 14
    @staticmethod
    def forward(x):
        sig = torch.sigmoid(x)
              ~~~~~~~~~~~~~ &lt;--- HERE
        res = x * sig
        return res
</code></pre>
<p>This is a costum SILU implementation that replaces the pytorch SILU as it creates issues issues when exporting to onnx/tflite or when trying to run</p>
<pre><code class="lang-auto">    nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(traced, input_tensor)

</code></pre>
<p>The full SiLU implementation is</p>
<pre><code class="lang-auto">class SiLU(nn.Module):
    """export-friendly version of nn.SiLU()"""
    @staticmethod
    def forward(x):
        sig = torch.sigmoid(x)
        res = x * sig
        return res
</code></pre>
<p>Any ideas how to correct this issue?</p>

tom: <p>Don’t make forward static (and add self as first argument) and you should be OK.</p>
Multi-GPU training to multi-node training

Rakshith_V: <p>If I have a training script which works well for multi-GPU training . What changes in the training script I should make to convert into multi-node training. (Aware about changes in launch command)</p>
<p>I have used local rank in my training script, should this be changed for multi-node training ?<br>
model= DDP(model, device_ids=[local_rank], output_device=local).<br>
Is this correct for multi-gPU training and what should be changed for multi-node training ?</p>

Manuel_Alejandro_Dia: <p>Please post some code to give context to your question.</p>
<p>In brief, for any DDP application you will need to specify the <code>world_size</code>, this is usually given by your number of GPUs if you are on one machine.</p>
<p>If you are using multiple machines, the world size would be determined by: <code>num_machones * gpus</code> , assumming that you have the same number of GPUs in each machine.</p>
<p>Have a look at this <a href="https://medium.com/codex/a-comprehensive-tutorial-to-pytorch-distributeddataparallel-1f4b42bb1b51" rel="noopener nofollow ugc">Medium post</a> where they explain what to do when using multiple nodes.</p>
Flow_to_image()

_guyyariv: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
given a RGB image:<br>
<code>image = torchvision.flow_to_image(optical_flow)</code><br>
How can I reconstruct optical_flow using image alone? (in other words to create image_to_flow)<br>
Thanks!</p>

ptrblck: <p>You could check if you could invert <a href="https://github.com/pytorch/vision/blob/fc838add3235d60feaea2251c1979fd253b26a77/torchvision/utils.py#L426-L458">this method</a>.</p>
DCGAN for data augmentation. Pairing generated images with labels

Martiiiiing: <p>I’m building a model that does emotion recognition ( give me a picture, i’ll tell you what is the emotion felt by the person in that picture).<br>
I want to do some data augmentation on my dataset using DCGAN.<br>
I’m using code from the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">tutorial</a>.<br>
I have 7 different classes. I’ll have to go up to 22 classes eventually.<br>
My dataset is structured like this:<br>
dataset/<br>
├── happy/<br>
│   ├── image1.png<br>
│   ├── image2.png<br>
│   ├── image3.png<br>
│   └── image4.png<br>
├── sad/<br>
│   ├── image1.png<br>
│   ├── image2.png<br>
│   ├── image3.png<br>
│   └── image4.png<br>
…</p>
<p>It properly imports all the images from the different folders.<br>
I just can’t find a way to retrieve the labels of the generated images.<br>
The only alternative I have for now is running the model for each emotion separately.<br>
If anyone knows the proper way, please let me know.<br>
Thanks.</p>
Why do we need to provide a vector as input for backward() when calculating gradient of vector-valued output?

oat: <p>In the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">PyTorch beginner tutorial</a>, why do we need to specify a <strong>vector</strong> as input for the <strong>backward()</strong> function to calculate the <strong>gradient/derivative</strong> of a <strong>vector-valued output y</strong> with respect to a vector input variable x, as shown below?</p>
<p>It seems that in the example below the vector v will “<strong>scale</strong>” the output of <strong>x.grad</strong>. So, why is this the case? Can we just specify v as a “<strong>dummy vector</strong>” composed of all ones [1.0, 1.0, 1.0] to avoid the “scaling”?</p>
<pre><code class="lang-auto">x = torch.randn(3, requires_grad=True)
y = x * 2

while y.data.norm() &lt; 1000:
    y = y * 2

v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(v)
print(x.grad)
</code></pre>

oat: <p>Let me try to answer myself. Found the following post which explains why:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/60700062/runtimeerror-occurs-in-pytorch-backward-function/60703287#60703287" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2945357/andreas-k" target="_blank" rel="noopener nofollow ugc">
    <img alt="Andreas K." src="https://www.gravatar.com/avatar/bd030641417e32d532056ed59e3340e3?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/60700062/runtimeerror-occurs-in-pytorch-backward-function/60703287#60703287" target="_blank" rel="noopener nofollow ugc">RuntimeError occurs in PyTorch backward function</a>
</h4>

<div class="tags">
  <strong>python, tensorflow, pytorch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/2945357/andreas-k" target="_blank" rel="noopener nofollow ugc">
    Andreas K.
  </a>
  on <a href="https://stackoverflow.com/questions/60700062/runtimeerror-occurs-in-pytorch-backward-function/60703287#60703287" target="_blank" rel="noopener nofollow ugc">09:23AM - 16 Mar 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


andreshmo: <p>Had same doutb. So then you do not need to scale? Why does tutorial do it then?</p>
Perform a 2D 'source pixel' lookup on a batch of 17 RGB images

TheMountain: <p>Hello,</p>
<p>I have a set of 3 channel (RGB) 512x512 images with a batch size of 17, stored in a tensor of shape (17, 3, 512, 512).</p>
<p>I also have a series of 2D lookup co-ordinates in a tensor of shape (17, 512, 512, 2), where that final 2 is the XY coordinate of the pixel to look up a source pixel in the original image, allowing me to rearrange the pixels of the 17 source images into 17 new resulting images.</p>
<p>Thanks to some help from this forum, I had this ‘2D pixel lookup’ working when I had a single 2D source image, which also was grayscale, so my image tensor was just (512, 512) and the ‘2d lookup co-ordinates’ tensor was (512, 512, 2).  The line that made that work was:</p>
<p><code>new_images = orig_images[..., lookup_coords[:, :, 0], lookup_coords[:, :, 1]]</code></p>
<p>But now I have 3 channels in my image and a batch size of 17, this line produces a resulting shape of (17, 3, 17, 512, 2) instead of (17, 3, 512, 512) - which matches the input textures - as I wanted.</p>
<p>I’ve messed with that line a lot, and tried squeezing and unsqueezing various items, but I just can’t get the right output shape.  Can anyone help?  Thanks if you can.</p>
<p>p.s. In case the above wasn’t clear, the 17 input images each have their own unique 2D lookup coordinate maps</p>

ptrblck: <p>Could you provide a (slow) reference implementation we could use to verify a potentially better approach?</p>

TheMountain: <p>Yes, good plan.  In fact, the ‘slow’ reference code I wrote isn’t even slow, it just has an iterator over each image in the batch.  With only 17 images in this case, I doubt there will be any speed loss, so I think this issue is fixed now, thanks.</p>
<p>For anyone wanting to see what I did, I added this to deal with each image in the batch separately:</p>
<pre><code class="lang-auto">    output_images = torch.zeros_like(batch_of_images)

    batch_size = batch_of_images.size(0)
    for batch_index in range(batch_size):

        orig_image  = batch_of_images[batch_index]
        lookup_coords = batch_of_coords[batch_index]
</code></pre>
<p>…and that brought my (17, 3, 512, 512) input images tensor down to just (3, 512, 512), which actually got this line of code working to perform the 2D pixel lookup:</p>
<pre><code class="lang-auto">output_images[batch_index] = orig_image[..., lookup_coords[..., 0], lookup_coords[..., 1]]
</code></pre>
<p>I could maybe optimize this code by getting rid of the zeros_like() bit, and just concatenating the result images, but I’ll leave that for another day when I know how to work with Tensors better.</p>
Multi threading memory leak in inference

abc_abc: <p>I am using <strong>Flask</strong> with Pytorch. Every new thread take 20 30mb memory and it is not released after thread is completed . If I do 100 threads in flask then it will take 20mb on 1st 100 calls. How to release that memory ? tried <code>empty_cache</code>, <code>torch.no_grad()</code> but got no luck.</p>
Image loading: Manual resize or using resize in Transform

phphuc: <p>Hi everyone,<br>
I am currently working with a dataset over 200,000 images in 4K resolution. I wonder if I should manually resize image (copy dataset but smaller size) before training or I could just use Resize in module Transform.<br>
Thank you in advance.<br>
P/s: I’m rather new to PyTorch. I hope to receive your all explanations.</p>

ptrblck: <p>Both would work and I would claim it depends on your use case and in particular if you want to randomly crop the images before resizing them etc.<br>
I would not expect to see a huge overhead of resizing the images on the fly, but this would also depend on your system.</p>

Kapil_Rana: <p>Both are fine and will work. I would suggest if you have specific resizing and good HDD space, do it prior to the training. It will save lot of time as at every the model will take time in resizing.</p>

phphuc: <p>In my case, these images are simply the input of training model to classify without any further augmentation. I am in question of whether the time to load and resize (4K) images in every batch is considerable to the training time.<br>
(I’m sorry for my slow response.)</p>

phphuc: <p>Thank you for your response. I am on the side of the preprocessing idea, too. I don’t have enough time to experiment, so if you have some insights about the pros and cons of these two approaches, I hope to hear more from you.</p>
How to pre-process time series classification data to feed in LSTM model

Nosin_Ibna_Mahbub1: <p>Hi, I have a time series dataset showing the bellow figure. How can i make this dataset ready for feeding into LSTM model. Here, “every_type” is a target column and  tt’s a binary classification problem.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cb7320737b222c00f293c04e3954f74430f551b.png" data-download-href="https://discuss.pytorch.org/uploads/default/9cb7320737b222c00f293c04e3954f74430f551b" title="Screenshot from 2023-05-24 14-45-16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cb7320737b222c00f293c04e3954f74430f551b_2_690x186.png" alt="Screenshot from 2023-05-24 14-45-16" data-base62-sha1="mmmYKfjcJ9YQ51JU6L26kbvntID" width="690" height="186" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cb7320737b222c00f293c04e3954f74430f551b_2_690x186.png, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cb7320737b222c00f293c04e3954f74430f551b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cb7320737b222c00f293c04e3954f74430f551b.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-05-24 14-45-16</span><span class="informations">877×237 22.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance</p>
Infinite inference when using multiprocessing with cuda for multiple models

enigma: <p>Hi!</p>
<p>I am trying to utilise AMD GPU for inference speedup for genetic learning: I have multiple servers that act like “game” environments, each server has its socket and is capable of “playing” a map, which I have multiple of, graph NN and “agents”: instances of GeneticLearner class, capable of fitting its internal weight vector, acting as the last layer of NN.</p>
<p>Originally I used multiprocessing to distribute the load, which worked fine, but when I try to pass the model and data to GPU, it results in an infinite waiting time in forward function and it looks like GPU is idle.</p>
<p>Is there any way I can fix this issue? CUDA+multiprocessing best practices didn’t help</p>
<p>my code looks roughly like this:</p>
<pre><code class="lang-py">def play_game(model, game_map, ws_queue: queue.Queue[websocket.Websocket]):
    ...
    # get socket from queue, play game, put socket back, return results

def init_fun():
    ...
    # assign torch model to static field "MODEL" of GeneticLearner

future_queue = queue.SimpleQueue()
with mp.Pool(proc_num, init_fun) as executor:
    for model, game_map in games:
        ...
        future = executor.apply_async(
            play_game,
            [model, game_map, max_steps, ws_urls], # ws_urls is a torch.multiprocessing.Manager.Queue
            ...
        )
        futures_queue.put(future)

    while not futures_queue.empty():
        ...  # get results
</code></pre>
<p>I load model like this:</p>
<pre><code class="lang-py">def load_model(path: str) -&gt; torch.nn.Module:
    model = ml.models.StateModelEncoder(hidden_channels=64, out_channels=8)
    model.load_state_dict(torch.load(path), strict=False)
    model.to(torch.device("cuda:0"))
    model.eval()
    return model
</code></pre>
<p>and use it like this:</p>
<pre><code class="lang-py">device = torch.device("cuda:0")
data.to(device)  # data is torch_geometric.data.HeteroData
out = model.forward(data.x_dict, data.edge_index_dict)
</code></pre>
<p>My environment:<br>
OS: Ubuntu 22.04<br>
GPU: AMD Radeon Vega Frontier Edition<br>
Conda Python3.10.9<br>
packages:<br>
pytorch-triton-rocm       2.0.1                    pypi_0    pypi<br>
torch                     2.0.1+rocm5.4.2          pypi_0    pypi<br>
torch-geometric           2.3.1                    pypi_0    pypi</p>
MultiheadAttention for Vision

Strange: <p>Hello,<br>
Can anyone help me in understanding how to use nn.MultiheadAttention for vision and how can I calculate query, key, value if I have (64, 16, 16) intermediate value.</p>
<p>I am Trying to implement <a href="https://arxiv.org/abs/2105.05233" rel="noopener nofollow ugc">Diffusion Models Beat GANs on Image Synthesis</a> paper. They used attention heads in UNets.</p>
<p>Thank you.</p>
Efficient Methods for Computing or Approximating the Condition Number of the Hessian Matrix in Neural Networks

blade: <p>Hello everyone,</p>
<p>I’m currently working on computing the condition number of the Hessian matrix for the weights of a neural network. While I have implemented the code for this computation, I have noticed that it is quite expensive and time-consuming. Therefore, I’m reaching out to the community to inquire if there are any more efficient methods or approximations available to tackle this problem.</p>
<p>Here is a snippet of my code for reference:</p>
<pre><code class="lang-auto"># -- predict
y, logits = _stateless.functional_call(model, params, x.squeeze(1))

# -- loss
loss = loss_func(logits, label.ravel())

for param in [v for k, v in params.items() if 'fc' in k]:
    # -- compute gradient
    grad = torch.autograd.grad(loss, param, retain_graph=True, create_graph=True, allow_unused=True)[0].reshape(-1)

    # -- compute Hessian
    Hessian = []
    n_params = grad.shape[0]
    for j in range(n_params):
        hess = torch.autograd.grad(grad[j], param, retain_graph=True, allow_unused=True)[0].reshape(-1)
        Hessian.append(hess)
    Hessian = torch.stack(Hessian)

    # -- compute condition number
    print(torch.linalg.cond(Hessian))
</code></pre>
<p>I would greatly appreciate any insights, suggestions, or alternative approaches that can help me optimize the computation of the Hessian matrix or find a more affordable approximation. Thank you in advance for your help!"</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/blade">@blade</a>,</p>
<p>Depending on where the bottleneck in your code is, you could always compute your derivatives with the <code>torch.func</code> package (in order to vectorize over the <code>for j in range(n_params):</code> loop, which will speed up your calc. In fact, the <code>torch.func</code> package will allow for an efficient Hessian calculation via their API and if multiple samples are needed, you can vectorize over the samples via <code>torch.func.vmap</code>. Both of these calls can be used compositionally too, i.e. <code>vmap(hessian(myFunc))</code></p>
<p>As the condition number is the ratio of the largest and smallest eigenvalues (depending on what norm you choose, of course). You could compute the largest eigenvalue via the <a href="https://en.wikipedia.org/wiki/Power_iteration" rel="noopener nofollow ugc">power-iteration method</a>, but I’m not sure on how to apply such method for the smallest eigenvalue (as that requires the inverse of your Hessian matrix, which scales cubically with the size of the Hessian matrix).</p>
[TorchDynamo] How to generate fx graph with nn.* layers?

zejphyr: <p>Hi Folks,</p>
<p>I try to generate a graph with nn.* layers in the “target” fields of FX nodes?</p>
<p>For example:</p>
<pre><code class="lang-auto">class WrappedBatchNorm(nn.Module):
    def __init__(self):
        super().__init__()
        self.mod = nn.BatchNorm2d(1)
    def forward(self, x):
        return self.mod(x)

class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Conv2d(1, 1, 1)
        self.layer2 = nn.BatchNorm2d(1)
        self.layer3 = nn.Conv2d(1, 1, 1)
        self.nested = nn.Sequential(
            nn.BatchNorm2d(1),
            nn.Conv2d(1, 1, 1),
        )
        self.wrapped = WrappedBatchNorm()

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.nested(x)
        x = self.wrapped(x)
        return x

def my_custom_backend(gm, example_inputs):
    print("My Custom Backend:")
    print(gm.graph)
    return gm.forward

#dyn.config.skipfiles_inline_module_allowlist.add(M)
torch._logging.set_logs(dynamo=logging.DEBUG)

mod = M()
mod.eval()
mod = torch.compile(mod, backend=my_custom_backend)
mod(torch.randn(1,1,4,4))

</code></pre>
<p>It generates a graph like:</p>
<pre><code class="lang-auto">My Custom Backend:
graph():
    %l_x_ : torch.Tensor [#users=1] = placeholder[target=L_x_]
    %l__self___layer1 : [#users=1] = call_module[target=L__self___layer1](args = (%l_x_,), kwargs = {})
    %l__self___layer2 : [#users=1] = call_module[target=L__self___layer2](args = (%l__self___layer1,), kwargs = {})
    %l__self___layer3 : [#users=1] = call_module[target=L__self___layer3](args = (%l__self___layer2,), kwargs = {})
    %l__self___nested_0 : [#users=1] = call_module[target=L__self___nested_0](args = (%l__self___layer3,), kwargs = {})
    %l__self___nested_1 : [#users=1] = call_module[target=L__self___nested_1](args = (%l__self___nested_0,), kwargs = {})
    %l__self___wrapped_mod : [#users=1] = call_module[target=L__self___wrapped_mod](args = (%l__self___nested_1,), kwargs = {})
    return (l__self___wrapped_mod,)
</code></pre>
<p>Since I want to do some patten match, I like it generate a graph with “target=nn.*" , e.g.:<br>
%xxx : [#users=1] = call_module[<em>target=nn.Conv2d</em>](args =…)</p>
<p>What should I do? can it trace into the self.xxx call and inline it?</p>
<p>Thanks</p>

marksaroufim: <p>It sounds like what you need is the torch IR - this tutorial might help <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

zejphyr: <p>Thank you! This makes sense</p>
Vision Transformer dataset

stanleygeorge: <p>On what dataset are the  pretrained models of <code>ViT_B_16</code> and <code>ViT_L_16</code> trained upon ? (For e.g. I could not understand from the description of <code>ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1</code> as which dataset was used for the training)</p>
<p>(<a href="https://pytorch.org/vision/main/models/generated/torchvision.models.vit_b_16.html#torchvision.models.ViT_B_16_Weights" class="inline-onebox" rel="noopener nofollow ugc">vit_b_16 — Torchvision main documentation</a>)</p>

Kapil_Rana: <p>The model is trained in Image-net dataset.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/datasets/imagenet-1k">
  <header class="source">

      <a href="https://huggingface.co/datasets/imagenet-1k" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://cdn-thumbnails.huggingface.co/social-thumbnails/datasets/imagenet-1k.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/datasets/imagenet-1k" target="_blank" rel="noopener nofollow ugc">imagenet-1k · Datasets at Hugging Face</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


stanleygeorge: <p>So, I did a bit of reading and this is what I understood.<br>
The models were pretrained using the IG 3.6B dataset following the SWAG approach of this paper (<a href="https://arxiv.org/pdf/2201.08371.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2201.08371.pdf</a>).<br>
And later, it was finetuned on Imagenet-1k.</p>
Pass batches to custom Python function

moth: <p>I have a custom Python function, which has several parameters and performs some logic. It only uses native Python functions or Numpy functions:</p>
<pre><code class="lang-python">def my_function(param_1, param_2, ..., param_N):
    # Logic
    ...
    return output
</code></pre>
<p>I want to speed up my function execution by using GPUs and passing a torch batch to this function instead of individual parameters. I could pass a numpy array to this function instead of individual parameters and <code>ravel()</code> the entries into variables inside the function. But I dont know how to go from here to passing a whole batch of data and process it as a batch.</p>
<p>Is this possible? Do I need to use only <code>torch</code> functions instead of numpy?</p>

ptrblck: <aside class="quote no-group" data-username="moth" data-post="1" data-topic="180559">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aca169/48.png" class="avatar"> moth:</div>
<blockquote>
<p>Do I need to use only <code>torch</code> functions instead of numpy?</p>
</blockquote>
</aside>
<p>Yes, if you want to use your GPU you should use PyTorch operations since <code>numpy</code> doesn’t support GPUs and you would have to move the data back to the CPU to apply <code>numpy</code> operations.<br>
Additionally, using <code>numpy</code> (or another 3rd party library) will detach the tensor from the computation graph as Autograd won’t be able to track these operations.<br>
PyTorch operations mostly accept batched inputs so you should be able to directly use them.</p>
Pytorch operation：get tensor from other index tensor

Jie_Zhang: <p>tensor A:shape （1024  128  3）<br>
tensor B: shape （1024  10）(the value in tensor B is index number of A at dimension 1)<br>
tensor C:shape （1024  10  3）<br>
how can I get tensor C from A with index B</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">A = torch.randn(1024, 128, 3)
B = torch.randint(0, 128, (1024, 10))
C = A[torch.arange(A.size(0)).unsqueeze(1), B]
print(C.shape)
# torch.Size([1024, 10, 3])
</code></pre>
Can't seem to be able to import Field and BucketIterator from torchtext

Emirates: <p>I’m new to PyTorch. I’m trying to follow a tutorial online but I’ve not been able to make progress since yesterday. I always get an error message when I try executing the code below on my laptop.</p>
<pre><code class="lang-py">from torchtext.datasets import TranslationDataset
from torchtext.data import Field, BucketIterator
</code></pre>
<p><strong>These are the type of error messages I get</strong></p>
<pre><code class="lang-py">ImportError: cannot import name 'TranslationDataset' from 'torchtext.datasets' (C:\ProgramData\Anaconda3\lib\site-packages\torchtext\datasets\__init__.py)

ImportError: cannot import name 'Field' from 'torchtext.data' (C:\ProgramData\Anaconda3\lib\site-packages\torchtext\data\__init__.py)
</code></pre>
<p>I read online that I should use <strong>torchtext.legacy</strong> instead. So when I tried the code below</p>
<pre><code class="lang-py">from torchtext.legacy.data import Field, BucketIterator, Iterator
</code></pre>
<p>I got this error message</p>
<pre><code class="lang-py">ModuleNotFoundError: No module named 'torchtext.legacy'
</code></pre>
<p>==============<br>
Please can someone kindly assist me on how to get my error fixed. I’m using <strong>torchtext</strong> version ‘0.15.1’ and <strong>torch</strong> version ‘2.0.0+cpu’</p>
Pytorch distributed multiple runs on same server - server socket has failed error

MLAlex1: <p>I have a server where there are 8 GPUs available. I want to run different runs where each run uses 2 GPU. After starting the first run, when I try to run the second with the following script:</p>
<pre><code class="lang-auto">torchrun --standalone --nproc_per_node=2 train.py --batch_size 16
</code></pre>
<p>I am getting the following error:</p>
<pre><code class="lang-auto">[W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:29400 (errno: 98 - Address already in use).
[W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:29400 (errno: 98 - Address already in use).
[E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
</code></pre>
<p>My understanding is that this happens because it’s trying to run on the same port. I tried to set different ports using <code>--master_port 29501 </code> and  setting ports in the script like:</p>
<pre><code class="lang-auto">os.environ['MASTER_PORT'] = "29500"
os.environ['MASTER_ADDR'] = '127.0.0.1'
</code></pre>
<p>but none of them worked.</p>
How to delete PyTorch objects correctly from memory

111: <p>I’m having an issue with properly deleting PyTorch objects from memory. With this Tensor:</p>
<blockquote>
<blockquote>
<blockquote>
<p>test = torch.Tensor(1000,1000)</p>
</blockquote>
</blockquote>
</blockquote>
<p>Then delete the object:</p>
<blockquote>
<blockquote>
<blockquote>
<p>del test</p>
</blockquote>
</blockquote>
</blockquote>
<p>CUDA memory is not freed up.</p>
<p>Is there a clean way to delete a PyTorch object from CUDA memory?</p>

albanD: <p>Hi,</p>
<p>It is because the cuda backend uses a caching allocator. This means that the memory is freed but not returned to the device.</p>
<p>if after running <code>del test</code> you allocate more memory with <code>test2 = torch.Tensor(1000,1000)</code>, you will see that the memory usage will stay exactly the same: it did not re-allocated memory but re-used the one that had been freed when you ran <code>del test</code>.</p>

111: <p>Ah Thanks a lot! It’s really helpful!</p>

nionjo: <p>Hi,<br>
Which function do you use to monitor the gpu memory usage?<br>
Thanks.</p>

albanD: <p>You can find them in the doc <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.max_memory_allocated">here</a>.</p>

SANTOSH_S: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,<br>
I am using different models for inference. But my gpu space is only 10gb.<br>
How do I unload a model from cuda and switch/load another model to cuda?</p>

albanD: <p>Hi,</p>
<p>Doing <code>model.cpu()</code> will move it back to the cpu.<br>
Assuming that nothing else references the weights, they will be freed and returned to our allocator to be used for other Tensors.</p>

ndvbd: <p>Here’s a miminal example showing that memory is not freed. I want to complete free the tensor memory from GPU, and to be able to see it in nvidia-smi:</p>
<pre><code class="lang-auto">
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "6"
import torch
from transformers import LlamaForCausalLM

model = LlamaForCausalLM.from_pretrained(
            pretrained_model_name_or_path='decapoda-research/llama-7b-hf',
            load_in_8bit=True,
            device_map={'': 0},
        )

del model
torch.cuda.empty_cache()

print('breakpoint here - is memory freed?')
</code></pre>

ptrblck: <p>Seems to be solved <a href="https://discuss.pytorch.org/t/how-to-delete-a-tensor-in-gpu-to-free-up-memory/48879/27">here</a>.</p>
ProcessGroup not converted to ProcessGroupNCCL

santurini: <p>I was trying to launch a distributed training for which a ProcessGroupNCCL is required but it seems like pybind11 is failing to convert my process group to NCCL.</p>
<p>The process is initialized as follows:</p>
<pre><code class="lang-auto">rank = int(os.environ["OMPI_COMM_WORLD_RANK"])
local_rank = int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])
world_size = int(os.environ["OMPI_COMM_WORLD_SIZE"])

dist.init_distributed(dist_backend="nccl", rank=rank, world_size=world_size)
</code></pre>
<p>And the error I get is the following:</p>
<pre><code class="lang-auto">TypeError: ensure_nccl(): incompatible function arguments. The following argument types are supported:
    1. (arg0: torch.distributed.distributed_c10d.ProcessGroupNCCL, arg1: torch.Tensor) -&gt; None

Invoked with: &lt;torch.distributed.distributed_c10d.ProcessGroup object at 0x7fa630e71cb0&gt;, tensor([[ 0.8793,  1.1869, -1.2402,  ...,  0.7247, -0.2372, -0.3371],
</code></pre>
<p>Looking at the source code of <code>init_distributed()</code> when I pass “NCCL” as backend the process group should be defined as <code> torch.distributed.distributed_c10d.ProcessGroupNCCL</code> but is not.</p>

H-Huang: <p><code>init_distributed</code> looks like a different API, so i assume this is a wrapper around <code>torch.distributed.init_process_group</code>. Since PyTorch 2.0 (<a href="https://github.com/pytorch/pytorch/releases/tag/v2.0.0#:~:text=%2389523" class="inline-onebox" rel="noopener nofollow ugc">Release PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever · pytorch/pytorch · GitHub</a>)-,Distributed%20(c10d),-Dispatchable%20collectives%3A%20An), <code>init_process_group</code> no longer returns specific processgroup instances (e.g. <code>ProcessGroupNCCL</code>, <code>ProcessGroupGloo</code>, etc.) but insteads returns a generic <code>ProcessGroup</code> instance. This <code>ProcessGroup</code> may have multiple backends. Probably the fix is to just update the <code>ensure_nccl</code> function to check for <code>torch.distributed.distributed_c10d.ProcessGroup</code> instead of <code>torch.distributed.distributed_c10d.ProcessGroupNCCL</code></p>

santurini: <p>Actually I typed wrong it is just:</p>
<pre><code class="lang-auto">import torch.distributed as dist
dist.init_process_group(...)
</code></pre>
<p>But I think your observation is still valid, I’ll check and see what happens.</p>
I have the problem when i use the horovod

loveafy: <p>I run my process in single computor have the problem:<br>
RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use). The server socket has failed to bind to 0.0.0.0:29500 (errno: 98 - Address already in use).<br>
the run commond is horovodrun -np 2 -H localhost:2 -p 12345 python -m torch.distributed.launch --nproc_per_node 2 train.py<br>
But i run the officia example pytorch_mnist.py don’t have that problem.</p>