akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error for affine_grid_generator when using 5D tensor

John1231983: <p>I have a tensor size of <code>1x2x32x32x32</code>. I want to feed it into spatial transformation network using the tutorial in pytorch. I have change the size of fc based on my input size. The final size before send to the <code>grid = F.affine_grid(theta, x.size())</code> is</p>
<pre><code class="lang-auto">theta: (1,6)
x (1,2,32,32,32)
</code></pre>
<p>However, I got the error. <strong>How should I fix it?</strong> You can run my code in the colab at <a href="https://colab.research.google.com/drive/1MPzto7IJ3Z9yj5wo1trl3Sk9JxlI7EQr#scrollTo=mYdLsM7Dkf1j" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<blockquote>
<p>ret = torch.affine_grid_generator(theta, size)<br>
RuntimeError: invalid argument 6: wrong matrix size at /opt/conda/conda-bld/pytorch-nightly_1555305720252/work/aten/src/THC/generic/THCTensorMathBlas.cu:494<br>
This is my code</p>
</blockquote>
<pre><code class="lang-auto">from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import numpy as np

plt.ion()   # interactive mode
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Spatial transformer localization-network
        self.localization = nn.Sequential(
            nn.Conv3d(2, 8, kernel_size=7),
            nn.MaxPool3d(2, stride=2),
            nn.ReLU(True),
            nn.Conv3d(8, 10, kernel_size=5),
            nn.MaxPool3d(2, stride=2),
            nn.ReLU(True)
        )
        # Regressor for the 3 * 2 affine matrix
        self.fc_loc = nn.Sequential(
            nn.Linear(10 * 4 * 4 * 4, 32),
            nn.ReLU(True),
            nn.Linear(32, 3 * 2)
        )

        # Initialize the weights/bias with identity transformation
        self.fc_loc[2].weight.data.zero_()
        self.fc_loc[2].bias.data.copy_(torch.tensor([1, 0, 0, 0, 1, 0], dtype=torch.float))

    # Spatial transformer network forward function
    def stn(self, x):
        xs = self.localization(x)
        print (xs.size())
        xs = xs.view(-1, 10 * 4 * 4 * 4)        
        theta = self.fc_loc(xs)
        theta = theta.view(-1, 2, 3)
        grid = F.affine_grid(theta, x.size())
        x = F.grid_sample(x, grid)
        return x

    def forward(self, x):
        # transform the input
        x = self.stn(x)
        
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = Net().to(device)
x =  torch.rand(1,2,32,32,32).to(device)
print('Input shape', x.shape)
x_stn = model(x)

</code></pre>

John1231983: <p>Ok. I fixed it. It should be <code>theta = theta.view(-1, 3, 4)</code></p>

Zac_Boussaid: <p>Do you have an explanation for this ? it works for me too  but i’m not able to understand it</p>
Got stuck in TorchScript Trace

iamexperimentingnow: <p>Hi,</p>
<p>Basically, I have built the deep learning model which takes dynamic inputs with different shapes,</p>
<p>so how do I trace the model here?</p>
<p>usually, we trace the model using the below command</p>
<pre><code class="lang-auto">example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
</code></pre>
<p>but then, in my scenario, I pass dynamic input ndarray with different shapes<br>
eg:</p>
<pre><code class="lang-auto">1. example = torch.rand(1, 3, 224, 224)
2. example = torch.rand(1, 3, 224, 550)
3. example = torch.rand(1, 3, 640, 480)
</code></pre>
<p>could you please help me understand and how to implement this?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/ppwwyyxx">@ppwwyyxx</a> <a class="mention" href="/u/driazati">@driazati</a><br>
<a class="hashtag" href="/c/jit/13">#<span>jit</span></a> <a class="hashtag" href="/c/deployment/14">#<span>deployment</span></a></p>

ptrblck: <p>You would have to <code>torch.jit.script</code> the model instead of tracing.<br>
The next stable release will use the <code>nvfuser</code> backend during scripting which has dynamic shape support.</p>

iamexperimentingnow: <p>thanks for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a> , I will try to save the model by torch.jit.script and get back to you.</p>
Make on model's parameters be .clone() of other's

mgheini: <p>Hi,</p>
<p>Is there a way to create a model where the parameters are clones of another model’s parameters? To emphasize, this is more than just value copying, which is, for instance, possible by means of loading the <code>state_dict</code>, etc. I need the second model’s parameters to be created/assigned using the <code>.clone()</code> operation as I need the gradients to be stored for the original l model as well.</p>
<p>Thanks.</p>
Can conv2d work with more than 4 dimension input?

evanthebouncy: <p>Hi,</p>
<p>As of now conv2d expect input of dimension [batch x channel x width x height], but how would I process a list of images (say of size M). As of now my input tensor has shape [M x batch x channel x width x height]. Pushing it through a conv2d layer cause errors.</p>
<p>I was able to work around it by un-stacking it into a python list of size M, calling conv2d over it in a for-loop and re-stacking them back. Let me know if there’s a more “torch-esque” way of doing it</p>
<p>much thanks ~</p>
<p>–evan</p>

justusschock: <p>Do you want to model dependencies/correlations between these M images?</p>

John1231983: <p>You can do it using conv3d. Your case like video processing. If you still want to use conv2d, so convert it to [batch x channel x height x ( M x width)] using <code>view</code> function</p>

evanthebouncy: <p>I want to compute attention over these M images.</p>
<p>essentially I’d like to convert [img1, img2, . . . imgM] into latent representations [lat1, lat2, …, latM]</p>
<p>I originally thought I can just create tensor [M x batch_image] which is essentially [M x batch x chan x W x H] once you expand it out.</p>
<p>and pass it through conv2d, which would result in [M x batch x latent]</p>
<p>no luck though xD</p>

justusschock: <p>You could really try it with 3d convolutions and maybe try something like <a href="https://github.com/bionick87/ConvGRUCell-pytorch" rel="nofollow noopener">this</a>.</p>

yashasvi97: <p>Hi <a class="mention" href="/u/evanthebouncy">@evanthebouncy</a>, I have a similar use case where I want to feed input of shape BXNXCXHXW, where B is batch size, N is number of images, and rest same. I am also trying to implement some attention mechanism, was wondering were you able to find a way to pass more than 4d input to conv2d? or some other solution maybe?</p>

hkingtswcbyy: <p>Use view make the code ugly, if conv2d support extra-dims will make the code clean and clear.</p>
Disable pytorch from reserving CUDA memory

rohithkrn: <p>I there a way to disable pytorch caching allocator from reserving cuda memory?</p>

ptrblck: <p>I assume you would like to disable the caching mechanism and force the allocator to use <code>alloc/free</code> for each (de-)allocation? If so, then you could use <code>PYTORCH_NO_CUDA_MEMORY_CACHING=1</code>. Note that this is a debugging env variable as you would see a large performance decrease.</p>
How can I increase the size of MNIST?

YuhskeHujisaki: <p>The size of MNIST is (1,28,28), but I would like to change this to (1,30,375). What should I do? Which should I fill in the enlarged part with 0 or 1?</p>

ptrblck: <p>You could resize the images using <code>transforms.Resize</code>.</p>
<aside class="quote no-group" data-username="YuhskeHujisaki" data-post="1" data-topic="154364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuhskehujisaki/48/48683_2.png" class="avatar"> YuhskeHujisaki:</div>
<blockquote>
<p>Which should I fill in the enlarged part with 0 or 1?</p>
</blockquote>
</aside>
<p>If you want to pad the images only, I would fill the padding with zeros as it’s representing the background.</p>
Get the last token tensor according the seq_len from a padded tensor

yipliu: <p>Hi,</p>
<p>Poor to my English, I can not google well about my question.</p>
<p>I have a padded tensor  tensor_a.shape=[B, S, H] , and I have a seq_len.shape = [B]</p>
<p>tensor_a =<br>
tensor([[[0.8906, 0.5816, 0.3081],<br>
[0.7228, 0.2029, 0.9706],<br>
[0.8582, 0.5991, 0.2073],<br>
[0.5673, 0.2016, 0.5965]],</p>
<pre><code>    [[0.7556, 0.4601, 0.5045],
     [0.5717, 0.2664, 0.0891],
     [0.2461, 0.0900, 0.6039],
     [0.6516, 0.0964, 0.6036]]])
</code></pre>
<p>seq_len= tensor([3, 1])</p>
<p>I want to get a new tensor</p>
<p>tensor([[  [0.8582, 0.5991, 0.2073],<br>
[<br>
[0.5717, 0.2664, 0.0891],<br>
]])<br>
I just want to get the last word’s tensor.  What can I do?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">tensor_a = torch.tensor([[[0.8906, 0.5816, 0.3081],
                          [0.7228, 0.2029, 0.9706],
                          [0.8582, 0.5991, 0.2073],
                          [0.5673, 0.2016, 0.5965]],
                         
                         [[0.7556, 0.4601, 0.5045],
                          [0.5717, 0.2664, 0.0891],
                          [0.2461, 0.0900, 0.6039],
                          [0.6516, 0.0964, 0.6036]]])

seq_len = torch.tensor([2, 1])
res = tensor_a[torch.arange(tensor_a.size(0)), seq_len]
print(res)
# tensor([[0.8582, 0.5991, 0.2073],
#         [0.5717, 0.2664, 0.0891]])
</code></pre>
torch.nn.DataParallel problem

hitbuyi: <p>I use cpu only, cuda not used here</p>
<p>device = torch.device(“cpu”)<br>
model = torch.nn.DataParallel(model).to(device)</p>
<p>y = model(x) # both y and x are cpu tensors</p>
<p>I found self.model.src_device_obj is cuda:0 , and impose error:<br>
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu self.src_device_obj</p>
<p>debug in pycharm, the erro is in nn.parallel.data.parallel.py ,see following picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c50f6a562b1952099613261def37875271ef536.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4c50f6a562b1952099613261def37875271ef536" title="E1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c50f6a562b1952099613261def37875271ef536_2_690x388.jpeg" alt="E1" data-base62-sha1="aT7KzuIP8DLv1WlbZDphB9wDxlQ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c50f6a562b1952099613261def37875271ef536_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c50f6a562b1952099613261def37875271ef536_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c50f6a562b1952099613261def37875271ef536_2_1380x776.jpeg 2x" data-dominant-color="3E4755"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">E1</span><span class="informations">1920×1080 271 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>it is said nn.DataParallel is not supported in CPU mode, so what is used  to replace nn.DataParallel in CPU mode?</p>
Super Resolution algorithms for very large images

zabaniiiyaaa: <p>Hello guys<br>
I’ve been wondering how is it possible to apply a super resolution model to very large images (2000x2000)<br>
I know and use pytorch distributed parallel training, but was wondering if there was something similar to it for the test<br>
The test code I’m trying to run is the following’s :</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">

      <a href="https://github.com/Lornatang/ESRGAN-PyTorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:130/65;"><img src="https://opengraph.githubassets.com/4b477e45ba942d146ee39f2081b6ccebac6c935ea1b2598266f70411a883a1e4/Lornatang/ESRGAN-PyTorch" class="thumbnail" width="130" height="65"></div>

<h3><a href="https://github.com/Lornatang/ESRGAN-PyTorch" target="_blank" rel="noopener nofollow ugc">Lornatang/ESRGAN-PyTorch</a></h3>

  <p>A simple implementation of esrgan, which uses the pytorch framework. - Lornatang/ESRGAN-PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
If the image I try to upscale surpasses a resolution (600x600) it is not possible to test on it because of out of memory issue.<br>
In terms of graphic prowess, I work on a server with multiple nvidia Tesla gpus on multiple nodes so I can use as many gpus as possible.<br>
Thank you</p>

Lornatang: <p>You can split into several small images for processing and then stitch them together.</p>
Split the dataloader and save / load, result are not identical

Leonmac: <p>Hello,<br>
I want to make a small tool that can do data-set pre-splitting before the train happen.<br>
What I did:</p>
<ol>
<li>split the original testloader to three sub-testloader return as a dataloader list–by using torch.utils.data.random_split,</li>
<li>save all testloaders to three *.pt file to disk – by using torch.save</li>
<li>reload three *.pt files to new testloaders from disk --by using torch.load</li>
</ol>
<p>Basically, I am following the suggestion of here for save/load the dataloader</p><aside class="quote quote-modified" data-post="3" data-topic="62813">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-save-dataloader/62813/3">How to Save DataLoader?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Maybe 
import random
from torch.utils.data import Sampler, DataLoader

class MySampler(Sampler):
    def __init__(self, data_source):
        self.seq = list(range(len(data_source)))
    def __iter__(self):
        return iter(self.seq)

dataset = LoadYourDataset()
sampler = MySampler(dataset)
dataloader = DataLoader(dataset, shuffle=False, sampler=sampler)

for epoch in range(0, 999):
    random.shuffle(dataloader.sampler.seq)
    for i, (x, y) in enumerate(dataloader):
        # some code.
   …
  </blockquote>
</aside>

<p>When I compare post_step1 testloader1 vs. post_step3 reload_testloader1 – I sample out the 1st batch of testloader and plot the image, then I find they are not the same.</p>
<p>Initially, I suspect if there are some random/shuffle flag impacts, but given at end of step1, all three testloader should be already confirmed, I couldn’t figure out where could be the problem.</p>
<p>Is there any tricky about the torch.save/torch.load for the testloader object? the saving/load is quite simple.</p>
<pre><code class="lang-auto">def save_dl(dataloader_obj:DataLoader, file_name: str) -&gt; None:
    """save the dataloader to pt file"""
    torch.save(dataloader_obj, file_name)

def load_dl(file_name: str) -&gt; DataLoader:
    """load the dataloader from  pt file"""
    return torch.load(file_name)
</code></pre>

Leonmac: <p>OK after some further test, I think I find the reason – i.e. the “shuffle” option defined when Dataloader obj is created, decides that each time you load the dataloader, whether the contents will be shuffled or not, this is preserved when you save it to disk.</p>
I want to stack a RGB images for 3D or 4D tensor

yoshimura.hisa: <p>Hi, pytorch support teams.</p>
<p>I want to construct a 3D or 4D RGB tensor.<br>
And, create a GAN model using these tensor.<br>
How do I define how to create such a tensor?<br>
I would like to stack the attached 2D RGB images.<br>
Or can you extract each RGB element from a 3D image as a 3D tensor?</p>
<p>Kind regards,<br>
yoshimura.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/255fe73f9869b33f9d095c556ff3a3f11f26bcca.png" alt="173480331-c74cf544-d349-4c10-a30e-d53735d7c00e" data-base62-sha1="5kD8zF25hCohq6p6VK6aOkb1JH4" width="257" height="252"></p>

harpone: <p>Typical CV image tensor has shape <code>[channels, height, width]</code> (<code>channels = 3</code> for RGB). You could stack these with <code>torch.stack(list_of_images, dim=-1)</code>, which gives a tensor of shape <code>[channels, height, width, num_images]</code>.</p>

yoshimura.hisa: <p>Hi, HIkkei.</p>
<p>I could stack my data. Thanks’ your advice.</p>
Cuda error GPU error, unable to allocate memory

iamexperimentingnow: <p>Hi,</p>
<p>I have built the semantic segmentation model and trying to deploy my model in a 16GB GPU machine with a 4vCPU machine.</p>
<p>when I try to set the workers as 4 I get a Cuda Memory error. After that, I try to reduce the number of workers to 2. It worked and I was able to deploy and host the model as an API.</p>
<p>can anyone tell me how to completely utilise 4 works to make 4 concurrent API calls?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="hashtag" href="/c/deployment/14">#<span>deployment</span></a> <a class="hashtag" href="/c/vision/5">#<span>vision</span></a></p>

ptrblck: <p>The number of workers in a <code>DataLoader</code> should not cause “CUDA Memory errors”, so could you explain which error you saw and post a minimal, executable code snippet to reproduce the issue?</p>

iamexperimentingnow: <p>thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a> , here I’m doing real-time inferencing. So, I will be passing only one image at a time. so I believe the number of workers in the <code>DataLoader</code>  during inference is 1.</p>
<p>please correct me if I’m wrong.</p>

ptrblck: <p>I’m not sure how a <code>DataLoader</code> would work with a real-time inference use case, but in any case no “Memory errors” should be raised, so I would still need more information about the actual error and a minimal code snippet to be able to debug it.</p>
BCE class weightings causing loss instability

azhx: <p>Hi,<br>
I’m using <code>BCEWithLogitsLoss</code> in a multilabel classification problem in the following way:</p>
<pre><code class="lang-auto">return F.binary_cross_entropy_with_logits(logits, targets, 
                      weight = self.freq_weight, pos_weight = self.pn_weight, reduction = 'mean')
</code></pre>
<p>where <code>self.freq_weight</code> is a tensor of weights for each class to balance the relative occurrences of each class and <code>self.pn_weight</code> is the weight of positive examples, as described in the docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">here</a></p>
<p>My intuition is that the extreme magnitude difference between relative class weights produced by the imbalance in my dataset is causing huge loss spikes during training, and preventing the loss from converging. What are some ways to deal with this?</p>

Andrei_Cristea: <p>Something you should definitely consider is to use a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="noopener nofollow ugc">WeightedRandomSampler</a> instead as an alternative to balancing out your classes without running into the exploding loss issue you described.</p>

azhx: <p>one reservation I have with applying a weighted random sample is that samples in my dataset often have multiple labels, and those labels which appear infrequently often also appear with labels that appear frequently. Hence, oversampling instances where the infrequent label appears also would oversample the frequent labels, which could end up being counterproductive.</p>
<p>Maybe there can be some more complex weighted sampling scheme where I decrease the weight of instances that don’t contain infrequent labels also, but this seems like a very convoluted path</p>

Andrei_Cristea: <p>That’s a legitimate reservation, however doing what you just described doesn’t seem that convoluted to me, since <code>WeightedRandomSampler</code> takes in custom weights. One way to come up with some weights would be to rank your classes by frequency (0 is least fequent and 1 is most frequent) and for each datapoint <code>i</code> assign a score that looks like:</p>
<p><code>score_i = product over class k of (1 - rank_k)</code>, for all classes k that appear in datapoint i</p>
<p>Then you use <code>score_i</code> to produce weights to pass to <code>WeightedRandomSampler</code>.</p>
<p>This very crude method will basically reward datapoints that have rare classes but not common classes. You can use this as a fine-tuning step at the end of model training for a few epochs (i.e. you can mostly train your model without this) with a validation set that focuses on the rare classes.</p>
Merging Convolutional & Batch Norm Layer

Amur_Saqib_Pal: <p>Hi. I’m trying to merge a convolutional layer and a batch norm layer so I dont have to create a separate representation of the batch norm layer in an FPGA implementation of the model. I merged the layers by replacing the weights and biases of the preceding convolutional layer with the appropriate transformed weights and biases required for the fusion.</p>
<p>The problem is that I cannot perform in-place operations on leaf-type variables i.e. the weights and biases, so I have to use torch.no_grad() while testing the model. However, that way, I can’t train it and the loss is never changing because the model can’t backpropagate. Any help on this would be appreciated.</p>
Send the output of your network to DataLoader

yukw777: <p>My training regime is set up so that the next batch depends on the output of the network on the previous batch. I’m currently saving the output on the previous batch on the model itself to be used for the next batch, but it doesn’t seem like the correct way to go about this, and I was wondering if there’s a way to send the output back to DataLoader so that I can keep the forward pass simpler. My initial thought was to use <code>IterableDataset</code> and send the output back via Python generator, but this doesn’t seem possible based on my reading of the DataLoader code as there is no way to access the generator created by <code>IterableDataset</code>. Is there a way to do this? Is my current solution of saving the output on the previous batch the best one, or is there a better solution?</p>
How to broadcast this slicing to a 3D tensor?

Marco_D: <p>Hi guys.<br>
Let’s have the following trivial scenario:</p>
<pre><code class="lang-auto">x = torch.randint(0,4, (2,3))
y = torch.randint(0,100, (5, 3))
y[x]
</code></pre>
<p>Here, I simply have two position vectors (x) and want to extract the elements of y for each position vector. In this way, y[x] has dimensionality (2x3x3).<br>
Now, I cannot replicate the same results if we add a third dimension, that is, the batch dimension at the beginning. In this case:</p>
<pre><code class="lang-auto">x = torch.randint(0,4, (4,2,3))
y = torch.randint(0,100, (4, 5, 3))
y[x]
</code></pre>
<p>I expect to broadcast the previous operation for each element in the batch, in order to obtain an array of dimensionality (4x2x3x3), but the output it’s not like that.</p>
<p>Any insight?</p>

Marco_D: <p>Basically, I can obtain the expected result with this:</p>
<pre><code class="lang-auto">torch.cat([y[i][x[i]] for i in range(4)], dim=0).view(4, 2, 3, 3)
</code></pre>
<p>But I don’t want to use a for loop</p>

ejguan: <p>You are indexing the <strong>first dimension</strong> of <code>y</code> using <code>x</code>. With <code>x</code> as shape of (4, 2, 3), each element of x will point to the first dimension of <code>y</code> and select the corresponding data of (5, 3). That’s why the result size should be (4, 2, 3, 5, 3)</p>
Is TorchScript compatible with CUDA Graph?

hyhuang00: <p>I am wondering the relationship between TorchScript and the newly introduced CUDA Graph integration with PyTorch. I tried to use CUDA Graph to accelerate my code, which is traced already, and I observe no speedup in my experiments. The trace between the two settings are almost the same. Is TorchScript compatible with CUDA Graph?</p>

ptrblck: <p>Yes, you can use CUDA graphs on a scripted model.<br>
Are you seeing any performance benefits on the standard model (i.e. before scripting)?<br>
As is explained in the docs, small workloads (e.g. via small batch sizes) would see the largest benefit in CUDA graphs.</p>

hyhuang00: <p>Thank you for your help. I haven’t got a chance to compare the performance with the standard model, but I did make some experiments w.r.t. small batch sizes, ranging from 8 to 64. When my batch size equals to 8, I observed a tiny improvement of about 2% in end to end latency, and such improvement diminishes when the batch size grows larger than 32. The improvement is so small that I don’t know whether it is just deviation or some real improvements.</p>
<p>What makes me more confusing is the trace I collected from my experiments. Here is a screenshot for one the trace I collected. I thought the CUDA graph will bring all the cudaLaunchKernel runtimes together to the very beginning of the model, but from the trace it seems like this is not the case. Each of the operators still has their own cudaLaunchKernel calls, and it takes a while for the process to launch these kernels. I’m happy to provide you with more information regarding the raw trace files if you need them.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7680ee9b7b85b366cce676000be4b5cadc52975.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7680ee9b7b85b366cce676000be4b5cadc52975" title="Screen Shot 2022-06-16 at 11.08.20 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7680ee9b7b85b366cce676000be4b5cadc52975_2_689x178.png" alt="Screen Shot 2022-06-16 at 11.08.20 AM" data-base62-sha1="qaujTqVKtHvf7DPC0KixAcBZLKJ" width="689" height="178" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7680ee9b7b85b366cce676000be4b5cadc52975_2_689x178.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7680ee9b7b85b366cce676000be4b5cadc52975_2_1033x267.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7680ee9b7b85b366cce676000be4b5cadc52975_2_1378x356.png 2x" data-dominant-color="E4DBE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-06-16 at 11.08.20 AM</span><span class="informations">2580×668 92.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Convert online stream of data into graph data(torch_geometric.data.data.Data)

DeepRobo: <p>I have an online stream of data that is coming in, I need to process them and from each data point in my batch, create a torch_geometric style data point. So if my batch has 128 points, I need to convert it into a batch of 128 graphs and pass it to my GNN.</p>
<p>I need to perform this without causing many hindrances in the training loop, so looping over each sample is the last thing one wants to do. I have tried functorch vmap, but since it is s returning non-tensor data type I’m getting an error.</p>
<p>Any help or ideas is much appreciated.</p>

ejguan: <p>Could you please clarify your problem?</p>
<p>Are you asking about converting stream of data to batches, or <code>vmap</code> over a batch Tensor?</p>
Bidirectional LSTM Seq2Seq with Residual connections in decoder

aaaaab: <p>I am trying to build a seq2seq model including bidirectional LSTMs in both encoder and decoder parts. I also want to add residual connections to the decoder part, i.e., to add the input of each LSTMCell to its output. Can anyone tell me how to change the outputs of LSTMCells in the decoder when I’m using the torch.nn.LSTM command in Pytorch? Enclosed you can see a figure of what I want to implement regarding the decoder part. (Assume my model includes 1 layer with 512 cells)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc2168a603c51d109003eaf9598caea03adf694a.png" alt="bi" data-base62-sha1="t7OYSEqYov4HRYUgeFpxOxYPbWa" width="441" height="246"></p>
TransformerEncoderLayer accuary changes in model after calling model.eval()

samuel_fipps: <p>I have model with a nn.TransformerEncoderLayer() with some lstm layers after it. When ever i go to test the model after saving it and loading it in another file, with the TransformerEncoderLayer in the model, depending on the shuffle of the input data I get different accuracy.</p>
<p>If I train the model without that layer the model save’s, loads, and evals() with the same accuracy no matter the order.</p>
<p>Will the TransformerEncoderLayer always give me different accuracy like this? I even printed out the weights of this layer and they all saved correctly.</p>
ESRGAN with Vulkan - Error Vulkan::convolution not available!

Burstaholic: <p>There seems to be some kind of incompatibility between PyTorch’s Vulkan build and the ESRGAN/Real-ESRGAN neural net architecture. I have an AMD card that isn’t supported by ROCm (RX 5700), so is there any way for me to run this?</p>
<p>I’m hoping there’s a fix that doesn’t require me to like, re-invent the ESRGAN architecture.</p>
<pre><code class="lang-auto">Error Vulkan::convolution not available! Reason: The provided (weight, bias, stride, padding, dilation, groups, transposed, output_padding, output_min, output_max) parameters are either invalid individually or their combination is not supported by Vulkan impl.
</code></pre>
<p>Running ‘classic’ ESRGAN has a similar-seeming issue:</p>
<pre><code class="lang-auto">...
File "/home/travis/development/ESRGAN/RRDBNet_arch.py", line 34, in forward
    return x5 * 0.2 + x
RuntimeError: Vulkan binary elementwise ops require channel dimension to be equal!
</code></pre>
<p>I know that this project can be compiled with NCNN to run with Vulkan - what’s the problem with the regular PyTorch implementation?</p>
Speeding up torch.matmul

cong_huang: <p>Hi, I would like to compute the matrix multiplication for two matrices.<br>
Using torch.matmul could get correct result but the speed is slow.<br>
Could you please give me some adavise to speed the matrix multiplication?<br>
I use the following code the measure the time.</p>
<pre><code class="lang-auto">cols = torch.randn(16,57600,1,108).cuda()
local_weight = torch.randn(16,57600,108,3).cuda()
    with torch.no_grad():
        for i in range(10):
            torch.cuda.synchronize()
            t0 = time.time()
            c = cols.matmul(local_weight)#torch.matmul(cols,local_weight)
            torch.cuda.synchronize()
            print(time.time() - t0)
</code></pre>
<p>Calculating the torch.matmul takes 5 ms. Is it possible to make it more faster?</p>

KFrank: <p>Hi Huang!</p>
<aside class="quote no-group quote-modified" data-username="cong_huang" data-post="1" data-topic="154294" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cong_huang/48/45755_2.png" class="avatar"> cong_huang:</div>
<blockquote>
<pre><code class="lang-auto">cols = torch.randn(16,57600,1,108).cuda()
local_weight = torch.randn(16,57600,108,3).cuda()
...
            c = cols.matmul(local_weight)#torch.matmul(cols,local_weight)
</code></pre>
<p>Calculating the torch.matmul takes 5 ms. Is it possible to make it more faster?</p>
</blockquote>
</aside>
<p>This sounds quite fast to me.</p>
<p>Please note that the (two-dimensional) batch of matrix multiplications that<br>
you are performing is quite large.</p>
<p>In your example, one call to <code>matmul()</code> performs just a little under 600<br>
million floating-point operations.  Doing so in 5 milliseconds corresponds<br>
to 120 gigaflops.  That’s very good.</p>
<p>In principle, you could further parallelize you batch of matrix multiplications<br>
across multiple gpus, but whether that would give you improved throughput<br>
for your actual use case would depend on a lot of details.</p>
<p>(As an aside, I can basically reproduce your timings, except that I get about<br>
50 mS per <code>matmul()</code> on my not-very-powerful GeForce GTX 1050 Ti, for a<br>
speed of 12 gigaflops.)</p>
<p>Best.</p>
<p>K. Frank</p>
Batch jaccard similarity calculation

saswat_72444: <p>I have a batch of n 1D vectors (int32) and I wish to calculate the Jaccard similarity of the vector at index 0 with all other (n-1) vectors in one operation. Any hints on how to approach the problem? Thanks in advance</p>

KFrank: <p>Hi Saswat!</p>
<aside class="quote no-group" data-username="saswat_72444" data-post="1" data-topic="154245" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat_72444/48/45962_2.png" class="avatar"> saswat_72444:</div>
<blockquote>
<p>I have a batch of n 1D vectors (int32) and I wish to calculate the Jaccard similarity of the vector at index 0 with all other (n-1) vectors in one operation.</p>
</blockquote>
</aside>
<p>You can use broadcasting (or <code>expand()</code>) to treat the first row of your<br>
batch as a whole batch of copies of the first row.  Then just perform your<br>
desired row-by-row computation.  (This will give you <code>n</code> results, and you can<br>
ignore – or not compute – the first one, if so desired.)  Here’s an example<br>
where we calculate the dot product of each row in the batch with the first row:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; t = torch.nn.init.orthogonal_ (torch.empty (3, 5))
&gt;&gt;&gt; (t * t[0]).sum (dim = 1)
tensor([ 1.0000e+00, -1.4901e-08,  0.0000e+00])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
RuntimeError: stride should not be zero

mdbb: <p>I have just learn PyTorch on MAC for weeks, the issue is I can run the VGG16 on cpu , but I can’t run on mps. the error information listed as follow.</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
Input In [21], in &lt;cell line: 6&gt;()<br>
6 for epoch in range(epochs):<br>
8     since = time.time()<br>
—&gt; 10     epoch_loss, epoch_acc, epoch_test_loss, epoch_test_acc = fit(epoch,model,train_dl,test_dl)<br>
12     train_loss.append(epoch_loss)<br>
13     train_acc.append(epoch_acc)</p>
<p>Input In [19], in fit(epoch, model, trainloader, testloader)<br>
8 for x, y in trainloader:<br>
9     x, y = x.to(device), y.to(device)<br>
—&gt; 10     y_pred = model(x)<br>
11     loss = loss_fn(y_pred, y)<br>
12     optimizer.zero_grad()</p>
<p>File ~/opt/anaconda3/envs/deeplearning/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/opt/anaconda3/envs/deeplearning/lib/python3.9/site-packages/torchvision/models/vgg.py:43, in VGG.forward(self, x)<br>
41 def forward(self, x):<br>
42     x = self.features(x)<br>
—&gt; 43     x = self.avgpool(x)<br>
44     x = x.view(x.size(0), -1)<br>
45     x = self.classifier(x)</p>
<p>File ~/opt/anaconda3/envs/deeplearning/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/opt/anaconda3/envs/deeplearning/lib/python3.9/site-packages/torch/nn/modules/pooling.py:1179, in AdaptiveAvgPool2d.forward(self, input)<br>
1178 def forward(self, input: Tensor) → Tensor:<br>
 → 1179     return F.adaptive_avg_pool2d(input, self.output_size)</p>
<p>File ~/opt/anaconda3/envs/deeplearning/lib/python3.9/site-packages/torch/nn/functional.py:1214, in adaptive_avg_pool2d(input, output_size)<br>
1212     return handle_torch_function(adaptive_avg_pool2d, (input,), input, output_size)<br>
1213 _output_size = _list_with_default(output_size, input.size())<br>
 → 1214 return torch._C._nn.adaptive_avg_pool2d(input, _output_size)</p>

mdbb: <p>the issue doesn’t happend when I use RESNET18.</p>
Getting flat curves when using torch.use_deterministic_algorithms

ayushexel: <p>I’m working on <a href="https://github.com/ultralytics/yolov5" rel="noopener nofollow ugc">YOLOv5</a> repo and we’re trying to make the training runs reproducible. Only setting random seeds doesn’t work so we’re trying to experiment with <code>torch.use_deterministic_algorithms</code>. Using it makes the experiments reproducible in terms of losses but all the validation metrics are always flat at 0.<br>
<a href="https://github.com/ultralytics/yolov5/pull/8213" rel="noopener nofollow ugc">The PR with changes</a><br>
<a href="https://wandb.ai/glenn-jocher/test-reproduce-pr2" rel="noopener nofollow ugc">Dashboard with losses</a> - Notice how all the losses decrease over time but the <code>metrics/</code> section stays flat.<br>
Here’s the init seed function:</p>
<pre><code class="lang-auto">def init_seeds(seed=0):
    # Initialize random number generator (RNG) seeds https://pytorch.org/docs/stable/notes/randomness.html
    # cudnn seed 0 settings are slower and more reproducible, else faster and less reproducible
    import torch.backends.cudnn as cudnn
    torch.use_deterministic_algorithms(True, warn_only=True)
    os.environ['PYTHONHASHSEED'] = str(seed)
    os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    # https://pytorch.org/docs/stable/_modules/torch/cuda/random.html#manual_seed
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)  # for multi GPU. Exception safe
    cudnn.benchmark, cudnn.deterministic = (False, True) if seed == 0 else (True, False)
</code></pre>
How to improve inference speed?

iamexperimentingnow: <p>Hi Everyone,</p>
<p>Can anyone please recommend to me what is an efficient way to reduce the inference timing? Currently, my model takes 50sec to do the inference.</p>
<p>Objective: I have built the flask API and it takes 50sec to do the inference. So, I would like to reduce the inference timing and return the result from the flask app.</p>
<p>This is basically for real-time inference, I want to pass a single image to the flask API for inferencing.</p>
<p>system details:<br>
using GPU p2.xlarge instance<br>
Ubuntu 18.04</p>
<p>Steps were already taken:</p>
<ol>
<li>attached model to GPU</li>
</ol>
<p>Note:<br>
Framework: PyTorch<br>
Application: semantic segmentation model</p>
<p>It would be really helpful if anyone recommends an efficient way to optimize.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you please help me.</p>

ptrblck: <p>I think it would be interesting to see which part of your full inference pipeline takes which amount of time. E.g. is the model itself using the majority of the 50s or is it the flask application?<br>
Based on this you could then check how to improve your use case e.g by speeding up the model itself via the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a> or by switching to <code>torchserve</code>, <code>Triton</code>, or any other inference serving application.</p>

iamexperimentingnow: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="153886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>s it the flask application?</p>
</blockquote>
</aside>
<p>firstly, thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a>. I did a check and I noticed model inference takes about 30sec and flask takes 20sec to convert the ndarray to list, json.dumps and return the result out of the application.</p>
<p>Now, I managed to reduce 20 sec to around 5sec in converting ndarray to list and json.dumps.</p>
<p>I’m literally trying to reduce the inference time.</p>
Loss showing NaN in 2nd epochs

Mohammed_Rizin: <p>I was training a model with images and loss Binary crossentropy.<br>
The loss seems to be genuine in 1st epoch. But in 2nd epoch after some amount of batches only it will raise nan loss but the accuracy increases. also i am using amp.</p>
<pre><code class="lang-auto">loss: 6.89388 ; accuracy: 0.72175: 100%|██████████| 3010/3010 [1:19:28&lt;00:00,  1.58s/it]
loss: 0.56813 accuracy: 0.84400: 100%|██████████| 751/751 [05:47&lt;00:00,  2.16it/s]

loss: nan ; accuracy: 0.84616: 100%|██████████| 3010/3010 [1:13:44&lt;00:00,  1.47s/it]
loss: nan accuracy: 0.84765: 100%|██████████| 751/751 [05:46&lt;00:00,  2.17it/s]
</code></pre>
<p>and when calculating AUC with 2nd epoch validation data<br>
<code>ValueError: Input contains NaN, infinity or a value too large for dtype('float32').</code></p>

shivammehta007: <p>There could be many reasons why this could be the case,<br>
I will suggest to turn on the flag <code>torch.autograd.set_detect_anomaly(True)</code><br>
This will drop in error wherever the autograd engine encounters <code>NaN</code> and give you the function where the error occurred, It could be advantageous to debug and see what is going on!</p>

Mohammed_Rizin: <p>I found what the error was.<br>
It was caused due to batch normalization</p>

hongxing_zhang: <p>can you explain in detail ？ I have the same problem. thanks</p>
Resources to understand PyTorch source code

sky-2002: <p>Hello ! I wanted to ask if there are any good resources (may include books, articles, blog posts, videos, courses etc) that discuss the implementation details and source code explanations(of PyTorch, example - how to get started with the source code) for beginners. Please let me know if there are any <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20">. Asking this because newbies get scared looking at such a large code base and it feels daunting to contribute (but there is just so much willingness to contribute).</p>
Training on a tensors without common DataLoader

Allukard: <p>I am sorry if my question sounds too noobish. I am beginner both in python and in torch.<br>
So I managed to iterate through my dataset by a custom way and created a function which is resulting<br>
me an a = InputDataTensor, OutputDataTensor<br>
My question is how to train a model based on usage of my function which is on every call gives me new input data of tensor[10,1] and corresponding output data of [100,1] (sizes I will adjust in future).<br>
The thing is that my dataset is just an input and corresponding output. I am not working with any prebuilt datasets from torch and guides on dataloaders are just confusing me with always working over MNIST.</p>

nivek: <p>Have you had the chance look at the <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html#optimizing-the-model-parameters" rel="noopener nofollow ugc">Quickstart guide</a>? This section shows how you can pass in data to train your model.</p>

Allukard: <p>This guide is showing work with dataloaders and MNIST, which I wanted to avoid. Thanks anyway. I am thinking now towards assembling conventional dataset from my func. so I just can swith to dataloaders.</p>

nivek: <p>If you do not wish to use the <code>DataLoader</code>, you can replace <code>enumerate(dataloader)</code> from the tutorial with your own iterator object.</p>
<p>Let me know if there is any issue with that.</p>

Allukard: <p>Thanks. I have a solution now. I just automated creation of a guide-like dataset from my data and now I just follow guides with a regular dataloader which is able now to load data I need. It sounds so simple, but was a hard to come up with for such a newibe like me.</p>
ETA for 1.12 with MPS support :)

ludwigwinkler: <p>Does anybody of the devs have an estimated time of arrival (ETA) for the stable 1.12 version? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Rough estimate would be much appreciated.</p>

ptrblck: <p>We are now in <a href="https://github.com/pytorch/pytorch/issues/78005">Phase 2 of the <code>1.12.0</code> release</a> and are focused on critical cherry-picks, stability testing etc. so are close to the release. You can track the progress in the same tracker.</p>

ludwigwinkler: <p>Yippie! Thanks for the info and thanks for the awesome work all of you are doing! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>