akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I have added class weights in CrossEntropyLoss, but accuracy reduced

Z_Huang: <p>I have added class weights in CrossEntropyLoss, but I figured the prediction accuracy has reduced. Any idea? Did I use class weights inappropriately in the loss function?</p>
<pre><code class="lang-auto">class_weights = compute_class_weight(
    class_weight = 'balanced',
    classes = np.unique(train_loader.dataset.labels),
    y = np.ravel(train_loader.dataset.labels))

class_weights = [ 1., 83.36, 90.92]

loss_function = torch.nn.CrossEntropyLoss(weight=torch.Tensor(class_weights)).to(device)
</code></pre>

ptrblck: <p>You might be running into the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">accuracy paradox</a> if your dataset is imbalanced. E.g. if class0 occurs in 99% of all samples, a simple <code>return 0</code> function would already achieve a 99% accuracy but would be quite useless.</p>
Why do torchvision models have different prediction layer's name?

SashimiDelicieux: <p>I wanted to make a function that take any torchvision model as argument and output the same model with a different output size, but I realized that a simple but annoying problem prevented me from directly doing so : The prediction layers of those models have different names. For example, the last layer of Resnet and Inception is called “fc”, while the last layer of AlexNet and VGG is called “classifier”.</p>
<p>I bypassed this problem by using <code>hasattr(model, "fc")</code>, but it seems a bit odd to me.</p>
<p>Is there a reason for this difference ? Wouldn’t be better to just have one name ?</p>

ptrblck: <p>I guess it would have been better to use a specific naming convention, but since the models were implemented in different releases, by different authors, each with (slightly) different coding styles, the layer names might differ and your check sounds like a good approach to check for the classifiers.</p>
Using nn.Conv1D when batch size varies in the loader

banikr: <p>I am using an input with size <code>batchsize x nFeatures</code>.<br>
The data loader is as follows:</p>
<pre><code class="lang-auto">train_loader = DataLoader(peakspectra_tic, batch_size=bSize, shuffle=True, num_workers=1, drop_last=False)
</code></pre>
<p>And the conv layer is:</p>
<pre><code class="lang-auto">    def convlayer_(self, inchannels, outchannels, outputsize):
        layer = nn.Sequential(
            nn.Conv1d(in_channels=inchannels, out_channels=outchannels, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm1d(num_features=outputsize),
            nn.ReLU()
        )
        return layer
</code></pre>
<p>When the batch size is equal to <code>bSize</code> it runs but in the last batch(drop last) which is not equal it runs into error.<br>
How to deal with this?</p>

ptrblck: <p>All layers are accepting a variable batch size, so I assume:</p>
<aside class="quote no-group" data-username="banikr" data-post="1" data-topic="164024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>I am using an input with size <code>batchsize x nFeatures</code>.</p>
</blockquote>
</aside>
<p>means you are using the actual batch size as the number of channels?<br>
If so, use <code>drop_last=True</code> in your <code>DataLoader</code> to drop the last (smaller) batch since your model depends on the fixed batch size and will fail otherwise (note that this approach is also uncommon since your inference use case would also depend on the same fixed batch size).</p>
Treat MNIST data set as a regular data set

kkladny: <p>I want to perform some pre-selection on the data of the MNIST data set. However, I notice that the data set does not behave the way I would like to. I load it like</p>
<pre><code class="lang-auto">train_data = MNIST('./files/', train=True, download=True,
                    transform=torchvision.transforms.Compose([
                    torchvision.transforms.ToTensor(),
                    torchvision.transforms.Normalize((0.1307,), (0.3081,)), 
                ]))
</code></pre>
<p>For example, trying to select the first five examples like</p>
<p><code>train_data[:5]</code></p>
<p>gives the error</p>
<pre><code class="lang-auto">ValueError: only one element tensors can be converted to Python scalars
</code></pre>
<p>So, to circumvene this, I create a new data set like</p>
<pre><code class="lang-auto">train_data = TensorDataset(torch.unsqueeze(train_data.data, dim=1), train_data.targets)
</code></pre>
<p>which does not seem very elegant, but it makes my data set behave like I want it to. However, I notice that this way all the transformations I applied to the original data set are lost. What can I do to make the data set behave like I want to and not loose the transformations? I know there exists this <code>Subset</code> class, however I do not like to use that either since I want to also modify the data later. For the later transformation I cannot just create a <code>transforms</code> function since the transformation will also depend on the label which the <code>transform</code> argument to <code>MNIST</code> does not allow me to do.</p>

ptrblck: <p>You would need to index the <code>Dataset</code> directly e.g. via a loop:</p>
<pre><code class="lang-python">for i in range(5):
    data, target = train_data[i]
</code></pre>
<p>instead of a slice.</p>
Matmul operation very slow

Py_son: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3c9766bafadf05dfb8205b35267a409c9c6bb88.png" alt="image" data-base62-sha1="yMDEJ9sipUQKRLqx48f6qFJBiKk" width="650" height="29"></p>
<p>Hey guys. The matmul operation I am performing is taking around 30-45 seconds just to perform an operation ( I am currently on CPU). Is this normal? I am not really to sure why it is taking so long. It is a fairly hefty operation but I did not expect it to take this long.</p>

ptrblck: <p>Could you post the shapes of both tensors, please?</p>
Tensor outcome after operation still a lead node

Yoo: <p>I could not get through z.backward() as it showed “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”</p>
<blockquote>
<p>a0 = torch.FloatTensor([1]).clone().detach()<br>
a0 = a0.to(device)<br>
a0.requires_grad_ = True # create a leaf tensor<br>
z = a0 * 2<br>
opt = torch.optim.SGD([a0], lr = 0.01)<br>
z.backward()<br>
opt.step()</p>
</blockquote>
<p>I believe a0 is a leaf node in device and should have computed a0.grad. After rounds of checking I found z was still a leaf node, why was that?</p>

ptrblck: <aside class="quote no-group" data-username="Yoo" data-post="1" data-topic="164428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yoo/48/53396_2.png" class="avatar"> Yoo:</div>
<blockquote>
<p>a0.requires_grad_ = True # create a leaf tensor</p>
</blockquote>
</aside>
<p>This call is wrong, since <code>.requires_grad_</code> is a function and not an attribute, so you are overriding it.<br>
Use:</p>
<pre data-code-wrap="python"><code class="lang-python">a0.requires_grad_()
</code></pre>
<p>and it should work.</p>
Detected that PyTorch and torchvision were compiled with different CUDA versions

ilayasis: <p>Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=11.5 and torchvision has CUDA Version=11.3. Please reinstall the torchvision that matches your PyTorch install.</p>

ptrblck: <p>Did you try to reinstall PyTorch and/or <code>torchvision</code> using the same CUDA runtime as explained in the error message? If so, could you describe where you are stuck?</p>
Time to init an optimizer

Yoo: <p>Suppose I define a tunable tensor. In order to tune it in CUDA, I have 3 steps:</p>
<ol>
<li>move it to CUDA</li>
<li>add <code>.requires_grad_()</code> to it</li>
<li>init an optimizer for it</li>
</ol>
<p>Is my order right? Can I switch the step order?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/yoo">@Yoo</a>,</p>
<p>I think you can swap step 1 and 2 around, but you can’t swap step 3 as if you had the parameters on the cpu and initialized them in an optimizer, then moved them to <code>CUDA</code> then your update (<code>.grad</code> attribute) would be on the GPU but the parameters would be on the cpu.</p>
<p>Also, sharing a minimal reproducible example will help illustrate your problem more easily!</p>

Yoo: <p>I tried three orders:</p>
<blockquote>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
<h1>1</h1>
<p>a0 = torch.FloatTensor([1]).clone().detach().requires_grad_(True) # require_grad<br>
print(“leaf”, a0.is_leaf)<br>
a0 = a0.to(device) # move to CUDA<br>
print(“leaf”, a0.is_leaf)<br>
opt = torch.optim.SGD([a0], lr = 0.01) # init opt</p>
<h1>2</h1>
<p>a0 = torch.FloatTensor([1]).clone().detach()<br>
print(“leaf”, a0.is_leaf)<br>
a0 = a0.to(device) # move to CUDA<br>
a0.requires_grad_ = True # require_grad<br>
print(“leaf”, a0.is_leaf)<br>
opt = torch.optim.SGD([a0], lr = 0.01) # init opt</p>
<h1>3</h1>
<p>a0 = torch.FloatTensor([1]).clone().detach().requires_grad_(True) # require_grad<br>
print(“leaf”, a0.is_leaf)<br>
opt = torch.optim.SGD([a0], lr = 0.01) # init opt<br>
a0 = a0.to(device) # move to CUDA<br>
print(“leaf”, a0.is_leaf)</p>
</blockquote>
<p>Case 1 gave error since .to(device) made the tensor differentiable.<br>
Case 2 and 3 worked but I was not sure whether the optimizer in 3 knew to update a0 in device instead of in CPU?</p>
C++ BatchNorm results differed in Python results

HyungJoo-Kwon: <p>Hi, There was a problem while I was converting the code. (python to c++)<br>
C++ BatchNorm results differed in Python results.</p>
<pre><code class="lang-auto"># Python 
class My_model(nn.Module):
    def init(self):
        super(My_model), self).__init__()
        self.conv1 = torch.nn.Conv1d(15, 64, 1)
        self.bn1 = nn.BatchNorm1d(64)
   def forward(self, x):
        x = self.conv1(x)        # result is same in python and C++
        x = self.bn1(x)            # result is different in python and C++
        print(x)
</code></pre>
<pre><code class="lang-auto">//C++
class My_model : public torch::nn::Module{
public:
   torch::Tensor forward(torch::Tensor x);
private:
    torch::nn::Conv1d conv1 = torch::nn::Conv1d(torch::nn::Conv1dOptions(15, 64, 1));   // result is same in python and C++
    torch::nn::BatchNorm1d bn1= torch::nn::BatchNorm1d(torch::nn::BatchNorm1dOptions(64));   // result is different in python and C++
}
My_model model = My_model();
model.forward(x);
</code></pre>
<p>I checked same Input x.<br>
Python model weight and model bias reflected in the weight and bias of the C++ model.<br>
Is there a solution or something mistake?</p>

ngageorange: <p>It’s likely related to the initialization of the conv1 layer. Ensure that the manual seed is the same on both versions if you’re not loading in the weights of a pre-trained network.</p>

HyungJoo-Kwon: <p>Thank you for your reply.<br>
I make pre-trained model in python and save .pt file including model_state_dict.<br>
And using .pt file</p>
<pre><code class="lang-auto">// C++
My_model model = My_model();
auto container = torch::jit::load(".pt path");
for (auto p : model.named_parameters().keys())                 // pt 데이터 가중치 모델에 넣기
	model.named_parameters()[p].data() = container.attr(p).toTensor();
</code></pre>
<p>I loaded the weights with the code above and can checked loaded the weights in C++…<br>
Using torch.manual_seed(0),  torch::cuda::manual_seed_all(0);<br>
And the above procedure was performed again, but the results were the same.</p>

HyungJoo-Kwon: <p>Finally I find it.<br>
In Test<br>
python : nn.BatchNorm1d(64)<br>
c++ : torch::nn::BatchNorm1d(torch::nn::BatchNorm1dOptions(64).track_running_stats(false);</p>
Predicted data is in a different scale

eyohadrian: <p>I’m trying to predict the sine wave with RNN. Even if the out put learns the trend of the curve, the output seems it is in other scale as you can see in the next plot:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea52f54ff88174887c0ae1b62e5b1bbc48d1f652.png" alt="image" data-base62-sha1="xqVsQGB4JWgnBjmhy5CxAkMnpnk" width="391" height="256"></p>
<p>Then a couple of questions jumped out of my mind:</p>
<ol>
<li>Since the range of the input is from 1 to -1, is it necessary to normalise?</li>
<li>Changing the optimizer from SGD to Adam changed the scale of the output drastically like from 4.5 to -4.5. I can’t understand why. (I wanted to post a second image showing it but I can’t since I’m a new user)</li>
</ol>
<p>I think I’m missing a super basic concept on Machine Learning and some Data Science skills as well to tackle this problem so any help would be greatly appreciated.</p>
<h1><a name="data-1" class="anchor" href="#data-1"></a>Data</h1>
<pre><code class="lang-auto">wave = np.sin(np.arange(0,8*np.pi,0.1))
wave = torch.from_numpy(wave).repeat(3).float()
</code></pre>
<h1><a name="model-2" class="anchor" href="#model-2"></a>Model</h1>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(RNN, self).__init__()

        self.rnn = nn.RNN(input_size=input_size, hidden_size=hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, 1)

    def forward(self, input, hidden):
        output, hidden = self.rnn(input, hidden)
        output = self.fc(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)
</code></pre>
<h1><a name="hyperparameters-3" class="anchor" href="#hyperparameters-3"></a>Hyperparameters</h1>
<pre><code class="lang-auto">model=RNN(input_size=4, hidden_size=256)
learning_rate = 1e-2
epochs=100
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
loss_fn = nn.MSELoss()
</code></pre>
<h1><a name="training-loop-4" class="anchor" href="#training-loop-4"></a>Training loop</h1>
<pre><code class="lang-auto">min_loss=np.inf
model.train()
loss_train=0.0
for epoch in range(0, epochs):
  
  hidden = torch.zeros(1, 256)

  optimizer.zero_grad()
  x, hidden = model(train.view(train.shape[0] // 4, 4), hidden)
  loss = loss_fn(x[-1], test[0].unsqueeze(0))

  loss.backward()
  optimizer.step()
  loss_train+=loss.item()

  if epoch%10 == 0:
        print('Epoch: {}/{}.............'.format(epochs, epoch), end=' ')
        print("Loss: {:.4f}".format(loss.item()))
</code></pre>
<h1><a name="evaluation-5" class="anchor" href="#evaluation-5"></a>Evaluation</h1>
<p>This is how I generate the output which I’ve shown before.</p>
<pre><code class="lang-auto">src = train[-4:].unsqueeze(0)
results = []

hidden = torch.zeros(1, 256)

model.eval()
with torch.no_grad():
  for i in range(0, test.size()[0]):
    src = torch.cat((train[i+1::], test[:i+1])).clone()
    o, h = model(src.view(src.shape[0] // 4, 4), hidden)
    results+=o
</code></pre>
<p>Thank you.</p>
Data labelling for two objects in one image file | Object detection

smile_0: <p>Hello all,</p>
<p>greetings! I have a query regarding object detection label data. Currently, I have trained data of images with objects such as cat and dog (both in one image). I have information such as (x_min_cat, y_min_cat, x_max_cat, y_max_cat, x_min_dog, y_min_dog, x_max_dog, y_max_dog) for these objects. I assume class_name and _id as cat,0 and dog,1 respectively…</p>
<p>I am writing the csv file in format |image_id|class_name|class_id|x_min|y_min|x_max|y_max|. since I have two objects in one image, I would like to ask, how one suppose to put label data information such as (class_name, class_id, x_min, y_min, x_max, y_max) for two objects in/under one image_id?</p>
<p>thank you in advance…</p>
<p>Kind regards</p>
How to make a DataLoader shuffle per batch

nestiank: <p>I know that a DataLoader will shuffle per epoch, to avoid duplicate sampling.</p>
<p>However, I need my DataLoader to shuffle per batch, to allow duplicate sampling.</p>
<p>Is this possible without a Sampler?<br>
Is this possible with a Sampler, or not possible even with a Sampler?</p>

nivek: <blockquote>
<p>However, I need my DataLoader to shuffle per batch, to allow duplicate sampling.</p>
</blockquote>
<p>I assume this means you would like to sample <code>n</code> times with replacement for a given batch of <code>n</code> size. You should be able to write your own <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.BatchSampler" rel="noopener nofollow ugc"><code>BatchSampler</code></a> to do that and pass your custom implementation to <code>DataLoader</code>.</p>
<p>Alternatively, you can write a <code>collate_fn</code> that re-samples and collates the batch, and pass that <code>collate_fn</code> to <code>DataLoader</code>.</p>
The result using CNN

kamilian: <p>Hi, when the train the CNN, the result of accuracy equal to 99, 40%,</p>
<p>And the curve loss  is</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25a5505542dee7723014066831bcdd78261c7f59.png" alt="image" data-base62-sha1="5n1QFPv5NDp8utDk9zaA7wsbjN7" width="360" height="377"></p>
<p>Is that right which gave the network?</p>

ptrblck: <p>Your description isn’t sufficient to understand which issue you are facing, so please feel free to add more details and describe where exactly you are stuck.</p>

kamilian: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  the curve of the loss is just?</p>
Increasing Magnitude of Gradients

fabian_schutze: <p>Hi,</p>
<p>I am training a 3d object detection network and I realize that the magnitudes of the gradients are increasing at the lower layers (input layers) of the networks. I use a Resnet50 network as a backbone, a FPN and then custom heads. I use BN in all layers of the network. The network has two input modalities, depth and rgb, which I ingest via two resnet-stems.</p>
<p>The mean and std of the activations are shown in the graph below: (the means on the left graph, the stds on the right graph).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff5070ce7ef83b8570a15471a2af50bc116cb89f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff5070ce7ef83b8570a15471a2af50bc116cb89f" title="activations"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff5070ce7ef83b8570a15471a2af50bc116cb89f_2_690x362.png" alt="activations" data-base62-sha1="AqC65J6uIGuyzLxYTo1mke19Mwn" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff5070ce7ef83b8570a15471a2af50bc116cb89f_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff5070ce7ef83b8570a15471a2af50bc116cb89f_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff5070ce7ef83b8570a15471a2af50bc116cb89f_2_1380x724.png 2x" data-dominant-color="F7F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">activations</span><span class="informations">1920×1009 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As can be seen, the gradients in the higher layers (towards the end of the network) have a comparatively low stds and mean, while the lower layers have a high std and (absolute) mean.</p>
<p>Does somebody know how I could have a more balanced gradient distribution? Would that even be desirable? I am using the default activations of pytorch.</p>
How to put tensors in tensor?

MikeMACintosh: <p>Hi, it’s a possibly the funniest question here but i stumbled.<br>
I have such tensors:<br>
image = torch.randn(1, 3, 224, 224)<br>
coords = torch.randn(1, 2)</p>
<p>and want to get mapping in the form of a tensor:<br>
mapping = torch.tensor((image, coords))<br>
That through mapping[0] get a picture and mapping[1] get a coords but obviously i have ValueError: only one element tensors can be converted to Python scalars.</p>
<p>Is there correct way to do this?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/mikemacintosh">@MikeMACintosh</a>,</p>
<p>I don’t think this is possible because they have different sizes, if they were both the same size you could use <code>torch.cat</code> or <code>torch.stack</code> to represent them as a single tensor. Can you share more about your use case?</p>

MikeMACintosh: <p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a><br>
Thanks for reply, yes i can’t use cat/stack because tensors have a different shape.<br>
I have many images and corresponding coordinate.<br>
I create a tensor from image and feed them to DINO model to get image embedding.<br>
Next i do some image manipulation and i want to be able to get the coordinates for the corresponding image tensor.<br>
Moreover, I want to be able to save embeddings to disk via torch.save(‘image_embedding.pth’) and later do torch.load(‘image_embedding.pth’) to continue working with them.</p>
<p>Right now I only save image embeddings, but I want to have an image-to-coordinate mapping and be able to save it to disk and load later.</p>

AlphaBetaGamma96: <p>If you’re wanting to save batches of images/coorindates, you could use <code>torch.stack</code> for all your images and all your coordinates separately. Then save the pair of tensors in a tuple via <code>torch.save</code>?</p>
Getting crashes when running on older (non-AVX) CPUs

davidw82: <p>Hi,<br>
I’m the developer of a plugin to the VFX compositing software Nuke. We are using libtorch internally in the plugin to run model inferences. libtorch and all relevant dependencies are statically linked into our dynamic library binary. It’s all working well, both using CUDA or CPU.<br>
But, we’ve encountered problems when it’s run on older CPUs that do not support the AVX and AVX2 instruction sets. It’s instantly crashing.<br>
I have read this page: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/README.md" class="inline-onebox" rel="noopener nofollow ugc">pytorch/README.md at master · pytorch/pytorch · GitHub</a><br>
It says that PyTorch should dynamically choose the correct CPU path (in this case the non-AVX/AVX2 one) during runtime if the following environment variable is set: ATEN_CPU_CAPABILITY=default<br>
However, we’re not seeing any difference in behaviour when we’ve set it, i.e. it’s still crashing.</p>
<p>Is there a limitation to this approach here, i.e. that it’s not working since libtorch is statically linked to my binary. Or is there some other problem? Should it just be working when the environment variable above is defined?</p>
<p>All the best,<br>
David</p>

davidw82: <p>Anybody? Somebody got to have some insight into this matter. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Maybe you can help me by pointing to the right person <a class="mention" href="/u/ptrblck">@ptrblck</a>, you’ve been so helpful in the past. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I would assume that a non-vectorized code path should be taken instead of a failure, so please feel free to create a GitHub issue for your use case.<br>
In the meantime, you might want to disable AVX for a special build e.g. as described in <a href="https://discuss.pytorch.org/t/how-to-build-pytorch-without-avx/158408/3">this post</a> for the non-AVX CPU workstations.</p>
Dataset from dictionary of images

timdnewman: <p>Hi,</p>
<p>I feel like this should be easy but I’m actually at a loss of where to start (I’m recovering from C19 so I’ll blame that if I’m being especially slow).</p>
<p>I’ve trained my models and used the data sets from folders fine it all makes sense.</p>
<p>For inference, the images I’m using are huge microsopy images (30-100k x 4-8K pixels) and not pre-split (which had to be done to label them for training)</p>
<p>I can easily split them into a dictionary of PIL images (~30K images) that lives comfortably in memory.</p>
<p>Is there any way to use that as the input to a data loader - so I can use transforms and do the inference without saving the intermediate files?</p>
<p>Thanks in advance!</p>

nivek: <p>You can create a custom <code>Dataset</code> similar to <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#creating-a-custom-dataset-for-your-files" rel="noopener nofollow ugc">this example</a>, where you can perform transformation within <code>__getitem__</code> and return the result without saving them.</p>

timdnewman: <p>Thanks <a class="mention" href="/u/nivek">@nivek</a> - that does make it a little clearer.</p>
<p>I think I can guess and hope I don’t blow anything up off the back of that. If I get it working I’ll post it back here so there is an actual example somewhere.</p>

timdnewman: <p>Realized I never did do this!</p>
<pre><code class="lang-auto">class DataFromDict(Dataset):
    def __init__(self,input_dict ):
        self.input_dict = input_dict
        self.input_keys = list(input_dict.keys())

    def __len__(self):
        return len(self.input_keys)

    def __getitem__(self,idx):
        item = self.input_dict[self.img_keys[idx]]['item_key']
        label = self.input_dict[self.img_keys[idx]]['label_key']
        return item, label
</code></pre>
<p>That’s not exactly how I did it since I didn’t need the label and so abused it by returning something slightly different - but that should work.</p>
<p>The list of keys shouldn’t be needed, I think, in python 3.7+ but means it is fixed. This should work as a starter for 10 when anyone comes looking for a similar problem.</p>
Too much c++ API overhead

junjihashimoto: <p>In the following example, it takes 900ns just to add 4 numbers.<br>
Is there a way to reduce this time?<br>
c++ without torch can calculate in less than 1ns.</p>
<pre data-code-wrap="c++"><code class="lang-nohighlight">#include &lt;benchmark/benchmark.h&gt;
#include &lt;torch/torch.h&gt;

static void BM_TorchAdd(benchmark::State&amp; state) {
  torch::NoGradGuard no_grad;
  torch::Tensor tensor = torch::rand({2, 2});
  for (auto _ : state)
    auto v = tensor + tensor;
}
// Register the function as a benchmark
BENCHMARK(BM_TorchAdd);

static void BM_SimpleAdd(benchmark::State&amp; state) {
  float tensor[4];
  float buf[4];
  for (auto _ : state)
    for (int i=0;i&lt;4;i++)
      buf[i] = tensor[i] + tensor[i];
}
// Register the function as a benchmark
BENCHMARK(BM_SimpleAdd);

BENCHMARK_MAIN();
</code></pre>
<pre><code class="lang-bash">$ g++ --version
g++ (Ubuntu 11.2.0-19ubuntu1) 11.2.0
$ cat /usr/build-version # The version of libtorch
1.11.0+cu113
$ g++ -O  libtorch-bench.cpp -ltorch -lc10 -lbenchmark -ltorch_cpu
$ ./a.out
2022-10-25T23:04:51+09:00
Running ./a.out
Run on (16 X 4890 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x8)
  L1 Instruction 32 KiB (x8)
  L2 Unified 512 KiB (x8)
  L3 Unified 16384 KiB (x1)
Load Average: 0.27, 0.46, 0.57
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
-------------------------------------------------------
Benchmark             Time             CPU   Iterations
-------------------------------------------------------
BM_TorchAdd         894 ns          894 ns       785041
BM_SimpleAdd      0.309 ns        0.309 ns   1000000000
</code></pre>
How to obtain model gradients with amp?

fabian_schutze: <p>I occasionally like to inspect the gradients of a model but since I use amp I do not know anymore how to do that. Can you kindly help me how to obtain the scaled gradients?</p>
<p>I usually do:</p>
<pre><code class="lang-auto">            grads = []
            for p in model.parameters():
                if (p.grad is None) or (not p.requires_grad):
                    continue
                else:
                    grads.append(p.grad.detach())
</code></pre>
<p>With amp, I do the following to train the model:</p>
<pre><code class="lang-auto">        self._scaler.scale(loss).backward()
        assert parameters is not None
        self._scaler.unscale_(optimizer)  # unscale the gradients
        torch.nn.utils.clip_grad_norm_(parameters, clip_value, norm_type=norm_type)
        self._scaler.step(optimizer)
        self._scaler.update()
</code></pre>
<p>Can somebody tell me how I can get the gradients of these model parameters in this case after I have scaled them? If I iterate again over <code>parameters</code> can I again get each gradient? Will it be scaled?</p>

ptrblck: <p>You can access the gradients in the same way as shown in your first approach after calling <code>unscale</code> manually as also shown in your second code snippet.</p>

fabian_schutze: <p>Thank you very much for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a> , which was very helpful indeed to me.</p>
How do I convert `nn.ModuleList()` to ordinary python `list`?

1chimaruGin: <p>I have</p>
<pre><code class="lang-auto">ModuleList(
          (conv1): ConvKXBN(
            (conv1): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (conv2): ConvKXBN(
            (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
            (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (conv3): ConvKXBN(
            (conv1): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
)
</code></pre>
<p>I want -</p>
<pre><code class="lang-auto">[ConvKXBN(
  (conv1): Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn1): BatchNorm2d(32, eps=1e-05, momentum=1.0, affine=True, track_running_stats=True)
), ConvKXBN(
  (conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(32, eps=1e-05, momentum=1.0, affine=True, track_running_stats=True)
), ConvKXBN(
  (conv1): Conv2d(32, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn1): BatchNorm2d(128, eps=1e-05, momentum=1.0, affine=True, track_running_stats=True)
)]
</code></pre>

ptrblck: <p>Creating a <code>list</code> directly should work:</p>
<pre><code class="lang-python">modules = nn.ModuleList([
    nn.Linear(1, 2),
    nn.Linear(2, 3),
    nn.Linear(3, 4)
])
print(modules)
# ModuleList(
#   (0): Linear(in_features=1, out_features=2, bias=True)
#   (1): Linear(in_features=2, out_features=3, bias=True)
#   (2): Linear(in_features=3, out_features=4, bias=True)
# )

l = list(modules)
print(l)
# [Linear(in_features=1, out_features=2, bias=True), Linear(in_features=2, out_features=3, bias=True), Linear(in_features=3, out_features=4, bias=True)]
</code></pre>
Semantic Segmentation with Captum, Feature Ablation

Sega: <p>Dear all,</p>
<p>when reading through the <a href="https://captum.ai/tutorials/Segmentation_Interpret" rel="noopener nofollow ugc">segmentation tutorial</a> I noticed that the <em>logits</em>, corresponding to pixels that are predicted to be a particular class, are summed. Since logits are unbounded, I was wandering why the output is not normalized by means of a softmax prior summation.</p>
<p>Many thanks in advance!</p>
Is get_signature or parse_type_line abandoned?

lyutoon: <p>As title, can someone please tell me that were <code>get_signature</code> and <code>parse_type_line</code> abandoned in the latest version of pytorch? If not, why I cannot find any caller of these two functions. Or is there any way for to trigger one of these two functions by a normal way not just call <code>torch.jit.annotations.xxx</code> such as using <code>@torch.jit.script</code> or something else? If they are abandoned, why they were not removed from the source code.</p>
<p>Hope to get reply! Thanks a lot!</p>
How to save and load specific layers of yolov5, to aplly in diferent heads during inference?

aspiranteaphd: <p>I have trained a yolov5 (Model A), and I want to train another model (Model B) using the backbone of “A”, and a specific head.</p>
<p>Example problem.<br>
I have a dataset of traffic scenes that include different types of vehicles. For example, the same image can have a class(Super category) of “Car” and class (category) of “BMW” or “Mercedes”[Model A], another image can have a class (Super category) of “Motorcycle”, and class of “Honda” or “Yamaha”[Model B].</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7be955b89608ce7aaf4c77c1ebce124f1777b38.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7be955b89608ce7aaf4c77c1ebce124f1777b38" title="imagem"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7be955b89608ce7aaf4c77c1ebce124f1777b38.png" alt="imagem" data-base62-sha1="x46CmBNvJmBogCiaJ8sG0dhPwRa" width="690" height="239" data-dominant-color="FAF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imagem</span><span class="informations">805×279 3.99 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>–<br>
Since the input images and therefore most of the features should be the same, I wish to train one model then freeze it’s feature-extractor, and sort of transfer learn the same model for the categories using the pretrained feature extractor.</p>
<p>During inference, I want to load Model A and do its prediction → Load Model B Head → Swap “Model A” head with “Model B” → Do the prediction of secondary model.</p>
<p>All of this process has the objective of saving computational resources, since the weights of the feature extracting backbone is the same, I only want to save the weights of the head B.</p>
Error: "caffe2::cuda" is ambiguous

jiapei100: <p>I’m trying to build <strong>pytorch</strong> with <strong>cuda-11.8</strong> on <strong>Ubuntu 22.04</strong>, but obtained the following <strong>ERROR</strong> messages:</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">....../pytorch/caffe2/core/context_gpu.cu(564): error: "caffe2::cuda" is ambiguous

....../pytorch/caffe2/core/context_gpu.cu(564): error: expected a ";"

....../pytorch/caffe2/core/context_gpu.cu(568): warning #12-D: parsing restarts here after previous syntax error

....../pytorch/caffe2/core/context_gpu.cu(569): error: "caffe2::cuda" is ambiguous

....../pytorch/caffe2/core/context_gpu.cu(628): error: "caffe2::cuda" is ambiguous
</code></pre>
<h3>
<a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy
➜  ~ python --version
Python 3.10.6
➜  ~ gcc --version
gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

➜  ~ 
</code></pre>

jiapei100: <p>Please refer to <a href="https://github.com/pytorch/pytorch/pull/87683" rel="noopener nofollow ugc"> error: “caffe2::cuda” is ambiguous</a></p>
Loss and Accuracy calculation

Lane: <p>Hi, I have a few questions.</p>
<ol>
<li>Where can I find this output variables (loss_mask, loss_rpn_box_reg, loss_classifier,loss_objectness, loss_box_reg) in which script are they defined?</li>
</ol>
<p>E<code>poch: [1]  [170/189]  eta: 0:00:07  lr: 0.000100  loss: 0.6814 (0.8748)  loss_classifier: 0.0554 (0.0661)  loss_box_reg: 0.0693 (0.0586)  loss_mask: 0.4453 (0.4971)  loss_objectness: 0.0848 (0.2369)  loss_rpn_box_reg: 0.0075 (0.0161)  time: 0.4024  data: 0.0084  max mem: 7751</code></p>
<ol start="2">
<li>Is this <code>loss</code> a total loss of trained model?</li>
<li>Does torchvision has <code>accuracy</code> calculated as it has <code>loss</code>?</li>
</ol>

ptrblck: <p>Your post doesn’t mention the used model or any other information, so I assume you are running the Object Detection tutorial. If so, then check <a href="https://github.com/pytorch/vision/blob/0d7807d59520289b2065b4db4a138b7fba2f61fd/references/detection/engine.py#L12-L60">this code</a> to see how the metric is calculated.</p>

Lane: <p>Yes I’m using this Object detection tutorial and I used this <code>fucntion train_one_epoch()</code> for show the loss, but can I from there calculate <code>accuracy</code>?</p>
How do I modify gradients of DDP models after calling backward()?

Chen_Zhang: <p>I’m implementing an algorithm for which I need to modify the gradients after calling zero_grad() and backward() of multiple loss. I wrote a structure in the model to record the gradient for each back-prop, and then called a method of that structure to modify the model’s gradients directly.<br>
The problem is, this step could be finished in only one gpu, and how do I synchronize the gradients of models on gpu?</p>

Yanli_Zhao: <p>you can call dist.broadcast() if you want to broadcast the gradient from rank 0 to other ranks; or you can call dist.all_reduce() if you want to sum the gradient from all the ranks</p>
Runing Multiple trainings (or inference) in parallel in a single GPU

bouzouita_wafa: <p>Is it possible to run Multiple trainings of the same deep learning model (pytorch) in parallel in a single GPU ? and How please ?</p>

ptrblck: <p>Yes, this would be theoretically possible if:</p>
<ul>
<li>enough memory and compute resources are free on the GPU</li>
<li>if you are using different streams or generally different processes</li>
<li>if the CPU is not blocking the execution.</li>
</ul>
Is it possible to build pytorch from source for arm with nvidia gpu

Ronannnn: <p>Is it possible to build pytorch from source for arm with nvidia gpu?</p>

ptrblck: <p>Yes, that’s possible and we are already building PyTorch for ARM e.g. via <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">these NGC containers</a>.</p>

Ronannnn: <p>Thank you! I’ll try it out. But I’m wondering why there is no official python lib for arm gpu.</p>

ptrblck: <p>I’m not sure what “Python lib for ARM GPU” means, since the linked container is compiled for an ARM CPU and works on any GPU with a compute capability &gt;= 5.2.</p>

Ronannnn: <p>I’m sorry I’m new to python. I tried to install torch with ‘pip install torch’ in arm with gpu, but torch.cuda.is_available() is false. Anyway, NGC containers work for me. Thanks!</p>
Find the max in sepcific dim and replace by 1 and other 0

jazin: <p>Dear programmer,<br>
I have the following tensors:<br>
"<br>
index = tensor([[1, 1, 1,  …, 0, 0, 1],<br>
[0, 0, 2,  …, 1, 1, 1],<br>
[1, 1, 0,  …, 0, 1, 0],<br>
…,<br>
[0, 1, 2,  …, 0, 2, 1],<br>
[2, 0, 1,  …, 0, 1, 0],<br>
[2, 1, 1,  …, 1, 1, 1]])<br>
"<br>
index.size() = torch.Size([20, 3001])<br>
Also, I have the source tensor which is:<br>
source = torch.zeros(20, 3, 3001)</p>
<p>source.size() = torch.Size([20, 3, 3001])<br>
I would like to use the index tensor and replace the 1 with the source tensor in dim=1.<br>
I know I should use scatter, but I could not able to replace it.<br>
Could you please help me?<br>
Thanks</p>

ptrblck: <aside class="quote no-group" data-username="jazin" data-post="1" data-topic="163447">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e19b73/48.png" class="avatar"> jazin:</div>
<blockquote>
<p>I would like to use the index tensor and replace the 1 with the source tensor in dim=1.</p>
</blockquote>
</aside>
<p>Could you describe this approach with a small example? I’m currently unsure if you want to replace values in <code>source</code> where the <code>index</code> tensor has values of <code>1</code> or if the <code>index</code> tensor should be used as a plain index to assign new values to <code>source</code>.</p>
Total estimated model params size is bigger than trainable params

GhTara: <p>Hello,</p>
<p>I have a problem with the model size. The description of my model size is as follows:</p>
<blockquote>
<p><em>| Name          | Type  | Params</em><br>
<em>----------------------------------------</em><br>
<em>0 | model_        | Model | 44.5 M</em><br>
*1 | loss_function | Loss  | 0     *<br>
<em>----------------------------------------</em><br>
<em>44.5 M    Trainable params</em><br>
<em>0         Non-trainable params</em><br>
<em>44.5 M    Total params</em><br>
<em>178.099   Total estimated model params size (MB)</em></p>
</blockquote>
<p>So I get this Runtime Error:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB</p>
</blockquote>
<p>I am curious why <em>Total estimated model params size</em> is so much bigger than <em>Trainable params</em>. And what is the difference?<br>
How can I reduce the Total params size to avoid this error?</p>
<p>I appreciate any help.</p>

ptrblck: <aside class="quote no-group" data-username="GhTara" data-post="1" data-topic="163357">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ghtara/48/53961_2.png" class="avatar"> GhTara:</div>
<blockquote>
<p>I am curious why <em>Total estimated model params size</em> is so much bigger than <em>Trainable params</em>. And what is the difference?</p>
</blockquote>
</aside>
<p>I don’t think the param size is too large, as it should be calculated as the number of elements in all parameters multiplied with their element size. Assuming you are using the default <code>float32</code> <code>dtype</code> a quic estimation would be:</p>
<pre data-code-wrap="python"><code class="lang-python">(44.5 * 1e6 * 4) / 1024**2
# 169.7540283203125
</code></pre>
<p>where the posted estimation uses <code> / 1000**2</code> to calculate the size in <code>MB</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">(44.5 * 1e6 * 4) / 1000**2
# 178.0
</code></pre>
poolINTERNAL ASSERT FAILED at "../aten/src/ATen/ParallelOpenMP.cpp":65, please report a bug to PyTorch. Invalid thread pool

jinluyang: <p>pytorch version: 1.11.0<br>
device: 96 CPU cores, used by many people, often slow.(Actually the problem goes away if I use a better CPU)<br>
OS: Centos 7<br>
The error is as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c25056735520d6491b6a68f5a2ceee38ac3f39ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/c25056735520d6491b6a68f5a2ceee38ac3f39ba" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c25056735520d6491b6a68f5a2ceee38ac3f39ba_2_690x99.png" alt="image" data-base62-sha1="rIYKCEesL6XsKovEwcblcLceGts" width="690" height="99" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c25056735520d6491b6a68f5a2ceee38ac3f39ba_2_690x99.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c25056735520d6491b6a68f5a2ceee38ac3f39ba_2_1035x148.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c25056735520d6491b6a68f5a2ceee38ac3f39ba.png 2x" data-dominant-color="0E0E0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1123×162 58.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The error occurs when the model is run for the first time. Then it disappears when run the second time and so on. So it is easy to work around the problem by warming up. But I’m curious about why it fails to create caffe2 threadpool.</p>

ptrblck: <p>Could you check if you are still seeing the same issue using the latest nightly binary? If so, could you create a GitHub issue with instructions to reproduce it, please?</p>
What does the tensor's "H","T" attributes mean?

111466: <p>Example codes as followed:</p>
<pre><code class="lang-python">import torch

T = torch.rand(3,3)
print(T.H)
print(T.T)
</code></pre>
<p>And the corresponding output:</p>
<pre><code class="lang-python">tensor([[0.9100, 0.4052, 0.8227],
        [0.9101, 0.5563, 0.8068],
        [0.5961, 0.8708, 0.0686]])
tensor([[0.9100, 0.4052, 0.8227],
        [0.9101, 0.5563, 0.8068],
        [0.5961, 0.8708, 0.0686]])

 tensor([[0.9100, 0.9101, 0.5961],
        [0.4052, 0.5563, 0.8708],
        [0.8227, 0.8068, 0.0686]])
</code></pre>

ptrblck: <p>From the docs:<br>
<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.T">Tensor.T</a>:</p>
<blockquote>
<p>Returns a view of this tensor with its dimensions reversed.<br>
If <code>n</code> is the number of dimensions in <code>x</code>, <code>x.T</code> is equivalent to <code>x.permute(n-1, n-2, ..., 0)</code>.</p>
</blockquote>
<p><a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.H">Tensor.H</a>:</p>
<blockquote>
<p>Returns a view of a matrix (2-D tensor) conjugated and transposed.<br>
<code>x.H</code> is equivalent to <code>x.transpose(0, 1).conj()</code> for complex matrices and <code>x.transpose(0, 1)</code> for real matrices.</p>
</blockquote>
How cloud I convert a quantized model into a lossy model of float32?

Rahul_Xie: <p>Given a quantized model (PTQ or QAT), I want to convert it into a model whose parameters representation is float32. Because I want to use GPU. How cloud I achieve it?</p>

jerryzh168: <p>you mean “dequantize” a model? interesting, I don’t think we support that. Why don’t you use the original floating point model in this case?</p>

Rahul_Xie: <p>Thank you for your suggestion. Actually, I want to perform experiments on quantized model. For some reason, the quantized model does not support GPU, which makes the experiment pretty slow.</p>

jerryzh168: <p>maybe you can try <a href="https://discuss.pytorch.org/t/quantization-frequently-asked-questions/161251#quantized-inference-in-gpu-3" class="inline-onebox">[quantization] Frequently Asked Questions</a> to see if this works for you or not</p>

Rahul_Xie: <p>Thank you very much.</p>
How could I pad packed data based on length

yllgl: <p>I have a packed data and each sequences’ length.<br>
Example:</p>
<pre><code class="lang-python">data = torch.tensor([4, 1, 3, 5, 2, 6])
lengths = torch.tensor([2,1,3])
</code></pre>
<p>I want to create a pad 2-D (batch_size,max_lengths) matrix like:</p>
<pre><code class="lang-auto">output = torch.tensor([[4,1,0], #length=2
                       [3,0,0],#length=1
                       [5,2,6])#length=3
</code></pre>
<p>And due to my training purpose, this operation should be able to track backward gradient if <code>data.requires_grad=True</code>.</p>

ptrblck: <p><code>pad_sequence</code> should work:</p>
<pre><code class="lang-python">data = torch.tensor([4, 1, 3, 5, 2, 6], dtype=torch.float32, requires_grad=True)
lengths = torch.tensor([2,1,3])

x = data.split(torch.tensor_split(lengths, len(lengths)))
out = torch.nn.utils.rnn.pad_sequence(x, batch_first=True)
print(out)
# tensor([[4., 1., 0.],
#         [3., 0., 0.],
#         [5., 2., 6.]], grad_fn=&lt;CopySlices&gt;)

out.mean().backward()
print(data.grad)
# tensor([0.1111, 0.1111, 0.1111, 0.1111, 0.1111, 0.1111])
</code></pre>
Torch.fx usage question

yichuan520030910320: <p>when I try to tansform the order of the operations in my module, I find the graph changed but the forward function remains thesame and it is said that ‘UnboundLocalError: local variable ‘getitem_5’ referenced before assignment’<br>
I just chage the args of some node but it failed in performe a result</p>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce this issue, please?</p>