akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Tools to visualize/debug parallelism?

Vadim: <p>Are there any tools that can help to visualize and debug parallelism problems in my models?</p>
<p>For example:</p>
<ul>
<li>I would love to see a chart showing when asynchronous kernels actually started and finished execution whether there are any data dependencies between them, whether kernel launch was delayed because of memory pressure, and so on.</li>
<li>I would like to make sure that my code is not accidentally causing CPU/GPU synchronization by converting result of GPU computation into a plain Python/NumPy type (or by performing any other operation that causes synchronization).</li>
</ul>

googlebot: <p>See <a href="https://pytorch.org/docs/master/autograd.html#profiler" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch master documentation</a>, chrome trace shows cpu/gpu timeline, emit_nvtx - cuda low-level timeline.</p>

Vadim: <p>Thanks, I’ve exported chrome trace from profiler, but I’m not sure how to interpret it.   What are those “cpu_to_cuda” event lines?   What are “Outgoing flow” and “Preceding/Following events”, which appear in the lower panel when I click something?</p>
<p>Also, this does not seem to help with my second problem (unintentional synchronization).</p>

googlebot: <p>I don’t know how exactly it links events, cpu “commands” are linked to cuda kernel launches, and dependencies are somehow reflected.</p>
<aside class="quote no-group" data-username="Vadim" data-post="3" data-topic="107452">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vadim/48/31272_2.png" class="avatar"> Vadim:</div>
<blockquote>
<p>Also, this does not seem to help with my second problem (unintentional synchronization).</p>
</blockquote>
</aside>
<p>Hm, you would normally see gaps in GPU timeline, if you have some bad cross-device interactions.</p>
Class Imbalance for Image Segmentation

moreshud: <p>I am working on image segmentation task with image and 7 [2, 5] labels given in .npy format. I checked the class frequency in each by taking the sum of ones since the labels are readily one-hot encoded. Based on suggested approach, I using the weighted (median sampling frequency) dice+CrossEntropy loss but the performance is not improving.</p>
<p>Are there other approach that can be used to handle such imbalance for improved performance?</p>

4158ndfkvBHJ1: <p>Class imbalance in image segmentation is a notoriously difficult problem. I got decent results with 2 things :</p>
<ul>
<li>using an architecture with multiscale representation, like HrNetv2</li>
<li>using the loss suggested in this paper : <a href="https://arxiv.org/abs/1708.02002" rel="noopener nofollow ugc">https://arxiv.org/abs/1708.02002</a>, the idea is that the loss isn’t decreased much for confident correctly classified pixels. This way, most of the parameters updates are concentrated on classes that are difficult to classify.</li>
</ul>

moreshud: <p>I appreciate your contributions.</p>
Update grad_fn of net.parameters()

afps: <p>Say I have a list of parameters called p_new and each parameter has a grad_fn which I care about.</p>
<p>I would like to update net.parameters() like this:</p>
<pre><code class="lang-auto">for param, pi in zip(net.parameters(), p_new):
        param = pi # I don't wanna update data only!

</code></pre>
<p>Of course this doesn’t work. Any alternatives?</p>

albanD: <p>Hi,</p>
<p>You can copy into a Tensor in a differentiable way with <code>param.copy_(pi)</code>.<br>
The problem is that Parameters are leafs (they never have history) so you won’t be allowed to do this.<br>
If you don’t want them to be Parameters anymore and want them to have history, you can simply delete the existing parameter <code>del net.foo</code> and set a plain Tensor to replace it <code>net.foo = your_tensor</code>.</p>

afps: <p>Thanks! But how do I do that for all parameters?<br>
Like this?</p>
<pre><code class="lang-auto">for param, pi in zip(net.parameters(), p_new):
        del param
        param = pi
</code></pre>
<p>I guess no.</p>
<p>Also, after the update I want to do a forward pass. Will I be able to do that?</p>

albanD: <p>When you do net.paramters(), you only get the Parameter, not the parent Module.<br>
You can check the <code>named_parameters()</code> to get both the Parameter and the name to access it.<br>
You can do something that looks like the following:</p>
<pre><code class="lang-auto">def del_obj(mod, name):
    names = name.split(".")
    for n in names[:-1]:
        mod = mod[n]
    del mod[names[-1]]

def set_obj(mod, name, new_val):
    names = name.split(".")
    for n in names[:-1]:
        mod = mod[n]
    mod[names[-1]] = new_val

for (name, _), pi in zip(net.named_parameters(), p_new):
        del_obj(net, name)
        set_obj(net, name, pi)
</code></pre>

afps: <p>I get the error: model object is not subscriptable.</p>
<p>FYI the names of the parameters are:</p>
<p>fc.0.weight<br>
fc.0.bias<br>
fc.1.weight<br>
fc.1.bias</p>
<p>And even if I try explicitly del fc.0.weight I get: TypeError: ‘Linear’ object does not support item deletion</p>

albanD: <p>You will need to get the linear first by doing <code>lin = fc[0]</code> then you can delete the weights as <code>del lin.weight</code>.</p>
Best Way to Compare Models

AlperEkmekci: <p>I wonder what is the best model to compare ?</p>
<p>Let’s say we have 2 models or same model with various differences (optimizer, layer depth changes etc.), how can we compare those models in a reliable way ?</p>
<p>Additionally when we use different criterion loss functions, how can we choose one and use one of them and be sure about which one is the better than the other ?</p>
<p>Greetings.</p>

googlebot: <p>Reducing training non-determinism and doing n-fold cross-validation is pretty reliable. But all such stuff is obviously slow…</p>
<aside class="quote no-group" data-username="AlperEkmekci" data-post="1" data-topic="106717">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alperekmekci/48/31270_2.png" class="avatar"> AlperEkmekci:</div>
<blockquote>
<p>Additionally when we use different criterion loss functions, how can we choose one and use one of them and be sure about which one is the better than the other ?</p>
</blockquote>
</aside>
<p>If you switch a loss function, you postulate a different probability distribution family for predicted values. If that’s your intent, it is possible to compare model log-likelihoods (for non-deep compact models that would be AIC). Simpler approach is just comparing some auxiliary metrics.</p>

AlperEkmekci: <p>As I understand AIC metric gives us which DL model is better at prediction relatively so I can use it on model selection thank you.</p>
<p>I think <code>torch.set_deterministic()</code> and <code>torch.manuel_seed(0)</code> are pretty useful for comparing how changes are affecting to the models. Is it a good way for DL model testing pipeline ?</p>

googlebot: <p>Use of AIC’s num. of parameters term is questionable with overparametrized DL models. But likelihood part is comparable across models, yes.</p>
<aside class="quote no-group" data-username="AlperEkmekci" data-post="3" data-topic="106717">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alperekmekci/48/31270_2.png" class="avatar"> AlperEkmekci:</div>
<blockquote>
<p>I think <code>torch.set_deterministic()</code> and <code>torch.manuel_seed(0)</code> are pretty useful for comparing how changes are affecting to the models. Is it a good way for DL model testing pipeline ?</p>
</blockquote>
</aside>
<p>Manual seed is good, as it has no computational side effects. Additional measures should be used with care, you could slow down stuff without achieving bit-wise reproducibility. IMO, it is more appropriate for continuous hyperparam optimization, to not confuse some search algorithms, discrete model/parameter choices can be more obvious.</p>

AlperEkmekci: <p>Thanks for valuable answers. I will try these steps you suggested.</p>
<p>Greetings.</p>
Saved CNN model giving different results

Arijeet_Choudhury: <p>I trained the batch normalized vgg16 CNN model and saved the best model parameters. During training, the best accuracy on the vaildation set was  97.65%. I loaded the saved parameters using model.load_state_dict and ran the model on the validation set again. I did set my model to eval mode using model.eval() and also used torch.no_grad(). But the accuracy differs now.<br>
original- 97.65%<br>
after loading- 96.45%, 97.12%, 96.87% on 3 different runs.<br>
What could be the issue?</p>

JamesDickens: <p>You could debug this by:</p>
<pre><code class="lang-auto">def areTorchModulesEqual(module1, module2):
    for index, (p1, p2) in enumerate(zip(module1.parameters(), module2.parameters())):
        if p1.data.ne(p2.data).sum() &gt; 0:
            return False
    return True

def whichModulesHaveBeenUpdated(model1_list, model2_list):
    for index, module1 in enumerate(model1_list):
        module2 = model2_list[index]
        if areTorchModulesEqual(module1, module2):
            print("At Index " + str(index) + " Module1 and Module2 are equal, below is the printed module")
            print(module1)
            print("\n")
        else:
            print("At index " + str(index) + " Module1 and Module2 are not equal, below is the printed module")
            print(module1)
            print("\n")
    print("End of Modules")
</code></pre>
<p>Also does your model have batch-normalization or dropout?<br>
It could also be an inconsistency with your test data iterator.</p>
Pytorch multiprocessing dataloader worker_init_fn problem

chaonan99: <p>I have a problem with this code piece</p>
<pre><code class="lang-auto">import os

import torch


class Dataset(torch.utils.data.Dataset):
    arg = {'batch_size': 1}
    def __init__(self, arg):
        print('__init__')
        self.arg.update(arg)
        print(self.arg)

    def _worker_init_fn(self, *args):
        print('worker init')
        print(self.arg)

    def get_dataloader(self):
        return torch.utils.data.DataLoader(self, batch_size=None,
                                         num_workers=3,
                                         worker_init_fn=self._worker_init_fn,
                                         pin_memory=True,
                                         multiprocessing_context='spawn')

    def __getitem__(self, idx):
        return 0

    def __len__(self):
        return 5


def main():
    dataloader = Dataset({'batch_size': 2}).get_dataloader()
    for _ in dataloader:
        pass


if __name__ == '__main__':
    main()
</code></pre>
<p>Basically I want the workers to have  <code>{'batch_size': 2}</code>, but actually they have <code>{'batch_size': 1}</code>. The code print the following:</p>
<pre><code class="lang-auto">__init__
{'batch_size': 2}
worker init
{'batch_size': 1}
worker init
{'batch_size': 1}
worker init
{'batch_size': 1}
</code></pre>
<p>How can I make the workers to have the correct <code>batch_size</code>?</p>

chaonan99: <aside class="quote no-group" data-username="chaonan99" data-post="1" data-topic="74826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chaonan99/48/22167_2.png" class="avatar"> chaonan99:</div>
<blockquote>
<p>self.arg.update(arg)</p>
</blockquote>
</aside>
<p>Solved. If I add <code>self.arg = self.arg</code> after this line it works.</p>

Euruson: <p>Set <code>multiprocessing_context</code> to <code>fork</code> works fine on linux.</p>
<p>I guess the reason is that the <code>arg</code> is a class variable (rather than an instance variable) and mutuable. As <code>spawn</code> pickles instances to use in new process, the old class variable is inaccessible in dataloader workers. And it makes sense  that <code>multiprocessing_context=fork</code> works fine, for forked processes share some resources.</p>
Find index of certain value in 3 dimension array in parallel way

hohohihi: <p>i made a ‘zero_mask2’ function(i copy and paste the code below) in the middle of my pytorch file. But this is too slow. So, i want to find the better way.</p>
<p>First, Let me explain the gist of the function below.</p>
<h1>input</h1>
<p>‘all_idx_before’ ( dimension of [batch_size, number of points, 20] ) : batch_size is a well-known definition of mini batch in deep learning society. In every batch, there are many points like 1024. And, in every point, a point has 20 values.</p>
<p>‘idx’ ( dimension of [batch_size, number of points, 5] ) : batch and number of points are same as before. But, in this case, a point has only 5 values.</p>
<p>‘p’ : just an any number, for example, 7.</p>
<h1>toy example</h1>
<p>lets say, a point in ‘all_idx_before’ has (9, 10, 11, 12, 6, 7, 8, 14, 2, 3, 4, 5, 18, 19, 15, 16, 17, 13, 1, 20).</p>
<p>And a point in ‘idx’ has (10, 6, 2, 4, 18).</p>
<p>In this setting, i want to find the index of each value in ‘idx’. for example, (10) in ‘idx’ is second element in ‘all_idx_before’, (6) in ‘idx’ is fifth element in ‘all_idx_before’…</p>
<p>So, potential output of this point would be like (1, 5, 8, 10, 12), (‘second’ becomes ‘1’ because of number system in python.)</p>
<p>I do this logic in every batch and every point. So, i used for iteration. But this is too slow. Is there any way of doing this in parallel? I use numpy format. But i can use tensor if it is better.</p>
<p>Thank you.</p>
<pre><code class="lang-auto">def zero_mask2(idx, all_idx_before, p):

size_batch = len(idx)
size_points = len(idx[0])
size_neighbor = len(idx[0][0])

mask = torch.empty(size_batch, size_points, size_neighbor)

for i in range(size_batch):
 for j in range(size_points):
  for l in range(size_neighbor):
   mask[i][j][l] = np.where(all_idx_before[i][j] == idx[i][j][l])[0][0] 

mask_t = mask &lt;= p # smaller than p or same : true , bigger than p : false
return mask_t
</code></pre>

Ruslan_Baynazarov: <p>I would add a new dimension to those tensors and compare them:</p>
<pre><code class="lang-auto">all_idx_before = all_idx_before.view([size_batch, size_points, 20, 1])
idx = idx.view([size_batch, size_points, 1, 5])

absdiff = torch.abs(idx - all_idx_before) # [batch, points, 20, 5]
</code></pre>
<p>Then find indices with <code>argmin</code>:</p>
<pre><code class="lang-auto">output = absdiff.argmin(dim=2)
</code></pre>

hohohihi: <p>Oh! That is a good way. I will try and let you know the result. Thank you.</p>
<p>----- after using your method ----<br>
Wow! it became much more faster than using 3 for iteration…<br>
But it seems it consumes more memory.</p>
<p>Thank you again.</p>
Keep the momentum in weights file

User4: <p>Hi, I have a small question</p>
<p>For example with SGD momentum, the momentum will be stacked by this formula (follow <a href="https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD" rel="noopener nofollow ugc">Pytorch source code</a>):</p>
<pre><code class="lang-auto">v{t+1} &amp; = momentum * vt + g{t+1}  
</code></pre>
<p>Does Pytorch keep the momentum from optimizer in pth file? Or we have to reset the momentum every time we train again?</p>

googlebot: <p>optimizer state is kept separately, you can save &amp; reload it, see optimizer.load_state_dict()</p>

User4: <p><a class="mention" href="/u/googlebot">@googlebot</a> Thank you for the fast reply,</p>
<p>I have another question, so if we use the pre-trained model for the new dataset, for example from ResNet50 trained by ImageNet dataset, so the momentum (in both cases of SGD and Adam) and the num_iterations (in case of Adam) are both a large number since the first epoch of new dataset, is it correct?</p>

googlebot: <p>Hm, I don’t think you would reload optimizer state in that case, as old gradient history no longer applies.</p>

User4: <p>Understood, thanks for your help.</p>
How to swap lines from a 2D tensor?

Aldebaran: <p>Given the following 2D tensor whit shape [N,E]:</p>
<pre><code class="lang-python">#N=3, E=3
r1 = torch.tensor([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])
</code></pre>
<p>how to swap the <code>r1</code> “rows” to get results like the one shown below:</p>
<pre><code class="lang-python">r1 = torch.tensor([
    [1, 2, 3],
    [7, 8, 9],
    [4, 5, 6]
    
])
</code></pre>
<p><strong>Note</strong>: any permutation between the “rows” would be valid.</p>

Ruslan_Baynazarov: <p>Hi!</p>
<pre><code class="lang-auto">&gt;&gt;&gt; r1[[0,2,1]]
tensor([[1, 2, 3],
        [7, 8, 9],
        [4, 5, 6]])
</code></pre>
<p>Is this what you want?</p>

Aldebaran: <p>Hello <a class="mention" href="/u/ruslan_baynazarov">@Ruslan_Baynazarov</a>,</p>
<p>It’s good answer, thank you! The following is a code snippet showing the complete example:</p>
<pre><code class="lang-python">batch_size = 3
embedding_size = 4
sample = torch.rand(batch_size, embedding_size)
# tensor([[0.8855, 0.1571, 0.2649, 0.6999],
#         [0.0864, 0.3912, 0.1669, 0.6979],
#         [0.3881, 0.5181, 0.4245, 0.3675]])

shuffle_index = torch.randperm(batch_size)
# tensor([2, 0, 1])

shuffled_sample = sample[[shuffle_index]]
# tensor([[0.3881, 0.5181, 0.4245, 0.3675],
#         [0.8855, 0.1571, 0.2649, 0.6999],
#         [0.0864, 0.3912, 0.1669, 0.6979]])
</code></pre>
Variable get updated at the same time

little_cat: <p>How could I achieve it in Pytorch if I want to optimize a variable x but I have this constrain x + y = 1.0<br>
When I optimize the x, I want to get y updated at the same time.<br>
For example,</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    x = torch.tensor(1.0, requires_grad=True)
    optimizer = SGD([x], lr=0.1)
    y = 1 - x
    z = y * 0.5 + x
    print('Beginning')
    print(x)
    print(y)
    print('---')
    optimizer.zero_grad()
    z.backward()
    optimizer.step()
    print('After')
    print(y)
    print(x)
</code></pre>
<p>I want to y get updated after the optimizer update x.</p>

Ruslan_Baynazarov: <p>Just run <code>y = 1 - x</code> again after <code>optimizer.step()</code>.<br>
There is no way that two trainable variables are hard constrained.</p>

Ruslan_Baynazarov: <p>You can add a soft constraint if you wish.</p>
<pre><code class="lang-auto">loss = z + some_coefficient * torch.abs(y + x - 1)
loss.backward()
</code></pre>
<p>In this case <code>y</code> must also be initialized as a variable requiring gradient.</p>
Custom Transformation

CalenDario13: <p>Hi there, I am new of Pytorch, I want to apply my own function to transform pictures, but duing that the process slows down a lot. I think the problem here is that for each image it calls a class that takes a while to be loaded (but not sure). As I said I am new, so if you think this is the wrong approach just tell which is the better solution even if it is far away from this one.</p>
<pre><code class="lang-auto">from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms
import torch

import numpy as np
from PIL import Image

from mtcnn.mtcnn import MTCNN # recognise key points
detector = MTCNN() # It takes a while to be laoded

# Define Custom Transformation:
def occlude(input, p=0.5):

  '''
  This function occlude one of the three possible key points (mouth, nose and eyes),
  with a certain probability p.

    Args:
        input (array): a picture
        p (int): the probability that an occlusion occur

    Returns:
        modified picture (array): an occluded picture with probability p
  '''

  targets = ['mouth', 'eyes', 'nose'] + [None for _ in range(int(3 / p) - 3)]
  faces = detector.detect_faces(input)
  for f in faces:
    choice = np.random.choice(targets)


    if choice == 'mouth':
      x1 = f['keypoints']['mouth_left'][1]
      x2 = f['keypoints']['mouth_right'][1]
      y1 = f['keypoints']['mouth_left'][0]
      y2 = f['keypoints']['mouth_right'][0]
      input[x1-5:x2+5,y1-5:y2+5] = 0

    if choice == 'eyes':
      x1 = f['keypoints']['left_eye'][1]
      x2 = f['keypoints']['right_eye'][1]
      y1 = f['keypoints']['left_eye'][0]
      y2 = f['keypoints']['right_eye'][0]
      input[x1-5:x1+5,y1-5:y1+5] = 0
      input[x2-5:x2+5,y2-5:y2+5] = 0

    if choice == 'nose':
      x1 = f['keypoints']['nose'][1]
      y1 = f['keypoints']['nose'][0]
      input[x1-5:x1+5,y1-5:y1+5] = 0

  return Image.fromarray(input)


train_transform = transforms.Compose([
            transforms.Lambda(lambda x: occlude(x, p=0.3)),
            transforms.RandomApply(torch.nn.ModuleList([transforms.RandomAffine(degrees=0, translate=(0.10,0.10))]), p=0.30),               
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize((0,0,0), (1,1,1)),
            transforms.Resize((197,197))
        ])

train = DataLoader('./FER/train',  transformations=train_transform)
</code></pre>

Ruslan_Baynazarov: <p>The class is loaded once, so this is not the problem.<br>
Face detector is quiet a heavy operation itself.<br>
It takes long to detect faces.<br>
If I wanted to do this kind of augmentation, I wound probably run face detector on the whole data set before training. And used the saved face detection results.</p>
A weird question about `+=` operation when changing model input

bat67: <p>Hi~<br>
I came across a weird issue these days, and the minimal code to reproduce is as below:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 3, kernel_size=3, stride=1, padding=1) 

    def forward(self, x):
        return self.conv1(x)

model = Net()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
model.train()

tot = 0
input = torch.rand(4, 3, 32, 32)
for _ in range(2):
        out = model(input)
        tot += out
        
        delta = torch.rand(out.shape)
        
        #&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
        input += delta            # ERROR
        # input = input + delta   # CORRECT
        #&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;

with torch.autograd.set_detect_anomaly(True):
    loss = tot.mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>When I change the input of the model using <code>+=</code> operation, it give me a runtime error when using <code>with torch.autograd.set_detect_anomaly(True)</code> at the <code>loss.backward()</code> line:</p>
<p>“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 3, 32, 32]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!”</p>
<p>This was so confused until I seperate <code>+=</code> to <code>+</code> and <code>=</code>. I’m still wondering, is there any differences to use these two method in this situation? As they are all model input without requiring grad. (I’m now using 1.5.1 version)</p>
<p>Thanks in advance!</p>

googlebot: <aside class="quote no-group" data-username="bat67" data-post="1" data-topic="107422">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bat67/48/8035_2.png" class="avatar"> bat67:</div>
<blockquote>
<p>As they are all model input without requiring grad</p>
</blockquote>
</aside>
<p>That makes no difference, inputs are used in the first layer in some operation that works like: layer1(input, weights1). That operation is normally multiplication-based, so weights1 gradient computation depends on input, and inplace += op rewrites it</p>

bat67: <p>Thanks! That’s clear.</p>
How to concatenate 2d tensors with 2 different dimensions

Hristo_Petkov: <p>I was wondering if it is possible to concatenate two different pytorch tensors with different shapes. one tensor is of shape torch.Size([247, 247]) and the other is of shape torch.Size([10, 183]). Is it possible to concatenate these using torch.cat() on dim=1?</p>

InnovArul: <p>I don’t think it is possible.<br>
Can you elaborate on your requirements that warrants this?</p>

Hristo_Petkov: <p>I fixed the error I had a wrong variable in place. Sorry for the trouble</p>
How could I rearrange order along certain dimension?

coincheung: <p>Hi,</p>
<p>Suppose I have a tensor <code>a</code> of shape (2, 3, 4), and my order tensor <code>idx</code> is (2, 4). I need to rearrange it along the <code>dim=2</code> dimension. Such that <code>a[0] = a[0][:,idx[0]], a[1] = a[1][:,idx[1]]</code>.<br>
My problem is that my tensor in reality is larger than the tensor in the example, and I would like to do it in single line which is faster and requires less memory.</p>
<p>How could I do this please ?</p>

googlebot: <p>perhaps something like</p>
<blockquote>
<p>a.gather(2, idx.unsqueeze(1).expand(-1,3,-1))</p>
</blockquote>
<p>will work</p>

coincheung: <p>Thanks !! What is the memory overhead of this method ? Will the memory usage gets doubled due the operation of <code>idx.unsqueeze(1).expand(-1, 3, 1)</code>?</p>

googlebot: <p>expand() doesn’t allocate memory, it does stride manipulations to create self-overlaps</p>

coincheung: <p>Thanks a lot for replying !!!</p>
Performance of contiguous vs. non-contiguous tensors

CDhere: <p>Hi all, I work mostly on computer vision problems and I’ve found that in CV-related computation there are usually tons of tensor flipping (e.g., reshape, switching axes, adding new axes), which might result in non-contiguous tensors (a super good explanation <a href="https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107">here</a>). Sometimes people will deliberately try to keep tensors to be contiguous, for example the following line from the popular detectron2’s <code>detectron2.data.dataset_mapper.DatasetMapper</code>:</p>
<pre><code class="lang-auto">dataset_dict["image"] = torch.as_tensor(np.ascontiguousarray(image.transpose(2, 0, 1)))
</code></pre>
<p>is making sure that an image tensor with contiguous memory is created.</p>
<p>But most of the time people don’t care much about keeping them contiguous, with crazy tensor flipping all over the place (no blaming but just to describe how people make full use of the flexibility provided by the framework lol).</p>
<p>I wonder if there are any general guidelines in dealing tensor’s memory? Is it always better to use contiguous tensors? Thanks!</p>

googlebot: <p>Thing is, operations on permuted tensors create contiguous tensors, so permuted tensors usually don’t stick around. One notable exception is torch.channels_last format, added some time ago, that one is “sticky”.</p>

CDhere: <p>Could you please give an example on how new contiguous tensors are created? I’ve tried timing forwarding contiguous vs. non-contiguous tensors, and forwarding contiguous tensors has lower latency than the other.</p>

googlebot: <pre><code class="lang-auto">x=torch.ones(2,3,4).permute(0,2,1)
(x * 2).stride()
Out[54]: (12, 1, 4)
(x.log()).stride()
Out[55]: (12, 1, 4)
</code></pre>
<p>So, elementwise ops don’t create contiguous tensors, but I believe they have the same speed on non-contiguous tensors (at least unary and scalar operand ops)</p>
<pre><code class="lang-auto">(x.matmul(torch.ones(3,5))).stride()
Out[56]: (20, 5, 1)
F.conv1d(x, torch.ones(5,4,1)).stride()
Out[58]: (15, 3, 1)
</code></pre>
<p>Contiguous tensors are created by usual “layer” ops. They can work slower on non-contiguous tensors, yes.</p>
<p>However:</p>
<pre><code class="lang-auto">nhwc = torch.ones(2,3,4,5,device="cuda")
nchw = nhwc.permute(0,3,1,2)
(nchw.stride(), nchw.is_contiguous(), nchw.is_contiguous(memory_format=torch.channels_last))
Out[69]: ((60, 1, 20, 5), False, True)
F.conv2d(nchw, torch.ones(6,5,1,1,device="cuda")).stride()
Out[72]: (72, 1, 24, 6)
</code></pre>
<p>here torch.channels_last format is auto-detected for a cuda 4d tensor, and is preserved</p>
How to create computational graphs for updated parameters

Surayuth_Pintawong: <p>I want to create computation graphs for updated parameters so that I could calculate their higher-order derivatives later. The code below shows how I solve the problem if the model is not complex.</p>
<pre><code class="lang-auto">import torch

# 1: Define model parameters(w, T)
w = torch.tensor(1., requires_grad = True)
T = torch.tensor(2., requires_grad = True)

# 2: Define output(L)
L = T * w

# 3: Calculate dL/dw (by fixing T)
grad = torch.autograd.grad(L, w, create_graph=True)[0]

# 4: Update w --&gt; w2
lr = 0.1
w2 = x - lr*grad
# I use w2 instead of using optimizer.step() to update w
# because I want to create a computational graph between w2, w and T

#5: Calculate the new value of output
L2 = T * w2
# Note that w2 depends on both w and T,
# w2 = w - lr*grad = w -lr*T

#6: Calculate dL2/dT (by fixing w)
dT = torch.autograd.grad(L2, T)[0]
#Because L2 = T*(w - lr*T), dL2/dT = w - 2*lr*T = 0.6
</code></pre>
<p>The problem is if my model is getting larger, I can’t create a new variable (<span class="hashtag">#4</span>) and assign it to a model (<span class="hashtag">#5</span>) without excluding the computational graphs. For example, I can only update the model parameters via</p>
<pre><code class="lang-auto">with torch.no_grad():
     for param in model.parameters():
        param -​= lr*param.grad
</code></pre>
<p>or use</p>
<pre><code class="lang-auto">optimizer.step()
</code></pre>
<p>instead. But both of these methods don’t create computational graphs for parameters. So, Is there a way that I can update parameters for a large model without excluding their computational graphs?</p>

albanD: <p>Hi,</p>
<p>You might want to take a look at the <a href="https://github.com/facebookresearch/higher">higher</a> library that is built to do just that.</p>

Surayuth_Pintawong: <p>Thank you. I didn’t know they made a library for meta learning.</p>
How to calculate minus set of two tensors?

Z_Mario: <pre><code class="lang-auto">A = torch.tensor([4,7,11])
B = torch.tensor([1,4])
</code></pre>
<p>How to get a tensor:<code>tensor([7,11])</code>, in which all element from A but not from B?</p>

Z_Mario: <p>I have find the answer from <a href="https://stackoverflow.com/questions/55110047/finding-non-intersection-of-two-pytorch-tensors" class="inline-onebox" rel="noopener nofollow ugc">python - Finding non-intersection of two pytorch tensors - Stack Overflow</a></p>
Autoencoder accuracy calculator positional argument error

MaxJohnson: <p>I got this autoencoder</p>
<pre><code class="lang-auto">class AE(nn.Module):
    def __init__(self, **kwargs):
        super(AE,self).__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=128
        )
        self.encoder_output_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_output_layer = nn.Linear(
            in_features=128, out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)
        code = self.encoder_output_layer(activation)
        code = torch.sigmoid(code)
        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)
        activation = self.decoder_output_layer(activation)
        reconstructed = torch.sigmoid(activation)
        return reconstructed
</code></pre>
<p>and I am trying to build a accuracy calculator</p>
<pre><code class="lang-auto">with torch.no_grad():
    for batch_features in test_loader:
        batch_features = batch_features[0]
        test_examples = batch_features.view(-1, 784)
        reconstruction = model(test_examples)
        
        images, labels = data[0].to(device), data[1].to(device)
        print(labels)
        
        outputs_auto = AE(images.to(device))
        _, predicted_auto = torch.max(outputs_auto.data, 1)
        total += labels.size(0)
        correct += (predicted_auto == labels).sum().item()
        break
    print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
</code></pre>
<p>but it shows</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-60-47c95a24e155&gt; in &lt;module&gt;
     19         print(labels)
     20 
---&gt; 21         outputs_auto = AE(images.to(device))
     22         _, predicted_auto = torch.max(outputs_auto.data, 1)
     23         total += labels.size(0)

TypeError: __init__() takes 1 positional argument but 2 were given
</code></pre>

InnovArul: <aside class="quote no-group" data-username="MaxJohnson" data-post="1" data-topic="107356">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maxjohnson/48/29916_2.png" class="avatar"> MaxJohnson:</div>
<blockquote>
<p><code>        outputs_auto = AE(images.to(device))</code></p>
</blockquote>
</aside>
<p>Here, you are calling <code>AE</code> class with the input.<br>
Instead, you should instantiate <code>AE</code> class to create model object and then call that object.</p>
<pre><code class="lang-auto">AE_model = AE(&lt;params&gt;)
AE_model.to(device)
...

with torch.no_grad():
    for batch_features in test_loader:
        ...
        outputs_auto = AE_model(images.to(device))

</code></pre>
Which one is faster (pytorch mobile or tflite)?

Johnson_Mark: <p>I am training a pytorch model with quantization aware and port it to mobile. I found only one solution is that using pytorch mobile <a href="https://pytorch.org/mobile/home/" class="inline-onebox" rel="noopener nofollow ugc">Home | PyTorch</a></p>
<p>I have a question about SPEED of pytorch mobile and tflite for the quantization model. Which one is faster (pytorch mobile or tflite)?</p>
Determinism of gradient accumulation

googlebot: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/873e667728615c3fb2bc9a797eda0cf1158ed33b.png" data-download-href="https://discuss.pytorch.org/uploads/default/873e667728615c3fb2bc9a797eda0cf1158ed33b" title="autograd-mix"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/873e667728615c3fb2bc9a797eda0cf1158ed33b.png" alt="autograd-mix" data-base62-sha1="jiq9kivPTry3jRAKjTbzvwx524j" width="690" height="254" data-dominant-color="DCE9D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">autograd-mix</span><span class="informations">822×303 21.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Am I correct in assuming that adding CPU operations like above makes gradient accumulation (for tensor shared_ctx) non-deterministic?</p>
<p>I.e. latent1backward &amp; latent2backward are serialized, as single cuda stream is used. But CPU backward functions (and/or memory transfers) are executed concurrently, and summands for shared_ctx’s gradient can come in different orders.</p>

albanD: <p>Hi,</p>
<p>Yes you are correct that when using multiple devices, the accumulation is not forced to run in a specific order.</p>
<p>A “simple” solution to fix that is to use a custom function that does the copy to the different devices in the forward and the accumulation in the backward (in a fixed order in your custom backward). Would that work for you?</p>

googlebot: <p>Thanks, at least this explains divergences I’m observing.</p>
<p>I’ll have to think, whether it is worth it to mitigate this with custom functions… One has to know all “client” nodes for that…</p>
<p>Good to be aware of this in general anyway.</p>

googlebot: <p>or will it be enough to produce two tensors in autograd.Function: shared_ctx_cpu, shared_ctx_gpu</p>
<p>is latent3backward and latent4backward accumulation order deterministic?</p>

albanD: <p>You can for a very generic custom function to do that:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.autograd import Function

class CustomScatter(Function):
    @staticmethod
    def forward(ctx, tensor, devices):
        ctx.inp_device = tensor.device
        outs = tuple(tensor.to(d) for d in devices)
        return outs

    @staticmethod
    def backward(ctx, *grads):
        res = 0.
        for g in grads:
            res += g.to(ctx.inp_device)
        return res, None

torch.manual_seed(123)
a = torch.rand(2, requires_grad=True)
devices = ["cpu", "cpu", "cuda", "cuda"]

all_t = CustomScatter.apply(a, devices)

loss = 0.
for t in all_t:
    loss += a * t.to("cpu")
loss.sum().backward()
print(a.grad)
</code></pre>
<blockquote>
<p>is latent3backward and latent4backward accumulation order deterministic?</p>
</blockquote>
<p>As long as they are “ready to run” (meaning that all their inputs are ready) at the same time and that they ran in the same order during the forward pass yes.</p>
Run two pre-trained models in parallel on the same GPU

Mamdouh_Aljoud: <p>I am trying to get an output from two different pre-trained models using the same input in parallel.<br>
I tried threading and multiprocessing, in threading, the code gets slower, in multiprocessing, the functions responsible for running the pre-trained models do not fire.</p>
<p>The code used is:</p>
<pre><code class="lang-auto">from multiprocess import Pool, Process, set_start_method, Queue


from threading import Thread
output1parallel = None
output2parallel = None

def getOutput1(q):

  print("IM HERE 1\n")
  global loader, resnet18, output1parallel
  for i,batch in enumerate(loader):
    currentBatch = batch.cuda()
    resnet18 = resnet18.cuda()
    output1parallel = resnet18(currentBatch).cpu()
    del currentBatch
  q.put('hello')

def getOutput2(q):
  
  print("IM HERE 2\n")
  global loader, densenet,output, output2parallel

  for i,batch in enumerate(loader):
    currentBatch = batch.cuda()
    densenet = densenet.cuda()
    output2parallel = densenet(currentBatch).cpu()

    del currentBatch
  q.put('hello')

if __name__ == '__main__':
  set_start_method('spawn', force=True)
  densenet.share_memory()
  resnet18.share_memory()
  start = time.time()
  q = Queue()
  p1 = Process(target = getOutput1, args=(q,) )
  p2 = Process(target = getOutput2, args=(q,) )

  p1.start() 
  p2.start() 
  print(p1, p1.is_alive())
  
  print(p2, p2.is_alive())
  p1.join()
  p2.join()
print("Time for parallel implementation: {}".format(time.time() - start))

</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/mamdouh_aljoud">@Mamdouh_Aljoud</a>, please use <code>torch.multiprocessing</code> instead. See examples here: <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.7.0 documentation</a></p>
<p>Besides, I would recommend first try using multiple CUDA streams in the same process, using one stream for each model. See examples here: <a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 1.7.0 documentation</a></p>
Compute full data gradient w.r.t model parameters

Amit_Chandak: <p>Hello All,<br>
I am new to Pytorch and  trying to compute gradient using whole dataset w.r.t given model_parameters. Can some one please tell me if the below function is correct.<br>
I have seen few related posts but those were not solving the problem which I am facing.<br>
Thank You in advance</p>
<pre><code class="lang-auto">def computeFullDataGradient(loader, model, criterion, cuda=True):

    data_size = len(loader.dataset)
    batch_size = loader.batch_size
    correction_factor = (batch_size * 1.0 / data_size)
    full_gradVec = list()
    for i, (input, target) in enumerate(loader):
        if cuda:
            input = input.cuda(non_blocking=True)
            target = target.cuda(non_blocking=True)

        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)
        output = model(input_var)
        loss = criterion(output, target_var)
        batch_gradVec = torch.autograd.grad(loss, model.parameters())
        if i == 0:
            for param in batch_gradVec:
                full_gradVec.append(torch.zeros_like(param))

        for param1, param2 in zip(full_gradVec, batch_gradVec):
            param1 += correction_factor * param2

    return full_gradVec
</code></pre>

albanD: <p>Hi,</p>
<p>Yes that should be computing the sum of the gradients wrt each sample.<br>
btw you don’t need to wrap things into Variables anymore as all Tensors are always Variables now.</p>

Amit_Chandak: <p>Thank You for your help.</p>
Do I need to also call detach() in a torch.no_grad() scope?

aaroswings: <p>My concern is memory leaks and cuda memory errors, and I’m still a new pytorch user, so I was curious about this.</p>
<p>I want to evaluate my model in a <code>no_grad</code> scope and save intermediate variables from it into lists outside of the scope of the block. I don’t want to accidentally store a reference to the computation block and cause cuda out of memory errors.</p>
<p>Do I need to call <code>detach()</code>, or will the context manager take care of it?</p>

albanD: <p>Hi,</p>
<p>No there is no need to use <code>.detach()</code> within the <code>no_grad</code> block because no autograd info is created.</p>
Train a GCN without any feature?

Lame: <p>I was using Graph convolutional network (GCN) in PyTorch to train a graph of nodes that has a couple of features, but I’m wondering if I have some of the nodes with no features (these nodes are kind of knowledge/metadata); can I still do the training on the graph including these nodes?</p>
<p>If not, what would be an alternative solution to keep training on them?</p>
<p>Any ideas?</p>

Lame: <p>Any suggestion or idea if that’s possible?</p>