akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Backprop Using Model Parallelism

username: lm_modeler
cooked: <p>I’ve distributed a very large transformer model across several GPUs and gotten it to generate predictions. It can do forward passes, but it gets hung up in backprop.</p>
<pre><code class="lang-auto">loss.backward()

c:\programdata\anaconda3\envs\context2\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--&gt; 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199 
    200     def register_hook(self, hook):

c:\programdata\anaconda3\envs\context2\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--&gt; 100         allow_unreachable=True)  # allow_unreachable flag
    101 
    102 

RuntimeError: expected device cuda:3 but got device cuda:0 (compute_types at ..\aten\src\ATen\native\TensorIterator.cpp:246)
(no backtrace available)
</code></pre>
<p>Moving the weights to the last GPU seems like it defeats the purpose of model parallelism. What’s the right approach to do this? It definitely won’t be able to train if I do it in a data parallel sort of way.</p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet so that we could reproduce this issue and debug it, please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: lm_modeler
cooked: <p>If someone else runs into this challenge and you don’t have a specific solution in mind, the best thing to do seems to be to carefully follow each and every tensor. In my case, I was using a transformer model whose wte layer was called on the lm_head, meaning that the lm_head had to (counter intuitively) be placed on the first gpu.</p>
Wav2Letter pretrained weights

username: robz
cooked: <p>Hello! I just read the Wav2Letter paper (<a href="https://arxiv.org/abs/1609.03193" rel="noopener nofollow ugc">https://arxiv.org/abs/1609.03193</a>), and would like to try reproducing it in pytorch.</p>
<p>I noticed that there’s a model available in torchaudio: <a href="https://pytorch.org/audio/models.html#wav2letter" rel="noopener nofollow ugc">https://pytorch.org/audio/models.html#wav2letter</a></p>
<p>But it doesn’t seem to be pretrained. My attempts to run it appear to produce nonsense output: <a href="https://colab.research.google.com/drive/1KpceNl5eT08kIpiX-TLjD5PkKZR7pK7u?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1KpceNl5eT08kIpiX-TLjD5PkKZR7pK7u?usp=sharing</a></p>
<p>Is there a way to use the weights that the paper used or some other set of pretrained weights? (Apologies if this is a silly question — I’m new to pytorch and ML frameworks, so I’m not sure if I’m missing some standard way to load pretrained weights)</p>
<p>Thanks!</p>
How to use Pinned Memory for ROS Node

username: soulslicer
cooked: <p>I have a network running within a ROS node. It produces a large tensor, which I have to copy to CPU and send it to another node</p>
<pre><code class="lang-auto"># Called once
pinned_tensor = torch.zeros((1,128,512,512)).pin_memory()

# Called in Loop
pinned_tensor[:] = self.network(input_tensor).cpu()
</code></pre>
<p>This is still slow even if i use pin memory. Am i using it incorrectly?</p>
When the next version of CUDA comes?

username: Sosom_Park
cooked: <p>I just wonder when the next CUDA version is available<br>
appreciate your reply</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You can install the nightly binaries with CUDA11 and <code>1.7</code> will be the next stable release with this CUDA version.</p>

username: Sosom_Park
cooked: <p>Thanks a lot for this quick reply. then when would I expect the next release of 1.7 ?</p>

username: ptrblck
cooked: <p>I would assume “soon”, as the <code>1.7</code> branch is already created: <a href="https://github.com/pytorch/pytorch/tree/1.7">https://github.com/pytorch/pytorch/tree/1.7</a><br>
However, I don’t know the exact schedule.</p>
How to use torchvision.transforms when I need to load 3D images

username: alan_ayu
cooked: <p>Now I need to train a network with 3D medical images,but when I use torchvision.transforms to do some operations such as ‘RandomCrop’ and ‘Normalize’,I get an error about number of arguments for <strong>init</strong> ,Can you give me some advice about that ?<br>
Thank you!</p>

username: bodokaiser
cooked: <p>torchvision does not support 3D volumes you will have to implement transforms yourself.</p>
<p>However you can easily copy code from <code>torchvision</code> and just extend it to support the extra dimension.</p>

username: alan_ayu
cooked: <p>ok,thanks for your reply</p>

username: fepegar
cooked: <p>Hi <a class="mention" href="/u/alan_ayu">@alan_ayu</a>,</p>
<p>This might come a bit late, but maybe you can use <a href="https://torchio.readthedocs.io/index.html" rel="nofollow noopener">TorchIO</a> for this. Someone asked about <code>RandomCrop</code> <a href="https://discuss.pytorch.org/t/data-agumentation-in-3d-images/86289">here</a>, and you have multiple normalization transforms in the <a href="https://torchio.readthedocs.io/transforms/preprocessing.html#preprocessing" rel="nofollow noopener">documentation</a>.</p>

username: agrija9
cooked: <p>Hi <a class="mention" href="/u/fepegar">@fepegar</a> fepegar,</p>
<p>I am facing a similar issue pre-processing 3D cubes from a custom turbulence data.</p>
<p>I have managed to compute the mean and std deviation of all my cubes (of dimensions 21x21x21) along the three channels by splitting the dataset in batches, then I compute mean and std per batch and finally average them by the total dataset size.</p>
<p>I know I can not apply transforms.Normalize directly to my cubes because that function works with 2D data. When I look at the TorchIO documentation link you provide, I don’t see explicitly any example of normalizing 3D data.</p>
<p>Do you know if there is such an example within the documentation?</p>

username: fepegar
cooked: <p>Hi, Alan. You have multiple normalization transforms here: <a href="https://torchio.readthedocs.io/transforms/preprocessing.html" rel="noopener nofollow ugc">https://torchio.readthedocs.io/transforms/preprocessing.html</a></p>
<p>TorchIO is optimized to work with 3D (and 4D) data! Don’t hesitate to open an issue in the TorchIO repository if you have any further questions.</p>

username: agrija9
cooked: <p>Hi Fernando. Thanks for the reply, I will have a closer look at the documentation.</p>
Tensor size error with gradients stored at each training iteration

username: NewDunce
cooked: <p>Hi everyone,<br>
I’m working on a project where I need to write an optimization algorithm for an MNIST program, and the update of the algorithm requires the gradient of the loss-max function at a previous iteration in addition to that at the current iteration. I tried storing the gradient at each iteration in a dictionary so that they can be added to the gradient at the subsequent iteration for the update. However, I keep getting the error message “RuntimeError: The size of tensor a (5) must match the size of tensor b (10) at non-singleton dimension 3,” and I don’t know what I’m doing wrong here. The algorithm is in my training function below:</p>
<pre><code class="lang-auto">def train(args, model, device, train_loader, optimizer, epoch, lr, iter):
    model.train()
    v0dict = {}
    grad0dict = {}
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()

        L = 2
        eta = 1/(2*L)
        S = pow(args.batch_size, 0.5)
        print(iter)
        for p in model.parameters():
            if iter == 1:
                grad0 = p.grad
                v0 = grad0/S
                v0dict[iter] = v0
                grad0dict[iter] = grad0
                p.data.add_(-eta*v0)
                print('grad size: {}\n'.format(p.grad.size()))
                print('data size: {}\n'.format(p.data.size()))
            else:
                v0 = v0dict[iter-1]
                grad0 = grad0dict[iter-1]
                print('grad size: {}\n'.format(p.grad.size()))
                print('data size: {}\n'.format(p.data.size()))
                v = v0 + (p.grad - grad0)/S
                v0dict[iter] = v
                grad0dict[iter] = grad0
                p.data.add_(-eta*v)

         optimizer.zero_grad()        
         iter += 1
</code></pre>
<p>Could someone show me where I’ve gone wrong? Any help would be greatly appreciated.</p>
Parameter updates with second order grads and in-place copy operation

username: Atsu3110
cooked: <p>Hi all , my question is related to the following question.</p>
<h1><a href="https://discuss.pytorch.org/t/example-of-second-order-derivative-single-variable/83399">Example of second order derivative (single variable)</a></h1>
<p>In the above post, the questioner proposed a MAML example.</p>
<p>My 1st question is : can torch.autograd work for in-place copy operation to update parameters with second order gradients ? For example, can the following toy MAML example work ?</p>
<pre><code class="lang-python">
import torch
from torch.autograd import grad

# f(x) = 2x + 3

x_train = torch.tensor(2., requires_grad=True)
y_train = torch.tensor(7., requires_grad=True)  # 2 * 2 + 3 = 7

x_test = torch.tensor(0.2, requires_grad=True)
y_test  = torch.tensor(3.4, requires_grad=True) # 0.2*2 + 3 = 3.4

W = torch.nn.Parameter(torch.tensor(1., requires_grad=True))

n_inner_update = 3

for step in range(20):
    y_pred = x_train * W + 3.
    loss = torch.abs(y_pred - y_train)
    inner_grad = grad(loss, W, create_graph=True)[0]

    for inner in range(n_inner_update - 1):
        y_pred = x_train * W + 3.
        loss = torch.abs(y_pred - y_train)
        inner_grad = grad(loss, W, create_graph=True)[0]
        with torch.no_grad():
            W.copy_(W - inner_grad* 0.002)

    y_pred_test = x_test * W + 3.
    outer_loss = torch.abs(y_pred_test - y_test)
    outer_loss_wrt_W = grad(outer_loss, W)[0]
    with torch.no_grad():
        W.copy_(W - outer_loss_wrt_W * 0.01)
    print(loss.item())

</code></pre>
<p>My Second question is : Why the following errors happns if torch.no_grad() is removed from the line ?<br>
Could a in-place copy operation be a node of computational graphs whose nodes have gradients ?</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
24         inner_grad = grad(loss, W, create_graph=True)[0]<br>
25         # with torch.no_grad():<br>
—&gt; 26         W.copy_(W - inner_grad* 0.002)<br>
27<br>
28     y_pred_test = x_test * W + 3.</p>
<p>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</p>
Multi class dice loss function

username: edshkim98
cooked: <p>Hello everyone, i am trying to use dice loss for my 3D point cloud semantic segmentation model.<br>
Although, I have implemented the function by referencing some of the codes, I am not sure whether it is correct as my IoU for my validation set does not increase compare to using cross entropy loss solely.<br>
Below is my function for multi class dice loss:</p>
<pre><code class="lang-auto">def diceLoss(prediction_g, label_g, num_class, epsilon=1):
    ls = []
    diceRatio_g = 0
    label_g= one_hot(label_g,15).to(device) #one hot encode 15 classes
    label_g = label_g.reshape(batch_size*16384, -1) #16384 is number of point cloud (batchsize*n_pts,15)
    prediction_g = prediction_g.reshape(batch_size*16384,-1) #reshape so (batchsize*n_pts,15)

    for i in range(num_class):
        pred = prediction_g
        label = label_g
        pred=torch.nn.functional.softmax(pred, dim=1)[:, i] #select ith index in softmax
        pred = pred.reshape(-1,1) #bs*pts*1

        
        diceLabel_g = label.sum(dim=0)
        diceLabel_g = diceLabel_g[i]
        
        dicePrediction_g = pred.sum(dim=0)
        
        diceCorrect_g = (pred * label)[:,0]
        diceCorrect_g = diceCorrect_g .sum(dim=0)
        #print(diceLabel_g,dicePrediction_g,diceCorrect_g)
        
        
        diceRatio_g += (2 * diceCorrect_g + epsilon) \
        / (dicePrediction_g + diceLabel_g + epsilon)
        
    loss = 1-(1/num_class)*diceRatio_g
    #print(loss)
    
    return loss
</code></pre>
<p>Please have a look and let me know if there is any problem. Thank you.</p>

username: Caruso
cooked: <p>Hi <a class="mention" href="/u/edshkim98">@edshkim98</a>,</p>
<p>I think it could be because of the <code>diceCorrect_g = (pred * label)[:,0]</code>, because <code>pred</code> is of shape <code>[B*N, 1]</code> and represents the models output for the selected class <code>i</code>, while <code>label</code> is still of shape <code>[B*N, C]</code>. So multiplying each other you get <code>diceCorrect_g</code>, which should be of shape <code>[B*N, C]</code> and then you select the first element of dim 1, which probably is mostly zero.<br>
Either you set <code>label = label_g[:, i]</code> (where i denotes your class) or I think you can actually remove the for loop totally and just do <code>diceCorrect_g = (label_g * softmax(prediction_g, dim=-1)).sum()</code> and <code>dicePrediction_g = dicePrediction_g .sum()</code> <code>diceLabel_g = diceLabel_g .sum()</code></p>
Save to a torchscript model in c++ (libtorch)

username: freezek
cooked: <p>I trained a model using libtorch, and want to save it still using libtorch. And I found <a href="https://github.com/pytorch/pytorch/issues/35464" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/35464</a>, indicate that</p>
<blockquote>
<p>we already have the ability to save to a  <code>std::ostream</code>  in C++</p>
</blockquote>
<p>Which refer to <code>torch::jit::Module::save()</code> ?<br>
And how to use it to save a torchscript model ?</p>
Using ConvTranspose2d and Conv2d on the same input

username: uertenli
cooked: <p>Hello,</p>
<p>I am currently working on HoughNet (<a href="https://arxiv.org/abs/2007.02355" rel="noopener nofollow ugc">https://arxiv.org/abs/2007.02355</a>) paper which has its code publicly available. I tried to change <code>ConvTranspose2d</code> in (<a href="https://github.com/nerminsamet/houghnet/blob/master/src/lib/models/networks/houghnet_large_hourglass.py#L280" rel="noopener nofollow ugc">https://github.com/nerminsamet/houghnet/blob/master/src/lib/models/networks/houghnet_large_hourglass.py#L280</a>) with Conv2d. However, I get this error message:</p>
<pre><code class="lang-auto">
RuntimeError: Given groups=1, weight of size [9, 1, 17, 17], expected input[2, 9, 128, 192] to have 1 channels, but got 9 channels instead
</code></pre>
<p>The original configuration is as follows:</p>
<pre><code class="lang-auto">Sequential(
  (0): ConvTranspose2d(9, 1, kernel_size=(17, 17), stride=(1, 1), padding=(8, 8), bias=False)
) 
input_size = torch.Size([2, 9, 1, 128, 192])
weight_size = torch.Size([9, 1, 17, 17])
</code></pre>
<p>The error occurs when the configuration is as follows:</p>
<pre><code class="lang-auto">Sequential(
  (0): Conv2d(9, 1, kernel_size=(17, 17), stride=(1, 1), padding=(8, 8), bias=False)
) 
input_size = torch.Size([2, 9, 1, 128, 192])
weight_size = torch.Size([9, 1, 17, 17])
</code></pre>
<p>As far as I understand, Conv2d should be applicable on this input. I tried replicate this on a small trial code. There I have to have a 4D input and I wasn’t able to replicate this error. Whenever, I have 4D inputs however, I can replace my conv layers with transposed conv and vice versa. Any idea why this might occur?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>The <code>weight_size</code> of the <code>nn.Conv2d</code> module is wrong, as it would be:</p>
<pre><code class="lang-python">conv = nn.Conv2d(9, 1, kernel_size=(17, 17), stride=(1, 1), padding=(8, 8), bias=False)
print(conv.weight.shape)
&gt; torch.Size([1, 9, 17, 17])
</code></pre>
<p>Are you changing the <code>.weight</code> attribute manually in your code?</p>

username: uertenli
cooked: <p>I simply changed the ConvTranspose2d with Conv2d and I get this error. However, the weight initially is assigned manually: <a href="https://github.com/nerminsamet/houghnet/blob/master/src/lib/models/networks/houghnet_large_hourglass.py#L276" rel="noopener nofollow ugc">https://github.com/nerminsamet/houghnet/blob/master/src/lib/models/networks/houghnet_large_hourglass.py#L276</a></p>

username: uertenli
cooked: <p>I would like to reduce 9 channels to 1, do I have to have the matrix ordering swapped manually as well?</p>

username: ptrblck
cooked: <p>As shown in my minimal code snippet, the weight shape is wrong in your code, so you would have to <code>permute</code> it to the expected shape, i.e. swap dim0 with dim1.</p>
`weight.grad.data` is not necessary

username: wsdgh
cooked: <p>One official tutorial has below comment</p>
<blockquote>
<pre><code># An alternative way is to operate on **weight.data** and **weight.grad.data**.
# Recall that tensor.data gives a tensor that shares the storage with
# tensor, but doesn't track history.
</code></pre>
</blockquote>
<p><a href="https://github.com/pytorch/tutorials/blob/7f48ac827389f3438113c1604ac416cca3c1090e/beginner_source/examples_autograd/two_layer_net_autograd.py#L59-L72" rel="noopener nofollow ugc">https://github.com/pytorch/tutorials/blob/7f48ac827389f3438113c1604ac416cca3c1090e/beginner_source/examples_autograd/two_layer_net_autograd.py#L59-L72</a></p>
<p>The <code>requires_grad</code> of <code>weight</code> is <code>True</code>, we get a tensor sharing same underlying data with <code>weight</code> by <code>weight.data</code>,  but this tensor’s <code>requires_grad</code> is <code>False</code>.</p>
<p>Since <code>weight.grad</code> is untracking tensor (<code>requires_grad = False</code>), It seems that <code>weight.grad.data</code> is same with <code>weight.grad</code>.</p>
<p>In a word, <code>.data</code> or <code>.detach()</code> is used for variable tensor i.e. <code>requires_grad</code> is <code>True</code>, for other tensor, it do nothing.</p>
<p>Please corret me.</p>

username: tom
cooked: <p>As you observe, in many circumstances, <code>a.grad</code> will not require gradient.<br>
This is, however, not universally true:</p>
<pre><code class="lang-auto">a= torch.randn((5,5), requires_grad=True)
(a**2).sum().backward(create_graph=True)
print(a.grad.requires_grad)
</code></pre>
<p>Note that <a href="https://pytorch.org/2018/04/22/0_4_0-migration-guide.html#what-about-data" rel="nofollow noopener"><code>.detach()</code> is universally better than <code>.data</code></a>, except when you use it as a shorthand for something involving <code>with torch.no_grad():</code> (which is the case for optimizers).</p>
<p>Best regards</p>
<p>Thomas</p>

username: wsdgh
cooked: <p>Got it,</p>
<p><code>weight.grad.data</code> can handle <code>backward of backward</code> scenario and don’t harm tensor that wont require gradient</p>

username: Sumit_Diware
cooked: <p>Hi,</p>
<p>I am trying to manually update weights, while using gradients from autograd (I need this as a base to implement my custom weight update algorithm later). So, based on the discussion above, is the following a good practice in general : using .grad.data and .grad.data.zero_() instead of .grad and .grad.zero_() ?</p>
<p>Note : I am using named_parameters() for debugging purposes only , the name is not used for compuatations inside the loop.</p>
<pre><code>    with torch.no_grad():
        for name, p in model.named_parameters():
            p.data = p.data - learning_rate*p.grad.data 
            # Manually zero the gradients after updating weights
            p.grad.data.zero_()
</code></pre>
<p>Regards, Sumit</p>
Pytorch training not stable with different batch size

username: cosinepi
cooked: <p>I have tried to run this pytorch code [<a href="https://github.com/oudalab/eeqa/blob/master/code/run_trigger_qa.py" rel="noopener nofollow ugc">https://github.com/oudalab/eeqa/blob/master/code/run_trigger_qa.py</a>]<br>
<a href="https://github.com/oudalab/eeqa/blob/master/code/script_args_qa.sh" rel="noopener nofollow ugc">https://github.com/oudalab/eeqa/blob/master/code/script_args_qa.sh</a></p>
<p>for event classification task with 8 K80 GPUs, it runs well with batch size set to 32, but when it set to batch size 16 or 8 it failed randomly at some epoch and steps.</p>
<p>Wonder any of you guys have similar issue and how to solve this problem.  the torch I use is 1.4.0, since there is error when update to pytorch 1.6.0 with this code which is know issue.</p>
<p>Is the code written above has any suspicious part?</p>
<p>Thank you guys for the help!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="cosinepi" data-post="1" data-topic="98143">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cosinepi/48/18824_2.png" class="avatar"> cosinepi:</div>
<blockquote>
<p>the torch I use is 1.4.0, since there is error when update to pytorch 1.6.0 with this code which is know issue.</p>
</blockquote>
</aside>
<p>What is the know issue preventing the update to <code>1.6</code> and what kind of errors are you getting?</p>
Custom loss function yields poor results

username: saul_santos
cooked: <p>Hi, basically I’m creating a custom loss function in the forward function of a nn.module. However when I add the function rk4_step the results are absurd. Without it they are ok. For instance, if I use one training point I get the same test loss with 100 points as I would get if I used 400 training points, which doesn’t make too much sense. So I think the problem may be related with the loss.backward() in the trainning which is not computing the gradients properly. Is everything in rk4_step compatible with backward?</p>
<pre><code>def forward(self, x):
    with torch.set_grad_enabled(True):
            time_step =torch.tensor(0.01)
            out=self._rk4_step(self.function, x, 0, time_step)
    return out
def function(self,x,t):
     self.n = n = x.shape[1]//2
     qqd = x.requires_grad_(True)
     L = self._lagrangian(qqd).sum()
     J = grad(L, qqd, create_graph=True)[0] ;
     DL_q, DL_qd = J[:,:n], J[:,n:]
     DDL_qd = []
     for i in range(n):
         J_qd_i = DL_qd[:,i][:,None]
         H_i = grad(J_qd_i.sum(), qqd, create_graph=True)[0][:,:,None]
         DDL_qd.append(H_i)
     DDL_qd = torch.cat(DDL_qd, 2)
     DDL_qqd, DDL_qdqd = DDL_qd[:,:n,:], DDL_qd[:,n:,:]
     T = torch.einsum('ijk, ij -&gt; ik', DDL_qqd, qqd[:,n:])
     qdd = torch.einsum('ijk, ij -&gt; ik', DDL_qdqd.inverse(), DL_q - T)

     return torch.cat([qqd[:,self.n:], qdd], 1)
        
def _lagrangian(self, qqd):
    x = F.softplus(self.fc1(qqd))
    x = F.softplus(self.fc2(x))
    # x = F.softplus(self.fc3(x))
    L = self.fc_last(x)
    return L
def _rk4_step(self, f, x, t, h):
    # one step of Runge-Kutta integration
    k1 = torch.mul(f(x, t),h)
    k2 = torch.mul(f(x + k1/2, t + h/2),h)
    k3 = torch.mul(f(x + k2/2, t + h/2),h)
    k4 = torch.mul(f(x + k3, t + h),h)
    return x + 1/6 * (k1 + 2 * k2 + 2 * k3 + k4)</code></pre>

username: saul_santos
cooked: <p>Just a new update. I realized that fc_last.bias is always None. Which makes me think that there is indeed an error.</p>
Why does DistributedDataParallel consume more GPU memory compared to DataParallel during AMP training?

username: Joe1Chief
cooked: <p>When I use DataParallel(), the maximum batch size can be set to <strong>512</strong>(cudnn.benchmark is disabled.), but DistributedDataParallel only supports setting batchSize to <strong>128</strong>.</p>
<p>Could cudnn cause such a problem?</p>
<p>This is the main structure of my code.</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    ...
    # Multi GPU
    print(f'Running DDP on rank: {args.local_rank}')
    torch.cuda.set_device(args.local_rank)
    dist.init_process_group(backend='nccl', init_method='env://')
    main()

def main():
    ...
    train_sampler = DistributedSampler(train_dataset)
        same_seeds(args.seed)

    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=args.batch_size,
        shuffle=(train_sampler is None),
        num_workers=args.num_workers,
        # worker_init_fn=_init_fn,
        pin_memory=True,
        sampler=train_sampler
        )
       ...
       # Creates a GradScaler once at the beginning of training.
       scaler = GradScaler()

       # Distribute model across all visible GPUs
       net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net)
       net = DDP(net, device_ids=[args.local_rank], output_device=args.local_rank)
       cudnn.benchmark = True  # enable cudnn
       ... 
       for epoch in range(start_epoch, args.epochs):
           train_sampler.set_epoch(epoch)
           train_loss, train_acc, batch_time = train(epoch, net, train_loader, criterion, optimizer,       
               warmup_scheduler, scaler)
     ...

def train(epoch, net, train_loader, criterion, optimizer, scheduler, scaler):
    """Train for one epoch."""
    net.train()
    for batch_idx, (images, targets) in enumerate(train_loader):
        with autocast():
            images, targets = images.cuda(), targets.cuda()
            logits = net(images)
            loss = criterion(logits, targets)
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
        ...
</code></pre>

username: ptrblck
cooked: <p>AMP shouldn’t use more memory and I assume you are trying to use the global batch size in each process and thus GPU.<br>
As explained <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769/2">here</a> you should set the <code>batch_size</code> for each GPU as the local batch size (by dividing the global batch size by the number of GPUs).</p>
Hello, I want to know why the loss of this linear regression has not been reduced.

username: zx_l
cooked: <p>import torch as t<br>
from matplotlib import pyplot as plt<br>
from IPython import display</p>
<p>def number(batch_size):<br>
x = t.rand(batch_size , 1)*5<br>
y = 2 * x + 3 + t.randn(batch_size , 1)<br>
return x , y<br>
x , y = number(500)</p>
<p>model = t.nn.Sequential(<br>
t.nn.Linear(1, 1)<br>
)</p>
<p>loss_fn = t.nn.MSELoss()<br>
optimizer = t.optim.Adam(model.parameters() , lr = 0.01)</p>
<p>for i in range(5000):<br>
y_pred = model(x)<br>
loss = loss_fn(y_pred , y)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
print(loss)</p>

username: ptrblck
cooked: <p>The loss is reduced from <code>~104</code> to <code>~1.01</code> in my environment.<br>
Note that you can post code snippets by wrapping them into three backticks ```, which makes debugging and helping easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Data Augmentation for my dataset for image classification task

username: abdulhafeezco
cooked: <p>I want to perform a dog bred classifier task. The number of dogs per breed that I have is different and I want to perform data augmentation to make the frequency to be the same or to make the data balanced. Below is my code. How can I modify the below to not just transform my original dataset but transform and append it to existing dataset and eventually have the same number of classes</p>
<pre><code class="lang-auto">train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                           transforms.RandomHorizontalFlip(),
                                           transforms.Resize(224),
                                           transforms.CenterCrop(224),
                                           transforms.ColorJitter(
                                                                    brightness=0.1*torch.randn(1),
                                                                    contrast=0.1*torch.randn(1),
                                                                    saturation=0.1*torch.randn(1),
                                                                    hue=0.1*torch.randn(1)),
                                           transforms.ToTensor(),
                                           transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

train_datasets = datasets.ImageFolder( os.path.join(data_dir,"train") , transform=train_transforms)
trainloader = torch.utils.data.DataLoader(train_datasets, batch_size=32, shuffle=True)
# dog_classes = trainloader.classes
data, target = iter(trainloader).next()
</code></pre>
Requires_grad becomes false after pooling layer

username: hzh2020
cooked: <p>Hi,<br>
when using eval mode, I need to use gradients in part of my model. In forward method, I set the requires_grad flag of parameters to True and expect the following module will calculate and store gradients, but after the maxpooling layer, requires_grad flag becomes false and I don’t know why…</p>
<pre><code class="lang-auto">        xl1 = self.conv_block1(xf3)
        xl2 = self.conv_block2(xf4)
        xl3 = self.conv_block3(xf5)
        
        for x in [xl1, xl2, xl3]:
            x.requires_grad_(True)
        xl1.register_hook(self.save_gradient)
        xl2.register_hook(self.save_gradient)
        xl3.register_hook(self.save_gradient)
        
        xl1 = self.max1(xl1) #xl1.requires_grad is False, here self.max1 = nn.MaxPool2d(kernel_size=56, stride=56)

</code></pre>
<p>Thanks in advance!</p>

username: hzh2020
cooked: <p>I find out that I made a mistake that I wrap these code in torch.no_grad() <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
When to implement backward()?

username: amirhf
cooked: <p>I am reading through the <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">‘Extending PyTorch’</a> that implements a new <code>nn.Module</code> and uses the function that has both <code>forward()</code> and <code>backward()</code> implemented. My question is if I want to define a new module that just adds a random weight to the input tensor and I define it like this:</p>
<pre><code class="lang-auto">class CustomLayer(nn.Module):
    def __init__(self, input_dim):
        super(CustomLayer, self).__init__()
        self.input_dim = input_dim
        self.weight = nn.Parameter(torch.Tensor(input_dim))
        self.weight.data.uniform_(-0.05, 0.05)

    def forward(self, input):
        return torch.add(input, self.weight)
</code></pre>
<p>Then do I need to implement a <code>backward()</code> as well? Because in the forward method, I am not using a <code>Function</code>, but just an already implemented operation in PyTorch.<br>
Does having an <code>nn.Parameter()</code> in my contructor requires me to write a <code>backward()</code> function? Do they get updated without havin a <code>backward()</code>?<br>
To my understanding if only you implement a custom function and you want to somehow manipulate the backward operations, then you need to write a <code>backward()</code> function. So for example, the module in the tutorial can be simply written as:</p>
<pre><code class="lang-auto">class Linear(nn.Module):
    def __init__(self, input_features, output_features, bias=True):
        super(Linear, self).__init__()
        self.input_features = input_features
        self.output_features = output_features
        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(output_features))
        else:
            self.register_parameter('bias', None)

        self.weight.data.uniform_(-0.1, 0.1)
        if self.bias is not None:
            self.bias.data.uniform_(-0.1, 0.1)

    def forward(self, input):
        return torch.add(torch.matmul(input, self.weight), self.bias)
</code></pre>
<p>without a <code>backward()</code> method!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes you only need to write a backward function if:</p>
<ul>
<li>You want to compute something that is not the “true” gradient that is computed by the autograd</li>
<li>Your forward function is not handled by the autograd (because you use custom function or third party library)</li>
</ul>
<p>And the example in the tutorial is indeed artificial but it is just an example on how to do that for a simple function.</p>

username: amirhf
cooked: <p>That makes sense. Thank you for the clarification!</p>
Need i to create a copy of the original image and output if there are multiple losses?

username: micklexqg
cooked: <p>I am working on an image-to-image translation task.<br>
There are multiple losses apart from L1 and GAN loss.<br>
I notice that the copies of the original image and output are created for GAN loss.<br>
I do not understand why  would we do that?<br>
if I computed another loss, need i also create a copy?<br>
I tried with several different cases.<br>
one case is without copy, but the result (it is computed using  the original image or output, and it will be used for computing the loss) is assigned  to a new object.<br>
the error is: element 0 of tensors does not require grad and  does not have a grad_fn.  (also came across: leaf variable has been moved into the graph interior )<br>
then  I tried with a copy, but came across: no grad accumulator for a saved leaf.<br>
what should i do?</p>

username: micklexqg
cooked: <p>Some code is as follows:<br>
<span class="hashtag">#loss_l1</span> = criterionL1(output, gt)<br>
loss_l1 = 0<br>
<span class="hashtag">#with</span> torch.no_grad():<br>
H_gt = gt<br>
H_gt = get_H(gt, H_gt)<br>
H_output =  output<br>
H_output = get_H(output, H_output)<br>
H_loss_l1 = criterionL1(H_output, H_gt)</p>
<p>if i use loss_l1 = criterionL1(output, gt),  the error is:   one of the variables needed for gradient computation has been modified by an inplace operation…<br>
but i just set  loss_l1 = 0, it is ok.<br>
why could it be so, what is the problem?</p>
How to score "normality" of given vector?

username: hadaev8
cooked: <p>I have an encoder what compress sample to 100d vector.<br>
And I have gan what generate samples from 100d vector sampled from normal distribution.<br>
So I want to enforce the first model to output values like from normal distribution.<br>
Should i just use loss = KLDivLoss(x, y) ?</p>
Unbalanced Data- Poor results with weights

username: krioux5
cooked: <p>Hello, I am using crossentropyloss to segment CT scans. My data has 8 small structures of labels 1,2…8, and label 0 corresponds to areas that do not belong to the structures. In total, the structures make up only about 1.1% of the data, with the smallest structure being only about 0.001% and the largest around 0.9%. For testing purposes to debug weighting I am doing training where I set all the structures to a label of 1 and achieved reasonable results with no weights:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b3f5c6f20b247dd9df243f1635208193f099b61.png" alt="image" data-base62-sha1="fiKQ4dg4c8a6xqooisokrKbDX8d" width="222" height="204"><br>
when compared to the input:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c8cc7c97cb5811ee732e6604e7bf7afd05ba8f3.png" alt="image" data-base62-sha1="aVbUjVjwfBHsrHgVvdxW0s8AkrV" width="252" height="224"><br>
But I assumed that better results could be attained by adding weights to CrossEntropyLoss. I heard that using the inverse proportionality of labels was the correct weighting( in this case 0.9 for structures, 0.1 for non structures), but this severely overrepresented the structure labels:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3a5167164c5698ec9fa47a5542c010a41cae147.png" alt="image" data-base62-sha1="yLnIT7iaerCW83cPlMZKpj4XIKH" width="248" height="197"><br>
I trained again with less severe weights (0.8 for structures, 0.2 for non structures) and the pattern held. Overrepresentation of structures but less so.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dce561558340a8c60ddd406b266e0f2845884ba.png" alt="image" data-base62-sha1="dnQtE29YHMyRAMdgOm6L51tbn5g" width="392" height="320"><br>
Any recommendations? Can I use any other techniques like weighted random samplers in segmentation, or is there a way I can make weights actually help my results?</p>
<p>Thanks, Kyle.</p>
Understanding Autograd function

username: kumar1997
cooked: <p>Can I get any study materials for understanding Autograd function?</p>

username: albanD
cooked: <p>Which part in particular?</p>
<p>This blogpost might give you a good introduction: <a href="https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd">https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd</a></p>
How to train model to avoid forgetting of previous knowledge?

username: vishalthengane
cooked: <p>Hi there,</p>
<p>I have a model with trained weights <code>W</code> and I want to train that model on different task without forgetting previous knowledge. I have another matrix <code>V</code> of the same size as <code>W</code>, and</p>
<pre><code class="lang-auto"># A.requires_grad=True
# B.requires_grad=False
V = torch.matmul(A, B) # then V.requires_grad=True
</code></pre>
<p>New Weights <code>W1 = W + V</code> so W1 has automatically <code>requires_grad=True</code> and I don’t want to update <code>W1, W, V, and B</code> I only want to update <code>A</code>, How I can do that?</p>
<p>Thanks!</p>

username: vishalthengane
cooked: <p>Sorry, Solve it. It’s already there. I got confused previously.</p>
<p>Thanks!</p>
Function torch.quantile cannot be found

username: Gil_Pinsky
cooked: <p>Using PyTorch version 1.6 (on Conda, Windows, CUDA 10.2)<br>
AttributeError: module ‘torch’ has no attribute ‘quantile’</p>

username: ptrblck
cooked: <p><code>torch.quantile</code> is not available in <code>1.6</code> so you would have to install the nightly binary using the instructions from <a href="https://pytorch.org/get-started/locally/">here</a> or wait for the release of <code>1.7</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Gil_Pinsky
cooked: <p>Aha, I see! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Is it possible to instantiate a model from its repr output?

username: Shisho_Sama
cooked: <p>I wonder is it possible to instantiate a model from its repr output in Pytprch?</p>

username: tom
cooked: <p>No. It might work if its only classes from the standard library (they (roughly?) advertise the parameters you could instantiate them with), but the moment you have anything custom, the lack of code for the <code>forward</code> would seem to hit you. You would need the state dict, too.<br>
Currently the main way to save models with complete information is via the JIT. If you stare down the git tree hard enough, you see that something is in the works that would allow to save models with source code and data, but it’s not ready yet, i.e. the state is described as follows and it won’t be in PyTorch 1.7:</p>
<blockquote>
<p>This module is a prototype and should not be used for anything real. The<br>
APIs and package format are subject to change without warning. Issues filed<br>
against this module may not receive support.</p>
</blockquote>
<p>Best regards</p>
<p>Thomas</p>

username: Shisho_Sama
cooked: <p>Thanks a lot tom. really appreciate it. Yes I know about the jit. but this is a pruned model, so I was kind of searching to see if I could avoid a retraining from scratch or manually creating a custom model definition for it.</p>
Mixed Precision or FP16 training in libtorch

username: a_d
cooked: <p>Hello all,<br>
Is there a way to train a model in mixed precision/ half-precision in libtorch using <code>amp</code>?<br>
If there is no <code>amp</code> package in libtorch, can we somehow use the Nvidia Apex library in C++?</p>
<p>TIA</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/a_d">@a_d</a><br>
I don’t think we support amp in libtorch at this moment. <a class="mention" href="/u/mcarilli">@mcarilli</a></p>
<p>You can find some workaround in this post.<br>
‘<a href="https://discuss.pytorch.org/t/deploy-mixed-precision-model-in-libtorch/89046" class="inline-onebox">Deploy mixed precision model in libtorch</a>’</p>
<p>I think Apex is a mix of its own libs and pytorch libs, not easy to use it purely from C++ I guess.</p>

username: a_d
cooked: <p>Thanks a lot for your reply <a class="mention" href="/u/glaringlee">@glaringlee</a>, I will have look at it</p>
About logsumexp function in C++ libtorch

username: Chen0729
cooked: <p>Hi ,All</p>
<p>I would like to convert the following line from<br>
Pytorch to libtorch in C++.</p>
<p>X1=log_sum_exp(x)</p>
<p>I try to convert it in c++ as</p>
<p>Torch::X1=torch::log_sum_exp(x)</p>
<p>But it says incorrect.</p>
<p>Could anyone give suggestions about how to correct it please ?</p>
<p>Thanks.<br>
Xu</p>

username: glaringlee
cooked: <p>Do you mean from torch.logsumexp?<br>
‘<a href="https://pytorch.org/docs/stable/generated/torch.logsumexp.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.logsumexp.html</a>’<br>
If so, we do have torch::logsumexp(tensor, {}, bool keepdim = false) where {} is a list of dimension you want to reduce.<br>
’</p>
How to go from XYZImpl to XYZ?

username: abhinavkulkarni
cooked: <p>Hi,</p>
<p>I have a nested sequential model (I subclassed <code>nn::Sequential</code> so as to have a non templated <code>forward</code> method).</p>
<pre><code class="lang-cpp">struct MySequentialImpl : torch::nn::SequentialImpl {
public:
    template&lt;typename... Modules&gt;
    explicit MySequentialImpl(Modules &amp;&amp;... modules);

    torch::Tensor forward(torch::Tensor x);
};

TORCH_MODULE(MySequential);
</code></pre>
<p>Let’s say I have a module like the following:</p>
<pre><code class="lang-cpp">    auto module = MySequential(MySequential(nn::Conv1d(6, 6, 3), nn::ReLU()), nn::Identity());

    for (auto &amp;child: module.ptr()-&gt;children()) {
        MySequentialImpl *ptr = child-&gt;as&lt;MySequential&gt;();
        if (ptr) {
            std::cout &lt;&lt; ptr-&gt;name() &lt;&lt; std::endl;

        }
    }
</code></pre>
<p>How do I convert <code>MySequentialImpl *ptr</code> above to a <code>std::shared_ptr&lt;MySequential&gt;</code>?</p>
<p>Thanks</p>

username: glaringlee
cooked: <p>You never have std::shared_ptr&lt;MySequential&gt;.<br>
The underlying is always std::shared_ptr&lt;MySequentialImpl&gt;.</p>
<p>MySequential is defined like this:<br>
‘<a href="https://github.com/pytorch/pytorch/blob/a54416d208cd024779455c7b44e7b86f51a3da45/torch/csrc/api/include/torch/nn/pimpl.h#L202" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/a54416d208cd024779455c7b44e7b86f51a3da45/torch/csrc/api/include/torch/nn/pimpl.h#L202</a>’</p>
<p>It is actually a ModuleHolder of MySequentialImpl, and ModuleHolder always hold a shared_ptr of MySequentialImpl.</p>
<p>If you want your MySequential obj (not shared_ptr) back, you can do<br>
auto obj = torch::nn::ModuleHolder&lt;MySequentialImpl&gt;(module.ptr&lt;MySequentialImpl&gt;(0));<br>
but you have to know the type of the module at each position in your module.</p>
<p>Normally, you just need to use the Impl ptr to call -&gt;forward(), no?</p>
Cifar10 Classifier Producing NaNs on my phone

username: SpaceCowboy850
cooked: <p>So this is just for funsies, but I got pytorch built and running in UserLand on my Samsung Galaxy S9 (Arm64).  It can even train a very very simple toy net that learns to multiply by 10.</p>
<p>However, when I train this:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" target="_blank" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I get NaNs after the first few iterations.  I <em>don’t</em> get NaNs on this same code if I’m training on the CPU on my Windows10 dekstop (same code, same batch size, etc).</p>
<p>I’ve tried to put in batch norm layers after the convs and linear layers.  Didn’t help.  I tried not using CrossEntropyLoss and instead did a softmax + NLLLoss, also didn’t help.</p>
<p>Nothing is “crucial” about getting this to work, I just assumed CPU mode would give same results on two different systems.  It would be interesting to know what the difference is.  Maybe my pytorch build is broken in some non-obvious way?</p>
<p>Anyone tried this?</p>
Trouble using cuda inside of Docker

username: dwiel
cooked: <p>Hello, I am trying to get pytorch running inside docker, but not having much luck. Here is some terminal output first from the host machine, then immediately after running inside of a base nvidia/cuda docker image:</p>
<pre><code class="lang-auto">$ nvidia-smi
...
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
...
$ nvidia-docker run --rm -it nvidia/cuda:10.2-base-ubuntu18.04 /bin/bash
# nvidia-smi
...
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
...
# apt-get update &amp;&amp; apt-get install -y python3 python3-pip &amp;&amp; pip3 install torch torchvision
...
# python3 -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.6.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.5 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 SUPER
Nvidia driver version: 440.44
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] Could not collect
</code></pre>
<p>I am using prepackaged nvidia/cuda docker containers and a minimal python install, so I’m not sure what could be going wrong. All of the problems I have found other people have so far have caused <code>nvidia-smi</code> to not work properly, but in my case, it is working fine, yet torch still says CUDA is not available.</p>

username: dwiel
cooked: <p>I wiped the OS, re-installed everything from scratch and it is now working.</p>