akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Terminating gradient descent when at minima

Omroth: <p>Hi guys,</p>
<p>The gradient descent part of my loop looks like this:</p>
<pre><code class="lang-auto">torch::Tensor loss = torch::nn::functional::mse_loss(guess, target);
optimizer.zero_grad();
loss.backward();
optimizer.step();
</code></pre>
<p>I’d like to terminate the loop when guess has stopped changing - would I just look for the size of the difference between previous guess and guess, or is there a better way?</p>
<p>Thanks.</p>

Omroth: <p>I suppose it could also be that change in loss is small - again, is this something to do manually, or is there an inbuilt function for it?</p>

tom: <p>You could also use the total norm of the gradients as a criterion and stop when it gets really small, indicating that you are close to a critical point. This saves you from needing to keep around a copy of the parameters.<br>
It won’t be exactly the same when you have momentum and adaptive optimizers, but it’s not an uncommon stopping criterion in gradient descent.</p>
<p>Best regards</p>
<p>Thomas</p>
Gradient print during backward call

cbd: <p>In below code, why the gradient prints number of times as epoch is increase?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class Net(torch.nn.Module):

    def __init__(self):       
        super(Net, self).__init__()
        self.w=torch.nn.Parameter(torch.ones([2,2]))
       
    def forward(self, x):
        out= x * self.w
        self.w.register_hook(lambda grad: print("GRADIENT OF  SELF.W IS"))
        self.w.register_hook(lambda grad:print(grad))
        return out


size1=2
learning_rate = 0.0005
net=Net()
criterion=torch.nn.L1Loss()
optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate,momentum=0)
x=torch.ones(([1,1,size1,size1]))
target=torch.zeros(([1,1,size1,size1]))#,requires_grad=True)
target[0,0,:,:]=0.95
x.requires_grad=True

for i in range(0,5):
  print("*** Epoch number is ",i)
  x1=net(x)
  l=criterion(target,x1)
  l.backward()
  optimizer.step()
  optimizer.zero_grad()</code></pre>

albanD: <p>Hi,</p>
<p>It’s because self.w is always the same Tensor, and every time you do a forward, you add another hook onto it.<br>
These hooks don’t go away by them selfves so you just have more and more of them here.</p>

cbd: <p>Is it consume memory or some diverse effect if we don’t remove it?</p>

albanD: <p>No.<br>
But if you add a new one at each iteration, it is expected that at iteration n, you will have n hooks being called.</p>
<p>I think what you want here is to move the hook registration to the <code>__init__</code> of your nn.Module so that it is done only once. and so at every iteration, the hook will be called once.</p>

cbd: <p>You just point out the right solution. I am not understood how defining in “<strong>init</strong>”, it calls during backward in every epoch? I just assumed till now “<em>init</em>_” doesn’t play role in forward or backward. It just instantiated once during model creation.</p>

albanD: <p>Your Tensor <code>self.w</code> is defined once during the init and then re-used for every forward in the <code>forward</code> method.<br>
So if you add a hook to that Tensor, it will be taken into account at every forward as the Tensor is re-used at every forward.</p>
<p>In your code above, if you register the hook on <code>out</code> you won’t see this behavior anymore because <code>out</code> is a new Tensor at every forward. The behavior you observe here happens because <code>self.w</code> is the same Tensor that is re-used at every forward.</p>
Calculate first model's gradients based on another model's loss

abhibha1807: <p>Hello,<br>
I am new to Pytorch and I want to implement an algorithm of the following format,</p>
<pre><code class="lang-auto">#input1...
output1 = model1(input1)

output = model1(some_input)
#calling output as input2 from now on

output2 = model2(input2)
loss2 = fn(output2)
loss2.backward()
</code></pre>
<p>If am not wrong, loss2.backward() should compute the gradients wrt the parameters of model1 as the ‘output’ from model1 is input to model2. But when I check the parameters using the code,</p>
<pre><code class="lang-auto">for params in model1.parameters():
   print(params.grad.norm())
</code></pre>
<p>The value of the ‘norm’ is constant even after the backward pass, indicating no gradients were calculated for model1’s parameters. I wonder why is this happening? I am using the<br>
<a href="https://huggingface.co/transformers/model_doc/encoderdecoder.html#encoderdecodermodel" rel="noopener nofollow ugc">hugging face</a> library for the models.<br>
This is my reproducible code</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
from transformers import BertModel, BertForMaskedLM, BertConfig, EncoderDecoderModel
model1 = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') 
model2 = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') 

#fake inputs 
en_input=torch.tensor([[1,2], [3,4]])
de_output=torch.tensor([[3], [4]])
lm_labels=torch.tensor([[5], [6]])

output1 = model1(input_ids=en_input, decoder_input_ids=de_output, labels=lm_labels)

output = model1.generate(input_ids=en_input, decoder_start_token_id=0) #input2

new_labels=torch.rand(2, output.shape[1])
new_labels=new_labels.long()

output2=model2(input_ids=en_input, decoder_input_ids=output, labels=new_labels)

prediction_scores = output2[1]
predictions = F.log_softmax(prediction_scores, dim=2)
loss2 = (predictions.sum() - output.sum()).sum() # some loss
loss2.backward(retain_graph=True)
#now I check model1's parameters
</code></pre>

ptrblck: <p>You are right in the assumption that passing the output of one model to the other would keep the computation graph intact and <code>model1</code> should get valid gradients as long as the computation graph is not detached.<br>
I don’t know what <code>model1.generate</code> does so could you check, if <code>output</code> has a valid <code>.grad_fn</code>?<br>
If that’s not the case, this would mean that <code>output</code> is detached and Autograd won’t backpropagate through it. On the other hand, if the <code>grad_fn</code> is set, could you check the <code>.grad</code> attributes of <code>model1</code>'s parameters before and after the first <code>backward()</code> call (they should be <code>None</code> before and contain valid values after)?</p>

abhibha1807: <p>Thanks a lot for replying! Turns out model1.generate() produces tensors of type integer and hence they have no ‘grad_fn’ associated with them. Hence I wasn’t able to back-prop till model1.</p>
Class Activation Map Final Layer

Jam: <p>I am trying to create a CAM for my binary classification model. My model has a CNN (ResNet50) + bidirectional LSTM structure for which the last layers look like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f61a688994be7d2b5f27e3bd8751dbf632d41697.png" data-download-href="https://discuss.pytorch.org/uploads/default/f61a688994be7d2b5f27e3bd8751dbf632d41697" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f61a688994be7d2b5f27e3bd8751dbf632d41697.png" alt="grafik" data-base62-sha1="z782jBrvGUJ0kiV7KjSQzvwG3gr" width="690" height="367" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">711×379 8.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I know that for CAM usually we grab the last Conv layer and then use a softmax. But which layer do I have to use considering my model structure?</p>
Prediction value of ResNet 3D Pytorch Multilabel classification is too small

Chamroukhi: <p>Hi,<br>
i trained ResNet3D using dataset multilabel and im using BCELoss with sigmoid layer i have 31% topKcategorical accuracy (i have 23 labels), i trying to predict test-set but model given all labels a proba too small(0.0…),<br>
example:<br>
[0.061, 0.032, 0.036, 0.009, 0.024, 0.009, 0.025, 0.017, 0.021,<br>
0.014, 0.013, 0.017, 0.078, 0.01 , 0.068, 0.014, 0.068, 0.035,<br>
0.04 , 0.044, 0.03 , 0.04 , 0.015]<br>
any help ?</p>
Loss.backward() found long, expected float

miturian: <p>I am trying to code a test case for multi task learning, using my own loss function. The idea is that the output layer is 3-dimensional,  the first output is used for 1D regression, the last two are used for 2-class classification. So, my combined loss function is a weighted sum of L1 loss and CELoss.</p>
<p>However, pytorch is complaining about my datatypes?</p>
<pre><code class="lang-auto">class my_loss:

    def __init__(self,weights):
      self.weights=weights

      #age loss:
      self.L1=nn.SmoothL1Loss(reduction='mean',beta=0.05)

      #gender loss:
      self.CE=nn.CrossEntropyLoss(reduction='mean')

    def __call__(self,output,target):
      loss = self.weights[0]*self.L1(output[:,0],target[:,0])+self.weights[1]*self.CE(output[:,1:3],target[:,1])

      return loss

#testing:
it=iter(dataloaders_dict['train'])
X,y=it.next()

criterion = my_loss(weights=(1,1))

out=model_ft(X)
loss=criterion(out,y)
print(y.dtype)
print(X.dtype)
print(out.dtype)
print(loss.dtype)

loss.backward()
</code></pre>
<p>This returns:</p>
<pre><code class="lang-auto">torch.int64
torch.float32
torch.float32
torch.float32
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-88-fc96d4e3072b&gt; in &lt;module&gt;()
     31 print(loss.dtype)
     32 
---&gt; 33 loss.backward()

1 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    145     Variable._execution_engine.run_backward(
    146         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--&gt; 147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    148 
    149 

RuntimeError: Found dtype Long but expected Float
</code></pre>
<p>If only my targets are long, how come loss.backward is finding long’s, and how do I fix it?</p>

ptrblck: <p>You are using a <code>target</code> tensor as a <code>LongTensor</code>, which is fine for <code>nn.CrossEntropyLoss</code>, but will fail for <code>nn.SmoothL1Loss</code>, so you would have to transform it into a <code>FloatTensor</code>:</p>
<pre><code class="lang-python">loss = self.weights[0]*self.L1(output[:,0],target[:,0].float())+self.weights[1]*self.CE(output[:,1:3],target[:,1])
</code></pre>

miturian: <p>thanks, yes, that solves it. But, do you have any idea why the error does not arise in nn.SmoothL1Loss?</p>

ptrblck: <p>The error is raised in <code>nn.SmoothL1Loss</code>, but unfortunately in the backward pass, and I’m unsure if accepting <code>LongTensor</code>s in the forward is a bug or a feature, so feel free to create an issue on GitHub, so that it can be tracked and fixed, if needed.</p>
Testing accuracy significantly greater than Training and Validation accuracies

Lucky_Magna: <p>I’m training a CNN using Transfer Learning, resnet34 to be precise, the accuracies of training and validation set are not improving. They are stuck at 72% to 18% respectively. I trained the model for 25 epochs, and it didn’t show any signs of improvement at all. But when I tested the model, the accuracy was nowhere near the training and validation set, it was 86%. Can anyone explain why this is happening?</p>
ImportError: cannot import name 'find_classes' from 'torchvision.datasets.folder'

Wissal_Guedri: <p>Hello,</p>
<p>When I m trying to create a trained-NN using <em>GDVB framework</em>,  I get the following error.</p>
<p>Traceback (most recent call last):<br>
File “/home/wg/anaconda3/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/wg/anaconda3/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/wg/Documents/GDVB/lib/R4V/r4v/<strong>main</strong>.py”, line 6, in <br>
from . import cli<br>
File “/home/wg/Documents/GDVB/lib/R4V/r4v/cli/<strong>init</strong>.py”, line 5, in <br>
from … import distillation<br>
File “/home/wg/Documents/GDVB/lib/R4V/r4v/distillation/<strong>init</strong>.py”, line 11, in <br>
from .data import get_data_loader<br>
File “/home/wg/Documents/GDVB/lib/R4V/r4v/distillation/data.py”, line 12, in <br>
from torchvision.datasets.folder import (<br>
ImportError: cannot import name ‘find_classes’ from ‘torchvision.datasets.folder’ (/home/wg/Documents/GDVB/.venv/lib/python3.8/site-packages/torchvision/datasets/folder.py)</p>
<p>Best regards<br>
Wissal</p>

eqy: <p>It looks like this depends on a very recent change to torchvision which may not have even made it to the conda packages yet.<br>
<a href="https://github.com/pytorch/vision/commit/f9af70a98244b30f645fb8a033b6e4ce59ca020f#diff-5d4aa766c846fb9465acd2f311d8f397f98f3baeb8c0b4f04d8a26863d9dd8e3" rel="noopener nofollow ugc">Make DatasetFolder.find_classes public (#3628) · pytorch/vision@f9af70a (github.com)</a></p>
<p>You can try updating your version of torchvision and if that doesn’t work you might need to build and install from source.</p>

ptrblck: <p>Additionally to <a class="mention" href="/u/eqy">@eqy</a>’s suggestions you could also install the nightly binary (select “Preview(nightly)” <a href="https://pytorch.org/get-started/locally/">here</a>) to get the most recent changes in the binaries.<br>
I’ve checked that a relatively new <code>torchvision</code> version <code>0.10.0.dev20210516</code> contains the <code>find_classes</code> method.</p>
Choose between tensorflow or pytorch

David_Smit: <p>I think in the future Peturch will replace Tensorflow. What do you think, friends? And which framework do you think will become more widespread in the future?</p>

David_Smit: <p>plz answer my topic…</p>

pascal_notsawo: <p>In my humble opinion I don’t think this is the right place to discuss this <a class="mention" href="/u/david_smit">@David_Smit</a><br>
There are discussions elsewhere on the subject, like on <a href="https://www.reddit.com/r/MachineLearning/comments/7ziagd/d_discussion_on_pytorch_vs_tensorflow/" rel="noopener nofollow ugc">reddit</a> for example.</p>
<p>But if you want to know if you have to use tensorflow or pytorch for a particular task, I could try to give my opinion on that</p>

ptrblck: <p>I agree that this discussion board might not be the best place to discuss these vague questions, as you might get biased answers in the best case.<br>
I’m sure a lot of ML practitioners could give you pro and cons for all frameworks and I see it in a pragmatic way: use whichever framework fits into your mindset and use case.<br>
As always, if you are missing something in PyTorch, let us know and we’ll certainly take a look into it.</p>

Bernardino152: <p>If you are in academia and are getting started, go for Pytorch. It will be easier to learn and use. If you are in the industry where you need to deploy models in production, Tensorflow is your best choice. You can use Keras/Pytorch for prototyping if you want<a href="https://www.facetimeapp.ltd/facetime-for-pc-windows-download/" rel="noopener nofollow ugc">.</a></p>
Second derivative with respect to input

kavka: <p>Hi, I’m trying to implement the second derivative with respect to input. However, the returned error confused me. The following are the sample code.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(2, 1)

    def forward(self, x):
        x = self.fc1(x)
        return x

dat = torch.tensor([[1.2,3.4], [5.6,1.6]], requires_grad=True)
model = Net()
out = model(dat)
tmp,  = torch.autograd.grad(outputs = out, inputs = dat, grad_outputs = torch.ones_like(out), retain_graph=True, create_graph=True)
second = torch.autograd.grad(outputs = tmp, inputs = dat, grad_outputs =torch.ones_like(tmp), retain_graph=True, create_graph=True)
</code></pre>
<p>The error message are the following</p>
<pre><code class="lang-auto">--------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-66-8ad432f34bca&gt; in &lt;module&gt;
----&gt; 1 torch.autograd.grad(outputs = tmp, inputs = dat, grad_outputs =torch.ones_like(tmp), retain_graph=True)

~/.local/lib/python3.8/site-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
    221         retain_graph = create_graph
    222 
--&gt; 223     return Variable._execution_engine.run_backward(
    224         outputs, grad_outputs_, retain_graph, create_graph,
    225         inputs, allow_unused, accumulate_grad=False)

RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
<p>I tried “Set allow_unused=True if this is the desired behavior.” But it returns a empty tensor</p>
<p>Can anyone help me to fix this?</p>
cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input

FangzheXu311: <p>Hi ! I’m training CNN and error comes up: “cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.”<br>
I first used Tensor with shape (1,1,96,96,36) and the model worked well, but what data I need to input have the shape (1,1,512,512,336). I know that this error may happen because of the lack of GPU memory. But I checked “nvidia-smi” and the memory was enough to abundant.<br>
I wonder whether there are any methods to solve this problem without changing firmware. Thanks in advance.</p>

ptrblck: <p>Could you post an executable code snippet as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

Kushaj: <p>As a temporary solution you can use <code>torch.backends.cudnn.enabled = False</code>.</p>
Model Training becomes very slow if I add one outer for loop

kavka: <p>I was trying to implement several runs for one model to see the confidence interval of my model prediction. However, it becomes very slow.  The following are the sample codes.</p>
<pre><code class="lang-auto">    for i in range(opt.ite + 1):
        model = getattr(models, opt.model)()
        if opt.load_model_path:
            model.load(opt.load_model_path)
        model.to(device)

        model.apply(weight_init)

        # optimizer
        op = Optim(model.parameters(), opt)
        optimizer = op._makeOptimizer()

        previous_err = torch.tensor(10000)
        best_epoch = 0
  
        for epoch in range(opt.max_epoch + 1):
          
            # load the data
            for data in zip(datI_loader, datB_loader):
                # train part                   
                loss.backward()
                optimizer.step()
       
            if epoch % 100 == 0:
                test_err = val(model, grid, sol)
                if test_err &lt; previous_err:
                    previous_err = test_err
                    best_epoch = epoch
        test_meter.add(previous_err.to('cpu'))
        epoch_meter.add(best_epoch)
</code></pre>
<p>For example, if I take opt.ite to be like 2 or 3, it will becomes extremely slow, which might take 5 or 6 hours, however, I remove the outer for loop, which, looks below,</p>
<pre><code class="lang-auto">        model = getattr(models, opt.model)()
        if opt.load_model_path:
            model.load(opt.load_model_path)
        model.to(device)

        model.apply(weight_init)

        # optimizer
        op = Optim(model.parameters(), opt)
        optimizer = op._makeOptimizer()

        previous_err = torch.tensor(10000)
        best_epoch = 0
  
        for epoch in range(opt.max_epoch + 1):
          
            # load the data
            for data in zip(datI_loader, datB_loader):
                # train part                   
                loss.backward()
                optimizer.step()
       
            if epoch % 100 == 0:
                test_err = val(model, grid, sol)
                if test_err &lt; previous_err:
                    previous_err = test_err
                    best_epoch = epoch
</code></pre>
<p>It only take about 20 mins to finish.</p>
<p>Any ideas about this problem?</p>

ptrblck: <p>I don’t know what <code>val</code> is doing exactly, but it seems you are appending its output to some object.<br>
If you are not running this code (inside <code>val</code>) in a <code>with torch.no_grad()</code> context, this could store the entire computation graph and thus increase the memory and yield to slowdowns.</p>
Model seems to compute model "score" on CPU from tensors on the GPU: "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"

krishnab: <p>I am training a Sequence to Sequence LSTM model. The problem is that during the training loop, the loss calculation is complaining with the error: “Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!”.</p>
<p>So it seems like the computing the “score” from the model generates a tensor on the CPU instead of the GPU. I can fix this by just adjusting the line <code>scores = model(data, targets)</code> to <code> scores = model(data, targets).to(device)</code>, but that seems like an unnecessary passing of a tensor back and forth from the GPU to the CPU and then back to the GPU.</p>
<pre><code class="lang-auto">model = Seq2SeqPF(encoder_net, decoder_net).to(device)

load_from_checkpoint = False

if load_from_checkpoint:
    load_checkpoint(torch.load(os.path.join(CHECKPOINT_DIRECTORY, CHECKPOINT_NAME)),model, device)

for epoch in range(EPOCHS):
    print(f"Epoch: {epoch + 1}/{EPOCHS}")
    kbar = pkbar.Kbar(target=batches_per_epoch, width=8)

    if epoch % 5 == 0:
      checkpoint = {'state_dict': model.state_dict(), 
                    'optimizer': optimizer.state_dict()}
      save_checkpoint(checkpoint, 
                      CHECKPOINT_DIRECTORY, 
                      CHECKPOINT_NAME)

    for batch_idx, (data, targets) in enumerate(train_loader):
        
        data = data.to(device=device)
        targets = targets.to(device=device)

        # forward pass and compute error
        scores = model(data, targets) 
        loss = criterion(scores, targets)    # &lt;---GENERATES ERROR ABOUT CPU AND GPU

    # backward pass and apply gradients.
        optimizer.zero_grad()
        loss.backward()

        # gradient descent step
        optimizer.step()

        kbar.update(batch_idx, values=[("loss", loss)])
</code></pre>
<p>The code seems standard. I also did push the model itself to the GPU device, and the <code>encoder_net</code>, and <code>decoder_net</code> are also network layers that are pushed to the GPU before the model code is pushed to the GPU.</p>
<p>Any suggestions on the right way to handle this?</p>

eqy: <p>Could there be a missing <code>criterion = criterion.cuda()</code> somewhere?</p>

krishnab: <p><a class="mention" href="/u/eqy">@eqy</a> Oh that makes sense. I did not know I needed that. I did not realize that I needed to explicitly indicate where the criterion is computed. Hmm. Thanks for finding that for me. I appreciate it.</p>
Dataloader for a very large dataset of 2D slices for segmentation

Geeks_Sid: <p>I have a dataset of size [40000, 512, 512] images (type np.int16) and I want to create a Dataloader for segmentation network.<br>
Although I have 200 Nifti images of 512x512x200 images, Since I am creating a 2D network, it would be efficient to shuffle between subject slices of different subjects as well.<br>
I had the following thoughts:<br>
i) Create multiple tensors and load them through dataloader.<br>
ii) Load image every single time and the slice and send the data(slow and useless and throws away a lot of data)<br>
iii) Create individual slices, store them as .npz and then use it (very very space consuming and very useless)</p>
<p>Can someone help me understanding what to do?</p>
<p>I can share thoughts and code for the dataloader that I made.</p>
<p>Thanks</p>

ptrblck: <p>Could you explain the relationship between the initial dataset of <code>[40000, 512, 512]</code> and the <code>200</code> Nifti images?<br>
Does each sample contain 200 images? In that case, could you deal with <code>[40000, 512, 512, 200]</code> images?</p>

Geeks_Sid: <p>Oh the initial Dataset is just of [200, 200, 512, 512] but I combined all the slices to make it a tensor of [40000, 512, 512] to have 40000 slices</p>

pchunduru: <p>Hi  I am looking for a solution to the same issue. Were you able to get it? If so can you please share it here. Thanks</p>
QNNPACK way slower than FP32 especially the add operation

Alexander_Soare: <p>This has already come up <a href="https://discuss.pytorch.org/t/slow-inference-on-quantized-mobilenetv3/86637">here</a> but I feel it was not addressed properly.</p>
<p>I’m facing this issue with FX tracing on <code>timm</code>'s <code>efficientnet_b3</code>. I get 10x speedup on regular backend but a 4x slow down on qnnpack (<code>torch.backends.quantized.engine = 'qnnpack'</code>). Here are the top 10 time eaters:</p>
<pre><code class="lang-auto">---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                             Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                  model_inference         1.91%       9.592ms       100.00%     503.021ms     503.021ms             1  
                   quantized::add        41.99%     211.224ms        42.02%     211.352ms      11.124ms            19  
                quantized::conv2d        20.23%     101.774ms        20.45%     102.887ms     791.436us           130  
                   quantized::mul        11.50%      57.855ms        11.56%      58.131ms       2.236ms            26  
                    aten::sigmoid        11.39%      57.277ms        11.40%      57.321ms       2.205ms            26  
                 aten::dequantize        11.26%      56.655ms        11.29%      56.795ms     530.796us           107  
                      aten::silu_         0.05%     238.570us         0.72%       3.618ms      46.382us            78  
                       aten::silu         0.67%       3.379ms         0.67%       3.379ms      43.323us            78  
        aten::quantize_per_tensor         0.34%       1.694ms         0.34%       1.694ms      20.918us            81  
                       aten::mean         0.14%     719.232us         0.16%     803.762us      29.769us            27  
---------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
</code></pre>
<p>It’s sad that so much time is being taken by <code>add</code> and <code>mul</code>. The average time for one <code>add</code> is 11 ms! The convs are taking about as long as the FP32 version.</p>
<p>I also tried this with MobileNet V2 and got a 20x slowdown from 41 ms to 806 ms.</p>

HDCharles: <p>Is this on mobile or on server? I know there is some work being done on the speed of add and mul here: <a href="https://github.com/pytorch/pytorch/issues/56187" class="inline-onebox" rel="noopener nofollow ugc">AVX512 and Vec512 · Issue #56187 · pytorch/pytorch · GitHub</a></p>
<p>if its on mobile, it may be necessary to involve the mobile team as well.</p>
Siamese network using Contrastive loss

Sushmita_Upadhyay: <p>I am creating positive and negative pairs for Contrastive loss calculation in Siamese network. For positive pairs I am taking two similar images. However for creating a negative pair I am randomly pairing two different images. I am still working on the model architecture. Every time I train a new model a random combination of dissimilar pair is picked. This causes different accuracy and loss function. Am I right to pick dissimilar pairs randomly or should I create fixed similar and dissimilar pairs for training and testing?<br>
Also what should be the ratio for similar and dissimilar pairs for every image?</p>

nicozhou: <p>It seems that your pairing is somehow good, could you share about your network architecture? Maybe the feature extractor?</p>
Advice on a model for a physical system

vicio: <p>Hello everyone,<br>
I’m kinda new to the Machine Learning world (I only did some basic models during my master) and I would like to receive some suggestions about some good approaches that I could take for my supervised learning task. My data is structured as following: the input is a 3-channels image and my output is a one channel image. Each channel in the input represents a snapshot of a certain quantity in a physical system. Each pixel in the image can be thought as a site in a lattice. The output contains the probability that a given pixel is active. The output image comes from a physical evolution of the input state. What are some good architectures for this kind of problem? Do you have any paper\article treating this kind of problem? Thanks in advance.</p>

KFrank: <p>Hi Vincenzo!</p>
<aside class="quote no-group quote-modified" data-username="vicio" data-post="1" data-topic="121731" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vicio/48/38473_2.png" class="avatar"> vicio:</div>
<blockquote>
<p>the input is a 3-channels image and my output is a one channel image.<br>
…<br>
The output contains the probability that a given pixel is active.</p>
</blockquote>
</aside>
<p>If I understand your description, you wish to perform binary (that is to say<br>
two-class) <em>semantic segmentation.</em></p>
<p><a href="https://en.wikipedia.org/wiki/U-Net" rel="noopener nofollow ugc">U-Net</a> is a well-established semantic-segmentation architecture.</p>
<p>Pytorch’s <code>torchvision</code> includes some pre-built semantic-segmentation<br>
models (but not U-Net), such as <a href="https://pytorch.org/vision/0.8/models.html#torchvision.models.segmentation.fcn_resnet50" rel="noopener nofollow ugc">fcn_resnet50</a>.</p>
<p>Best.</p>
<p>K. Frank</p>

vicio: <p>Thanks for the tip, this might be the way to go!</p>
How to use torch.autograd.backward for a matrix?

axolotl: <p>Hello, I want to use .backward() to calculate grad w.r.t each input of my matrix X_ij, however , it is huge and iterating over each position specifying grad argument in .autograd(grad=) takes way to much time. I have not yet found a better way rather then iterate through columns i and rows j but it takes nested for-loop like so:</p>
<pre><code class="lang-auto">output.shape # [60,120]
elemet_wise_grads = []
for i in range(60):
    for j in range(120):
        external_grad = torch.zeros_like(output)
        external_grad[i,j] = 1
        output.backward(gradient=external_grad, retain_graph=True)
        gradient = embeddings.grad
        elemet_wise_grads.append(gradient)
</code></pre>
<p>is there an efficient way to do that?<br>
thank you</p>
How to optimize the term with "gradient wrt input"?

phantom90: <p>Hi there,</p>
<p>I encounter with this problem, how to compute:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61af5e392e5364f44d611ac1914a88f1b7b9f89e.png" alt="image" data-base62-sha1="dWa2pNtEnOxpaIOlVHIU2AaPqxM" width="34" height="42"></p>
<p>in pytorch? Here, X is the model input, Y is the prediction, \theta is the parameter.</p>
<p>Thanks in advance~</p>
Training an ASR Model with NLLLoss or CrossEntropyLoss

knoriy: <p>I’m new to ASR models. Could someone kindly help me understand how to train a simple asr model?</p>
<p><strong>Background:</strong><br>
I have created an attention-based ASR model with an encoder that takes in mel-spec as inputs, with shape <code>(B, C, T)</code> or <code>(16, 80, 873)</code>, the decoder is passed a sequence of IDs corresponding to the symbols in the text of shape <code>(B, T)</code> or <code>(16, 168)</code> this is embedded with <code>nn.Embedding(  n_symbols, 256) -&gt; (16, 168, 256)</code>.</p>
<p>After some RNN layers the model outputs a tensor of shape <code>(16, 168, 256)</code>.</p>
<hr>
<p>My lack of knowledge comes in the Loss calculation for ASR models. I have tries both NLLLoss and CrossEntropyLoss with little success and many different errors:</p>
<pre><code class="lang-auto">CrossEntropyLoss &amp; NLLLoss -&gt; RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15
</code></pre>
<p>Following another thread <a href="https://discuss.pytorch.org/t/runtimeerror-multi-target-not-supported-newbie/10216/2">HERE</a></p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

Unity05: <p>Hi,</p>
<p>for ASR models, I would usually go for the connectionist temporal classification loss (<a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html" rel="noopener nofollow ugc">CTC loss</a>).</p>
<p>Regards,<br>
Unity05</p>

knoriy: <p>Thank you, I’ll look into it.</p>
<p>Could you elaborate on why you use CTCLoss?</p>

Unity05: <p>It is really usefull you get for instance duplicates from the horizontal time stamps and so on because CTC uses blank characters (i.e. <code>heelll[blank]llllloooo</code> is the same as <code>hellll[blank]lo</code> would get encoded to <code>hello</code> but <code>hello</code> would not.).<br>
The loss (CTCLoss) is computed as illustrated here (the blank labels are not removed for loss calculation): <img src="https://miro.medium.com/max/1200/1*1_5KnLvaTkGUFoyat2jHcQ.png" alt="" width="600" height="441"><br>
(<a href="https://miro.medium.com/max/1200/1*1_5KnLvaTkGUFoyat2jHcQ.png" rel="noopener nofollow ugc">https://miro.medium.com/max/1200/1*1_5KnLvaTkGUFoyat2jHcQ.png</a>)<br>
During inference, you get the prediction by best path decoding.</p>

knoriy: <p>Thank you very much <a class="mention" href="/u/unity05">@Unity05</a>!</p>
<p>That’s super helpful.</p>

Unity05: <p>Thanks, and yeah, CTC is quite nice. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=9" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>
Metric for multi-label classification

Chamroukhi: <p>Dear all,<br>
any help for which metric in Pytorch used for multi-labeling ?</p>

KFrank: <p>Hi Chamroukhi!</p>
<aside class="quote no-group" data-username="Chamroukhi" data-post="1" data-topic="121721" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/d6d6ee/48.png" class="avatar"> Chamroukhi:</div>
<blockquote>
<p>any help for which metric in Pytorch used for multi-labeling ?</p>
</blockquote>
</aside>
<p>The appropriate loss function for a multi-label, multi-class classification<br>
task is <a href="https://pytorch.org/docs/1.8.1/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a>.</p>
<p>Best.</p>
<p>K. Frank</p>

Chamroukhi: <p>yeah but for accuracy what metric should used (topKcategorical, accuracy ,…)?</p>
Where do you guys Store your Data (Checkpoints etc) while training on Kaggle, Colab

JoeExotic: <p>Hello Guys,</p>
<p>where, how do you store, download/upload your files to while training your Models in the Cloud (Kaggle, Colab)?<br>
Im currently using Google Cloud Storage, but the 300 free USD Bonus is simply used up immediately just for Storing the Checkpoints. I bought 100GB Gdrive Storage, but thats just not enough if you use Colab Pro and you can train multiple Models at once.</p>
<p>Ty in advcanded for your suggestions</p>
What is different between FusionGroup and CudaFusionGroup?

alanzhai219: <p>Different API will result in different group such as FusionGroup and CudaFusionGroup. Why? And what is different?</p>

tom: <p>These are actually different fuser generations.<br>
The “classic” 1st-gen fuser only did pointwise ops and created <code>FusionGroup</code> nodes. The newer fuser developed by a team at NVIDIA creates <code>CUDAFusionGroup</code>. To round off the trio, there is <code>TensorExprGroup</code> nodes created by the TensorExpr/NNC fuser developed by a team at FB. The latter two also support some reductions.<br>
A while ago, <a href="https://lernapparat.de/jit-optimization-intro/" rel="noopener nofollow ugc">I wrote a blog on the various fusers</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

alanzhai219: <p>Thanks for your reply. I read your blog first. It seems that the mechanism is not easy to figure out.</p>

tom: <p>The JIT optimization steps probably are among the most sophisticated bits in PyTorch (along with the dispatcher,…). For a deep dive on one of the fusers, I can also enthusiastically recommend <a href="https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s31952/" rel="noopener nofollow ugc">Christian Sarofeen’s talk</a> (I think you need to register to see it).</p>
<p>Best regards</p>
<p>Thomas</p>

alanzhai219: <p>Excellent! I will look into it and will email you when encounter any question. Much thanks.</p>
About nn.parallel.DistributedDataParallel

shinra: <p>I trained my module with 4 gpus ,like:</p>
<p><code>mymodule = nn.parallel.DistributedDataParallel(mymodule, device_ids=[local_rank])</code><br>
But I saved my module  by</p>
<p><code>torch.save(mymodule.state_dict() , '%s/modelG_%d.pth' % (opt.outf, epoch))</code><br>
When I load it  by:<br>
<code>    mymodule.load_state_dict(torch.load(f))</code><br>
I got:<br>
<code>RuntimeError: storage has wrong size: expected -4763383137013773690 got 128</code></p>
<p>What is wrong? And if there is any way to deal with it without re-training?<br>
Thanks.</p>

liyz15: <p>Have you ensured that only one process is writing to checkpoint? Multiple processes writing to the same checkpoint will corrupt it.</p>

shinra: <p>I did not consider this problem <img src="https://discuss.pytorch.org/images/emoji/apple/cold_sweat.png?v=9" title=":cold_sweat:" class="emoji" alt=":cold_sweat:">.<br>
Looks like that I have to modify my code and re-train my module.<br>
Thanks a lot.</p>
Point in polygon

Ilia_Karmanov: <p>I am trying to predict points that lie on a 2D Cartesian map constrained by a floor-plan which contains rooms of arbitrary shapes. I am currently calculating l2-distance to the bounding-box around a room as a loss (since I only need to consider the four sides), however I would prefer to more precisely penalise the model especially for non-rectangular rooms.</p>
<p>Is anyone aware of a point-in-polygon loss I could use to train my model?</p>
<p>Some methods I have considered (but have not worked well are):</p>
<ol>
<li>Generating many points and training an MLP to know whether the point belongs to the room or not, then using this as a loss-term; however this is very slow</li>
<li>Precomputing the closest-distance on the exterior of the room and using L2-distance to that when training, and masking out the loss if inside, however then there is no gradient information when inside and the point can jump outside</li>
</ol>
Is there any alternative to numpy.add.at in PyTorch?

seva100: <p>There is a function in NumPy called <code>np.add.at</code> which allows to add values to elements accessed by multi-index with repeated elements, such that for each repeated element all its corresponding values will be considered in the summation. E.g. see an example:</p>
<pre><code class="lang-auto">A = np.zeros(5)
np.add.at(A, [1, 1, 2], 1)
A
</code></pre>
<p>produces:</p>
<pre><code class="lang-auto">array([0., 2., 1., 0., 0.])
</code></pre>
<p>Right now I am in a very large need for the same thing in PyTorch (I can’t avoid repeated indices in my task), as the plain summation has different behavior:</p>
<pre><code class="lang-auto">A = torch.zeros(5)
A[[1, 1, 2]] += 1
A
</code></pre>
<p>produces:</p>
<pre><code class="lang-auto">tensor([ 0.,  1.,  1.,  0.,  0.])
</code></pre>
<p>Is there any way to simulate behavior of <code>np.add.at</code> by PyTorch operations?<br>
Thank you!</p>

seva100: <p>Looks like I have already found a solution myself:</p>
<pre><code class="lang-auto">A = torch.zeros(5)
A.index_add_(0, torch.LongTensor([1, 1, 2]), torch.FloatTensor([1, 1, 1]))
A
</code></pre>
<p>It seems that this can also be used for multi-dimensional tensors, if they are flattened beforehand. It would be very handy to have such a function for multi-dimensional tensors also (if there is no one already).</p>

Yasser_Khalil: <p>Hello,</p>
<p>Is there any scalable solution to this problem? I am using multi-dimensional tensors I would like to sum elements using indices in another tensor, and some indices appear more than once. In NumPy <code>np.add.at</code> does the job.<br>
Thanks</p>

NagabhushanSN95: <p>Check this <a href="https://stackoverflow.com/a/65584479/3337089" rel="noopener nofollow ugc">StackOverflow answer</a></p>
Feeding A Grayscale Image To Pretrained Models

Adwaitt: <p>Hello guys, I have been looking around in the discussion forums on how to feed a grayscale image to imagenet models. I wanted to know that, is it possible to simply use a convolution with padding = 1 so as to preserve its dimension and then feed it to the model. Will there be any side effects of it?</p>

ptrblck: <p>It is certainly possible to use a conv layer in order to transform the grayscale input images to images containing 3 channels. To keep the spatial dimension you could use e.g. a <code>1x1</code> kernel or any other setup with the appropriate padding (since you’ve mentioned a padding value of 1 I assume you would like to use a <code>3x3</code> kernel with <code>stride=1</code> and <code>dilation=1</code>).<br>
“Side effects” would be that you are adding more trainable parameters at the beginning of the model. Assuming you would like to train this layer, Autograd would need to backpropagate through the entire model, which could be avoided in case you would freeze the pretrained model and only train the classifier.</p>

Adwaitt: <p>Thank you, I don’t think I want to train that layer so I should freeze it.</p>
Unexpected behavior of synchronization while running DistributedDataParallel

yurii: <p>Hi guys!<br>
I have one PC with two GPUs on board. Below is schematic code for distributed training:</p>
<pre><code class="lang-auto">dist.init_process_group(backend="nccl", .....)
.....
model = DistributedDataParallel(model, device_ids=[rank])  # rank is 0 or 1
.....
for i in range(len(batches))
    .....    
    outputs = model(inputs, .....)
    loss = criterion(outputs, ....)
    loss.backward()
    optimizer.step()
    print(rank, "-", i)
    .....
</code></pre>
<p>I expected to see following print outputs (to save space, I quote them horizontally):<br>
<code>0-0 1-0, 0-1 1-1, 0-2 1-2, 0-3 1-3,  ...,  0-n 1-n</code><br>
However I got something unexpected:<br>
<code>0-0 1-0,  0-1 1-1, ...,   0-60 1-62, ...,  0-3125 1-3145,  ...</code><br>
Sometimes gpu_1 is up by 20 iterations ahead of gpu_0!<br>
I see two possibilities here:</p>
<ol>
<li>my code has error somewhere</li>
<li>it is normal behavior for synchronization</li>
</ol>
<p>If 2. is the case could you please explain why is it so?<br>
From what I’ve understood in documentation, synchronization between processes happens at each iteration in loss.backward() call. Suppose I have some model parameter w. At each iteration this parameter must be the same in gpu_0 and gpu_1 model replicas.<br>
For example:<br>
at first iteration w is:<br>
<code>w + (upd0_0 + upd1_0)/2</code><br>
at second iteration:<br>
<code>w + (upd0_0 + upd1_0)/2 + (upd0_1 + upd1_1)/2</code><br>
at i-th iteration:<br>
<code>w + (upd0_0 + upd1_0)/2 + (upd0_1 + upd1_1)/2 + ... + (upd0_i + upd1_i)/2</code><br>
where <code>upd0_i</code> and <code>upd1_i</code>- parameter updates calculated during backprop at i-th iteration in gpu_0 and gpu_1 correspondingly.</p>
<p>Thanks!</p>

wayi: <aside class="quote no-group" data-username="yurii" data-post="1" data-topic="121396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/cdc98d/48.png" class="avatar"> yurii:</div>
<blockquote>
<p><code>print(rank, "-", i)</code></p>
</blockquote>
</aside>
<p>Can you change the print statement as print(rank, “-”, i, <strong>flush=True</strong>)? I guess some output is buffered, so it gives you the impression that one ranks is behind the other.</p>

yurii: <p>Hi, Yi Wang<br>
Thanks for suggestion, I tried flush=True, but situation remains the same. Looks like I have bug in my code…</p>

wayi: <p>Realize that the root cause is that, your print statement only requires the input from host, not from device. If you print results from CUDA tensors, you should see synced outputs.</p>
<p>This is because although DDP syncs across devices at each step, the allreduce communication from the host perspective is just a non-blocking enqueue operation. There isn’t anything wrong with your DDP code. Just your print statement gives you an illusion caused by the enqueue operation from the host.</p>

yurii: <p>I see, got it, thanks.</p>
CNN Model Error

Sahil_Chhillar: <p>class Net(Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()</p>
<pre><code>    self.cnn = Sequential(
        Conv2d(in_channels=1, out_channels=4, kernel_size=3, stride=1),
        ReLU(inplace=True),
        MaxPool2d(kernel_size=3, stride=1),
        Conv2d(in_channels=4, out_channels=4, kernel_size=3, stride=1),
        ReLU(inplace=True),
        MaxPool2d(kernel_size=3, stride=1),
        Linear(in_features=1000, out_features=500),
        ReLU(inplace=True),
        Linear(in_features=500, out_features=250),
        ReLU(inplace=True),
        Linear(in_features=250, out_features=4)
    )

def forward(self, x):
    x = self.cnn(x)
    x = x.view(x.size(0), -1)
    return x
</code></pre>
<p>Please tell me what is wrong, I don’t get it.<br>
Error is: RuntimeError: Given groups=1, weight of size [4, 1, 3, 3], expected input[2190, 64, 64, 3] to have 1 channels, but got 64 channels instead</p>

ptrblck: <p>It seems you are passing the input tensor in a channels-last memory format, while PyTorch expects channels-first by default, so you would have to <code>permute</code> the dimensions into <code>[batch_size, channels, height, width]</code>. In case you are changing the memory format via <code>tensor.to(memory_format=torch.channels_last)</code> note that you should not change the dimensions manually, but let PyTorch do it internally.</p>
<p>Once you’ve permuted the input tensor you would run into the next error of a wrong number of <code>in_channels</code> in the first conv layer.<br>
While it seems that your input tensor uses 3 channels, the first conv layer uses 1 input channel, so you would have to change this next.</p>

Sahil_Chhillar: <p>Thanks a lot. I will try that.</p>