akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Representing heterogeneous data in memory

username: michalsustr
cooked: <p>Suppose I have a training set of <em>set of sequences of variable size</em>, i.e.</p>
<pre><code class="lang-auto">D = { (x^i, y^i) | i \in (1...N)}
where
x^i = { x^{i,j}  | j \in (1...N_i)}  and  y^i \in \R^N_i
where
x^{i,j} \in \R^N_{i,j}
</code></pre>
<p>with some sizes <code>N_i, N_{i,j}</code>.</p>
<p>What is the best way to save them in memory? Does <code>torch::Tensor</code> allow to save block of heterogeneous data via pointers to (dynamically allocated) other <code>torch::Tensor</code>s, which would hold the actual data? (And do so possibly recursively?) Most importantly, does it allow all the nice PyTorch things like tracking gradients?</p>
<p>While I could manage this by storing tensors into <code>std::vector&lt;torch::Tensor&gt;</code> (and do so recursively), I lose the ability to use the torch interface when interacting with the data.</p>
<p>What is the recommended way to do this? I imagine similar problems arise often for graph neural networks.</p>

username: tom
cooked: <p>What you describe is called <a href="https://github.com/pytorch/pytorch/issues/22169" rel="noopener nofollow ugc">NestedTensors</a> in PyTorch parlance and it isn’t there yet.</p>
<p>Until that materializes, the options are mostly padding (to regular shape) or <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.PackedSequence.html#torch.nn.utils.rnn.PackedSequence" rel="noopener nofollow ugc">packing</a>. Either might be good to combine with “stratifying” your batches to have similar sizes, e.g. <a href="https://torchtext.readthedocs.io/en/latest/data.html#bucketiterator" rel="noopener nofollow ugc">torchtext does this</a>, I have done that before for CT scans.<br>
Whether packing or batching is more appropriate and how the stratification is best done depends on your data and your task.</p>
<p>Best regards</p>
<p>Thomas</p>

username: michalsustr
cooked: <p>Great, thank you for the pointers!</p>
Problem encountered when tested with batchnorm layers

username: MrChenFeng
cooked: <p>Hi guys, when tested with batchnorm layers now, i was confused by something:</p>
<pre><code class="lang-auto">input = torch.rand(10,5)
model = nn.BatchNorm1d(5)
out = model(input)
print(out.mean(0), out.var(0))
</code></pre>
<p>Here I would expect the mean be zero while var be 1 if I didn’t misunderstand the original batchnorm operation cause we havent update the gamma and beta value of the batchnorm layer. However, i always get a strange value such as 1.1102 for var and ~0 for mean. Any suggestions?</p>
<p>Besides, I will get 1 and 0 for weight and bias(gamma and beta) when:</p>
<pre><code class="lang-auto">print(model.weight, model.bias)
</code></pre>

username: MrChenFeng
cooked: <p>anyone has similar questions?</p>

username: ptrblck
cooked: <p>The number of elements might be too small and you might get values closer to the zero mean and unit variance by using a larger input, such as <code>[100, 5]</code>.</p>

username: MrChenFeng
cooked: <p>Thanks, I tried and it does have some changes. I just wonder why pytorch cannot normalize it correctly. A bit weird.</p>

username: ptrblck
cooked: <p>I don’t think PyTorch cannot “normalize it correctly”, but batchnorm layers don’t use the unbiased variance calculation, which is probably why you might think it’s incorrect. If you want to use the biased one, you could create a custom layer by e.g. following <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">this implementation</a>.</p>

username: MrChenFeng
cooked: <p>Thanks so much. You are right, pytorch used 1/n not 1/(n-1). I can now match the output with the manual calculated one.</p>
Performance degrade in CPU vs CUDA

username: soroosh_99
cooked: <p>Is it normal that when I use a pretrained model on GPU and transform it to CPU the performance decreases?<br>
And is there a why to equalize this difference?</p>
<p>As can be seen, the CPU can not detect the <strong>America</strong> box<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62756841bff74105691b5d238075345a3dcab9f1.jpeg" alt="a" data-base62-sha1="e30kPwoI3D5cgv7TRhmbRBNGVd7" width="342" height="335"></p>
<p>While the GPU model can:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f5af18591d90b413de98023dbaf557e20c416a9.jpeg" alt="b" data-base62-sha1="dBycIuH3ZSqxODhn0kMzR5Vf9yh" width="342" height="335"></p>
Resize pictures

username: Samuel_Bachorik
cooked: <p>Hello i want to ask if you can show me best way to resize pictures in numpy array and also torch tensor.<br>
I have picture of shape (480, 700, 3) and I need (350, 480, 3)</p>
<p>What is best way to do that ?</p>

username: ptrblck
cooked: <p>For tensors you could use <code>F.interpolate</code> or the <code>torchvision.transforms</code>, which are now also working on tensors.</p>
How to apply multiple data loaders to one model?

username: ysleer
cooked: <p>I am trying to train one model using multiple data loaders.</p>
<p>An example I thought of is below.</p>
<pre><code class="lang-auto">for epoch in epochs:
train_loader = zip(train_loader_list[0], train_loader_list[1], train_loader_list[2])
    for batch_idx, sample in enumerate(train_loader):
        """
    for batch_idx in range(len(train_loader)):

        sample = batch_queue.get()
        batch_queue.task_done()
        """
        data_time.update(time.time() - end)

        for i in range(0, len(sample)):
            input = sample[i]['image'].cuda()
            target = sample[i]['label'].cuda()
            target_exist = sample[i]['exist'].cuda()            

            output, output_exist = model(input)  # output_mid           
            loss_seg = criterion(torch.nn.functional.log_softmax(output, dim=1), target)
            loss_exist = criterion_exist(output_exist, target_exist)
            loss = loss_seg + loss_exist * 0.1            

            loss_avg += loss
        loss_avg = loss_avg / len(sample)

        losses.update(loss_avg.data.item(), input.size(0))

        optimizer.zero_grad()
        loss_avg.backward()
        optimizer.step()
</code></pre>
<p>I want to average the loss values of 3 data loaders and backward them.</p>
<p>I use DistributedDataParallel</p>
<p>“RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32]] is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).”</p>
<p>An above error occurs when proceeding in the following way.</p>
<p>I don’t know which part is the problem.</p>
<p>Please help me.</p>
<p>Thank you</p>
Pytorch automatically used the tensor core?

username: sangheonlee
cooked: <p>Hi, I was running the imagenet in PyTorch example.</p>
<p>Do I need the add code or give the option for using tensor core? or Pytorch use the tensor core as default mode?</p>
<p>Thanks</p>

username: Dwight_Foster
cooked: <p>No you need to use the pytorch automatic mixed precision library <a href="https://pytorch.org/docs/stable/amp.html" rel="noopener nofollow ugc">here</a>.</p>

username: ptrblck
cooked: <p>Besides what <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> mentioned, on Ampere GPUs the tensor cores will be used through <a href="https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/">TF32</a>.</p>
How is torchserve scalable? (using multiple replicas)

username: gtimothee
cooked: <p>Hi,</p>
<p>I am trying to compare torch serve and ray serve performances. I started with ray serve, and with ray serve I can decide how many replicas of a given model to launch at the same time. Now for torch serve I successfully served a model but I cannot find how to deploy multiple replicas of the same model on a single machine (say I have 2 gpus and my model uses 1 gpu, why not deploying 2 times the model so that it can handle more requests ?).</p>
<p>Edit: using it, it seems that is uses replicas, but I just don’t control it.</p>
<p>Thank you in advance for your help.</p>
Weird build issue on linux when using libtorch

username: wolfviking0
cooked: <p>I have a weird error when building an example using libtorch on linux Ubuntu 18.04:</p>
<p>In file included from /home/external/torch_cuda/include/c10/core/Scalar.h:9:0,<br>
from /home/external/torch_cuda/include/ATen/core/TensorBody.h:7,<br>
from /home/external/torch_cuda/include/ATen/Tensor.h:11,<br>
from /home/external/torch_cuda/include/ATen/Context.h:4,<br>
from /home/external/torch_cuda/include/ATen/ATen.h:5,<br>
from /home/external/torch_cuda/include/torch/csrc/api/include/torch/types.h:3,<br>
from /home/external/torch_cuda/include/torch/script.h:3,<br>
from /home/libs/test/src/test.cpp:21:<br>
/home/external/torch_cuda/include/c10/core/ScalarType.h:62:3: error: expected identifier before ‘int’<br>
AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS(DEFINE_ENUM)<br>
^<br>
/home/external/torch_cuda/include/c10/core/ScalarType.h:62:3: error: expected ‘}’ before ‘int’<br>
/home/external/torch_cuda/include/c10/core/ScalarType.h:62:3: error: expected unqualified-id before ‘,’ token<br>
AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS(DEFINE_ENUM)<br>
^<br>
/home/external/torch_cuda/include/c10/core/ScalarType.h:62:3: error: two or more data types in declaration of ‘QInt8’<br>
AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS(DEFINE_ENUM)<br>
^<br>
/home/external/torch_cuda/include/c10/core/ScalarType.h:62:3: error: two or more data types in declaration of ‘QUInt8’<br>
AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS(DEFINE_ENUM)<br>
^<br>
/home/torch_cuda/include/c10/core/ScalarType.h:62:3: error: two or more data types in declaration of ‘QInt32’<br>
AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS(DEFINE_ENUM)<br>
^<br>
/home/torch_cuda/include/c10/core/ScalarType.h:62:3: error: two or more data types in declaration of ‘BFloat16’<br>
AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS(DEFINE_ENUM)</p>
<p>It seem I have an issue with this Macro AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS but not sure why.<br>
Any suggestion or idea ?</p>
<p>Thanks</p>

username: Carlos_Alberto_Toro
cooked: <p>Good evening, I have a similar problem, any ideas to solve it? Thank you.</p>
Simple JIT scripting fails with "Tried to access nonexistent attribute or method..."

username: alannnna
cooked: <p>I am new to JIT scripting and don’t know what this error means, and have not found this zero_grad error online nor any related errors whose solutions seem to make sense here. What am I doing wrong here?</p>
<p>The error is this (see gist for full error):</p>
<pre><code class="lang-auto">Tried to access nonexistent attribute or method 'zero_grad' of type 'Tensor (inferred)'.
</code></pre>
<p>The full repro is here:</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/alannnna/008106cdee4ced848780ed1d829f57d2" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/alannnna/008106cdee4ced848780ed1d829f57d2" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/alannnna/008106cdee4ced848780ed1d829f57d2</a></h4>
<h5>bench.py</h5>
<pre><code class="Python">import time

import torch

from mnist import Net, DEFAULTS, train_example


def make_net():
    torch.manual_seed(1234)
    model = Net()</code></pre>
This file has been truncated. <a href="https://gist.github.com/alannnna/008106cdee4ced848780ed1d829f57d2" target="_blank" rel="noopener nofollow ugc">show original</a>
<h5>error.txt</h5>
<pre><code class="Text">$ python bench.py 

7.002076864242554
Traceback (most recent call last):
  File "bench.py", line 50, in &lt;module&gt;
    try_jit()
  File "bench.py", line 31, in try_jit
    scripted_train_example = torch.jit.script(train_example)
  File "/Users/me/blah/blah/venv/lib/python3.7/site-packages/torch/jit/_script.py", line 940, in script
    qualified_name, ast, _rcb, get_default_args(obj)</code></pre>
This file has been truncated. <a href="https://gist.github.com/alannnna/008106cdee4ced848780ed1d829f57d2" target="_blank" rel="noopener nofollow ugc">show original</a>
<h5>mnist.py</h5>
<pre><code class="Python">from __future__ import print_function
import argparse
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms

</code></pre>
This file has been truncated. <a href="https://gist.github.com/alannnna/008106cdee4ced848780ed1d829f57d2" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>I think the error is raised, because scripting assumes tensors as the input, if no annotations are used as described <a href="https://discuss.pytorch.org/t/error-with-my-custom-concat-class-with-torchscript/112440/2">here</a>.<br>
I don’t know, if it’s possible to script the complete optimization, but you should be able to script the model at least.</p>
How to convert tif format mast to png format mast during image segmentation

username: Deep_learning_labora
cooked: <p>How to convert tif format mast</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a98ec05fa5790c288550a856c5e705f5958583b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2a98ec05fa5790c288550a856c5e705f5958583b" title="Screenshot_20210303_142808"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a98ec05fa5790c288550a856c5e705f5958583b_2_690x271.jpeg" alt="Screenshot_20210303_142808" data-base62-sha1="64PGCYUaBjB56WMePzqPJBcvZqj" width="690" height="271" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a98ec05fa5790c288550a856c5e705f5958583b_2_690x271.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a98ec05fa5790c288550a856c5e705f5958583b_2_1035x406.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a98ec05fa5790c288550a856c5e705f5958583b.jpeg 2x" data-dominant-color="747474"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20210303_142808</span><span class="informations">1080×425 63.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>to png format mast during image segmentation</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9d6c4ab73b66980211c6fbe2d46969f3e3c3a46.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e9d6c4ab73b66980211c6fbe2d46969f3e3c3a46" title="Screenshot_20210302_174104"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9d6c4ab73b66980211c6fbe2d46969f3e3c3a46_2_690x319.jpeg" alt="Screenshot_20210302_174104" data-base62-sha1="xmDo92DzLBZ5ipWIElKLjyfYo7A" width="690" height="319" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9d6c4ab73b66980211c6fbe2d46969f3e3c3a46_2_690x319.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9d6c4ab73b66980211c6fbe2d46969f3e3c3a46.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9d6c4ab73b66980211c6fbe2d46969f3e3c3a46.jpeg 2x" data-dominant-color="21258B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20210302_174104</span><span class="informations">960×444 33.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Deep_learning_labora
cooked: <p>tif format mast can not user in<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#</a></p>
Handling for loops and if conditions to get better performance

username: FA_mn
cooked: <p>I need to calculate the gradient of each element of loss w.r.t parameters and  use it to compute gradient vector and hessian matrix.<br>
my code is below:</p>
<pre><code class="lang-auto">        optimiser.zero_grad()
        Error_Train_1 = PDE_loss(train_x, train_y, loss_func)
        
        
        size_hess = q*p
        MyHess = torch.zeros([size_hess,size_hess])
        MyGrad = torch.zeros([size_hess,1])
        for j in range(Error_Train_1.shape[0]): 
            for i in range(q+1):
                jac = []
                optimiser.zero_grad()
                (Error_Train_1[j,i]).backward(retain_graph=True)
                        
                for params in self.parameters():
                    # in order to prevent stopping algorithm
                    if params.grad is None:
                       Pg = torch.zeros_like(params).reshape(-1)
                    else:
                       Pg = params.grad.reshape(-1)
                            
                    if len(jac) == 0: 
                       jac = Pg
                    else:
                       jac = torch.cat((jac, Pg)) 

                jac = torch.reshape(jac, (1, -1))     
                Hess = torch.matmul(torch.transpose(jac, 0, 1), jac)  
                Par_grad = torch.transpose(jac, 0, 1)*(Error_Train_1[j,i])  
                MyGrad +=  Par_grad
                MyHess += Hess
</code></pre>
<p>There are a lot of for loops and if conditions, which is not good for performance, and it takes a lot of time.<br>
Could you please help me and let me know if there is any better way to handle if conditions or for loops?</p>
<p>Thanks.</p>
How to go from 1d index that results from flatten operation to 2d index?

username: NadiaMe
cooked: <p>Say we have a matrix M that is n by n and after we flatten it we get n^2 sized array.<br>
Then we find max at index k.<br>
How to convert k to (x, y) coordinate for pre-flatten array?<br>
Is there torch function for this?</p>

username: Eta_C
cooked: <pre><code class="lang-python">n = 5
M = torch.rand(n, n)
flatten_M = M.view(-1)
k = torch.argmax(flatten_M)
print(k, flatten_M[k])
i, j = divmod(k.item(), n)
print(M[i, j])
</code></pre>
How to set a>b constraint for my a and b predictions

username: seyeeet
cooked: <p>I have two predications <code>a</code> and <code>b</code>, how can I set constraints or define a loss that minimizes for condition <code>a&gt;b</code>, where <code>a</code> and <code>b</code> are regression outputs of my network.<br>
In other words, I want to teach the network that <code>a</code> should always be greater than <code>b</code>.</p>

username: Naruto-Sasuke
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.MarginRankingLoss.html?highlight=ranking%20loss#torch.nn.MarginRankingLoss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.MarginRankingLoss.html?highlight=ranking%20loss#torch.nn.MarginRankingLoss</a></p>

username: seyeeet
cooked: <p>Thanks for your reply, do you know how can I find the source code for it? I reached to this in pytorch:</p>
<pre><code class="lang-auto">def margin_ranking_loss(input1, input2, target, margin=0, size_average=None,
                        reduce=None, reduction='mean'):
    # type: (Tensor, Tensor, Tensor, float, Optional[bool], Optional[bool], str) -&gt; Tensor
    r"""margin_ranking_loss(input1, input2, target, margin=0, size_average=None, reduce=None, reduction='mean') -&gt; Tensor

    See :class:`~torch.nn.MarginRankingLoss` for details.
    """  # noqa
    if not torch.jit.is_scripting():
        tens_ops = (input1, input2, target)
        if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
            return handle_torch_function(
                margin_ranking_loss, tens_ops, input1, input2, target, margin=margin,
                size_average=size_average, reduce=reduce, reduction=reduction)
    if size_average is not None or reduce is not None:
        reduction_enum = _Reduction.legacy_get_enum(size_average, reduce)
    else:
        reduction_enum = _Reduction.get_enum(reduction)
    if input1.dim() == 0 or input2.dim() == 0 or target.dim() == 0:
        raise RuntimeError(("margin_ranking_loss does not support scalars, got sizes: "
                            "input1: {}, input2: {}, target: {} ".format(input1.size(), input2.size(), target.size())))
    return torch.margin_ranking_loss(input1, input2, target, margin, reduction_enum)
</code></pre>
<p>but iI cannot find the source code to see how they defined it, any suggestion/</p>
How to debug a detected anomoly

username: William_Ravenscroft
cooked: <p>Hi, I have this error - I think it was raised because I have torch.autograd.set_anomoaly_detect(True).</p>
<pre><code class="lang-auto">[W python_anomaly_mode.cpp:104] Warning: Error detected in CudnnRnnBackward. Traceback of forward call that caused the error:
  File "/share/miniconda3/envs/cs21/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/share/miniconda3/envs/cs21/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/share/miniconda3/envs/cs21/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/share/usr/will/sw/Conv-TasNet/src/multi_conv_tasnet.py", line 58, in forward
    spatial_mixture = self.spatial_encoder(mixture)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/share/sw/Conv-TasNet/src/multi_conv_tasnet.py", line 132, in forward
    mixture_w, (self.h, self.c) = self.lstm(mixtures_l)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 581, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
 (function _print_stack)
</code></pre>
<p>I assumed it maybe a vanishing/exploding gradient issue but I am already using gradient clipping… Does anyone know how I could debug it?</p>
<p>Thanks in advance</p>
Is topk broken?

username: sparseinference
cooked: <p>I just notice that when specifying <code>sorted=False</code> , the values are still sorted.<br>
Have I misunderstood the documentation?</p>
<pre><code class="lang-auto">x = torch.rand(5,5)
&gt;&gt;&gt; x
tensor([[0.6847, 0.0039, 0.1289, 0.9117, 0.6705],
        [0.4684, 0.0572, 0.3332, 0.7579, 0.8104],
        [0.9675, 0.7871, 0.9898, 0.3934, 0.4836],
        [0.0480, 0.1028, 0.0757, 0.7755, 0.7489],
        [0.1737, 0.0918, 0.9962, 0.4362, 0.7884]])

&gt;&gt;&gt; x.topk(2, sorted=False).values
tensor([[0.9117, 0.6847],
        [0.8104, 0.7579],
        [0.9898, 0.9675],
        [0.7755, 0.7489],
        [0.9962, 0.7884]])

</code></pre>

username: ptrblck
cooked: <p><code>sorted=True</code> will make sure that the results are indeed sorted. Setting it to <code>False</code> does not “unsort” them, but doesn’t guarantee they will be sorted (they still could be depending on the backend implementation).</p>

username: sparseinference
cooked: <p>Thanks for your answer!</p>
<p>It is certainly a surprise though. On both cpu and cuda the <code>sorted</code> argument does’nt change the result at all.</p>

username: ptrblck
cooked: <p>It does change the output on my setup:</p>
<pre><code class="lang-python">x = torch.rand(5,5).cuda()
print(torch.topk(x, 5, sorted=False).indices == torch.topk(x, 5, sorted=True).indices)
&gt; tensor([[False, False, False, False,  True],
          [ True, False, False, False,  True],
          [ True, False, False,  True,  True],
          [False, False, False, False,  True],
          [False, False, False,  True,  True]], device='cuda:0')
</code></pre>

username: sparseinference
cooked: <p>So, it seems  to depend on <code>k</code>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.topk(x, 2, sorted=False).indices == torch.topk(x, 2, sorted=True).indices)
tensor([[True, True],
        [True, True],
        [True, True],
        [True, True],
        [True, True]], device='cuda:0')
</code></pre>
Pytorch 1.7 slower on CUDA 11.0 than CUDA 10.2?

username: amirhf
cooked: <p>Hello,</p>
<p>I am experiencing slower <strong>distributed</strong> training with the new PyTorch 1.7 built with CUDA 11.0 compared to 10.2! Has anyone benchmarked anything yet? I use the same script in two different environments one with CUDA 11.0 and the other with CUDA 10.2. The same script that takes 21 hrs for one epoch on CUDA 10.2 takes 24 hours on CUDA 11.0.</p>

username: ptrblck
cooked: <p>I guess the potential slowdown is not coming from distributed training (and thus NCCL) not from CUDA11, but might be coming from e.g. cudnn (which also depends on the device you are using).</p>
<p>Are you only seeing the slowdown using DDP or also using a single device? The latter case would point towards my assumption.</p>
<p>Could you give more information about your setup (GPU, model architecture) and also profile the training on a single device?</p>

username: benfei
cooked: <p>I’m having similar issue with pytorch 1.7 w/ CUDA 11.0 compared to CUDA 10.1. I’m using 2080Ti as GPU.</p>
<p>Simple example that demonstrates this (only conv2d):</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

x = torch.randn(10, 64, 128, 128).cuda()
w = torch.randn(64, 64, 5, 5).cuda()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

# warmup
y = []
for _ in range(10):
    y.append(F.conv2d(torch.randn_like(x), w))

# measure
start.record()
y = []
for _ in range(10):
    y.append(F.conv2d(torch.randn_like(x), w))
end.record()
torch.cuda.synchronize()
print('time = %.2f' % (start.elapsed_time(end),))
</code></pre>
<p>Results:</p>
<pre><code class="lang-auto"># pytorch 1.7 w/ cuda 10.1
# time = 21.05 +/- 0.05
# pytorch 1.7 w/ cuda 11.0
# time = 25.40 +/- 0.05
</code></pre>

username: ptrblck
cooked: <p>Could you update to PyTorch <code>1.7.1</code> with CUDA11.0 and cudnn8.0.5 and recheck the performance, please?</p>

username: benfei
cooked: <p>Thanks for the quick response!<br>
I used 1.7.1 in my tests. Sorry for not providing patch-version in my previous comment.<br>
I’ve uploaded my collect-env logs to a relevant github issue: <a href="https://github.com/pytorch/pytorch/issues/47908#issuecomment-745140208" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/47908#issuecomment-745140208</a></p>

username: jucor
cooked: <p>Interesting. Is the problem still present with Cuda 11.2? And have others been able to see the same problem?</p>
Does casting one datatype to another break the computaton graph?

username: someAdjectiveNoun
cooked: <p>I have a simple NER model that uses a word2vec embeddings layer followed by a linear layer and a softmax function.</p>
<p>To compute the loss after the <code>softmax</code> function, I use the <code>BCELoss</code> function.</p>
<pre><code class="lang-auto">class SemanticCRF(nn.Module):

    def __init__(self):
        super(SemanticCRF, self).__init__()

        # log reg
        self.hidden2tag = nn.Linear(300, 2)

        # crf
        self.crf_layer = CRF(2, batch_first=True)

        # binary cross entropy loss
        self.criterion = nn.BCELoss()


    def forward(self, input_ids=None, attention_mask=None, labels=None):

        # log reg
        probablities_ = F.softmax ( self.hidden2tag( input_ids.float() ) )
        probablities__ = torch.argmax(probablities_, dim=2)
        print(probablities__)
        
        probablities___ = torch.masked_select(probablities__, attention_mask)
        labels = torch.masked_select(labels, attention_mask)

        print('Logits: ', probablities__.dtype)
        print('Labels: ', labels.dtype)

        loss = self.criterion(probablities___.float() , labels.float())
        # print(loss)
        print(loss.requires_grad) 
        # return loss, torch.Tensor(probablities.float()), labels.float()
        return loss, probablities___, labels

</code></pre>
<p>However, this causes the following error. The model worked perfectly fine with a CRF layer that output loss which was backpropagated without error. The error comes up only with BCELoss. I am using BCELoss because there are only two entities in my NER task. With BCELoss, <code>loss.requires_grad</code> is set to <code>False</code>.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "TrainModel.py", line 264, in &lt;module&gt;
    train(model, optimizer, scheduler, train_dataloader, development_dataloader, args)
  File "TrainModel.py", line 128, in train
    abs(b_loss).backward()
  File "/home//lib/python3.6/site-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home//lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: ptrblck
cooked: <p>Changing the <code>dtype</code> won’t break the computation graph as long as you are using floating point tensors.<br>
However, this operation <code>probablities__ = torch.argmax(probablities_, dim=2)</code> will break it, as <code>argmax</code> is not differentiable. Casting the output back to <code>.float()</code> won’t restore the graph.</p>
Take gradient of loss with respect to weight

username: Mari
cooked: <p>I am wondering how I can take gradient of residual and BCs losses w.r.t. weights?<br>
class model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(model, self).<strong>init</strong>()<br>
self.layers = nn.Sequential(<br>
nn.Linear(input_n,h_n),<br>
Swish(),<br>
nn.Linear(h_n,h_n),<br>
Swish(),<br>
nn.Linear(h_n,h_n),<br>
…<br>
def forward(self,x):<br>
output = self.layers(x)<br>
return  output<br>
optimizer = optim.Adam(model.parameters(), lr=learning_rate, betas = (0.9,0.99),eps = 10**-15)</p>
<p>loss = loss_res + Lambda_BC* loss_bc<br>
loss.backward()</p>
<p>I extracted the weights of the model with this approach:<br>
for layers in model.modules():<br>
if isinstance(layers, nn.Linear):<br>
weight = layers.weight</p>
<p>Do I need to do something like this:<br>
weight.require_grads = True<br>
loss_res_grad= torch.autograd.grad(loss_res, weight)[0]</p>
<p>Thanks,<br>
Mari</p>
Equivalence between DP and DDP

username: CDhere
cooked: <p>I was using DP for training my models and later swtiched to DDP, however I noticed a significant performance drop after switching to DDP. I’ve double checked and made sure that data batches (size, sampling, random seeds, etc.) are consistent in two senarios, and have modified learning rate according to the “proportional to batch size” guideline as in the “Train ImageNet in 1 hour” paper. However I still got the performance drop with DDP.</p>
<p>Is this expected? My understanding is that if we make sure the model sees the same data and learning rate (and of course start from the same initialization), DP and DDP training should produce the same model? Am I missing anything? Are there other factors that will like lead to differences, say, loss function, batch norms?</p>
<p>Thanks!</p>

username: wayi
cooked: <p>Could you provide a script that can reproduce this issue?</p>
<p>Are you sure that you used the same learning rate, loss function, optimizer, same number of epochs, etc?</p>
<p>Both DP and DDP should be able to produce a model that can be also output by PyTorch w/o using DP or DDP, as long as the gradients are synced at every step. What about the performance w/o using DP or DDP?</p>

username: wayi
cooked: <p>What’s the loss function? If the loss function is not commutative, then it make result in a difference.</p>
How can I more efficiently reduce noise in the gradient by using a continuous relaxation of the gradient for max pooling layers?

username: ProGamerGov
cooked: <p>The code I have below was adapted from <a href="https://github.com/tensorflow/lucid/blob/master/lucid/optvis/overrides/smoothed_maxpool_grad.py" rel="noopener nofollow ugc">code written for the original TensorFlow 1.x</a>, and thus is probably not as efficient as it could be in PyTorch. When it replaces the MaxPool2d layers in a model, it reduces the noise in the gradient by using a continuous relaxation of the gradient for max pooling.</p>
<p>The issue is that it seems to be really slow and can result in out of memory errors (especially when using CUDA), but I need it or something like it to calculate attributions for specific spatial positions. I am a bit out of my depth here with this issue and thus any help would be appreciated!</p>
<pre><code class="lang-auto">class MaxPool2dSmoothed(torch.nn.Module):
    """
    Pooling layer where, if we backprop through it,
    gradients get allocated proportional to the input activation.
    Then we backprop through that instead.
    
    #https://github.com/tensorflow/lucid/blob/master/lucid/optvis/overrides/smoothed_maxpool_grad.py
    #https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/building-blocks/AttrSpatial.ipynb
    """

    def __init__(
        self,
        kernel_size: Union[int, Tuple[int, ...]],
        stride: Optional[Union[int, Tuple[int, ...]]] = None,
        padding: Union[int, Tuple[int, ...]] = 0,
        ceil_mode: bool = False,
    ) -&gt; None:
        super().__init__()
        self.avgpool = torch.nn.AvgPool2d(
            kernel_size=kernel_size,
            stride=stride,
            padding=padding,
            ceil_mode=ceil_mode,
        )

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x_avg = self.avgpool(x ** 2) / (1e-2 + self.avgpool(torch.abs(x)))

        x_smoothed = torch.autograd.grad(
            outputs=[x_avg],
            inputs=[x],
            grad_outputs=[torch.ones_like(x_avg)],
            create_graph=True,
        )[0]
        return x_smoothed


pool = MaxPool2dSmoothed(kernel_size=3, stride=2, padding=0, ceil_mode=True)
test_input = torch.nn.Parameter(torch.randn(1, 6, 64, 64))
test_output = pool(test_input.clone())
</code></pre>
How to train domain adaptive model

username: Hwarang_Kim
cooked: <p>output_source = model(source_image)<br>
loss_s = crossentropyloss(output_source,0)<br>
loss_s.backward()<br>
output_target = model(target_image)<br>
loss_t = crossentropyloss(output_target,1)<br>
loss_t.backward()</p>
<p>or</p>
<p>output_source = model(source_image)<br>
loss_s = crossentropyloss(output_source,0)<br>
output_target = model(target_image)<br>
loss_t = crossentropyloss(output_target,1)<br>
loss = loss_s + loss_t<br>
loss.backward()</p>
<p>which one is right approach?<br>
thank you in advance</p>

username: ptrblck
cooked: <p>Both approaches should work.<br>
The first one would use less memory and more compute, since you would be using two backward calls (more compute), but Autograd will be able to delete the intermediate tensors from the first forward pass, as they are not needed anymore (gradients were already computed using them).<br>
The latter approach would store both computation graphs and would thus use more memory.</p>
Split parameters from a model

username: avinash_kori
cooked: <p>Is there a way to use split model parameters and use separate optimizers for each parameter set?</p>
<p>I currently have model like:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, arg4, arg3, arg2, arg1=1):
        super(Net, self).__init__()
        self.customout = CustomLayer(args1, args2) 
        self.hidden = torch.nn.Linear(args2, args3) 
        self.predict = torch.nn.Linear(args3, args4) 

    def forward(self, x):
        out = ...
        return out

</code></pre>
<p>I would like to split model parameters as:</p>
<pre><code class="lang-auto">param1 = {customLayer.weight, customLayer.bias}
param2 = {*.Linear.weight, *.Linear.bias}
</code></pre>
<p>So that I can maximize over paramset1 and minimize over paramset2</p>
<p>I tried something like this:</p>
<pre><code class="lang-auto">param1 = set()
param2 = set()
for name, m in net.named_parameters():
    if name.lower().startswith('custom'):
        param1 |= set(m)
    else:
        param2 |= set(m)

optimizer1 = torch.optim.Adam(param1, lr=learning_rate)
optimizer2 = torch.optim.Adam(param2, lr=learning_rate)
</code></pre>
<p>But this results in error:</p>
<pre><code class="lang-auto">    raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>please let me know how this can be addressed</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Are you seeing the same error, if you pass all parameters to the optimizer?<br>
If so, I guess you might be applying a differentiable operation on (some) parameters inside the <code>CustomLayer</code>, such as <code>to()</code>.<br>
Could you post the <code>CustomLayer</code> definition in this case, so that we could have a look?</p>

username: avinash_kori
cooked: <p>When using net.parameters() in the optimizer, it works fine.</p>
<p>customlayer defn is just element-wise product, as defined below:</p>
<pre><code class="lang-auto">class CustomLayer(nn.Module):
    def __init__(self, args1, args2):
        super(CustomLayer, self).__init__()
        self.weights = nn.Parameter(torch.Tensor(args1, args2))

    def forward(self, x):
        return x * self.weights
</code></pre>

username: ptrblck
cooked: <p>Thanks for the update. Could you check which objects are stored in <code>param1</code> and <code>param2</code>? Are you seeing valid <code>nn.Parameters</code> in them?</p>
Batches of the training set from two datasets

username: marcomatt92
cooked: <p>Hello everyone, I’m pretty new to Pytorch.<br>
I have two different datasets, and I need to build my training set where every batch is made of N images, where N/2 are from dataset1 and N/2 from dataset2.<br>
So reading something from here:</p><aside class="quote quote-modified" data-post="35" data-topic="649">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martin_pflaum/48/12877_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/train-simultaneously-on-two-datasets/649/35">Train simultaneously on two datasets</a> 
  </div>
  <blockquote>
    If you are looking for using multiple dataloaders at the same time this should work 

class cat_dataloaders():
    """Class to concatenate multiple dataloaders"""

    def __init__(self, dataloaders):
        self.dataloaders = dataloaders
        len(self.dataloaders)

    def __iter__(self):
        self.loader_iter = []
        for data_loader in self.dataloaders:
            self.loader_iter.append(iter(data_loader))
        return self

    def __next__(self):
        out = []
        for d…
  </blockquote>
</aside>

<p>I tried to do something similar:</p>
<pre><code class="lang-auto">concat = data.ConcatDataset(datasets=(train_set1, train_set2))

mixed_training = data.DataLoader(concat, batch_size=opts.batch_size,
                                   sampler=DistributedSampler(ConcatDataset(train_set1, train_set2),
                                  num_replicas=world_size, rank=rank),
                                 num_workers=opts.num_workers)
</code></pre>
<p>where data.ConcatDataset is the pytorch method, ConcatDataset is implemented using the code in the previous link.</p>
<p>I need a DistributedSampler to use the set_epoch attribute in the training part.<br>
The performances seem pretty good doing so, I just wanted to know if it’s correct or if there’s a simpler/more correct way.<br>
Thank you for the help!</p>
Reparametrization in Keras to Pytorch

username: anotherone_one
cooked: <p>Hello everyone! Does anyone know how to convert this reparametrization function to pytorch? In particular, I don’t know how to properly do the <code> tf.random.normal(shape=(batch, latent_dim))</code> in pytorch.</p>
<pre><code class="lang-auto">batch, latent_dim = z_mean.shape
epsilon = tf.random.normal(shape=(batch, latent_dim))
 z = z_mean + tf.math.exp(0.5 * z_logsigma) * epsilon
</code></pre>
<p>On another topic, does anyone know which is the best alternative for <code>tf.nn.sigmoid_cross_entropy_with_logits()</code> in pytorch?</p>
<p>Thank you</p>

username: Dwight_Foster
cooked: <p>Ok so for the first function you could do something like this</p>
<pre><code class="lang-auto">epsilon = torch.normal(0,1, size=(batch, latent_dim))
z = z_mean + torch.exp(0.5 * z_logsigma) * epsilon
</code></pre>
<p>and for the loss you can use BCE with logits loss <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">here</a>.</p>
Is my format for loading MNIST data from the root directory correct?

username: bing
cooked: <p>Hi,<br>
I am trying to modiy a tutorial where instead of downloading the MNIST data, it is reading from the saved root directory which I have already downloaded.<br>
Am I doing it correctly because I am getting <code>Dataset not found</code>  error.<br>
Below is my code for train and test loader,  hvd.rank() modification is for horovod changes and are taken from  <a href="https://github.com/horovod/horovod/blob/master/examples/pytorch/pytorch_mnist.py" rel="noopener nofollow ugc">here</a></p>
<pre><code>train_dataset = \
   datasets.MNIST('/scratch/netra/MNIST/processed/training.pt-%d' % hvd.rank(), train=True, download=False,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ]))
train_dataset = \
   datasets.MNIST('/scratch/netra/MNIST/processed/test.pt-%d' % hvd.rank(), train=True, download=False,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ]))</code></pre>

username: ptrblck
cooked: <p>The <code>MNIST</code> dataset expects a path to the folder containing the dataset, not the path to specific <code>*.pt</code> files, so you would need to change this.</p>

username: bing
cooked: <p>I changed it, still the same problem. I am suspecting the problems are coming due to horovod changes in the path.</p>
Suggestion of model

username: Samuel_Bachorik
cooked: <p>Hello I want to ask if you can suggest me some good model for road segmentation. I have dataset with 2000 images from city and my small models just cant do this job for me. They have small capacity.<br>
I leave one photo and mask from dataset.<br>
Image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc6334f8ec75a00cc9946e95c160a93409a8f6b9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cc6334f8ec75a00cc9946e95c160a93409a8f6b9" title="Druhe_Moment1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc6334f8ec75a00cc9946e95c160a93409a8f6b9.jpeg" alt="Druhe_Moment1" data-base62-sha1="ta5X8v4zViPklV9sl3m866olYBX" width="690" height="473" data-dominant-color="717478"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Druhe_Moment1</span><span class="informations">700×480 65 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Mask<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22786695127403b81b7183ecd582acb450844ae4.png" data-download-href="https://discuss.pytorch.org/uploads/default/22786695127403b81b7183ecd582acb450844ae4" title="Druhe_Moment1_color_mask"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/22786695127403b81b7183ecd582acb450844ae4.png" alt="Druhe_Moment1_color_mask" data-base62-sha1="4UWbRtCNKcoxsLzInTQF1nh6tnu" width="690" height="473" data-dominant-color="B542C6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Druhe_Moment1_color_mask</span><span class="informations">700×480 820 Bytes</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Dwight_Foster
cooked: <p>You could look at detectron2 <a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">here</a> or maybe unet <a href="https://github.com/milesial/Pytorch-UNet" rel="noopener nofollow ugc">here</a>.</p>
Is it possible to calculate the Hessian of a network while using gradient checkpointing?

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I just have a general question about the use of gradient checkpointing! I’ve recently discussed this method and it seems it’d be quite useful for my current research as I’m running out of CUDA memory.</p>
<p>After reading the <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">docs</a>, it looks like it doesn’t support the use of <code>torch.autograd.grad()</code> but only <code>torch.autograd.backward</code>. Within my model, I used both <code>torch.autograd.grad</code> and <code>torch.autograd.backward</code> as my loss function depends on the Laplacian (trace of the Hessian) of the network with respect to the inputs and another terms. This would be something like this (in terms of actual code),</p>
<pre><code class="lang-auto">output = model(input) #R^N to R^1 function (nn.Module)
loss_values = calc_laplacian(model, input) + input.pow(2) #M number of samples  in batch
loss = loss_values.mean() #take mean of loss_values to get loss for given batch

optim.zero_grad() #clear grad cache
loss.backward()   #calculate loss
optim.step()      #update parameters
</code></pre>
<p>where <code>calc_laplacian</code> is my own custom function which calculates the Laplacian (via use of <code>torch.autograd.grad</code>) of an <code>nn.Module</code> function with respect to <code>input</code> which is the input Tensor to the <code>model</code>.</p>
<p>Would it be possible to implement gradient checkpointing for such a model, and if so, are there any examples available online?</p>
<p>Any help is appreciated! Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Linear Layer optimisation with Adam: is there an order?

username: Jean-Eric_Campagne
cooked: <p>Dear experts</p>
<p>I wander if for instance considering a very simple Single nn.Linear(1,N) layer, Adam is optimising the weights/bias (2xN parameters) in a certain order? This means that the converging solution results in some way of a solution for the first couple (w1,b1) then (w2,b2) and so on.</p>
<p>best.</p>