akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
TypeError: The outputs of the user-provided function given to hessian must be either a Tensor or a tuple of Tensors but the given outputs of the user-provided function has type <class 'Functions.Determinant'>

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’m trying to calculate the hessian of a custom autograd Function via the use of <code>torch.autograd.functional.hessian(Determinant. A)</code>. However, I’m getting the following error!</p>
<pre><code class="lang-auto">import torch
from Functions import Determinant

A = torch.randn(2,2, requires_grad=True)

psi_t = torch.det(A)
psi_c = Determinant.apply(A)

print(psi_t, psi_c)

grad1_t = torch.autograd.grad(psi_t, A)
grad1_c = torch.autograd.grad(psi_c, A)

print(grad1_t, grad1_c)

grad2_t = torch.autograd.functional.hessian(torch.det, A) #works
print(grad2_t)
grad2_c = torch.autograd.functional.hessian(Determinant, A) #fails
</code></pre>
<p>where <code>Determinant</code> is a <code>torch.autograd.Function</code> with its own Backward and DoubleBackward method. How exactly can I calcuate the hessian via <code>torch.autograd.functional.hessian</code> with a custom <code>torch.autograd.Function</code>?</p>
<p>The full stacktrace is,</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test_custom_det.py", line 18, in &lt;module&gt;
    grad2_c = torch.autograd.functional.hessian(Determinant, A)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 701, in hessian
    res = jacobian(jac_func, inputs, create_graph=create_graph, strict=strict, vectorize=vectorize)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 482, in jacobian
    outputs = func(*inputs)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 697, in jac_func
    jac = jacobian(ensure_single_output_function, inp, create_graph=True)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 482, in jacobian
    outputs = func(*inputs)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 685, in ensure_single_output_function
    is_out_tuple, t_out = _as_tuple(out, "outputs of the user-provided function", "hessian")
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 21, in _as_tuple
    raise TypeError("The {} given to {} must be either a Tensor or a tuple of Tensors but the"
TypeError: The outputs of the user-provided function given to hessian must be either a Tensor or a tuple of Tensors but the given outputs of the user-provided function has type &lt;class 'Functions.Determinant'&gt;.
</code></pre>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

AlphaBetaGamma96: <p>Solution: I need to wrap my custom function within a <code>nn.Module</code> or a function to make it callable, for example,</p>
<pre><code class="lang-auto">def custom_det(A):
  return Determinant.apply(A)
</code></pre>
<p>then,</p>
<pre><code class="lang-auto">hessian = torch.autograd.functional.hessian(custom_det, A)
</code></pre>
Extract batch of square submatrices from batch of square matrices

Samue1: <p>The following piece of code extracts a batch of square submatrices <code>sub_S</code> from a batch of square matrices <code>S</code>. This piece of code is currently a bottleneck in a larger system. Does PyTorch provide any functionality to get rid of the for loops? So far I didn’t find anything that could improve it.</p>
<p>Here is a minimal working example</p>
<pre><code class="lang-auto"># Parameters
batch_size = 2
size = 24
n_idx = 6

# Mock data
S = torch.arange(end=batch_size*size*size).reshape(batch_size, size, size)

# Mock indices
indices = list()
for _ in range(batch_size):
    indices.append(torch.randint(high=size, size=(n_idx**2, 2)))

sub_S = list()
for b in range(batch_size):
    sub_S.append(S[b][indices[b][:, 0], indices[b][:, 1]].reshape(n_idx, n_idx))

sub_S = torch.stack(sub_S)
</code></pre>

Mazhar_Shaikh: <p>Hi Samue1,</p>
<p>To eliminate the second for loop in your code, you can use broadcasted indexes in the batch dimension as well. An example is shown below</p>
<pre><code class="lang-auto">batch_size = 100
size = 24
n_idx = 4

# Mock data
S = torch.arange(end=batch_size*size*size).reshape(batch_size, size, size)
indices = list()
for _ in range(batch_size):
    indices.append(torch.randint(high=size, size=(n_idx**2, 2)))

def forlooping(S, indices):
    # Mock indices
    sub_S = list()
    for b in range(batch_size):
        sub_S.append(S[b][indices[b][:, 0], indices[b][:, 1]].reshape(n_idx, n_idx))
    sub_S = torch.stack(sub_S)
    return sub_S

def indexing(S, indices):
    indices = torch.stack(indices, dim=0).view(-1, n_idx, n_idx, 2)
    bidx = torch.arange(batch_size).view(-1,1,1)
    sub_S = S[bidx, indices[...,0], indices[...,1]]
    return sub_S
</code></pre>
<p>The speed up is noticeable with large batch sizes like so</p>
<pre><code class="lang-auto">In [2]: %timeit indexing(S, indices)
155 µs ± 1.98 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [3]: %timeit forlooping(S, indices)
1.9 ms ± 14.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
</code></pre>
<p>To eliminate the first for loop, simply include the batch size in the size parameter like below</p>
<pre><code class="lang-auto">indices = torch.randint(high=size, size=(batch_size, n_idx**2, 2)))
</code></pre>
<p>Hope this helps!</p>

Samue1: <p>Could you explain a bit more the logic behind reshaping <code>indices</code> like <code>(-1, n_idx, n_idx, 2)</code>? Why does that in combination with <code>[...,0]</code> and <code>[...,1]</code> work? Thanks!</p>
Torch.clamp backward got nan values

wangjuan313: <p>In my codes, I have used torch.clamp as follows:<br>
epsilon = 1e-6<br>
ypred =  torch.clamp(ypred, epsilon, 1-epsilon)</p>
<p>and got error message as follows:<br>
Error: Function ‘ClampBackward’ returned nan values in its 0th output.</p>
<p>I have no idea what the problem is. Any suggestions?</p>

googlebot: <p>Non-finite value is introduced somewhere later.</p>
<p>Following gives the same error; Inf values are not considered anomalous for some reason…</p>
<pre><code class="lang-auto">with torch.autograd.set_detect_anomaly(True):
	torch.ones(1).requires_grad_().clamp(0.01, 0.99).div(0).backward()
</code></pre>

wangjuan313: <p>Thanks. Your reply is very helpful.</p>

fengqh: <p>how to solve this problem, please?</p>

googlebot: <p>it is a generic numerical problem, topic’s case was about Inf values produced in forward</p>

fengqh: <p>i download codes from github, i check code and do not find nan, how to deal with this problem, thank you</p>

fengqh: <p>i check codes and do not find div zero</p>

googlebot: <aside class="quote no-group" data-username="fengqh" data-post="6" data-topic="107632">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/71e660/48.png" class="avatar"> fengqh:</div>
<blockquote>
<p>how to deal with this problem</p>
</blockquote>
</aside>
<p>use a debugger, ensure that your loss (forward output) contains non-finite values (perhaps at some epoch &gt; 1), re-run forward() step-by-step to find the problem (you can use conditional breakpoints and “jump to cursor” in pycharm’s debugger). Sorry, I cant think of an easier way.</p>

fengqh: <p>i meet this problem at about 20 epoch, debug may need one day <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=10" title=":sob:" class="emoji" alt=":sob:"> <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=10" title=":sob:" class="emoji" alt=":sob:"></p>
How do I know randperm is performed on GPU

kevin_Xiong: <p>Hi, I am wondering how PyTorch ensures Ops like arange or randperm which creates Tensors by taking scalars are running on GPU. Because they don’t have a context where for some op the input tensors can provide such a device context or parametric op whose parameters can provide it as well.</p>
<p>I’ve run the following and the output is CPU. I know that because by default the output tensor will be transferred to CPU. I want to know whether the generation process is conducted on GPU. And it seems that Ops like these are exported directly from cpp.</p>
<pre><code class="lang-auto">class _test_gpu_(torch.nn.Module):
    def __init__(self):
        super(_test_gpu_, self).__init__()
        self.f=torch.nn.Linear(4,4)
        new_Wight = torch.Tensor(np.ones([4,4])) 
        self.f.weight = torch.nn.Parameter(new_Wight)

    def forward(self,x):
        ind=torch.randperm(4)
        print(ind.device)
        y=self.f(x)   
        return  y[ind]
t= _test_gpu_().cuda()
x=torch.Tensor(np.ones([1,4])).cuda()
res=t(x)
</code></pre>

ptrblck: <p>You can specify the desired <code>device</code> in <code>torch.ramdperm</code> and check the used CUDA kernel e.g. via:</p>
<pre><code class="lang-python">nsys nvprof python -c "import torch; torch.randperm(10, device='cuda')"
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">CUDA Kernel Statistics:

 Time(%)  Total Time (ns)  Instances  Average  Minimum  Maximum                                                  Name                                                
 -------  ---------------  ---------  -------  -------  -------  ----------------------------------------------------------------------------------------------------
    42.1            7,425          1  7,425.0    7,425    7,425  void at::cuda::detail::cub::DeviceRadixSortSingleTileKernel&lt;at::cuda::detail::cub::DeviceRadixSortP…
    16.3            2,880          1  2,880.0    2,880    2,880  void at::native::(anonymous namespace)::distribution_elementwise_grid_stride_kernel&lt;unsigned int, 4…
    14.3            2,528          1  2,528.0    2,528    2,528  void at::native::vectorized_elementwise_kernel&lt;4, at::native::FillFunctor&lt;long&gt;, at::detail::Array&lt;…
    13.8            2,432          1  2,432.0    2,432    2,432  void (anonymous namespace)::elementwise_kernel_with_index&lt;int, at::native::arange_cuda_out(c10::Sca…
    13.4            2,368          1  2,368.0    2,368    2,368  void (anonymous namespace)::randperm_handle_duplicate_keys_kernel&lt;int, at::native::(anonymous names…
</code></pre>
`torch.utils.checkpoint` with BatchNorm

guozhiyao4: <p>When I use the <code>checkpoint</code> to warp a module with BatchNorm, will it change the running_average of BN?</p>

guozhiyao4: <p>I use a toy model to test the value</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.utils.checkpoint import checkpoint
from torch import optim

class bn_linear(nn.Module):
    def __init__(self):
        super(bn_linear, self).__init__()
        self.bn = nn.BatchNorm1d(10, momentum=0.001)
        self.linear = nn.Linear(10, 1)
    def forward(self, x):
        return self.linear(self.bn(x))

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.s = bn_linear()
    def forward(self, inp):
        return checkpoint(self.s, inp)
        # return self.s(inp)

model = Model()

for i in range(1000):
    x = torch.ones(2,10)*i
    x.requires_grad_(True)
    loss = torch.mean(model(x))

print({n:p for n, p in model.named_buffers()})
</code></pre>
<p>But the result shows the same.</p>
Using Dynamic Parallelism in C++

jimoskar: <p>Hi! I am working on a project where we load a TorchScript model into C++. We have tried to make a model utilizing dynamic parallelism as described in <a href="https://pytorch.org/tutorials/advanced/torch-script-parallelism.html" rel="noopener nofollow ugc">this</a> tutorial. When we do inference with the model in Python, we see a reduction in run time when parallelism is used. However, when we load the model into C++, there is no improvement in run time. We have used torch.jit.fork to parallelize the forward pass in the model. When we run the torch.autograd.profiler in Python, we see that the forward pass runs in parallell, while in C++ we do not see this. Is even dynamic parallelism of jit::Modules supported in C++?</p>
Loss function dosen't change from epoch to an other

el_youssfi_azeddine: <p>Hello,<br>
I working on a huge and complicated pytorch model to solve PDEs using torch geometric graphs and so on, I do computations on GPU RTX3090, but when I lunch the training I have memory error , I start searching, one of the thing I remarked is that when calculating error if I store it as a scalr instead of a tensor I use much less memory, but I convert it to a tensor just before calling backward() , but the problem is that in this case the training in starting but with no change from an epoch to an other.<br>
here is the training loop:<br>
an the epochs:<br>
Is there a way to resolve this, remmeber my forst problem is cuda memory error.<br>
thanks</p>

ptrblck: <p>You are detaching the losses from the computation graph by wrapping them into a new tensor.<br>
Setting <code>requires_grad = True</code> on this new tensor will then hide the expected error message.<br>
You can verify it by checking the <code>.grad</code> attributes of all parameters after the <code>backward()</code> op and would see that they are <code>None</code>.<br>
To fix this sum the losses directly without using <code>torch.tensor(err...)</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier.</p>

el_youssfi_azeddine: <p>By doing .item() to err_fe and err_eq I just gained a lot of memory and the cuda memory error just disappeared, is there a way to keep use item()?<br>
The code is really big ( more thank 400 line just for the model) and also it’s not my code I can’t publish it, I hope that you can help me otherwise; I can provide more precisions as much as needed.<br>
thanks</p>

el_youssfi_azeddine: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="128023">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>To fix this sum the losses directly without using <code>torch.tensor(err...)</code>.</p>
</blockquote>
</aside>
<p>By doing so  cuda memory error message appears then before the training starts.</p>

ptrblck: <p>It seems you might be running out of memory by fixing the issues, i.e. by avoiding to detach the tensor.<br>
You also cannot use <code>item()</code>, as it would also detach the result from the computation graph and would return a plain Python scalar.<br>
To lower the memory usage you could e.g. decrease the batch size etc.</p>
Unexpected tensor value between different cuda streams of different threads

3yuan: <p>In my libtorch C++ project, two threads are used, and each thread has its own cuda stream. when the  stream of thread2 tried to use the tensor output by the stream of thead1, in most cases, the tensor value was not expected in thread2. Any ideas? Thanks</p>

ptrblck: <p>This sounds as if you are missing synchronizations in your code.</p>

3yuan: <p>Is synchronization necessary between cuda streams?</p>

ptrblck: <p>Yes, as explained in the <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-streams">docs</a>:</p>
<blockquote>
<p>Operations inside each stream are serialized in the order they are created, but operations from different streams can execute concurrently in any relative order, unless explicit synchronization functions (such as <a href="https://pytorch.org/docs/stable/generated/torch.cuda.synchronize.html#torch.cuda.synchronize"><code>synchronize()</code></a> or <a href="https://pytorch.org/docs/stable/generated/torch.cuda.Stream.html#torch.cuda.Stream.wait_stream"><code>wait_stream()</code></a>) are used. For example, the following code is incorrect:</p>
</blockquote>
<pre data-code-wrap="python"><code class="lang-python">cuda = torch.device('cuda')
s = torch.cuda.Stream() # Create a new stream. 
A = torch.empty((100, 100), device=cuda).normal_(0.0, 1.0)
with torch.cuda.stream(s):
    # sum() may start execution before normal_() finishes!
    B = torch.sum(A)
</code></pre>
<blockquote>
<p>When the “current stream” is the default stream, PyTorch automatically performs necessary synchronization when data is moved around, as explained above. However, when using non-default streams, it is the user’s responsibility to ensure proper synchronization.</p>
</blockquote>
Learnable hidden state plus truncated back-propagation through time

leckofunny: <p>Hello everyone,</p>
<p>I’m currently examining the potential of learning a hidden state initialization (instead of zeros) for a recurrent neural net (i.e. GRU/LSTM). However, I would like to combine the learnable initialization with truncated back-propagation through time.</p>
<p>Starting from this simple <a href="https://colab.research.google.com/github/dlmacedo/starter-academic/blob/master/content/courses/deeplearning/notebooks/pytorch/Time_Series_Prediction_with_LSTM_Using_PyTorch.ipynb" rel="noopener nofollow ugc">Time Series Perdiction Problem</a> I gave it a try and implemented a learnable hidden state initialization given truncated bptt. My adjustments can be found in this <a href="https://colab.research.google.com/drive/1SidGPovRXjZW73-PsNOr_HsY86I32AiX?usp=sharing" rel="noopener nofollow ugc">colab notebook</a>.</p>
<p>Conceptually, I have to make use of nn.Parameter(torch.zeros(hidden_state_size), requires_grad=True) to make it learnable. This is pretty simple if no truncation is used. Once chunks of sequences are fed, while feeding the latest hidden state make things more complicated. I’m wondering now, whether my code is conceptually correct. Could anybody take a brief look at it or could suggest an approach to assess it (e.g. visualizing the graph, plotting gradients)?</p>
<p>In the end, I’d like to add it to a Deep Reinforcement Learning algorithm that does use truncated bptt. Collecting the hidden states and their gradients inside the replay buffer will be another challenge. But that’s not worth to tackle if I’m uncertain concerning the learnable hidden state initialization.</p>
<p>Big thanks in advance to any opinions and advice on this matter. I hope that this thread will be helpful to others as well.</p>

leckofunny: <p>From a theoretical point of view, achieving a learnable hidden state initialization while doing truncated BPTT is not valid, because gradients are truncated and thus cannot be backpropagated to the original initialization.</p>
NaNs in loss temporarily go away after reloading checkpoint

wes10: <p>After training for about 1000 iterations I start getting NaNs in my loss. I reloaded the most recent checkpoint (at 900 iterations)  to debug. After starting training at 900 iterations I didn’t see NaNs in the loss again until around 2000 iterations. Reloaded the checkpoint at 1900 iterations and then saw NaNs again around 3000. Anyone have ideas on why this behavior could be occurring?</p>

ptrblck: <p>Your model seems to be diverging non-deterministically.<br>
I also don’t know, if you are resetting e.g. the optimizer, which could also delay the divergence.</p>
Accurate performance estimation for CUDA platforms

Ajinkya.Bankar: <p>Is there any procedure to find the Cuda platform-agnostic execution time of a certain inference operation? I want to compare the execution time of two different inference operations. But, the execution time depends on the already existing load on the GPU and many other factors. I tried the profiler example given <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">here</a>. But, it shows different CUDA times for each independent trial on the same GPU. And, the CUDA times are completely different if I change the GPU. Therefore, it is not an accurate estimation. In computer science, we find the number of CPU cycles to compare the performance of two algorithms. Is there a similar analogy for GPU using PyTorch?</p>

ptrblck: <p><code>torch.utils.benchmark</code> would probably give you the most accurate estimate, as it will add warmup iterations and synchronize for you.<br>
Different performance on different devices might be expected especially if you are using libs to accelerate the workload, e.g. cuDNN.</p>
Reduced Precision

Geri: <p>Hello,  I am trying to implement reduced precision on my code to explore the performance of training. However, I am trying what is introduced in the tutorials but I am not able to get it working.</p>

ptrblck: <p>What exactly are you using and which errors are you seeing?</p>

Geri: <p>I just cannot understand how to implement it in my code. Do you think I could get some help if I can share my code somewhere ?</p>

ptrblck: <p>Yes, the forum is a place to share code snippets tp get some help.<br>
I don’t know which reduced precision approach you are using, but in case via <code>torch.cuda.amp</code> take a look at the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">examples</a>.</p>
Get batched tensor elements by index

Samue1: <p>I want to use the indices returned by <code>max_pool2d_with_indices(x)</code> to extract elements from a tensor <code>m</code> with the same dimensions as <code>x</code>. However, the returned values are wrong. Here is a minimal working example:</p>
<pre><code class="lang-auto">import torch

def main():

    x = [[[[0, 0, 0, 0],
           [0, 1, 1, 0],
           [0, 1, 1, 0],
           [0, 0, 0, 0]],

         [[1, 0, 0, 1],
          [0, 0, 0, 0],
          [0, 0, 0, 0],
          [1, 0, 0, 1]]]]

    x = torch.tensor(x).float()

    m = [[[[10, 11, 12, 13],
           [14, 15, 16, 17],
           [18, 19, 20, 21],
           [22, 23, 24, 25]],

          [[26, 27, 28, 29],
           [30, 31, 32, 33],
           [34, 35, 36, 37],
           [38, 39, 40, 41]]]]

    m = torch.tensor(m).float()

    y, idx = torch.nn.functional.max_pool2d_with_indices(x, kernel_size=(2, 2))

    m_select = torch.take(m, idx)
    print(m_select)


if __name__ == "__main__":
    main()
</code></pre>
<p>I want to <code>m_select</code> to be</p>
<pre><code class="lang-auto">tensor([[[[15., 16.],
          [19., 20.]],

         [[26., 29.],
          [38., 41.]]]])
</code></pre>
<p>but currently I get</p>
<pre><code class="lang-auto">tensor([[[[15., 16.],
          [19., 20.]],

         [[10., 13.],
          [22., 25.]]]])
</code></pre>
<p>Meaning, I want the elements in <code>m</code> where the activations in <code>x</code> are the highest. But currently only elements from the first feature map are returned by <code>torch.take()</code>. How can I get the desired output?</p>

ptrblck: <p>You could flatten the tensors and use <code>gather</code> to get the desired output:</p>
<pre><code class="lang-python">out = torch.gather(torch.flatten(m, 2), 2, torch.flatten(idx, 2)).view(idx.size())
print(out)
&gt;  tensor([[[[15., 16.],
            [19., 20.]],

           [[26., 29.],
            [38., 41.]]]])
</code></pre>

Samue1: <p>Thank you!</p>
<p>Would it also be possible to use <code>torch.take()</code> by modifying <code>idx</code> in such a way that the indices do not start for every feature map from 0? Is there a simple way to do that?</p>

ptrblck: <p><code>take</code> would treat the tensor as 1D, so you could try to add the offsets to the indices so that the values can be accessed in a sequential/flattened manner.</p>
Do we need to define a softmax function in the case of cross entropy

OBouldjedri: <p>Hi I am running a CNN on pythorch and using the cross entropy loss I was wondering if I should define the softmax inside the neural network itself or is it fine to feed that to the loss and then get the predicted values for the accurcy calculus: what would be the difference ? thank ypou</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = ConvLayer()<br>
self.conv2 = ConvLayer2()<br>
self.conv3 = ConvLayer3()</p>
<pre><code>  ##########################################
  self.fc0 =nn.Linear(96*12*1,500)
  self.dr=nn.Dropout(0.2)
  self.fc1=nn.Linear(500,12)
  self.act=nn.ReLU()
 
###############################################

# x represents our data
def forward(self, x):
  # Pass data through conv1
  ############################## part1
  
  
  x =self.conv1(x)
  x=self.conv2(x)
  x=self.conv3(x)
  x = x.view(x.size(0), -1)
  #x = torch.flatten(x, 1)
  x=self.fc0(x)
  x=self.act(x)
  x=self.dr(x)
  x=self.fc1(x)
 

  return x
</code></pre>
<p>def train(model, optimizer, train_loader, epoch , writer,IMU):</p>
<pre><code>    model.train()
    train_loss=0
    #train_reconstruction_loss=0
    #train_margin_loss=0
    
    correct=0
    for batch_idx, (data, target) in enumerate(train_loader):
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data), Variable(target, requires_grad=False)
        optimizer.zero_grad()
        
      
        data=data[:,None,:]
     
        #output, probs = model(data)
        
        target=target.type(torch.LongTensor)  #switch it to long

        target=torch.squeeze(target,)
 
        #data, target = Variable(data), Variable(target)
        
        
        target=target.cuda()
        
        output=model(data)
        #print()
        print('output',output)
        
        input()
        
        target=target.long()
        
        
        
        loss=criterion(output,target)
        
        predicted_softmax =F.softmax(output, dim = 1)
        print('predicted softmax',predicted_softmax)
        print()
        
        _, predicted = torch.max(predicted_softmax, 1)
        print('predicted',predicted)
        input()
        output=predicted

        loss.backward()
        train_loss+=loss.data
        optimizer.step()
        
      
        
        correct += (output == target).sum().item()</code></pre>

ayalaa2: <p>If you’re using this loss specifically: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.9.0 documentation</a></p>
<p>Then you do not need to do a softmax operation. The loss, internally, will use a logsoftmax for computational stability reasons before the NLL. It mentions this at the top of the documentation.</p>
<p>You can just feed it the output of your network along with the labels.</p>
Single image and text input CNN with no ground truth

electric93: <p>Hello there!<br>
I’m very new to PyTorch and using its functionalities, but I’m working on a thesis regarding 3d facial animation. What I need is an algorithm that takes a single image (image shows which part of the face is not posed correctly yet) and maybe even the name of the controller (which is known) that controls this part of the face, and outputs two 3-dimensional vectors with one being the translation and the other being the rotation of the controller. I do not have a ground truth value to compare against, but I do have a very precise percentage of how close to perfect the animation is.<br>
So here was my plan:<br>
Create a CNN that takes an image and a text as the input, predicts some values for the two 3-dim vectors, then I get the percentage (instead of a loss function) and then do the backward propagation.<br>
Does this make any sense? And am I even correct in using a CNN for this kind of task?<br>
I also had a lot of issues understanding CNNs so how exactly would a text input work with the image? And how would such an algorithm be structured?</p>
<p>Thank you so much for your time and have a great day!!</p>
Multiplication equivalent to torch.sum()

JosAr: <p>Hey there,<br>
So i am looking for a quick and clean method to do some multiplication but i can not find a good solution.<br>
I have a vector with the size <code>size() = [8,16,1,2]</code> and would like to receive and vector of <code>size() = [8,16,1,1]</code> where in the last dimension the product of the two entries is taken.</p>
<p>If i wanted to take the sum it would be easy using <code>torch.sum(x, 3)</code> but is there a nice way to take the product?</p>
<p>Thanks for your help in advance!</p>

JosAr: <p>shame on me, i found an answer. Using <code>torch.prod()</code> does exactly the job.</p>
Is it possible to turn off gradients for individual input samples

subhashnerella: <p>I am trying to use a model which combines both Resnet and Transformer architectures. I will provide the input image to the resnet backbone and extract the features of the intermediate layer.<br>
for example:</p>
<p><code>input(size: 1,3,256,256) --------&gt; Resnet50layer3-----------&gt;features(1,1024,16,16)</code></p>
<p>from the output features (16X16), I take 10 pixels by some criteria which  results in tensor of size(1,1024, 10).</p>
<p>Now, this is provided as input to the transformer encoder with batch size 1, source sequence length 10, and embedding size 1024.</p>
<p>This works alright when I don’t use any data transforms other than <code>toTensor()</code>.</p>
<p>Now I am trying to use the <code>RandomErase</code> transform with erasing value as 0. When I select the pixels from the output of the Resnet50 to provide them as input to the transformer, some of the pixels I am interested in might lie in the corresponding erased region of the input image. So I create a key_padding_mask of size (batch_size, src_sequence_length) to ignore these erased region pixels. During the backpropagation step because of this mask, it works alright for the transformer part but the gradient backpropagates to resnet block and the weights of resnet become <code>nan</code>.</p>
<p>How can I prevent backpropagation for these specific pixels extracted from the erased region in the resnet block? The key_padding_mask has boolean values that represent which pixels should be avoided for gradient backpropagation. As I am using RandomErase the pixels which should be ignored change from sample to sample.</p>
View/print parameters values

Rami_Nasser: <p>I have a custom network, how can print the parameters along the network with informative names that denote the type of the param and the layer it belongs to ?</p>

JuanFMontesinos: <p>The parameters’ names are the ones you use when you set the network. Soo whether they are informative or not is your bussiness.<br>
The str representation of the model will display they all.</p>

Rami_Nasser: <p>I use for example nn.Linear as some of the parameters, what do you mean by str representation of the model ?</p>

JuanFMontesinos: <p>Python classes has a<code> __str__</code> attribute and a<code> __repr__</code> attribute such that if you use <code>print(model)</code> you will get a string.<br>
If you have a class inherited from a nn.Module and you print an instance of that class you will get a list of the layers sorted in the same way you defined them and with the name you used to define the layers inside the nn.Module.</p>
<p>So basically if you have</p>
<pre><code class="lang-auto">class Audio_Model(nn.Module):
    def __init__(self):
        super(Audio_Model, self).__init__()
        self.watermelon = nn.Linear(A,B)
</code></pre>
<p>Watermelon will appear when you do</p>
<pre><code class="lang-auto">model = Audio_Model()
print(model)
</code></pre>
<p>If you give a meaningful name, then that name will appear.</p>
Problem to load the torch.profiler module

Ajinkya.Bankar: <p>I am studying the tutorial given <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">here</a> to profile the CPU time of the model. I get an error message <code>ModuleNotFoundError: No module named 'torch.profiler'</code>. I have torch version 1.7.1 installed. May I know what’s the problem?</p>

ptrblck: <p>The new profiler was released in <code>1.8.1</code> as described in <a href="https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/">this blog post</a> so you would have to update PyTorch.</p>

Ajinkya.Bankar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> It worked. Thanks!</p>
Distributed Training PyTorch Gloo [socket: Too many open files]

msalvaris: <p>Hi,</p>
<p>I am trying to get a distributed training to work across multiple nodes using gloo backend. I am able to get everything working on a single node with 4 GPUs but once I go across nodes I get the following error and the processes hang:</p>
<p>Exception in thread Thread-46:Traceback (most recent call last): File "/opt/conda/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/opt/conda/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 476, in _reduction_thread_fn _process_batch() # just to have a clear scope File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 469, in _process_batch dist.all_reduce(coalesced, group=group_id) File "/opt/conda/lib/python3.6/site-packages/torch/distributed/<strong>init</strong>.py", line 324, in all_reduce return torch._C._dist_all_reduce(tensor, op, group)RuntimeError: [/opt/conda/conda-bld/pytorch_1524586445097/work/third_party/gloo/gloo/transport/tcp/pair.cc:142] socket: Too many open files</p>
<p>I have searched around and the only references I find are to issues with the data loader. In this example I am using synthetic data so that can’t be it. Any help would be much appreciated.</p>
<p>Thanks</p>

ztichigo: <p>I also faced the same issue with pytorch gloo backend.  My code works without this error after I reset the max number of socket in my linux like<code> ulimit -n 65535</code>, usually the defualt value is 1024.</p>
How to adjuist the learning rate in a custome way?

songyuc: <p>Hi, guys,<br>
I want to realize adjusting the learning rate in a custom way, while the Docs only provide certain fashioned methods as <a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" rel="noopener nofollow ugc">PyTorch – How to adjust learning rate</a>. So, I am wandering whether this is a tutorial or demo which I can follow to define a custom strategy of adjuisting the learning rate?</p>
<p>Any guide or answer will be appreciated!</p>

ptrblck: <p>You could check, if <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LambdaLR.html#torch.optim.lr_scheduler.LambdaLR"><code>LambdaLR</code></a> would fit your use case and if not, you could derive a custom learning rate scheduler using any of the <a href="https://github.com/pytorch/pytorch/blob/4ed885881704e96e61dca063e2d3bf881b753558/torch/optim/lr_scheduler.py">provided schedulers</a> as a template.</p>

songyuc: <p>Let me see this reference material.</p>

songyuc: <pre><code class="lang-python">&gt;&gt;&gt; # Assuming optimizer has two groups.
&gt;&gt;&gt; lambda1 = lambda epoch: epoch // 30
&gt;&gt;&gt; lambda2 = lambda epoch: 0.95 ** epoch
&gt;&gt;&gt; scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
&gt;&gt;&gt; for epoch in range(100):
&gt;&gt;&gt;     train(...)
&gt;&gt;&gt;     validate(...)
&gt;&gt;&gt;     scheduler.step()
</code></pre>
<p>If I use the scheduler like the given example, would the <code>lr</code> be processed sequentially by <code>lambda1</code> and   <code>lambda2</code> like a pipeline manner?</p>
Optimizer schedule step

OBouldjedri: <p>Hi  I am training a neural network on pythorch using the training / validation/ test configuration.<br>
for the optimizer I am doing this:</p>
<p>optimizer = optim.Adam(model.parameters(), lr=args.lr)</p>
<p>scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, verbose=True, patience=15, min_lr=1e-6)</p>
<p>but I was wondering if the  scheduler step should be taked in respect with the test loss or the validation loss, I mean after every epoch should I do:</p>
<p>scheduler.step(test_loss)</p>
<p>or</p>
<p>scheduler.step(val_loss)</p>
<p>Thank you in advance.</p>

ptrblck: <p>I would use the validation loss and would not use the test loss at all (neither for the scheduler, nor early stopping etc., as I would consider it a data leak).<br>
Once your training is finished using the training and validation datasets, you would use the test dataset once and deploy the model.</p>
Is there a way to add tensors to an existing dataloader?

jungminc88: <p>The dataloader I have is derived like this:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader,TensorDataset
dataset = TensorDataset(tensor0, tensor1)
dataloader = DataLoader(
                dataset,  
                sampler = RandomSampler(dataset)
                batch_size = batch_size
)
</code></pre>
<p>Then I realized I had to add <code>tensor2</code> when creating dataset: It should have been</p>
<pre><code class="lang-auto">dataset = TensorDataset(tensor0, tensor1, tensor2)
</code></pre>
<p>But I didn’t save <code>tensor0</code>, <code>tensor1</code>, or <code>dataset</code>. Creating these all over again is very costly.<br>
Can I make the desired modification to <code>dataloader</code> if I only have <code>dataloader</code>?</p>

ptrblck: <p>The <code>DataLoader</code> uses the provided <code>Dataset</code> to create the batches. Based on your description it seems that you would like to add the new tensor to the <code>Dataset</code>. Outside of the <code>DataLoader</code> loop, you could access it via <code>dataloader.dataset</code> and could also try to reassign a new <code>Dataset</code>. However, the <a href="https://github.com/pytorch/pytorch/blob/4ed885881704e96e61dca063e2d3bf881b753558/torch/utils/data/dataset.py#L213-L231"><code>TensorDataset</code></a> doesn’t provide a method to add new tensors, so you would have to recreate it.</p>

jungminc88: <p>Thank you very much!</p>
How can I do these operations in one call?

Omroth: <p>mask has shape [125]<br>
dots has shape [100, 125]</p>
<p>First I want to multiply each element in column c in dots by mask[c]</p>
<p>Second I want to sum each column c of dots to end up with a tensor of shape [100]</p>
<pre><code>        for (int i = 0; i &lt; 125; i += 1)
        {
            for (int j = 0; j &lt; 100; j+= 1)
            {
                dots.index({j, i}) = dots.index({j, i}) * mask.index({i});
            }
        }

        torch::Tensor final_add = torch::zeros({100}, torch::dtype(torch::kFloat64).requires_grad(false));

        for (int i = 0; i &lt; 125; i += 1)
        {
            for (int j = 0; j &lt; 100; j += 1)
            {
                final_add.index({j}) = final_add.index({j}) + dots.index({j, i});
            }
        }</code></pre>

ptrblck: <p>You could use broadcasting for the first operation and <code>sum(1)</code> for the second one as seen here:</p>
<pre><code class="lang-python"># setup
mask_ref = torch.randn(125)
dots_ref = torch.randn(100, 125)

# loop approach
mask = mask_ref.clone()
dots = dots_ref.clone()

for i in range(125):
    for j in range(100):
        dots[j, i] = dots[j, i] * mask[i]
    

final_add = torch.zeros(100)

for i in range(125):
    for j in range(100):
        final_add[j] = final_add[j] + dots[j, i];
    
# broadcasting
mask = mask_ref.clone()
dots = dots_ref.clone()

dots = dots * mask.unsqueeze(0)
result = dots.sum(1)

print((result - final_add).abs().max())
&gt; tensor(7.6294e-06)
</code></pre>
Multi-label classification on quantitative data

prantik: <p>Hi all,<br>
I am new to Deep Learning and I want to train a deep learning model preferably a CNN with data that looks like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/6123ed8dc7e8c2fd309c18ab676f477397b5020e.png" data-download-href="https://discuss.pytorch.org/uploads/default/6123ed8dc7e8c2fd309c18ab676f477397b5020e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/6123ed8dc7e8c2fd309c18ab676f477397b5020e.png" alt="image" data-base62-sha1="dRlhYnhqYdeCiYp7ZbdcJG0c4uq" width="690" height="95" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1000×138 6.68 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Right now I am trying to load this data to the Dataloader but getting errors. I am just wondering if it is possible to create some kind of one-hot encoding so that the data can be passed to an embedding layer and then convolution layers.</p>
<p>Thanks.</p>
Node, rank, local_rank

Ardeal: <p>Hi,</p>
<p>in torch.distributed:<br>
node means the machine(computer) id in the network.<br>
rank, means global_rank, means the process id in the network<br>
local_rank, means the process id in local machine(computer).</p>
<p>Is the my understanding above correct?</p>
<p>in my code,<br>
local_rank could be read from args.local_rank if torch.distributed is applied. otherwise local_rank doesn’t exist in args, right?</p>
<p>how could I get node id and rank in my code?</p>

H-Huang: <p>Hi, your understanding is correct. Here are the definitions we also refer to in documentation (<a href="https://pytorch.org/docs/master/elastic/run.html#:~:text=a%20future%20version.-,Definitions,-%3A" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed.run (Elastic Launch) — PyTorch master documentation</a>)</p>
<p>I assume you are using torch.distributed.launch which is why you are reading from args.local_rank. If you don’t use this launcher then the local_rank will not exist in args.</p>
<p>As of torch 1.9 we have a improved and updated launcher (<a href="https://pytorch.org/docs/master/elastic/run.html#module-torch.distributed.run" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed.run (Elastic Launch) — PyTorch master documentation</a>) in which you can read local_rank and rank from the environment variables. If you need the node id you’re code is on you can identify it through its <code>hostname</code>?</p>

Ardeal: <p><a class="mention" href="/u/h-huang">@H-Huang</a> ,<br>
Thank you!</p>
<p>Yes, I am using torch.distributed.launch. Thank you for your reply!</p>
How to correctly 'exclude' a module from the state dict of its parent module?

wentian_zhao: <p>There is a pre-trained model in a bigger model, and the parameters of the pre-trained model are always fixed during training. I want to exclude the parameters of the pre-trained model from the state_dict of the bigger model to save some disk space, since the pre-trained model could have a large number of parameters.</p>
<p>Here is a little example. The model to be trained is ‘MyModel’, and the fixed pre-trained model is the ‘resnet’ in the ‘Encoder’. I wonder how to exclude the parameters of the ‘resnet’ from the state_dict of the ‘Encoder’ model? I tried to override the ‘state_dict’ method and the ‘load_state_dict’ method of the ‘Encoder’, but the ‘load_state_dict’ method is never called when the state of ‘MyModel’ is being loaded.</p>
<pre><code class="lang-auto">import torch.nn as nn
from torchvision.models import resnet152

class Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.resnet = resnet152()   # pre-trained model to be excluded

    def forward(self, x):
        return self.resnet(x)

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = Encoder()
        self.decoder = ...          # some other network

    def forward(self, x):
        with torch.no_grad():
            feat = self.encoder(x)  # feature extraction only
        y = self.decoder(feat)      # do something else
        return y
</code></pre>

JuanFMontesinos: <p>So then why don’t you simply remove what you don’t want from the dictionary before saving?</p>
Is there a way to create a nn.Module without explicitly defining the class?

kyy: <p>Hi, I am trying to define a local nn.Module (inside a function) which cannot be easily expressed as <code>nn.Sequential</code>, and I am wondering if there is a way to define it without explicitly defining the class within the function?</p>
<p>Thanks!</p>

JuanFMontesinos: <p>Can you develop the question (maybe with an example). I find the lexicon you used a bit confusing. One rarely defines a class within a function. Do you mean instantiate? What’s the relationship between nn.Module and nn.Sequential? Sequential is just a type of module.</p>
Balance memory for DataParallel

dyukha: <p><code>nn.DataParallel(model)</code> efficiently parallelizes batches for me. However, when looking at memory, I see that device0 is almost full, while other devices have some memory to spare. Is there a way to balance memory load (e.g. split batches non-equally across devices)?</p>

wanchaol: <p>Thanks for posting question <a class="mention" href="/u/dyukha">@dyukha</a> Yeah DDP supports uneven inputs starting from pytorch 1.8.1, you can take a look at the details in the doc <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html?highlight=join#torch.nn.parallel.DistributedDataParallel.join" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 1.9.0 documentation</a></p>

wanchaol: <p><a class="mention" href="/u/dyukha">@dyukha</a> please also use DDP instead of Data Parallel, DDP is better to use even in a single process, and we are trying to deprecate Data Parallel in long term as well. see <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html" class="inline-onebox" rel="noopener nofollow ugc">DataParallel — PyTorch 1.9.0 documentation</a></p>

dyukha: <p>Thanks for the reply! I have to use DataParallel because of issues with DDP: <a href="https://discuss.pytorch.org/t/distributed-data-parallel-example-process-0-terminated-with-exit-code-1/127531/3" class="inline-onebox">Distributed Data Parallel example - "process 0 terminated with exit code 1" - #3 by dyukha</a></p>
Joint Training Dicussion

nicozhou: <p>Hi supposed I have two networks,<br>
unet → for segmentation<br>
resnet → for classification</p>
<p>How do I do joint training? is the code below correct?</p>
<pre><code class="lang-auto">
def train_models(inputs, targets, labels):
    unet.train()
    resnet.train()
    segment_output = unet(inputs)
    output = resnet(segment_output)
   
    optimizer1.zero_grad()
    optimizer2.zero_grad()
    loss1 = MSE(segmentation_output, labels)
    loss2 = BCE(output, target)
    loss1.backward()
    loss2.backward()

    optimizer1.step()
    optimizer2.step()
</code></pre>
<p>Or this one is the correct way?</p>
<pre><code class="lang-auto">optimizer = optim.Adam([
    {'params': unet.parameters()},
    {'params': resnet.parameters(), 'lr': 1e-3}
], lr=1e-4)

def train_models(inputs, targets, labels):
    unet.train()
    resnet.train()
    segment_output = unet(inputs)
    output = resnet(segment_output)
   
    optimizer.zero_grad()
    loss1 = MSE(segmentation_output, labels)
    loss2 = BCE(output, target)
    loss = loss1 + loss2
    loss.backward()
    optimizer.step()
</code></pre>
<p>If I happened to have weighted loss, is the 2nd approach better?</p>

JuanFMontesinos: <p>What do you mean by joint training?<br>
Are the networks totally independent?<br>
Does the sum of losses make sense in any way?<br>
If you just want to train two different models you can run two different scripts. No need to run them in the same script.</p>
<p>Both codes will “run”, the difference is the theory in behind depending on what you are doing.</p>

nicozhou: <p>i want to implement a segmentation based classification. The reason why i want to joint it together is to achieve end-to-end fashion.<br>
First, I will warm up unet<br>
Second, i will warm up resnet, while unet is frozen<br>
Third, i will train both network together with the sum of their loss.</p>
<p>Question is in the third phase, is summing loss and separating loss has the same meaning? I hope that unet learning from the classification network too.</p>
<p>By the way, you said both code will run, so i can say the two implementation above have different meaning?</p>

JuanFMontesinos: <p>Yeh then you can sum both losses at the same time, as it will jointly optimize the problem.<br>
Think that classifier’s gradients affect the segmentation too. If you see that classifier info degradates the segmentation then you can train them appart. But in that case it would be better to fully train the segmentator and then the classifier. I don’t find the point of iterate between both.</p>