akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
`.cpu()` or `to.("cpu")` doesn't free gpu memory

chois_good: <h2>Code</h2>
<pre><code class="lang-auto">import torch

a = torch.randn(1000000, 1000, device=0)

#
# current gpu usage = 4383M
#

b = a.cpu()

#
# current gpu usage is still = 4383M
#
</code></pre>
<p>I’d like to free gpu memory(<code>a</code>) after convert the tensor to cpu.</p>
<p>What I’ve tried:</p>
<pre><code class="lang-auto">import gc
del a
gc.collect()
</code></pre>
<p>But it still occupies 4383M of gpu.</p>
<p>How can I do this?</p>

JuanFMontesinos: <p>I’m not an expert but you need to consider than when you call .cpu() you are making a copy in RAM, but It doesn’t imply you are removing the gpu version.</p>
<p>You can delete the GPU variable to achieve that. The memory will be available but it is not overwritten until new variables use that space. So in short memory manager is aware it can use those memory addresses, but doesn’t delete them as it would require time and can just overwrite when it is necessary.<br>
Anyway this is just what I understood from other people’s explanations.</p>

G.M: <ol>
<li>Variable <code>a</code> is still in use. Pytorch won’t free memories of variables that are still in use.</li>
<li>To free the memory, I guess you would have to first delete the variable. Then you can try <code>torch.cuda.empty_cache()</code>.</li>
</ol>
[JIT] Const ModuleList length

zetyquickly: <p>Hello,</p>
<p>I’ve found the issue that cannot understand.</p>
<p>In the TorchScript doc said that we can use <code>ModuleList</code> as long as they are constant. It is possible to iterate over it, but cannot get the length of <code>ModuleList</code></p>
<pre><code class="lang-auto">class SubModule(torch.nn.Module):
    def __init__(self):
        super(SubModule, self).__init__()
        self.weight = nn.Parameter(torch.randn(2))

    def forward(self, input):
        return self.weight + input

class MyModule(torch.nn.Module):
    __constants__ = ['mods']

    def __init__(self, x):
        super(MyModule, self).__init__()
        self.mods = torch.nn.ModuleList([SubModule() for i in range(10)])

    def forward(self, v):
        print(len(self.mods))
        return v


m = torch.jit.script(MyModule(3))
m(torch.rand(3))
</code></pre>
<p>Gives an error:</p>
<pre><code class="lang-auto">~/anaconda2/envs/pytorch-nightly/lib/python3.7/site-packages/torch/jit/_recursive.py in create_methods_from_stubs(concrete_type, stubs)
    278     rcbs = [m.resolution_callback for m in stubs]
    279     defaults = [get_default_args(m.original_method) for m in stubs]
--&gt; 280     concrete_type._create_methods(defs, rcbs, defaults)
    281 
    282 def create_script_module_for_tracing(nn_module, stubs):

RuntimeError: 
Tried to access nonexistent attribute or method '__len__' of type '__torch__.torch.jit.___torch_mangle_65._ConstModuleList'. Did you forget to initialize an attribute in __init__()?:
at &lt;ipython-input-26-4c61e0830c6c&gt;:17:14
    def forward(self, v):
        print(len(self.mods))
              ~~~ &lt;--- HERE
        return v
</code></pre>

BobChen: <p>I have meet the same problem:( Can anyone give some suggestions?</p>

G.M: <p>Just use another variable to store the length.</p>
CUDA out of memory while running UniLM

richylyq: <p>I am trying to run the code on UniLM on <a href="https://github.com/microsoft/unilm" rel="nofollow noopener">https://github.com/microsoft/unilm</a> for fine-tuning, under Abstractive Summarization - [Gigaword] (10K)<br>
but as I was running the code, i encountered this error</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 10.92 GiB total capacity; 10.01 GiB already allocated; 67.56 MiB free; 10.29 GiB reserved in total by PyTorch)
</code></pre>
<p>Can anyone assist me in explaining what is going on and even better, solve it with me.</p>
<p>Much Appreciated!</p>

ptrblck: <p>Your model and training might use too much device memory, which will yield this error.<br>
Try to reduce the batch size to lower the memory footprint.<br>
If that’s not possible, you could try to use <a href="https://pytorch.org/docs/stable/checkpoint.html">torch.utils.checkpoint</a> to trade compute for memory.</p>
<p>Also, check the available memory of your GPU via <code>nvidia-smi</code> and, if possible, stop other processes which might take some memory.</p>
Compiling from Source with Pre-Installed ONNX

forwardmeasure: <p>Hello all,<br>
I was wondering if it would be possible to compile LibTorch against a pre-installed ONNX library. I am currently able to compile LibTorch from scratch, against a pre-installed version of Protobuf, but didn’t see an obvious way to do the same with ONNX.</p>
<p>This is a requirement for me, as I have some applications that use ONNX and do not use LibTorch. I don’t want to have multiple, possibly incompatible ONNX libraries installed.</p>
<p>Thanks in advance,<br>
Prashanth</p>
Using Torch.Meshgrid with missing data

HappyPants: <p>I am working with a directed acyclic graph that models the <code>nx</code> random variables over <code>nt</code> time steps. I built a mesh grid representing positional coordinates in this graph to retrieve the value of the variable x<sub>t,i</sub> where <em>t</em> is the time coordinate and <em>i</em> is one of the <code>nx</code> random variables.</p>
<p>How can I obtain a list of key-value pair coordinates where the key is a pair of time (<code>t_idx</code>) and positional (<code>x_idx</code>) coordinate and the value is the next <strong>imediate</strong> coordinate where the variable is <strong>observed</strong>? It is possible to compute a boolean mask to know where the data is observed.</p>
<pre><code class="lang-python"># time and positional coordinate
t_idx , x_idx = torch.meshgrid(torch.arange(arg.nt, out=torch.LongTensor()),
                               torch.arange(arg.nx, out=torch.LongTensor())
                              )

# time and positional coordinate with observed data 
mask_data = (~torch.isnan(train_data)).squeeze(-1).nonzero()

</code></pre>

HappyPants: <p>I’ve found a way to do build a list that permits retrieving this key-pair value but this may not be the best solution…</p>
<pre><code class="lang-auto">coordinate = (torch.isnan(train_data) != True).squeeze(-1).nonzero()
# tensor([[0, 0], [0, 1],[0, 2],[4, 1], [5, 2],[6, 0]])
idx_pair = []
for id_va in set(coordinate[:, 1]):
    all_tidx = torch.where((coordinate == id_va))[0]
    for x, y in zip(all_tidx, all_tidx[1:]):
        idx_pair.append(torch.stack((coordinate[x][0], coordinate[y][0], coordinate[y][1])))
idx_pair = torch.stack(idx_pair)
# output = tensor([[0, 3, 0],
#        [1, 4, 1],
#        [2, 5, 2],
#        [5, 6, 2]])
</code></pre>
Method for better utilization of GPU memory for Kmeans clustering

Sabra_Ossen: <p>I have implemented K means clustering algorithm in GPU using PyTorch. I have a Tesla K80 GPU (11GB memory).</p>
<p>I have used the following methods to be able to increase the number of data points and clusters.</p>
<ul>
<li>
<p>Explicitly delete variables initialized once they are out of scope, this releases GPU memory that has no use.</p>
</li>
<li>
<p>Used half precision floating point.</p>
</li>
</ul>
<p>Still. I was only able to classify a maximum of 3 million data points into 500 clusters. You will be able to find the code below.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/SabraO/c59edbaeb9141d88db3f1e3a0e4d3ccb" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/SabraO/c59edbaeb9141d88db3f1e3a0e4d3ccb" target="_blank" rel="nofollow noopener">https://gist.github.com/SabraO/c59edbaeb9141d88db3f1e3a0e4d3ccb</a></h4>
<h5>kmeans.py</h5>
<pre><code class="Python">import torch
import numpy as np
from datetime import datetime

n_clusters = 500 # Max Possible 485 (32-bit)
n_data_samples = 2950000 # Max possible 2M (32-bit)
n_iterations = 2

with torch.cuda.device(0):
    # Generate sample values</code></pre>
This file has been truncated. <a href="https://gist.github.com/SabraO/c59edbaeb9141d88db3f1e3a0e4d3ccb" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Is there any better method supported by PyTorch to utilize the GPU memory such that the GPU memory is used for mostly calculation while the data is being streamed from CPU to GPU. This way while the data (matrices) is being transferred to the GPU, calculation on some other parts are done. This way the whole dataset doesn’t need to completely be present in the GPU. Then I will be able to cluster more data points into more clusters. Please let me know if it is possible.</p>
<p>Any help on this issue is appreciated.</p>

subhadarshi: <p>check out this <a href="https://github.com/subhadarship/kmeans_pytorch" rel="nofollow noopener">github repo</a>.</p>
<p>find documentation <a href="https://subhadarship.github.io/kmeans_pytorch" rel="nofollow noopener">here</a>.</p>
<p>On a GPU (in google colab), clustering 10 million 2D samples into 3 clusters takes about 25 seconds.</p>
Pooling over a stack of tensors

ed-fish: <p>If I have a stack of tensors of shape <code>(20, 2048, 1, 1) </code> How can I perform adaptive pooling over the stack to return a tensor of shape <code>(1, 2048, 1, 1)</code> Thank you.</p>

JuanFMontesinos: <p>just permute the tensor to get a (1,2048,1,20) or whatever shape which fits and pool.</p>

ed-fish: <p>Thanks - here is the working example</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

t = torch.rand(20, 2048, 1, 1)
print(t.shape)
t = t.permute(2, 1, 0, 3)
print(t.shape)
t = F.adaptive_avg_pool2d(t, 1)
print(t.shape)
t = t.view(1, 2048)
print(t.shape)
</code></pre>
<p>returns</p>
<pre><code class="lang-auto">torch.Size([20, 2048, 1, 1])
torch.Size([1, 2048, 20, 1])
torch.Size([1, 2048, 1, 1])
torch.Size([1, 2048])
</code></pre>
SM3 for PyTorch (Memory-efficient Adapative Optimizer)

enealor: <p>Hello all,</p>
<p>I recently learned about the <a href="https://arxiv.org/abs/1901.11150" rel="nofollow noopener">SM3 optimization algorithm</a>. I implemented it in PyTorch to better understand the paper, and I am sharing the code here. It’s available at <a href="https://github.com/Enealor/PyTorch-SM3" rel="nofollow noopener">PyTorch-SM3</a>. The implementation itself includes the features of the TensorFlow version (support for dense and sparse tensors) as well as a feature they mention but do not include in their final version (exponential moving averages). The authors designed the algorithm to be a memory-efficient alternative to Adam and Adagrad for large NLP architectures such as Transformer-Big and BERT.</p>
<p>I have tested my version numerically and tried it for a few simple networks to verify that nothing arose during training. However, I am not working with models as large as the optimizer is intended for. If you do work with models that large and notice any issues, please let me know.</p>

Yaroslav_Bulatov: <p>Looks nice! I’m curious how performance compares against the AdaFactor optimizer – <a href="https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py" rel="nofollow noopener">https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py</a> . AdaFactor was used to train the 1.5B GPT-2 model. SM3 seems to give extra extra flexibility in choosing factors, whereas AdaFactor is limited to output dimension,input dimension split.</p>

enealor: <p>The authors compare SM3 to Adafactor in their article. I’ll quote the relevant paragraph as I don’t think I can say it better.</p>
<blockquote>
<p>Comparison with Adafactor: Adafactor is a very effective method for space-efficient adaptive optimization. SM3 and Adafactor differ in a number of important ways. First, Adafactor is only defined for matrix-shaped parameters while SM3 applies to tensors of arbitrary dimensions, and even more generally, to any predefined cover of the parameters. Second, Adafactor is in essence a fixed learning-rate algorithm, being a memory-constrained variation of Adam, and often requires a manually devised learning-rate schedule to ensure convergence. In contrast, SM3 adapts its learning rates in an adaptive, data-driven manner similar to Adagrad. Finally, SM3 comes with rigorous convergence guarantees in stochastic convex optimization settings.</p>
</blockquote>
<p>And yeah, it seems that SM3 offers amazing flexibility in choice of cover. They end up choosing similar to Adafactor despite this flexibility as their experiments showed that rows/columns had correlated activations.</p>
What could cause loss jumping up for CNN + RPN for 1 epoch?

martinr: <p>I am training a CNN with RPN at the end, rather deep, but straightforward model. I am training with focal loss and saw such a peak on my training loss. How could I explain it? I mean, it happened around 80th epoch when the model was already rather stable. I am just conceptually wondering what could possible push the loss up for 1 epoch and then back to normal again? Haven’t seen such behaviour before, only much smaller jumps. The loss is an average for thousands of training examples, not just for one minibatch.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cc5dcf1d622d7c281288fa6fa7d4d883f0f9229.png" alt="Opera Snapshot_2020-02-04_220933_localhost" data-base62-sha1="8FCzcXP4e2nHVjPt8rOZ2jeG1oB" width="318" height="287"></p>
<p>The peak is actually very high (see below), but capped by 2 on the plot:</p>
<pre><code class="lang-auto">Epoch 84 mean loss 0.05488062464491334 LR: 0.001
Epoch 85 mean loss 0.05492022068288196 LR: 0.001
Epoch 86 mean loss 198953985.74916962 LR: 0.001
Epoch 87 mean loss 0.05357671019970312 LR: 0.001
Epoch 88 mean loss 0.05381619437844289 LR: 0.001
</code></pre>
<p>Could it be that one of the minibatches went to INF (buffer overflow?) and it is the result of averaging this over batch_count? Any thoughts or ideas appreciated.</p>
IndexError: Target 1 is out of bounds (NLLoss)

kamathhrishi: <pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.lstm=nn.LSTM(10,120)
        self.output=nn.Linear(120,10)

    def forward(self, x):
        x,h = self.lstm(x)
        x = F.softmax(self.output(x))
        return x,h
    
net=Net()

inputs=[]
outputs=[]

for i in range(0,9):
    
    one=torch.zeros([10])
    one[i]=1
    inputs.append(one)
    
    one=torch.zeros(10)
    one[i+1]=1
    outputs.append(one)
    
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
    
for j in range(0,10):    

     optimizer.zero_grad()
     forward,hidden=net(torch.stack(inputs).view((9,1,10)))
     print(forward.shape)
     #print(outputs)
     loss = criterion(forward,torch.stack(outputs).long())
     print(loss)
</code></pre>
<p>Complete Error trace</p>
<pre><code class="lang-auto">
  File "&lt;ipython-input-70-ffecb9d328f7&gt;", line 1, in &lt;module&gt;
    runfile('/Users/hrishikesh/Hrishikesh/Projects/Major Project/ASR/ann.py', wdir='/Users/hrishikesh/Hrishikesh/Projects/Major Project/ASR')

  File "/Users/hrishikesh/opt/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "/Users/hrishikesh/opt/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/hrishikesh/Hrishikesh/Projects/Major Project/ASR/ann.py", line 42, in &lt;module&gt;
    loss = criterion(forward,torch.stack(outputs).long())

  File "/Users/hrishikesh/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)

  File "/Users/hrishikesh/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)

  File "/Users/hrishikesh/opt/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File "/Users/hrishikesh/opt/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1863, in nll_loss
    input, target, weight, reduction_enum, ignore_index)

IndexError: Target 1 is out of bounds.
</code></pre>

h3li05: <p>CrossEntropyLoss() requires a 1-D tensor of class indices of the target (outputs in your case) but not one-hot vectors of the target.<br>
The shape of the target should be [mini_batch_size], but in your case it happens to be [mini_batch_size,num_classes].<br>
That is why you are getting  <code>IndexError: Target 1 is out of bounds.</code><br>
Go through the official documentation for more details(<a href="https://pytorch.org/docs/master/nn.html#crossentropyloss" rel="nofollow noopener">link</a>)</p>
Detach variable from computed dynamic graph

bananacode: <p>Consider a simple GAN model as shown here <a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/gan/gan.py" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/gan/gan.py</a></p>
<p>L144 and L157 have a common computation: <code>discriminator(gen_imgs)</code>. They only differ in that one detaches the generator from the computational graph. However, the forward pass of the discriminator is still computed twice.</p>
<p>Is it possible to compute <code>discriminator(gen_imgs)</code> only once, but detach <code>gen_imgs</code> from the graph after computation?</p>
<p>E.g something like</p>
<pre><code class="lang-auto">d_out = discriminator(gen_imgs)
g_loss = adversarial_loss(d_out, valid)
(...)
fake_loss = adversarial_loss(d_out.detach(gen_imgs), fake)
</code></pre>

albanD: <p>Why not just use <code>d_out.detach()</code> for the second one?  It is a Tensor that has no gradient history.</p>

bananacode: <p>If I call <code>d_out.detach()</code>, then the gradient wouldn’t flow back to the discriminator right?</p>

albanD: <p>You’re right it won’t flow back to the discriminator.</p>

bananacode: <p>So there isn’t a way to achieve what I asked?</p>

albanD: <p>Isn’t that what you asked?<br>
You want it to flow back to discriminator but not above?</p>

bananacode: <p>Yes, I want it to flow back to the discriminator. What you proposed will prevent the gradient to flow back to the discriminator.</p>

albanD: <p>Ho ok. So no this is not possible to do.You will need to forward through the <code>discriminator</code> again.</p>

bananacode: <p>I see. That’s a pity. Thank you for your responses!</p>
Integrated gradients using with pack_padded_sequence returns error

Mostafa_Alishahi: <p>Hi all,</p>
<p>I am using integrated gradient (IG) package from Captum package, which I apply one LSTM on varying length sequences and then I try to get IG from the trained model using the following line of code:</p>
<blockquote>
<p>attr, delta = ig.attribute((data, seq_lengths), target=1, return_convergence_delta=True)</p>
</blockquote>
<p>but I am getting the following error:</p>
<blockquote>
<p>RuntimeError: <code>lengths</code> array must be sorted in decreasing order when <code>enforce_sorted</code> is True. You can pass <code>enforce_sorted=False</code> to pack_padded_sequence and/or pack_sequence to sidestep this requirement if you do not need ONNX exportability.</p>
</blockquote>
<p>however, I have sorted the lengths of the array in each batch in decreasing order.<br>
please note that If I use this IG without using pack_padded_sequence it works perfectly.</p>
<p>regarding the previous error, I set enforce_sorted=False in pack_padded_sequence but I am getting another error:</p>
<blockquote>
<p>RuntimeError: Length of all samples has to be greater than 0, but found an element in ‘lengths’ that is &lt;= 0</p>
</blockquote>
<p>Here is the length of all the samples which none of them are less than zero:</p>
<blockquote>
<p>tensor([23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,<br>
23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,<br>
23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,<br>
23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,<br>
23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,<br>
23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,<br>
22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 21, 21, 21, 20,<br>
14, 10])</p>
</blockquote>
<p>any help would be much appreciated.</p>

Narine_Kokhlikyan: <p>Thank you for the question <a class="mention" href="/u/mostafa_alishahi">@Mostafa_Alishahi</a>!<br>
Replied on github: <a href="https://github.com/pytorch/captum/issues/275" rel="nofollow noopener">https://github.com/pytorch/captum/issues/275</a></p>
Split tensor based on segments

3dextended: <p>Hi!</p>
<p>I am looking for the most efficient way to split a tensor into multiple different tensors based on a 1-D index tensor.</p>
<p>Here is the full problem description:</p>
<p>I have a point cloud tensor with dimension (batchsize, 3, 2048) and an index tensor of size (batchsize, 2048) (LongTensor). This index array contains integer values from 0 to 3 (inclusive). I want to split the point cloud based on those indices such that I end up with 4 different tensors of shape (batchsize, 3, N_i) where the sum over all 4 different N_i equals 2048.</p>
<p>I’ve looked into torch.index_select but I can’t figure out, how I could use this function to my advantage.</p>
<p>Your help would be much appreciated!</p>

ptrblck: <p>I’m not sure to understand the use case completely.<br>
Could you post some examples using smaller tensor shapes, e.g. <code>[1, 3, 10]</code>, and demonstrate how you would like to split this tensor using the indices?</p>

phisku: <p>Coming here having a similar question. If I understood the OP correctly, given a tensor A of<br>
(1, 3, 5) and a corresponding tensor B of shape (5,) where for example</p>
<p>A=<br>
[[[3, 0, 0, 1, 8, 9, 5, 4, 1, 6],<br>
[0, 0, 0, 8, 0, 7, 1, 0, 0, 8],<br>
[1, 0, 2, 5, 3, 6, 3, 2, 4, 7]]]</p>
<p>and B=[0, 0, 1, 1, 0, 2, 2, 3, 3, 3]</p>
<p>the goal is to get 4 tensors where the first contains all elements of A whose corresponding B is 0.<br>
Thus<br>
[[[3, 0, 8],<br>
[0, 0, 0],<br>
[1, 0, 3]]]</p>
<p>the second one would be<br>
[[[0, 1],<br>
[0, 8],<br>
[2, 5]]]</p>
<p>and so forth.</p>
<p>Is there an easy was of doing it. And to answer my actual question, is there also a way to generalize it to arbitrary index values and then split in a similar fashion such that each resulting tensor contains values with the same corresponding index?</p>
nn.CrossEntropyLoss's Gradient is Zeroed out by My customized loss

Dichoto: <p>Hi, Pytorcher:</p>
<p>I used nn.Module to write my own loss, which is combined with nn.CrossEntropyLoss.</p>
<p>class MyLoss(nn.Module):</p>
<pre><code>def __init__(self):
     XXXX

def forward(self, input):
    """
    Args:
        Input: tensor from a network
   """
   loss = Compute(input)

   """
   Here has some numerical issue, loss should be greater than 0, but may get loss &lt;0
   """
    return (torch.sqrt(loss) if loss.item()&gt;0 else 0) 
</code></pre>
<p>mymodel = Net()<br>
myloss = MyLoss()<br>
ce = CrossEntropyLoss()<br>
total_loss  = myloss + ce</p>
<p>When MyLoss returns 0. The optimizer should backpropagate on nn.CrossEntropyLoss. But it turns out that the gradient is zero. The problem might be a constant return. But cross-entropy should have gradient.</p>
<p>Does anyone come across this type of problem?</p>
<p>Thanks.</p>

Lornatang: <p>If you don’t initialize the parameters in the network, you’re likely to have a gradient problem</p>

ptrblck: <p>Your code snippet should work, even if you return a zero in your custom loss function as seen here:</p>
<pre><code class="lang-python">output = torch.randn(10, 10, requires_grad=True)
target = torch.randint(0, 10, (10,))

criterion = nn.CrossEntropyLoss()

loss = 0 + criterion(output, target)
loss.backward()
print(output.grad)
</code></pre>
<p>Could you check the value of <code>ce</code> before calling <code>total_loss.backward()</code>?</p>

Dichoto: <p>Thanks for the reply. I am double-checking the numerical issue. Currently, it seems like tensor broadcasting introduces a non-negligible error, an entry is expected to be zero, but we got something like 7E-7.</p>
<p>The loss should be greater than or equal to 0, but sometimes got a very small negative value. I haven’t figured out the reason.</p>
Max of a tensor and a scalar

vadimkantorov: <p>Is there a PyTorch analogue of Lua’s <code>torch.cmax(tensor, value)</code>?</p>
<p>I checked the docs <code>torch.max</code> seems to want another tensor as an argument and doesn’t cut it.</p>
<p>There are workarounds, but an easy direct way of doing it would be more concise and probably efficient.</p>
<p>Thanks!</p>

elanmart: <p>One way to accomplish this would be with the <code>torch.max()</code> function you mentioned:</p>
<pre><code class="lang-python">X = th.FloatTensor([
    [-1, 2],
    [2, -1]
])
scalar = th.FloatTensor([1])

th.max(X, scalar.expand_as(X))
</code></pre>
<p>gives you</p>
<pre><code class="lang-python"> 1  2
 2  1
[torch.FloatTensor of size 2x2]
</code></pre>
<p>Please keep in mind that I’m playing with PyTorch for few days only, so that may not be the optimal solution.</p>
<p><strong>EDIT</strong> I remember one of the devs mentioned they’re considering adding a <code>autograd.Scalar</code>, I guess it would make things nicer in this case.</p>

vadimkantorov: <p>Yep, that’s workaround I’m currently using, but it’s a little too verbose IMO compared to <code>cmax</code>.</p>

smth: <pre><code class="lang-python">torch.clamp(tensor, max=value) # cmin
torch.clamp(tensor, min=value) # cmax
</code></pre>
<p>Edited to reflect what vadim said below</p>

vadimkantorov: <p>Seems it’s the other way around:</p>
<p><code>torch.clamp(tensor, min=value)</code> is <code>cmax</code> and<br>
<code>torch.clamp(tensor, max=value)</code> is <code>cmin</code>.</p>
<p>It works but is a little confusing at first.</p>

fmassa: <p><code>cmax</code>/<code>cmin</code> was removed to simplify the API, after some comments we received.<br>
<a href="https://github.com/pytorch/pytorch/pull/455" class="onebox" target="_blank">https://github.com/pytorch/pytorch/pull/455</a></p>

HectorAnadon: <p>Maybe someone is looking for this. Is there a way to do this without that much code?</p>
<pre><code class="lang-auto">def min_(value1, value2):
    if is_number(value1) and isinstance(value2, torch.Tensor):
        return torch.clamp(value2, max=value1)
    elif is_number(value2) and isinstance(value1, torch.Tensor):
        return torch.clamp(value1, max=value2)
    elif isinstance(value1, torch.Tensor) and isinstance(value2, torch.Tensor):
        return torch.min(value1, value2)


def max_(value1, value2):
    if is_number(value1) and isinstance(value2, torch.Tensor):
        return torch.clamp(value2, min=value1)
    elif is_number(value2) and isinstance(value1, torch.Tensor):
        return torch.clamp(value1, min=value2)
    elif isinstance(value1, torch.Tensor) and isinstance(value2, torch.Tensor):
        return torch.max(value1, value2)

min_(2, torch.tensor(3))
min_(torch.tensor(3), 2)
min_(torch.tensor(3), torch.tensor(2))</code></pre>
Tracking gradient

nzinc: <p>How can I get gradients from backward() or optimizer to track those by comet.ml?</p>
<pre><code class="lang-auto">    generator_loss.backward()
    optG.step()
    experiment.log_metrics({'Generator loss': generator_loss.item(),
                            'Critic loss': critic_loss.item()},
                            step = i)
</code></pre>
<p>Like this for example</p>

tux: <p>After applying <code>generator_loss.backward()</code> you can access the gradient with <code>input.grad</code>. See gradient section of <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py</a> for instance.</p>
Logical Indexing Broadcasing

pyscorcher: <p>It seems logical indexing does not support broadcasting, is this correct?</p>
<p>I’ve tried following: Let’s say we have a batch of masks and a batch of images, and we want to extract the masked pixels:</p>
<pre><code class="lang-auto">mask = torch.randn((10, 1, 32, 32)) &gt; 0
img = torch.rand((10, 3, 32, 32))
img[mask] # error
</code></pre>
<p>(We can of course use workarounds using a product of mask and image where broadcasting seems to work.)</p>
Float vs. Int in torchvision.utils.save_image

CCL: <p>I predict a binary segmentation mask using an array with values 1 and 0, multiply this by 255, and try to save the resulting array as an image. However, I’m getting the following error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “test.py”, line 71, in <br>
torchvision.utils.save_image(predicted, path_ + idx[0])<br>
File “C:\Users\CCL\Anaconda3\lib\site-packages\torchvision\utils.py”, line 107, in save_image<br>
ndarr = grid.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to(‘cpu’, torch.uint8).numpy()      RuntimeError: result type Float can’t be cast to the desired output type Int</p>
</blockquote>
<p>Here is my code:</p>
<pre><code class="lang-auto">mask_pred = net(img)
predicted = torch.sigmoid(mask_pred) &gt; 0.5
predicted = torch.tensor(predicted, dtype=torch.int32)*255
torchvision.utils.save_image(predicted, path_ + idx[0])
</code></pre>
<p>I’m using torch version 1.4.0 and torchvision version 0.5.0. The above code works on torch v1.1.0 and torchvision v0.3.0.</p>

ptrblck: <p>Internally the output array will be created via:</p>
<pre><code class="lang-python">ndarr = grid.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to('cpu', torch.uint8).numpy()
</code></pre>
<p>so you don’t need to multiply your predictions with <code>255</code> and can pass it as a float tensor.<br>
The error is thrown at the <code>.add_(0.5)</code> step.</p>
<p>The documentation doesn’t mention any required ranges, so you might open a GitHub issue and link to this topic.</p>

CCL: <p>Thank you! Directly passing in <code>mask_pred</code> worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Will open issue.</p>

pmeier: <p>GitHub issue for reference: <a href="https://github.com/pytorch/vision/issues/1847" rel="nofollow noopener">https://github.com/pytorch/vision/issues/1847</a></p>
How to use torch.cat and preserve the grad history unchanged?

Deeply: <p>Simply, I have a few variables, say <code>x1, x2, x3</code>, with <code>requires_grad</code> set to <code>True</code>. Using torch.Tensor resulted in removing all the <code>grad</code> history,</p>
<pre><code class="lang-auto">ipdb&gt; x1
tensor(0.6684, device='cuda:0', grad_fn=&lt;L1LossBackward&gt;)

ipdb&gt; x2
tensor(0.7662, device='cuda:0', grad_fn=&lt;L1LossBackward&gt;)

ipdb&gt; torch.Tensor((x1, x2))
tensor([0.6684, 0.7662])
</code></pre>
<p>I need to concatenate the variables into one tensor and preserve the grad history, so that I can use <code>backward</code> pass later.</p>
<p>I have tried  using torch.Tensor( (x1, x2, x3) ) but all the grad history has been lost.<br>
Using <code>torch.cat((x1,x2,x3))</code> did not work too, throwing the error message “RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated” .</p>
<p>One way around this, is to define the tensor<br>
<code>z = torch.ones(3, dtype=torch.float32, requires_grad=True).to(device)</code></p>
<p>then, multiply <code>z</code> by <code>x</code></p>
<pre><code class="lang-auto">z[0] = z[0]*x1
z[1] = z[1]*x2
z[2] = z[2]*x3
</code></pre>
<p>Any, ideas, alternatives?</p>

ptrblck: <p>Could you try to use <code>torch.stack</code> instead?</p>

AntoineC: <p>What about explicitly setting the flag?</p>
<pre><code class="lang-python">x = torch.tensor(1., requires_grad=True)
torch.tensor([x**3, 2*x], requires_grad=True)
</code></pre>
<p><strong>Edit</strong>: sorry, that won’t work, as it will create new variables (one in the position of <code>x***3</code> and one in the position of <code>2*x</code>) with their own grad history.</p>
Object has no attribute 'data'

S_M: <p>Hi, I am trying to get access to the gradient of my image by using this line<br>
data_grad = data[1].grad.data<br>
then I get some error:(</p>
<p>how shall I handle this error?<br>
thanks</p>
<p>for batch_idx, (data, target) in enumerate(train_loader):<br>
target = target if len(target) &gt; 0 else None<br>
if not type(data) in (tuple, list):<br>
data = (data,)<br>
if device:<br>
data = tuple(d.device() for d in data)<br>
if target is not None:<br>
target = target.device()</p>
<pre><code>    data[0],data[1],data[2].requires_grad_(True)
    outputs = model(*data)
    
    if type(outputs) not in (tuple, list):
        outputs = (outputs,)
        
    loss_inputs = outputs
    if target is not None:
        target = (target,)
        loss_inputs += target

    loss_outputs = loss_fn(*loss_inputs)
    loss = loss_outputs[0] if type(loss_outputs) in (tuple, list) else loss_outputs
    optimizer.zero_grad()
    loss.backward()
    data_grad = data[1].grad.data#here I got some error</code></pre>

G.M: <p><code>data[1]</code> doesn’t require grad, so its <code>.grad</code> is <code>None</code>.</p>

S_M: <p>What do you mean? Shall I drop this line out?<br>
data[0],data[1],data[2].requires_grad_(True)</p>

G.M: <p>try running this code:</p>
<pre><code class="lang-auto">a=[]
b=[]
c=[]
a,b,c.append(1)
print(a,b,c)
</code></pre>
<p>you’ll see what’s wrong :).</p>
Question for (TRANSFER LEARNING FOR COMPUTER VISION TUTORIAL)

SrJ: <p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<pre><code class="lang-auto">def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    *inp = np.clip(inp, 0, 1)*
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated


# Get a batch of training data
inputs, classes = next(iter(dataloaders['train']))

# Make a grid from batch
out = torchvision.utils.make_grid(inputs)

imshow(out, title=[class_names[x] for x in classes])
</code></pre>
<p>Why the bolded line(inp = np.clip(inp,0,1)) is needed??<br>
I hae visualized by commenting out that line and it still worked the same way. Then why do i need to clip??</p>

mailcorahul: <p><code>np.clip</code> is to make sure the tensor values lie in the range <code>[0, 1]</code>. You need the input images normalized and be in a specific range(either <code>[0, 1]</code> or <code>[0, 255]</code>) which is what <code>np.clip</code> ensures.</p>
<p>Visualizing such unbounded tensors may not be a problem though, but they need to be in a certain bound in order to train a network.</p>

SrJ: <p>This function is for visualize only. So, i guess i need not normalize the input .</p>
<p>But Why plt.imshow() i s showing a normalized image correctly??</p>

mailcorahul: <aside class="quote no-group" data-username="SrJ" data-post="3" data-topic="68418">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e9c0ed/48.png" class="avatar"> SrJ:</div>
<blockquote>
<p>This function is for visualize only. So, i guess i need not normalize the input .</p>
</blockquote>
</aside>
<p>Right.</p>
<aside class="quote no-group" data-username="SrJ" data-post="3" data-topic="68418">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e9c0ed/48.png" class="avatar"> SrJ:</div>
<blockquote>
<p>But Why plt.imshow() i s showing a normalized image correctly??</p>
</blockquote>
</aside>
<p>plt.imshow supports both 0-1 and 0-255 arrays</p>
Autograd in Variational Autoencoder and Classification

Giuseppe_Russp: <p>`Hi guys, sorry but I have a question about how the gradient is propagated in the following architecture.</p>
<p>I have a VAE where both the encoder and the decoder are GRUs. In particular the encoder takes as input the embedding of a sentence (x), while the decoder takes as input the sentences (x) and the hot one encoding of some attributes (y) of the sentence (e.g, the verb tense).</p>
<p>I want to connect a classifier (e.g, fully connected) which takes as input the hidden state of the decoder and outputs some y_hat probabilities, to actually check if the reconstruction contains information about the original attributes of x.<br>
Now when my classifier is not predicting correctly I would like that the encoder/decoder vae tries to learn hidden representations which contains information about that attribute.</p>
<p>So my question is when in my training loop I suppose I  have something like the following code, does the gradient go back until the encoder or not? If someone can explain me the path of the gradient I would really appreciated it. Thanks again</p>
<pre><code class="lang-auto">vae = SentenceVAE()
classifier = Classifier()
model_opt = torch.optim.Adam(vae.parameters(), lr=0.001) 


for epoch in range(epochs):
        for iteration, batch in enumerate(data_loader):
            sent_recon, hidden_decoder  = vae(batch['input'], batch['length'], batch['label'])
            attributes_probability = classifier(hidden_decoder)

#computing the classifier loss using cross_entropy and the vae_loss

            loss = vae_loss - (disc_weight*classifier_loss)

            if split == 'train':
                model_opt.zero_grad()
                oss.backward()
                model_opt.step()

</code></pre>