Understanding tensor.backwards()

sad_robot: <p>Hello, so I don’t really get why it is that we need to give a grad tensor to tensor.backwards(). They say the grad should be the gradient of the tensor w.r.t itself but wouldn’t that just be a tensor of all ones?<br>
If not could you please give an example where the gradient wouldn’t be all ones?</p>
<p>I feel like I’m missing something basic here. I’ve never done a vector calc course to be fair.<br>
Also I’ve read <a href="https://discuss.pytorch.org/t/what-does-tensor-backward-do-mathematically/27953/4">this</a> and many similar posts, I still don’t understand it…</p>

ptrblck: <aside class="quote no-group" data-username="sad_robot" data-post="1" data-topic="85269">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sad_robot/48/25510_2.png" class="avatar"> sad_robot:</div>
<blockquote>
<p>but wouldn’t that just be a tensor of all ones?</p>
</blockquote>
</aside>
<p>In the default use case, yes. If your loss is a scalar value, you don’t need to pass the gradient and it will be set to 1 by default.<br>
However, if your loss is a tensor with more than a single value, you have to pass the gradient manually. If you want to use 1s, you could just use <code>loss.backward(torch.ones_like(loss))</code>.</p>

sad_robot: <p>Thanks for your reply.</p>
<p>I understand the API enough I don’t understand the underlying mathematical foundation of it. I’m saying that I thought the derivative of a tensor w.r.t itself would HAVE TO BE a tensor of all 1s.</p>
<p>I assumed this because when you take the gradient you treat all other variables as constants and then my understanding is that you would get a 1 each time for each variable then fill the tensor with that.</p>
<p>Another way of saying it is I don’t see how to the gradient could be anything but a tensor of all 1s.</p>

ptrblck: <p>If you are working on a use case, where you are calling the <code>backward</code> method on the loss and want to use the gradient as <code>dLoss/dLoss</code>, then you shouldn’t care about the gradient value and just use the default 1.<br>
However, other use cases don’t necessarily call <code>backward</code> only on the loss and might call it e.g. on an intermediate activation, so that limiting the gradient to strict ones would be an unnecessary restriction.</p>

sad_robot: <p>But correct me if I’m wrong wouldn’t that mean that if it were the derivative w.r.t. itself it would have to be all ones? And it would only be non-unit if it were the derivative w.r.t. to something earlier (like the loss or a previous layer)?</p>

ptrblck: <p>If you are trying to calculate the <code>dLoss/dLoss</code>, then it would be <code>torch.ones</code>, that’s correct.<br>
For other use cases it might be different. Recently such a use case was described <a href="https://discuss.pytorch.org/t/propagate-manually-computed-gradients/85884">here</a>.</p>

sad_robot: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="85269" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If you are trying to calculate the <code>dLoss/dLoss</code> , then it would be <code>torch.ones</code> , that’s correct.<br>
For other use cases it might be different. Recently such a use case was described <a href="https://discuss.pytorch.org/t/propagate-manually-computed-gradients/85884">here</a>.</p>
</blockquote>
</aside>
<p>Ah ok, I think that the docs for <a href="https://pytorch.org/docs/stable/autograd.html?highlight=tensor%20backward#torch.Tensor.backward" rel="noopener nofollow ugc">tensor.backwards()</a> confused me because they said that the arg: gradient := “Gradient w.r.t. the tensor.” (which makes it sound like gradient w.r.t. itself).</p>

saba: <p>Hi Ptrblck,</p>
<p>I am trying to update the discriminator by multiplying the coffecient to the all computed error. I used thsi code but it gives me the following error.I would apprecite your help.</p>
<pre><code class="lang-auto">        Param=0.2
        netD.zero_grad()
        images1=images1.unsqueeze(1)
        real_cpu = images1.to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        label=label.to(device)
        netD=netD.float()
        output = netD(real_cpu).view(-1)
        
        errD_real = criterion(output, label)
## ------- Creating fake --------------
        noise = torch.randn(b_size, nz,1, 1, 1, device=device)
        netG=netG.float()
        fake = netG(noise).to(device)
        
## -----------------FakeFinal ---------------
        FakeFinal=torch.mul(fake,Negpach1)

        label.fill_(fake_label)
        label=label.to(device)
        output = netD(FakeFinal.detach()).view(-1)
        
        errD_fake = criterion(output, label)        
        eraDWhole=(1-Param)*(errD_fake+errD_real)

        eraDWhole.backward()

</code></pre>
<p>Error:<br>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>

ptrblck: <p>Try to enable anomaly detection and check the operation which is shown in the stack trace.<br>
Based on this line of code check, if inplace operations were executed before this line and remove them.</p>

saba: <p>I understand that I should use this “torch.autograd.set_detect_anomaly(True)”  , where in the code I should add this?</p>

ptrblck: <p>Add it at the beginning of your script (after the imports, before any model is executed).</p>

saba: <p>it is solved by</p>
<pre><code class="lang-auto">ss=1-Param
errD_wh=(ss)*(criterion(output1,label1)+criterion(output, label))

</code></pre>