Loss.backward() raises error 'grad can be implicitly created only for scalar outputs'

ajeetk: <p>Hi,</p>
<p><code>loss.backward()</code> do not go through while training and throws an error when on multiple GPUs using torch.nn.DataParallel</p>
<p><code>grad can be implicitly created only for scalar outputs</code></p>
<p>But, the same thing trains fine when I give only <code>deviced_ids=[0]</code> to torch.nn.DataParallel.<br>
Is there something I am missing here?</p>
<p>Addendum:</p>
<p>While running on two gpus, the loss function returns a vector of 2 loss values. If I run the backward only on the first element of the vector it goes fine.</p>
<p>How can I make the backward function work with vector containing two or more loss values?</p>
<p>Thanks.</p>

albanD: <p>when you do <code>loss.backward()</code>, it is a shortcut for <code>loss.backward(torch.Tensor([1]))</code>. This in only valid if <code>loss</code> is a tensor containing a single element.<br>
<code>DataParallel</code> returns to you the partial loss that was computed on each gpu, so you usually want to do <code>loss.backward(torch.Tensor([1, 1]))</code> or <code>loss.sum().backward()</code>. Both will have the exact same behaviour.</p>

kmyfoer: <p>If I want to get average loss of each sample, and every single element in <code>loss</code> has been averaged on batch,  should I use <code>loss.mean().backward()</code>?</p>

albanD: <p>If all the batch are the same size, it will work.</p>

kmyfoer: <p>I got it. Thank you   : )</p>

gradwolf: <p>When I try loss.mean.backward() or loss.sum.backward() I am getting this warning? UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. warnings.warn('Was asked to gather along dimension 0, but all ’</p>
<p>How do I suppress this one?</p>

ptrblck: <p>Let’s continue the discussion in the topic you’ve created and have a look at <a class="mention" href="/u/simonw">@SimonW</a>’s <a href="https://discuss.pytorch.org/t/how-to-suppress-this-warning/50301/2">answer</a>.</p>

mxn.wls: <p>Why is backward limited in this way? Is it an oversight or some important reason?</p>
<p>Thanks!</p>

albanD: <p>The gradient is computed as a vector jacobian product. So the size of the vector has to match the size of a dimension jacobian (which is the size of the output).</p>

mxn.wls: <p>Sure, the number of grads needs to equal the number of variables.</p>
<p>What I meant was it seems weird that the “backward” function is defined as on 1 variable unless otherwise stated, even though it is implemented on a vector with &gt; 1 variables.</p>
<p>e.g. if I have vec, a 2 element tensor of 2 variables, and call vec.backward() it won’t work, but if vec is a 1 element tensor it will. I can’t see an obvious reason why backward should be default limited to 1 variable (unless explicitly told otherwise), especially seeing as it is a method of the variable.</p>
<p>Is there a reason for this limitation?</p>

albanD: <p>You can call it with more elements, it’s just that you have to specify the  grads yourself.</p>

Duo_Li: <p>can you write a example?</p>

albanD: <p>For example, if you want to compute the gradient for the sum of the elements in <code>x</code>. You can do either:<br>
<code>x.sum().backward()</code> or <code>x.backward(torch.ones_like(x))</code>.</p>

Prograndma: <p>I know that this is an old thread, but it’s what showed up when I googled my error. The way I fixed it was different, and might help somebody else. The issue is that I admittedly copied code from somewhere online and it had:</p>
<pre><code class="lang-auto">nn.BCELoss(reduction='none')  # No reduction to allow masking
</code></pre>
<p>this line in it. My issue came from the reduction=‘none’ part. nn.BCELoss usually takes all the losses from a batch and whatever your output is and finds the average and returns that as a scalar. The reduction=‘none’ makes it so that doesn’t happen. So instead of a scalar I was getting a tensor. Hope this helps other people like me!</p>