Should i do loss.backward() or loss.mean().backward()

isalirezag: <p>Can someone please make it clear for me what is the difference the <code>loss.backward()</code> and <code>loss.mean().backward()</code>? confusion overloads <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"><br>
The  loss is <code>loss = criterion(output, label)</code></p>
<p>where/when should i do   l<code>oss.backward</code> and in what senario should i do <code>loss.mean().backward()</code>?<br>
does it have anything to do with batchsize and number of GPUs that are being used?</p>
<p>can it cause a problem or is it wrong if I use one instead of another carelessly?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vmirly1: <p>If your loss is not a scalar value, then you should certainly use either <code>loss.mean()</code> or <code>loss.sum()</code> to convert it to a scalar before calling the backward. Otherwise, it will cause an error as in the following example:</p>
<pre><code class="lang-python">&gt;&gt;&gt; loss = criterion(m(input), target)
&gt;&gt;&gt; loss
tensor([1.2377, 0.6949, 0.6477], grad_fn=&lt;BinaryCrossEntropyBackward&gt;)
&gt;&gt;&gt; loss.backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/vahid/anaconda3/envs/py37/lib/python3.7/site-packages/torch/tensor.py", line 102, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/vahid/anaconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/__init__.py", line 84, in backward
    grad_tensors = _make_grads(tensors, grad_tensors)
  File "/home/vahid/anaconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/__init__.py", line 28, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>

chenglu: <p>For what I know most of the loss function of pytorch will return a scalar by default unless you mannually change the <code>reduce</code> parameter.</p>

MariosOreo: <p>Hi,</p>
<p>Why do you want add  <code>.mean()</code> on loss function, if you call the method in <code>torch.nn</code> or <code>torch.nn.functional</code> it has already call <code>.mean()</code>. For example, if you use <code>torch.nn.CrossEntropyLoss()</code> as your loss function, and input two tensors, one is your model’s output, the other is correspond target, and the loss is also a tensor with only one element like this.</p>
<pre><code class="lang-auto">tensor(1.6975, device='cuda:1', grad_fn=&lt;NllLoss2DBackward&gt;)
... ...
</code></pre>
<p>And you can do backpropagation in your network normally.</p>

vmirly1: <p>Yes, that’s true. The important thing is that calling <code>loss.backward()</code> only works on scalar values. Using the <code>reduction=mean</code> or <code>reduction=sum</code> can take care of this. But sometimes, one may need to use <code>reduction=none</code> in order to apply different mean/sum along different dimensions, for example, applying mean over the batch but taking the sum over pixels.</p>

isalirezag: <p>Yes I agree, but these are all correct for one gpu case.<br>
how about when we are using multiple gpu, should we stick to loss.backward() if our loss is scalar or we should use .mean.backward()?</p>

vmirly1: <p>It does not matter how many GPUs you use. If <code>loss</code> is already a scalar, then you can just call backward <code>loss.backward()</code> but if it is not scalar, then you can convert that to a scalar and then call backward: <code>loss.mean().backward()</code>. The number of GPUs do not play any role on how to call the backward function.</p>

isalirezag: <p>Just FYI, we can also use <code>.mean().backward()</code>, whenever we define the loss inside our <code>nn.model</code>.<br>
It usually being used for the cases that we have multiple gpus and dont want one of the GPUs be unbalance in terms of memory.<br>
More can be found <a href="https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551/4">here</a></p>

vmirly1: <p>O interesting! So if <code>loss</code> is defined inside the model, then <code>loss</code> can have independent values across different GPUs, and in that case we would need to add them up (or use mean) to convert them to a scalar value, and then call <code>.backward()</code>.</p>

shirui-japina: <p>Your question is exactly the solution to <a href="https://discuss.pytorch.org/t/different-samples-use-different-learning-rates/58234">my question</a>.<img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=9" title=":upside_down_face:" class="emoji" alt=":upside_down_face:"><img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>