How is the MSELoss() implemented?

username: Peter_Ham
cooked: <p>I’m trying to understand how MSELoss() is implemented. Usually people will think MSELoss is (input-target)**2.sum()/batch_size, but when I explicitly write this as the loss function, it turns out that it actually leads to very different training curve from if I use nn.MSELoss()</p>

username: richard
cooked: <p><code>nn.MSELoss()</code> is implemented by default as: <code>((input-target)**2).mean()</code></p>

username: Peter_Ham
cooked: <p>Thanks. What’s the difference between this and <code>(input-target).pow(2).sum()/input.size()[0]</code>?</p>

username: richard
cooked: <p><code>((input-target)**2).mean()</code> is equivalent to</p>
<pre><code class="lang-auto">sizes = input.size()
(input-target).pow(2).sum()/ input.numel()
</code></pre>
<p><code>input.numel()</code> doesn’t have to equal <code>input.size()[0]</code> unless <code>input</code> is a 1D tensor.</p>

username: rasemailcz
cooked: <p>Hi,</p>
<p>I tried the following loss functions</p>
<pre><code class="lang-auto">output = model(data)
train_loss1 = F.mse_loss(output, target.cuda(), True)
train_loss2 = ((output - target.cuda())**2).mean()
</code></pre>
<p>and got different results. Also, the training curve is different.</p>
<p>How is that possible, please?</p>

username: richard
cooked: <p>If training is different, the computation of gradients is probably different, assuming you’re using the same optimizer.</p>
<p>Can you check if your gradients (or the output) is different? Are there any random parts to your model (random initialization, etc?) Weights to <code>nn</code> layers are usually randomly initialized.</p>

username: rasemailcz
cooked: <p>Training is different, but that does not matter actually.</p>
<p>What matters is that <code>train_loss1</code> and <code>train_loss2</code> are different numbers, though I would expect them to be equal. How can this happen?</p>

username: richard
cooked: <p><a class="mention" href="/u/rasemailcz">@rasemailcz</a> could you show me some inputs where <code>train_loss1</code> and <code>train_loss2</code> are different?</p>

username: rasemailcz
cooked: <p><a class="mention" href="/u/richard">@richard</a></p>
<p>output = <code>-0.0226 -0.1922 0.1461 -0.0481 -0.0526 -0.0449 0.1586 -0.0980 0.2177 0.0402 -0.1200 0.0010 0.0022 0.0430 0.1930 0.2304 0.0043 0.0659 0.2427 0.1580 -0.0449 -0.0477 0.1961 0.2336 -0.0308 0.0648 0.0669 0.0072 0.0353 -0.2793 0.0105 -0.1510 -0.0942 -0.1761 0.0477 -0.0564 -0.1628 0.0467 -0.0819 -0.2643 0.1066 -0.0952 0.0918 -0.0934 0.1405 -0.1959 -0.0477 -0.1138 -0.1032 -0.0622 -0.0658 0.2957 -0.1170 -0.1541 0.1663 0.2635 -0.1477 0.2634 0.0940 -0.0477 -0.1920 0.0104 0.3450 -0.0514 -0.1592 0.2188 -0.3998 -0.1696 -0.1194 -0.3216 -0.0702 0.0074 -0.0223 0.0597 0.0329 -0.1500 0.2207 0.1900 -0.0688 0.3510 0.1114 -0.0829 0.0919 -0.1787 0.0266 -0.2059 0.0821 -0.1061 0.1190 0.0090</code></p>
<p>target.cuda() = <code>0.3100 0.0638 -0.1865 -0.4252 -0.6371 -0.8090 -0.9300 -0.9922 -0.9919 -0.9289 -0.8072 -0.6343 -0.4212 -0.1814 0.0700 0.3170 0.5440 0.7365 0.8823 0.9720 0.9999 0.9641 0.8669 0.7143 0.5162 0.2850 0.0355 -0.2163 -0.4543 -0.6634 -0.8300 -0.9435 -0.9965 -0.9855 -0.9113 -0.7784 -0.5955 -0.3742 -0.1287 0.1252 0.3711 0.5931 0.7769 0.9105 0.9854 0.9965 0.9431 0.8286 0.6604 0.4493 0.2089 -0.0451 -0.2962 -0.5282 -0.7259 -0.8763 -0.9697 -1.0000 -0.9649 -0.8669 -0.7122 -0.5110 -0.2762 -0.0232 0.2313 0.4707 0.6794 0.8435 0.9523 0.9986 0.9792 0.8954 0.7526 0.5602 0.3308 0.0794 -0.1772 -0.4222 -0.6394 -0.8143 -0.9355 -0.9947 -0.9880 -0.9158 -0.7828 -0.5979 -0.3731 -0.1234 0.1345 0.3836</code></p>

username: richard
cooked: <p>The following script:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
from torch.autograd import Variable

output = Variable(torch.Tensor(
[-0.0226, -0.1922, 0.1461, -0.0481, -0.0526, -0.0449, 0.1586, -0.0980, 0.2177, 
0.0402, -0.1200, 0.0010, 0.0022, 0.0430, 0.1930, 0.2304, 0.0043, 0.0659, 0.2427, 
0.1580, -0.0449, -0.0477, 0.1961, 0.2336, -0.0308, 0.0648, 0.0669, 0.0072, 0.0353,
-0.2793, 0.0105, -0.1510, -0.0942, -0.1761, 0.0477, -0.0564, -0.1628, 0.0467, 
-0.0819, -0.2643, 0.1066, -0.0952, 0.0918, -0.0934, 0.1405, -0.1959, -0.0477,
-0.1138, -0.1032, -0.0622, -0.0658, 0.2957, -0.1170, -0.1541, 0.1663, 0.2635,
-0.1477, 0.2634, 0.0940, -0.0477, -0.1920, 0.0104, 0.3450, -0.0514, -0.1592, 0.2188,
-0.3998, -0.1696, -0.1194, -0.3216, -0.0702, 0.0074, -0.0223, 0.0597, 0.0329, 
-0.1500, 0.2207, 0.1900, -0.0688, 0.3510, 0.1114, -0.0829, 0.0919, -0.1787, 0.0266, 
-0.2059, 0.0821, -0.1061, 0.1190, 0.0090]), requires_grad=True)

target = Variable(torch.Tensor(
[0.3100, 0.0638, -0.1865, -0.4252, -0.6371, -0.8090, -0.9300, -0.9922, -0.9919, 
-0.9289, -0.8072, -0.6343, -0.4212, -0.1814, 0.0700, 0.3170, 0.5440, 0.7365, 0.8823,
0.9720, 0.9999, 0.9641, 0.8669, 0.7143, 0.5162, 0.2850, 0.0355, -0.2163, -0.4543, 
-0.6634, -0.8300, -0.9435, -0.9965, -0.9855, -0.9113, -0.7784, -0.5955, -0.3742,
-0.1287, 0.1252, 0.3711, 0.5931, 0.7769, 0.9105, 0.9854, 0.9965, 0.9431, 0.8286, 
0.6604, 0.4493, 0.2089, -0.0451, -0.2962, -0.5282, -0.7259, -0.8763, -0.9697, 
-1.0000, -0.9649, -0.8669, -0.7122, -0.5110, -0.2762, -0.0232, 0.2313, 0.4707, 
0.6794, 0.8435, 0.9523, 0.9986, 0.9792, 0.8954, 0.7526, 0.5602, 0.3308, 0.0794,
-0.1772, -0.4222, -0.6394, -0.8143, -0.9355, -0.9947, -0.9880, -0.9158, -0.7828,
-0.5979, -0.3731, -0.1234, 0.1345, 0.3836]))

train_loss1 = F.mse_loss(output, target, True)
train_loss2 = ((output - target)**2).mean()
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">In [8]: train_loss1
Out[8]:
Variable containing:
 0.5171
[torch.FloatTensor of size (1,)]

In [9]: train_loss2
Out[9]:
Variable containing:
 0.5171
[torch.FloatTensor of size (1,)]
</code></pre>
<p>I’d also expect it to give the same on the gpu. Does your problem occur when your inputs are on the cpu?</p>

username: rasemailcz
cooked: <p><a class="mention" href="/u/richard">@richard</a></p>
<p>I found the reason, but I must have missed it in the documentation - can you explain me (redirect me to the documentation), why the output of the</p>
<pre><code class="lang-auto">train_loss1 = F.mse_loss(output, target, True)
train_loss2 = ((output - target)**2).mean()
</code></pre>
<p>is different where the sizes of the tensors are:</p>
<pre><code class="lang-auto">output.data.size()
Out[9]: torch.Size([90, 1, 1, 1])
</code></pre>
<p>and</p>
<pre><code class="lang-auto">target.data.size()
Out[8]: torch.Size([90, 1])
</code></pre>
<p>?</p>
<p>When the <code>output</code> size is changed to [90, 1], the results are the same…</p>

username: richard
cooked: <p>That appears to be a bug, thank you for pointing it out. I’ve opened an issue here: <a href="https://github.com/pytorch/pytorch/issues/4938">https://github.com/pytorch/pytorch/issues/4938</a>.</p>
<p>Relevant reading is <a href="http://pytorch.org/docs/master/notes/broadcasting.html">broadcasting semantics</a>; something seems off about <code>mse_loss</code>.</p>

username: Peter_Ham
cooked: <p><a class="mention" href="/u/rasemailcz">@rasemailcz</a> make sure your output and target have exactly the same dimension. For example, if you have output of size [32,] but target of size [32,1], then the calculated loss is wrong. You can do this by <code>output.view(target.size())</code></p>

username: Peter_Ham
cooked: <p><a class="mention" href="/u/richard">@richard</a> it does not make sense to me since <code>input.size()[0]</code> is just the batch size, so you divide your total loss by the batch size. What’s wrong with this?</p>

username: richard
cooked: <pre><code class="lang-auto">loss = nn.MSELoss()
out = loss(x, t)
</code></pre>
<p>divides by the total number of elements in your tensor, which is different from the batch size.</p>

username: Peter_Ham
cooked: <p>thanks. usually people only divide that by the batch size I think?</p>

username: richard
cooked: <p>My above explanation was for how <code>nn.MSELoss</code> is implemented. If you want to divide only by the batch size, you can do the following:</p>
<pre><code class="lang-auto">loss = nn.MSELoss(reduce=False)
out = loss(x, t).sum() / batch_size
</code></pre>

username: Benlin_Liu
cooked: <p>it’s divided by the total number of elements in the input, i.e. n<em>c</em>h*w for a 4d input, not batch size.</p>

username: xuef
cooked: <p><a class="mention" href="/u/richard">@richard</a><br>
do you have any comment to this <a href="https://discuss.pytorch.org/t/difference-between-nn-mseloss-and-torch-mean-op-target-2/48843">related post</a></p>