How to Implement L2 Loss in Pytorch(for CPU)

username: Py_Buddy
cooked: <p>I was trying to implement using</p>
<p>criterion = torch.nn.MSELoss()<br>
lr = 1e-4<br>
weight_decay = 0  # for torch.optim.SGD<br>
lmbd = 0.9  # for custom L2 regularization</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)<br>
for data in test_loader:<br>
images, labels = data<br>
test = Variable(images.view(tuple(images.shape)))<br>
outputs = model(test)<br>
true_y = labels.max(1)[1]<br>
predicted_y = outputs.max(1)[1]</p>
<pre><code># Compute and print loss.
loss = criterion(predicted_y, true_y)

optimizer.zero_grad()

reg_loss = None
for param in model.parameters():
    if reg_loss is None:
        reg_loss = 0.5 * torch.sum(param**2)
    else:
        reg_loss = reg_loss + 0.5 * param.norm(2)**2

    loss += lmbd * reg_loss
    print('Loss: ', loss)
</code></pre>
<p>but getting below error</p>
<p>RuntimeError: _thnn_mse_loss_forward not supported on CPUType for Long</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>You can only pass <code>float</code> tensors to calculate gradient using <code>MSELoss</code>. Try to add <code>float()</code> at the end of <code>predicted_y</code> and <code>true_y</code> tensors like below:</p>
<aside class="quote no-group quote-modified" data-username="Py_Buddy" data-post="1" data-topic="52854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/py_buddy/48/12035_2.png" class="avatar"> Py_Buddy:</div>
<blockquote>
<p>loss = criterion(predicted_y.float(), true_y.float())</p>
</blockquote>
</aside>
<p>The reason is when you use <code>.max()</code> it returns Long or simply integer not float numbers. So you have to cast them before passing to the loss function.</p>
<p>Good luck<br>
Nik</p>

username: Py_Buddy
cooked: <p>Thanks for your help</p>
<p>but need only loss value(single value only) , but getting something below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/eb09b2109583efa77421adc2160b3a26a5288305.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb09b2109583efa77421adc2160b3a26a5288305" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/eb09b2109583efa77421adc2160b3a26a5288305.png" alt="image" data-base62-sha1="xxeYEezH8X7y697MEb4fCxedPVj" width="690" height="351" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1112×566 36.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Nikronic
cooked: <p>Oh, I forgot to say which is weird that you are not getting any error.<br>
<code>torch.max(tensor, dim)</code> returns a tuple of values and corresponding indices. So change your code in this way:</p>
<aside class="quote no-group quote-modified" data-username="Py_Buddy" data-post="1" data-topic="52854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/py_buddy/48/12035_2.png" class="avatar"> Py_Buddy:</div>
<blockquote>
<p>_, true_y = labels.max(1)[1]<br>
_, predicted_y = outputs.max(1)[1]</p>
</blockquote>
</aside>
<p>Finally, if you want to get loss values and aggregate them or anything that only related to its value as you said, you must use <code>loss.item()</code>.</p>
<aside class="quote no-group quote-modified" data-username="Py_Buddy" data-post="1" data-topic="52854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/py_buddy/48/12035_2.png" class="avatar"> Py_Buddy:</div>
<blockquote>
<p>print('Loss: ', loss.item())</p>
</blockquote>
</aside>

username: Nikronic
cooked: <p>Anything, If you want to just print the loss value and do not change it in anyway, use <code>.item()</code> and it will return the corresponding value. In your case, just <code>.item()</code> to the print function.</p>
<p>But if you want to change the loss itself, for instance, merging two different losses by weighted sum, something like <code>loss = 10*loss1 + 5*loss2</code>, you should not use <code>.item()</code> because you will lose <code>grad_fn</code> which is your backward function.</p>

username: Py_Buddy
cooked: <p>Yes, got it. Thank you</p>