Loss functions differences

JDE65: <p>I was trying to tailor-make the loss function to better reflect what I was trying to achieve.<br>
As the results were not convincing I therefore tested an MSE loss self-coded versus the nn and F versions with :</p>
<ol>
<li>criterion = nn.MSELoss() and loss1 = criterion1(outputs, targets)</li>
<li>loss2 = F.mse_loss(outputs, targets)</li>
<li>criterion3 = AdjMSELoss() and loss3 = criterion3(outputs, targets)<br>
where :</li>
</ol>
<blockquote>
<p>class AdjMSELoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(AdjMSELoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, outputs, targets):
    outputs = torch.squeeze(outputs)
    loss = (outputs - targets)**2
    return torch.mean(loss)
</code></pre>
</blockquote>
<p>As long as it test this with 2 tensors outside a backprop training, it gives the exact same results… BUT when I run a model with these losses for the backprop, it diverges between the AdjMSELoss custom function and the 2 nn.MSELoss and F.mse_loss (that gave identical results).<br>
NB. All trials share the exact same seed and are reproducible <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Could someone explain why (and advise on the best approach to create an efficient custom loss function) ?<br>
THX in advance</p>
<p>The code for the custom loss function is given above.<br>
The code for the training is as follows :</p>
<blockquote>
<p>def train_model1D(model, criterion, optimizer, epochs, learning, verbose, train_dl):<br>
model.train()<br>
torch.set_grad_enabled(True)<br>
total_step = len(train_dl)<br>
curr_lr = learning<br>
if str(criterion) != ‘CrossEntropyLoss()’:<br>
criterion2 = AdjRegrLoss()<br>
history = dict(train=<span class="chcklst-box fa fa-square-o fa-fw"></span>, train_regr=<span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
else:<br>
criterion2 = nn.CrossEntropyLoss()<br>
accuracy = 0<br>
history = dict(train=<span class="chcklst-box fa fa-square-o fa-fw"></span>, train_regr=<span class="chcklst-box fa fa-square-o fa-fw"></span>, acc=<span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
best_model_wts = copy.deepcopy(model.state_dict())<br>
best_loss = 1e64<br>
for epoch in range(epochs):<br>
train_losses = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
trregr_losses = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
y_preda = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
y_truea = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i, (inputs, targets) in enumerate(train_dl):<br>
y_pred = model(inputs)<br>
loss = criterion(y_pred, targets)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
train_losses.append(loss.item())<br>
loss2 = criterion2(y_pred, targets)<br>
trregr_losses.append(loss2.item())<br>
if str(criterion) == ‘CrossEntropyLoss()’:<br>
_, y_predb = torch.max(y_pred, 1)<br>
y_predb = y_predb.cpu()<br>
y_predb = y_predb.detach().numpy()<br>
y_trueb = targets.cpu()<br>
y_trueb = y_trueb.detach().numpy()<br>
y_preda.extend(y_predb)<br>
y_truea.extend(y_trueb)<br>
train_loss = np.mean(train_losses)<br>
trregr_loss = np.sum(trregr_losses)<br>
history[‘train’].append(train_loss)<br>
history[‘train_regr’].append(trregr_loss)<br>
if str(criterion) == ‘CrossEntropyLoss()’:<br>
accuracy = accuracy_score(y_preda, y_truea)<br>
history[‘acc’].append(accuracy)<br>
if trregr_loss &lt; best_loss:<br>
best_loss = trregr_loss<br>
best_model_wts = copy.deepcopy(model.state_dict())<br>
if ((verbose != 0) and (epoch == 0)):<br>
print('Epoch 0  - Train Loss = ’ + str(round(train_loss, 6)) + ’ // Cumulative error training = ’ + str(round(trregr_loss, 6)))<br>
if ((verbose != 0) and (((epoch+1) % verbose) ==0)):<br>
print('Epoch ’ + str(epoch+1) + ’ - Train Loss = ’ + str(round(train_loss, 6)) + ’ // Cumulative error training = ’ + str(round(trregr_loss,6)))<br>
torch.set_grad_enabled(False)<br>
model.eval()<br>
return model, history</p>
</blockquote>
<p>You’ll notice that</p>
<ol>
<li>I torch squeeze the output to get a tensor of size [batch, ] instead of [batch, 1]</li>
<li>I compute another loss function trregr_loss that is not relevant and not used for the back propagation. It computes the effective loss of prediction if yhat is of the opposite sign of ytrue.</li>
</ol>
<blockquote>
<p>class AdjRegrLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(AdjRegrLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, outputs, labels):
    outputs = torch.squeeze(outputs)
    loss = torch.abs(labels)
    adj = torch.mul(outputs, labels)
    adj[adj&gt;0] = 0
    adj[adj&lt;0] = 1
    loss = loss * adj
    return torch.sum(loss)
</code></pre>
</blockquote>

huahuanZ: <p>Can you give the minimum codes to reproduce the error? And btw, if you want to implement the default init function of your custom class, that should be surrounded with two underscores on each side, like <code>def __init__(self, *args)</code></p>

JDE65: <p>Hello David, I modified the original post with the extended code… I hope it helps <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
and, of course,  my init() are indeed surrounded by 2 __, even if it is not evident from the code shared ?</p>
<p>NB. I don’t understand why the code does not appear as code with the identification <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=9" title=":open_mouth:" class="emoji" alt=":open_mouth:"></p>

huahuanZ: <p>What is the shape of the output <code>y_pred</code> and your labels <code>targets</code>? I think you can’t conduct both <code>CrossEntropyLoss</code> and <code>MSELoss</code> on the same pair of data. As the document states, the shape of input and target to these loss functions should be:</p>
<ol>
<li>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropy#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a>: input<code>[N,C]</code>; target<code>[N]</code> (where N is the batch size, and C is the number of classes.)</li>
<li>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss" rel="noopener nofollow ugc">MSELoss</a>: input and target should be the identical in shape, like <code>[N,*]</code>.</li>
</ol>
<p>So, only when <code>C=1</code>, it’s possible to use the two loss functions at the same time, but that would be meaningless for a classification task.<br>
BTW, if your want to paste codes, use three dots (```) to surround them, so the format won’t be changed.<br>
```<br>
print("")<br>
```</p>

JDE65: <p>Hello Harvey,</p>
<p>I don’t use the topic functions at the same time. When the model is a regression, there is one neuron in the output layer and the loss function is nn.MSE or nn.L1 (MAE) or the custom AdjMSE.</p>
<p>When the model classifies, there are as many neurons in the output layer as there are classes, and the loss function is nn.CrossEntropy…</p>
<p>The function is generic and allows both uses (regression and classification), hence the confusion.</p>
<p>My issue is when I do a regression, I don’t get the same backward propagation if I use nn.MSELoss or if I use the custom AdjMSE while there are assumed to be perfectly identical…</p>

huahuanZ: <p>Can your confirm the <code>targets</code> is of shape <code>[N,]</code> instead of <code>[N,1]</code> when doing regression ?<br>
With your implementation of <code>AdjMSELoss</code>, I notice that if <code>targets</code> is of shape <code>[N,1]</code>, since you conduct squeeze on <code>outputs</code> (which will be <code>[N,]</code>), then the <code>loss</code> will be wrongly broadcast to shape <code>[N, N]</code>.</p>
<p>So, if this is the problem, just squeeze the <code>targets</code> and the loss would be identical to nn.MSELoss.</p>
<pre><code class="lang-auto">class AdjMSELoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, outputs, targets):
        outputs = torch.squeeze(outputs)
        targets = torch.squeeze(targets)
        loss = (outputs - targets)**2
        return torch.mean(loss)
</code></pre>

JDE65: <p>David,</p>
<p>Thanks for looking into this.</p>
<p>The outputs are of shape [N, 1] (therefore, I squeeze it) and the targets are of shape [N, ].</p>
<p>The 2 loss functions nn.MSELoss() and AdjMSELoss() provide the exact same numerical results when I test them on two tensors*… BUT they do not provide similar results when the backward propagation is applied with loss.backward(). And I don’t understand why and which one is the right one…</p>
<p><em>* I created randomly two numpy arrays of respective size [N, 1] and [N, ]. I converted them into Torch tensors and computed the nn.MSELoss and the AdjMSELoss. These were systematically identical. The problem only arise when I apply loss.bacward()</em></p>

huahuanZ: <p>I can’t reproduce what you said, in my test, when outputs and targets are of same shapes, then whatever w/ or w/o <code>loss.backward()</code>. Outcomes of <code>AdjMSELoss()</code> and <code>MSELoss</code> are identical.<br>
Here is my script</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class TestModel(nn.Module):
    def __init__(self, idim: int):
        super().__init__()
        self.linear = nn.Linear(idim, 1)

    def forward(self, x: torch.Tensor):
        return self.linear(x)

class AdjMSELoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, outputs, targets):
        outputs = torch.squeeze(outputs)
        loss = (outputs - targets)**2
        return torch.mean(loss)


def test():
    model = TestModel(5)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
    model.train()
    torch.set_grad_enabled(True)
    criterion = nn.MSELoss()
    criterion2 = AdjMSELoss()

    for i in range(1):
        inputs = torch.randn(4, 5)
        targets = torch.randn(4)
        y_pred = model(inputs)

        loss = criterion(y_pred.squeeze(), targets)
        optimizer.zero_grad()
        # invoke the backward function here
        loss.backward()
        optimizer.step()

        loss2 = criterion2(y_pred, targets)
        print(loss==loss2)
        print("iter {}: {} loss: {:.3e}".format(i, criterion, loss))
        print("iter {}: {} loss: {:.3e}".format(i, criterion2, loss2))


if __name__ == "__main__":
    test()
</code></pre>
<p>And output is</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor(True)
&gt;&gt;&gt; iter 0: MSELoss() loss: 3.582e-01
&gt;&gt;&gt; iter 0: AdjMSELoss() loss: 3.582e-01</code></pre>