Extremely negative large losses during training for facial landmark prediction problem

Mona_Jalal: <p>This block of code reports very bad losses:</p>
<pre><code class="lang-auto">model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.MultiLabelSoftMarginLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, sample_batched in enumerate(train_loader):
        #print(i, sample_batched['image'].size(),
        # sample_batched['landmarks'].size())
        
        images_batch, landmarks_batch = \
            sample_batched['image'], sample_batched['landmarks']
        
        images = images_batch
        labels = landmarks_batch.reshape(-1, 68 * 2)
        
        images = Variable(images.float())
        labels = Variable(labels)
        
        images = images.to(device)
        labels = labels.to(device)
        
        # Forward pass
        outputs = model(images)
        
        #print("Label Shape", labels.shape, "Output Shape", outputs.shape)
        
        
        loss = criterion(outputs, labels.float())
        
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if (i+1) % 5 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' 
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))
</code></pre>
<p>The losses are:</p>
<pre><code class="lang-auto">Epoch [1/15], Step [5/17], Loss: -19443.7852
Epoch [1/15], Step [10/17], Loss: -48699.7461
Epoch [1/15], Step [15/17], Loss: -71593.1797
Epoch [2/15], Step [5/17], Loss: -126392.4062
Epoch [2/15], Step [10/17], Loss: -119797.7344
Epoch [2/15], Step [15/17], Loss: -234912.5000
Epoch [3/15], Step [5/17], Loss: -271398.0000
Epoch [3/15], Step [10/17], Loss: -428674.2500
Epoch [3/15], Step [15/17], Loss: -285817.3125
Epoch [4/15], Step [5/17], Loss: -387621.0938
Epoch [4/15], Step [10/17], Loss: -488975.9375
Epoch [4/15], Step [15/17], Loss: -593710.8750
Epoch [5/15], Step [5/17], Loss: -565681.6250
Epoch [5/15], Step [10/17], Loss: -633298.5000
Epoch [5/15], Step [15/17], Loss: -712257.2500
Epoch [6/15], Step [5/17], Loss: -708618.5000
Epoch [6/15], Step [10/17], Loss: -627527.0625
Epoch [6/15], Step [15/17], Loss: -996953.1875
Epoch [7/15], Step [5/17], Loss: -956897.9375
Epoch [7/15], Step [10/17], Loss: -1206485.2500
Epoch [7/15], Step [15/17], Loss: -1061303.5000
Epoch [8/15], Step [5/17], Loss: -1044490.9375
Epoch [8/15], Step [10/17], Loss: -1279828.0000
Epoch [8/15], Step [15/17], Loss: -1121125.0000
Epoch [9/15], Step [5/17], Loss: -1569353.0000
Epoch [9/15], Step [10/17], Loss: -1250445.8750
Epoch [9/15], Step [15/17], Loss: -1473160.0000
Epoch [10/15], Step [5/17], Loss: -1510457.2500
Epoch [10/15], Step [10/17], Loss: -2036726.8750
Epoch [10/15], Step [15/17], Loss: -1585122.0000
Epoch [11/15], Step [5/17], Loss: -2009158.3750
Epoch [11/15], Step [10/17], Loss: -1488373.8750
Epoch [11/15], Step [15/17], Loss: -1942903.0000
Epoch [12/15], Step [5/17], Loss: -2108666.7500
Epoch [12/15], Step [10/17], Loss: -2771515.0000
Epoch [12/15], Step [15/17], Loss: -2192605.7500
Epoch [13/15], Step [5/17], Loss: -2039266.5000
Epoch [13/15], Step [10/17], Loss: -3268939.7500
Epoch [13/15], Step [15/17], Loss: -2243900.0000
Epoch [14/15], Step [5/17], Loss: -3102693.5000
Epoch [14/15], Step [10/17], Loss: -2941674.0000
Epoch [14/15], Step [15/17], Loss: -3125370.7500
Epoch [15/15], Step [5/17], Loss: -2586744.7500
Epoch [15/15], Step [10/17], Loss: -1802294.8750
Epoch [15/15], Step [15/17], Loss: -4690499.0000
</code></pre>
<p>What are some ways to make them more sensible? This is for facial landmark prediction.</p>
<p>I understand the following ConvNet is super simple but is that to be the only reason for these terrible losses?</p>
<pre><code class="lang-auto">num_classes = 68 * 2 #68 coordinates X and Y flattened

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 18, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(18),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0))
        self.layer2 = nn.Sequential(
            nn.Conv2d(18, 32, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2, padding=0))
        self.fc = nn.Linear(32 * 56 * 56, num_classes)
        
    def forward(self, x):
        out = self.layer1(x)
        #print("After Layer 1", out.shape)
        out = self.layer2(out)
        #print("After Layer 2", out.shape)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        #print("Final Layer Output Shape", out.shape)
        return out
</code></pre>

KFrank: <p>Hello Mona!</p>
<p>You are probably passing values for <code>target</code> (your <code>labels</code>) that are<br>
either negative or larger than one to <code>MultiLabelSoftMarginLoss</code>,<br>
causing it to return invalid negative values.</p>
<aside class="quote no-group quote-modified" data-username="Mona_Jalal" data-post="1" data-topic="98453" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<pre><code class="lang-auto">criterion = nn.MultiLabelSoftMarginLoss()
...
        loss = criterion(outputs, labels.float())
</code></pre>
<p>The losses are:</p>
<pre><code class="lang-auto">Epoch [1/15], Step [5/17], Loss: -19443.7852
...
Epoch [15/15], Step [15/17], Loss: -4690499.0000
</code></pre>
</blockquote>
</aside>
<p>Conceptually, <code>MultiLabelSoftMarginLoss</code> should never return a<br>
negative loss value.  Its <code>input</code> (the <code>outputs</code> of your model) should<br>
be <em>logits</em> that run from <code>-inf</code> to <code>inf</code> that are, in effect, converted to<br>
probabilities inside of <code>MultiLabelSoftMarginLoss</code>.  (This part is<br>
correct in your code as your model returns the result of its last<br>
<code>Linear</code> layer for your <code>outputs</code>.)</p>
<p>Its <code>target</code> (your <code>labels</code>), however, should be probabilities that run<br>
from <code>0.0</code> to <code>1.0</code>*.  If the <code>target</code> values are outside of this range, you<br>
violate the requirements of <code>MultiLabelSoftMarginLoss</code>, which can<br>
then return (invalid) negative values, and your training will likely diverge<br>
(which it appears to be doing).</p>
<p>Check that your <code>labels</code> lie within <code>[0.0, 1.0]</code>, and if not, ask what they<br>
mean conceptually, and how (and whether) they can be converted to<br>
probability-like values.</p>
<p>*)  The documentation for <a href="https://pytorch.org/docs/1.6.0/generated/torch.nn.MultiLabelSoftMarginLoss.html#torch.nn.MultiLabelSoftMarginLoss" rel="noopener nofollow ugc">MultiLabelSoftMarginLoss</a> states that</p>
<pre><code class="lang-auto"> y[i]∈{0,  1},
</code></pre>
<p>that is, that <code>y</code> (the <code>target</code>) is equal to either <code>0</code> or <code>1</code>.  But I believe<br>
that <code>MultiLabelSoftMarginLoss</code> is meaningful for values of <code>y</code> that<br>
range from <code>0.0</code> to <code>1.0</code>, and the code accepts such values.</p>
<p>Best.</p>
<p>K. Frank</p>