Negative output from neural network?

SarahTeoh: <p>I’m currently trying to build a network for Human Activity Recognition using pyTorch and DSADS dataset.</p>
<p>This is my neural network definition.</p>
<pre><code class="lang-auto">import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d((1, 2), stride=2)

        self.conv1 = nn.Conv2d(6,16, (1, 6))
        self.conv2 = nn.Conv2d(16,32, (1, 6))

        self.fc1 = nn.Linear(32 * 1 * 3, 70)
        self.fc2 = nn.Linear(70, 30)
        self.fc3 = nn.Linear(30, 19)

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = self.relu(x)
        x = self.pool(x)
        x = x.view(x.size()[0], -1)
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.relu(x)
        x = self.fc3(x)
        return x

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.0001, momentum=0.9, weight_decay=0.005)
</code></pre>
<p>This is how I load train and test data.</p>
<pre><code class="lang-auto">class data_loader(Dataset):
    def __init__(self, samples, labels, t):
        self.samples = samples
        self.labels = labels
        self.T = t
        
    def __getitem__(self, index):
        sample, target = self.samples[index], self.labels[index]
        if self.T:
            return self.T(sample), target
        else:
            return sample, target

    def __len__(self):
        return len(self.samples)

# train_data_reshaped: numpy array of dimension (57000, 6, 1, 30)
# test_data_reshaped: numpy array of dimension (22800, 6, 1, 30)
train_set = data_loader(train_data_reshaped, train_labels, transform)
    test_set = data_loader(test_data_reshaped, test_labels, transform)
    train_loader = DataLoader(train_set, batch_size=64, shuffle=True, drop_last=True)
    test_loader = DataLoader(test_set, batch_size=64, shuffle=False)
</code></pre>
<p>Training and testing:</p>
<pre><code class="lang-auto">for epoch in range(30):
    for (inputs, labels) in train_loader:
        inputs, labels = inputs.to(device).float(), labels.to(device)
        inputs = inputs.view(-1, 6, 1, 30)
        labels = labels[:, -1].long()
        labels = labels - 1        
        optimizer.zero_grad()
        outputs = net(inputs)
        print("outputs: ", outputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    for (inputs, labels) in test_loader:
        inputs, labels = inputs.to(device).float(), labels.to(device)
        inputs = inputs.view(-1, 6, 1, 30)        
        labels = labels[:, -1].long()
        labels = labels - 1
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
</code></pre>
<p>I expect the net will produce non-negative outputs each of them represents probability. However, my net produces negative output like this:</p>
<pre><code class="lang-auto">outputs:  tensor([[-0.2586, -0.0550,  0.2156,  ...,  0.0132, -0.1516,  0.0052],
        [-0.2586, -0.0952,  0.2352,  ...,  0.0199, -0.1925,  0.0107],
        [-0.4870, -0.0813,  0.4605,  ...,  0.0353, -0.1569,  0.0322],
        ...,
        [-0.2445, -0.1426,  0.1625,  ...,  0.0872, -0.1863,  0.0826],
        [-0.3659, -0.1266,  0.3297,  ...,  0.0860, -0.1500,  0.0035],
        [-0.3237, -0.0675,  0.2883,  ...,  0.0384, -0.2692,  0.0562]],
       grad_fn=&lt;AddmmBackward&gt;)
</code></pre>
<p>Any idea of what I did wrong? Thanks in advance.</p>

simaiden: <aside class="quote no-group" data-username="SarahTeoh" data-post="1" data-topic="99191">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/67e7ee/48.png" class="avatar"> SarahTeoh:</div>
<blockquote>
<p><code>nn.CrossEntropyLoss()</code></p>
</blockquote>
</aside>
<p>It’s okay that the net produces negative output, to get the actual prediction you can use argmax, or if you prefer the probabilities use softmax after net output</p>

SarahTeoh: <p>Thank you for your reply. However, my accuracy and loss did not change much in the whole learning process. I have tried changing the optimizer to Adam optimizer, learning rate, but the accuracy and loss still do not change.<br>
Any idea where I can start inspecting?</p>
<p>This is how I calculated accuracy and loss</p>
<pre><code class="lang-auto">for (inputs, labels) in test_loader:
        inputs, labels = inputs.to(device).float(), labels.to(device)
        inputs = inputs.view(-1, 6, 1, 30)        
        labels = labels[:, -1].long()
        labels = labels - 1
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        sum_loss += loss.item()
        predicted = torch.argmax(outputs, dim=1)
        sum_total += labels.size(0)
        sum_correct += (predicted == labels).sum().item()
        print("test  mean loss={}, accuracy={}"
            .format(sum_loss*BATCH_SIZE/len(test_loader.dataset), float(sum_correct/sum_total)))
    test_loss_value.append(sum_loss*BATCH_SIZE/len(test_loader.dataset))
    test_acc_value.append(float(sum_correct/sum_total))
</code></pre>
<p>Accuracy:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dcf22594de94fed2965f9c402c350703f6fdb94.png" data-download-href="https://discuss.pytorch.org/uploads/default/2dcf22594de94fed2965f9c402c350703f6fdb94" title="Screen Shot 2020-10-13 at 19.21.04"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2dcf22594de94fed2965f9c402c350703f6fdb94_2_532x500.png" alt="Screen Shot 2020-10-13 at 19.21.04" data-base62-sha1="6xfgGoAEVhzkqNVtubwMRo4gZM0" width="532" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2dcf22594de94fed2965f9c402c350703f6fdb94_2_532x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2dcf22594de94fed2965f9c402c350703f6fdb94_2_798x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dcf22594de94fed2965f9c402c350703f6fdb94.png 2x" data-dominant-color="FBFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-10-13 at 19.21.04</span><span class="informations">828×778 32 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

SarahTeoh: <p>Loss looks like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23b7524a9cb44f8b34d2ca3c2eb4c371852ca0a.png" data-download-href="https://discuss.pytorch.org/uploads/default/a23b7524a9cb44f8b34d2ca3c2eb4c371852ca0a" title="Screen Shot 2020-10-13 at 20.02.35"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a23b7524a9cb44f8b34d2ca3c2eb4c371852ca0a_2_532x500.png" alt="Screen Shot 2020-10-13 at 20.02.35" data-base62-sha1="n9aJJ04aq2Yc5rXXQvdLHgFubB8" width="532" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a23b7524a9cb44f8b34d2ca3c2eb4c371852ca0a_2_532x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a23b7524a9cb44f8b34d2ca3c2eb4c371852ca0a_2_798x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a23b7524a9cb44f8b34d2ca3c2eb4c371852ca0a.png 2x" data-dominant-color="FBFDFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-10-13 at 20.02.35</span><span class="informations">828×778 34.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Try to scale down the use case and overfit a small dataset, e.g. just 10 samples by playing around with the hyperparameters.<br>
If that doesn’t work at all there might be an issue in the code we haven’t seen yet.</p>

timiderinola: <p>Hi <a class="mention" href="/u/sarahteoh">@SarahTeoh</a>. I’m facing a problem almost exactly like the one you have described here. Have you found out what was wrong?</p>

zuliani99: <p>I’m getting a similar situation, where going through the training batch my model return as output bigger and bigger negative tensor, like it so:</p>
<pre><code class="lang-auto">0 tensor([[ 0.5556,  0.1423, -0.0081,  ..., -0.0284,  0.2894, -0.1392],
        [ 0.0117, -0.3541, -0.0547,  ..., -0.0909, -0.0544,  0.0516],
        [ 0.1631, -0.4689, -0.1981,  ...,  0.0536,  0.1466, -0.1272],
        ...,
        [ 0.0324, -0.1406, -0.0363,  ..., -0.2817,  0.1431, -0.1592],
        [ 0.3237, -0.5117,  0.2351,  ..., -0.2581,  0.2067,  0.1143],
        [ 0.2187, -0.0261, -0.4586,  ..., -0.0784,  0.0513, -0.1654]],
       grad_fn=&lt;AddmmBackward0&gt;)
1 tensor([[-8.6231e+14, -8.6537e+14, -8.6190e+14,  ..., -8.6262e+14,
         -8.6389e+14, -8.6389e+14],
        [-8.3022e+14, -8.3316e+14, -8.2987e+14,  ..., -8.3053e+14,
         -8.3178e+14, -8.3181e+14],
        [-8.2274e+14, -8.2565e+14, -8.2240e+14,  ..., -8.2306e+14,
         -8.2429e+14, -8.2431e+14],
        ...,
        [-8.5651e+14, -8.5954e+14, -8.5610e+14,  ..., -8.5683e+14,
         -8.5811e+14, -8.5814e+14],
        [-8.5923e+14, -8.6227e+14, -8.5883e+14,  ..., -8.5957e+14,
         -8.6087e+14, -8.6089e+14],
        [-8.5363e+14, -8.5664e+14, -8.5321e+14,  ..., -8.5394e+14,
         -8.5523e+14, -8.5525e+14]], grad_fn=&lt;AddmmBackward0&gt;)
2 tensor([[-2.0069e+16, -2.0111e+16, -2.0064e+16,  ..., -2.0073e+16,
         -2.0092e+16, -2.0092e+16],
        [-1.9857e+16, -1.9899e+16, -1.9852e+16,  ..., -1.9861e+16,
         -1.9879e+16, -1.9880e+16],
        [-1.8977e+16, -1.9017e+16, -1.8971e+16,  ..., -1.8981e+16,
         -1.8998e+16, -1.8998e+16],
        ...,
        [-2.0098e+16, -2.0140e+16, -2.0092e+16,  ..., -2.0102e+16,
         -2.0120e+16, -2.0121e+16],
        [-1.9166e+16, -1.9207e+16, -1.9161e+16,  ..., -1.9170e+16,
         -1.9188e+16, -1.9188e+16],
        [-1.9767e+16, -1.9809e+16, -1.9762e+16,  ..., -1.9771e+16,
         -1.9789e+16, -1.9789e+16]], grad_fn=&lt;AddmmBackward0&gt;)
3 tensor([[-1.0240e+17, -1.0257e+17, -1.0238e+17,  ..., -1.0242e+17,
         -1.0249e+17, -1.0249e+17],
        [-1.0369e+17, -1.0386e+17, -1.0367e+17,  ..., -1.0371e+17,
         -1.0378e+17, -1.0378e+17],
        [-9.4790e+16, -9.4943e+16, -9.4769e+16,  ..., -9.4806e+16,
         -9.4871e+16, -9.4872e+16],
        ...,
        [-9.9082e+16, -9.9240e+16, -9.9060e+16,  ..., -9.9098e+16,
         -9.9169e+16, -9.9169e+16],
        [-1.0332e+17, -1.0349e+17, -1.0330e+17,  ..., -1.0333e+17,
         -1.0341e+17, -1.0341e+17],
        [-9.4997e+16, -9.5149e+16, -9.4977e+16,  ..., -9.5012e+16,
         -9.5076e+16, -9.5077e+16]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>