Using Sigmoid Function in CNN Classifier

username: thymolect
cooked: <p>Hey @ everybody,</p>
<p>I´m trying to use a CNN Classifier on clinical data (input shape 39,12, rows for values and columns for time intervals)  to predict a categorial statement as good / bad result; so i´m using conv2d with the size (1,3) or (1,5) to find time sensitive patterns (each row == 1 clinical parameter) and a max pool pattern of (1,2).</p>
<p>After running the forward path i´m using a sigmoid function on the output node of the last linear network layer to receive a propability between 0 and 1 for category good; using these propabilties to discriminate the loss between them and the labels [0,1] and backpropagate on them.</p>
<p>I ask myself if the architecture ist correct and if it is whatelse can i do to improve performance, because los isnt really decreasing and accuracy is random at the begining of training and ends up getting worse.</p>
<pre><code class="lang-auto"># CNN Model
# Output Size of Convolutional Layer by ((Width - FilterSize + (Padding * 2)) / Stride) + 1

import torch.nn as nn
import torch.nn.functional as F


class ConvNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, (1,5), padding = (0,1), dilation = 1).double()
        # could use avg_pool2d 
        self.pool = nn.MaxPool2d((1,2), 1)
        self.conv2 = nn.Conv2d(6, 16, (1,3), padding = (0,1), dilation = 1).double()
        self.fc1 = nn.Linear(16 * 39 * 8, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 1)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.float()
        x = x.view(-1,16*39*8)
        # or use x = torch.flatten(x, 1) to flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        # Propabilities through Sigmoid Function
        x = torch.sigmoid(x)
        return x
    
    def binary(self,y):
        # adding function of sigmoid function and round to classify and calculate accuracy; KODIETZ 221017
        y = torch.round(y)
        return y
    
# Define model
net = ConvNet()

import torch.optim as optim
# define optimizer &amp; Loss Function; get model
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)

# training
total_datapoints = len(trainloader)

for epoch in range(num_epochs):
    running_loss = 0.0
    epoch_acc = 0.0
    epoch_loss = 0.0
    print(f'-----------Start Epoch {epoch + 1} -----------------------')
    for i,(input_torch,labels) in enumerate(trainloader):
        
        # origin shape: [batch,channels,height,weight]
        # here we got [5,1,12,39] = 5,1,468
        input_torch = input_torch.double().to(device)
        labels = labels.float().to(device)
        
        # Zero the gradients
        optimizer.zero_grad()
        
        # Forward pass
        outputs = net(input_torch).reshape(1,len(input_torch))[0]
        loss = criterion(outputs, labels)

        # Accuray for Batch
        y_pred = net.binary(outputs)
        correct = sum(labels == y_pred).item()
        acc = correct / len(input_torch)
        epoch_acc += acc
        
        # Backward Propagation and Optimization
        loss.backward()
        optimizer.step()
        
        # print statistics
        epoch_loss += loss.item()
        if i % 5 == 0:
            print(f'{epoch + 1}, {i + 1}\t|\t loss: {loss.item():.5f} \t| acc: {acc}')
            
    print(f'-----------End Epoch {epoch + 1}: | Loss: {epoch_loss/len(trainloader):.5f} | Acc: {epoch_acc/len(trainloader):.3f}')

</code></pre>
<p>thanks for any help!<br>
best regards</p>

username: KFrank
cooked: <p>Hi Obrey!</p>
<p>Short story:  Get rid of the <code>sigmoid()</code> and use <code>BCEWithLogitsLoss</code>.</p>
<aside class="quote no-group" data-username="thymolect" data-post="1" data-topic="163712" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thymolect/48/54086_2.png" class="avatar"> thymolect:</div>
<blockquote>
<p>predict a categorial statement as good / bad result;</p>
</blockquote>
</aside>
<p>This is a <em>binary classification</em> problem.  You could treat it as a two-class<br>
multi-class problem (and use <code>CrossEntropyLoss</code>), but you’re better off<br>
using pytorch’s support for the “special case” of binary classification (and<br>
use <code>BCEWithLogitsLoss</code>).</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        self.fc3 = nn.Linear(84, 1)
...
        x = self.fc3(x)
        # Propabilities through Sigmoid Function
        x = torch.sigmoid(x)
        return x
...
criterion = nn.CrossEntropyLoss()
</code></pre>
</blockquote>
</aside>
<p>You have a mismatch here between your final layer, <code>fc3</code>, and your loss<br>
criterion <code>CrossEntropyLoss</code>.  If you wanted to treat your problem as<br>
a two-class multi-class problem (but you don’t really), you would want<br>
your final layer to have <em>two</em> outputs (e.g., <code>Linear (84, 2)</code>).   As it stands,<br>
your model won’t work.</p>
<p>Leave <code>fc3</code> as it is, get rid of <code>sigmoid()</code>, and use <code>BCEWithLogitsLoss</code>.</p>
<p>(You could keep the <code>sigmoid()</code> and use <code>BCELoss</code>, but doing so would<br>
be less numerically stable and would have no offsetting advantages.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: thymolect
cooked: <p>Thank you very much <a class="mention" href="/u/kfrank">@KFrank</a></p>
<p>I came from a 2 class calssifier and forgot to change the loss criterion <em>outch</em></p>
<p>Anyhow the classifier is still not satisfying <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> - because of an imabalanced dataset i provided a weigth attribute for the loss; but still not performing properly;<br>
while training loss is accurately decreasing the accuracy is increasing;</p>
<p>any suggestions for proper tuning ?</p>
<p>best, obrey</p>

username: KFrank
cooked: <p>Hi Obrey!</p>
<aside class="quote no-group" data-username="thymolect" data-post="3" data-topic="163712" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thymolect/48/54086_2.png" class="avatar"> thymolect:</div>
<blockquote>
<p>because of an imabalanced dataset i provided a weigth attribute for the loss;</p>
</blockquote>
</aside>
<p>Just to confirm, it you are using <code>BCEWithLogitsLoss</code>, you should be<br>
using its <code>pos_weight</code> constructor argument to increase the weight of<br>
the positive samples if they are underrepresented (or decrease their<br>
weight if they are overrepresented).</p>
<aside class="quote no-group">
<blockquote>
<p>while training loss is accurately decreasing the accuracy is increasing;</p>
</blockquote>
</aside>
<p>Could you clarify this?  “Training loss is decreasing” – good, this is what<br>
training is supposed to do.  “The accuracy is increasing.”  To me, “accuracy”<br>
means the percentage of predictions that are correct.  So this would also<br>
be good.</p>
<p>Did you mean that the accuracy was <em>decreasing?</em>  Were you contrasting<br>
training vs. test (or validation) loss?</p>
<p>Best.</p>
<p>K. Frank</p>