Model accuracy is stuck at exact 0.5, loss decreases consistently

copperwiring: <p><strong>This is using PyTorch</strong></p>
<p>I have been trying to implement UNet model on my images, however, my model accuracy is always exact 0.5. Loss does decrease.</p>
<p>I have also checked for class imbalance. I have also tried playing with learning rate. Learning rate affects loss but not the accuracy.</p>
<p>My <strong>architecture</strong> below ( from <a>here</a> )</p>
<pre><code class="lang-python">""" `UNet` class is based on https://arxiv.org/abs/1505.04597

The U-Net is a convolutional encoder-decoder neural network.
Contextual spatial information (from the decoding,
expansive pathway) about an input tensor is merged with
information representing the localization of details
(from the encoding, compressive pathway).

Modifications to the original paper:
(1) padding is used in 3x3 convolutions to prevent loss
    of border pixels
(2) merging outputs does not require cropping due to (1)
(3) residual connections can be used by specifying
    UNet(merge_mode='add')
(4) if non-parametric upsampling is used in the decoder
    pathway (specified by upmode='upsample'), then an
    additional 1x1 2d convolution occurs after upsampling
    to reduce channel dimensionality by a factor of 2.
    This channel halving happens with the convolution in
    the tranpose convolution (specified by upmode='transpose')


    Arguments:
        in_channels: int, number of channels in the input tensor.
                     Default is 3 for RGB images. Our SPARCS dataset is 13 channel.
              depth: int, number of MaxPools in the U-Net. During training, input size needs to be 
                     (depth-1) times divisible by 2
        start_filts: int, number of convolutional filters for the first conv.
            up_mode: string, type of upconvolution. Choices: 'transpose' for transpose convolution 

"""

class UNet(nn.Module):

    def __init__(self, num_classes, depth, in_channels, start_filts=16, up_mode='transpose', merge_mode='concat'):

        super(UNet, self).__init__()

        if up_mode in ('transpose', 'upsample'):
            self.up_mode = up_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for upsampling. Only \"transpose\" and \"upsample\" are allowed.".format(up_mode))
    
        if merge_mode in ('concat', 'add'):
            self.merge_mode = merge_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for merging up and down paths.Only \"concat\" and \"add\" are allowed.".format(up_mode))

        # NOTE: up_mode 'upsample' is incompatible with merge_mode 'add'
        if self.up_mode == 'upsample' and self.merge_mode == 'add':
            raise ValueError("up_mode \"upsample\" is incompatible with merge_mode \"add\" at the moment "
                             "because it doesn't make sense to use nearest neighbour to reduce depth channels (by half).")

        self.num_classes = num_classes
        self.in_channels = in_channels
        self.start_filts = start_filts
        self.depth = depth

        self.down_convs = []
        self.up_convs = []

        # create the encoder pathway and add to a list
        for i in range(depth):
            ins = self.in_channels if i == 0 else outs
            outs = self.start_filts*(2**i)
            pooling = True if i &lt; depth-1 else False

            down_conv = DownConv(ins, outs, pooling=pooling)
            self.down_convs.append(down_conv)

        # create the decoder pathway and add to a list
        # - careful! decoding only requires depth-1 blocks
        for i in range(depth-1):
            ins = outs
            outs = ins // 2
            up_conv = UpConv(ins, outs, up_mode=up_mode, merge_mode=merge_mode)
            self.up_convs.append(up_conv)
            

        self.conv_final = conv1x1(outs, self.num_classes)

        # add the list of modules to current module
        self.down_convs = nn.ModuleList(self.down_convs)
        self.up_convs = nn.ModuleList(self.up_convs)

        self.reset_params()

    @staticmethod
    def weight_init(m):
        if isinstance(m, nn.Conv2d):
            
            #https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/ 
            ##Doc: https://pytorch.org/docs/stable/nn.init.html?highlight=xavier#torch.nn.init.xavier_normal_ 
            init.xavier_normal_(m.weight)
            init.constant_(m.bias, 0)



    def reset_params(self):
        for i, m in enumerate(self.modules()):
            self.weight_init(m)
        

    def forward(self, x):
        encoder_outs = []
         
        # encoder pathway, save outputs for merging
        for i, module in enumerate(self.down_convs):
            x, before_pool = module(x)
            encoder_outs.append(before_pool)

        for i, module in enumerate(self.up_convs):
            before_pool = encoder_outs[-(i+2)]
            x = module(before_pool, x)
        
        # No softmax is used. This means we need to use
        # nn.CrossEntropyLoss is your training script,
        # as this module includes a softmax already.
        x = self.conv_final(x)
        return x
</code></pre>
<p>Parameters are :</p>
<pre><code class="lang-python">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x,y = train_sequence[0] ; batch_size = x.shape[0]
model = UNet(num_classes = 2, depth=5, in_channels=5, merge_mode='concat').to(device)
optim = torch.optim.Adam(model.parameters(),lr=0.01, weight_decay=1e-3)
criterion = nn.BCEWithLogitsLoss() #has sigmoid internally
epochs = 1000
</code></pre>
<p>The <strong>function</strong> for training is :</p>
<pre><code class="lang-auto">import torch.nn.functional as f


def train_model(epoch,train_sequence):
    """Train the model and report validation error with training error
    Args:
        model: the model to be trained
        criterion: loss function
        data_train (DataLoader): training dataset
    """
    model.train()
    for idx in range(len(train_sequence)):        
        X, y = train_sequence[idx]             
        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device) 
        
        outputs = model(images)
        print(masks.shape, outputs.shape)
        loss = criterion(outputs, masks)
        optim.zero_grad()
        loss.backward()
        # Update weights
        optim.step()
    # total_loss = get_loss_train(model, data_train, criterion)
</code></pre>
<p>My <strong>function for calculating loss and accuracy</strong> is below:</p>
<pre><code class="lang-python">def get_loss_train(model, train_sequence):
    """
        Calculate loss over train set
    """
    model.eval()
    total_acc = 0
    total_loss = 0
    for idx in range(len(train_sequence)):        
        with torch.no_grad():
            X, y = train_sequence[idx]             
            images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
            masks = Variable(torch.from_numpy(y)).to(device) 

            outputs = model(images)
            loss = criterion(outputs, masks)
            preds = torch.argmax(outputs, dim=1).float()
            acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])
            total_acc = total_acc + acc
            total_loss = total_loss + loss.cpu().item()
    return total_acc/(len(train_sequence)), total_loss/(len(train_sequence))

</code></pre>
<p><strong>Edit</strong> : Code which runs (calls) the functions:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    train_model(epoch, train_sequence)
    train_acc, train_loss = get_loss_train(model,train_sequence)
    print("Train Acc:", train_acc)
    print("Train loss:", train_loss)

</code></pre>
<p>This is my first post on the forum and first time playing with PyTorch.<br>
Can someone help me identify as why is accuracy always exact 0.5?</p>

jankunze: <p>Hi,</p>
<p>what resolution do your images have? Right now you are creating an UNet with depth=10. That means according to the model that</p>
<aside class="quote no-group" data-username="copperwiring" data-post="1" data-topic="72210">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> copperwiring:</div>
<blockquote>
<pre><code class="lang-auto">input size needs to be (depth-1) times divisible by 2
</code></pre>
</blockquote>
</aside>
<p>which equals 512. The original UNet has a depth of 4 if I’ve counted correctly. Maybe decreasing the depth of your model will solve the problem, but that’s only a wild guess.</p>
<p>EDIT:<br>
You are using BCEWithLogitsLoss which seems to be okay <a href="https://discuss.pytorch.org/t/loss-function-crossentropyloss-vs-bcewithlogitsloss/16089/4">according to this post</a>. But your UNet-model says in the comments, that</p>
<pre><code class="lang-auto">        # No softmax is used. This means we need to use
        # nn.CrossEntropyLoss is your training script,
        # as this module includes a softmax already.
</code></pre>
<p>Could this be a problem?</p>

copperwiring: <p>Hi <a class="mention" href="/u/jankunze">@jankunze</a>, I updated my code with more code (for clarity).</p>
<p>Also, I tried using depth 5 (updated) too, it doesn’t help.  Loss decreases but not doesn’t affect accuracy.</p>
<p>Also, as for <code>BCEWithLogitsLoss()</code> , it has a sigmoid internally as discussed <a href="https://discuss.pytorch.org/t/bceloss-vs-bcewithlogitsloss/33586">here</a>.</p>
<p>Thank you for the response. I’ll look for more insights.</p>

copperwiring: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can you please provide some input by any chance? I see you have answered questions like <a href="https://discuss.pytorch.org/t/how-to-use-binary-cross-entropy-with-logits-in-binary-target-and-3d-output/52784/5">these</a> and I feel your input would be really helpful.</p>

copperwiring: <p><a class="mention" href="/u/k_frank">@K_Frank</a> I see you have answered questions (<a href="https://discuss.pytorch.org/t/why-is-beclosswithlogits-compute-different-value-from-crossentropyloss/40818">example</a>) related to BCELoss (which is where I think the problem is). Can you give some insights into my problem please?</p>
<p>Thank you for you time.</p>

copperwiring: <p><a class="mention" href="/u/alband">@albanD</a> Sorry for tagging you here but I was reading few of your answers (<a href="https://discuss.pytorch.org/t/f-cross-entropy-vs-torch-nn-cross-entropy-loss/25505/6">1</a>, <a href="https://discuss.pytorch.org/t/passing-variable-in-bce-loss-function/71881/6">2</a> ). I would be grateful if you can look at my above problem.</p>
<p>Thank you so much!</p>

albanD: <p>Hi,</p>
<p>Could you give more details on the loss value evalution? Both for training and validation sets?<br>
Also you should try with a much simpler model just to make sure that you are able to learn something on your data that is better than random.<br>
You might have some bugs in your code that will be easier to catch with a simpler model with a more predictable behavior.</p>

jankunze: <p>Hi,</p>
<p>can you show the method for calculating accuracy?</p>
<aside class="quote no-group">
<blockquote>
<p>accuracy_check_for_batch</p>
</blockquote>
</aside>
<p>Two more things you can try (if not already done):<br>
Test your model with a standard dataset to see if the problem lies in your data or somewhere else.</p>
<p>Visualize the prediction images. Maybe that can give you a hint.</p>

copperwiring: <aside class="quote no-group" data-username="copperwiring" data-post="1" data-topic="72210">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> copperwiring:</div>
<blockquote>
<p>criterion = nn.BCEWithLogitsLoss() <span class="hashtag-raw">#has</span> sigmoid internally</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/alband">@albanD</a>, I used <code>criterion = nn.BCEWithLogitsLoss() #has sigmoid internally</code> for loss calculation. Pardon me but I didn’t get what exactly did you mean by <code>loss value evaluation</code></p>