More than 100 % Training Accuracy at end of each epoch

Aarya: <p>Hello,</p>
<p>I am training a Siamese network with Binary Cross Entropy Loss and calculating classification accuracy.<br>
I have more than 100 % training accuracy at the end of each epoch. I am not able to fix this.</p>
<p>Following is the screenshot of the issue. (Please look at Epoch 1/20 : Accuracy : 116.98)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c17bf5b517d10bbbdcdd0407dda65c6c099d2d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c17bf5b517d10bbbdcdd0407dda65c6c099d2d4" title="test"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c17bf5b517d10bbbdcdd0407dda65c6c099d2d4_2_533x500.png" alt="test" data-base62-sha1="fqerCKDlycPRq2y0obcK8jiKj52" width="533" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c17bf5b517d10bbbdcdd0407dda65c6c099d2d4_2_533x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c17bf5b517d10bbbdcdd0407dda65c6c099d2d4_2_799x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c17bf5b517d10bbbdcdd0407dda65c6c099d2d4.png 2x" data-dominant-color="343434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test</span><span class="informations">848×795 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Following is the code snippet of train epoch and fit function:</p>
<pre><code class="lang-auto">def fit(train_loader, val_loader, model, loss_fn, optimizer, scheduler, n_epochs, cuda, log_interval, metrics,
        start_epoch=0, message=None) -&gt; object:
    """
    Loaders, model, loss function and metrics should work together for a given task,
    i.e. The model should be able to process data output of loaders,
    loss function should process target output of loaders and outputs from the model

    Examples: Classification: batch loader, classification model, NLL loss, accuracy metric
    Siamese network: Siamese loader, siamese model, contrastive loss
    Online triplet learning: batch loader, embedding model, online triplet loss
    """
    for epoch in range(0, start_epoch):
        scheduler.step()
    #log_data['epoch'] = []
    #log_data['train_losses'] = []

    for epoch in range(start_epoch, n_epochs):
        #log_data['epoch'].append(epoch)
        scheduler.step()

        # Train stage
        train_loss, metrics = train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics)
        #log_data['train_loss'].append(train_loss)

        message = 'Epoch: {}/{}. Train set: Average loss: {:.4f}'.format(epoch + 1, n_epochs, train_loss)
        for metric in metrics:
            message += '\t{}: {}'.format(metric.name(), metric.value())

        val_loss, metrics = test_epoch(val_loader, model, loss_fn, cuda, metrics, log_interval)
        val_loss /= len(val_loader)

        message += '\nEpoch: {}/{}. Validation set: Average loss: {:.4f}'.format(epoch + 1, n_epochs,
                                                                                 val_loss)
        for metric in metrics:
            message += '\t{}: {}'.format(metric.name(), metric.value())

        print(message)
</code></pre>
<pre><code class="lang-auto">def train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics):
    for metric in metrics:
        metric.reset()

    model.train()
    losses = []
    total_loss = 0


    for batch_idx, ((x0, x1), y) in enumerate(train_loader):

        x0, x1, y_true = x0.cpu(), x1.cpu(), y.cpu()
        optimizer.zero_grad()
        output1, output2 = model(x0, x1)

        p_dist = torch.nn.PairwiseDistance(keepdim=True)

        dy = p_dist(output1, output2)
        dy = torch.nan_to_num(dy)
        y_true = torch.nan_to_num(y_true)

        '''2 lines indicated the normalization of dy to 0 and 1 by dividing it with max value'''

        maximum_dy = torch.max(dy)
        maximum_dy = torch.nan_to_num(maximum_dy)
        dy = dy/maximum_dy

        maximum_y_true = torch.max(y_true)
        maximum_y_true = torch.nan_to_num(maximum_y_true)

        y_true = y_true/maximum_y_true

        dy = torch.squeeze(dy, 1)
 
        input_dy = torch.empty(dy.size(0), 2)
        input_dy[:, 0] = 1 - dy
        input_dy[:, 1] = dy
      

        y_true_2 = torch.zeros(dy.size(0), 2)
        y_true_2[range(y_true_2.shape[0]), y_true.long()] = 1

  
        m = nn.Sigmoid()
        loss = loss_fn(m(input_dy), y_true_2)

        loss.backward()
        optimizer.step()

        losses.append(loss.item())
        total_loss += loss.item()

        input_dy_metric = torch.round(input_dy)
        for metric in metrics:
            metric(input_dy_metric, y_true_2)
        for metric in metrics:
            metric.total = (batch_idx+1)*y_true_2.shape[0]

        if batch_idx % log_interval == 0:
            message = 'Train: [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                batch_idx, len(train_loader),
                100. * batch_idx / len(train_loader), np.mean(losses))
            for metric in metrics:
                message += '\t{}: {}'.format(metric.name(), metric.value())

            print(message)
            losses = []

    total_loss /= (batch_idx + 1)
    return total_loss, metrics
</code></pre>
<p>Following is the Accuracy meric definition :</p>
<pre><code class="lang-auto">class Metric:
    def __init__(self):
        pass

    def __call__(self, outputs, target, loss):
        raise NotImplementedError

    def reset(self):
        raise NotImplementedError

    def value(self):
        raise NotImplementedError

    def name(self):
        raise NotImplementedError
</code></pre>
<pre><code class="lang-auto">class Binary_accuracy(Metric):
    """
    Works with classification model
    """

    def __init__(self):
        self.correct = 0
        self.total = 0

    def __call__(self, input_dy_metric, y_true_2):

        self.correct += (input_dy_metric == y_true_2).sum().float()/2

    def reset(self):
        self.correct = 0
        self.total = 0

    def value(self):
        return 100 * float(self.correct) / self.total
        #print(len(self.correct))
        #print(self.total)
    def name(self):
        return 'Accuracy'
</code></pre>
<p>Any help is most appreciated.<br>
Thank you in advance <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/eqy">@eqy</a></p>

gphilip: <aside class="quote no-group" data-username="Aarya" data-post="1" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<pre><code class="lang-auto">for metric in metrics:
            metric.total = (batch_idx+1)*y_true_2.shape[0]
</code></pre>
</blockquote>
</aside>
<p>Should that <code>=</code> be <code>+=</code> ?</p>

Aarya: <p><a class="mention" href="/u/gphilip">@gphilip</a> thank you for your response. But I am not sure if this is working. Are you sure about this solution?</p>

gphilip: <aside class="quote no-group" data-username="Aarya" data-post="3" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p>But I am not sure if this is working.</p>
</blockquote>
</aside>
<p>I don’t understand what you mean by this. In what way are you not sure that this is working?</p>
<aside class="quote no-group" data-username="Aarya" data-post="3" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p>Are you sure about this solution?</p>
</blockquote>
</aside>
<p>What do you mean? You reported a problem, and I suggested something which may help. You are the only person here who is in a position to check the solution. Why are you asking <em>me</em> if I am sure about this solution?</p>

Aarya: <p>It is not working because denominator (total) gets bigger and bigger leading the accuracy in one digit as the training progresses.</p>

gphilip: <aside class="quote no-group" data-username="Aarya" data-post="5" data-topic="128911" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p>denominator (total) gets bigger and bigger leading the accuracy in one digit as the training progresses.</p>
</blockquote>
</aside>
<p>So that step does seem to have solved your original problem?</p>
<aside class="quote no-group" data-username="Aarya" data-post="1" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p>I have more than 100 % training accuracy at the end of each epoch.</p>
</blockquote>
</aside>
<p>Edited to add:</p>
<p>Could you tell me the following? What is happening in the line</p>
<aside class="quote no-group" data-username="Aarya" data-post="1" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p><code>metric.total = (batch_idx+1)*y_true_2.shape[0]</code></p>
</blockquote>
</aside>
<p>?</p>

Aarya: <p>No, unfortunately not. I don’t want the original accuracy to be changed which is seen in between of training, I just need to get rid of the last training accuracy that goes above 100% as per the snippet.</p>
<p>Do you find any other solution to this ?</p>

Aarya: <p>metric.total is collecting all predictions, which is used to calculate the accuracy,<br>
i.e. correct predictions / total predictions.</p>

gphilip: <aside class="quote no-group" data-username="Aarya" data-post="8" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p>metric.total is collecting all predictions,</p>
</blockquote>
</aside>
<p>How does the right-side of that expression do this?</p>

Aarya: <p>I don’t really get that line. But my<code> batch_size</code> is 4 and <code>y_true_2</code> is that target or true label, which is of shape [4,2]</p>

gphilip: <aside class="quote no-group" data-username="Aarya" data-post="10" data-topic="128911" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p>I don’t really get that line.</p>
</blockquote>
</aside>
<p>Figuring that out may help you with the problem.</p>

Aarya: <aside class="quote no-group" data-username="Aarya" data-post="1" data-topic="128911">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarya/48/30746_2.png" class="avatar"> Aarya:</div>
<blockquote>
<p><code>y_true_2[range(y_true_2.shape[0]), y_true.long()] = 1</code></p>
</blockquote>
</aside>
<p>Can you please explain this line, it’s a bit confusing to me.</p>