Network forgets everything during learning (strange behavior of loss function)

priscacheese: <p>Hi,</p>
<p>I am training a model using Pytorch and Pytorch Geometric and I’ve noticed a weird behavior of the loss function:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016.png" data-download-href="https://discuss.pytorch.org/uploads/default/eaf8003a9f9d79e97da553a5be3486eefd938016" title="20K_samples"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_690x345.png" alt="20K_samples" data-base62-sha1="xwD4bFjvHFONAjcNZweRjcdKYt0" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_1380x690.png 2x" data-dominant-color="FDFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20K_samples</span><span class="informations">2400×1200 96.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems that during training the network forgets everything. Moreover, the more samples I have in the dataset, the more often this kind of “reset” occurs (and when the dataset is large, the network doesn’t learn at all).</p>
<p>These are the optimizer and the loss function that I am using</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()

lr_start = 2e-4
l2_weight = 1e-10 # L2 regularization weight
optimizer = optim.Adam(net.parameters(), lr=lr_start, weight_decay=l2_weight)

lr_decay = torch.tensor(0.99, dtype=torch.float64)
lr_decay_steps = torch.tensor(5, dtype=torch.float64)
gamma = torch.pow(lr_decay, torch.div(1, lr_decay_steps))

scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=gamma, last_epoch=-1)
</code></pre>
<p>I am not an expert of neural network and Pytorch, so I was wondering if maybe there are some mistakes in the code above or what else could cause this issue…</p>

gphilip: <p>If you could share the code for your training loop then I can try debugging it.</p>

priscacheese: <p>This is my training loop:</p>
<pre><code class="lang-auto">    for epoch in range(load_epoch, load_epoch+n_epochs):
        start = time.time()
        loss, accuracy = train_epoch(net,
                                     optimizer,
                                     criterion,
                                     device,
                                     train_dataloader,
                                     clip_val
                                     )
        end = time.time()

        wandb.log({'training loss':loss})
        wandb.log({'training accuracy':accuracy})

        if (epoch+1) % print_every == 0:
            logger.info(f"Training loss at epoch {epoch+1} : {loss}")
            logger.info(f"Training accuracy at epoch {epoch+1} : {accuracy}")
            logger.info(f"Runtime for one epoch : {end-start}")

        if (epoch+1) % test_every == 0:
            start = time.time()
            val_loss, accuracy = test(net, device, test_dataloader, criterion)
            end = time.time()
            wandb.log({'validation loss':val_loss})
            wandb.log({'validation accuracy':accuracy})

            logger.info(f"Validation loss at epoch {epoch+1} : {val_loss}")
            logger.info(f"Validation accuracy at epoch {epoch+1} : {accuracy}")
            logger.info(f"Runtime for testing : {end-start}")

        if (epoch+1) % save_every == 0:
            logger.info(f"Saving training at epoch {epoch+1}")
            torch.save({
                'epoch': epoch+1,
                'model_state_dict': net.state_dict(),
                'optimizer_state_dict': optimizer.state_dict(),
                'scheduler_state_dict': scheduler.state_dict() if scheduler is not None else None,
                'losses': losses,
                'val_losses': val_losses,
                'accuracies': accuracies,
                'val_accuracies': val_accuracies
                }, join(save_folder,"epoch_{:04d}.tar".format(epoch+1)))
</code></pre>
<p>where:</p>
<pre><code class="lang-auto">def train_epoch(net, optimizer, loss, device, dataloader, clip_val):
    # training step on all dataset
    net.train()

    running_loss = 0.0
    accuracy = 0.0
    for batch in dataloader:
        # zero the parameter gradients
        optimizer.zero_grad()

        batch.to(device)
        pred = net(batch)
        pred_binary = torch.sigmoid(pred.squeeze()) &gt;= 0.5
        pred_correct = pred_binary == batch.y.type_as(pred)
        accuracy_temp = torch.sum(pred_correct)/len(pred_correct)

        accuracy += accuracy_temp.item()

        logit = loss(pred.squeeze(), batch.y.type_as(pred))
        logit.backward()

        if clip_val is not None:
            torch.nn.utils.clip_grad_value_(net.parameters(), clip_val)
        optimizer.step()

        running_loss += logit.detach().item() 

    running_loss /= len(dataloader)
    accuracy /= len(dataloader)

    return running_loss, accuracy

def test(net, device, test_dataloader, val_criterion):
    net.eval()
    net = net.to(device)

    with torch.no_grad():
        val_loss = 0.0
        accuracy = 0.0

        for batch in test_dataloader:
            batch.to(device)
            pred = net(batch)
            pred_binary = torch.sigmoid(pred.squeeze()) &gt;= 0.5
            pred_correct = pred_binary == batch.y.type_as(pred)
            accuracy_temp = torch.sum(pred_correct)/len(pred_correct)

            accuracy += accuracy_temp.item()

            val_logit = val_criterion(pred.squeeze(),
                                      batch.y.type_as(pred)).item()
            val_loss += val_logit

    val_loss /= len(test_dataloader)
    accuracy /= len(test_dataloader)

    return val_loss, accuracy
</code></pre>
<p>(I’ve tried with <code>clip_val = 0.65</code> and <code>clip_val = None</code>)</p>
<p>I hope this is enough code, and I am happy to share more if it can be useful.</p>

gphilip: <p>Thank you for the code.</p>
<p>Could you tell me the following?</p>
<ul>
<li>What is the meaning of the term <code>accuracy_temp</code>? What do you intend this term to contain?</li>
<li>Why does it make sense to accumulate <code>accuracy_temp</code> values across batches by <em>adding</em> them?</li>
<li>Why does it make sense to divide the summed-up  <code>accuracy_temp</code> values by the number of things in <code>dataloader</code>?</li>
</ul>
<p>For instance: suppose your <code>net</code> predicted half of the targets correctly in every batch. What would computing <code>accuracy</code> in this manner give you as the final <code>accuracy</code> value? Does this match the <code>accuracy</code> value that you expect to get, if your <code>net</code> was good at predicting exactly half of the targets in each batch correctly?</p>
<p>I have the same questions about <code>logit</code> as well: why does it make sense to add them up over batches, and why does it make sense to divide this sum by the total number of items in the input?</p>

priscacheese: <p>Thank you a lot for taking the time to help me.</p>
<p>My idea was to use <code>accuracy_temp</code> to compute the average accuracy over each batch and then sum over all the computed <code>accuracy_temp</code> and divide by the number of batches (which I thought to be equal to <code>len(dataloader)</code>) to compute the overall accuracy.</p>
<p>I realize now that this is correct only if the number of elements in each batch is constant, which is not necessarily true. I will change the code in order to count the total number of correctly predicted samples along all batches, and divide by the total number of samples only after the for loop.</p>
<p>Regarding your question, I think that if the network predicted half of the elements in every batch correctly, I would get 0.5 as accuracy as expected, but maybe the code is not doing what I thought…</p>
<p>With respect to the loss function, I applied a similar reasoning: I wanted to compute the loss for each element and divide it by the number of elements in the dataset, and since I am working with batches, I thought of averaging it over the number of batches. What would be the common way of computing the loss value?</p>
<p>I hope this answer your questions.</p>

gphilip: <aside class="quote no-group" data-username="priscacheese" data-post="5" data-topic="131503">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/priscacheese/48/37798_2.png" class="avatar"> priscacheese:</div>
<blockquote>
<p>My idea was to use <code>accuracy_temp</code> to compute the average accuracy over each batch and then sum over all the computed <code>accuracy_temp</code> and divide by the number of batches (which I thought to be equal to <code>len(dataloader)</code>) to compute the overall accuracy.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="priscacheese" data-post="5" data-topic="131503">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/priscacheese/48/37798_2.png" class="avatar"> priscacheese:</div>
<blockquote>
<p>I think that if the network predicted half of the elements in every batch correctly, I would get 0.5 as accuracy as expected,</p>
</blockquote>
</aside>
<p>You can easily check this assumption by working out the numbers <em>by hand</em>  for a simplified scenario: there are a 1000 inputs in total, which are split into 2 equal-sized batches with a 500 elements in each batch, and in each batch the network predicts half of the values correctly. What is the accuracy that will be reported in this case, if you follow your scheme? Does this match what you expected it to be? Does this correctly reflect how good the network is?</p>

priscacheese: <p>Yes, this is what I did: with respect to your example the expected accuracy is 0.5 and the accuracy in each batch is 0.5. Summing over the accuracy computed in each batch and dividing by the number of batches we would get (0.5+0.5)/2 = 0.5. Is this different from what you would expect?</p>
<p>Regarding the loss function, is it correct that you suggested that there is something wrong with respect to the computation of the logits?</p>

gphilip: <aside class="quote no-group" data-username="priscacheese" data-post="7" data-topic="131503">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/priscacheese/48/37798_2.png" class="avatar"> priscacheese:</div>
<blockquote>
<p>Is this different from what you would expect?</p>
</blockquote>
</aside>
<p>My mistake, I assumed that <code>len(dataloader)</code> denotes the number of items in the underlying dataset. If <code>len(dataloader)</code> denotes the number of <em>batches</em> delivered by the dataloader then this computation is correct, except in the case—which you pointed out—when batches have different sizes.</p>