Improve the performance of my model

moreshud: <p>Hi everyone,</p>
<p>I am working with 3D dataset on 2D Unet network fed per slices. Having gathered insight from the forum over challenges, I finally with some samples and the results show that my model is overfitting. I would like you suggesting in improving the performance of my model</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28c761e40a6bb7d306ae2f9533f56b64e3bd98d2.png" alt="2021-03-19 23_52_07_UNet_pathologies_loss_plot" data-base62-sha1="5OKgXKPeTogU1mlMohHsk53iNxg" width="640" height="480"></p>

KFrank: <p>Hi Moreshud!</p>
<aside class="quote no-group" data-username="moreshud" data-post="1" data-topic="115536" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f17d59/48.png" class="avatar"> moreshud:</div>
<blockquote>
<p>the results show that my model is overfitting.</p>
</blockquote>
</aside>
<p>I wouldn’t say that your model is overfitting (yet).</p>
<p>It is true that your training loss is lower that your validation loss – and<br>
is going down more rapidly.  But to me overfitting means that your<br>
validation loss starts going up (even while the training loss continues to<br>
decrease).  The idea is that your model is fitting the “random” specifics<br>
of your training set so much that it degrades your model’s previous<br>
ability to make good predictions on your validation set.</p>
<p>So keep training for more epochs as long as your validation loss is<br>
still going down – even if it’s larger than your training loss.</p>
<p>Another observation of greater concern to me is that your training<br>
loss starts out significantly larger than your validation loss.  This is<br>
something you should get to the bottom of.  Is it just a statistical fluke?<br>
You could rerun your training with different random partitions of your<br>
data into the training and validation sets.  Are you computing your<br>
training and validation losses in slightly different ways so that you’re<br>
not making a direct comparison?  Is it an outright bug?</p>
<p>Best.</p>
<p>K. Frank</p>

aguennecjacq: <p>+1 for what KFrank just said, you need to wait a bit more for your model to overfit.<br>
If you want a good starting point to improve your model, start by playing with your learning rate.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/0782e7e59e16d83214184d7a371f6c0a210933f1.jpeg" alt="image" data-base62-sha1="14rO1fJDCgGGrxvOCu66NJEJxbb" width="459" height="414"><br>
Related article: <a href="https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10" class="inline-onebox" rel="noopener nofollow ugc">Understanding Learning Rates and How It Improves Performance in Deep Learning | by Hafidz Zulkifli | Towards Data Science</a></p>

moreshud: <p>Thanks for the explanation</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="115536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>So keep training for more epochs as long as your validation loss is<br>
still going down – even if it’s larger than your training loss.</p>
</blockquote>
</aside>
<p>I am training for more epoch as you suggested to improve the model performance.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="115536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You could rerun your training with different random partitions of your<br>
data into the training and validation sets</p>
</blockquote>
</aside>
<p>Due to the nature of the dataset (given as compressed <code>.nii.gz</code>), I converted each of the datasets to <code>.npy</code> in order to train the volumetric data in slices. Doing this, I split the dataset into training, validation, and testing since some of the given files are either without pathologies, with just one pathology, or both (where there is a bit overlap). Currently rerunning with a different splitting ratio of the training to validation.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="115536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are you computing your<br>
training and validation losses in slightly different ways so that you’re<br>
not making a direct comparison? Is it an outright bug?</p>
</blockquote>
</aside>
<p>This is also one of my worries on seeing the result and I have checked what might likely be the cause. I have computed the loss by initializing to zero on every epoch and followed the example given on the PyTorch site though for a classification task.</p>
<pre><code class="lang-auto">print('Training &amp; Validation Started......')
    for epoch in epoch_ranger:
        start_time = time.time()
        
        epoch_loss = {"train":{"pathologies":0, "lunglobes":0}, 
                      "valid":{"pathologies":0, "lunglobes":0}}
        epoch_score = {"train":{"pathologies":0, "lunglobes":0},
                       "valid":{"pathologies":0, "lunglobes":0}}
        epoch_channel = {"train":{}, "valid":{}}
        
        # Each epoch has a training and validation phase
        for phase in ['train', 'valid']:
            if phase == 'train':
                model.train() 
            else:
                model.eval() 

            running_pathologies_loss, running_lunglobes_loss = 0.0, 0.0
            running_pathologies_score, running_lunglobes_score  = 0.0, 0.0

            # Iterate over data.
            for data in dataloaders[phase]:                
                inputs, pathology_target = data

                inputs = inputs.to(device) 
                pathology_target = pathology_target.to(device)
                
                # zero the parameter gradients
                optimizer.zero_grad()
               
                # forward
                with torch.set_grad_enabled(phase == 'train'): 
                    outputs = model(inputs) 

                    pathology_loss = criterion["pathologies"](outputs, pathology_target)#.mean()  # [:, [0,1,2], :,:]
                    pathology_pred = torch.nn.Sigmoid()(outputs) &gt;= .5
                    
                    dice_coefficient_pathologies = evaluation_metric(pathology_pred, pathology_target)
                                       
                    # backward + optimize only if in training phase
                    if phase == 'train':
                        pathology_loss.backward() #retain_graph=True 
                        optimizer.step()
                           
                # statistics
                running_pathologies_loss += pathology_loss.item() * inputs.size(0)
                
                running_pathologies_score += dice_coefficient_pathologies.item() * inputs.size(0)
            
                        
            epoch_loss[phase]["pathologies"] = running_pathologies_loss/len(dataloaders[phase].dataset)
            epoch_score[phase]["pathologies"] = running_pathologies_score / len(dataloaders[phase].dataset)
            
            # storing experiment result for visualization
            container[phase]["loss"]["pathologies"].append(epoch_loss[phase]["pathologies"])
            container[phase]["score"]["pathologies"].append(epoch_score[phase]["pathologies"])
                        
            if phase == "valid":
                container["learning_rate"].append([param_group['lr'] for param_group in optimizer.param_groups][0])         
                average_val_loss = (epoch_loss["valid"]["pathologies"] + epoch_loss["valid"]["lunglobes"])/2
                # scheduler.step(average_val_loss)    #ReduceOnPlateau                          
                # scheduler.step()                             
            
                # saving model checkpoint of inference and/or resuming training            
                checkpoint = {
                    'epoch': epoch + 1,
                    'avg_min_val_loss': average_val_loss,
                    'model': model,
                    'model_state_dict': model.state_dict(),
                    'optimizer': optimizer,
                    'optimizer_state_dict': optimizer.state_dict(),
                    'container': container
                }
            
                save_checkpoint(checkpoint, False, checkpoint_path, best_model_path)
            
                if epoch != 0 and average_val_loss &lt; avg_min_val_loss: # pathologies or lunglobes?
                    print('Validation loss decreased ({:.6f} --&gt; {:.6f}).  Saving model ...'.format(avg_min_val_loss, average_val_loss))
                    avg_min_val_loss = average_val_loss
         
                    save_checkpoint(checkpoint, True, checkpoint_path, best_model_path)             

        training_time = str(datetime.timedelta(seconds=time.time() - start_time))[:7]
        
        print("Epoch: {}/{}".format(epoch+1, num_epochs),
              "Training | pathologies - loss: {:.4f}".format(epoch_loss["train"]["pathologies"]),
              "score: {:.4f}".format(epoch_score["train"]["pathologies"]),
              "Validation | pathologies - loss: {:.4f}".format(epoch_loss["valid"]["pathologies"]),
              "score: {:.4f}".format(epoch_score["valid"]["pathologies"]),
              "|Time: {}".format(training_time))
</code></pre>

moreshud: <p>I will check other value of my learning rate with few epoch to see which gives a good starting point. Can I handle this better with learning rate scheduling?</p>

aguennecjacq: <p>I would recommend getting the starting learning rate right at first, then do some fine tuning with a learning rate scheduler. For other parameters, I would suggest copying them from the professionals (check a few papers and github codes)  when starting. After a while you will be able to roughly estimate how your parameters should be from experience.<br>
Unfortunately, what I’m describing is the ideal situation. This depends a lot on what you are doing and your resources. If your model takes a long time to train or you don’t have the resources to train it quickly (contrarily to popular belief, not every researcher has &gt;100 gpus <img src="https://discuss.pytorch.org/images/emoji/apple/neutral_face.png?v=9" title=":neutral_face:" class="emoji" alt=":neutral_face:">), you may not be able to do a lot fine tuning with your hyperparameters.</p>