Discussion about train loss and valid loss

banikr: <p>Hi I am a beginner in deep learning and pytorch. Would like to clarify the following outcomes I got.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/ddddfb07740dded424accd346b1cdb2089348caf.png" alt="image" data-base62-sha1="vEJ6Xc9JNt4Jf4L1fY9z1krAfnh" width="320" height="240"><br>
This graph contains both train loss and valid loss. The train is red and valid is blue. My question is:</p>
<ol>
<li>What can I learn from this graph about the model I am working with.</li>
</ol>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2587239b492b356d340583e9870fe8f15f5d7dd0.png" alt="image" data-base62-sha1="5lZcqYVrP08GrOP9iRZsRCZGRgs" width="320" height="240"><br>
This is just the valid loss from the upper one.<br>
2. What can I infer from this one?</p>
<p>Thanks</p>

ptrblck: <p>Are you summing the loss or is it the mean value?<br>
Which loss are you using and what target (i.e. how many classes, if classification use case)?</p>

banikr: <p>This is for training…</p>
<pre><code class="lang-auto"> for epoch in range(num_epochs):
       running_loss = 0
        for i,sample in enumerate(train_loader):
            .
            .
            .
            output = net(mr)
            optimizer.zero_grad()
            loss = criterion(ct,output)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
            mean_loss = running_loss/(i+1)

</code></pre>
<p>and this is for validation:</p>
<pre><code class="lang-auto">        for i, sample in enumerate(valid_loader):
            running_loss = 0
            with torch.no_grad():
                mr = sample[0].float().to(device)
                ct = sample[1].float().to(device)
                output = net(mr)
                loss = criterion(ct, output)
                running_loss += loss.item()
                mean_loss = running_loss / (i + 1)
                print(
                    'epoch: ...
</code></pre>
<p><strong>1.</strong> from the code I am using above I am summing the loss per epoch in <code>running_loss</code> and the dividing it to get <code>mean_loss</code> per epoch.<br>
<strong>2.</strong> I am using   <code>criterion = nn.MSELoss()</code> loss.<br>
<strong>3.</strong> input size is <code>5,1,32,32,32</code> and trying to synthesize output.</p>

ptrblck: <p>Thanks for the information.<br>
What is the range of your target?</p>

banikr: <p>You mean data values?<br>
Target voxel intensities range from -1010 to 200/300</p>

ptrblck: <p>Does the train and validation set contain the same intensity values?<br>
Also, do the training inputs contain a lot of air (-1000HU), while the validation data contain more organs?</p>
<p>Based on the loss, it <em>seems</em> your model might start with E(output)=0, while the target is a lot of -1000s.</p>
<p>Normalizing the output to e.g. <code>[-1, 1]</code> might help.<br>
For the validation/test you could unnormalize the targets again and compute the real MSE.</p>

banikr: <p><strong>Does the train and validation set contain the same intensity values?</strong><br>
Out of 20 3D subject images(172x220x156), I have split the train-valid-test as 16-2-2. So the train and validation sets are not from the same subject.</p>
<p><strong>Also, do the training inputs contain a lot of air (-1000HU), while the validation data contain more organs?</strong><br>
The train and validation patches are taken in a similar pattern, so it should not be dissimilar drastically.</p>
<p>Could you be a little specific about the normalization?<br>
Shall I normalize both input and output from <code>[-1,1]</code> and train the network?</p>

ptrblck: <aside class="quote no-group" data-username="banikr" data-post="7" data-topic="51301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>Shall I normalize both input and output from <code>[-1,1]</code> and train the network?</p>
</blockquote>
</aside>
<p>I would start by standardizing the input (zero mean, unit variance) similar to the preprocessing pipeline for images.<br>
Regarding the target, I would also start by normalizing it to e.g. <code>[0, 1]</code> and check both outputs.</p>
<p>Could you run a small test by printing the mean value of the first input training sample and the first validation sample as well as the mean values of the outputs of your model for these targets?<br>
The discrepancy between these losses is so huge, that I would assume a total domain mismatch.</p>

banikr: <p>from the code below:</p>
<pre><code class="lang-auto">    for epoch in range(max_epochs):
            .
            .
       for i,sample in enumerate(train_loader):
            .
            .
            mr = sample[0].float().to(device)   #torch.Size([batch_size, 1, 32, 32, 32])
            print(mr.mean())
            ct = sample[1].float().to(device)
            print(ct.mean())
</code></pre>
<p>and the mean of the train and validation samples are.</p>
<pre><code class="lang-auto">train directory has 8000 samples
valid directory has 1000 samples
training ...
train input: 46.612483978271484
train target: 120.2796859741211
train output: 0.11282896995544434

validation input: 46.15531921386719
validation target: -71.48211669921875
validation output: -0.17590099573135376
</code></pre>

ptrblck: <p>Do you have 20 samples in total and split them in 16-2-2?<br>
Since your targets seem to have a high variance I’m not sure, if your model can learn anything useful from the data.</p>
<p>You could try to normalize it, as described, and hope for the best.</p>

banikr: <p>I worked with the same images in 2D network. I didnt normalize or standardize the data but it worked pretty well. I had used metrics like MAE,MSE and PSNR those were in workable range.<br>
The 2D architecture I had used were UNet and GAN with GDL loss.<br>
So I wanted to transfer the same pipeline to 3D network.<br>
I will try the normalizing and see how it works.</p>

banikr: <aside class="quote no-group" data-username="banikr" data-post="1" data-topic="51301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>This graph</p>
</blockquote>
</aside>
<p>I was wondering why for the same code and data validation losses are so smaller than training losses??<br>
Any idea?</p>

ptrblck: <p>It’s a bit hard to tell, but you might just be unlucky in that your validation data values are closer to the current output (<code>0</code>) as indicated my the mean values.</p>

banikr: <p>Hey,<br>
I took <code>mean</code> and <code>std</code> of the train and validation dataset and standardized both input and target to zero mean, unit variance.</p>
<pre><code class="lang-auto">MR = (MR - meanMR) / stdMR
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e9ae145c5a6fb3e077f45d26165e1e3e5f73f2c8.png" alt="image" data-base62-sha1="xledhKRZ195mYlE9KgJvY6dzWuA" width="320" height="240"><br>
Train(red) and validation(blue) loss</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c9539a4a94cd3d4cd65bfc6531ffee3aa2054a1b.png" alt="image" data-base62-sha1="sJ15y78bCoTXYPp7nRcDSt7qZ0D" width="320" height="240"></p>
<p>After that I tried with normalization [0,1] the ground truth and input train and validation set.<br>
The results are kinda similar.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/04a3e189321c16018d1b769b1c29cfd4efcf4676.png" alt="image" data-base62-sha1="F31rA4pbaUmygi5rc0IsGGOcLk" width="320" height="240"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1acd1ede19c14a3d8c1b75836158d2368ed1aa23.png" alt="image" data-base62-sha1="3P5SKp9zLrSiwnDJc5Pd9zqQfzd" width="320" height="240"></p>
<p>any thoughts?</p>

ptrblck: <p>Could you print histograms for the pixel/voxel values of your training and validation images?</p>

banikr: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2345adc9ecfefcc11ed920a03fc15383facd3beb.png" data-download-href="https://discuss.pytorch.org/uploads/default/2345adc9ecfefcc11ed920a03fc15383facd3beb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2345adc9ecfefcc11ed920a03fc15383facd3beb_2_345x208.png" alt="image" data-base62-sha1="521ZRjlT4fJguUrs9sbW9tZ1eX1" width="345" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2345adc9ecfefcc11ed920a03fc15383facd3beb_2_345x208.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2345adc9ecfefcc11ed920a03fc15383facd3beb_2_517x312.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2345adc9ecfefcc11ed920a03fc15383facd3beb_2_690x416.png 2x" data-dominant-color="F7F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2080×1255 74.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<code>max 441.0800</code><br>
<code>min -11.1639</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f352f3a7580982d5be6794446bbc97222b2c39d4.png" data-download-href="https://discuss.pytorch.org/uploads/default/f352f3a7580982d5be6794446bbc97222b2c39d4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f352f3a7580982d5be6794446bbc97222b2c39d4_2_345x204.png" alt="image" data-base62-sha1="yIxKpTmKcfpCJknTaGmgkkQnY3y" width="345" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f352f3a7580982d5be6794446bbc97222b2c39d4_2_345x204.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f352f3a7580982d5be6794446bbc97222b2c39d4_2_517x306.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f352f3a7580982d5be6794446bbc97222b2c39d4_2_690x408.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2078×1233 68.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<code>max 429.9766</code><br>
<code>min -9.8587</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/54559e99ed946d19b67cf5a73f4ae8796e15217d.png" data-download-href="https://discuss.pytorch.org/uploads/default/54559e99ed946d19b67cf5a73f4ae8796e15217d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/54559e99ed946d19b67cf5a73f4ae8796e15217d_2_345x205.png" alt="image" data-base62-sha1="c23xSQXnmxLpeSMzWqYRYZz7hGR" width="345" height="205" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/54559e99ed946d19b67cf5a73f4ae8796e15217d_2_345x205.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/54559e99ed946d19b67cf5a73f4ae8796e15217d_2_517x307.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/54559e99ed946d19b67cf5a73f4ae8796e15217d_2_690x410.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2059×1226 72.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<code>max 3582</code><br>
<code>min -1281</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4aa2453345a4c7bb30485d4edfce134620be4fbc.png" data-download-href="https://discuss.pytorch.org/uploads/default/4aa2453345a4c7bb30485d4edfce134620be4fbc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4aa2453345a4c7bb30485d4edfce134620be4fbc_2_345x205.png" alt="image" data-base62-sha1="aEeZI9nfh0PYQjAeEz5rZf4o0Ta" width="345" height="205" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4aa2453345a4c7bb30485d4edfce134620be4fbc_2_345x205.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4aa2453345a4c7bb30485d4edfce134620be4fbc_2_517x307.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4aa2453345a4c7bb30485d4edfce134620be4fbc_2_690x410.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2057×1225 72.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<code>max 3536</code><br>
<code>min -1181</code></p>

banikr: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I did some debugging in the validation codes and corrected the <code>running_loss</code> line which should start before validation epoch iteration.<br>
Now the loss looks like the following based on 100 patches/image. (1600 train and 200 valid)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d4fae1eace0caa962cfca62673749ce93d563897.png" alt="image" data-base62-sha1="uo6JGZ8mxKVNNXvKGOkl3dOvtm7" width="394" height="275"></p>
<p>What do you think about it?</p>
<p>Is it overfitting as the training loss is lower than the validation loss? But how to define <strong>lower</strong>?</p>

ptrblck: <p>Good to hear you’ve found a bug!</p>
<p>The loss curves now look like usual loss curves. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Yes, your model is overfitting, as the training loss decreases while the validation loss hits a plateau.<br>
If you continue training, the validation loss will probably even increase again.</p>
<p>You could counter it with adding some regularization or reduce the model capacity (e.g. using dropout, fewer parameters etc.).</p>

banikr: <p>Right now I am using <strong>minimum validation loss</strong> as my chosen criteria to save the model.<br>
What do you think about it?<br>
Do you think training the model with more number of patches gonna change the loss graph above like reducing the valid loss or so?</p>

ptrblck: <aside class="quote no-group" data-username="banikr" data-post="19" data-topic="51301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>Right now I am using <strong>minimum validation loss</strong> as my chosen criteria to save the model.<br>
What do you think about it?</p>
</blockquote>
</aside>
<p>That makes sense, as it’s basically using early stopping.</p>
<aside class="quote no-group" data-username="banikr" data-post="19" data-topic="51301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<p>Do you think training the model with more number of patches gonna change the loss graph above like reducing the valid loss or so?</p>
</blockquote>
</aside>
<p>I’m really not sure, as I don’t know how the number of patches affects the capacity of the model.<br>
Try to change it and observe if the gap between the loss curves closes or spreads.</p>