Changed pytorch model behavior after loading pre-trained weights

hrnademi: <p>I was just trying to train UNet from scratch with a mammography dataset to detect tumor tissue in mammograms. After training the model for 25 epochs I achieved the following results on TRAIN set:<br>
<strong>Epoch 25, Loss: 0.0168, mean_IoU: 0.4271, mean_Dice: 0.3559, Elapsed time: 403.4554 sec</strong></p>
<p>After saving model’s weights and trying to evaluate the model on TEST set, I got a very bad result:<br>
<strong>mean_Iou: 0.007, mean_Dice: 0.006</strong></p>
<p>At first, I thought the model is suffering from overfitting, but the model performed badly even on TRAIN set:<br>
<strong>mean_Iou: 0.006, mean_Dice: 0.008</strong></p>
<p>The model after loading weights was supposed to reproduce the result in Epoch 25 (am I correct?), but came with very bad results. I doubled check saving and loading model but nothing seems wrong.</p>
<p>Here is the link to implementation on GitHub: <a href="https://github.com/hrnademi/Mammography" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hrnademi/Mammography</a><br>
Here is the link to the reported log till epoch 25: <a href="https://github.com/hrnademi/Mammography/blob/master/logs.txt" class="inline-onebox" rel="noopener nofollow ugc">Mammography/logs.txt at master · hrnademi/Mammography · GitHub</a></p>
<p>Further information about the dataset:<br>
Training images: 2400 images with the size of 256x256 png<br>
Test images: 600 images with the size of 256x256 png</p>
<p>In eval.py the pre-trained model is loaded.</p>
<p>Many thanks for your attention.</p>

user_123454321: <p>In the eval.py file you are loading the ‘Epoch_10_model.pth’ file. If I check in the logs at 10 epoch<br>
Epoch 10, Loss: 0.0396, mean_Iou: 0.0, mean_Dice: 0.0911, Elapsed time: 403.3541 sec</p>
<p>But, yes, 0.008 is much less than 0.0911, however, iou is higher than during training. Hmm…not able to find anything else wrong…<br>
Can you once evaluate on the model at 25th epoch. For that I guess you need to save the model at every 5th iteration. Or at least try with 20th epoch model.</p>

hrnademi: <p>I removed the background from the mammography images and start training again.<br>
<a href="https://github.com/hrnademi/Mammography/tree/master/Report" rel="noopener nofollow ugc">Here</a> you can find the results after 20 epochs. I just updated <a href="https://github.com/hrnademi/Mammography/blob/master/logs.txt" rel="noopener nofollow ugc">logs.txt</a> too. Unfortunately, nothing has changed, results on the test set are very bad.</p>

hrnademi: <p>@ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> Could you help me with this problem?</p>

ptrblck: <p>If I understand the issue correctly, you are getting worse training and test results by loading the trained model.<br>
Could you pass a constant input to the model (e.g. <code>torch.ones</code>) after training it (and before saving the <code>state_dict</code>) and save the output as a reference, and compare it to the result in your test script after loading the <code>state_dict</code>?<br>
If these results are equal, then I guess the difference comes from the data loading and processing in both scripts.</p>

hrnademi: <p>Unfortunately, I got different results:<br>
I think something is wrong with batch normalization in UNet</p>
<p>After training model and before saving state_dict (model.train())<br>
tensor([[[[ -8.4269, -12.0579, -11.8041,  …, -10.0886,  -8.6157,  -5.6419],<br>
[ -5.6116, -10.0398,  -8.1797,  …,  -6.5734,  -5.6017,  -8.4666],<br>
[ -7.3116, -12.3692, -10.6413,  …,  -7.6334,  -4.2797,  -6.6685],<br>
…,<br>
[ -3.6545,  -5.0984,  -5.1562,  …,  -3.8056,  -4.3882,  -9.5862],<br>
[ -7.2274,  -4.3889,  -4.6838,  …,  -3.6184,  -3.2181, -13.1806],<br>
[ -7.8890,  -4.6818,  -6.3442,  …,  -5.1584,  -4.0778,  -6.4020]]]],<br>
device=‘cuda:0’, grad_fn=)</p>
<p>After saving state_dict and set model to evaluation mode (model.eval())</p>
<p>tensor([[[[-5.5660, -5.7427, -6.7510,  …, -6.1302, -6.1012, -7.4343],<br>
[-5.5984, -5.3072, -4.8963,  …, -6.6118, -5.3268, -5.8644],<br>
[-6.6022, -5.5419, -5.0204,  …, -6.9750, -4.1107, -5.6634],<br>
…,<br>
[-4.4239, -4.8906, -6.2160,  …, -5.3613, -4.3033, -3.7815],<br>
[-6.3954, -5.0958, -6.3577,  …, -5.2069, -4.2653, -3.2605],<br>
[-6.4229, -4.5915, -4.6084,  …, -3.9113, -3.7404, -3.7629]]]],<br>
device=‘cuda:0’, grad_fn=)</p>

ptrblck: <p>It’s expected that the results differ if you pass the same input to the model in training and evaluation mode.<br>
In the former case, the batchnorm layers will normalize the input activation using the batch statistics and will update the running stats with it, while the running stats will be used to normalize the activation in the latter case.<br>
Could you compare the output in the training and evaluation script using <code>model.eval()</code> only?</p>

hrnademi: <blockquote>
<p>Could you compare the output in the training and evaluation script using <code>model.eval()</code> only?</p>
</blockquote>
<p>I experimented again the way you said and outputs were equal (<a href="https://drive.google.com/file/d/1ONnDQIXBBblv13WGPCoSbYanK6wwEV5B/view?usp=sharing" rel="noopener nofollow ugc">Click here to see</a>).</p>
<p>Following curves achieved after training model for 50 epochs, I set <code>batch_size=1</code> and <code>track_running_stats=False</code> in every batch normalization that existed in UNet.<br>
When I use the training set for both training and evaluating model, I get good results, but when use test set for evaluation I get weird results.</p>
<p>Link to training log: <a href="https://github.com/hrnademi/Mammography/blob/master/logs.txt" rel="noopener nofollow ugc">https://github.com/hrnademi/Mammography/blob/master/logs.txt</a><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/478a0228a178a83f8055b4937400770cd2ee3c5c.png" alt="1" data-base62-sha1="acRA5BS0IUZSitXfyWCDm8LjdbC" width="480" height="360"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b215d638c8ed3d8a726bdb45e969e7d9aad978d.png" alt="2" data-base62-sha1="1AsJCUFRYtw1BSHz42bQ0ORq80J" width="480" height="360"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23fb8576ac752e2ef3d1db7e7fff09762800c17d.png" alt="3" data-base62-sha1="58jANf8fZPQP010f6OK2nkOGAAB" width="480" height="360"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccdc012e3160b4f4acdde1bcb820e4d896e9ba77.png" alt="4" data-base62-sha1="tegLdSXlkCdMCd0eFsgGlGFZ1QP" width="480" height="360"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/534540d838152e5e6f49093f193166c79c8a2249.png" alt="5" data-base62-sha1="bSE0hvXQHFOtLYYVXJ0iUenwxkZ" width="480" height="360"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6716507919d07cb98e8aef459b1412a78d79c49c.png" alt="6" data-base62-sha1="eHWZeOLMQyVQUmMhRinIFuLTeQk" width="480" height="360"></p>

ptrblck: <aside class="quote no-group" data-username="hrnademi" data-post="9" data-topic="109138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hrnademi/48/12442_2.png" class="avatar"> hrnademi:</div>
<blockquote>
<p>I experimented again the way you said and outputs were equal</p>
</blockquote>
</aside>
<p>That’s good to hear and indicates that saving and loading the model works properly.</p>
<p>If you are seeing bad results for the validation dataset, it points towards the overall training routine, the dataset splits etc. but not necessarily a software bug.</p>