How to interpret weird loss/accuracy behavior in cross validation

username: jdlin
cooked: <p>I’m using transfer learning to classify binary medical MRI brain images. I used 5-fold cross validation, SGD optimizer and scheduler with different learning rate with <code>1e-4</code> and <code>1e-5</code> to get the following results. The learning curve and the accuracy have weird behavior and I don’t know how to interpret and improve. Please help me indicate the problems and the things to try on.</p>
<p>The optimizer and scheduler settings I used:</p>
<pre><code class="lang-auto">params = [
            { 'params': parameters['base_parameters'], 'lr': sets.learning_rate }, 
            { 'params': parameters['new_parameters'], 'lr': sets.learning_rate }
            ]
        
optimizer = torch.optim.SGD(params=params, momentum=0.9, weight_decay=1e-4)
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)
</code></pre>
<p>Averaged learning curve and accuracy results with <code>lr = 1e-4</code>(average over 5 folds):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48a337e3a916bc9988a7a84a4f5b099db65ca8c7.png" data-download-href="https://discuss.pytorch.org/uploads/default/48a337e3a916bc9988a7a84a4f5b099db65ca8c7" title="Screen Shot 2021-04-29 at 12.28.57 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48a337e3a916bc9988a7a84a4f5b099db65ca8c7_2_690x212.png" alt="Screen Shot 2021-04-29 at 12.28.57 PM" data-base62-sha1="amA4qRDmILfP3tlXknwknKGrbAr" width="690" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48a337e3a916bc9988a7a84a4f5b099db65ca8c7_2_690x212.png, https://discuss.pytorch.org/uploads/default/original/3X/4/8/48a337e3a916bc9988a7a84a4f5b099db65ca8c7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/8/48a337e3a916bc9988a7a84a4f5b099db65ca8c7.png 2x" data-dominant-color="F9F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-04-29 at 12.28.57 PM</span><span class="informations">909×280 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Averaged learning curve and accuracy results with <code>lr = 1e-5</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/185bcea2ca57434d8e014624bc329c463b9ff85f.png" data-download-href="https://discuss.pytorch.org/uploads/default/185bcea2ca57434d8e014624bc329c463b9ff85f" title="Screen Shot 2021-04-29 at 12.29.09 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/185bcea2ca57434d8e014624bc329c463b9ff85f_2_690x212.png" alt="Screen Shot 2021-04-29 at 12.29.09 PM" data-base62-sha1="3tu9CM4rKwLLG86kkqhCBx2ENYj" width="690" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/185bcea2ca57434d8e014624bc329c463b9ff85f_2_690x212.png, https://discuss.pytorch.org/uploads/default/original/3X/1/8/185bcea2ca57434d8e014624bc329c463b9ff85f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/8/185bcea2ca57434d8e014624bc329c463b9ff85f.png 2x" data-dominant-color="F9F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-04-29 at 12.29.09 PM</span><span class="informations">917×283 28.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I have a few questions regarding the results.</p>
<ol>
<li>Will the result be different if I change from SGD to Adams with the same learning rate?</li>
<li>In <code>lr = 1e-4</code> situation, the model predicts single class mostly. As I searched in the forum, people said with lower learning might save the issue and indeed with  <code>lr = 1e-5</code>, the model gives more diverse results (still poor accuracy). However, the train/validation accuracy stops improving and yet fluctuating after certain epochs. What might be the reason for this?</li>
<li>The loss  curve stops decreasing around 2.5, but in normal situations, isn’t the loss value lower than 1? Why is my loss value stops at 2.5?</li>
<li>The data augmentation method I used was randomly select the brain regions during training (i.e. some regions might be ignored). Will this cause the accuracy not improving as shown in the curve? Or should I use other data augmentation method (e.g. train on the whole brain with rotations)?</li>
</ol>
<p>It might be complicated questions to answer, but i would be so appreciated if anyone could help me with this. Thank you!!</p>