Which is better? Loss + optimizer question

username: newtopytorch
cooked: <p>I am testing my program on a subset of my data to make sure everything works. I’m experimenting with different loss functions and optimizers. I’m getting the following results:</p>
<p>The figure below uses MSE as the loss, Adam as the optimizer, and takes 250-300 epochs for the loss to plateau. The plateau sits around a loss of 225.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b4ff252755b90f4768d537442d991b9ef1c92d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b4ff252755b90f4768d537442d991b9ef1c92d9" title="loss_rmse_adam"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b4ff252755b90f4768d537442d991b9ef1c92d9_2_666x500.png" alt="loss_rmse_adam" data-base62-sha1="hAS0QhZ1fxMBwK6n0oj3kLZeHd7" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b4ff252755b90f4768d537442d991b9ef1c92d9_2_666x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b4ff252755b90f4768d537442d991b9ef1c92d9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b4ff252755b90f4768d537442d991b9ef1c92d9.png 2x" data-dominant-color="F6F6FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_rmse_adam</span><span class="informations">800×600 24.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The figure below uses MSE as the loss, Adam as the optimizer, and takes 50-100 epochs for the loss to plateau. The plateau sits around a loss of 214.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aa2dbc22b408ee012728aedfb92afe6bbdbb7e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/2aa2dbc22b408ee012728aedfb92afe6bbdbb7e5" title="loss_rmse_sgd"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aa2dbc22b408ee012728aedfb92afe6bbdbb7e5_2_666x500.png" alt="loss_rmse_sgd" data-base62-sha1="65aYx2Ya03ujlLSEDCBUV46Va5L" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aa2dbc22b408ee012728aedfb92afe6bbdbb7e5_2_666x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aa2dbc22b408ee012728aedfb92afe6bbdbb7e5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aa2dbc22b408ee012728aedfb92afe6bbdbb7e5.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_rmse_sgd</span><span class="informations">800×600 21.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The figure below uses L1 as the loss, Adam as the optimizer, and takes ~250 epochs for the loss to plateau.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bc9da6c13e5e8e7b68a164558170467973f6fc1.png" data-download-href="https://discuss.pytorch.org/uploads/default/0bc9da6c13e5e8e7b68a164558170467973f6fc1" title="loss_rmse_L1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bc9da6c13e5e8e7b68a164558170467973f6fc1_2_666x500.png" alt="loss_rmse_L1" data-base62-sha1="1GhIGjJzXwPMHRXMkbBAmUozqil" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bc9da6c13e5e8e7b68a164558170467973f6fc1_2_666x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bc9da6c13e5e8e7b68a164558170467973f6fc1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bc9da6c13e5e8e7b68a164558170467973f6fc1.png 2x" data-dominant-color="F6F6FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_rmse_L1</span><span class="informations">800×600 27.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What am I supposed to prefer? Fewer epochs? Or lower loss? Edit: batch size and learning rate for all of the above are exactly the same.</p>
<p>I also am not sure about why my RMSE dips slightly, then increases a little and then levels out. Is this dip normal?</p>

username: newtopytorch
cooked: <p>I’m following the procedure outlined in this article below:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://rosenfelder.ai/assets/img/favicon/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://rosenfelder.ai/multi-input-neural-network-pytorch/" target="_blank" rel="noopener nofollow ugc" title="05:40PM - 19 July 2020">Python Tutorials for Machine Learning, Deep Learning and Data Visualization – 19 Jul 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:131/63;"><img src="https://rosenfelder.ai/assets/img//post3/teaser.png" class="thumbnail" width="131" height="63"></div>

<h3><a href="https://rosenfelder.ai/multi-input-neural-network-pytorch/" target="_blank" rel="noopener nofollow ugc">Multi-Input Deep Neural Networks with PyTorch-Lightning - Combine Image and...</a></h3>

<p>A small tutorial on how to combine tabular and image data for regression prediction in PyTorch-Lightning.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>However, I’m using smaller images of 40x40 pixels, and only 3 columns of data in my table.</p>