Why do SGD and Adam behave so differently for a dataset with 1 sample?

username: agt
cooked: <p>Here are two graphs of training my model with only 1 data sample. SGD is perfectly monotonic, while Adam has these weird oscillations. Why is this?</p>
<p>After creating the model, I saved <code>model.state_dict()</code>, trained with SGD, then reloaded the original state dict, and finally trained with Adam.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d718ce392f2972287d904991ff09ba1589d776c7.png" data-download-href="https://discuss.pytorch.org/uploads/default/d718ce392f2972287d904991ff09ba1589d776c7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d718ce392f2972287d904991ff09ba1589d776c7_2_632x500.png" alt="image" data-base62-sha1="uGPNIaHddSG53BGCOneRbrTtIvJ" width="632" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d718ce392f2972287d904991ff09ba1589d776c7_2_632x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d718ce392f2972287d904991ff09ba1589d776c7_2_948x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d718ce392f2972287d904991ff09ba1589d776c7_2_1264x1000.png 2x" data-dominant-color="FBFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1417×1120 57.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73a966c2b0a5375afad1caa34727f30d2d70fef3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/73a966c2b0a5375afad1caa34727f30d2d70fef3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73a966c2b0a5375afad1caa34727f30d2d70fef3_2_595x500.jpeg" alt="image" data-base62-sha1="gvbQHbFFYUYhwH51E8vXUlHwGnF" width="595" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73a966c2b0a5375afad1caa34727f30d2d70fef3_2_595x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73a966c2b0a5375afad1caa34727f30d2d70fef3_2_892x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73a966c2b0a5375afad1caa34727f30d2d70fef3_2_1190x1000.jpeg 2x" data-dominant-color="FCFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1553×1305 82.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For reference, here’s 100 samples (batch_size = 1):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6d094945f06069fa90c7e1c8d258b0a53e7b4ec.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a6d094945f06069fa90c7e1c8d258b0a53e7b4ec" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d094945f06069fa90c7e1c8d258b0a53e7b4ec_2_668x500.jpeg" alt="image" data-base62-sha1="nNI8LZrktL8PC2tG7kBMxZEtb6k" width="668" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d094945f06069fa90c7e1c8d258b0a53e7b4ec_2_668x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d094945f06069fa90c7e1c8d258b0a53e7b4ec_2_1002x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6d094945f06069fa90c7e1c8d258b0a53e7b4ec_2_1336x1000.jpeg 2x" data-dominant-color="F9FAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1795×1343 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e3122e306dc71e8269768cd7cb88ec8e18f6fae.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8e3122e306dc71e8269768cd7cb88ec8e18f6fae" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e3122e306dc71e8269768cd7cb88ec8e18f6fae_2_610x500.jpeg" alt="image" data-base62-sha1="khT4DuMCSHc9NMMjLNqwLHO1OPk" width="610" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e3122e306dc71e8269768cd7cb88ec8e18f6fae_2_610x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e3122e306dc71e8269768cd7cb88ec8e18f6fae_2_915x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e3122e306dc71e8269768cd7cb88ec8e18f6fae_2_1220x1000.jpeg 2x" data-dominant-color="F1F5F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1516×1241 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Prompted by this: <a href="https://discuss.pytorch.org/t/training-with-batch-size-1-all-outputs-are-the-same-and-trains-poorly/102477/8" class="inline-onebox">Training with batch_size = 1, all outputs are the same and trains poorly - #8 by agt</a></p>