Adaptive l2 regularization experiment

derby: <p>hi all, i am an amateur getting caught up with pytorch, started tinkering around and had a little idea to make the regularization adapt with respect to the loss per epoch. take these results with a grain of salt.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/derbydefi/al2r">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/derbydefi/al2r" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/9c877d931559c36f49d7d2f0148dbc26e4a11b420c99b3d04fe4da2c47663c5a/derbydefi/al2r" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/derbydefi/al2r" target="_blank" rel="noopener nofollow ugc">GitHub - derbydefi/al2r: Adaptive L2 regularization experiment</a></h3>

  <p>Adaptive L2 regularization experiment. Contribute to derbydefi/al2r development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/7168c5de3ba9d628220382c6586b5d00cfb4ab56.png" data-download-href="https://discuss.pytorch.org/uploads/default/7168c5de3ba9d628220382c6586b5d00cfb4ab56" title="Screenshot from 2024-03-15 15-24-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/7168c5de3ba9d628220382c6586b5d00cfb4ab56_2_602x500.png" alt="Screenshot from 2024-03-15 15-24-46" data-base62-sha1="gbgqBvgpbbPuWgWNSYqzcVFipeK" width="602" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/7168c5de3ba9d628220382c6586b5d00cfb4ab56_2_602x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/7168c5de3ba9d628220382c6586b5d00cfb4ab56_2_903x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/1/7168c5de3ba9d628220382c6586b5d00cfb4ab56.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-03-15 15-24-46</span><span class="informations">1195×991 83.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>maybe not enough data/runs to be make any conclusions, but this was mainly an exercise in learning for me but who knows maybe this is useful to someone else.<br>
here is a paper that confirmed my suspicions and gave me some more empirical backing (though i admit my implementation is much simpler)…</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1907.06288">
  <header class="source">
      <img src="https://arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1907.06288" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail" width="500" height="500"></div>

<h3><a href="https://arxiv.org/abs/1907.06288" target="_blank" rel="noopener nofollow ugc">Learning Neural Networks with Adaptive Regularization</a></h3>

  <p>Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis. While most previous works aim to diversify the representations, we explore the complementary direction by performing an...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>any feedback would be appreciated, thanks</p>

ptrblck: <p>I haven’t read the paper but would be interested to know how many runs were performed to create these plots? Do these lines represent a single run?</p>

derby: <p>yeah they are single run, should be very easily reproducible to test (like copy paste and run the github code if you have all the dependencies) though results may vary (i hope not a lot though lol). this was with MNIST dataset and just on cpu</p>