Weight Decay Parameter

username: julioeu99
cooked: <p>Hi, can someone explain me in newbie words (i´m new at deep learning word), what does the parameter weight decay on torch adam? And whats the impact if i change it from 1e-2 to 0.<br>
Thank you.</p>

username: SANTOSH_S
cooked: <p>Hi<br>
<a class="mention" href="/u/julioeu99">@julioeu99</a> weight decay in simple terms just reduces weights calculated with a constant(here 1e-2). This ensures that one does not have large weight values which sometimes leads to early overfilling.<br>
Weight decay sometimes makes the model to converge slower.</p>
<p>By default pytorch has <code>weight_decay=0</code></p>
<p>Some useful discussions on the same:</p><aside class="quote quote-modified" data-post="4" data-topic="33105">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/48/60_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-sgd-weight-decay-work/33105/4">How does SGD weight_decay work?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    The part that I circled doesn’t seem right to me: 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e1761b0ce81dd7cff5679379168d2e5f1b6bec70.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1761b0ce81dd7cff5679379168d2e5f1b6bec70" title="39a9f417cb836820174d71e9a9318d5ccc6191f3.png">[39a9f417cb836820174d71e9a9318d5ccc6191f3]</a> 
In L2 regularization, you modify the cost as follows 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/44ad32d9502e88846fc36bb31c06c7df8ef97c69.png" data-download-href="https://discuss.pytorch.org/uploads/default/44ad32d9502e88846fc36bb31c06c7df8ef97c69" title="text_Regularized.png">[text_Regularized]</a> 
The weight update should be then 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e831c3e6a017fcf2fd3ae177c71c928274772d57.png" data-download-href="https://discuss.pytorch.org/uploads/default/e831c3e6a017fcf2fd3ae177c71c928274772d57" title="text_updatedWeig.png">[text_updatedWeig]</a> 
The way PyTorch applied the weight decay seems correct to me (you can drop the factor 2)
  </blockquote>
</aside>
<aside class="quote" data-post="4" data-topic="8436">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-pytorch-implement-weight-decay/8436/4">How pytorch implement weight_decay?</a> 
  </div>
  <blockquote>
    That line means, in other notation: 
d_p = d_p + weight_decay * p.data. 
Here’s a good article about why the L2 penalty is implemented by adding weight_decay * weight_i to the gradient: <a href="https://stats.stackexchange.com/questions/29130/difference-between-neural-net-weight-decay-and-learning-rate" rel="noopener nofollow ugc">https://stats.stackexchange.com/questions/29130/difference-between-neural-net-weight-decay-and-learning-rate</a>
  </blockquote>
</aside>