How to stop weight-decay from affectnig PReLU?

username: Kalamaya
cooked: <p>Hi, so I would like to use the PReLU function, however as can be seen here, it warns us that we should not use weight-decay with it.</p>
<p>Now in my code, the weight-decay is being used, and it is called like this:</p>
<pre><code># Optimizer
optimizer = optim.Adam(net.parameters(), 
                      lr = float(args.learningRate), 
                      weight_decay = float(args.l2_weightDecay)
                      ) 
</code></pre>
<p>What I do not understand though, is how can I specify explicitly, that I do not want my PReLU layers to be affected by weight decay?</p>
<p>Thanks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/72d8a5d7d778bef3abd37c803f10c42dc3bd7b03.png" data-download-href="https://discuss.pytorch.org/uploads/default/72d8a5d7d778bef3abd37c803f10c42dc3bd7b03" title="image.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/72d8a5d7d778bef3abd37c803f10c42dc3bd7b03_2_690x458.png" width="690" height="458" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/72d8a5d7d778bef3abd37c803f10c42dc3bd7b03_2_690x458.png, https://discuss.pytorch.org/uploads/default/original/1X/72d8a5d7d778bef3abd37c803f10c42dc3bd7b03.png 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/72d8a5d7d778bef3abd37c803f10c42dc3bd7b03.png 2x" data-dominant-color="EEF0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image.png</span><span class="informations">843×560 45.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: apaszke
cooked: <p>See <a href="http://pytorch.org/docs/optim.html#per-parameter-options">per-parameter optimizer options</a>.</p>

username: xuanzhangyang
cooked: <p>per-parameter optimizer options like this:<br>
optim.SGD([<br>
{‘params’: model.base.parameters()},<br>
{‘params’: model.classifier.parameters(), ‘lr’: 1e-3}<br>
], lr=1e-2, momentum=0.9)</p>
<p>but i don’t know which attribute of model include the PReLU,<br>
optim.SGD([<br>
{‘params’: model.conv.parameters()},<br>
{‘params’: model.bias.parameters()},<br>
{‘params’: model.prelu.parameters(), weight_decay=0}<br>
], weight_decay=0.0001,lr=1e-2, momentum=0.9)<br>
it’s right ?</p>

username: falmasri
cooked: <aside class="quote quote-modified" data-post="4" data-topic="4946">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/uapatira/48/670_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dont-regularize-me-bro/4946/4">Don't Regularize Me, Bro</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Thanks for the tips, guys. Your suggestions, and looking at the .step() method of <a href="http://pytorch.org/docs/master/_modules/torch/optim/sgd.html#SGD.step" rel="noopener nofollow ugc">an optimizer</a>, provide some key insight. Here’s a compact example for reference purposes, in case anyone else has this need: 
model = nn.Sequential(
  nn.Conv2d(...),
  nn.PReLU(...),
  nn.Conv2d(...),
  nn.PReLU(...),
  nn.Conv2d(...),
)

params = [
  { 'params' : model[0].parameters() },
  { 'params' : model[2].parameters() },
  { 'params' : model[4].parameters() },

  { 'params' : model[1].parameters(), 'weight_d…
  </blockquote>
</aside>