Changing the set of trainable params in the optimizer during training?

username: jayz
cooked: <p>Hey,</p>
<p>I have this idea in mind that I would like to try and see what happens when I randomly “dropout” the training of entire layer’s weights during different training steps. E.g. in epoch one train all layers but the first one, in epoch two maybe drop out layer three and five, again in some other epoch train all layers, and so on.</p>
<p>The idea is that it might on the one hand add a kind of regularization to the model, as well as save some computational effort, if implemented correctly, because the gradients of the corresponding parameters that are “turned off” in training during an epoch do not need to be computed.</p>
<p>I don’t know whether this is a common practice, and I am interested in hearing opinions on this.</p>
<p>What would be the typical way to achieve this?</p>
<p>Best,<br>
JZ</p>

username: rfmac
cooked: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"></p>
<p>I can understand this in two ways:</p>
<ol>
<li>You’re talking about the dropout technique, applyied to all weights of an entire layer</li>
<li>You want that the weights of a layer do not contribute to the weights update.</li>
</ol>
<p>The first option can be achieved by applying a dropout with a drop probability equals to 1.0, but it doesn’t make sense for me, because after zeroing the weights of an entire layer, all the next layers would be zeroed too, including the output.</p>
<p>The second option possibly can be achieved by freezing the layer weights dinamically, setting the <code>requires_grad</code> attribute of the parameters of a single layer to <code>False</code>, then reverting it back to <code>True</code>.</p>
<p>Best Regards,<br>
Rafael Macedo.</p>

username: jayz
cooked: <p>Hello once more, Rafael, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>yes, I meant the second option, was not sure however how to realize it. I should have probably not mixed up my explanation with the terminology of dropout, as I realize now, it is a bit confusing.</p>
<p>So, I will simply randomly access layers like <code>model.layer1.weight/bias.requires_grad=False</code>, then train like this for one epoch, and after that set requires_grad=True, again. This should introduce some interesting form of regularization, no?</p>
<p>Best, JZ</p>

username: rfmac
cooked: <p>Hello!!</p>
<p>Yeah, I think the way you explained is going to do what you want.</p>
<p>About introducing interesting form of regularization, I have absolutely no idea hahaha<br>
Deep Learning is an experimental research area, hardly some one can predict how even a minor change in the model will affect it, and as I’ve never seen some one using this technique, I’ll not try to guess.</p>
<p>So go on, try it, and if possible, come back and let your conclusions registered here in the forum <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"><br>
See ya!</p>

username: jayz
cooked: <p>Hey,</p>
<p>just wanted to give some quick feedback on this, for people interested in this topic. I recently discovered that the method I proposed is close to a regularization method called Zoneout. Find the paper here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1606.01305">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1606.01305" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1606.01305" target="_blank" rel="noopener nofollow ugc">Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations</a></h3>

  <p>We propose zoneout, a novel method for regularizing RNNs. At each timestep,
zoneout stochastically forces some hidden units to maintain their previous
values. Like dropout, zoneout uses random noise to train a pseudo-ensemble,
improving...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Best, JZ</p>