ULMFit fine-tuning freeze

pattiJane: <p>Hello,</p>
<p>I have trained a language model and now I want to fine-tune this pre-trained model.</p>
<pre><code class="lang-auto">model.summary()
model.load_encoder('lmtest') 
model.freeze()
model.summary()
</code></pre>
<p>Before loading the encoder I look at model summary and once again after loading and freezing. However in both summaries trainable modules and number of parameters are still same. I would expect freeze() to set everything other than last layer to non-trainable if I understood correctly. So why does not freeze() change anything (visible)?</p>
<p>I am quite new to pytorch and I would appreciate your guidance.</p>

ecdrid: <p>After freeze, try printing,</p>
<pre><code class="lang-python">for name, params in model.named_parameters():
    if params.requires_grad:
        print(name)
</code></pre>

pattiJane: <p>Thanks a lot! I tried it and got the following error:</p>
<p><code> 'RNNLearner' object has no attribute 'named_parameters'</code></p>
<p><a href="https://stackoverflow.com/questions/11685936/why-am-i-getting-attributeerror-object-has-no-attribute" rel="nofollow noopener">This post</a> says it might be an indentation error but I checked and there was none.</p>

ecdrid: <p>What’s <code>RNN Learner</code>? Also if it’s a model which inherits from <code>nn.module</code> then the above 3 line will definitely run.</p>

pattiJane: <p>It is from ULMFit architecture and takes Sequential.RNN which inherits nn.Sequential as an input as I understand.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/fastai/fastai/blob/564896d7b84b59bee40db19ee298a1028235442b/fastai/text/learner.py#L45" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/fastai/fastai/blob/564896d7b84b59bee40db19ee298a1028235442b/fastai/text/learner.py#L45" target="_blank" rel="nofollow noopener">fastai/fastai/blob/564896d7b84b59bee40db19ee298a1028235442b/fastai/text/learner.py#L45</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="35" style="counter-reset: li-counter 34 ;">
<li>    for i,w in enumerate(itos_new):</li>
<li>        r = stoi_wgts[w] if w in stoi_wgts else -1</li>
<li>        new_w[i] = enc_wgts[r] if r&gt;=0 else wgts_m</li>
<li>        if dec_bias is not None: new_b[i] = dec_bias[r] if r&gt;=0 else bias_m</li>
<li>    wgts['0.encoder.weight'] = new_w</li>
<li>    if '0.encoder_dp.emb.weight' in wgts: wgts['0.encoder_dp.emb.weight'] = new_w.clone()</li>
<li>    wgts['1.decoder.weight'] = new_w.clone()</li>
<li>    if dec_bias is not None: wgts['1.decoder.bias'] = new_b</li>
<li>    return wgts</li>
<li>
</li>
<li class="selected">class RNNLearner(Learner):</li>
<li>    "Basic class for a `Learner` in NLP."</li>
<li>    def __init__(self, data:DataBunch, model:nn.Module, split_func:OptSplitFunc=None, clip:float=None,</li>
<li>                 alpha:float=2., beta:float=1., metrics=None, **learn_kwargs):</li>
<li>        is_class = (hasattr(data.train_ds, 'y') and (isinstance(data.train_ds.y, CategoryList) or</li>
<li>                                                     isinstance(data.train_ds.y, LMLabelList)))</li>
<li>        metrics = ifnone(metrics, ([accuracy] if is_class else []))</li>
<li>        super().__init__(data, model, metrics=metrics, **learn_kwargs)</li>
<li>        self.callbacks.append(RNNTrainer(self, alpha=alpha, beta=beta))</li>
<li>        if clip: self.callback_fns.append(partial(GradientClipping, clip=clip))</li>
<li>        if split_func: self.split(split_func)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/fastai/fastai/blob/564896d7b84b59bee40db19ee298a1028235442b/fastai/text/models/awd_lstm.py#L162" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/fastai/fastai/blob/564896d7b84b59bee40db19ee298a1028235442b/fastai/text/models/awd_lstm.py#L162" target="_blank" rel="nofollow noopener">fastai/fastai/blob/564896d7b84b59bee40db19ee298a1028235442b/fastai/text/models/awd_lstm.py#L162</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="152" style="counter-reset: li-counter 151 ;">
<li>        self.output_dp = RNNDropout(output_p)</li>
<li>        if bias: self.decoder.bias.data.zero_()</li>
<li>        if tie_encoder: self.decoder.weight = tie_encoder.weight</li>
<li>
</li>
<li>    def forward(self, input:Tuple[Tensor,Tensor])-&gt;Tuple[Tensor,Tensor,Tensor]:</li>
<li>        raw_outputs, outputs = input</li>
<li>        output = self.output_dp(outputs[-1])</li>
<li>        decoded = self.decoder(output)</li>
<li>        return decoded, raw_outputs, outputs</li>
<li>
</li>
<li class="selected">class SequentialRNN(nn.Sequential):</li>
<li>    "A sequential module that passes the reset call to its children."</li>
<li>    def reset(self):</li>
<li>        for c in self.children():</li>
<li>            if hasattr(c, 'reset'): c.reset()</li>
<li>
</li>
<li>def awd_lstm_lm_split(model:nn.Module) -&gt; List[List[nn.Module]]:</li>
<li>    "Split a RNN `model` in groups for differential learning rates."</li>
<li>    groups = [[rnn, dp] for rnn, dp in zip(model[0].rnns, model[0].hidden_dps)]</li>
<li>    return groups + [[model[0].encoder, model[0].encoder_dp, model[1]]]</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ecdrid: <p>Try this,</p>
<pre><code class="lang-auto">for name, params in your_learner_object.model.named_parameters():
    if params.requires_grad:
        print(name)
</code></pre>

pattiJane: <p>Thanks a lot! It works now!<br>
It only prints parameters from the last layer so it means <code> freeze()</code> works I think <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>1.layers.0.weight<br>
1.layers.0.bias<br>
1.layers.2.weight<br>
1.layers.2.bias<br>
1.layers.4.weight<br>
1.layers.4.bias<br>
1.layers.6.weight<br>
1.layers.6.bias</p>
</blockquote>