Initializing parameters of a multi-layer LSTM

dpernes: <p>I have a <code>nn.Module</code> that contains an LSTM whose number of layers is passed in the initialization. I would like to do Xavier initialization of its weights and setting the bias of the forget gate to 1, to promote learning of long-term dependencies. My problem is how to iterate over all the parameters in order to initialize them.</p>
<p>Doing something like</p>
<pre><code>for name, param in lstm.named_parameters():
  if 'bias' in name:
     nn.init.constant(param, 0.0)
  elif 'weight' in name:
     nn.init.xavier_normal(param)
</code></pre>
<p>does not work, because <code>param</code> is a copy of the parameters in <code>lstm</code> and not a reference to them. This kind of loop can be used, for instance, to print the values of the parameters but not to modify them (as far as I know). Thank you in advance.</p>

tom: <aside class="quote no-group quote-modified" data-username="dpernes" data-post="1" data-topic="5791">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dpernes/48/16250_2.png" class="avatar"> dpernes:</div>
<blockquote>
<p>Doing something like<br>
…<br>
does not work, because param is a copy of the parameters in lstm and not a reference to them.</p>
</blockquote>
</aside>
<p>Did you try? Your snippet works perfectly well for me.</p>
<p>I was surprised at the other thread. The reason you cannot assign the loop variable is because then the loop variable name will just point to something else. You most certainly can modify the element you are looping over and it is exactly that what is sometimes thought of as surprising when you loop over a list of lists and append to the inner variables or somesuch.</p>
<p>In fact, looping over <code>model.parameters()</code> way above is also how the optimizers get the parameters they are optimizing…</p>
<p>Best regards</p>
<p>Thomas</p>

dpernes: <p>I have just tried it now  in a toy example and in fact it works! <img src="//discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:">  No idea why it was not working before… Some stupid mistake, for sure. Thank you again!</p>

yzexeter: <p>It works in my experiments. Thanks!</p>

fallcat: <p>Hi! I have a question. Don’t you need to do ‘xavier_uniform_’ instead of ‘xavier_uniform’? Aren’t functions with underscore the ones that modify the weights in-place?</p>

krishna: <p>Thank you <a class="mention" href="/u/dpernes">@dpernes</a> , your code works. Yes <a class="mention" href="/u/fallcat">@fallcat</a> we need to use   xavier_uniform_ for in-place initialization of weights.</p>

ishan_pytorch: <p>Seems like the functions with the underscore are deprecated (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/init.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/init.html</a>). Hence dpernes’ solution worked for everyone.</p>

sajastu: <p>Works for me as well!</p>

cc4df22877b7876e50f0: <p>I want to load a rnn pretrained model  from a dict(named model_pretrained), param changed. But when I print the model parametes, I found rnn parameters not change.</p>
<pre><code class="lang-auto">for name, param in model.rnn.named_parameters():
            print("0", param)
            param = torch.nn.Parameter(model_pretrained["rnn."+name])
            print("1", param)
</code></pre>

cc4df22877b7876e50f0: <p>print(id(param)) get different identity</p>
<pre><code class="lang-auto">for name, param in at.rnn.named_parameters():
    print(id(param))
    param = torch.nn.Parameter(model_pretrained_itbl["rnn."+name].to("cuda"), requires_grad = False)
    print(id(param))
</code></pre>
<blockquote>
<p>140360628406128<br>
140360701517152</p>
</blockquote>