How to initialize weights/bias of RNN LSTM GRU?

username: EthanZhangYi
cooked: <p>I am new to Pytorch and RNN, and don not know how to initialize the trainable parameters of nn.RNN, nn.LSTM, nn.GRU. I would appreciate it if some one could show some example or advice!!!</p>
<p>Thanks</p>

username: SherlockLiao
cooked: <pre><code class="lang-python">net = nn.LSTM(10, 20, 1)
net.weight_hh_l0.data.fill_(0)
</code></pre>
<p>make a 1 layer lstm, input_dim = 10, hidden_state = 20, this can make weight in first layer is 0</p>

username: EthanZhangYi
cooked: <p>Thanks <a class="mention" href="/u/sherlockliao">@SherlockLiao</a><br>
There are four weights/bias for a LSTM layer, so all need to be initialized in this way?  Is there a common initialization distribution for LSTM? Like Gaussian or Uniform distribution.</p>
<blockquote>
<p>weight_ih_l[k] – the learnable input-hidden weights of the k-th layer (W_ii|W_if|W_ig|W_io), of shape (input_size x 4<em>hidden_size)<br>
weight_hh_l[k] – the learnable hidden-hidden weights of the k-th layer (W_hi|W_hf|W_hg|W_ho), of shape (hidden_size x 4</em>hidden_size)<br>
bias_ih_l[k] – the learnable input-hidden bias of the k-th layer (b_ii|b_if|b_ig|b_io), of shape (4<em>hidden_size)<br>
bias_hh_l[k] – the learnable hidden-hidden bias of the k-th layer (b_hi|b_hf|b_hg|b_ho), of shape (4</em>hidden_size)</p>
</blockquote>

username: SherlockLiao
cooked: <p>I only know how to do it one by one, maybe this way, u can write a for loop</p>
<pre><code class="lang-python">net.weight.data.normal_(0, math.sqrt(2. / n))
</code></pre>

username: lakehanne
cooked: <p>How about this?</p>
<pre><code class="lang-python">dict = {}       #we can store the weights in this dict for convenience
import torch.nn.init as weight_init
for name, param in net.named_parameters(): 
      weight_init.normal(param); 
      dict[name] = param
</code></pre>

username: Rafael_Valle
cooked: <p>How about these solutions below. Are they viable?</p>
<pre><code>init.xavier_normal(GRU)
# or
for p in layer.params():
    init.xavier_normal(p)</code></pre>

username: HappyStorm
cooked: <p>Hi, I found a way as follows (but I’m not sure is it correct or not):</p>
<pre><code class="lang-auto">a = nn.GRU(500, 50, num_layers=2)

from torch.nn import init
for layer_p in a._all_weights:
    for p in layer_p:
        if 'weight' in p:
            # print(p, a.__getattr__(p))
            init.normal(a.__getattr__(p), 0.0, 0.02)
            # print(p, a.__getattr__(p))
</code></pre>
<p>This snippet of the code could initialize the weights of all layers.<br>
Hope this could help you <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: EsraaElelimy
cooked: <p>To initialize the weights for nn.RNN, you can do the following :<br>
In this example, I initialize the weights randomly.</p>
<pre><code class="lang-auto">rnn = nn.RNN(input_size=5,hidden_size=6, 
                      num_layers=2,batch_first=True)  
num_layers = 2
for i in range(num_layers):
     rnn.all_weights[i][0] = torch.randn(size=(5,6)) # weights connecting input-hidden
     rnn.all_weights[i][1] = torch.randn(size=(6,6)) #weights connecting hidden-hidden
</code></pre>

username: piggjf
cooked: <p>I think this will work for initializing weights and biases.</p>
<pre><code class="lang-python">for layer in range(num_layers):
    for weight in rnn._all_weights[layer]:
        if "weight" in weight:
            nn.init.xavier_uniform_(getattr(rnn,weight))
        if "bias" in weight:
            nn.init.uniform_(getattr(rnn,weight))
</code></pre>
<p>If you want to use a different initialization, just change it.  The key is that <code>getattr(rnn, weight)</code> is referencing each of the attributes in question.</p>