Set forget gate bias of LSTM

kfzn: <p>Hi,<br>
I’d like to set forget gate bias of LSTM to a specific value, for example, 1.0. I’ve been looking at torch.nn.LSTM but could not find a way to do it. Do I have to write my own LSTM layer to do this? Could someone kindly provide a quick example how this can be done? Thank you so much!<br>
Max</p>

apaszke: <p>It’s not super convenient, but we guarantee that a bias vector of each LSTM layer is structured like this:</p>
<pre><code class="lang-auto">[b_ig | b_fg | b_gg | b_og]
</code></pre>
<p>You can find that in the <code>Variables</code> section of the <a href="http://pytorch.org/docs/nn.html#torch.nn.LSTM">LSTM docs</a>.</p>
<p>So, to set the forget gate bias, you’d need to filter out the bias parameters, and set all indices from 1/4 to 1/2 of the length to the desired value.</p>

kfzn: <p>Adam, thank you so much for getting back to me. May I ask another follow up question (this may sound stupid) - how do I access the Variables of LSTM? The API does not seem to provide direct access to modify Variables. Do I have to modify the source code of RNNbase? Best,</p>

Tudor_Berariu: <p>Suppose you have an LSTM model with two layers:</p>
<pre><code>l = LSTM(10, 20, 2)
</code></pre>
<p>Then all its parameters names are in this list of lists:</p>
<pre><code>&gt;&gt;&gt; l._all_weights
Out[35]: 
[['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0'],
 ['weight_ih_l1', 'weight_hh_l1', 'bias_ih_l1', 'bias_hh_l1']]
</code></pre>
<p>And you can acces them by name:</p>
<pre><code>l.bias_ih_l0.data[20:40].fill_(0)
</code></pre>
<p>(In this case biases are arrays of length 80.)</p>
<p>If you want to set the bias for all forget gates to 1:</p>
<pre><code>for names in l._all_weights:
    for name in filter(lambda n: "bias" in n,  names):
        bias = getattr(l, name)
        n = bias.size(0)
        start, end = n//4, n//2
        bias.data[start:end].fill_(1.)</code></pre>

kfzn: <p>Thank you so much. I appreciate it!</p>

akhil_reddy: <p>I know this is not the right group but, is there a similar procedure to do this in torch.I am having a tough time figuring it out. Any help is deeply apreciated.</p>

Kaixhin: <p><a class="mention" href="/u/akhil_reddy">@akhil_reddy</a>  Check the example code for the <a href="https://github.com/Kaixhin/nninit#example" rel="nofollow noopener"><code>nninit</code> library</a> with <code>nn.FastLSTM</code>. There’s a line with the comment “High forget gate bias”. I think it’s the same for <code>nn.LSTM</code> but not 100% sure.</p>

akhil_reddy: <p><a class="mention" href="/u/kaixhin">@Kaixhin</a><br>
Can the command be used after initialising all the parameters of the model (which consists of the lstm layer) using getParameters() ?</p>
<p>Out of curiosity , how are the weights and biases of each gate ordered in FastLSTM<br>
I see that fast gate bias is the fourth quadrant of the biases from the command</p>

Kaixhin: <p><a class="mention" href="/u/akhil_reddy">@akhil_reddy</a>  Yes it should be fine, but I would still check manually by initialising a LSTM, using <code>:getParameters()</code>, updating the bias, and then checking the unrolled weights.</p>
<p>Ah actually that’s an error on my part which I will correct - seems like it’s the 3rd out of 4 blocks according to the <a href="https://github.com/Element-Research/rnn/blob/master/FastLSTM.lua#L182" rel="nofollow noopener">source code</a>. Your best bet is to check the source code if you have any doubts, and raise an issue with the <code>rnn</code> repo if you’re still stuck as this is not a PyTorch issue.</p>

akhil_reddy: <p><a class="mention" href="/u/kaixhin">@Kaixhin</a><br>
Thank you so much for the help, will proceed accordingly.<img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

zanaa: <p>When using a GRU, I would like to initialize the bias reset gate to -1 and would appreciate tips for finding the proper start and end values!</p>

cherry: <p><a class="mention" href="/u/apaszke">@apaszke</a><br>
Hi Adam,<br>
would you be able to specify what do you mean by</p>
<aside class="quote no-group" data-username="apaszke" data-post="2" data-topic="1745">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<p>set all indices from 1/4 to 1/2</p>
</blockquote>
</aside>
<p>Why those specific indices? Wouldn’t be <code>lstm.bias_hh_l0</code> already be all biases of the forget gate of the first layer?</p>

pourya_farzi: <p>GREAT ! That’s what we all need <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>