Bidirectional lstm, why is the hidden state randomly initialized?

billconan: <p>I’m looking at a lstm tutorial. In this tutorial, the author seems to initialize the hidden state randomly before performing the forward path.</p>
<pre><code class="lang-auto">        hidden_a = torch.randn(self.hparams.nb_lstm_layers, self.batch_size, self.nb_lstm_units)
        hidden_b = torch.randn(self.hparams.nb_lstm_layers, self.batch_size, self.nb_lstm_units)
</code></pre>
<p>it makes more sense to me to initialize the hidden state with zeros.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/williamFalcon/f27c7b90e34b4ba88ced042d9ef33edd" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/williamFalcon/f27c7b90e34b4ba88ced042d9ef33edd" target="_blank" rel="nofollow noopener">https://gist.github.com/williamFalcon/f27c7b90e34b4ba88ced042d9ef33edd</a></h4>
<h5>Pytorch_LSTM_variable_mini_batches.py</h5>
<pre><code class="Python">import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.nn import functional as F

"""
Blog post:
Taming LSTMs: Variable-sized mini-batches and why PyTorch is good for your health:
https://medium.com/@_willfalcon/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e
"""</code></pre>
This file has been truncated. <a href="https://gist.github.com/williamFalcon/f27c7b90e34b4ba88ced042d9ef33edd" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>is random initialization the correct practice?</p>

sharvil: <p>In this case, the author is treating the initial state as a learned value (<a href="https://gist.github.com/williamFalcon/f27c7b90e34b4ba88ced042d9ef33edd#file-pytorch_lstm_variable_mini_batches-py-L65-L66" rel="nofollow noopener">see this block</a> of code). In that case, it makes sense to use a randomly initialized vector to break symmetry, just like any other parameter.</p>
<p>Learned initial states are atypical – most architectures I’ve come across use a zero initial state. In PyTorch, you would just omit the second argument to the <code>LSTM</code> object.</p>

vdw: <p>I can’t see the model learning the initial state. <code>init_hidden()</code> gets called for every call of the <code>forward()</code> method, i.e., for each batch. So each batch starts with a new random initial state. What exactly is learned here?</p>
<p>As far as I can tell, the learning the initial state is done by initializing the hidden state once when creating the model, and then you only <code>detach()</code> the hidden state for each new batch.</p>
<p>Creating a new random hidden state for each batch probably doesn’t hurt much – I don’t know, to be honest. But it certainly results in the case where the same example – input sequence and target sequence/class) is trained with different initial initial hidden states. Not sure which effects this has.</p>

sharvil: <p>Can’t be sure without consulting the author, but I think the <em>intent</em> was to treat the initial state as a learned value. But you’re right that the implementation doesn’t do that since <code>init_hidden()</code> is called in <code>forward()</code> (which I missed). What they probably should’ve done is called <code>init_hidden()</code> once inside <code>__build_model()</code> and not reassigned <code>self.hidden</code>.</p>

vdw: <p>Out of curiosity, I trained a simple binary classifier (LSTM with attention) on a text dataset of mine. Once with random initialization, once with zero’d initialization of the hidden state for each batch:</p>
<pre><code>ZEROES
Training accuracy: 0.970, test accuracy: 0.840
Training accuracy: 0.975, test accuracy: 0.841
Training accuracy: 0.970, test accuracy: 0.840

RANDOM
Training accuracy: 0.927, test accuracy: 0.847
Training accuracy: 0.923, test accuracy: 0.845
Training accuracy: 0.926, test accuracy: 0.849
</code></pre>
<p>The result are not unexpected, I think. Using zero’d hidden states yields a higher training accuracy since the same sentence never starts with a different hidden state. In fact, all sentences are treated equally given the initial hidden state is the same – I don’t think it’s important that the initial state is all zero’s, it’s just important that it’s the same for each batch (even if it’s set randomly at the very beginning).</p>
<p>The test accuracy is a tad better for a random initialization. I guess one could argue that a random initialization introduces some kind of regularization that avoids overfitting (lower training accuracy) but generalizes a bit better (higher test accuracy).</p>
<p>Disclaimer: This was just a quick-and-dirty test with a simple model and small-ish dataset. Please don’t use these results to make any deeper conclusions :).</p>

david.waterworth: <p>I’ve been confused by this exact example myself - because <code>init_hidden</code> is on <code>forward</code>, it means that not only during training is the initial state (per batch) random, but also during validation and testing?</p>
<p>It seems to me that it’s something you should call in the training loop (per batch or per epoch), but then I’m not sure what initial state you’d use for inference. You probably want to use the final state from the previous batch if you’re predicting from a windowed time-series?</p>