Bidirectional RNN predicts padding for each timestep

montyhall: <p>Hi</p>
<p>I have the following Bidirectional LSTM language model</p>
<pre><code>drop = nn.Dropout(dropout)
word_embeddings = nn.Embedding(vocabSize, input_size)
rnn = nn.LSTM(input_size,
              hidden_size,
              n_layers,
              dropout=dropout,
              bidirectional=True)

fc = nn.Linear(hidden_size*numDir, vocabSize)
</code></pre>
<p>Assume the following</p>
<pre><code>vocabSize=107
batchsize=4
input_size=100
hidden_size=200
n_layers=1
dropout=0.1
bidir=True
numDir = 2 
</code></pre>
<p>My vocab is a list of <strong>107</strong> tokens with 0th position as <code>&lt;pad&gt;</code>.</p>
<p>I am using <em><strong>pack_padded_sequence</strong></em> and <strong>pad_packed_sequence</strong> to pack and unpack my input in the forward()</p>
<pre><code>emb = drop(word_embeddings(input))
packed_input = pack_padded_sequence(emb,seqLengths)
packed_output, (hT, cT) = rnn(packed_input, hidden)
ht, _ = pad_packed_sequence(packed_output)
logits = fc(ht.view(ht.size(0)*ht.size(1), ht.size(2)))
</code></pre>
<p>My input dimensions are <strong>(T,B,*)</strong>, where <strong>T</strong> is sequence length, <strong>B</strong> is batch size. The inputs are left aligned 0padded and sorted by their input lengths:</p>
<p><code>seqLengths = [19, 16, 16, 16]</code></p>
<p>for a particular batch. The dimensions I observe after forward() are</p>
<pre><code>input	= (19,4)
emb 	= (19,4,100)
ht	    = (19,4,400)
logits	= (76,107)
</code></pre>
<p>which all make perfect sense. And the model seems to work fine in left-right single direction. However, in the bidirectional mode the model predicts <code>&lt;pad&gt;</code> for every position of the sequence.</p>
<p>One hypothesis I was working with was that the padding, being the last element (and 0th position in the vocab) is killing the gradients in the backward pass. So I attempted to use the packed_output directly for the FC layer directly (i.e not running pad_packed_sequence()). But that creates dimension mismatches. I also tried these dimension selection methods that pass the sequence lengths but select only the last element of the sequences ( [<a href="https://discuss.pytorch.org/t/about-the-variable-length-input-in-rnn-scenario/345/22">About the variable length input in RNN scenario</a>](<a href="https://discuss.pytorch.org/t/about-the-variable-length-input-in-rnn-scenario/345/22">About the variable length input in RNN scenario</a> and <a href="https://discuss.pytorch.org/t/indexing-multi-dimensional-tensors-based-on-1d-tensor-of-indices/1391">Indexing Multi-dimensional Tensors based on 1D tensor of indices</a>. However, I need the full elements of the tensor, not just the last none-zero.</p>
<p>Am I thinking incorrectly about this problem? why is the bidirectional lstm always predicting the padding token in every timestep in the setup above?</p>
<p>thank you</p>

sumeet_baghel: <p>I came across this post beacuse i had a problem in taking the output of the padded input with bidirectional lstm and your case is the same.<br>
I might not have have understood your problem completely but here’s what i think,  You don’t have to pass the ‘ht’ as it is because it is a collection of hidden states over the timesteps.What you could do is select first and last timestep vectors and remove padding from both of them, concatenate and then pass it to the FC.<br>
Please correct if i’m wrong and also reply if you found another way around.</p>