LSTM as decoder

harshraj22: <p>consider the case of machine translation using encoder decoder architecture. Consider the case when nn.Lstm is used as encoder as well as decoder. Assume the number of nn.LstmCell in both is 5.</p>
<p>In case of encoder, during the forward propagation, we send a batch of sentences, and for each sentence, word_i is passed as input to LstmCell_i. This Lstm finally returns the hidden state to decoder.</p>
<p>In case of decoder, for each sentence, a loop is created, whose length is the length of sentence we want to generate. Each word is generated by passing previous word (and hidden state) through Lstm. The lstm with 5 layer is used to <strong>encode a sentence in encoder</strong>, while it is used to <strong>generate a word</strong> in decoder. Does this mean, while decoding, each word has to pass through each 5 LstmCell and the output of final cell predicts the word ?</p>
<p>Please correct me if I am wrong.</p>
<p>ps. I have already gone through <a href="https://discuss.pytorch.org/t/seq2seq-lstm-difference-between-decoders-that-loop-over-sequence-and-those-that-dont/53262">this</a> discussion.</p>

miro: <aside class="quote no-group" data-username="harshraj22" data-post="1" data-topic="108831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harshraj22/48/30954_2.png" class="avatar"> harshraj22:</div>
<blockquote>
<p>In case of encoder, during the forward propagation, we send a batch of sentences, and for each sentence, word_i is passed as input to LstmCell_i.</p>
</blockquote>
</aside>
<p>This isn’t how forward prop of an LSTM works. The word is always passed into LstmCell_0, then the output of LstmCell_0 goes into LstmCell_1, etc.</p>

harshraj22: <p>Can you give a little more detail ?<br>
and so, the final prediction for each word comes from the output of last LstmCell ? And does this mean images like <a href="https://images.app.goo.gl/cK1wFMVG4ho3ePZN8" rel="noopener nofollow ugc">this</a> about Lstm are completely misleading ? (which show that each word is fed to each LstmCell) ?</p>

miro: <blockquote>
<p>each word is fed to each LstmCell</p>
</blockquote>
<p>Those are in fact the same LstmCell unrolled, not different LstmCells.</p>

vdw: <aside class="quote no-group" data-username="harshraj22" data-post="1" data-topic="108831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harshraj22/48/30954_2.png" class="avatar"> harshraj22:</div>
<blockquote>
<p>Assume the number of nn.LstmCell in both is 5.</p>
</blockquote>
</aside>
<p>That’s probably your misunderstanding. There is not a list of cells, but only cell that is recurrently used until a stop condiation is met – for the encoder, when the last word/item in the sequences as been processed.</p>
<p>The output of the cell at time i is used as input of the cell at the next time step i+1.</p>
<p>The illustrations you’ve linked just show the “unrolled” network to better visualize the sequential nature of the processing. All the cells are indeed the same cell. See the image below: both representations show the same network – left with the recurrent connection and right unrolled to better see how the sequence is processed.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a86a0e70fe66032e749e3ece6d108ef23dbfd188.png" data-download-href="https://discuss.pytorch.org/uploads/default/a86a0e70fe66032e749e3ece6d108ef23dbfd188" title="RNN-unrolled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a86a0e70fe66032e749e3ece6d108ef23dbfd188_2_690x181.png" alt="RNN-unrolled" data-base62-sha1="o1RrgMrd4rpRgNjYsGvDESTU44g" width="690" height="181" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a86a0e70fe66032e749e3ece6d108ef23dbfd188_2_690x181.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a86a0e70fe66032e749e3ece6d108ef23dbfd188_2_1035x271.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a86a0e70fe66032e749e3ece6d108ef23dbfd188_2_1380x362.png 2x" data-dominant-color="9BA69C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RNN-unrolled</span><span class="informations">2706×711 77.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>