Questions about the implement of nn.LSTM

MarcusNerva: <p>The shape of input into LSTM is (seq_len, batch, inputsize). And in the course of deep learning, I was told that step t will utilize the information from step t - 1. But after read both the Python and C++ source code, I didn’t find the iteration on ‘time’(i.e. the step on seq_len). And it seemed that all the time step of input features are calculated in parallel. So how exactly the information is pass from time step t to step t - 1?  And what is the difference between nn.LSTM and nn.LSTMCell? Does nn.LSTM is just a multi-layer version of nn.LSTMCell?</p>
<p>I’m so confused <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>

AbdulsalamBande: <p>Lets take an example . If the input dimension was  <code>(batch,seq_len, inputsize)</code>. Each batch goes through the below formular in parallel with other batches. Example. If we have an input of shape <code>(2,2,2)</code>. From left, batch=2, seqlen=2, and inputsize=2.</p>
<pre><code class="lang-auto">tensor([[[0.6451, 0.8479],
         [0.3435, 0.2441]],

        [[0.0185, 0.4722],
         [0.8744, 0.2785]]])
</code></pre>
<pre><code class="lang-auto">[0.6451, 0.8479], = SEQ1   BATCH 1
         [0.3435, 0.2441] = SEQ2 BATCH 1
</code></pre>
<pre><code class="lang-auto">[0.0185, 0.4722],  SEQ1  BATCH 2
         [0.8744, 0.2785].   SEQ2  BATCH 2
</code></pre>
<p><strong>TIME1:</strong>  SEQ1 BATCH1 Will go into the below formular AND at the same time SEQ1 BATCH2 will go in the below formular separately. So They will output two <code>h&lt;t&gt;</code></p>
<p><strong>TIME2:</strong> SEQ2 BATCH1 Will go into the below formular AND at the same time SEQ2 BATCH2 will go in the below formular BUT the <code>h&lt;t-1&gt;</code>  for SEQ2 BATCH1 will be the <code>h&lt;t&gt;</code> of  SEQ1 BATCH1. Similarly the <code>h&lt;t-1&gt;</code> for SEQ2 BATCH2 will be the <code>h&lt;t&gt;</code> of  SEQ1 BATCH2</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8b9c42dd8a508f58fad108458967ad37d0aebf8.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8b9c42dd8a508f58fad108458967ad37d0aebf8" title="Screen Shot 2021-03-16 at 10.51.49 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8b9c42dd8a508f58fad108458967ad37d0aebf8_2_689x285.png" alt="Screen Shot 2021-03-16 at 10.51.49 PM" data-base62-sha1="sDHuMWMz8qcMvPhhp2h3sieYMgo" width="689" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8b9c42dd8a508f58fad108458967ad37d0aebf8_2_689x285.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8b9c42dd8a508f58fad108458967ad37d0aebf8_2_1033x427.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8b9c42dd8a508f58fad108458967ad37d0aebf8.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-03-16 at 10.51.49 PM</span><span class="informations">1196×494 33.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I hope it helps. Similar to torch.lstm is torch.rnn. You can watch my youtube video on how it works with pen and paper<br>
<a href="https://www.youtube.com/watch?v=4p9SXDSwxqA&amp;ab_channel=AbdulsalamBande" rel="noopener nofollow ugc">torch.rnn explained with Pen and Paper</a></p>
<p>For the difference between <code>nn.LSTM</code>  and    <code>nn.LSTMCell</code> : <code>nn.LSTMCell</code> is just a single implementation of the above formular while <code>nn.LSTM</code> is a layer that applies <code>nn.LSTMCell</code> in a for loop.<br>
Reference <a href="https://stackoverflow.com/questions/57048120/pytorch-lstm-vs-lstmcell" rel="noopener nofollow ugc">Pytorch LSTM vs LSTMCell</a></p>

googlebot: <aside class="quote no-group" data-username="MarcusNerva" data-post="1" data-topic="114999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marcusnerva/48/35468_2.png" class="avatar"> MarcusNerva:</div>
<blockquote>
<p>But after read both the Python and C++ source code, I didn’t find the iteration on ‘time’(i.e. the step on seq_len)</p>
</blockquote>
</aside>
<p>this loop exists, at the deepest level usually</p>
<aside class="quote no-group" data-username="MarcusNerva" data-post="1" data-topic="114999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marcusnerva/48/35468_2.png" class="avatar"> MarcusNerva:</div>
<blockquote>
<p>And it seemed that all the time step of input features are calculated in parallel.</p>
</blockquote>
</aside>
<p>This is correct, non-recurrent input-to-hidden computations can be performed separately for better performance. Results of this are later used in hidden-to-hidden computation loop.</p>

MarcusNerva: <p>It seems that this is the deepest layer I find, but still got no loops on time.<br>
In pytorch/aten/src/ATen/native/RNN.cpp</p>
<pre><code class="lang-auto">template &lt;typename cell_params&gt;
struct LSTMCell : Cell&lt;std::tuple&lt;Tensor, Tensor&gt;, cell_params&gt; {
  using hidden_type = std::tuple&lt;Tensor, Tensor&gt;;

  hidden_type operator()(
      const Tensor&amp; input,
      const hidden_type&amp; hidden,
      const cell_params&amp; params,
      bool pre_compute_input = false) const override {
    const auto&amp; hx = std::get&lt;0&gt;(hidden);
    const auto&amp; cx = std::get&lt;1&gt;(hidden);

    if (input.is_cuda()) {
      TORCH_CHECK(!pre_compute_input);
      auto igates = params.matmul_ih(input);
      auto hgates = params.matmul_hh(hx);
      auto result = at::_thnn_fused_lstm_cell(
          igates, hgates, cx, params.b_ih(), params.b_hh());
      // applying projections if w_hr is defined
      auto hy = params.matmul_hr(std::get&lt;0&gt;(result));
      // Slice off the workspace argument (it's needed only for AD).
      return std::make_tuple(std::move(hy), std::move(std::get&lt;1&gt;(result)));
    }

    const auto gates = params.linear_hh(hx).add_(
        pre_compute_input ? input : params.linear_ih(input));
    auto chunked_gates = gates.unsafe_chunk(4, 1);
    auto ingate = chunked_gates[0].sigmoid_();
    auto forgetgate = chunked_gates[1].sigmoid_();
    auto cellgate = chunked_gates[2].tanh_();
    auto outgate = chunked_gates[3].sigmoid_();
    auto cy = (forgetgate * cx).add_(ingate * cellgate);
    auto hy = outgate * cy.tanh();
    hy = params.matmul_hr(hy);
    return std::make_tuple(std::move(hy), std::move(cy));
  }

};
</code></pre>
<p>However, I can find loops on lstm layers:</p>
<pre><code class="lang-auto">template&lt;typename io_type, typename hidden_type, typename weight_type&gt;
LayerOutput&lt;io_type, std::vector&lt;hidden_type&gt;&gt;
apply_layer_stack(const Layer&lt;io_type, hidden_type, weight_type&gt;&amp; layer, const io_type&amp; input,
                  const std::vector&lt;hidden_type&gt;&amp; hiddens, const std::vector&lt;weight_type&gt;&amp; weights,
                  int64_t num_layers, double dropout_p, bool train) {
  TORCH_CHECK(num_layers == (int64_t)hiddens.size(), "Expected more hidden states in stacked_rnn");
  TORCH_CHECK(num_layers == (int64_t)weights.size(), "Expected more weights in stacked_rnn");

  auto layer_input = input;
  auto hidden_it = hiddens.begin();
  auto weight_it = weights.begin();
  std::vector&lt;hidden_type&gt; final_hiddens;
  for (int64_t l = 0; l &lt; num_layers; ++l) {
    auto layer_output = layer(layer_input, *(hidden_it++), *(weight_it++));
    final_hiddens.push_back(layer_output.final_hidden);
    layer_input = layer_output.outputs;

    if (dropout_p != 0 &amp;&amp; train &amp;&amp; l &lt; num_layers - 1) {
      layer_input = dropout(layer_input, dropout_p);
    }
  }

  return {layer_input, final_hiddens};
}
</code></pre>
<p>Still very confused. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>

googlebot: <p>You’re looking at a convoluted c++ implementation, where cells (i.e. 1 step code) are “functors” and the loop is higher.<br>
In the same file, look at two FullLayer::operator() overloads, notice that inputs.unbind(0) produces a list, and there is a loop over step_inputs in other overload.</p>