LSTM dropout - Clarification of Last Layer

username: ronrest
cooked: <p>In the <a href="http://pytorch.org/docs/master/nn.html#torch.nn.LSTM" rel="noopener nofollow ugc">documentation for LSTM</a>, for the <code>dropout</code> argument, it states:</p>
<blockquote>
<p>introduces a dropout layer on the outputs of each RNN layer except the last layer</p>
</blockquote>
<p>I just want to clarify what is meant by “everything except the last layer”.</p>
<p>Below I have an image of two possible options for the meaning.</p>
<ul>
<li><strong>Option 1:</strong>  The final cell is the one that does not have dropout applied for the output.</li>
<li><strong>Option 2:</strong>  In a multi-layer LSTM, all the connections between layers have dropout applied, except the very top layer.  So a single layer LSTM would not have any dropout applied.</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/62f94ceee433b693ef73be231f51ae4291e53880.png" data-download-href="https://discuss.pytorch.org/uploads/default/62f94ceee433b693ef73be231f51ae4291e53880" title="diagram.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62f94ceee433b693ef73be231f51ae4291e53880_2_288x500.png" width="288" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62f94ceee433b693ef73be231f51ae4291e53880_2_288x500.png, https://discuss.pytorch.org/uploads/default/original/2X/6/62f94ceee433b693ef73be231f51ae4291e53880.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/62f94ceee433b693ef73be231f51ae4291e53880.png 2x" data-dominant-color="C8C7AC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">diagram.png</span><span class="informations">301×521 27.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried looking at the source code for LSTM and RNNBase, but I can’t figure out how it is being applied.</p>

username: dgriff
cooked: <p>LSTM is a rolled up LSTMCell and to think of each layer as Performing one LSTMCell action and that should help make sense of it. Number layers being the number of time steps performed</p>

username: tom
cooked: <p>Hello,</p>
<p>from both the text you quoted and the code, I’d say likely option 2. This is my understanding of layers.</p>
<p>If you believe that, after disabling cudnn, this line is relevant, it might also give a hint.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/rnn.py#L91" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/rnn.py#L91" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/nn/_functions/rnn.py#L91</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="81" style="counter-reset: li-counter 80 ;">
<li>    for j, inner in enumerate(inners):</li>
<li>        l = i * num_directions + j</li>
<li>
</li>
<li>        hy, output = inner(input, hidden[l], weight[l])</li>
<li>        next_hidden.append(hy)</li>
<li>        all_output.append(output)</li>
<li>
</li>
<li>    input = torch.cat(all_output, input.dim() - 1)</li>
<li>
</li>
<li>    if dropout != 0 and i &lt; num_layers - 1:</li>
<li class="selected">        input = F.dropout(input, p=dropout, training=train, inplace=False)</li>
<li>
</li>
<li>if lstm:</li>
<li>    next_h, next_c = zip(*next_hidden)</li>
<li>    next_hidden = (</li>
<li>        torch.cat(next_h, 0).view(total_layers, *next_h[0].size()),</li>
<li>        torch.cat(next_c, 0).view(total_layers, *next_c[0].size())</li>
<li>    )</li>
<li>else:</li>
<li>    next_hidden = torch.cat(next_hidden, 0).view(</li>
<li>        total_layers, *next_hidden[0].size())</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
My understanding of this code is that it does a single timestep (your horizontal axis) for many layers.</p>
<p>Best regards</p>
<p>Thomas</p>

username: dgriff
cooked: <p>Time steps I’m referring from going from each element in the input. Like if you fed a sentence as sequence the order of the words in that sentence matters. Hence why the pytorch docs say:</p>
<p>“For each element in the input sequence, each layer computes the following function”</p>
<p>Nn.LSTM is a window LSTM Aka rolled up LSTM</p>
<p>And would be option 2</p>
<p>Which sorry I misread you Tom thought you were disagreeing with that but read incorrectly and what I wrote was wrong it is multilayer for each time step😲</p>

username: STU
cooked: <p>Hi, if it means the Option-1, why does the num_layers must great than 1?</p>