How to use the hidden state in Pytorch for classification vs seq2seq problems

Zorojuro: <p>Hi everyone, am new to pytorch…</p>
<p>What type of problems require the following setup (<em>1 and 2</em>) for a <strong>LSTM</strong>?</p>
<pre><code class="lang-auto">   self.lstm = nn.LSTM(**args)

   hidden = None

1. out, _= self.lstm(x, hidden )

2. out, hidden = self.lstm(x, hidden )
</code></pre>
<p>Which ones can be used for <em>classification</em> and one for <em>sequence to sequence</em> between 1 and 2?</p>

Zorojuro: <p><a class="mention" href="/u/aladdinperzon">@AladdinPerzon</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> may you kindly assist me?</p>

vdw: <p>I think, the picture below helps the most:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7496a33d835f085d800ee17c0ade05895a89551" title="lstm-output-vs-hidden"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" alt="lstm-output-vs-hidden" data-base62-sha1="x03AlD3QYEjqGqTmkDe4za9w3df" width="583" height="500" data-dominant-color="EDEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm-output-vs-hidden</span><span class="informations">640×548 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Your <code>out</code> is the <code>output</code> in this image and contains the <strong>hidden states for each timestamp</strong>. If you have an LSTM with multiple layers, they will be the hidden states of last layer; and it gets a tad more complicated in case of a bidirectional LSTM. This is why the default shape of <code>output</code> is <code>(seq_len, batch, num_directions * hidden_size)</code></p>
<p>Your <code>hidden</code> is the tuple <code>(h_n, c_n)</code> in the image. For further processing, you use <code>h_n</code> which contains the <strong>last hidden state w.r.t. to your sequence</strong>, i.e., the hidden state after the last time step. If you use multi layers, it will contain the last hidden states for all layers. Again, Bi-LSTM adds a little complexity. The default shape of <code>h_n</code> – <code>(num_layers * num_directions, batch, hidden_size)</code> – reflects this.</p>
<p>There are no hard rules when to use <code>output</code> or <code>h_n</code>. It depends on  your task and your network model, i.e., what kind of information you want to consider. For example, for basic classification, the default method is to use the last hidden state <code>h_n</code> (technically, <code>h_n[-1]</code> to get the last hidden state to the last layer). But you can also use <code>output</code> and sum or average the hidden states of all time steps.</p>

tommywolfheart: <p>Hi <a class="mention" href="/u/vdw">@vdw</a> . What would be the purpose of using the sum or average of the hidden states? Does it capture a better representation of the sequence somehow? I’m trying to figure out what to do with my implementation of my LSTM and can’t really find any theory discussing what to feed into the fully connected layer or why. Is there any method to the madness?</p>

tommywolfheart: <p>I’ll answer my question with, from <a href="https://web.stanford.edu/~jurafsky/slp3/9.pdf" rel="noopener nofollow ugc">https://web.stanford.edu/~jurafsky/slp3/9.pdf</a>, this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed4855e68c335ee2359f78200ee83da7c9a34f18.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed4855e68c335ee2359f78200ee83da7c9a34f18" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed4855e68c335ee2359f78200ee83da7c9a34f18_2_590x500.png" alt="image" data-base62-sha1="xR68B93YJxsqFbHaoTtkFxqY40E" width="590" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed4855e68c335ee2359f78200ee83da7c9a34f18_2_590x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed4855e68c335ee2359f78200ee83da7c9a34f18.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed4855e68c335ee2359f78200ee83da7c9a34f18.png 2x" data-dominant-color="E8E7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">705×597 97.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>