Understanding RNN

Luca_Pamparana: <p>I have not had much experience with RNNs and have been  looking at some examples in the  pytorch repository and I have a question about the example provided here: <a href="https://github.com/pytorch/examples/blob/master/word_language_model/model.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/word_language_model/model.py</a></p>
<p>In this example, the <code>RNNModel</code> forward function looks as follows:</p>
<pre><code class="lang-auto">def forward(self, input, hidden):
        emb = self.drop(self.encoder(input))
        output, hidden = self.rnn(emb, hidden)
        output = self.drop(output)
        decoded = self.decoder(output)
        return decoded, hidden
</code></pre>
<p>I have 2 questions:</p>
<p>1: Why is the <code>output</code> being used in the decoder step? I thought the decoder should take the hidden state as input? Should this not be <code>self.decoder(hidden)</code>?</p>
<p>2: Assuming there is an explanation for (1), why is usually the dropout used on the output? I am guessing the output here is a tensor. So, how is dropout applied on a tensor? I thought dropout is a property of the weights?</p>
<p>I hope my questions make some sense.</p>

ptrblck: <ol>
<li>
<code>output</code> will contain the output features of the last layer in your rnn for all time steps, while <code>hidden</code> will contain the features for the last step.<br>
This small code example shows the equality:</li>
</ol>
<pre><code class="lang-python">rnn = nn.RNN(10, 20)
input = torch.randn(5, 3, 10)
h0 = torch.randn(1, 3, 20)
output, hn = rnn(input, h0)

print((output[-1] == hn).all())
&gt; tensor(true)
</code></pre>
<ol start="2">
<li>Dropout will drop output activations randomly as shown here:</li>
</ol>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
x = torch.randn(1, 10)
out = lin(x)
print(out)
print(F.dropout(out, 0.5))
</code></pre>

Luca_Pamparana: <p>Thank you for your answer! This really clarifies a few things for me. Many thanks.</p>
<p>Looking at the code again last night, there is also one more thing that stuck out. Usually these auto encoder like things have a RNN for encoding and another one for decoding (recon). However, in this code, the decoder is just a linear layer. I wonder why that is?</p>
<pre><code class="lang-auto">self.decoder = nn.Linear(nhid, ntoken)
....
decoded = self.decoder(output)
</code></pre>
<p>Why should the reconstruction be a linear combination of the output? I am trying to wonder what this pytorch example is trying to achieve  with this? I also ask because I have seen similar things in RNN based anomaly detection but am confused about the role of this linear decoder layer.</p>

ptrblck: <p>I’m not sure about the architecture decision for this model and I would guess <a class="mention" href="/u/vdw">@vdw</a> might know more about this particular decoding strategy. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>However, if you are looking for an encoder and decoder model with RNNs in both parts, have a look at the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html">Seq2Seq tutoorial</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vdw: <p>Well, I had a look at the code. While I’m not familiar with this setup for a Language Model (LM) either, a look at the training data made a bit clearer to me. Still, everything that follows are not much more than educated guesses.</p>
<p>Most fundamentally, an LM aims to predicts the next word given an input sequence of words. Most LM architectures reflect this “directly” in the training date. For example, given a long document like “A B C D E F G H I J K L M N O P …”, the training data (input =&gt; output) is generates as (assuming an input sequence of 5)</p>
<pre><code>A B C D E =&gt; F
B C D E F =&gt; G
C D E F G =&gt; H
D E F G H =&gt; I
...
</code></pre>
<p>So yeah, in this case you would simple pump input through the RNN and use the hidden state as input for the final linear layers, e.g.:</p>
<pre><code>output, hidden = self.rnn(emb, hidden)
# optional dropout or what not
decoded = self.decoder(hidden)
</code></pre>
<p>That essentially simply an RNN-based classifier where the classes are all the word in the vocabulary.</p>
<p>The architecture here works differently. If I understand in <code>example/main.py</code> currectly, the training data is generated differently:</p>
<pre><code>A B C D E =&gt; B C D E F
...
</code></pre>
<p>(the structure of subsequent training examples) depend on the batch size; check the methods <code>batchify()</code> and <code>get_batch()</code>. Anyway, the output sequence is the input sequence shifted by one word to the left.</p>
<p>No they treat the whole training as a sequence labeling task, which is kind of a prediction for/after each time time step and not just the last. As <a class="mention" href="/u/ptrblck">@ptrblck</a> said, <code>output[i]</code> is the hidden state of step <code>i</code>. I always use the figure below to check if I use the right return values of an LSTM/GRU:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7496a33d835f085d800ee17c0ade05895a89551" title="lstm-output-vs-hidden"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e7496a33d835f085d800ee17c0ade05895a89551.png" alt="lstm-output-vs-hidden" data-base62-sha1="x03AlD3QYEjqGqTmkDe4za9w3df" width="583" height="500" data-dominant-color="EDEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lstm-output-vs-hidden</span><span class="informations">640×548 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Anyway, that’s how understand the proposed architecture. I assume it yields better results compared to more traditional architectures. I hope that helps…well, at least gives some food for thoughts.</p>

Luca_Pamparana: <p>Thank you for such a detailed reply. This really clears a lot of doubts!</p>