RNN Architecture Questions

username: Wesley_Neill
cooked: <p>I’m learning about RNNs from the Udacity PyTorch class, and I think I understand the basic concept. Now I’m trying to get myself familiar with the details, starting with understanding the dimensions of inputs and outputs of different layers in an RNN.</p>
<ul>
<li>
<strong>input_size</strong>  –  The number of expected features in the input x.</li>
<li>
<strong>hidden_size</strong>  – The number of features in the hidden state h.</li>
<li>
<strong>num_layers</strong>  – Number of recurrent layers.</li>
</ul>
<p>I copied these parameter descriptions from the docs, but I’m having a hard time visualizing my network based on these parameters.</p>
<ol>
<li>What do the docs mean by “features in the input x”? Does it refer to having multiple variables? Or like an image might have multiple channels?</li>
<li>What are “features in the hidden state”? Again, the word “features” seems pretty ambiguous, and I don’t know what it means at all.</li>
</ol>
<p>I get that the hidden state has some “memory” of its previous self and is updated by combining the current input to the hidden layer with the previous hidden state. I don’t get the specifics of what’s going on though. For example, here is a question from the class, which I have no idea how to answer:</p>
<p>Say you’ve defined a GRU layer with  <code>input_size = 100</code> ,  <code>hidden_size = 20</code> , and  <code>num_layers=1</code> . What will the dimensions of the hidden state be if you’re passing in data, batch first, in batches of 3 sequences at a time?</p>
<p>Unfortunately, RNNs are not explained as well as CNNs were in the previous lesson. Thanks for your help!</p>

username: vdw
cooked: <p>Very crudely speaking, number of features refer to the size of the vectors/tensors.</p>
<p>Outside end-to-end neural networks, the term feature had a more tangible meaning, since feature engineering was an import processing step you had to do “manually”. For example, to use a SVM to classify a text document had extract the text document into a set of features. This could some very naive features such as <span class="hashtag">#words</span>, <span class="hashtag">#characters</span>. In this case, each document would be represented by 2 numerical features. In practice, you would have more meaningful features but they would have a clear semantic meaning.</p>
<p>With end-to-end neural networks, this semantic meaning is usually latent and not obvious. For example, each word in a text may be represented by a 300-dim vector of numerical values, i.e., the word has 300 features. This vector places a word in a 300-dim space in relation to other words. However, you usually don’t really know what an individual value in the vector means. For example, The 25th entry in the 300-dim vector does not tell you that the word is a noun, verb, adjective, etc.</p>
<p>So in case you use an LSTM for text processing where the LSTM processes a sequences of words, each words is represented by a vector of size <code>input_size</code>. This representation is needed since word are symbolic representations. For example, the words “cat” and “kitten” are only similar to you because your mental models of both concepts (animal, 4 legs, furry, meows, etc.) are similar. For a computer, these are completely different things. A vector representation now allows to map “cat” and “kitten” (and all other word) to a numerical representation where the vector for “cat” and the vector for “kitten” are closer together compared to, say, the vector of “cat” and the vector of “train”.</p>
<p>In contrast, if you use an LSTM for time series prediction of already scalar numerical values, <code>input_size</code> is just 1.</p>
<p>For the resulting dimensions of the hidden state, you best consult the PyTorch docs, but it definitely does not depend on the <code>input_size</code>.</p>

username: Wesley_Neill
cooked: <p>I’m afraid I don’t quite understand your explanation. I’m sorry!</p>
<p>I’ve scoured the PyTorch RNN docs, and there is little to no explanation of how data should be formatted for input, and certainly no information about how the shape of the data is changing as it moves through the network.</p>
<p>For example, I THINK my forward pass I need to have my data in the shape <code>(batch_size, seq_length, input_size)</code>, but there are no concrete examples of what <code>seq_length</code> is or what <code>input_size</code> is. How are they different?</p>

username: vdw
cooked: <p>Since you posted in <code>nlp</code>, I assume you work with text.</p>
<p>A very common application is sentence classification (e.g., for sentence classification), where each sentence is a sequence of words. Let’s say you have a batch 3 sentences, each containing 10 words (<code>nn.LSTM</code> and <code>nn.GRU</code> require by default sequences of the same length; you can look up <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284/15"><code>padding</code> and packing</a>)</p>
<p>That means your batch has the shape <code>(batch_size, seq_len)</code>, i.e., <code>(3, 10)</code> with the numbers above. Not that each sentence/sequence is a vector of integers reflecting the index of a word in your vocabulary.</p>
<p>The next step is to push the batch through a <code>nn.Embedding</code> layer to map words (represented by ther indices) to word vectors of size, say, <code>100</code>. The output shape after the embedding layer is then <code>(batch_size, seq_len, embed_dim)</code>, i.e., <code>(3, 10, 100)</code> with the numbers above.</p>
<p>This tensor can now serve as input for your <code>nn.LSTM</code> or <code>nn.GRU</code> which expect as input <code>(batch_size, seq_len, input_size)</code> – not that by default, they actually expect <code>(seq_len, batch_size, input_size)</code>; so either you <code>transform()</code> for tensor or you define your RNN layer with <code>batch_first=True</code>.</p>
<p>Anyway, <code>embed_dim</code>, i.e., the size of your word vectors defines <code>input_size</code>, 100 in the example above. Summing up</p>
<ul>
<li>
<code>batch_size</code> is the number of sentences in your batch (e.g., 3)</li>
<li>
<code>seq_len</code> is the number of items in your sequences such as words in a sentence (e.g., 10)</li>
<li>
<code>input_size</code> is the size of the tensor/vector that represents a single(!) item in your sequence such as 100-dim word vectors for each word in a sentence.</li>
</ul>
<p>The shape of inputs and outputs are very well defined; see, for example, for <a href="https://pytorch.org/docs/master/generated/torch.nn.LSTM.html" rel="nofollow noopener"><code>nn.LSTM</code></a>.</p>