LSTM input shape with "batches"

username: zabulon
cooked: <p>Hello!</p>
<p>I am trying to understand how the “N = batch size” option works for a LSTM (<a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">doc</a>) and I find it a bit confusing.</p>
<p>Apparently, this works:</p>
<blockquote>
<p>import torch<br>
from torch.nn import Embedding, LSTM</p>
<p>num_chars = 8<br>
batch_size = 2<br>
embedding_dim = 3<br>
hidden_size = 5<br>
num_layers = 1</p>
<p>embed = Embedding(num_chars, embedding_dim)<br>
lstm = LSTM(input_size=embedding_dim, hidden_size=hidden_size)</p>
<p>hiddens = torch.zeros(num_layers, 8 // batch_size, hidden_size)<br>
cells = torch.zeros(num_layers, 8 // batch_size, hidden_size)</p>
<p>sequence = torch.tensor([0,1,2,3,4,5,6,7])</p>
<p>batched = sequence.view((batch_size, -1))</p>
<p>embedded = embed(batched)<br>
last_hidden, (hiddens, cells) = lstm(embedded, (hiddens, cells))</p>
</blockquote>
<p>The hidden state shape is (2, 4, 5) and the cell shape is (1, 4, 5) which sounds right.<br>
But I had to use <code>8/batch_size</code> when setting up the initial hidden and cell states, when the doc says <code>h_0</code> should be of shape (1, <code>batch_size</code>, H_out). My <code>batch_size</code> is 2, not 4.</p>
<p>What am I missing?</p>

username: ptrblck
cooked: <p>Your <code>embedded</code> activation has a shape of <code>[2, 4, 3]</code> corresponding to <code>[batch_size, seq_len, embedding_dim]</code>. However, <code>nn.LSTM</code> expects inputs in <code>[seq_len, batch_size, input_size]</code> by default.<br>
Thus, <code>dim1</code> with a size of <code>4</code> is used as the batch size raising the error.<br>
You should <code>permute</code> the <code>embedded</code> activation or use <code>batch_first=True</code> in the <code>nn.LSTM</code> module:</p>
<pre><code class="lang-python">hiddens = torch.zeros(num_layers, batch_size, hidden_size)
cells = torch.zeros(num_layers, batch_size, hidden_size)

sequence = torch.tensor([0,1,2,3,4,5,6,7])

batched = sequence.view((batch_size, -1))

embedded = embed(batched).permute(1, 0, 2).contiguous()
last_hidden, (hiddens, cells) = lstm(embedded, (hiddens, cells))
</code></pre>
<pre><code class="lang-python">embed = Embedding(num_chars, embedding_dim)
lstm = LSTM(input_size=embedding_dim, hidden_size=hidden_size, batch_first=True)

hiddens = torch.zeros(num_layers, batch_size, hidden_size)
cells = torch.zeros(num_layers, batch_size, hidden_size)

sequence = torch.tensor([0,1,2,3,4,5,6,7])

batched = sequence.view((batch_size, -1))

embedded = embed(batched)
last_hidden, (hiddens, cells) = lstm(embedded, (hiddens, cells))
</code></pre>

username: zabulon
cooked: <p>Actually, the code runs perfectly well - I’m just wondering if the doc is correct about <code>N</code>. Or maybe I got lucky it runs, although I have the wrong setup? But I don’t see an error on my side.</p>
<p>For example, with no error raised, I see <code>last_hidden</code> come out as having shape <code>(2,4,5)</code>.</p>

username: zabulon
cooked: <p>Oh, and one more question: when using batches that way with an LSTM, is it simply equivalent to running multiple separate LSTM’s in parallel? The <code>hiddens</code> and <code>cells</code> take on one extra dimension, which seems to suggest there are just <code>batch_size</code> LSTM’s running in parallel <strong>independently</strong>.</p>

username: J_Johnson
cooked: <p>Basically, it is equivalent to multiple instances running in parallel. Albeit, there are not multiple copies of the model made when performing the forward pass with a batch size &gt; 1.</p>
<p>Generally speaking, a neural network will train more efficiently and reduce overfitting by accumulating the gradients over different examples before back propagation.</p>
<p>Additionally, the higher the batch size, the higher the learning rate you can use without getting exploding gradients.</p>
<p>And, lastly, running batches in parallel allows you to take advantage of multiprocessing in parallel, which is especially beneficial if you’re using a GPU.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="zabulon" data-post="3" data-topic="189843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zabulon/48/64539_2.png" class="avatar"> zabulon:</div>
<blockquote>
<p>I’m just wondering if the doc is correct about <code>N</code>. Or maybe I got lucky it runs, although I have the wrong setup?</p>
</blockquote>
</aside>
<p>Yes, the docs are correct and you are lucky by changing the shape randomly executing a wrong setup. Check my code snippet and print all shapes to understand where the issue is.</p>

username: zabulon
cooked: <p>Got it, makes sense. Yeah, I just got lucky that the mixed-up dimensions were just still working by chance. Thanks!</p>

username: zabulon
cooked: <p>I’m confused - am I training 1 LSTM or 4 if my batch_size is 4? Are the internal W and gate matrices shared between the 4 batches, or separate?</p>

username: J_Johnson
cooked: <p>“Shared”. I use that word very loosely here. What really happens is no different than any other network with a batch size &gt; 1.</p>
<p>The loss function determines what occurs with autograd on the batch dim. For instance, L1Loss has a reduction parameter whose default is “mean”. That means the mean of the gradients across the batch dim is what is computed for that loss function configuration. See here:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html</a></p>
<p>You calculate the loss prior to backpropagation. Nothing with the parameters change until backprop(i.e. <code>loss.backward()</code> and <code>optimizer.step()</code>).</p>

username: zabulon
cooked: <p>Yeah, ok, makes sense, but when backprop takes place with <code>.backward</code>, is it going to update as many internal W matrices in the LSTM as there are batches? Or just one, no matter what the batch size is?</p>

username: J_Johnson
cooked: <p>Just one. There is only one copy of the parameters. The gradients get averaged over the batch and sequence dims for each data sampling given to the model.</p>