Input and Target size mismatch in RNNs

username: scarecrow21
cooked: <p>I am trying to implement a univariate time series model using vanilla RNNs. Following is the error I encounter:</p>
<pre><code class="lang-auto">/home/ggmu/anaconda3/envs/toothless/lib/python3.6/site-packages/torch/nn/modules/loss.py:443: UserWarning: Using a target size (torch.Size([5, 1, 1])) that is different to the input size (torch.Size([5, 10, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<p>Input size = (90, 10)<br>
output size = (90, 1)</p>
<p>Config:</p>
<pre><code class="lang-auto">batch_size = 5
input_size = 1
sequence_length = 10
hidden_size = 1
num_layer = 3
</code></pre>
<p>The RNN class:</p>
<pre><code class="lang-auto">class ModelRnn(nn.Module):
    def __init__(self):
        super(ModelRnn, self).__init__()
        
        self.rnn = nn.RNN(input_size=input_size, hidden_size=hidden_size, batch_first=True, num_layers=num_layer)
        
    def forward(self, x, hidden):
        hidden = torch.zeros(num_layer, batch_size, hidden_size)

        x = x.view(batch_size, sequence_length, input_size)
        out, hidden = self.rnn(x, hidden)
        
        return hidden, out
</code></pre>
<p>The data loader:</p>
<pre><code class="lang-auto">train_data = trainData(torch.FloatTensor(X_train), torch.FloatTensor(y_train))
train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=False, num_workers=2)
model = ModelRnn()
print(model)

#    ModelRnn(
#    (rnn): RNN(1, 1, num_layers=3, batch_first=True)
#   )

</code></pre>
<p>Optimizer:</p>
<pre><code class="lang-auto">optimizer=torch.optim.Adam(model.parameters(),lr=0.01)
criterion=nn.MSELoss()
</code></pre>
<p>Train loop:</p>
<pre><code class="lang-auto">for epoch in range(20):
    for x_batch, y_batch in train_loader:
        optimizer.zero_grad()
        hidden, output = model(x_batch, hidden)
        y_batch = y_batch.view(-1, 1, 1)
        loss = criterion(output,  y_batch)
        loss.backward()
        optimizer.step()
        
    print(f"{epoch+1} epoch | loss = {loss}")
</code></pre>
<p>Following is the output + error:</p>
<pre><code class="lang-auto">1 epoch | loss = 9323.71484375

/home/ggmu/anaconda3/envs/toothless/lib/python3.6/site-packages/torch/nn/modules/loss.py:443: UserWarning: Using a target size (torch.Size([5, 1, 1])) that is different to the input size (torch.Size([5, 10, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

2 epoch | loss = 9261.478515625
3 epoch | loss = 9235.8857421875
4 epoch | loss = 9227.794921875
5 epoch | loss = 9224.625
6 epoch | loss = 9222.990234375
7 epoch | loss = 9221.98046875
8 epoch | loss = 9221.2890625
9 epoch | loss = 9220.783203125
10 epoch | loss = 9220.400390625
11 epoch | loss = 9220.099609375
12 epoch | loss = 9219.857421875
13 epoch | loss = 9219.6591796875
14 epoch | loss = 9219.4951171875
15 epoch | loss = 9219.3564453125
16 epoch | loss = 9219.23828125
17 epoch | loss = 9219.1357421875
18 epoch | loss = 9219.0478515625
19 epoch | loss = 9218.9697265625
20 epoch | loss = 9218.9013671875
</code></pre>
<p>I can’t figure out the problem. The target size should be different from the input size. Because a sequence of 10 inputs give out 1 output. So,  target size = (torch.Size([5, 1, 1])) should be different from input size = (torch.Size([5, 10, 1])).</p>

username: Crohn_eng
cooked: <p>Hey <a class="mention" href="/u/scarecrow21">@scarecrow21</a>,</p>
<blockquote>
<p>The target size should be different from the input size. Because a sequence of 10 inputs give out 1 output. So, target size = (torch.Size([5, 1, 1])) should be different from input size = (torch.Size([5, 10, 1])).</p>
</blockquote>
<p>I think the “input” the error you are reporting is referring to is the input of the <code>F.mse_loss</code> function, which in this case is the <code>output</code> of your RNN, not the input you are giving to the network.<br>
There is probably a mismatch between the size of the output of your RNN and of the target you are giving to the loss function.<br>
Following the docs, the output of a RNN will consist of two tensors <code>[output, h_n]</code>, the first with shape <code>(seq_len, batch, num_directions * hidden_size)</code>, the second with shape <code>(num_layers * num_directions, batch, hidden_size)</code>.<br>
So, it is not true that a sequence of 10 inputs gives out 1 output: your <code>output</code> will contain the output features <em>(h_t)</em> from the last layer of the RNN, for each <em>t</em>. It would be something like <code>[seq_len=10, batch_size=5, hidden_size=1]</code>, using the parameters of your configuration.<br>
Now, take a look at the <code>forward</code> function of your <code>ModelRNN</code>:</p>
<pre><code class="lang-auto">def forward(self, x, hidden):
        hidden = torch.zeros(num_layer, batch_size, hidden_size)

        x = x.view(batch_size, sequence_length, input_size)
        out, hidden = self.rnn(x, hidden)
        
        return hidden, out
</code></pre>
<p>Since you “swap” the <code>hidden</code> and <code>out</code> tensors of position before returning, according to your configuration the tensors <code>hidden, output = model(x_batch, hidden)</code> in the <code>for</code> cycle of the training, would have as shapes <code>[num_layers*num_directions = 3, batch_size=5,  hidden_size=1]</code> and <code>[batch_size=5, seq_len=10, num_directions*hidden_size=1]</code> (since you instantiated your RNN with <code>batch_first=True</code>).<br>
When you compute the loss, you are doing it therefore between two tensors one of shape <code>[5, 10, 1]</code>, and one of shape <code>[5, 1, 1]</code>.<br>
In poor words, you are computing the loss between the whole sequence of output for each <em>t</em> and the last element of the sequence of the target.<br>
If you are interested only in the last element of the sequence for each batch, I would suggest you to modify the <code>forward</code> function in order to return only the last element of the sequence, or to manually modify the tensor before you compute the loss.</p>
<p>Hope everything is clear!</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: scarecrow21
cooked: <p>I changed the output variable in the training loop before passing it in the loss function.<br>
<code>output = output[:, -1, :]</code></p>
<p>This selects the last element of the sequence in each batch.<br>
I don’t get the size mismatch error anymore.</p>
<p>Thanks a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Crohn_eng
cooked: <p>You’re welcome buddy <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>