Cross Entropy Loss: Target size and Output size mismatch

username: bibekx
cooked: <p>I have problem using Categorical Cross Entropy loss<br>
Target data is imported from a numpy array containing label indices for 3 classes (0,1,2)</p>
<p><strong>Dataset definition</strong></p>
<pre><code class="lang-auto">class Tr_dataset(Dataset):
    def __init__(self, windowed_input, classification_target):
        self.windowed_input = windowed_input
        self.classification_target = classification_target
        
    def __len__(self):
        return len(self.windowed_input)
    
    def __getitem__(self, index):      
        x_input = self.windowed_input[index]
        x_target = self.classification_target[index]
        
        x_input_tensor = torch.Tensor(x_input)
        x_input_tensor= x_input_tensor.view(SEQUENCE_LENGTH, INPUT_SIZE)
        
        # Ground truth 
        x_target_tensor = torch.LongTensor(x_target)
        return x_input_tensor, x_target_tensor
</code></pre>
<p><strong>Model</strong></p>
<pre><code class="lang-auto">class classification_RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers):
        super(classification_RNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        self.rnn = nn.RNN(input_size = input_size, hidden_size = hidden_size,num_layers= num_layers, 
                          batch_first= True)
        
        self.out = nn.Linear(hidden_size, 3)

    def forward(self, x):

        # The first hidden layer is automatically initialized to zeros if not passed
        rnn_out, hidden = self.rnn(x)
        
        class_label = self.out(rnn_out)
        
        return class_label
</code></pre>
<p><strong>Model relevant information</strong><br>
<code>model = classification_RNN(input_size=INPUT_SIZE, hidden_size=HIDDEN_SIZE, num_layers=NUM_LAYERS)</code></p>
<p><strong>Where:</strong></p>
<pre><code class="lang-auto"># Model params
BATCH_SIZE = 12

# The sequence length of the windowed data input
SEQUENCE_LENGTH = 12

# This is actually the number of dimensions/ features in the input
INPUT_SIZE = 1 

# The number of features in the last hidden state (Which should definitely be one)
# This is also equal to the number of outpu time steps to predict?
HIDDEN_SIZE = 1

#The number of RNN layers to stack
NUM_LAYERS = 3
</code></pre>
<p><strong>Since it is already getting too long: Partial Training loop:</strong></p>
<pre><code class="lang-auto">for bi, (x_input, x_target) in enumerate(train_loader):
        model.train()
  
        x_input_batch, x_target_batch = x_input.to(device),  x_target.to(device)
    
        optimizer.zero_grad() 
    
        output_batch = model(x_input_batch)
        
        loss = criterion(output_batch, x_target_batch)
</code></pre>
<p>The 2 inputs for the criterion seem to have a size mismatch</p>
<p>I read in many places, I seem to be doing everything fine. But,<br>
I get the error: <code>ValueError: Expected target size (12, 3), got torch.Size([12, 1])</code></p>

username: KarthikR
cooked: <p>Can you print the shapes of output_batch and x_target_batch, before they are passed to the loss function?</p>

username: bibekx
cooked: <p>Output Batch has a size:<br>
torch.Size([12, 12, 3])</p>
<p>Target Batch has a size<br>
torch.Size([12, 1])</p>
<p>I change a line in <strong>def forward:</strong> to</p>
<p><code>class_label = self.out(rnn_out.contiguous().view(-1, self.hidden_size))</code></p>
<p>to meet input expectation of linear</p>
<p>now,<br>
Output Batch has a size:<br>
torch.Size([144, 3])</p>
<p>I want it to be [12, 3]</p>

username: KarthikR
cooked: <p>Let us assume you did not reshape and the original error was about output shape:</p>
<p>Output Batch has a size:<br>
torch.Size([12, 12, 3])</p>
<p>Target Batch has a size<br>
torch.Size([12, 1])</p>
<p>import torch<br>
import torch.nn as nn<br>
ce = nn.CrossEntropyLoss()<br>
a = torch.randn((12,12,3))<br>
b = torch.randn((12,1))<br>
<span class="hashtag">#Expected</span> target size (12, 3), got torch.Size([12, 1])<br>
<span class="hashtag">#print</span>('Input Shape: ', a.shape)<br>
<span class="hashtag">#print</span>('Target Shape: ', b.shape)<br>
<span class="hashtag">#print</span>('Loss: ', ce(a,b))</p>
<p><span class="hashtag">#This</span> works as I reshape the input batch.<br>
a = torch.randn((12,12,1))<br>
b = torch.randint(0,3,(12,1))<br>
print('Input Shape: ', a.shape)<br>
print('Target Shape: ', b.shape)<br>
print('Loss: ', ce(a,b))</p>
<p>In your original code, can you change “self.out = nn.Linear(hidden_size, 3)” to "self.out = nn.Linear(hidden_size, 1), print the shapes and try?</p>

username: bibekx
cooked: <p>But, would I not want the logits as one of the inputs to Cross Entropy loss?</p>
<p>Also, I am not trying to do binary classification, I have to predict among 3 classes</p>
<p>I think the real problem I am facing is interfacing the output of RNN to a linear layer</p>
<p>Do you think this is correct?</p>
<pre><code class="lang-auto">self.rnn = nn.RNN(input_size = input_size, hidden_size = hidden_size,num_layers= num_layers, 
                          batch_first= True)
        
self.out = nn.Linear(hidden_size, 3)
</code></pre>

username: pchandrasekaran
cooked: <p>Although, I’m not too familiar with the workings of RNNs, your implementation looks correct.</p>
<p>CrossEntropyLoss expects a input of dim = (N, C) and a target of dim = (N,). Additional dimensions are used for “K-dimensional loss” as stated in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">docs</a>. Since your output batch is of dim (12, 12, 3), the target expected shape is (12, 3), but your targets are (12, 1), which explains your error. You need to perform 2 reshapes:</p>
<p>The first one is</p>
<pre><code class="lang-auto">output_batch = output_batch[:, -1, :]
</code></pre>
<p>It works, but I have no idea why this specific “reshape”. Here’s a link to an <a href="https://www.kaggle.com/kanncaa1/recurrent-neural-network-with-pytorch" rel="noopener nofollow ugc">RNN Implementation for MNIST</a> where I looked it up.</p>
<p>The second one is</p>
<pre><code class="lang-auto">x_target_batch = x_target_batch.view(-1)
</code></pre>
<p>This one is to satisfy the target shape required for the loss function.</p>

username: Caruso
cooked: <aside class="quote no-group" data-username="pchandrasekaran" data-post="7" data-topic="99031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pchandrasekaran/48/29605_2.png" class="avatar"> pchandrasekaran:</div>
<blockquote>
<p>It works, but I have no idea why this specific “reshape”.</p>
</blockquote>
</aside>
<p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html#torch.nn.RNN" rel="noopener nofollow ugc">RNN Module </a> returns 2 output tensors, the outputs after each iteration and the last hidden state. We only use first, which is of shape <code>[Batch, Seq, Hidden]</code> with <code>batch_first=True</code> and <code>num_directions=1</code>. bibekx most likely only wants the output of the last iteration, so we slice it with <code>[:, -1, :]</code>. Best use of this slicing would be in the forward call in <code>classification_RNN</code>, right before we feed it into the linear layer.</p>

username: pchandrasekaran
cooked: <aside class="quote no-group" data-username="Caruso" data-post="8" data-topic="99031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/caruso/48/24651_2.png" class="avatar"> Caruso:</div>
<blockquote>
<p>bibekx most likely only wants the output of the last iteration, so we slice it with <code>[:, -1, :]</code> .</p>
</blockquote>
</aside>
<p>That makes sense now. Thank You for the explanation!</p>

username: bibekx
cooked: <p>Thank you guys for the help,<br>
I did implement it as recommended by <a class="mention" href="/u/caruso">@Caruso</a> and <a class="mention" href="/u/pchandrasekaran">@pchandrasekaran</a></p>
<p>it seems to have initiated the training but… Its not training as expected. as the training accuracy increases, so does the loss. doesn’t make sense.</p>
<p>The link to code: (Please note some class names might be different)</p>
<p><a href="https://github.com/Bibek-Poudel/Traffic_flow_simple_RNN/blob/master/Vanilla_RNN_Classification.ipynb" rel="noopener nofollow ugc">Link to my GitHub</a></p>

username: pchandrasekaran
cooked: <p>You’ve used the same variable in the format statement for the (acc, loss).</p>

username: bibekx
cooked: <p>Hi <a class="mention" href="/u/pchandrasekaran">@pchandrasekaran</a>, Thanks! I fixed it. But now, the accuracy is not improving.</p>

username: pchandrasekaran
cooked: <p><a class="mention" href="/u/bibekx">@bibekx</a> That’s a bit weird. I can’t tell how a variable change may have caused that.</p>
<p>What I would suggest is:</p>
<ol>
<li>Check the model weights in the test_model function to ensure that an updated model is in fact passed.</li>
<li>Check that the dataloader in test_model is iterating through correctly.</li>
<li>Play around with the optimizer params (and different optimizers) and epochs. The loss flatlines at only the 4th epoch.</li>
</ol>
<p>Personally, I don’t think 1 or 2 is an issue. It has to be the optimizer. Also, before any retraining, run cells 6, 10 and 11 just to ensure all the weights get reinitialized.</p>