Unsure of output dimension and Loss type (Newbie)

Harshit: <p>Hi<br>
I’m trying to train a character level RNN with a difference that each token(character) gets classified into one of m classes (multi class classification). My Input is of the shape (batch_size, seq_len, num_classes) because I’m using OHEncoding and output is of shape (batch_size, seq_len,num_classes) with softmax applied to dim 2. Here’s my model</p>
<pre><code class="lang-python">class CharRNN(nn.Module):
    def __init__(self, seq_len, input_size, hidden_size, output_size):
        super(CharRNN, self).__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True) 
        self.op = nn.Linear(hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=2)
        
    def forward(self, x):
        x, h = self.rnn(x)
        return self.softmax(self.op(x))
</code></pre>
<p>but when I use <code>NLLLoss()</code> as the criterion and <code>batch_size=32, seq_len=101, num_classes=32</code>,  I get the following error</p>
<pre><code class="lang-python">ValueError: Expected target size (32, 32), got torch.Size([32, 101, 32])
</code></pre>
<p>Please help if my understanding is correct or am I making a logical mistake somewhere. Thanks.</p>

vdw: <p>After</p>
<pre><code>x, h = self.rnn(x)`
</code></pre>
<p><code>x</code> has the shape <code>(seq_len, batch, hidden_size)</code>, in your case <code>(101, 32, hidden_size)</code>, since the output contains the hidden states for ALL 101 time steps.</p>
<p>The most obvious solution would to to use only the last hidden state. So you can either do</p>
<ul>
<li>
<code>self.op(x[-1])</code> or</li>
<li><code>self.op(h[-1])</code></li>
</ul>
<p>In your case, both <code>x[-1]</code> and <code>h[-1]</code> will have the shape <code>(batch, hidden_size)</code>, in your case <code>(32, hidden_size)</code>.</p>
<p>The output of the <code>self.op</code> layer will be <code>(batch_size, num_classes)</code>, in your case <code>(32, 32)</code>, exactly what you want.</p>
<p>The last change you need to do is to change <code>dim=2</code> to <code>dim=1</code> in your definition of the Softmax layer.</p>

Harshit: <p>Hi <a class="mention" href="/u/vdw">@vdw</a><br>
thanks for your response. If I only use the last output or hidden layer from the RNN I’m only getting one prediction for one sequence. Whereas I want predictions for each character in the sequence (101 seq_length). Also to clarify the output shape I mentioned is actually the shape of the labels. So my input and labels are of same shape <code>(batch_size, seq_len, num_classes)</code>.</p>

vdw: <p>You might want to look a <a href="https://discuss.pytorch.org/t/please-help-lstm-input-output-dimensions/89353">this post</a>, it seems very related. The link Udacity tutorial is also exactly about a character RNN.</p>

Harshit: <p><a class="mention" href="/u/vdw">@vdw</a>, The suggestion worked<br>
I used the Udacity’s Notebook to change the output and final activations in my model. The outputs were changed from <code>(batch_size, seq_len, num_classes)</code> to <code>(batch_size, seq_len)</code>, after I removed OHEncoding for the output. During the training time, the raw logit outputs were used in the loss function and in the inference stage I applied Softmax to change the model outputs to probabilities. I guess that’s the only way to do it.</p>