Categorical LSTM confused with shape of data and batching

username: Tank
cooked: <p>Creating a part of speech LSTM. Not sure how to shape the data if I am batching sentences of similar length.</p>
<p>X.shape = (100, 60, 4), [batch, length of sentence, features per word]<br>
output.shape = (100, 60, 10) [batch, length of sentences, type of word (10 potential types)]<br>
y.shape = (100, 60)</p>
<p>The error I get: Expected target size (100, 10), got torch.Size([100, 60])</p>
<p>Below the code of my network:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.lstm1 = nn.GRU(input_size = 4, hidden_size = 32, bidirectional= True)
        self.fcn1 = nn.Linear(64, 512)
        self.fcn2 = nn.Linear(512, 512)
        self.fcn3 = nn.Linear(512, 10)
        
        self.softmax = nn.LogSoftmax(dim=2)

    def forward(self, x):
        x, _ = self.lstm1(x)
        x = self.fcn1(x)
        x = self.fcn2(x)
        x = self.fcn3(x)
        x =x.squeeze(1)
  
       
        return x


net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=0.0001)

for epoch in range(5000):
    
    optimizer.zero_grad()
    
    running_loss = 0.0
    outputs = net(X)
    print(outputs.shape, y.shape)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
    running_loss += loss.item()
        
    if epoch % 100 == 0: 
        print("Epoch: ", epoch, "Loss: " , running_loss) 
</code></pre>
<p>Thanks for the help !</p>

username: ptrblck
cooked: <p><code>nn.GRU</code> expects the input in the shape <code>[seq_len, batch_size, input_features]</code> by default.<br>
You could use <code>batch_first=True</code> to pass the inputs as <code>[batch_size, seq_len, input_features]</code>, which seems to match your current input shape.<br>
The output will have the shape <code>[batch_size, seq_len, hidden*num_directions]</code>.<br>
The following linear layers will apply their operations on <code>dim1</code> “in a loop”, i.e. the linear transformation will be applied on each sample in the <code>seq_len</code> dimension.<br>
The last <code>x.squeeze(1)</code> won’t have any effect, as the temporal dimension will stay at 60 in <code>dim1</code>.</p>
<p>Also, you don’t have any non-linearities between the linear layers, so you might want to add them.</p>

username: Tank
cooked: <p>Thanks for the reply. I also had to change the loss function. Cross Entropy does not seem to work for many-to-many LSTM. Instead I am using</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
</code></pre>
<p>shape of output  = [100, 60, 10 ] and shape of y = [100, 60, 10] where the 10 is one hot encoded. I can’t get cross entropy criterion to work on this form of data.</p>

username: ptrblck
cooked: <p>What is your number of classes and what do the dimensions of the output represent?<br>
Are you working on a multi-class classification, where each temp. step would correspond to a single class?<br>
If so, <code>nn.CrossEntropyLoss</code> expects the model output to have the shape <code>[batch_size, nb_classes, seq_len]</code> and the target as <code>[batch_size, seq_len]</code> containing the class indices in the range <code>[0, nb_classes-1]</code>.</p>

username: Tank
cooked: <p>I am working on a multi-class classification where each t_i needs to be classified.</p>
<p>Not sure how i would go about transforming output:</p>
<p>current output is <code>[number of batches, sequence length, number of classes]</code><br>
target is correct <code>[number of batches, sequence length]</code></p>

username: harsha_g
cooked: <p><code>output = output.permute(0, 2, 1)</code></p>

username: Tank
cooked: <p>Thank you! It worked!!</p>