Loss function format for sequence (NER/POS) tagging

username: scarecrow21
cooked: <p>I am trying to implement an NER tagger and I’m stuck in the implementation of loss function. I’m using cross entropy loss.</p>
<p>The <strong>y_pred</strong> size is = <code>torch.Size([2, 49, 9])</code><br>
The <strong>y_batch</strong> size is = <code>torch.Size([2, 49])</code></p>
<pre><code class="lang-auto"># Config
BATCH_SIZE = 2
EMBEDDING_SIZE = 5
VOCAB_SIZE = len(word2idx)
TARGET_SIZE = len(tag2idx) # number of output tags is 9
HIDDEN_SIZE = 4
STACKED_LAYERS = 3
</code></pre>
<p><strong>length of sentences in the batch</strong> = 49</p>
<p>Sample IP/OP:</p>
<pre><code class="lang-auto">Sentence: Ronaldo is from portugal. 
Tags:      PER     O    O     LOC
</code></pre>
<pre><code class="lang-auto">loss(y_pred, y_batch)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">ValueError: Expected target size (2, 9), got torch.Size([2, 49])
</code></pre>
<p>I know that CrossEntropy requires y_pred to be of size (N,C) and the y_batch to be of size (N) where N is the batch size and C is the number of classes.</p>
<p>The issue is, I have 9 probabilities per word here. So, I need to calculate the loss for each of them. That’s where I’m tripping up. How do I go about implementing the loss function for this?</p>

username: ptrblck
cooked: <p>Try to permit <code>y_pred</code> so that you pass it as <code>[2, 9, 49]</code>.</p>

username: scarecrow21
cooked: <p>I changed the shape of <code>y_pred</code> to <code>[2, 9, 49]</code> using <code>y_pred.view(BATCH_SIZE, TARGET_SIZE, -1)</code>.</p>
<p>I then did <code>loss(y_pred.view(BATCH_SIZE, TARGET_SIZE, -1), y_batch)</code>. The network started training but my loss kept increasing.</p>
<p>Here’s the training loop:</p>
<pre><code class="lang-auto"># Train loop
gru_model.train()
for e in range(1, EPOCHS+1):
    epoch_loss = 0
    epoch_acc = 0
    for batch in train_loader:
        x_batch, y_batch = map(list, zip(*batch))
        x_batch = [torch.tensor(i).to(device) for i in x_batch]
        y_batch = [torch.tensor(i).to(device) for i in y_batch]

        y_batch = pad_sequence(y_batch, batch_first=True)
        
        y_pred = gru_model(x_batch)
                               
        loss = criterion(y_pred.view(BATCH_SIZE, TARGET_SIZE, -1), y_batch)
        
        loss.backward()
        optimizer.step()
        
        epoch_loss += loss.item()
        
    print(f'Epoch {e+0:03}: | Loss: {epoch_loss/len(train_loader):.5f}')



# Output
'''
Epoch 001: | Loss: 2.11514
Epoch 002: | Loss: 2.12977
Epoch 003: | Loss: 2.16030
Epoch 004: | Loss: 2.17899
Epoch 005: | Loss: 2.17955
Epoch 006: | Loss: 2.18188
Epoch 007: | Loss: 2.19973
Epoch 008: | Loss: 2.19941
Epoch 009: | Loss: 2.20499
Epoch 010: | Loss: 2.19535 
'''
</code></pre>
<p>There was one other issue which I faced - when I change my batch_size=64, stacked_layers=4, hidden_size=8, and embedding_size=128 to bigger numbers; I get the following error.</p>
<pre><code class="lang-auto">
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-62-c943c869e71b&gt; in &lt;module&gt;
     20 #         print(y_pred.view(BATCH_SIZE, TARGET_SIZE, -1))
     21 
---&gt; 22         loss = criterion(y_pred.view(BATCH_SIZE, TARGET_SIZE, -1), y_batch)
     23 
     24         loss.backward()

RuntimeError: shape '[64, 9, -1]' is invalid for input of size 108900
</code></pre>
<p>Please tell me if you need any additional information/code.</p>

username: ptrblck
cooked: <p>Sorry for the typo, but apparently autocorrect “corrected” permute to permit. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Could you <code>.permute</code> the dimensions, since <code>view</code> will not yield the desired results:</p>
<pre><code class="lang-python">x = torch.arange(2*3*4).view(2, 3, 4)
y = x.view(2, 4, 3)
z = x.permute(0, 2, 1)

print(x)
&gt; tensor([[[ 0,  1,  2,  3],
         [ 4,  5,  6,  7],
         [ 8,  9, 10, 11]],

        [[12, 13, 14, 15],
         [16, 17, 18, 19],
         [20, 21, 22, 23]]])
print(y)
&gt; tensor([[[ 0,  1,  2],
         [ 3,  4,  5],
         [ 6,  7,  8],
         [ 9, 10, 11]],

        [[12, 13, 14],
         [15, 16, 17],
         [18, 19, 20],
         [21, 22, 23]]])
print(z)
&gt; tensor([[[ 0,  4,  8],
         [ 1,  5,  9],
         [ 2,  6, 10],
         [ 3,  7, 11]],

        [[12, 16, 20],
         [13, 17, 21],
         [14, 18, 22],
         [15, 19, 23]]])
</code></pre>

username: scarecrow21
cooked: <p>I tried out permute. Did not know something like that existed. Everything works perfectly now.<br>
I changed the shape of <code>y_pred</code> to <code>[2, 9, 49]</code> using <code>y_pred.permute(0, 2, 1)</code>.</p>
<p>Thank you!</p>