text: 
  
  TITLE
  Loss function inputs for DistilBertForTokenClassification-like model using DistilBertModel
  
  COMMENTS
  5
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I want to fine tune my `DistilBertModel` just like `DistilBertForTokenClassification` for NER task by using nn.Module and building classifier on top myself. 
  
  But the problem is - I do not understand how to calculate loss function. In [official tutorial](https://huggingface.co/transformers/custom_datasets.html) it is explained only for seq classification which has multiple labels for input. But token classification is different!
  
  I am trying to do something like
  
  ```
          # For each batch of training data...
          for step, batch in enumerate(train_dataloader):
              batch_counts +=1
              # Load batch to GPU
              # b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)
              b_input_ids, b_attn_mask, b_labels = \
                batch['input_ids'].to(device), batch['attention_mask'].to(device), batch['labels'].to(device)
  
              # Zero out any previously calculated gradients
              model.zero_grad()
  
              # Perform a forward pass. This will return logits.
              logits = model(b_input_ids, b_attn_mask)
  
              # Compute loss and accumulate the loss values
              print('[DEBUG]', logits.shape, b_labels.shape)
              loss = loss_fn(logits, b_labels)
  ```
  
  The last line:
  
  > loss = loss_fn(logits, b_labels)
  
  will definitely raise error.
  
  I don't know how the expected label should look like and even the labels have `-100` extra instead of instead of indices.
  
  Full code(fairly straightforward with comments) : https://colab.research.google.com/drive/1FWPEV_5eOhveiT2AQyuSYm1Ka1pgeY2f?usp=sharing