Loss function inputs for DistilBertForTokenClassification-like model using DistilBertModel
Author: rozeappletree
Pull request: no
Labels: name: wontfix

I want to fine tune my `DistilBertModel` just like `DistilBertForTokenClassification` for NER task by using nn.Module and building classifier on top myself. 

But the problem is - I do not understand how to calculate loss function. In [official tutorial](https://huggingface.co/transformers/custom_datasets.html) it is explained only for seq classification which has multiple labels for input. But token classification is different!

I am trying to do something like

```
        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):
            batch_counts +=1
            # Load batch to GPU
            # b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)
            b_input_ids, b_attn_mask, b_labels = \
              batch['input_ids'].to(device), batch['attention_mask'].to(device), batch['labels'].to(device)

            # Zero out any previously calculated gradients
            model.zero_grad()

            # Perform a forward pass. This will return logits.
            logits = model(b_input_ids, b_attn_mask)

            # Compute loss and accumulate the loss values
            print('[DEBUG]', logits.shape, b_labels.shape)
            loss = loss_fn(logits, b_labels)
```

The last line:

> loss = loss_fn(logits, b_labels)

will definitely raise error.

I don't know how the expected label should look like and even the labels have `-100` extra instead of instead of indices.

Full code(fairly straightforward with comments) : https://colab.research.google.com/drive/1FWPEV_5eOhveiT2AQyuSYm1Ka1pgeY2f?usp=sharing

Comments:
Hello! Have you taken a look at how we compute the loss in the [DistilbertForTokenClassificationModel](https://github.com/huggingface/transformers/blob/master/src/transformers/models/distilbert/modeling_distilbert.py#L807-L819)? If you pass the `labels` to the model, your loss will get computed automatically. 

If you want to compute your loss yourself, I would advise to copy/paste the loss computation as shown here and adapt it to your own loss!

Hey @LysandreJik thank you for helping me out. I've implemented it and it is working perfectly fine.

```
 Epoch  |  Batch  |  Train Loss  |  Val Loss  |  Val Acc  |  Elapsed 
----------------------------------------------------------------------
   1    |   20    |   0.000000   |     -      |     -     |  273.43  
   1    |   40    |   0.000000   |     -      |     -     |  262.14  
   1    |   60    |   0.000000   |     -      |     -     |  258.93  
   1    |   80    |   0.000000   |     -      |     -     |  266.20  
   1    |   84    |   0.000000   |     -      |     -     |   50.22  
----------------------------------------------------------------------
   1    |    -    |   0.000000   |  0.299704  |   19.37   |  1201.87 
----------------------------------------------------------------------


 Epoch  |  Batch  |  Train Loss  |  Val Loss  |  Val Acc  |  Elapsed 
----------------------------------------------------------------------
   2    |   20    |   0.000000   |     -      |     -     |  273.85  
   2    |   40    |   0.000000   |     -      |     -     |  264.77  
   2    |   60    |   0.000000   |     -      |     -     |  263.98  
   2    |   80    |   0.000000   |     -      |     -     |  263.12  
   2    |   84    |   0.000000   |     -      |     -     |   50.64  
----------------------------------------------------------------------
   2    |    -    |   0.000000   |  0.230533  |   19.39   |  1207.72 
----------------------------------------------------------------------
```

Notebook: https://colab.research.google.com/drive/1FWPEV_5eOhveiT2AQyuSYm1Ka1pgeY2f?usp=sharing


What I noticed is

1. It is taking too long even in colab. Is it usual? ( `1207.72/60 = 20mins`)
2. Accuracy (sum/total even if there will be more `O` tags) is not improving that much on [wnut17train.conll](http://noisy-text.github.io/2017/files/wnut17train.conll). Is there something I might be doing wrong?

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

> If you think this still needs to be addressed please comment on this thread.

Yeah, need to be.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.