Problem with Token Classification models 
Author: Joerg99
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.1
- Platform: Win10
- Python version:3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.3 CPU
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

 examples/token-classification: @stefan-it
 

## Information

I followed this tutorial https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities for token classification but results were really bad. So I changed the dataset to conll2003 and simplified the data a little (remove sentences without entities, keep only sentences with a certain length) as I saw that the model should perform well on this data.  Unfortunately the results are still bad for example after epoch two with the bert model set trainable=True:

Conf mat: (rows are prediction, columns are the  labels)
[[  1   3  21  10   5  10   1  16 172]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   1   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  2   7   7  13   7   6   0   9 162]]

Classification report
('              precision    recall  f1-score   support\n'
 '\n'
 '           0       0.33      0.00      0.01       239\n'
 '           1       0.00      0.00      0.00         0\n'
 '           2       0.00      0.00      0.00         0\n'
 '           3       0.00      0.00      0.00         1\n'
 '           4       0.00      0.00      0.00         0\n'
 '           5       0.00      0.00      0.00         0\n'
 '           6       0.00      0.00      0.00         0\n'
 '           7       0.00      0.00      0.00         0\n'
 '           8       0.49      0.76      0.59       213\n'
 '\n'
 '    accuracy                           0.36       453\n'
 '   macro avg       0.09      0.08      0.07       453\n'
 'weighted avg       0.40      0.36      0.28       453\n')


I tried a lot of things and checked pre-processing and post-processing multiple times and can't find a bug in there. 
The model is close to the tutorial(in the tutorial it's a DistilBert model but as if performed in the same manner I changed to the bigger brother) but it seems like it's not learning at all. Though it should perform well with conll data and in other tutorials this model has shown good results (for example: https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Here's the code is use (not complete preprocessing): 

```

import tensorflow as tf
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')

train_encodings = tokenizer(train_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)
val_encodings = tokenizer(val_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)

train_dataset = tf.data.Dataset.from_tensor_slices((dict(train_encodings), train_labels))
val_dataset = tf.data.Dataset.from_tensor_slices((dict(val_encodings), val_labels))

from transformers import TFBertForTokenClassification

model = TFBertForTokenClassification.from_pretrained('bert-base-cased', num_labels=len(unique_tags)) #unique tags are infered from training data
model.layers[0].trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)

model.compile(optimizer=optimizer, loss=model.compute_loss)  #or tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

for epoch in range(8):
    model.fit(train_dataset.shuffle(64).batch(16), batch_size=16, verbose=1, epochs=10)
    predictions = model.predict(val_dataset)
    # some post-processing. predictions is a list with all logits. for calculating the metrics I only consider
    # the tags that are not -100 (which are supposed to be ignored ).
    good_indexes = [i for i, l in enumerate(val_labels) if l != -100]
    list_preds = []
    for logi in predictions['logits']:
        list_preds.append(np.argmax(logi))
    pred_post = [list_preds[j] for j in good_indexes]
    print(confusion_matrix(pred_post, label_post))
    report = classification_report(pred_post, label_post)
    pprint(report)


```


## Expected behavior

Better performance  i.e. reasonable F1 Scores   (for example: https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests, rather than help with training-related issues.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead? You'll have better answers over there.

Thanks!