TokenClassification with DestillBert does not learn
Author: KyloPrem
Pull request: no

Hey guys,

I need to classify a sequence of tokens in a given sentence as either 0: **unrelevant** or 1: **relevant**. I tried to follow the tutorial for TokenClassification with DestillBert/Bert from Huggingface for NER, but the transfer does not seem to result in a model that learns to make predictions.  I am not sure where the problem lays, does find a pointer to how I could debug the training process with trainer?

Data is of the following format:

**tokens**: List(String)  ['I', 'am' 'an' 'example' , '.' ]
**labels**: List(Integer) [0, 1, 1, 1, 0]

Here is the Dataset format I use. As the tokens from the dataset were extracted with Bert, I just convert them to their IDs and stitch them together with the special tokens [CLS] and [SEP]. For the special tokens I assign a label of -100 to ignore them during the loss computation.
```
from torch.utils.data import Dataset

class relDataset(Dataset):
    
    def __init__(self, tokens, labels, tokenizer):
        self.tokens = tokens
        self.labels = labels
        self.tokenizer = tokenizer

    def __getitem__(self, idx):

        encoding = {}
        encoding["input_ids"] = [101] + tokenizer.convert_tokens_to_ids(self.tokens[idx]) + [102]
        encoding["attention_mask"] = [1]*len(encoding["input_ids"])
        encoding["labels"] = [-100] + self.labels[idx] + [-100] 
        
        return encoding

    def __len__(self):
        return len(self.label)
```


```
train = rel_Dataset(tokens = train_df["tokens"].values,
                                     labels = train_df["labels"].values,
                                     tokenizer = tokenizer)
```
```
test = rel_Dataset(tokens = test_df["tokens"].values,
                                     labels = test_df["labels"].values,
                                     tokenizer = tokenizer)
```
Then I set up tokenizer, model, and trainer


```
from transformers import AutoTokenizer
from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer
from torch.nn.parallel import DataParallel
import torch 

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForTokenClassification.from_pretrained("distilbert-base-uncased", num_labels=2)
device = torch.device("cuda")
model.to(device)

training_args = TrainingArguments(
    evaluation_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    num_train_epochs=6,
    weight_decay=0.01
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_A8,
    eval_dataset=eval_A8,
    tokenizer=tokenizer,
    data_collator=data_collator,
    #compute_metrics=compute_metrics
)
```
Then using Trainer to train the model I achieve the following performance
trainer.train()

![image](https://user-images.githubusercontent.com/26967090/172064517-d0f0a35c-5b97-4c37-9f70-9c44772755e8.png)


The Training loss seems to be static, while the predictions on the test data do not seem to make a difference

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) as well?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.