DistilBertForSequenceClassification 0% accuracy when fine-tuning (using from_pretrained())
Author: johannes-garstenauer
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES - 1x NVIDIA RTX A6000
- Using distributed or parallel set-up in script?: NO

### Who can help?

@ArthurZucker 
@younesbelkada 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Custom Script:
```python
import datasets
import evaluate
import torch
from tqdm import tqdm
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer,
    DistilBertConfig,
    DistilBertForSequenceClassification
)

model_name = "johannes-garstenauer/distilbert-heaps-masked" # This doesn't work
#model_name = "distilbert-base-uncased" # This works
#model_name="AyoubChLin/distilbert_cnn_news" # This works
do_finetune = True

print("\n")
print(f"Finetuning {do_finetune}")
if do_finetune:
    print(f"Model: {model_name}")
print("\n")

tokenizer = AutoTokenizer.from_pretrained(model_name)


def preprocess_function(examples):
    return tokenizer(examples["struct"], truncation=True, max_length=512)


ds_name = "johannes-garstenauer/balanced_factor_3_structs_reduced_5labelled_large"
raw_dataset = datasets.load_dataset(ds_name, split="train[:1%]")

tokenized_datasets = raw_dataset.map(preprocess_function, batched=True)
tokenized_datasets = tokenized_datasets.train_test_split(test_size=0.05)

if do_finetune:
    print("Finetuning")
    # When using model 'AyoubChLin/distilbert_cnn_news', might have to adapt num_labels=6 to avoid error
    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=5)
else:
    config = DistilBertConfig(output_hidden_states=True, num_labels=5)
    model = DistilBertForSequenceClassification(config)
print(model.config)

args = TrainingArguments(
    f"distilbert-finetuned",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    num_train_epochs=3,
)

metric = evaluate.load("accuracy")


def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = torch.argmax(predictions, dim=-1)
    return metric.compute(predictions=predictions, references=labels)


trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    compute_metrics=compute_metrics,
    tokenizer=tokenizer
)

# Overfitting the model on one batch
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

for batch in trainer.get_train_dataloader():
    break

batch = {k: v.to(device) for k, v in batch.items()}
trainer.create_optimizer()

for _ in tqdm(range(40)):
    outputs = trainer.model(**batch)
    loss = outputs.loss
    loss.backward()
    trainer.optimizer.step()
    trainer.optimizer.zero_grad()

with torch.no_grad():
    outputs = trainer.model(**batch)
preds = outputs.logits
labels = batch["labels"]
print(compute_metrics((preds, labels)))
print(batch['labels'])

```

Using the official script:
```python
python run_classification.py \
    --model_name_or_path  johannes-garstenauer/distilbert-heaps-masked \
    --dataset_name johannes-garstenauer/balanced_factor_3_structs_reduced_5labelled_large \
    --metric_name accuracy \
    --text_column_name struct \
    --label_column_name label \
    --do_train \
    --do_eval \
    --max_seq_length 512 \
    --per_device_train_batch_size 64 \
    --learning_rate 2e-5 \
    --num_train_epochs 3 \
    --output_dir ~/tmp/script_model/
```
### Expected behavior

**What the code does (Minimal Required Example + Official Classification Script):**
- The custom script fine-tunes a DistilBertForSequenceClassification model on a slightly unbalanced custom dataset with 5 labels. 
- The custom script can also be used to instantiate an empty DistilBertForSequenceClassification
- The custom script will overfit the model on a single batch for 40 iterations
- The official classification script is configured to fine-tune DistilBertForSequenceClassification on the mentioned dataset

**The expected behavior is:**
- An accuracy of 100% after overfitting
- A decreasing loss and accuracy when using the official script

**The issue:**
- Depending on which model the sequence classifier is fine-tuned on, the resulting model will either work as expected (high accuracy, low loss) or not work at all, meaning it will have stable high loss and low accuracy throughout training and always predict the same class 
    -  The predicted class will always be '0'. Using a weighted loss function can force the model to always predict another class, however not make it accurately predict
    - Some investigating has shown that the models embeddings of the [CLS] token (which is used for the final classification internally) is the same for whatever input is given. This is likely the reason for why the model will always predict the same label.
- I have not come to an understanding of why using some models as foundation fro fine-tuning models will cause this error. Some examples of models where the issue arises and some of where it doesn't are:
    - "johannes-garstenauer/distilbert-heaps-masked"
        - This is the model I intend to use for fine-tuning in my project. The issue arises with this model
    - "distilbert-base-uncased"
        - This is the standard DistilBertModel. Here the issue doesn't arise.
    - "AyoubChLin/distilbert_cnn_news"
        - This is a DistilBertForSequenceClassification model from the Hub. Here the issue doesn't arise.
    - To me it seems like the issue mysteriously arises sometimes. I could not determine f.e. that it would arise for certain types of models (f.e. MaskedLM, Sequence Classification) and not for others.
    - The logs of when the model is instantiated using 'from_pretrained()' are also unsuspicious and do not indicate any errors
- A theory of mine has been that the issue is related to loading a model with a conflicting configuration. To me especially 'vocab_size' and 'dtype_torch' had been suspects. Experimenting with different configurations has however not proven conclusive.

In conclusion, the documentation for the 'from_pretrained()' method claims, that it is "suited for loading models of different tasks", however, a bug seems to occur .
Sorry for the long issue. I hope someone is able to help.
Thank you,
 
Johannes

**Notes on reproducing the official classification script:**
- When trying to reproduce the official classification script as outlined above be aware that an exception may occur:
- TypeError: __init__() got an unexpected keyword argument ‘token'
- I resolved this by commenting out line 494 in https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_classification.py
- (this is just a hint and unrelated to the larger issue)



Comments:
Hey! Thanks a lot for providing such a detailed issue 🤗 
As the documentation about the examples mentions:
> While we strive to present as many use cases as possible, the scripts in this folder are just examples. It is expected that they won’t work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs. To help you with that, most of the examples fully expose the preprocessing of the data. This way, you can easily tweak them.

This means that it's expected that it might not work on every single model out there. Since the script works for the models it was designed to work on, you probably have to adapt it to work on you own specific model. 
None of us really has the bandwidth to debug your particular training script, but we welcome and appriciate bug fix PR! 
Would recommend you to post something on[ the forum](https://discuss.huggingface.co/) as well, to get input from the community! 

TLDR; Almost 100% sure there is nothing wrong with `from_pretrained` but rather with the way you finetune the model no? The logs can also help us debug if maybe you dont't have the latest version of transformers, etc. I could help you if you provided a full reproducer with loading the models, and showing the issue with from pretrained! 

Hey! 
Thank you for the quick response and the input. 

It is right I was not expecting the official classification script to work out of the box. If I find out why I'll make sure to create a bug fix PR.
Regarding my custom script, I am also not sure if the issue lies with `from_pretrained` however something seems to go wrong when initializing certain models, that prevents the `[CLS]` token embedding from being learned and therefore prevents classification from producing sensible results. I have reduced the script as much as necessary I believe therefor I am not sure where to look next for a solution. I'll make sure to consult the forum, though ;)

Anyways thanks again

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @johannes-garstenauer , were you able to resolve this problem?

Hello @gadregayatri, yes the issue lies with the Tokenizer.
When adding special tokens (https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__.add_special_tokens) it is important to keep in mind, that the order of the list you pass here determines the id of the tokens. It might be, that the model expects the pad token to be a certain ID (you can check this by printing the config of your model) but the tokenizer has a different ID for the pad token, based on the order of the special tokens list. This mismatch caused the bug for me. 

Hope that helps,
Johannes