Saving and reloading DistilBertForTokenClassification fine-tuned model
Author: smith-nathanh
Pull request: no

I am trying to reload a fine-tuned DistilBertForTokenClassification model. I am using transformers 3.4.0 and pytorch version 1.6.0+cu101. After using the Trainer to train the downloaded model, I save the model with trainer.save_model() and during my trouble shooting I saved the model in a **different** directory via model.save_pretrained(). I am using Google Colab and saving the model to my Google drive. Before closing out my session, I evaluated the model and got good test results, however, when I return to the notebook (or Factory restart the colab notebook) and try to reload the model, the predictions are terrible. Upon checking the both directories, the config.json file is there as is the pytorch_mode.bin. It seems the trained model is not getting saved in this directories, but rather just the original model is? The model will work just fine if I don't close out my notebook session, but upon returning (or factory resetting) the reloading of the model yields a model that does not give good predictions. Is the trained model getting saved in a cache temporarily? But the save_model() function saves the original model?

```
from transformers import DistilBertForTokenClassification

# load the pretrained model from huggingface
model = DistilBertForTokenClassification.from_pretrained('distilbert-base-uncased', num_labels=len(uniq_labels)) 

model.to('cuda');

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir = model_dir +  'mitmovie_pt_distilbert_uncased/results',          # output directory
    #overwrite_output_dir = True,
    evaluation_strategy='epoch',
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir = model_dir +  'mitmovie_pt_distilbert_uncased/logs',            # directory for storing logs
    logging_steps=10,
    load_best_model_at_end = True
)

trainer = Trainer(
    model = model,                         # the instantiated 🤗 Transformers model to be trained
    args = training_args,                  # training arguments, defined above
    train_dataset = train_dataset,         # training dataset
    eval_dataset = test_dataset             # evaluation dataset
)

trainer.train()

trainer.evaluate()

model_dir = '/content/drive/My Drive/Colab Notebooks/models/'
trainer.save_model(model_dir + 'mitmovie_pt_distilbert_uncased/model')

# alternative saving method and folder
model.save_pretrained(model_dir + 'distilbert_testing')
```

Coming back to the notebook after restarting...

```from transformers import DistilBertForTokenClassification, DistilBertConfig, AutoModelForTokenClassification

# retreive the saved model 
model = DistilBertForTokenClassification.from_pretrained(model_dir + 'mitmovie_pt_distilbert_uncased/model', 
                                                        local_files_only=True)

model.to('cuda')
```
Model predictions are now terrible loading the model from either of the directories. 


Comments:
I'm encountering the same problem. Were you able to solve it?

Could you add the following lines:

```py
from transformers import logging as hf_logging

hf_logging.set_verbosity_info()
```
before reloading the model, and paste the results here?

cc @sgugger 

loading configuration file trained_models/checkpoint-8000/config.json
Model config DistilBertConfig {
  "_name_or_path": "distilbert-base-uncased",
  "activation": "gelu",
  "architectures": [
    "DistilBertForSequenceClassification"
  ],
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "id2label": { ... },
  "initializer_range": 0.02,
  "label2id": { ... },
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "vocab_size": 30522
}

loading weights file trained_models/checkpoint-8000/pytorch_model.bin
All model checkpoint weights were used when initializing DistilBertForSequenceClassification.

All the weights of DistilBertForSequenceClassification were initialized from the model checkpoint at trained_models/checkpoint-8000.
If your task is similar to the task the model of the checkpoint was trained on, you can already use DistilBertForSequenceClassification for predictions without further training.

When I go and evaluate the model from this point (either manually or by making a Trainer and using trainer.evaluate()) I get terrible scores. 

If I make a Trainer and try to continue training, I get terrible loss scores _except_ if I provide the checkpoint directory as part of the input to trainer.train(). If I supply the checkpoint directory there, the training appears to continue from the checkpoint, and if I train for ~300 more iterations, trainer.evaluate() gives decent performance but still not what I was seeing during the initial run. 

Okay, that's interesting. Do you mind sharing with us your environment? You can run `!transformers-cli env` and put the result here, we'll look into it.

Thanks @LysandreJik.

- `transformers` version: 3.5.0
- Platform: Linux-5.4.0-1030-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: Yes, single K80 on AWS
- Using distributed or parallel set-up in script?: No

There is little we can do to debug without a reproducer, which we don't have as the initial code contains a `train_dataset` and an `eval_dataset` we don't have access to. I just tried the notebook on {GLUE](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) and ran until the end of training (before hyperparameter-search), saved the model with `trainer.save_model(some_path)`, the restarted the notebook, ran all the cells up until the training then a new one with
```
model = AutoModelForSequenceClassification.from_pretrained(some_path, local_files_only=True)
trainer.model = model.cuda()
trainer.evaluate()
```
and it gave the exact same results as the end of training, so the `from_pretrained` method works well with the distilbert models.

As an update, I find that it's not just Distilbert models which will not save/reload for me, but also an Albert model gives the same behavior. Evaluation at the end of training gives 68% accuracy on my problem, whereas save/reload/reevaluate gives <1% accuracy.  Currently trying transformers 4.0 rather than 3.5.

Thanks for the update. As mentioned above, it does not help us fix this problem. We need a reliable reproducer for that.

@sgugger Turns out the problem was my fault. I was not keeping a consistent mapping of label names to integers across my runs. Once I corrected this my models performed identically after reload. Perhaps the OPs problem was similar. In any case, thanks for the help (the benchmark you linked helped me to debug) and sorry for the wild goose chase.

Glad you found the reason to your issue!

@mkreisel How are you "keeping a consistent mapping of label names to integers" across your runs now? Do you use a huggingface dataset ClassLabel ? I noticed in this original problem that it might be that the label mapping is somehow different after reloading the model when using num_labels in from_pretrained().

> There is little we can do to debug without a reproducer, which we don't have as the initial code contains a `train_dataset` and an `eval_dataset` we don't have access to. I just tried the notebook on {GLUE](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) and ran until the end of training (before hyperparameter-search), saved the model with `trainer.save_model(some_path)`, the restarted the notebook, ran all the cells up until the training then a new one with
> 
> ```
> model = AutoModelForSequenceClassification.from_pretrained(some_path, local_files_only=True)
> trainer.model = model.cuda()
> trainer.evaluate()
> ```
> 
> and it gave the exact same results as the end of training, so the `from_pretrained` method works well with the distilbert models.

@sgugger The issue is strictly with tokenclassification class. The index of the labels gets misaligned somehow when reloading a tokenclassification model. The problem happens across many model types: bert, distilbert, roberta, etc. If just giving num_labels = x when loading the model. I believe the issue has to do with the tokenizers and the fact that setting subwords equal to -100 creates another class when training the model, but that class is no longer available when you reload a pretrained tokenclassification model using from_pretrained(local_path). 

@nhsmith85 I was doing index -> class mapping using my own dictionary, not using anything internal to HuggingFace. I created a dataset class as an extension of torch.utils.data.Dataset:

```
class RecordsDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item

    def __len__(self):
        return len(self.labels)
```

At this point the text labels had already been mapped to integers. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello, Did anyone manage to get a solution for this? I am facing a very similar issue on `ViTForImageClassification` on using "google/vit-base-patch16-224".  Upon training, I am getting an accuracy of 0.75 and a very low loss. However, once I save and reload it, say after a day, the loss is back to ~10 and accuracy is 0.

Please find the necessary parts of my code here: https://gist.github.com/thevishnupradeep/d5efc0b0510d8a30d997cadd836d2c61

Also encountering the exact same problem with DistilBERT for QA.

As of July 2023 facing same issue with Bert model. Some one suggest a fix

Saved model performance is very bad compared to online model. Why????