F1 decreases resuming from last saved checkpoint of fine-tuning
Author: paulthemagno
Pull request: no
Labels: name: wontfix

I'm doing a simple fine-tuning on a Camembert-like model on SQUAD task. Because of different bugs (not of the script, but of my internal storage) the fine-tuning crashed twice and so I restarted from the **last saved checkpoint**. When I evaluated the results I noticed that, **considering every first checkpoint after resuming the fine-tuning, F1-score decreased a little**. 

I have reported logs about the trend of F1 score during steps:
```bash
'f1_2000': 66.24313812273859,
'f1_4000': 70.1527380093159,
'f1_6000': 71.3234894754696, # <---after that checkpoint it crashes
'f1_6748': 69.69809710267471 # <---restarted from 6000 steps checkp. F1 score decreases :( . After this checkpoint I stopped the process, being the end of the first epoch
# end 1st epoch
'f1_8748': 68.93893417511639, # <---restarted from 6748 steps checkp. F1 score decreases :(
'f1_10748': 70.01292832499436,
'f1_12748': 71.69395205306222,
'f1_14748': 72.05253624620924, 
# end 2nd epoch
'f1_16748': 72.47768542161042,
'f1_18748': 72.88693616477238,
'f1_20244': 73.42437500678734
# end 3rd epoch
```

It seems that doing a full fine-tuning without stopping the process could bring better results than restarting from the last checkpoint. Is it a normal behaviour or not? Is there some information about the optimizer or something that can be passed to the script _to avoid loss of F1 score between a fine-tuning and its continuation_?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.14.186-110.268.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.10
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

## Information

Model I am using: custom version of **Camembert**

The problem arises when using:
- [x] the official example scripts: [run_squad.py](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)

The tasks I am working on is:
- [x] SQUAD task

## To reproduce

Steps to reproduce the behavior:

1. I'm working on a SageMaker notebook executing the following python script:

```bash
!python3 run_squad.py \
  --model_type camembert \
  --model_name_or_path my_model_path \ # at the beginning, then I used the name of the folder of last saved checkpoint
  --do_train \
  --do_eval \
  --eval_all_checkpoints \
  --train_file train.json \
  --predict_file test.json \
  --per_gpu_train_batch_size 8 \
  --per_gpu_eval_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 3.0 \
  --max_seq_length 512 \
  --doc_stride 128 \
  --save_steps 2000 \
  --output_dir output_folder
```

2. Print results in a file to check which is the best model


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I expected F1 score increased every checkpoint or at least didn't decrease exactly when I restarted the training because its seems that the loss of F1 is caused by the restarting from the last checkpoint.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.