Results are different when fine-tuning continues after loading model from checkpoint 
Author: schwabmi
Pull request: no

## Environment info
     
- `transformers` version: 4.0.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes (device: cuda:0, n_gpu: 1)
- Using distributed or parallel set-up in script?: False

### Who can help

@sgugger
@stefan-it
## Information

Model I am using (Bert, XLNet ...): bert-base-cased

The problem arises when using:
* [x] the official example scripts: run_ner_old.py


The tasks I am working on is:
* [x] my own task or dataset:  token classification for a rhetoric device

## To reproduce

Steps to reproduce the behavior:

1. Run run_ner_old script and save model after one epoch (282 steps):

```
python3  ./run_ner_old.py \
--data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path bert-base-cased \
--output_dir ./output/ \
--max_seq_length  128 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--save_steps 282 \
--seed 1 \
--do_train \
--do_eval 
```
2. Run ner_old_script from checkpoint-282:
```
python3  ./run_ner_old.py \
--data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path ./output/checkpoint-282 \
--tokenizer bert-base-cased \
--output_dir ./output2/ \
--max_seq_length  128 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--save_steps 282 \
--seed 1 \
--do_train \
--do_eval 
```
3. Compare evaluation results

**First experiment:** 
Run the script `run_ner_old.py` as showed above to fine-tune BERT.
I saved the model after the first epoch (282 steps).

**Second experiment:**
Run the script `run_ner_old.py` as showed above to fine-tune BERT, starting from checkpoint-282 from the first experiment:
```
[INFO|trainer.py:662] 2020-12-01 14:35:09,848 >> ***** Running training *****
[INFO|trainer.py:663] 2020-12-01 14:35:09,848 >>   Num examples = 4501
[INFO|trainer.py:664] 2020-12-01 14:35:09,848 >>   Num Epochs = 2
[INFO|trainer.py:665] 2020-12-01 14:35:09,849 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:666] 2020-12-01 14:35:09,849 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:667] 2020-12-01 14:35:09,849 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:668] 2020-12-01 14:35:09,849 >>   Total optimization steps = 564
[INFO|trainer.py:681] 2020-12-01 14:35:09,851 >>   Continuing training from checkpoint, will skip to saved global_step
[INFO|trainer.py:682] 2020-12-01 14:35:09,851 >>   Continuing training from epoch 1
[INFO|trainer.py:683] 2020-12-01 14:35:09,851 >>   Continuing training from global step 282
[INFO|trainer.py:684] 2020-12-01 14:35:09,851 >>   Will skip the first 0 batches in the first epoch
```
This seems right as the training continues from step 282 and it trains one complete epoch ("skip the first 0 batches"). 

 But when I **compare the results**, they are slightly different:
1. experiment: eval_f1 = 0.9226747985188413
2. experiment: eval_f1 = 0.9211328976034858

Also the loss after 500 steps is already different:
1. experiment:
`{'loss': 0.09096851348876953, 'learning_rate': 5.673758865248227e-06, 'epoch': 1.773049645390071}
`
2. experiment:
`
{'loss': 0.010856814384460449, 'learning_rate': 5.673758865248227e-06, 'epoch': 1.773049645390071}
`

## Expected behavior

I would have expected that both trained models should produce the same results since the second experiment does exactly the same but in two steps. (The model is saved and loaded between the two epochs).


The *checkpoint-282* directory consists of the following files:
```
config.json
optimizer.pt
pytorch_model.bin
scheduler.pt
trainer_state.json
training_args.bin
vocab.txt
```
It does not seem that there is any random initialization since I added the seed and the results do not change when running again.

Did I forget to save or load anything? 

Cheers

Comments:
Hi there. The results are slightly different because your dataloader has some randomization (the train dataloader has `shuffle=True` ) and the `Trainer` currently does not go through your dataloader for the past epochs when resuming training. So it trains starting from the global step 282 with the data of the epoch 0 of the initial training (hope that makes sense).

Let me see if we can support full reproducibility without a big drop in performance (cause we don't want to loop through that epoch 0 without doing anything either).

Hej
thanks for the fast answer!
I tried it out, but the results still differ slightly (leaving out: `--ignore_data_skip`).
Your change in the code should make the results exactly the same when continue training, right?
Is it because of the training data sampler (`RandomSampler`)?

Cheers


> Your change in the code should make the results exactly the same when continue training, right?

Yes, and this is enforced by tests in the CI. If your results still differ slightly, there might be another source of randomness not properly seeded that is responsible for those changes.