Repository: huggingface/transformers

Issue:
Results are different when fine-tuning continues after loading model from checkpoint 
## Environment info
     
- `transformers` version: 4.0.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes (device: cuda:0, n_gpu: 1)
- Using distributed or parallel set-up in script?: False

### Who can help

@sgugger
@stefan-it
## Information

Model I am using (Bert, XLNet ...): bert-base-cased

The problem arises when using:
* [x] the official example scripts: run_ner_old.py


The tasks I am working on is:
* [x] my own task or dataset:  token classification for a rhetoric device

## To reproduce

Steps to reproduce the behavior:

1. Run run_ner_old script and save model after one epoch (282 steps):

```
python3  ./run_ner_old.py \
--data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path bert-base-cased \
--output_dir ./output/ \
--max_seq_length  128 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--save_steps 282 \
--seed 1 \
--do_train \
--do_eval 
```
2. Run ner_old_script from checkpoint-282:
```
python3  ./run_ner_old.py \
--data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path ./output/checkpoint-282 \
--tokenizer bert-base-cased \
--output_dir ./output2/ \
--max_seq_length  128 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--save_steps 282 \
--seed 1 \
--do_train \
--do_eval 
```
3. Compare evaluation results

**First experiment:** 
Run the script `run_ner_old.py` as showed above to fine-tune BERT.
I saved the model after the first epoch (282 steps).

**Second experiment:**
Run the script `run_ner_old.py` as showed above to fine-tune BERT, starting from checkpoint-282 from the first experiment:
```
[INFO|trainer.py:662] 2020-12-01 14:35:09,848 >> ***** Running training *****
[INFO|trainer.py:663] 2020-12-01 14:35:09,848 >>   Num examples = 4501
[INFO|trainer.py:664] 2020-12-01 14:35:09,848 >>   Num Epochs = 2
[INFO|trainer.py:665] 2020-12-01 14:35:09,849 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:666] 2020-12-01 14:35:09,849 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:667] 2020-12-01 14:35:09,849 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:668] 2020-12-01 14:35:09,849 >>   Total optimization steps = 564
[INFO|trainer.py:681] 2020-12-01 14:35:09,851 >>   Continuing training from checkpoint, will skip to saved global_step
[INFO|trainer.py:682] 2020-12-01 14:35:09,851 >>   Continuing training from epoch 1
[INFO|trainer.py:683] 2020-12-01 14:35:09,851 >>   Continuing training from global step 282
[INFO|trainer.py:684] 2020-12-01 14:35:09,851 >>   Will skip the first 0 batches in the first epoch
```
This seems right as the training continues from step 282 and it trains one complete epoch ("skip the first 0 batches"). 

 But when I **compare the results**, they are slightly different:
1. experiment: eval_f1 = 0.9226747985188413
2. experiment: eval_f1 = 0.9211328976034858

Also the loss after 500 steps is already different:
1. experiment:
`{'loss': 0.09096851348876953, 'learning_rate': 5.673758865248227e-06, 'epoch': 1.773049645390071}
`
2. experiment:
`
{'loss': 0.010856814384460449, 'learning_rate': 5.673758865248227e-06, 'epoch': 1.773049645390071}
`

## Expected behavior

I would have expected that both trained models should produce the same results since the second experiment does exactly the same but in two steps. (The model is saved and loaded between the two epochs).


The *checkpoint-282* directory consists of the following files:
```
config.json
optimizer.pt
pytorch_model.bin
scheduler.pt
trainer_state.json
training_args.bin
vocab.txt
```
It does not seem that there is any random initialization since I added the seed and the results do not change when running again.

Did I forget to save or load anything? 

Cheers


Discussion:
Hi there. The results are slightly different because your dataloader has some randomization (the train dataloader has `shuffle=True` ) and the `Trainer` currently does not go through your dataloader for the past epochs when resuming training. So it trains starting from the global step 282 with the data of the epoch 0 of the initial training (hope that makes sense).

Let me see if we can support full reproducibility without a big drop in performance (cause we don't want to loop through that epoch 0 without doing anything either).

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -665,12 +665,12 @@ def train(self, model_path: Optional[str] = None, trial: Union["optuna.Trial", D
         )
 
         logger.info("***** Running training *****")
-        logger.info("  Num examples = %d", num_examples)
-        logger.info("  Num Epochs = %d", num_train_epochs)
-        logger.info("  Instantaneous batch size per device = %d", self.args.per_device_train_batch_size)
-        logger.info("  Total train batch size (w. parallel, distributed & accumulation) = %d", total_train_batch_size)
-        logger.info("  Gradient Accumulation steps = %d", self.args.gradient_accumulation_steps)
-        logger.info("  Total optimization steps = %d", max_steps)
+        logger.info(f"  Num examples = {num_examples}")
+        logger.info(f"  Num Epochs = {num_train_epochs}")
+        logger.info(f"  Instantaneous batch size per device = {self.args.per_device_train_batch_size}")
+        logger.info(f"  Total train batch size (w. parallel, distributed & accumulation) = {total_train_batch_size}")
+        logger.info(f"  Gradient Accumulation steps = {self.args.gradient_accumulation_steps}")
+        logger.info(f"  Total optimization steps = {max_steps}")
 
         self.state.epoch = 0
         epochs_trained = 0
@@ -680,13 +680,20 @@ def train(self, model_path: Optional[str] = None, trial: Union["optuna.Trial", D
         if model_path and os.path.isfile(os.path.join(model_path, "trainer_state.json")):
             self.state = TrainerState.load_from_json(os.path.join(model_path, "trainer_state.json"))
             epochs_trained = self.state.global_step // num_update_steps_per_epoch
-            steps_trained_in_current_epoch = self.state.global_step % (num_update_steps_per_epoch)
-            steps_trained_in_current_epoch *= self.args.gradient_accumulation_steps
+            if not self.args.ignore_data_skip:
+                steps_trained_in_current_epoch = self.state.global_step % (num_update_steps_per_epoch)
+                steps_trained_in_current_epoch *= self.args.gradient_accumulation_steps
+            else:
+                steps_trained_in_current_epoch = 0
 
             logger.info("  Continuing training from checkpoint, will skip to saved global_step")
-            logger.info("  Continuing training from epoch %d", epochs_trained)
-            logger.info("  Continuing training from global step %d", self.state.global_step)
-            logger.info("  Will skip the first %d batches in the first epoch", steps_trained_in_current_epoch)
+            logger.info(f"  Continuing training from epoch {epochs_trained}")
+            logger.info(f"  Continuing training from global step {self.state.global_step}")
+            if not self.args.ignore_data_skip:
+                logger.info(
+                    f"  Will skip the first {epochs_trained} epochs then the first {steps_trained_in_current_epoch} "
+                    "batches in the first epoch."
+                )
 
         # Update the references
         self.callback_handler.model = self.model
@@ -712,6 +719,13 @@ def train(self, model_path: Optional[str] = None, trial: Union["optuna.Trial", D
 
         self.control = self.callback_handler.on_train_begin(self.args, self.state, self.control)
 
+        # Skip the first epochs_trained epochs to get the random state of the dataloader at the right point.
+        if not self.args.ignore_data_skip:
+            for epoch in range(epochs_trained):
+                # We just need to begin an iteration to create the randomization of the sampler.
+                for _ in train_dataloader:
+                    break
+
         for epoch in range(epochs_trained, num_train_epochs):
             if isinstance(train_dataloader, DataLoader) and isinstance(train_dataloader.sampler, DistributedSampler):
                 train_dataloader.sampler.set_epoch(epoch)
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -189,6 +189,10 @@ class TrainingArguments:
         model_parallel (:obj:`bool`, `optional`, defaults to :obj:`False`):
             If there are more than one devices, whether to use model parallelism to distribute the model's modules
             across devices or not.
+        ignore_skip_data (:obj:`bool`, `optional`, defaults to :obj:`False`):
+            When resuming training, whether or not to skip the epochs and batches to get the data loading at the same
+            stage as in the previous training. If set to :obj:`True`, the training will begin faster (as that skipping
+            step can take a long time) but will not yield the same results as the interrupted training would have.
     """
 
     output_dir: str = field(
@@ -350,6 +354,12 @@ class TrainingArguments:
     greater_is_better: Optional[bool] = field(
         default=None, metadata={"help": "Whether the `metric_for_best_model` should be maximized or not."}
     )
+    ignore_data_skip: bool = field(
+        default=False,
+        metadata={
+            "help": "When resuming training, whether or not to skip the first epochs and batches to get to the same training data."
+        },
+    )
 
     def __post_init__(self):
         if self.disable_tqdm is None:


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -554,6 +554,20 @@ def test_can_resume_training(self):
             self.assertEqual(b, b1)
             self.assertEqual(state, state1)
 
+            # Now check with a later checkpoint that it also works when we span over one epoch
+            checkpoint = os.path.join(tmpdir, "checkpoint-15")
+
+            # Reinitialize trainer and load model
+            model = RegressionPreTrainedModel.from_pretrained(checkpoint)
+            trainer = Trainer(model, trainer.args, train_dataset=trainer.train_dataset)
+
+            trainer.train(model_path=checkpoint)
+            (a1, b1) = trainer.model.a.item(), trainer.model.b.item()
+            state1 = dataclasses.asdict(trainer.state)
+            self.assertEqual(a, a1)
+            self.assertEqual(b, b1)
+            self.assertEqual(state, state1)
+
         # With a regular model that is not a PreTrainedModel
         with tempfile.TemporaryDirectory() as tmpdir:
             trainer = get_regression_trainer(
@@ -578,6 +592,22 @@ def test_can_resume_training(self):
             self.assertEqual(b, b1)
             self.assertEqual(state, state1)
 
+            # Now check with a later checkpoint that it also works when we span over one epoch
+            checkpoint = os.path.join(tmpdir, "checkpoint-15")
+
+            # Reinitialize trainer and load model
+            model = RegressionModel()
+            state_dict = torch.load(os.path.join(checkpoint, WEIGHTS_NAME))
+            model.load_state_dict(state_dict)
+            trainer = Trainer(model, trainer.args, train_dataset=trainer.train_dataset)
+
+            trainer.train(model_path=checkpoint)
+            (a1, b1) = trainer.model.a.item(), trainer.model.b.item()
+            state1 = dataclasses.asdict(trainer.state)
+            self.assertEqual(a, a1)
+            self.assertEqual(b, b1)
+            self.assertEqual(state, state1)
+
     def test_resume_training_with_gradient_accumulation(self):
         if torch.cuda.device_count() > 2:
             # This test will fail for more than 2 GPUs since the batch size will get bigger and with the number of