Resume training from checkpoint: not progressing
Author: mattivi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.0
- Platform: Linux-3.10.0-514.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.2
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
### Who can help

@sgugger


## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [x] the official example scripts: /examples/language-modeling/run_mlm.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: BERT MLM pre-training with own dataset

## To reproduce

Steps to reproduce the behavior:

1. Run script run_mlm.py, training from scratch, and save a checkpoint.
2. Stop the training.
3. Restore the training from the checkpoint, e.g. with the code below
4. When restoring, the pre-training process is not progressing (since hours).

```cmd
python run_mlm.py     --model_type bert     --model_name_or_path /bert-base-v2/checkpoint-204516/     
--overwrite_output_dir     --config_name /bert-base-v2-config/     --tokenizer_name /bert-base-v2-config/    
 --train_file /train_subset.txt     --validation_file /eval_subset.txt      --do_train     --do_eval     --line_by_line     
--output_dir /bert-base-v2/     --cache_dir /tmp/     --save_total_limit 300     --num_train_epochs 10   
  --warmup_steps 10000     --logging_steps 5000     --save_steps 11362    
 --per_device_train_batch_size 128     --per_device_eval_batch_size 128     --seed 42
```
Output is 
```
12/01/2020 15:43:28 - INFO - __main__ -   Loading tokenized dataset from file...
12/01/2020 15:47:22 - INFO - __main__ -   Done.
[INFO|trainer.py:357] 2020-12-01 15:47:29,458 >> The following columns in the training set don't have a corresponding argument in `BertForMaskedLM.forward` and have been ignored: special_tokens_mask.
[INFO|trainer.py:357] 2020-12-01 15:47:29,459 >> The following columns in the evaluation set don't have a corresponding argument in `BertForMaskedLM.forward` and have been ignored: special_tokens_mask.
[INFO|trainer.py:662] 2020-12-01 15:47:32,843 >> ***** Running training *****
[INFO|trainer.py:663] 2020-12-01 15:47:32,843 >>   Num examples = 145434960
[INFO|trainer.py:664] 2020-12-01 15:47:32,843 >>   Num Epochs = 10
[INFO|trainer.py:665] 2020-12-01 15:47:32,843 >>   Instantaneous batch size per device = 128
[INFO|trainer.py:666] 2020-12-01 15:47:32,843 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:667] 2020-12-01 15:47:32,843 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:668] 2020-12-01 15:47:32,843 >>   Total optimization steps = 11362110
[INFO|trainer.py:681] 2020-12-01 15:47:32,846 >>   Continuing training from checkpoint, will skip to saved global_step
[INFO|trainer.py:682] 2020-12-01 15:47:32,846 >>   Continuing training from epoch 0
[INFO|trainer.py:683] 2020-12-01 15:47:32,846 >>   Continuing training from global step 204516
[INFO|trainer.py:684] 2020-12-01 15:47:32,846 >>   Will skip the first 204516 batches in the first epoch
  0%|                                                                       | 0/11362110 [00:00<?, ?it/s]
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Would expect the training to restore from 204516 and continue training.


Comments:
It is expected that this would take some time, since it has to skip through `204,516` batches before continuing training. It will continue progressing after that skip is done.

In the PR mentioned above, I'm adding a flag to ignore that step if you're prepared to pay the price of having the training be slightly different from a training from scratch to go faster.

That would do for my case, thanks!

With that PR everything worked as expected, thanks for the very quick turnaround!

Happy to hear!

@sgugger Sorry to bother, but I am wondering why skipping steps takes computing. I mean, the random_seed is specified, so the trainer just need to find the breakpoint of an epoch and resume, I shouldn't take much time.

So does any parts of my understand are wrong?

Yes, but there is no way to be in the exact sample place in the dataloaders (that have randomness with the shuffling) without going through the first epochs and then batches.

@sgugger thanks for the information, sorry to revive this issue. How long does it usually take to go through the first epochs and then batches? half of what it took to train until that point or less?

It depends on your data and the time needed for your preprocessing. Note that there is a progress bar in the newer versions of Transformers so you can get a sense of the remaining time. You can also skip this with the [flag `ignore_data_skip`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.ignore_data_skip) though the model will train on already seen data in this case.

I have the same issue,
```
***** Running training *****
  Num examples = 2,560,000
  Num Epochs = 9,223,372,036,854,775,807
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 16
  Gradient Accumulation steps = 2
  Total optimization steps = 160,000
  Number of trainable parameters = 332,891,919
  Continuing training from checkpoint, will skip to saved global_step
  Continuing training from epoch 0
  Continuing training from global step 77000
  Will skip the first 0 epochs then the first 154000 batches in the first epoch.
  0%|                                                                                  | 0/160000 [00:00<?, ?it/s]
  The following columns in the training set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: locale, audio, input_length, sentence. If locale, audio, input_length, sentence are not expected by `Wav2Vec2ForCTC.forward`,  you can safely ignore this message.
There seems to be not a single sample in your epoch_iterator, stopping training at step 77000! This is expected if you're using an IterableDataset and set num_steps (160000) higher than the number of available samples.


Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 1869.2894, 'train_samples_per_second': 1369.504, 'train_steps_per_second': 85.594, 'train_loss': 0.0, 'epoch': 43.01}                                                
  0%|                                                     | 0/160000 [31:09<?, ?it/s]Saving model checkpoint to /usr/local/bin/source/output
Configuration saved in /usr/local/bin/source/output/config.json
Model weights saved in /usr/local/bin/source/output/pytorch_model.bin
Feature extractor saved in /usr/local/bin/source/output/preprocessor_config.json
tokenizer config file saved in /usr/local/bin/source/output/tokenizer_config.json
Special tokens file saved in /usr/local/bin/source/output/special_tokens_map.json
added tokens file saved in /usr/local/bin/source/output/added_tokens.json
trainer save model!
metric:  train_runtime
***** train metrics *****
  epoch                    =      43.01
  train_loss               =        0.0
  train_runtime            = 0:31:09.28
  train_samples_per_second =   1369.504
  train_steps_per_second   =     85.594
06/19/2023 08:28:34 - INFO - __main__ - *** Evaluate ***
***** Running Evaluation *****
  Num examples: Unknown
  Batch size = 8
The following columns in the evaluation set don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: locale, audio, input_length, sentence. If locale, audio, input_length, sentence are not expected by `Wav2Vec2ForCTC.forward`,  you can safely ignore this message.
Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Automatic Speech Recognition', 'type': 'automatic-speech-recognition'}, 'metrics': [{'name': 'Wer', 'type': 'wer', 'value': 1.0006485084306096}]}
```

max_steps= 160000
last checkpoint=77000

if **ignore_data_skip**=True is set, it can resume training correctly.

> It depends on your data and the time needed for your preprocessing. Note that there is a progress bar in the newer versions of Transformers so you can get a sense of the remaining time. You can also skip this with the [flag `ignore_data_skip`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.ignore_data_skip) though the model will train on already seen data in this case.

Thank you for your answer. I have another question, if I set ignore_data_skip, what will be the effect? Currently I'm using streaming to read the data, will ignore_data_skip cause to read the data from the beginning to train it, making those data to be duplicated for traini