Skip batches fast with accelerate
Author: sgugger
Pull request: yes

# What does this PR do?

This PR uses the lastest from `Accelerate` to quickly skip batches when resuming training (it's only going to be quicker for a regular dataset, iterable datasets will still require a manual pass though the first batches).

Note that the RNG seeds can't be reloaded before we have started the iteration of the data loader because the random sampler used by the data laoder needs to use the seed at this stage, so there is a small path to load it after the iteration has started

cc @stas00 if you want to experiment (needs to use Accelerate main until v0.16 is out).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for implementing this, Sylvain,

Is this PR going to sit for a bit until the new Accelerate comes out? 

Currently I have to work on some other urgent things and SLURM makes it hard to quickly test things, but would be happy to test once I get the opportunity. 

The new release is just out actually, so this should be merged as soon as positively reviewed :-)

ok, great! congrats!

let me then just do a quick test on my desktop.

Hmm, on a small test it worked fine, but I see it fails on the real training. It appears to be restarting on resume rather than fast-forwarding. Perhaps when I did the small test I only checked that it printed the fast-forwarding message and thought it was doing the right thing.

Here is the TB:

![snapshot_88](https://user-images.githubusercontent.com/10676103/216860918-00645132-12b9-471f-abaf-385ae03d1725.png)

The straight line is a similar training before this change, the restarting one is after. So as you can see it restarts the iteration counter rather than continuing (it's hard to tell what was done to the data). Both trainings were run in 2 parts each with resume due to the slurm environment.

I was training with `--log_level warning` so didn't get the info log.

Any thoughts to what might have gone wrong?

The example `run_clm.py` script that was running with this config:  https://github.com/huggingface/m4/pull/922/files#diff-27452e2e8d112cbfd59f7900ef3b39dc35a4a0faf2d967fd86e399fe7ccb1ba2R192-R217

During the problematic run I used  `accelerate==0.16.0` and `transformers@main` (from Feb 1)
The good one was `accelerate==0.15.0` and some recent `transformers`

On the other hand it finished training at the same iteration as before, so the dataloader issues `StopIteration` at the same iteration as before.

which means some accounting was off - that is epoch isn't calculated correctly.

The training should finish at the same loss (this is tested with and without randomness in the model for small training), so normally you're good with the data. I think the problem lies with [this line](https://github.com/huggingface/transformers/blob/182afb7dc6f40aea5f5bb41710cb5207d187b022/src/transformers/trainer.py#L1909) which uses the `step` variable which now goes from 0 to len(data_loader) - num_batches_skipped (instead of num_batch_skipped to len(data_loader) before my PR). Will push a fix today or tomorrow!

super! thank you, Sylvain.

Should be fixed by the PR mentioned above.