issue warning about different batch size being used for --resume_from_checkpoint
Author: tohara-PandoLogic
Pull request: no

### Feature request

According to #7198 pretraining must always use the same batch size:

> [15 Dec 21]
> [bowen-n] When I resume training from a checkpoint, I use a new batch size different from the previous training and it seems that the number of the skipped epoch is wrong.
> ...
> [sgugger] That feature is not supported, you should resume training with the exact same hyperparameters or start a new training if you want to change them.

This should be made explicit in a warning. As is, the system seems to just hang up.

This occurred specifically with the following script:
> ./examples/pytorch/language-modeling/run_mlm_no_trainer.py

The following is a sketch of the new. behavior. Batch size is not recoded in the checkpoint, so additional support is required:

```
if args.resume_from_checkpoint is not None or args.resume_from_checkpoint != "":
    accelerator.print(f"Resumed from checkpoint: {args.resume_from_checkpoint}")
    accelerator.load_state(args.resume_from_checkpoint)
    path = os.path.basename(args.resume_from_checkpoint)
    # Make sure same batch sizes uses
    # TODO: add support for storing batch size in checkpoint
    if accelerator...batch_size != args.per_device_eval_batch_size):
       logger.warning("Different batch size specified: previous %d vs. current %s; result unpredictable.",
                      accelerator...batch_size, args.per_device_eval_batch_size), 

```

Note that restarting is a not a practical solution given the time involved in pretraining. Given such consequences, it would be also be good for the argument description for --resume_from_checkpoint to warn about this limitation.

```
parser.add_argument(
    "--resume_from_checkpoint",
    type=str,
    default=None,
    help="If the training should continue from a checkpoint folder".       \
         " (n.b., using same hyperparameters in particular batch size).",
    )

```



----------

Environment information:
- `transformers` version: 4.26.0.dev0
- Platform: Linux-5.15.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.13
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


### Motivation

Again restarting is a not a practical solution given the time involved in pretraining.

Thus, the system should detect when two incompatible batch sizes are being used as well as mention this current restriction in the usage documentation. (It might not be apparent for users who have used other pretraining packages such such as Google [BERT](https://github.com/google-research/bert).)

Note that I had reviewed the script --help mentions of batch size before running the pretraining to make sure it was specified correctly. Thus I would have been alerted much sooner about this limitation, in particular before a costly pretraining run was made.

### Your contribution

I am more comfortable with the no_trainer-style scripts, having customized both the MLM and CLM ones as well as the code parrot example (i..e, under ./examples/research_projects/codeparrot). Therefore I could do the change outlined above as a PR once the support for recording the batch size is implemented. 

I could also add a test case for this, but there only seems existing tests for the Trainer style script (e.g., run_mlm.py) as in tests/trainer/test_trainer.py.

Comments:
The examples presented in this repo are not feature-complete apps with ironclad error messages, but just that... examples. We keep them with as little code as possible so they can be easily understood and customized. That's why they won't contain warnings for everything that could go wrong.

OK, so this turns out to be a two part feature request:
1. Issue warning
2. Include more informative usage for --resume_from_checkpoint

The first can be written off as won't fix as per keeping it simple, but the second should be addressed because it is straightforward to do so (e.g., minimal risk).

These examples are critical for doing non-trivial tasks with Hugging Face (e.g., pretraining and fine-tuning). The accelerator-based scripts are particularly non-trivial, so fleshing them out a little will be beneficial.

Sylvain: can you provide some tips (e.g., relatively safe batch size adjustments)?

Can you also un-hide the forum post I just made?
> https://discuss.huggingface.co/t/resuming-accelerate-based-pretraining-with-different-batch-size/30845

I put in a few links to the issue and code samples, so it got flagged as potential spam.

Best,
Tom

@tohara-PandoLogic Your post is un-hidden since yesterday, I rejected the spam flag. I still don't understand why you are using `--resume_from_checkpoint` for two different training with different hyperparameters. You can start a new training from any checkpoint by passing the model folder in `--model_name_or_path`.


OK, thanks. I take it the seed needs to be changed as well.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.