Again: RuntimeError: unscale_() has already been called on this optimizer since the last update()
Author: FartyPants
Pull request: no
Labels: name: solved

### System Info

- `transformers` version: 4.30.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.12
- Huggingface_hub version: 0.16.2
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've seen the PR where this was supposed to be fixed.

Now I think from my experiments the issue still happens when 
gradient_accumulation_steps is larger than dataset.num_rows divided by per_device_train_batch_size
These are obviously not very good input data and it's kinda obvious that it should blow - but this could happen to people if the dataset is too small and gradient_accumulation_steps set arbitrary  - but no relevant info is given just a RuntimeError. So it's kinda bug and kinda user error.
On my end the issue can be fixed if I lower the gradient_accumulation_steps so it satisfies the above requirement.
I had not looked at the transformers code where this gets to play. This is literally based on my own hunch - if I would forget to safeguard the data, I would make an error there.

The debug log
File "\env\lib\site-packages\transformers\trainer.py", line 1645, in train
    return inner_training_loop(
  File "\env\lib\site-packages\transformers\trainer.py", line 1987, in _inner_training_loop
    self.accelerator.clip_grad_norm_(
  File "\env\lib\site-packages\accelerate\accelerator.py", line 1893, in clip_grad_norm_
    self.unscale_gradients()
  File "\env\lib\site-packages\accelerate\accelerator.py", line 1856, in unscale_gradients
    self.scaler.unscale_(opt)
  File "env\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 275, in unscale_
    raise RuntimeError("unscale_() has already been called on this optimizer since the last update().")
RuntimeError: unscale_() has already been called on this optimizer since the last update().
### Expected behavior

safe guard this if it is indeed an issue
or give error that tells you why this happens (too high gradient_accumulation_steps for the amount of data)

If this is wrong place to bring this, let me know. 

Comments:
I have the same errors (I opened an issue). However I don't have the gradient_accumulation_steps larger than the rows divied by per_device_train_batch_size. I mean I have this parameters:
GRADIENT_ACCUMULATION_STEPS = 16
MICRO_BATCH_SIZE = 4
dataset.num_rows = 53131

Could it be that one of the batch size is less than the micro_batch_size?

I think this is fixed on the main branch now.
cc @muellerzr and @pacman100 

Hello @FartyPants, please see this: https://github.com/huggingface/transformers/issues/24849#issuecomment-1638272113

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.