RuntimeError: unscale_() has already been called on this optimizer since the last update().
Author: kunaldeo
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- `accelerate` version: 0.21.0.dev0 
- `peft` version:  0.4.0.dev0
- Platform: Linux-6.3.9-zen1-1-zen-x86_64-with-glibc2.37
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Run LoRA training
```py
trainer = Trainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    args=TrainingArguments(
        per_device_train_batch_size=4,
        auto_find_batch_size=True,
        gradient_accumulation_steps=32,
        warmup_steps=100,
        num_train_epochs=EPOCHS,
        learning_rate=LEARNING_RATE,
        fp16=True,
        logging_steps=1,
        evaluation_strategy="steps" if VAL_SET_SIZE > 0 else "no",
        save_strategy="steps",
        eval_steps=50 if VAL_SET_SIZE > 0 else None,
        save_steps=500,
        output_dir=OUTPUT_DIR, #output_dir=repository_id,
        save_total_limit=3,
        load_best_model_at_end=True if VAL_SET_SIZE > 0 else False,
        ddp_find_unused_parameters=False if ddp else None,
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),
)
model.config.use_cache = False

old_state_dict = model.state_dict
model.state_dict = (
    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
).__get__(model, type(model))

if torch.__version__ >= "2" and sys.platform != 'win32':
    model = torch.compile(model)

trainer.train(resume_from_checkpoint = False)
```
2. Sometime after 1st epoch I run into the following error
```sh
{'loss': 2.2014, 'learning_rate': 0.0002538461538461538, 'epoch': 0.99}                                                                                                                                                              
{'loss': 2.24, 'learning_rate': 0.0002492307692307692, 'epoch': 1.0}                                                                                                                                                                 
{'loss': 2.2383, 'learning_rate': 0.0002446153846153846, 'epoch': 1.01}                                                                                                                                                              
                                                                                                                                                                                                                                   
raceback (most recent call last):███████████████████████████████████▏                                                                                                                            | 112/333 [42:21<1:21:32, 22.14s/it]
  File "/home/kunal/ml/train.py", line 234, in <module>                                                                                                                                            
    trainer.train(resume_from_checkpoint = False)                                                                                                                                                                                    
  File "/home/kunal/miniconda3/envs/lora/lib/python3.10/site-packages/transformers/trainer.py", line 1530, in train                                                                                                                  
    return inner_training_loop(                                                                                                                                                                                                      
  File "/home/kunal/miniconda3/envs/lora/lib/python3.10/site-packages/accelerate/utils/memory.py", line 132, in decorator                                                                                                            
    return function(batch_size, *args, **kwargs)                                                                                                                                                                                     
  File "/home/kunal/miniconda3/envs/lora/lib/python3.10/site-packages/transformers/trainer.py", line 1843, in _inner_training_loop                                                                                                   
    self.accelerator.clip_grad_norm_(                                                                                                                                                                                                
  File "/home/kunal/miniconda3/envs/lora/lib/python3.10/site-packages/accelerate/accelerator.py", line 1913, in clip_grad_norm_                                                                                                      
    self.unscale_gradients()                                                                                                                                                                                                         
  File "/home/kunal/miniconda3/envs/lora/lib/python3.10/site-packages/accelerate/accelerator.py", line 1876, in unscale_gradients                                                                                                    
    self.scaler.unscale_(opt)                                                                                                                                                                                                        
  File "/home/kunal/miniconda3/envs/lora/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 275, in unscale_                                                                                                          
    raise RuntimeError("unscale_() has already been called on this optimizer since the last update().")                                                                                                                              
RuntimeError: unscale_() has already been called on this optimizer since the last update().  
```
This training works fine on `transformers@de9255de27abfcae4a1f816b904915f0b1e23cd9`. 

### Expected behavior

Training should succeed. 

Comments:
cc @pacman100 and @muellerzr 

Hello, this is a duplicate issue. Please search the already existing ones. This is fixed via PR https://github.com/huggingface/transformers/pull/24415

Yes this is fixed. Thanks.