Repository: huggingface/peft

Issue:
gradient_accumulation_steps problem in peft_lora_embedding_semantic_search.py
### System Info

accelerate==0.21.0
peft==0.5.0
transformers==4.31.0

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder
- [ ] My own task or dataset (give details below)

### Reproduction

```
accelerate launch \
--mixed_precision="fp16" \
peft/examples/feature_extraction/peft_lora_embedding_semantic_search.py \
--dataset_name="smangrul/amazon_esci" \
--max_length=70 --model_name_or_path="meta-llama/Llama-2-7b-hf" \
--per_device_train_batch_size=1 \
--per_device_eval_batch_size=1 \
--learning_rate=1e-4 \
--lr_scheduler_type="cosine"\
--weight_decay=0.05 \
--num_train_epochs 1 \
--gradient_accumulation_steps=2 \
--output_dir="results/peft_lora_e5_ecommerce_semantic_search" \
--seed=42 \
--with_tracking \
--use_peft \
--checkpointing_steps "epoch"
```

when for fine-tuning LLMs usually needs to use `gradient_accumulation_steps`.

### Expected behavior

In the script `peft_lora_embedding_semantic_search.py`, the `args.gradient_accumulation_steps` argument is not being used properly, I believe.

- On line 418 (in the example script from the specified version of PEFT), there is a misspelled variable name gradient_accumulation_stepp that should be replaced with gradient_accumulation_steps.
- The gradient_accumulation_steps argument is not being passed to the definition of the Accelerator class. It should be defined as follows:
```
accelerator = (
    Accelerator(log_with=args.report_to, project_dir=args.output_dir, gradient_accumulation_steps=args.gradient_accumulation_steps)
    if args.with_tracking else Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps)
)
```
This will ensure that the gradient_accumulation_steps argument is used when creating an instance of the Accelerator class, and thus will be available for use in the` with accelerator.accumulate(model):` statement.


Discussion:
Indeed, those look like mistakes. Do you want to create a PR to fix them?

Resolving patch:
diff --git a/examples/feature_extraction/peft_lora_embedding_semantic_search.py b/examples/feature_extraction/peft_lora_embedding_semantic_search.py
--- a/examples/feature_extraction/peft_lora_embedding_semantic_search.py
+++ b/examples/feature_extraction/peft_lora_embedding_semantic_search.py
@@ -207,9 +207,13 @@ def get_loss(cosine_score, labels):
 
 def main():
     args = parse_args()
-    accelerator = (
-        Accelerator(log_with=args.report_to, project_dir=args.output_dir) if args.with_tracking else Accelerator()
-    )
+
+    accelerator_kwargs = {"gradient_accumulation_steps": args.gradient_accumulation_steps}
+    if args.with_tracking:
+        accelerator_kwargs["log_with"] = args.report_to
+        accelerator_kwargs["project_dir"] = args.output_dir
+    accelerator = Accelerator(**accelerator_kwargs)
+    
     # Make one log on every process with the configuration for debugging.
     logging.basicConfig(
         format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
@@ -402,7 +406,7 @@ def preprocess_function(examples):
             resume_step = int(training_difference.replace("step_", "")) * args.gradient_accumulation_steps
             starting_epoch = resume_step // len(train_dataloader)
             resume_step -= starting_epoch * len(train_dataloader)
-            completed_steps = resume_step // args.gradient_accumulation_stepp
+            completed_steps = resume_step // args.gradient_accumulation_steps
 
     # update the progress_bar if load from checkpoint
     progress_bar.update(completed_steps)
diff --git a/examples/int8_training/peft_adalora_whisper_large_training.py b/examples/int8_training/peft_adalora_whisper_large_training.py
--- a/examples/int8_training/peft_adalora_whisper_large_training.py
+++ b/examples/int8_training/peft_adalora_whisper_large_training.py
@@ -422,16 +422,11 @@ def evaluation_loop(model, eval_dataloader, processor, normalizer, metric, force
 def main():
     args = parse_args()
 
-    # initialize accelerator
-    accelerator = (
-        Accelerator(
-            log_with=args.report_to,
-            project_dir=args.output_dir,
-            gradient_accumulation_steps=args.gradient_accumulation_steps,
-        )
-        if args.with_tracking
-        else Accelerator(gradient_accumulation_steps=args.gradient_accumulation_steps)
-    )
+    accelerator_kwargs = {"gradient_accumulation_steps": args.gradient_accumulation_steps}
+    if args.with_tracking:
+        accelerator_kwargs["log_with"] = args.report_to
+        accelerator_kwargs["project_dir"] = args.output_dir
+    accelerator = Accelerator(**accelerator_kwargs)
 
     # Make one log on every process with the configuration for debugging.
     logging.basicConfig(


Test patch: