LoRA-PEFT - HuggingFace examples - LoRA does not train faster/nor does it allow a bigger batch size (essentially no improvement)
Author: CalinLucian
Pull request: no

### System Info

```
Platform system: Windows-10-10.0.22621-SP0
Python version: 3.9.13
Available RAM : 31.872 GB
CPU count: 20
CPU model: Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz
GPU count: 1
GPU model: 1 x NVIDIA GeForce RTX 3090

```
```
transformers==4.32.0
peft==0.5.0
accelerate=0.22.0
```

I am struggling to understand why I get no faster training nor can I use a bigger batch size when using LoRA.

I tried using the two tutorial available here (literally just running shift+enter the code, one time using `model `in the `Trainer()` one time using `lora_model`in the `Trainer()`:

1) https://huggingface.co/docs/peft/task_guides/image_classification_lora
2) https://huggingface.co/docs/peft/task_guides/semantic_segmentation_lora

I also tried using a custom PyTorch() module for the same segmentation model and got the same results (no improvement) (like 2.1)


I am doing **exactly** what is provided in the tutorials and indeed the following:
- `print_trainable_parameters(model) `
- `print_trainable_parameters(lora_model)`

print correctly (as in 100% trainable parameters for model and between 1%-10%) depending on the `LoraConfig() ` below.


```
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=32,
    lora_alpha=32,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="lora_only",
    modules_to_save=["decode_head"],
)
lora_model = get_peft_model(model, config)
print_trainable_parameters(lora_model)
```

Indeed `lora_model.save_pretrained(...)` saves a smaller model according to the trainable parameters.

**However, my stupefaction is that the training time stays exactly the same, as if no optimization happens. I cannot increase the batch_size (max batch_size in normal training is exactly like batch_size in lora_training, as if nothing happens), nor does the speed of the training increase (literally the same speed).**

Does anyone have any idea? I use exactly the links provided above, and I have the platform and library versions provided at the beginning.


@muellerzr @pacman100 @amyeroberts 

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just run the notebooks on LoRA given the platform + the dependencies.

### Expected behavior

I expect to actually use a bigger batch_size, to have the model train much faster given LoRA, essentially see the improvements that are brought by LoRA.

Comments:
@pacman100 do you have any idea? Is this : https://github.com/huggingface/transformers/issues/25572 remotely related to the issue? (given that I see same memory usage when training with initial_model and lora_model)?

Hello @CalinLucian, the models in those examples are too small (85M params) for noticing large gains, I still notice gains though. Below is the comparison between peft vs full fine-tuning with the image classification notebook

|method | GPU VRAM |
| --- | --- |
|peft lora | 13.71GB |
|full fine-tuning | 17.87GB|

So, I don't see why a larger batch size wasn't possible. If I increase the batch size from 128 to 175, the GPU VRAM usage for LoRA goes from 13.71Gb to 18.05GB which is comparable to the full fine-tuning memory consumption.

|method | Wall time |
| --- | --- |
|peft lora with batch size 175 | 5min 12s |
|full fine-tuning with batch size 128 | 6min 29s |

PEFT wall time screenshot:
![Screenshot 2023-08-30 at 4 51 03 PM](https://github.com/huggingface/transformers/assets/13534540/7c1ba73a-3d37-4145-b9ca-48e407c24b7e)

Full Finetuning wall time screenshot:
![Screenshot 2023-08-30 at 4 50 47 PM](https://github.com/huggingface/transformers/assets/13534540/3577698a-1057-4fdf-90bb-a2af15ea50ee)


Also, a reminder that the main purpose for PEFT methods are as following:
1. PEFT enables large model training because the optimizer states and gradients are limited to the tiny percentage of trainable params while keeping the pretrained model frozen. This results in huge VRAM savings while training. Compute wise, forward+backward will still be the same (peft will have overhead of additional computations for adapter layers), but only the step will be a bit faster since there will be less grads to add. So there is little difference compared to unfrozen model compute-wise.The only real saving is memory (since you don't need to allocate grads+optim states for frozen weights). So, training speed improvement depends on a lot of factors and this is not something PEFT promises to improve. It either enables one to fit large models on existing hardware or fit larger batches on it (this probably leads to faster training) while trying to have comparable performance to full fine-tuning. Quantization techniques and gradient checkpointing further drive down the memory requirements. 
2. PEFT enables one to have tiny checkpoint, thus enabling one to fine-tune a base model on a lot of downstream tasks while keeping the storage in check and enabling easy portability of the fine-tuned models.



@pacman100 Do you think it's possible I did not see/it is very negligible in my case because multiprocessing on Windows does create a bottleneck between the CPU/GPU and thus I cannot properly see the difference? I will try as you mentioned to reproduce your results, thank you for your time. Thank you for the insights on points 1 and 2, insightful!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.