SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The memory consumption is not solely dictated by the model size but is also influenced by various other factors<sup>6</sup>. Our approach with PEQA inherently offers memory advantages during fine-tuning by striving to minimize both the model size and the number of training parameters. To provide a clear understanding of these benefits, we conducted tests using a single NVIDIA A100-80GB GPU and the causal language modeling code from the HuggingFace repository<sup>7</sup>. Both LoRA and PEQA fine-tuned the LLaMA-7B on the Wikitext2 dataset with a batch size of 2 without gradient accumulation. Our findings indicated that while LoRA peaked at a memory usage of 59GB during optimization, PEQA used just 43GB. Remarkably, this disparity (16GB, 7B) escalates as the model size increases; for instance, a 65B full-precision model under LoRA occupies 130GB, whereas PEQA remarkably uses just 33GB. Additionally, LoRA encountered Out-Of-Memory (OOM) issues at a batch size of 4, whereas PEQA, due to its efficiency, continued training seamlessly.
$^{6}$ https://huggingface.co/docs/transformers/perf_traingpu_one  
 $^{7}$ https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm_no Trainer.py