RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
(a); (b); (c); (a) Stable rank for the input activations using a different number of groups, with =1 indicating no sub-division of the tokens into smaller sub-tokens. (b) Gradient sparsity for both vanilla fine-tuning and VeLoRA. (c) Approximate probability of the feature similarity diverging by at least k .
(d); (e); (f)
Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Comparison of our method with full fine-tuning, GaLore and LORA on GLUE benchmark using pre-trained RoBERTa-Base. Our method reaches the best overall results while showing significant memory improvements, especially compared to GaLore. We bold the best results from the considered PEFT methods. The GPU memory is measured on-device.
Mean 5-shot MMLU test accuracy for LLaMA models finetuned with adapters on Alpaca. The GPU memory estimate consists of the frozen weights, trainable adapters, and input activations.
Comparison with low-rank algorithms on pre-training various sizes of LLaMA models on C4 dataset. Validation perplexity is reported, along with the on-device GPU memory usage.
All three ablations are done using LLama-7B model. (a) VeLoRA has no loss in performance when trained for fewer or more training epochs than QLoRA despite both reducing the memory footprint. (b) Importance in choosing the correct number of sub-token size to find an optimal memory v.s. accuracy trade-off. Using a GPU memory estimate for the input activations only. (c) Ablating various initialisation strategies for a rank-1 projection and with M=D~{}/~{}32 .
Memory v.s. accuracy trade-off for VeLoRA on different layers. We use a LLaMA-7B trained on alpaca and evaluated on MMLU. We report the GPU memory estimate from the input activations only.
Hyperparameters of fine-tuning RoBERTa base.
The optimal task-specific hyper-parameters proposed for the Hydra method [ 20 ] .
The hyper-parameters in VTAB-1K experiments. These parameters are the same for all tasks to avoid any potential overfitting to a specific task and to maintain a constant memory usage.