ICML 2025

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Tianyi Zhang * 1 2 Junda Su * 1 Aditya Desai 3 Oscar Wu 2 Zhaozhuo Xu 4 Anshumali Shrivastava 1 2 5 6
Abstract
Adapting pre-trained large language models
(LLMs) is crucial but challenging due to their
enormous size. Parameter-efficient fine-tuning
(PEFT) techniques typically employ additive
adapters applied to frozen model weights. To fur-
ther reduce memory usage, model weights are
often compressed through quantization. How-
ever, existing PEFT methods often yield subop-
timal model quality because they rely on restric-
tive assumptions, such as low-rank constraints
on adapters to limit the number of trainable pa-
rameters. We find that sketching, a popular data
compression technique, can serve as an efficient
LLM adaptation strategy while avoiding the low-
rank assumption. We introduce SketchTune, a
compressive adaptation strategy that compresses
LLM weights into compact fine-tunable sketches,
integrating compression and adaptation into a
unified framework. This integration eliminates
the need for complex two-path computation in
existing PEFT techniques, enabling faster and
more memory-efficient training and inference.
SketchTune is supported by mathematical insights
into matrix classes that are better approximated
using sketching rather than low-rank methods.
Our extensive evaluations with Llama and Mis-
tral models demonstrate that SketchTune outper-
forms leading PEFT methods across diverse tasks
while using substantially smaller base models and
comparable trainable parameters. As a highlight,
SketchTune outperforms LoRA, DoRA, and S2FT
on commonsense and math benchmarks using 2.6-
3.5× smaller base models and exceeds LoftQ in
accuracy by 14.48% on GSM8K with 7.3× fewer
trainable parameters.
*Equal contribution 1Rice University, Houston, TX 2xMAD.ai
3University of California, Berkeley, Berkeley, CA 4Stevens Insti-
tute of Technology, Hoboken, NJ 5ThirdAI Corp. 6Ken Kennedy
Institute. Correspond...