Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Authors: Tianyi Zhang
Junda Su
Aditya Desai
Oscar Wu
Zhaozhuo Xu
Anshumali Shrivastava
Venue: ICML 2025 poster (2025)
Keywords: large language models
fine-tuning
sketching
Primary area: deep_learning->everything_else

Abstract: Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size. Parameter-efficient fine-tuning (PEFT) techniques typically employ additive adapters applied to frozen model weights. To further reduce memory usage, model weights are often compressed through quantization. However, existing PEFT methods often yield suboptimal model quality because they rely on restrictive assumptions, such as low-rank constraints on adapters to limit the number of trainable parameters. We find that sketching, a popular data compression technique, can serve as an efficient LLM adaptation strategy while avoiding the low-rank assumption. We introduce SketchTune, a compressive adaptation strategy that compresses LLM weights into compact fine-tunable sketches, integrating compression and adaptation into a unified framework. This integration eliminates the need for complex two-path computation in existing PEFT techniques, enabling faster and more memory-efficient training and inference. SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods. Our extensive evaluations with Llama and Mistral models demonstrate that SketchTune outperforms leading PEFT methods across diverse tasks while using substantially smaller base models and comparable trainable parameters. As a highlight,  SketchTune outperforms LoRA, DoRA, and S2FT on commonsense and math benchmarks using 2.6-3.5$\times$ smaller base models and exceeds LoftQ in accuracy by 14.48\% on GSM8K with 7.3$\times$ fewer trainable parameters.

Sketch to Adapt:
Sketches for Efficient LLM Adaptation
Tianyi Zhang 12
Junda Su
Aditya Desai
Oscar Wu?
Zhaozhuo Xu
Anshumali Shrivastava 1256
Abstract
1. Introduction
Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size. Purameter-efffcient fine-tuming (PEFT) techniques typically employ additive adapters applied to frozen model weights. To fur- ther reduce memory usage, model weights are often compressed through quantization. How- ever, existing PEFT methods often yield subop- timal model quality because they rely restric- tive assumptions, such as low-rank constraints on adapters to limit the number of trainable pa- rameters. We find that sketching, a popular data compression technique, can serve as an efficient LLM adaptation strategy while avoiding the low- rank assumption. We introduce SketchTune, compressive adaptation strategy that compresses LLM weights into compact fine-tunable sketches, integrating compression and adaptation into a unified framework. This integration eliminates the need for complex two-path computation in existing PEFT techniques, enabling faster and more memory-efficient training and inference. SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods. Our extensive evaluations with Llama and Mis- tral models demonstrate that SketchTune outper- forms leading PEFT methods across diverse tasks while using substantially smaller base models and comparable trainable parameters. As a highlight, SketchTuun outperforms LoRA, A, DoRA, and S2FT commonsense and math benchmarks using 2.6- 3.5x smaller base models and exceeds LoftQ in accuracy by 14.48% on GSMSK with 7.3 fewer trainable parameters.
Recent advancements in Large Language Models (LLMs) have demonstrated their potential to drive significant progress in various fields, including natural language pro- cessing (Min et al., 2023), reasoning (Wei et al., 2022), and problem-solving (Kojima et al., 2022). These pre-trained LLMs can tackle wide range of challenges thanks to the extensive knowledge acquired during pre-training, but they still require fine-tuning for optimal performance on specific downstream tasks (Longpre et al., 2023). Unfortunately. fine-tuning LLMs can be prohibitively resource-intensive due their large size. Many existing works address this is- sue by adding a small set of additional trainable parameters while fixing the pre-trained parameters (Han et al., 2024).
Restrictive Linear Algebraic Assumptions in LLM Adapters. Parameter-efficient fine-tuning (PEFT) methods aim reduce the number of trainable parameters by impos- ing specific linear algebraic assumptions on LLM weight updates. For instance, sparsity- based approaches assume that only a small subset of weights undergo updates (Sung et al., 2021; Yang et al., 2024b), while the more popular low-rank adapter-based methods (Hu et al., 2022; Liu et al., 2024b) enforce the restrictive assumption that weight up- dates are inherently low rank. However, recent studies (Liu et al., 2024a) challenge this assumption, showing that fully fine-tuned weight updates exhibit high-rank patters.. Our empirical findings further support this, revealing that low-rank representations may not be optimal for capturing weight updates, as illustrated in Figure 1. Additionally, compressing weight updates alone may not be sufficient for fine-tuning LLMs under resource constraints. To address this, existing methods incorporate weight quantization to further lower memory for fine-tuning (Dettmers et al., 2023; Yin et al., 2023; Li et al., 2023b).
Quantized Fine- Hane--Tningg Produces Sub-Optimal Results. To reduce the memory usage for adapting full LLM pa- rameters, quantized fine-tuning methods freeze the low- bit quantized base weights and update additional low-rank adapters (Dettmers al., 2023; Yin al., 2023). How- ever, this combination of the low-rank adapter assumption and quantized weights results in sub-optimal performance compared to PEFT methods using full base models (Li et al., 2023b; Yin et al., 2023). Additionally, since quan-
'Equal contribution 'Rice University, Houston, TX 'xMAD.ai ai 3 University of California, Berkeley, Berkeley, CA "Stevens Insti- tute of Technology, Hoboken, NJ TThirAl Corp. 'Ken Kennedy Institute. Correspondence to: Tianyi Zhang <tz21 @rice.edu>> Anshumali Shrivastava <annhumall rice.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
tized model weights and trainable adapters use different bit widths, input tensors must pass through two unmerge- able computation paths during forward pass, leading to increased latency and lower throughput.
Rank Required to Explain Varlance In A
Llama-3-88, OpenChat fine tuned
Qwen2.5-78. Code fine tuned
3200
3020
2530
2300
2020
1520
SketchTune: Fine-Tunable Sketches for Unified Com- pression and Adaptation. We introduce SketchTune, a method that unifies compression and adaptation of LLMs with sketching. SketchTune uses a learned sketching al- gorithm to compress the LLM into a small set of shared sketched parameters. These sketched parameters are fully differentiable, allowing us to directly update them for adap- tation. The original parameters can be approximately recon- structed from the shared sketched parameters via a mapping matrix that projects each original parameter to a shared one. By leveraging a carefully designed, learned sketching pro- cedure, SketchTune preserves the pre-traine pre-trained capabilities of the full model while drastically reducing the model size by 3-8x. Furthermore, SketchTune goes beyond the restrictive low-rank or sparse assumption on weight updates. We pro- vide mathematical insights into the scenarios where weight updates are better approximated by sketching, as well as em- pirical evidence for why sketching can approximate weight updates with lower errors.
1500
1030
530
52
5055
25
50
75
5095
% of Variance Explained In A
Error of Approximating
Llama-3-88, OpenChat fine-tuned Llama-3-88, OpenChat fine-tuned 256 compression
64 compression
1.2
lla 6ir
0.5
0.8
Low Rank Nat ces (ranicd)) Sketching (0PP-1)
Law Rank Nau ces ('anc332)) Sketching (GP9w4)
0.1
Qwen2.5-78. Code fine-tuned 256 compression
OwenZ
Qwen2.5-78, 5-78, Code fine-tuned 64 x compression
1.02
0.95
14. Ailr
0:92
0.90
0.85
0.85
0.80
Low Rank Nat cas (ranc-3) Stetcring [CPR-11
0.00
Law Rank MaU cas (amc-32) Sectching (GP)
antention Query
Attention Koy
Adtention Valuo
Altenton Out
M.P-Ca
MLPUp
Down
atrenton Quary
Attentionape Salue Attention-e Kay
Attantion-au Out
MLP-Date
MLP-ij
Down
Through extensive experiments, we show that Sketch'Tune models achieve higher average accuracy on commonsense and math reasoning benchmarks compared to competitive PEFT baselines, while utilizing sketched models that are 2.6-3.6x smaller than the full base models used by these baselines. When compared to competitive quantized fine- tuning methods the 2-bit region, Sketch'Tune achieves 14.48% better accuracy while using 7.4x fewer trainable parameters. By leveraging dedicated CUDA kernels, Sketch- Tune demonstrates better training and inference efficiency than PEFT and quantized fine-tuning methods. Our code and model checkpoints are available publicly
Figure I. (Top 2) Minimum rank required by low-rank matrices to explain varying percentages of variance in fine- tuned LLM weight updates. (Bottom 4) Optimal approximation errors for sketching and low-rank matrices under different compression ratios.
2.1. Weight Updates Are Far from Low-Rank: Sketching Provides Superior Alternative
In this section, we provide empirical evidence that weight updates resulting from full fine-tuning of LLMs are high- rank, which limits the effectiveness of low-rank approxima- tions for capturing these updates. We further demonstrate that our proposed sketching compression technique achieves substantially lower approximation errors than con- ventional low-rank methods when representing the weight updates.
2. SketchTune: Fine-Tunabi Sketches for LLM Adaptation
Fine-tuning LLMs presents significant computational and memory challenges. To address this, we propose Sketch- Tune, novel approach that compresses LLM weights into memory-efficient sketches and fine-tunes these sketches di- rectly to adapt to downstream tasks. We begin by motivating the use of sketching for compressing model weight updates through empirical evidence. We then describe our method for learned sketching of LLM weights and the techniques for ttnninuning sketched models efficiently on GPUs.
Let W represent the original model weights and W' rep- resent the fine-tuned weights. The weight update, defined as A A W w, typically requires as much storage as the original weights. LoRA (Hu al., 2022), a prevalent PEFT method, compresses weight updates by representing them a product of two low-rank matrices. To evaluate the effectiveness of this approach, we examine the capacity of low-rank matrices to approximate weight updates from two fine-tuned LLMs: Llama-3-8B (Dubey et al., 2024), fine- tuned on OpenChat (Wang et al., 2024), and Qwen2.5-7B. fine-tuned on source code (Yang et al., 2024a).
Pttpps: // thub. com www.///////ooo/ooooooooooo httmm///eaantolllsss/teteteeeee htt.tecchsseetttteeeeee.a
2

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
projection (Raskutti & Mahoney, 2016), but these stochas- tic methods can corrupt the pre-trained knowledge embed- ded in LLM weights. To prevent such degradation, we introduce a learned sketching approach that preserves the model's pre-trained capabilities. The core idea is to com- press each weight matrix W E R" into a sketched matrix W sketched E Ikr where k < c, without significantly af- fecting the loss of the model. During inference, we recover approximate weight matrix W E RR"x from W sketched on-the-fly for matrix multiplications.
W
Layer Input
Sketching Matrix
X
Kept in GPU Memory
X
During Training Inforonce
W
W sketched
Tralnable,
W
-0.1
0.2
-0.7
0.3
Sketched
W
Weight Matrix
Parameters
Reconstructed Weight Matrix
10
10
11
Integer, Parameter-mapping Codes
c Trainable Fixed
For learning sketched matrices, our goal is to keep the loss of the sketched model as close as possible to that of the original network. Let Wa and W.r denote the tensors representing the weights of network A and its recovered weights from sketches, respectively. The learning objective for the sketched weights is to minimize L(W x))) L(W,), where L(W Ar) represents the loss evaluated at Wxr. To make the learning tractable, we employ an approximation proposed by Nagel et al. (2020):
Figure 2. An illustration of SketchTune's process of sketching for model compression and fine-tuning.
We perform singular value decomposition (SVD) on each weight update matrix A to determine the matrix rank re- quired capture a specified percentage of variance. Figure illustrates the rank (and its standard deviation) necessary to account for different levels of variance in A. The results show that an average rank exceeding 1000 is required to ex- plain merely 75% of the variance. This finding suggests that standard low-rank approaches, typically employing ranks ranging from to 64, may inadequately capture the com- plexity of fine-tuned weight updates.
((WN) - C(WN)-C(WN) ~ E WWX-WXI3 (I)
WEN
where W denotes a weight matrix the network, and X represents its corresponding input matrix. This approxi- mation offers two key advantages: 1. it decomposes the learning into a layer-wise convex problem, making it com- putationally feasible, 2. it enables layer-by-layer learning, allowing large models to be processed on a single GPU. Furthermore, since ||W - W can be expressed as a sum over the products of all row vectors of W and X, the learning process can be further decomposed into row-wise independent problems. This leads to the learning objective:
In contrast, we propose SketchTune, a novel approach that uses sketching to compress and represent weight updates. Unlike traditional PEFT methods that rely on adapters or additional parameters added to frozen weights, SketchTun directly compresses the entire model through a set fine- tunable sketched parameters. Specifically,, the sketched model utilizes shared parameters along with mapping matrix that associates each original parameter with these shared parameters.
arg min || w
wX w 2
where w and w represent row in the original weight ma- trix and its corresponding approximation recovered from row sketches, respectively. We learn the sketched weights by minimizing output distortions introduced by the recon- structed weights, using a small sample drawn from the em- pirical distribution of X.
We compare the quality of SketchTunee and low-rank matri- ces in approximating the weight updates on the previously mentioned Llama and Qwen models. We measure the ap- proximation quality using the normalized approximation error defined as |A|F LAlم where Á represents the approxi- mated weight update (see Appendix B). Figure shows that SketchTune achieves lower approximation errors than low- rank matrices across most layers, suggesting its superior effectiveness in capturing weight updates.
2.3. A Row-Wise Learning Strategy
For each weight matrix W E Re in the LLM,, we inde- pendently compress each row vector w E RRixe using a learned sketching process. The process involves learning two matrices: sketching matrix S for compression and mapping matrix M for reconstruction.
2.2. Formulation: Learning to Sketch LLM Weights
Sketching is data compression technique that preserves essential properties of the original data while substan- tially reducing memory requirements. Conventional ma- trix sketching techniques employ randomized algorithms such as column/row sampling (Liberty, 2013) and random
The sketching matrix S E cx projects w into a lower- dimensional space, producing a sketched row w Waerohed E Rixk The mapping matrix M E {0, 1)*xx,, a column- wise oot binary matrix, recovers an approximation w
3

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
RIxc of the original row for use in matrix multiplications. Formally, this process is defined as:
1
H produced by the weighted k-means
a algorithm. Then, the i-th row and j-th column of the sketch- ing matrix S is given as follows:
W Waketched = wS,
W W sletched M.
(3)
Wx
After sketching, the original row w and the sketching ma- trix S are longer required. Only the sketched weights Waketched and the mapping matrix M need to be stored in GPU memory during training and inference, leading to re- source efficiency.
if jj
S..
Σ
(5)
0
otherwise
The final learning procedure for model sketching proceeds as follows: first learn the sketching matrix S through weighted k-means, and obtain the sketched parameters w sketched as wS. We then initialize the mapping matrix M to be empty, and iteratively learn the columns of M. In each learning step, we fix the next column of M to map the next parameter to its nearest sketched parameter in w skatched, and apply an update to the unmapped parameters to compensate for the errors. During the iterative process, the error in the weight update can accumulate, degrading sketched model quality. To address this, we apply the strat- egy proposed by Frantar et al. (2022b) to use the Cholesky reformulation for inverse Hessian calculations and apply the weight updates in a block-wise manner. Specifically, we di- vide the weights into blocks of B 128 columns and keep weight updates contained to those columns. Once all param- eters within the block have been mapped, we apply a global weight update to the rest of the unmapped parameters.
To learn the mapping matrix M for parameter reconstruc- tion, we apply the iterative quantization strategy proposed by Frantar et al. (2022b) and Frantar& & Alistarh (2022) to pre- serve model quality. Each column of M is a binary one-hot vector that maps an original parameter (from a row of size c) to one of the k entries in the sketched parameters W Waketcheds where k < c. This mapping inevitably introduces some error in the model output. To minimize this error, we learn the columns of M sequentially and iteratively update the remaining unmapped parameters to compensate for the in- troduced error after each step. Concretely, for each original parameter, we identify the entry in w Waketched that is closest to it and assign the corresponding one-hot value to M. After fixing this column of M, we apply an update 5 (Equation 13 in the Appendix) to the remaining unmapped parameters to absorb the approximation error. This process is repeated until all columns of M are assigned.
For learning the sketched parameters Waketched, straight- forward method is to perform clustering on the original parameters to obtain a set of k centers as the sketched pa- rameters. To minimize the impact on model quality, we prioritize preserving the precision of parameters with large, outlying inverse Hessian diagonals h... where H is the second-order derivative of Equation 2. We achieve this by adopting the learning objective proposed by Zhang & Shrivastava, which emphasizes the preservation of more influential parameters:
2.4. Scaling Up Number of Trainable Parameters
To scale up the learning capacity of the sketched models, we need to increase the count of trainable parameters. Unlike adapter-based methods, the number of trainable parameters in SketchTune models is fixed after model sketching. Hence, to allow more flexibility in the amount of sketched parame- ters, we propose to divide each row into multiple sub-rows, and sketch each sub-row independently. Specifically, we divide each row E Ixe into g non-overlapping, con- tiguous groups of sub-rows w' E R 1x5 5, With g groups per row, we are increasing the number of trainable parameters g-fold compared to row-wise sketching. We use the nota- tion SketchTuneerPR- g to represent a sketched model with I groups per row (GPR). With everything put together, the final algorithm for weight sketching is given in Algorithm 1. We present quality comparison between the sketched mod- els and the original models in Appendix G.
Σ H )' RTT((w;, Waketehea) w;
arg min
Watetobed E
(4)
where RTN(W,, W Woketched) (round-to-nearest operator) rounds the value of w, to its nearest sketched parameter in Wsketched, and s is hyperparameter controlling the em- phasis on preserving outliers of H.T,, which we set s 3. More details on the mathematical derivations are presented in Appendix D.
2.5. Fine-Tuning Sketches
Once the model weights have been sketched, the original weights w and the sketching matrix S are no longer needed for training or inference. During training and inference, we use the sketched weights w Wakechhed and the mapping matrix M to reconstruct weights w as w Waketcheg Thus, with X being the layer input, the forward pass computes
Using Equation 4 as objective, we learn W Wkketched by lever- aging the weighted k-means (Lloyd, 1982; Zhang & Amini, 2021) algorithm. Consequently, we learn the sketching matrix as follows. Let aI,...,ae E {l....,k) k} be the cluster indices of parameters Wiy.rr, We weighted by
4

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Algorithm 1 Learning Sketch LLM Weights
2.6. Custom CUDA Kernel for Efficient Training and Inference
1: Function LearsShethhinggMaariiiwww X)
Input: sub-row weights w', layer imput X
We develop dedicated CUDA kernels for efficient training and inference of sketched models GPUs by leveraging the shared memory (see Appendix E for details). In Sec- tion 4.2, we perform a comprehensive evaluation of the efficiency of SketchTune during training and inference and compare against competitive methods. To ensure efficient storage, we store the mapping matrix M E {0, 1)*xx, column-wise one-hot binary matrix, as an integer matrix. Due to its one-hot nature, each column of M can be com- pactly represented with the index of its one-hot entry using [log2 k] -bit integer. To leverage the full bit widths, we take k E (16, 8, 4) to use the data types INT4, INT3, and INT2, respectively.
Output: sketching matrix
2: air..., ag WeightedKKeens
(w/
w' w'gh
w.
a jj
3: let S E Rack Stn +
2:
otherwise
4: return S
5: Function Learn LoarnToSketch(w, X,
Input: row weights w, layer input X, groups per row G
Output:
sketched weights of all G group G-1
w aakecched sketched the mapping matrix M
3. Theoretical Analysis
6: M < Oaxe
b initialize mapping matrix b initialize reconstructed weights initialize weight errors
7: w + 01xe
The properties of the true update matrix A, i.e. the update matrix obtained after full fine-tuning, determine good as- sumption for compression of weight update. However, the true is not known apriori. In this section, we analyze characteristics of A and the effect of sketching-basee meth- ods such as SketchTune, especially against popular low-rank approximation alternatives. Since A and the mappings Ms for each row derived from W can be unrelated, it is safe to assume (M) is random w.r.t A. For ease of exposition, we assume that [M]s belong to a specific kind of random sketching matrices derived from random-foo fold hashing (De- sai Shrivastava, 2023). Our result is presented below:
8: e 01xB
9: H cholesky (oooosky(22Xx")
[2XX I-)
10: for + O,..., 1 do
11: w' <
get current sub-row 12: S - LearnSketchingMattiin (
13: w ekeechhdd f -WS
14: for + 96 + +B.. 2B, .... (g+ 1 do 1
for j ,,..,
16:
m - arg min; [wa sketched
17:
Mmj + b set the current mapping column
Wj + [w9 Sketthhadmm
19:
Oj-i
W
H 3.3
Theorem 3.1. Consider a matrix A x n with sorted (descending) singular values {p: squares of which are drawn from power law i parameterized by coefficient n. Under the compression factor a (i.e. using ne fa parame- ters), let low-rank approximation and sketch approximation be A and A respectively. Then, the low-rank error is
20:
W)i(+B) f Wj((+B) Cj-; BJ
21: end for
22:
B)+ + W(+ eH ((++B)
23: end for
24:
end for
25: return [w) sketched w
G-1 sketched M
n/2k
| - - =
P
The expected error of random-fold sketching approximation is,
y = WsketckedMIX. For adaptation, we freeze M and perform propaaaaonn to update the sketched parameters W aaketched- The gradients of the sketched parameters are given as:
1 &
E(IA -
Σ
P
(8)
For large enough n, the expected sketching approximation error smaller than low-rank approximation error if
ac oy
(MX)*
T
(6)
17 E 0,
log(a)
8w sketched
log (2x)
The proof of the theorem presented in Appendix F. The above theorem characterizes matrices that are well approxi- mated by sketching instead of low-rank decomposition.
We present an illustration of the sketching and fine-tuning process of Sue Figure 2.
5

---

Sketch to Adapt:
Sketches for Efficient LLM Adaptation
Table Accuracy of SketchTune compared competitive PEFT methods for fine-tuning Llama models math datasets. Baseline results
are taken from Yang et al. (2024b). SketchTune achieves better comparable accuracy while using sketched models that are 2.6-3.6x smaller than the full base models used by other PEFT methods.
Base Model Trainable
Model
Method
MultiArith GSM8K AddSub AQuA SingleEq SVAMP MAWPS Avg.
(GB)
Param (M)
GPT-3.5
83.8
56.4
85.3
38.9
88.1
69.9
87.4
72.8
Full FT
13.48
6,738.4
98.8
43.1
91.1
20.9
94.3
60.6
88.2
71.0
LoRA
13.48
55.9
98.0
40.0
91.2
21.7
93.1
56.7
85.3
69.7
DoRA
13.48
56.6
97.3
38.9
89.6
22.4
93.9
58.4
85.3
69.4
LLaMA-7B
S
13.48
54.6
98.8
41.3
91.4
21.3
93.5
58.4
86.1
70.1
SketchTuneGrReI
3.89
21.8
97.8
36.5
89.9
25.2
90.7
55.7
86.6
68.9
SketchTune<rR-22
3.93
43.5
96.8
39.0
92.2
20.1
55.5
86.6
69.0
4.02
98.3
39.7
90.9
22.0
58.0
87.4
70.0
SketchTuneGpR-a
4.19
174.1
98.3
40.6
91.9
19.7
95.1
57.5
88.7
70.3
Full FT
26.03
13,015.9
98.3
47.6
92.9
26.0
95.1
65.7
88.7
73.5
LoRA
26.03
87.2
97.5
47.8
89.9
20.5
94.3
61.2
87.4
71.2
DoRA
26.03
88.5
97.2
48.1
90.6
20.9
93.9
63.8
88.2
71.8
LLaMA-13B
S'FT
26.03
84.6
97.7
48.4
90.4
22.8
95.5
63.9
87.8
72.4
SketchTunegrre-
7.14
34.1
97.2
44.0
88.6
26.0
91.7
64.9
85.7
71.2
Sketch/luneGrR-2
7.21
68.2
98.2
46.9
91.1
27.2
93.9
61.8
86.6
72.2
SketchTuneGPR--
7.36
136.3
98.5
91.9
24.0
95.9
64.2
89.1
73.1
Sketch/TunegpR-s
7.67
272.6
98.8
47.6
92.2
25.2
95.5
66.8
87.4
73.4
Full FT
13.48
6,738.4
99.3
47.5
91.1
24.4
96.7
62.5
89.1
72.9
LoRA
13.48
55.9
97.5
44.0
91.2
20.9
94.1
59.2
85.7
70.4
DoRA
13.48
56.6
98.2
43.8
90.1
24.4
94.5
59.1
89.1
71.3
LLaMA2-7B S'FT
13.48
54.6
98.5
44.3
91.1
25.2
94.7
61.8
88.2
72.0
SketchTTunGGPe:
3.92
21.8
98.0
41.4
89.6
26.4
92.9
59.3
89.1
71.0
SketchTunegrr-4
3.97
43.5
98.8
43.5
92.2
20.5
95.3
59.9
89.1
71.3
SketchTunegrra.
4.05
87.0
99.3
46.5
23.2
94.5
59.8
88.2
71.8
SkeichTunerPR-a
4.23
174.1
98.7
46.5
93.9
24.0
61.7
90.3
73.1
Full FT
16.06
8,030.3
99.2
62.0
93.9
26.8
96.7
74.0
91.2
77.7
LoRA
56.2
99.5
61.6
92.7
25.6
96.3
73.8
90.8
DoRA
16.06
57.0
98.8
62.7
92.2
26.8
96.9
74.0
91.2
77.5
LLMMA3-BB
S2FT
16.06
56.2
99.7
65.8
93.7
31.5
97.8
76.0
92.4
79.6
SkeichTunegree-
5.77
22.0
97.8
66.3
90.1
26.8
95.5
79.8
90.8
78.2
SketchTuneerre-2
5.81
44.0
98.3
69.4
90.6
29.5
94.3
76.8
91.2
78.6
SketchTunecrre-
5.92
88.1
99.2
68.2
91.4
30.7
97.0
76.2
92.4
79.3
SkeichTuneurR-s
6.10
176.2
99.7
68.8
92.7
77.5
92.9
79.9
implies the update-matrix A close to full-rank, i.c. 77 is closer 0, then SketchTune well suited approximate A, whereas v) closer to 1 would make low-rank superior alternative. Clearly, as far from being low rank, which indicates the superiority of SketchTune over low-rank approximations.
Models and Benchmarks. We fine-tune and evaluate the following models: Llama-7B, 2. Llama-13B (Touvron et al., 2023a), 3. Llama-2-7B, 4. Llama-2-13BB (Touvron et al., 2023b), 5. Llama-3-8B (Dubey al., 2024), 6. Mis- tral-7B (Jiang et al., 2023). For math problem-solving, fine-tune these models on the Mathl0K dataset and evaluate on 7 different math reasoning datasets (Hu et al., 2023). For commonsense reasoning, we fine-tune on the Common- sense 17KK dataset and evaluate on different commonsense reasoning datasets (Hu al., 2023). For instruction fine- tuning, fine-tune Mistral-7B (Jiang et al., 2023) on the Alpaca-GPT4 dataset (Peng al., 2023) for one epoch and evaluate it on MT-Bench (Zheng et al., 2023) using GPT-4o as a judge. Detailed descriptions of the datasets are provided in Appendix H. To compare Sketch Tune against efficient quantized model fine-tuning methods, we follow the settings in Li al. (2023b) fine-tune and Llama-2 models
from Figure that A is
4. Experiments
We conduct comprehensive experiments evaluate the adaptation capabilities of SketchTune against competitive baselines across diverse tasks, including math problem solv- ing, commonsense reasoning, and instruction following. Below, we introduce the datasets, models, and baselines used for evaluation, as well as the software and hardware for conducting the experiments.
6

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Table Accuracy of SketchTune compared competitive PEFT methods for fine-tuning Ilama models commonsense reasoning datasets, Baseline results are taken from Yang et al. (2024b). SketchTune achieves better or comparable accuracy while using sketched models that are 2.7-3.5x smaller than the full models used by other PEFT methods.
Base Model Trainable Param (M)
Model
Method
BoolQ
PIQA SIQA HellaSwag Wino ARC-e
ARC-e
OBQA
Avg.t
(GB)
ChatGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
Full FT
13.48
6,738.4
70.3
84.2
80.1
92.3
85.4
86.6
72.8
83,4
81.9
LoRA
13.48
55.9
69.2
81.7
78.4
83.4
80.8
79.0
62.4
78.4
76.7
DoRA
13.48
56.6
68.5
82.9
79.6
80.8
81.4
65.8
81.0
78.1
Galore
13.48
55.9
68.6
79.0
78.5
84.7
80.1
80.3
62.1
77.3
76.3
LoReFT
13.48
2.0
69.3
84.4
80.3
93.1
84.2
83.2
68.2
78.9
80.2
Llama-7B
LISA
13.48
667.8
70.4
82.1
78.7
92.4
82.9
84.9
70.2
78.4
80.0
F'FT
13.48
54.6
72.7
83.7
79.6
93.4
83.5
86.1
72.2
83.4
81.8
SketchTunearees
4.02
87.0
72.1
85.6
80.2
84.6
86.2
71.0
84.8
82.3
Full FT
26.03
13,015.9
74.5
86.3
81.3
94.4
86.9
89.7
77.9
88.8
85.0
LoRA
26.03
87.2
72.1
83.5
80.5
83.7
82.8
82.4
80.5
DoRA
26.03
88.5
72.4
84.9
81.5
92.4
84.2
84.2
69.6
82.8
81.5
LoReFT
26.03
3.9
72.1
86.3
81.8
95.1
87.2
86.2
73.7
84.2
83.3
Llama- 13B
F
26.03
84.6
74.2
85.7
80.7
94.9
86.4
88.4
76.3
87.8
84.3
Skeich T
7.36
136.3
73.9
87.4
82.5
95.6
90.3
75.7
89.4
85.1
Full FT
13.48
6,738.4
74.7
84.9
78.7
93.7
84.1
87.5
75.2
85.0
83.0
LoRA
13.48
55.9
69.8
79.9
79.5
83.6
82.6
79.8
81.0
77.6
Llama-2-75
DoRA
13.48
56.6
71.8
83.7
76.0
82.6
83.7
68.2
82.4
79.7
S'FT
13.48
54.6
72.9
86.1
80.2
94.3
85.5
87.2
74.6
83,4
83.0
SketchTunecer:
4.05
87.0
73.3
86.2
81.2
94.1
85.4
87.6
75.2
85.8
83.6
Full FT
16.06
8,030.3
73.9
86.2
79.1
93.1
85.8
88.1
78.2
84.0
83.6
LoRA
16.06
56.2
70.8
85.2
79.7
92.5
84.9
88.9
78.7
84,4
82.5
Llama-3-8B
DoRA
16.06
57.0
74.6
89.3
79.9
85.6
90.5
85.8
85.2
F'FT
16.06
56.2
75.0
89.0
80.7
96.5
88.0
92.5
83.4
87.8
86.6
Sketch Tunerreu
5.92
88.1
75.0
90.2
82.7
95.9
88.2
92.6
82.1
89.4
87.0
Table MT-Bencc scores for Mistral-7B fine-tuned on the Alpaca-GPTV training set. The baseline results are taken from Yang et al. (2024b). Despite using 3.1 smaller base model, SketchTune achieves better average than baselines.
Method
Base Model (GB) Writing
Roleplay Reasoning
Code
Math
Extraction
STEM
Humanities
Avg.
Full FT
14.48
5.50
4.45
5.45
2.50
3.25
5.78
4.75
5.45
4.64
S'FT
14.48
6.95
4.40
5.50
2.70
3.55
5.95
6.35
6.75
5.27
SketchTune<pa-
4.66
4.60
5.20
9.23
3.05
4.80
7.45
8.13
8.45
6.36
on the language modeling dataset WikiText-2 (Merity al., 2022) and the math reasoning dataset GSM8K (Cobbe et al., 2021). Additional instruction following evaluations against quantized model fine-tuning methods are presented in Ap- pendix K.
ods that fine-tune quantized models, we use QLoRA (Detmers al., 2023). 2. LoftQ (Li al., 2023b) and com- pare with SketchTune different bit widths. We optimize SketchTune's cyyprr-prrmmete including learning rate and batch size, through parameter sweep, and we report the perpaaaareetr for training in Appendix I.
Baselines. We compare SketchTune against the following PEFT baselines: 1. Galore (Zhao et al., 2024), 2. LoReFT (Wu al., 2024), 3. LISA (Pan et al., 2024), LoRA (Hu et al., 2022), 5. DoRA (Liu al., 2024b), 6. S²FT (Yang et al., 2024b). We also report the results of full fine- and ChatGPT
Software and Hardware. We implement Sketch Tune using Toroh (Paszke al., 2019) and the Transformers library (Wolf al., 2020). We develop custom CUDA kernels op- timized for the specific operations required in SketchTune ensuring herf-peeformance execution modern GPU ar- chitectures. We sketch each model using single Quadro RTX 8000-48GB GPU. For model training, we train each model using a single NVIDIA A100-40GB GPU.
tuning, GPT-3.5 Seext-Davinci-oe (gpt-3.5-turbo. from Hu et al. (2023). For Sketch- Tune, we use sketched models compressed with the INT4 data type and compare them against baselines that use the original weights as the base model. For baseline meth-

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Table 4. Perplexity and accuracy of SketchTune compared to QLoRA and LoftQ. two efficient fine-tuning methods for quantized models, various bit- widths. Baseline results are taken from Li et al. (2023b). N.A denotes that the model failed to converge. SketchTune achieves better comparable perplexity and accuracy while using 1.8-29.4x less trainable parameters than baseline methods.
WikiText-2
GSM8K
Data Type
Llama-2-7B
Llama-2-135 2-1BB
Data Type
Llama-2-7B
Llama-2-13B
Method
Method
Trainable Param (M)
Trainable Param (M)
Trainable Param (M)
Trainable Param (M)
PPL. +
PPL J
ACC1
ACC1
LoRAnni-64
FP16
159.91
5.08
250.35
5.12
64RAmn2=64
FP16
159.91
36,90
250.35
43.10
MoRAmml=64++4
FP16
159.91
250.35
LoRAmmt-64+ Rog
FP16
159.91
34.40
250.35
45.30
QLoRArank=s4
NF4
159.91
5.70
250.35
5.22
QLoRArank=54
NF4
159.91
35.10
250.35
39.90
rantQ
NF4
159.91
5.24
250.35
5.16
LoftQuan:644
NF4
159.91
35.00
250.35
45.00
SketchTuneGPR=1
INT4
21.76
5.32
34.08
4.81
SketchTunecrr-4
INT4
87.03
39.73
136.31
50.34
QooRAran--64
NF3
159.91
5.73
250.35
5.22
QLoRAnnn-644
NF3
159.91
32.10
250.35
40.70
LoftQrank-G4
NF3
159.91
5.63
250.35
5.13
LoftQuank-64
NF3
159.91
32.90
250.35
44.40
Sketch TUneGPR-1
INT3
10.90
5.63
17.04
5.05
Sketch -uneCRTun
INT3
43,50
37.15
68.16
47.54
QLoRArank-G4
NF2
159.91
N.A
250.35
N.A.
QLoRArank-G4
NF2
159.91
N.A.
250.35
N.A.
LankQrank-44
NF2
159.91
7.85
250,35
7.69
LoftQrank-6a
NF2
159.91
20.90
250.35
25.40
Sketch TUnCGPR-1
INT2
5.44
7.40
8.52
6.22
SketchTunecPre
INT2
21.75
29.95
34.08
39.88
Time to Taken (bstch size=1)
Partoken Decading Latency (context length= 2000)
Training Batch (betch size==1
Peak GPU Memnryy Usage (batch size=1)
GPU Mamory (CB)
600
Latency (mnc)
Time (ms)
Latency
402
220
80-
150
202-
100-
20-
100
contect wo2th-40200
contect (rnghh 1000
batch
baich
conteat 1e1ghh-512
Thirring
Infoeare
(romrst lengf 5-512) (corteal Inngth-103000
LoRA 32
QLoRA
ZZ
SketchTuncun-11
SketchTunegm--2 -2
SketchTunCupx.-
SketchTune
Figure 3. A comparison the training and inference efficiency of SketchTune, LoRA, and QLoRA.
4.1. Results
efficiency and final performance, all SketchTune models use the INT4 representation and GPR=4 for model sketching, while the baseline methods employ the original weights as the base. Despite using 2.7-3.5x x smaller base model size, SketchTune consistently outperforms competitive PEFT methods and even full fine-tuning in the average accuracy across benchmarks.
1. COMPARISON WITH PEFT METHONS
Math Problem-Solving. Table 1 presents the accuracy results of models fine-tuned on the Math dataset (Hu et al., 2023). SketchTune models are compressed to use INT4 representation for weights, while baselines use full models for fine-tuning. We explore the effectiveness of SketchTune at different compression rates by varying groups per row (GPR) in the sketched models. We highlight that SketchTunegpr-& consistently achieves the best accuracy compared to PEFT baselines despite using 2.6-3.4x smaller models, and is on par with full fine-tuning and GPT-3.5. Regarding parameter efficiency, SketchTunearR-2 performs better or on par with LoRA and DoRA, despite using fewer trainable parameters. We also conclude that scaling up the number of parameters by increasing GPR leads to a consis- tent improvement in the average accuracy across datasets.
Instruction Fine-tuning Table 3 shows the MT-Bench scores, judged by GPT-4o, for SketchTune F and fully fine-tuned models. SketchTune outperforms both baselines on most tasks, while using 67.8% smaller base model.
4.1.2. COMPARISON WITH COMPRESSIVE TUNE-TUNIGG
Table 4 reports the perplexity WikiText-2 and accuracy GSM8K for models fine-tuned with QLORA, LofiQ, and SketchTune at different bit widths. QLORA and LoftQ use 4/t/2-bit NormalFloat (NF4/NF//NF2) (Dettmers et al., 2023), while SketchTune uses 4/3/2-bit integers (INTW/NT3//NTT). SketchTune achieves lower perplex- ity on WikiText-2 in most cases and higher accuracy on GSM8K across all bit widths. Notably, it outperforms LoftQ
Commonsense Reasoning. Table 2 presents the accuracy results of models fine-tuned on the Commonsensel 70K dataset. To maintain a practical trade-off between memory
8

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
by 4.73 and 5.34 points on GSM8K (4-bit) with 1.8x fewer trainable parameters and by 9.05 and 14.48 points (2-bit) with 7.3x fewer parameters. Moreover, SketchTune sur- passes LoftQ on WikiText-2 (2-bit) while using 29.4x fewer trainable parameters. Details on calculating SketchTune's s number of trainable parameters can be found in Appendix C.
have been explored as effective methods for compressing neural networks (Xu, 2025), aiming to reduce computa- tional and storage requirements while maintaining perfor- mance. Random sketching for model compression has been explored recently in a variety of settings, including embed- ding compression (Chen et al., 2015; Desai et al., 2022; Desai & Shrivastava, 2022) and general model compression (Desai al., 2023; Desai & Shrivastava, 2023). However, these methods not suitable for compressing models after training; they are most effective for training compressed models from scratch. SketchTunne is among the first to ex- plore sketching techniques for post-training model compres- sion. Other techniques include multi- halti-hhsshig (Eban et al., 2020), tensor sketching (Kasiviswanathan et al., 2017), ran- dom projection (Ravi, 2019), linear sketches (Daniely et al., 2016), and higher-order count sketch (Shi & Anandkumar, 2020).
4.2. Efficiency Analysis
In Figure 3, we report the training and inference efficiency of SketchTune. Moreover we compare SketchTune with LoRA and QLoRA. All experiments are performed on an NVIDIA A100-40GB GPU. The results are averaged over 10 runs, each with 10 warmup steps. LoRA and QLoRA- based methods keep base and adapter weights separately, which requires two matrix multiplications for each layer, leading to inefficiencies and difficulties in implementation. SketchTune is free of adapters and requires a single ma- trix multiplication for each layer. SketchTune demonstrates 2.0-2.4x lower time to first token (TTFT) than LoRA, and 2.9-3.3x lower TTFT than QLORA. For decoding latency, SketchTune is consistently faster than LoRA and QLoRA. For training time, SketchTune is lower or on par with base- lines. Regarding GPU memory usage, SketchTune con- sumes 1.6-2.7x less memory than LoRA during training and inference due to the smaller size of sketched models.
6. Conclusion
In this work, we introduced SketchTune, a novel approach that unifies model compression and adaptation through weight sketching. Our method addresses the fundamental limitations of existing PEFT approaches by eliminating low- rank constraints and avoiding the computational overhead of separate adapter paths. Through theoretical analysis and comprehensive empirical evaluation across diverse tasks, we demonstrated that SketchTune achieves superior perfor- mance while using significantly smaller base models than competitive baselines. These results establish SketchTune a promising direction for efficient adaptation of LLMs.
5. Related Works
Resource-Efficien Fine-Tuning of LLMs. As fine-tuning LLMs is resource intensive, existing works reduce the computational and memory resources demands through parameter-efficient adapters, optimizer state compression, base model quantization, and more (Han et al., 2024). Adapter-basee based methods attempt to reduce the parameters for capturing model weight updates through low-rank methods (Hu et al., 2022; Liu et al., 2024b), vector-based random matrix adaptation (Kopiczko et al., 2024), sparsity (Guo et al., 2021; Yang et al., 2024b), orthogonal fine-tuning (Qiu et al., 2023; Liu et al., 2024c), etc. Optimizer states typ- ically consume twice the amount of memory as trainable parameters (Loshchilov & Hutter, 2019). and existing works reduce this overhead through low-rank approximation (Zhao al., 2024) and quantization (Dettmers et al., 2022; Li et al., 2023a). To reduce the memory demand for the base model, existing works (Dettmers et al., 2023; Li et al., 2023b; Qin et al., 2024; Yin et al., 2023) quantize the full model to inte- gers and fine-tune full-precision adapters added on top. To make L.MSS more memory efficient, existing works apply compression the model weights (Frantar et al., 2022a; Zhang et al., 2025), activations (Xiao et al., 2023), and KV cache (Zhang et al., 2024a;b), while maintaining model accuracy.
A Acknowledgements
This work was supported by National Science Foundation SHF-2211815 and Ken Kennedy Institute Cluster Grants.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Ansell, A., Vulié, L., Sterz, H., Korhonen, A., and Ponti, E. M. Scaling sparse fine-tuning to large language mod- els, 2024. URL https: //axivv.. org/ abs/ 2401. 16405.
Bisk, Y., Zellers, R., Gao, Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence,
Sketching for Model Compression. Sketching techniques

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
volume 34, pp. 7432-7439, 2020.
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. Docu- menting large webtext corpora: A case study the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, 2021.
Chen, W. Wilson, J.. Tyree, S., Weinberger, K., and Chen, Y. Compressing neural networks with the hashing trick. In International conference on machine learning, pp. 2285- 2294. PMLR, 2015.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL https: //arxiv. , org/ abs/aaass/90 10044.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Eban, E., Movshovitz-Attias, Y., Wu, H., Sandler, M., Poon, A., Idelbayev, Y., and Carreira- Perrpina,, M. Struc- tured multi- for model compression. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11903-11912 2020.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A. Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: // /arxiv.....//////////////////tv/ org/ abs/ 0003.05457.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 1221110.111100 14168, 2021.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022a.
Daniely, A., Lazic, N., Singer, Y., and Talwar, K. Sketching and neural networks. arXiv preprint arXiv:1604.05753 2016.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Optq: Accurate quantization for generative pre-trained trans- formers. In The Eleventh International Conference on Learning Representations, 2022b.
Desai, and Shrivastava, A. The trade-off's of model size in large recommendation models: 100gb to 10mb criteo-tb dirm model. Advances in Neural Information Processing 2022.
Systems,
Guo, D., Rush, A., and Kim, Parameter-efficient transfer learning with diff pruning. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Pupers), pp. 4884-4896, Online, August 2021. Association for Com- putational Linguistics. doi: 10. 100.18663/11/20011ac./olo 378. URL acl-long. 378//
Desai, A. and Shrivastava, A. In defense of param- eter sharing for model-compression. arXiv preprint arXiv:2310.11611, 2023.
Desai, A., Chou, L., and Shrivastava, A. Random offset block embedding (robe) for compressed embedding tables in deep learning recommendation systems. Proceedings of Machine Learning and Systems, 4:762-778, 2022.
wwwp:/y...rrr///// 2021.
Desai, A., Zhou, K., and Shrivastava, A. Hardware-aware compression with random operation access specific tile (roast) hashing. In International Conference on Machine Learning, pp. 7732-7749. PMLR, 2023.
Han, Z. Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-effiienn fine-tuning for large models: A com- prehensive survey, 2024. URL https://anms
org/ abs 2200/44033..0 14608.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations, 2022. URL https::/ httprriiww.eeee.tteeeeeeeee id=shpkpVXzo3h.
He, H., Li, J. B., Jiang. X., and Miller, H. Sparse matrix in large language model fine-tuning, 2025. URL https: //arxiv. org. aaaaab///a0 /at.//240.....55.
Hosseini, M. J., Hajishirzi, H., Etzioni, O., and Kushman, N. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523-533, 2014.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms, 2023.
URL https: /ttp/viivvvv.////////.//////aavw www.oorgaab.... 2305. 14314.
10

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// // openreview
Liberty, E. Simple and deterministic matrix sketching. In Proceedings of the 19th ACM SIGKDD international con- ference Knowledge discovery and data mining, pp. 581-588, 2013.
Ling, W., Yogatama, D., Dyer, C., and Blunsom, Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Anmual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 158-167, 2017.
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P., Bing, L.,, Xu, X., Poria, S., and Lee, R. Lim-adapters: An adapter family for parameter-efficient fine-tuning of large lan- guage models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, PP- 5254-5276, 2023.
Liu, J., Xiao, G., Li, K., Lee, D., Han, S., Dao, T., and Cai, Bitdelta: Your fine-tune may only be worth one bit, 2024a. URL https:: /twxivaivivvv.///////e/////aaa org/ abs/2402. 10193.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.,, Lavaud, L.. R., Lachaux, M.- A., Stock, P. Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //arxiv. org/ abs/ 06825.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-. F., Cheng, K.T,, and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXh::2402.09355 2024b.
Kasiviswanathan, S. P., Narodytska, N., and Jin, H. Deep neural network approximation using tensor sketching. arXiv preprint arXiv: 1710.007 2017.
Liu, W. Qiu, Z., Feng. Y., Xiu, Y., Xue, Y., Yu, L.., Feng, H., Liu, Z., Heo, J., Peng, S., Wen, Y., Black, M. J., Weller, A., and Schölkopf, B. Parameter-efficient orthog- onal finetuming via butterfly factorization. In The Twelfth International Conference on Learning Representations, 2024c. URL https: / wpw.reveewweeeeee////p net/ forum? id= 1d=7Nz 7N2gkk.gg....
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Sencel-Kezzioski, R., Hajishirzi, H., Sabharwal, A., Et- zioni, O., and Ang, S. D. Parsing algebraic word prob- lems into equations. Transactions of the Association for Computational Linguistics, 3:585-597, 2015.
Lloyd. S. Least squares quantization in pem. IEEE transac- tions on information theory, 18(2):129-1)) 137, 1982.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 22631-22666 PMLR, 2023.
Koncel Kenzil-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. Mawps: A math word problem reposi- tory. In Proceedings of the 2016 conference of the north american chapter of the association for computational lin- guisties: human language technologies, pp. 1152-1157, 2016.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL https: /apes///epeeeeeeee net/ pett/forum?ice met//forumm1dcBBgggRCCC)
Kopiczko, J., Blankevoort, T., and Asano, Y. M. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL https: https: http://ppppeeeeeteeeeteeee netriiiwww.eeeeeeeeeeeeeeeeeeeee forum? id=NjNfLdxx33.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A.
Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313- 330, 1993. URL https://aclanthan J93-2004/.
.org/
Li, B., Chen, J., and Zhu, J. Memory efficient optimiz- ers with 4-bit states. Advances in Neural Information Processing Systems, 1115136-51771 2023a.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016. Utthttps: //arxiv. org aaaaabbs/1000 07843.
Li, Y., Yu, Y., Liang C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. Loftq: Lora- trra-fine-tumee tine-tuning-wwrr quantization for large language models, 2023b. URL httpps:/aan //arxiv.. www.iiiorrrg//eoo arg/abs/// abs//3101100 08659.
Merity, S., Xiong, C., Bradbury. J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.

---

Sketch to Adapt:
Sketches for Efficient LLM Adaptation
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Ravi, S. Efficient on-device models using neural projections. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 5370-5379. PMLR, 09-15 Jun 2019. URL https::p///.... http:///pocceeeeeeeeeeeee , mllr pww....ressv/vv ravil9a. html.
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I,, and Roth, D. Recent advances in natural language processing via large pre-trained language models: A survey, ACM Computing Surveys, 56(2): 2023.
Roy, S. and Roth, D. Solving general arithmetic word problems. arXiv preprint arXiv:1608.0144 2016.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? Adaptive rounding for post-training quantization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7197-7206. PMLR, 13- 18 Jul 2020. URL httpp::/tptpeeeeeeeeeee http://pooceeee...//eee// mlr. press https//v1n///aaeaaaaaa html.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: http///xxvvvvr// abs/ 1907. 10641.
Sap. M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pp. 2019.
Pan, R., Liu, X., Diao, S., Pi, R., Zhang, J., Han, C., and Zhang, T. Lisa: Layerwise importance sampling for memory-efficient large language model mun-tuning,a4 2024. URL https: //ae:////xiivii org/ abs /200s/2403.0 17919.
Shi, Y. and Anandkumar, A. Higher-order count sketch: Dimensionality reduction that retains efficient tensor op- erations. In 2020 Data Compression Conference (DCC), pp. 394-394, IEEE, 2020.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Sung, Y-I.,, Nair, V., and Raffel, C. A. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 14:24193-24200 2021.
Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems? In Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, 2021.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followingg llama model. https: /ttt:/ggttttt www/tatsu-lall stanford palooddaapppca.. 2023.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL https: http://aaxvv.///./...../ / 2302. 13971.
Qin, H., Ma, X., Zheng, X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., and Magno, M. Accurate lora-finetuning quantization of Ilms via information retention, 2024. URL https:/ /ttxxvvvvvv//////////eeeeeaaav org/ abs 2202 05445.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-.,, Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R.,
Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Schölkopf, B. Controlling text- to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320- 79362, 2023.
Raskutti, G. and Mahoney, M. W. A statistical perspec- tive on randomized sketching for ordinary least-squares. Journal of Machine Learning Research, 17(213)11311, 2016.
12

---

Sketch to Adapt:
Sketches for Efficient LLM Adaptation
Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z.,, Zarov, I., Zhang, Y., Fan, A., Kambadur, M.. Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https: http///xxxvvvrro//
Zhang, L. and Amini, A. Label consistency in overfitted generalized k-means. Advances in Neural Information Processing Systems, 34:7965-7977, 2021.
Zhang. and Shrivastava, A. Leanquant: Accurate and seal-
able large language model quantization with loss-error- aware grid. In The Thirteenth International Conference Learning Representations.
abbs2337.092288
Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y. Openchat: Advancing open-source language models with mixed- quality data. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview www.eeeeeeei..eeeeeeee
Zhang, T., Yi, J., Xu, Z.,, and Shrivastava, A. Kv cache is I bit per channel: Efficient large language model in- ference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304-3333 2024a.
deee2ldd--ooo AOJYfhWYHE.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of of-thhughht prompting elicits reasoning in large language models. Advances in neural information processing systems, 45:24824-24837 2022.
Zhang, T., Yi, J., Yao, B., Xu, Z., and Shrivastava, A. Nomad-attention: Efficient Ilm inference on cpus through multiply-add-free attention. Advances in Neural Informa- tion Processing Systems, 37:1 177112270-112777 112330, 2024b.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-ar natural language processing. EMNLP 2020, pp. 38, 2020.
Zhang, T., Sui, Y., Zhong, S., Chaudhary, V., Hu, X., and Shrivastava, A. 70% size, 100% aceuracy: Lossless Ilm compression for efficient gpu inference via dynamic- length float. arXiv preprint arXiv:2504.11651 2025.
Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D., and Potts, C. Reft: Representation finetuning for language models, 2024. URL https: //arxiv. org/ abs/ 2404. 03592.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Galore: Memory-efficien I1m training by gradient low-rank projection, 2024. URL https: / wex.iv.orgeo aba/abaa/22 03507.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https: //arxiv.org//////...t.... wrtp:/s///aa///0//// 030..////.3..6.5..5... 05685.
Xu, Z. Compression-aware computing for scalable and sustainable ai. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28733-28733, 2025.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a.
Yang, X., Leng, J., Guo, G., Zhao, J., Nakada, R., Zhang, L., Yao, H., and Chen, B. S²ft: Efficient, scalable and generalizable Ilm fine-tuning by structured sparsity. arXiv preprint ar X 2262998,, 2024b.
Yin, J., Dong, J., Wang, Y., De Sa, C., and Kuleshov, V. Modulora: Finetuning bit llms on consumer gpus by integrating with modular quantizers. arXiv preprint 1xXi:23009161111 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. URL https: ttttps://arxx/ //arxvvvv.../////////a.////avtva org/ abs/ 1905. 07830.
13

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Appendix
A. Mathematical Notations
A summary of the mathematical notations used in the paper is presented in Table 5.
Table 5. Notations used in the paper.
Notation
Type
Explanations
C,C
Z+ Z+
The number of rows and columns in weight matrix
k
Z
The number of compressed columns in sketches
W, W
Rrxc, IR xx
A weight matrix, a row of the weight matrix The sketched parameters of a weight matrix, the sketched parameters of a row
W sketched, W sketeked
R 1x k
W,
W
R IR lxce
The reconstructed weight matrix and row from sketched parameters
S
cxck
The sketching matrix for compression, where wS = w skitthhed
M
{0. 1
The mapping matrix for reconstruction, where w = Wsketched M
B. Calculating Errors of Weight Update Approximation
In this section, we describe the details for calculating the approximation errors of weight updates using low-rank matrices and sketching. We use the update A of two fully fine-tuned models: 1. openchat hatph/chhtttpecchhhheeee wpwencatt--tteeeeeeeeeeeeccaeeee 3.5 5-1210 (Wang et al., 2024) from the base model meta- ama aMetaMMLllllll L. ama---8 (Dubey et al., 2024), 2. Qwen @wen/Qwen2, Qwen2. 5-Coder- from the base model /wven/Qwen2 7B7B (Yang et al., 2024a).
|| AA- where Å is the best approximation of A achievable using ||
We use the metric of normalized approximation error
low-rank matrices or sketching. For low-rank matrices, we calculate using the first r dominant entries of singular value decomposition (SVD), where is the rank of the low rank matrices. For sketching, we first sketch the models using the SketchTune algorithm to derive the row-wise mapping matrix M. Then, we calculate row of the optimal approximation
as
= M(M -
(10)
C. Calculating Total Number of Sketched Parameters
In this section, detail the method for calculating the number of trainable parameters in SketchTune model. Our approach involves sketching all linear projection layers within LLM, excluding the token embedding layer and the final prediction head layer.
Each row of the weight matrix in these projection layers is divided into sketching groups, where g represents the Groups Per Row (GPR) configured prior to sketching. Utilizing an n-bit weight representation, each parameter within a sketching group is encoded using an n-bit integer, allowing for 2" distinct values. Consequently, the number of trainable parameters for each linear projection layer is calculated as:
Trainable Parameters = Number of Rows g x 2"
To illustrate, consider sketching the LLaMA-2-7B model with GPR 4 and INT4 weight representation. Each transformer layer in the LLaMA-2-7B model includes key, query, value, and output projection layers with dimensions 4096 x 4096, as well as Multi-Layer Perceptron (MLP) layers sized 4096 x 11008 (down), 11008 x 4096 (up), and 11008 x 4096 (gate).
For all 32 transformer layers, the total number of trainable parameters is:
Total Trainable Parameters 32 4 x 2 (4096 x + 4096 + 11008 2) 87, 031, 808
Given that the full model comprises 6,738 million parameters, this results in a compression ratio of approximately 77x.
14

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
D. Details of Row-Wise Sketch Learning
We denote the weight reconstruction error ò and the loss error e, where
8 _ w w,
L(w + L(w).
(11)
Our objective is to solve for the optimal weight update 8 to apply to the weights w such that the loss error e is minimized. The learning process then proceeds as follows: we sequentially learn and fix the i-th column of the mapping matrix, denoted as M. during the i-th step. At each step, based on the loss error c, introduced by the current mapping, we solve for the optimal weight update ò that minimizes e,, and apply the update to the unmapped original weights. This process is repeated until all columns of M have been fixed.
To simplify solving for 8, we use second-order Taylor expansion to approximate the loss emror (Nagel al., 2020):
ac Ow
0² C
8
Ho,
where H
dw2'
Oc in pre-trained LLM.
8 Hō,
since
(12)
öw
For the Hessian H. approximate it by leveraging the second-order derivative of Equation 2, where H = 2xx". In practice, we compute X using small calibration dataset consisting of 128 sequences of 2048 tokens sampled from the C4 dataset (Dodge et al., 2021). Equation 12 can now be solved with a Lagrangian (Frantar & Alistarh, 2022), which yields the following solutions for 6; and 6;
6; W Waketched M. W:
(( sketchhd -w,) w,)
=
(13)
2
H
where H H., " is the i-th diagonal entry, and the i-th column of the Hessian, respectively. From the solution of G, we gain two key insights towards minimizing the loss error:
1
E; X (W sketohed M. w
E X
(14)
H
These two facts lead to two takeaways: 1. Since the loss error is proportional to the squared difference between the sketched parameter and the original parameter, a column of M should always be set to map original parameter to its nearest sketched parameter. 2. As the loss error is proportional to the inverse Hessian diagonals, the sketched parameters should be optimized to prioritize preserving the precision of weights with larger inverse Hessian diagonals. Therefore, to learn the set of & sketched parameters, we perform clustering to derive & centers from the e original parameters. We further leverage the learning objective proposed by Zhang Shrivastava to emphasize preserving parameters with outlier inverse Hessian diagonals:
E(
arg min Wateichee ERR
RTN(WW; W Waketched) W;
(15)
where RTN (Wi, W Wsketehad) (round-to-nearest operator) maps the parameter w; to the nearest sketched parameter in (W,,, W skkttched, and is a hyperparameter controlling the emphasis on preserving outliers in the inverse Hessian diagonals. In our experiments, we set
E. CUDA Kernel Implementation Details
We develop custom CUDA kernels for efficient training and inference of SketchTune. Specifically, we develop two dedicated CUDA kernels: 1. A kernel for weight reconstruction, which computes W from W Wketchedd 2. A kernel for gradient computation of the sketched parameters, which caleulates
9W siinted from oW AL
During training, the approximate weights W are only reconstructed when needed and not kept in memory to save memory usage. For the weight reconstruction kernel, each thread block responsible for reconstructing a single row weights by computing w- W Waketcked M. The mapping matrix M is stored in a packed integer format. Each thread block allocates
15

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
sufficient shared memory to cache sketched (utilizing only 32 bytes for k 16). It then reads the integer indices from M perform low-lateney lookups from sketched in shared memory and writes the retrieved entries of w to global memory.
For the gradient computation kernel, each thread block handles the computation of the gradient for a single row of sketched parameters, specifically calculating
OC
Oc dw
M
Öw sketched
Each thread block allocates enough memory to cache sitshtd times, where is the number of threads in each thread 8C and accumulates the values into its own copy of block. Threads within the same thread block reads different entries of ow
Oc
Since each thread maintains its own accumulator, atomic operations are unnecessary for ensuring consistency,
Finally, an aggregation step combines the intermediate results from all threads to produce the final gradient.
F. Theory
For the sake of analysis, consider a square matrix W n x n of post-quantized weights, where e = r = n. Let the true update under full-scale fine-tuning be A. Let A be the low-rank approximation learned via Low-rank methods. Let A, be -eeech-based adaptation learned via methods such as SketchTune. Let the compression factor be a. For simplicity, we and amn.
assume that
Errors with low-rank approximation Under the learning, the best approximation of the A under low-rank methods can be given by Eckart- Eckart-Young-We Young-Mirsky theorem.
n /2
A
Σ
Pi =
Σ
P
(16)
i=n/2k+1
where pj is the 7th largest singular value.
Errors with SketchTune Consider a single row of the matrix w : x n, the sketched matrix Waketched 1 x k. the mapping discovered M k x n. Consider how SketchTune approximates the true corresponding row in A, say 6. It tries to learn the best possible solution inside the row-subspace(MD) To analyze the errors, we will make the following assumptions,
For ease of exposition, we assume the mapping M is a balanced mapping where each parameter in Waketched is used equal number of times.
Since 8 is not known apriori, we assume that mapping M is a random w.r.t 5 with the distribution over M being the random-fold mapping defined (Desai & Shrivastava, 2023).
Under these assumptions, we can analyze the error incurred by SketchTune while approximating A. The load of each parameter W skkttched a = n/kk The approximation error can be written as follows.
Σ EmM
||8 - -112 = Σ
8, -g((
(17)
(a 1)2
E (115, - 811))
Σ
82
+
Σ
83
(18)
a
a
The 5y appears in 11 other terms. So aggregating,
(a 1)2
E (115, - 811) = Σ
8
+
a
82
(19)
a²
a
16

---

Sketch to Adapt: Fine-Tunable Sketches for Efficientt LLM Adaptation
(a
1) of
E (110, -012) =
Σ
(20)
1) 1181
E (1 8113)
(21)
Considering all the rows together and linearity of expectations,
E (IIA. - A (a
12
(22)
It is important to note that depending on the A, one of the approximations will be better than the other. For instance, if A is indeed low-rank, then LoRA will be the best approximation to use. We will show that is near full rank, then Sketching dominates LoRA-based approximations. We quantify this observation below,
Let the squared singular values, indexed by and sorted in the descending order, be represented by power-law i- parameterized by coefficient n,. If then all the singular values are 1, and rj increases, the A becomes more low-rank. " implies logarithmic sparsity. So we assume v) E [0, 1)
0 = -1) i1)
(23)
Since pj is monotonically decreasing, it can be bounded follows,
m+1
{" x=1
(x)
Σe
p 11 pz())dx
(24)
s=1
x
r
+
(25)
1 m
m
i=1
1 1- 1 ((n 11) <
p
< 1 +
1
(n' in -1)
(26)
1-
i=1
I(n)- = 1-n - 1) < Σ
2 1+
(n'- - 1) = =R(n)
(27)
1-n
Let us quantify the for which Sketching dominates Low-rank. Under the budget ls for each row, a = n/k. Sketching is superior in expectation if,
E (IIA, - 1 < A - A
(28)
which requires,
n/20x
1
(29)
i=1
1=1
Using the bounds defined above,
[(n) R(n/2a)
(30)
a
For large n, we can approximate the ratio as
17

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
6000
Sketch Tune dominates
Low-rank dominates
1024
400
C 2
L(n)
2200
R(n/288 2
n Coefficient of power law of singular values
Figure Depending on the power-law coefficient singular values proxy for rank) the unknown matrix, one method
dominates the other. We show this classification for sample case of 1024 and k 2
L())
(nn + 1)2-9-11
(31)
R(n/2a)
1 I-a
ni-a
1
(2a)1-n
1 I[(n)
a
T-
(32)
R(n/2ox
I- (2a)
[.(n) 0
1
(33)
R( ((n/2x)
(2a)-
This fraction is equal to when
(2a)²-1 1-
(34)
Thus,
log(a)
(35)
log(22 (2a)
Thus Sketching dominates the low-rank approximation on average in the region 7) E Jo. bounds is shown in figure 4
log(a) log(2a)
example of the
Theorem E.1 Consider a matrix A n X n with sorted (descending) singular values (m:) squares of which are drawn from power law i- parameterized by coefficient n. Under the compression factor a (i.e. using n² foo parameters). let low-rank approximation and sketch approximation be Ai and A respectively. Then, the low-rank error is
n/2k
| A
Σ
p
(36)
18

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
The expected error of random-fold sketching approximation is
B(IA
A
Ps
(37)
For large enough the expected sketching opproximation error is smaller than the low-rank approximation error if
17 0, log(a)
(38)
log 22a
G. Model Sketching Details
Table Perplexity of SketchTune, without fine-tuning, on WikiText-2, PTB, C4 compared full model. also report overhead of sketching time.
Model Size (GB)
Sketching Time (minutes)
Model
Data Type
GPR
WikiText-2 1
PTB
C4
FP16
5.68
41.15
INT2
4
3.89
34.93
12.37
199.47
15.10
3.93
37.86
48.92
Llama-7B
INT4
1
3.89
35.03
5.82
44.22
7.53
INT4
2
3,93
40.06
5.81
43.49
7.51
INT4
4
4.02
49.58
5.82
43.44
7.51
INT4
4.19
106.28
5.77
43.12
7.50
FP16
5.09
28.10
6.80
INT2
4
7.14
60.58
9.23
63.68
11.08
INT3
4
7.21
64.18
5.55
30.15
7.30
Llama-133
INT4
7.14
62.70
5.20
27.67
INT4
2
7.21
71.28
5,20
28.34
6.91
INT4
4
7.36
88.97
5.18
28.51
6.90
INT4
8
7.67
112.43
5.20
28.04
6.90
FP16
5,47
37.91
7.26
INT2
4
3.92
35,20
15.91
166.08
18.48
INT3
4
3.97
38.13
6.14
42.46
8.13
Illma-2-78
INT4
I
3.92
33.24
5.67
52.39
7.46
INT4
2
3.97
38.91
5.62
47.09
7.45
INT4
48.10
43.07
7.43
INT4
8
4.23
66.91
5.62
38.93
7.44
BF16
6.14
11.18
9.45
INT2
5.77
74.63
28.52
36.04
30.79
INT3
5.81
78.83
7.73
12.87
12.09
Llama-3-8B
INT4
1
5.77
69.56
6.59
11.59
10.18
INT4
2
5.81
79.00
6.54
11.63
10.12
INT4
5.92
88.83
6.52
11.52
10.09
INT4
106.28
11.49
10.00
We performed model sketching using C4 (Dodge et al., 2021) as calibration dataset (for computing the Hessian H for model sketching), consisting of 128 sample sequences each with 2048 tokens.
All model are sketched using single Nvidia Quadro RTX8000 GPU. In Table 6, provide details model sketching overhead (sketching time), as well as perplexity comparisons against the original models on WikiText-2 (Merity et al., 2022), PTB (Marcus et al., 1993), and C4 (Dodge et al., 2021) dataset, using 128 sequences of 2048 tokens each.
While Sketch Tune introduces an additional sketching step before fine-tuning, this preprocessing is fast, resource-efffcient, and one-time per base model. In Table 7, we report additional end-to-end sketching time and memory usage (INT4, GPR=4) for different sized models, using single A A100-40GB GPU and the aforementioned calibration setup.
19

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Table End to end sketching time (using INT4, GPR=4 setup) for different sized models on single A A100-40GB GPU. Thanks our layer-wise optimization objective (equation (1)), the sketching process scales efficiently to large models (70B).
Model
Original Size (GB)
Sketched Size (GB)
Max GPU Mem (GB)
Sketching Time (min)
Llama-3.2-3P
6.43
3.18
9.92
20.70
Llama-3.1
16.07
5.92
18.37
41.62
Llama-3,1-700
141.12
40.15
28.05
266.87
H. Dataset Information
Math Problem-Solving To fine-tune and evaluate math problem solving tasks, we fine-tuned models the Math10K dataset (Hu et al., 2023), which includes the training set from GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), and MAWPS (Koncel-Kedziones
et al., 2016) and agumented with language model generated chain-of-thoughts steps. We performed evaluation on 7 math datasets: MultiArith (Roy & Roth, 2016), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), SingleEQ (Koncel-Kedziorsk et al., 2015), SVAMP (Patel et al., 2021), and MAWPS (Koncel-Kezzionn et al., 2016). For each test sample, the model performs generation. And a final answer extracted calculate model's response accuracy.
Commonsense Reasoning The commonsense reasoning tasks consists of questions from 8 different datasets: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019), Arc-e, Arc-c (Clark et al., 2018), and OBQA (Mihaylov et al., 2018). The training set consists of training data from all datasets, formatted using a consistent pre-defined template (Hu et al., 2023), resulting in 170K samples. The test set from each dataset then used individually to evaluate the fine-tuned model's performance.
WikiText-2 The WikiText-2 dataset (Merity et al., 2016) consists of 44.8k training data, consisting of 36.7K training data, 3.76K validatiaon data, and 4.36K test data. Following LoftQ (Li et al., 2023b), we used the training set to perform fine-tuning, and the validataion set to evaluate fine-tuned model's 's performance.
MT-Bench The MT-Bench dataset (Zheng et al., 2023) a set of 80 challenging multi-turn open-ended ques- tions across 8 categories: writing, humanities, STEM, extraction, coding, math, reasoning, and roleplay.
Alpaca We used the Alpaca dataset (Taori al., 2023) to evaluate SketchTune's performance language gener- ation tasks. The Alpaca dataset consist of 52K training samples, with no test validation split. The responses generated with the text-davinci-O0 -03 engine. Alpaca-GPT4 (Peng et al., 2023) is similar dataset, which contains outputs generated by GPT-4 using the same prompts as the original Alpaca dataset. We adopted the FastChat framework from Zheng et al. (2023) to perform pair-wise compitition, using GPT-4o as a judge for model generation quality.
I. Experimental Settings
In this section, we provide comprehensive overview of the training and evaluation settings employed in our experiments and benchmarks. We begin by detailing the hyperparameter configurations used for each experiment. Following this, we describe the training and generation settings utilized to profile the training and inference efficiency of SketchTune.
I.1. Hyperparameter Selection
1.1.1. COMPARISON WITH PEFT METHODS
We followed experimental settings described in F'FT (Yang al., 2024b). We used AdamW (Loshchilov & Hutter, 2019) optimizer for all our experiments. The optimal hyperparamters chosen to produce the final results are provided in Table 8.
1.2. COMPARISON WITH COMPRESSIVE FINE-TUNING
For comparing with QLoRA and LoftQ. we followed the experimental settings described LoftQ (Li et al., 2023b). We used AdamW (Loshchilov Hutter, 2019) optimizer for all our experiments. The optimal hyperparameters chosen to
20

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Table 8. Hyperparameter selections for fine-tuning SketchTune on math reasoning and commonsense reasoning tasks.
Task
Model
LR
Optimizer
Batch Size
Epochs
LR Scheduler
Warmup Steps
Llama-77 7B
Math
Llama- 13B
8x10
AdamW
16
linear
Reasoning
Llama- 2-7B
Alama-3-88
3x10
AdamW
16
4
linear
100
Llama- 7B
Commonsense Reasoning
Ilama- 13B
8x
AdamW
64
2
linear
Alama-2-7B
Llama-3-8B
$
AdamW
64
2
linear
100
produce the final results are provided in Table
Table Hyperparameter selections for fine-tuning SketchTune WikiText-2 and GSM8K tasks.
Task
Model
LR
Optimizer
Batch Size
Epochs
LR Scheduler
Llama-2-7B
WikiTex1-2
3 10-5
AdamW
4
4
cosine
Llama-2-13B
Llama-2-7B
GSM8k
8x 10-s
AdamW
16
4
cosine
Llama-2-13B
1.1.3. INSTRUCTION TUNN-TUNING
For instruction fine-tuning tasks on Mistral-7B, we trained on the Alpaca- GPaa dataset for one epoch. We employed learning rate of 8 10-6, AdamW (Loshchilov & Hutter, 2019) for optimizer, linear LR scheduler, and batch size of 16 with 100 warmup steps.
1.2. Efficiency Evaluation Settings
Details on memory and speed efficiency evaluation settings are provided in Table 10.
Table 10. Efficiency evaluation settings for inference and training
Stage
Metric
Context Length
Batch Size
Warmup
4000
10
Time to First Token
8000
10
Inference
2000
10
Decoding Latency
2000
10
Peak Memory
10000
10
Training Latency
512
10
Training
Training Peak Memory
512
10
J. Comparison against Sparse Adapters
Table 11 presents additional accuracy results of fine-tuned Llama models on the commonsense reasoning tasks using SketchTune and sparsity based PEFT methods, including SpIEL (Ansell et al., 2024) and SMT (He et al., 2025). For SketchTune, we use the INT4 data representation and GPR=4 for model sketching, while baseline methods use the original weights. SketchTune is able to achieve better or comparable accuracy consistently across different tasks while using
21

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
2.71 3.54 smaller base models.
Table 11. Accuracy of Sketch compared to sprasity-based PEFT methods for fine-tuning Llama models on commonsense reasoning datasets. Baseline results taken from He et al. (2025). SketchTune achieve better comparable accuracy while using sketched models that are smaller than the full base models used by baseline methods.
Base Model
Trainable (aram(M)
Model
Method
BoolQ
PIQA
SIQA HellaSwag Wino ARC-e ARC-c OBQA
Avg.
(GB)
SpIEL
13.48
56.6
67.7
81.2
78.6
84
80.2
78.3
62.8
78.8
76.5
LLaMA-7B
SMT(Best)
13,48
330.9
72
82.9
80.7
93.3
82,4
86.
70.6
83
81.4
SketchTuneapes
4.02
87.0
72.1
85.6
80.2
93.7
84.6
86.2
71.0
84.8
82.3
SpIEL
26.03
45.8
73.2
84.3
81.4
91.2
84.1
83.1
68.8
82.8
81.1
LLaMA-13B
(MT(Best)
26.03
330.9
72.6
86.1
81.9
95
86.1
88.2
77.1
87.4
84.3
SkntchTuneopg-4
7.36
136.3
73.9
87.4
82.5
95.6
86.1
90.3
75.7
89.4
85.1
SplEL
13.48
55.9
70.5
80.6
80.8
85.8
83.4
81.2
65.8
81.8
78.3
LLAMA-2-78
SMT(Best)
13.48
330.9
72.6
85.2
82
85.7
87.8
74.5
85
83,4
Sketch
4.05
87.0
73.3
86.2
81.2
94.1
85.4
87.6
75.2
85.8
83.6
SplEL
16.06
47.2
72.1
83.6
80
91.8
85,4
91.2
76,8
80.8
82,7
LLaMA-3-8B
SMT(Best)
16.06
202.8
75.1
89.9
82.4
96.3
88.8
92.6
82.8
89.6
87.2
Sketch +4
5.92
88.1
75.0
90.2
82.7
95.9
88.2
92.6
82.1
89.4
87.0
K. LLM-as-a-judge Comparison with LoftQ
Table LLMM-as--judde evaluation between SketchTune LoftQ
Method
Data Type
Win
Loss
Tie
Win Rate
Loss Rate
Tie Rate
LoftQrank=64
NF4
93
147
60
0.31
0.49
0.39
SketchTuneapRes
INT4
147
93
60
0.49
0.31
0.61
In this section, evaluate the performance of SketchTune in comparison to LoftQ on language generation tasks using
the LLM-aas-ajudge framework (Zheng et al., 2023). Both methods were fine-tuned on the Llama-3-8B model (Dubey et al., 2024) using 4,096 randomly selected inputs from the Alpaca dataset (Taori et al., 2023). The fine-tuned models then generated responses on 300 distinct test inputs, which were also randomly sampled from the same dataset. GPT-4o was used as a judge to evaluate the models' generation quality. illustrated Table 12, SketchTune achieved win-loss ratio of 0.61 against LoftQ, demonstrating superior language generation capabilities.
L. Math Evaluations: Regular Expression Matching vs. LLM-Based Judging
For the experimental results in Table 1, we adopt the evaluation approach used in prior works (Hu et al., 2023; Yang et al., 2024b), where the last number an LLM''s response is extracted via regular expression matching and treated as the predicted answer. For instance, if an LLM outputs "Thus, Alice would need 570 tiles to cover 36 sqft area.", the method extracts as the answer. However, this extraction incorrect, as the intended answer is 570, but the regex- approach mistakenly identifies 36.
To address this issue, we propose more reliable evaluation method using LLM as the answer extractor. Specifically, use o3-mini to extract the final answer from each response. We observe that this LLM-based judging improves accuracy on math datasets by several percentage points compared to regular expression matching. Table 13 presents detailed comparison of results for LoRA, DoRA, and our proposed method, SketchTune.
22

---

Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Table 13. Accuracy score comparison between LLM-based judgement and regex-baser extraction.
Sketch- Tunegra-4
Sketch-
Dataset Eval Method
L2 LoRA,a1 LoRA,s LoRA,=16 DoRA,=2 DoRA,= DoRA, DoRA,st
Tuneera-s
Regex 03-mini
28.3
28.7
25.2
24.4
25.6
26.8
26.0
25.6
28.7
29.1
AQuA
39.8
34.6
39.0
38.6
35.0
35.4
38.2
37.4
37.0
39.0
Regex
66.9
66.3
69.2
68.8
66.3
67.2
68.8
69.4
68.2
68.8
GMSSK 03-mini
68.8
69.4
71.0
71.0
68.3
69.4
70.9
72.0
71.6
71.4
23


Reviews:
[{"review_id": "moIBJ666li", "reviewer": "Reviewer_WZr9", "summary": "The paper proposes an alternative to parameter efficient fine-tuning of LLMs by using sketching to create a low-dimensional representation of the weight matrices which is theoretically shown to be better for certain classes of matrices. Experiments on Llama models shows that the algorithm is able to outperform PEFT while using smaller base models and comparable trainable parameters.\n\n## update after rebuttal\n\nThe authors have addressed my concerns by adding experiments showing that the overhead of sketch generation is acceptable and clarifying that their choice of hyperparameters allows for significant compression despite grouping. Therefore, I have increased my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and for acknowledging the empirical effectiveness of our method. We address your concerns and questions below:\n\n## **[W1 - Sketch generation process appears to be more expensive than LoRA]**\n\nWhile SketchTune introduces an additional sketching step before fine-tuning, this preprocessing is **fast, resource-efficient, and one-time** per base model. In the table below, we report additional end-to-end sketching time (INT4, GPR=4) for different sized models, using a single A100-40GB GPU. Thanks to our layer-wise optimization objective (Equation 1), sketching scales efficiently to large models (**70B**) with a single GPU.\n\n| Model         | Original Size | Sketched Size | Max GPU Memory | Sketching Time |\n| ------------- | ------------- | ------------- | --------------- | --------------- |\n| Llama-3.2-3B  | 6.43G         | 3.18G         | 9.92 GB         | 20.7 mins       |\n| Llama-3.1-8B  | 16.07G        | 5.92G         | 18.37 GB        | 41.62 mins      |\n| Llama-3.1-70B | 141.12G       | 40.15G        | 28.05 GB        | 266.87 mins     |\n\nThe sketching overhead is comparable to existing compressed PEFT baselines. For example, LoftQ [1] reports a quantization overhead of **21 seconds** for a 4096 $\\times$ 4096 matrix, while SketchTune’s overhead is only **5 seconds (4.2x speedup)**.\n\nMoreover, a single sketched model can be reused across multiple downstream tasks. As demonstrated in Tables 1 and 2 of the paper, the same sketched models are effective across different domains, i.e. math and commonsense reasoning. We also plan to release these models on the HuggingFace Model Hub, enabling users to directly download and fine-tune without repeating the sketching step.\n\n\n## **[W2 - Learning separate sketches for each of the $g$ subgroups may reduce memory efficiency]**\n\nOur sketching approach is memory-efficient and effectively compresses the model weights. Specifically, each row $\\mathbf{w} \\in \\mathbb{R}^{1 \\times c}$ is divided into $g$ non-overlapping sub-rows $\\mathbf{w}' \\in \\mathbb{R}^{1 \\times \\frac{c}{g}}$, each of which is sketched into a $k$-dimensional vector. The resulting sketched row $\\mathbf{w}_{\\text{sketched}} \\in \\mathbb{R}^{1 \\times gk}$ leads to a compression factor of $\\frac{c}{gk}$.\n\nFor Llama 2 7B, the row size $c$ is either 4096 or 11008. In our experiments, we set $k \\in \\\\{4, 8, 16\\\\}$ and $g \\in \\\\{1, 2, 4, 8\\\\}$, resulting in a compression factor of at least $32\\times$ for Llama 2 7B. Notably, the compression factor increases with larger models, due to larger $c$ and fixed $k$ and $g$.\n\nThus, even with subgrouping, the sketching approach remains highly memory-efficient across model scales.\n\n\n## **[Q1 - The derivation of (6) is not clear to me. Please explain how it is derived.]**\n\nThank you for pointing this out. We realize that Equation (6) contains a typo: the derivative fractions are upside down. We apologize for the oversight and will correct this in the camera-ready version. The correct expression is:\n\n$\\frac{\\partial \\mathcal{L}}{\\partial w_{\\text{sketched}}} = \\frac{\\partial \\mathcal{L}}{\\partial y} (MX)^\\top$\n\nThe equation is derived as follows. Let $X$ be the layer input, $w_{\\text{sketched}}$ the sketched row, and $M$ the mapping matrix. The forward pass computes:\n\n$y = w_{\\text{sketched}}MX$\n\nDuring backpropagation, we apply the chain rule to compute the gradient of the loss $\\mathcal{L}$ with respect to $w_{\\text{sketched}}$:\n\n$\\frac{\\partial \\mathcal{L}}{\\partial w_{\\text{sketched}}} = \\frac{\\partial \\mathcal{L}}{\\partial y} \\cdot \\frac{\\partial y}{\\partial w_{\\text{sketched}}} = \\frac{\\partial \\mathcal{L}}{\\partial y} (MX)^\\top$\n\nWe will clarify this derivation in the revised paper.\n\n**References**\n\n[1] Li, Yixiao, et al. \"LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I checked the proofs at a high-level and they appear correct to me.", "experimental_designs_or_analyses": "I checked the experiments in the main paper, and they appear correct to me.", "relation_to_broader_scientific_literature": "Strengths:\n\n1. Sketching significantly reduces the model size while preserving the pre-trained capabilities of the model.\n\n2. Experiments show that it achieves higher accuracy than PEFT baselines with comparable number of trainable parameters.\n\nWeaknesses:\n\n1. The process of generating the sketches appears to be more expensive than LoRA\n\n2. It seems like learning separate c-dimensional sketches for each of the 'g' subgroups in a k-dimensional row can reduce the memory efficiency since we need kg << c to get significant memory saving and that might be difficult to satisfy", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "See questions below", "questions_for_authors": "The derivation of (6) is not clear to me. Please explain how it is derived.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "B7irnwIRUj", "reviewer": "Reviewer_v96N", "summary": "The paper proposes SketchTune, which uses a learned sketching algorithm to compress the LLM into a small set of shared sketched parameters and fine-tune those parameters for adaptation. The proposed approach reduces model size while preserving the pre-trained capabilities of the full model.\n\n## update after rebuttal\nI appreciate the authors’ efforts in providing a response, and most of my concerns have been addressed. Accordingly, I will increase my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and insightful feedback. We address your concerns as follows.\n\n## **[Theoretical Claims 1 - Ambiguous Title in Section 2.1 & More in-depth Empirical Observation Analysis]**\n\nWe thank the reviewer for the insightful suggestion. We conducted additional analysis to examine whether the weight updates in LLMs are close to full rank. In the table below, we report **the minimum rank and the standard deviation**, across all layers, required to explain a given percentage of the variance in the weight updates. This was computed by performing SVD on each weight update and counting how many top singular values are needed to reach the target variance threshold.\n\nThe results show that the weight updates are relatively **high-rank**. This suggests that standard LoRA configurations (e.g., rank 32) capture only a limited portion of the variance, and may not effectively approximate the full weight updates. We will incorporate this analysis into the final version of the paper to better align with the section title.\n\n| Model                                | Max Rank      | 25% Variance        | 50% Variance         | 75% Variance         | 90% Variance         | 95% Variance         |\n|--------------------------------------|---------------|----------------------|------------------------|------------------------|------------------------|------------------------|\n| Llama 2 7B, fine-tuned on Vicuna     | 4096          | 140.4 ± 62.7         | 460.9 ± 178.2          | 1089.0 ± 358.5         | 1866.4 ± 520.5         | 2354.0 ± 581.4         |\n| Llama 3 8B, fine-tuned on OpenChat   | 1024 or 4096  | 197.5 ± 111.8        | 562.5 ± 304.7          | 1180.5 ± 607.7         | 1848.8 ± 903.9         | 2221.8 ± 1051.3        |\n| Qwen 2.5 7B, fine-tuned on Code      | 512 or 3584   | 235.8 ± 164.1        | 594.9 ± 385.6          | 1138.2 ± 691.5         | 1675.3 ± 960.5         | 1958.1 ± 1085.4        |\n\n## **[Experimental Designs 1 – Additional Instruction-Following Benchmark]**\nWe conducted additional experiments by instruction-tuning Mistral-7B on the Alpaca-GPT4 dataset for one epoch, following the experimental setup in S2FT. In the table below, we report MT-Bench scores evaluated by GPT-4, with baseline results taken from the S2FT paper. Despite using a smaller base model, SketchTune outperforms the baselines.\n\n| Model | Method | Base Model (GB) | Writing | Roleplay | Reasoning | Code | Math | Extraction | STEM | Humanities | Avg. |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| Mistral-7B | Full FT | 14.48 | 5.50 | 4.45 | 5.45 | 2.50 | 3.25 | 5.78 | 4.75 | 5.45 | 4.64 |\n|  | $\\text{S}^2\\text{FT}$ | 14.48 | 6.95 | 4.40 | 5.50 | 2.70 | 3.55 | 5.95 | 6.35 | 6.75 | 5.27 |\n|  | $\\text{SketchTune}_{\\text{GPR}=4}$ | 4.66 | 4.60 | 5.20 | 9.23 | 3.05 | 4.80 | 7.45 | 8.13 | 8.45 | **6.36** |\n\n##  **[Experimental Designs 2 – GPR Choice in Tables 1 & 2]**\nWe reported results for GPR=4 in Table 2 due to the time-consuming nature of fine-tuning on the Commonsense170K dataset, which is 17× larger than the Math10K dataset used in Table 1. A single training run takes several days to complete. We selected GPR=4 as a practical trade-off between memory efficiency and final performance. We are happy to conduct additional experiments with other GPR values and include the results in the camera-ready version.\n\n## **[Comments 1 – Overhead of Sketching Time and Memory Requirement]**\n\nIn Table 9 of the paper, the reported sketching time refers to the time required to compress a base model into a sketched model using a single RTX 8000-48GB GPU. In the table below, we provide additional sketching time (INT4, GPR=4) and peak GPU memory usage for models of various sizes, using a single A100-40GB GPU. Thanks to the layer-wise optimization objective (Equation 1), model sketching scales efficiently to large models (e.g., 70B) on a single GPU. Additionally, we plan to upload sketched models to the HuggingFace Model Hub, enabling users to directly download and fine-tune without repeating the sketching step.\n\n| Model         | Original Size | Sketched Size | Max GPU Memory | Sketching Time |\n| ------------- | ------------- | ------------- | --------------- | --------------- |\n| Llama-3.2-3B  | 6.43G         | 3.18G         | 9.92 GB         | 20.7 mins       |\n| Llama-3.1-8B  | 16.07G        | 5.92G         | 18.37 GB        | 41.62 mins      |\n| Llama-3.1-70B | 141.12G       | 40.15G        | 28.05 GB        | 266.87 mins     |", "claims_and_evidence": "Yes, the paper provides extensive experiments and theoretical analysis to support its claims.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are reasonable.", "theoretical_claims": "I find Section 2.1 to be slightly misleading, as its title, \"Motivation: Weight Updates Are Far from Low-Rank,\" suggests that the section will primarily discuss the fundamental characteristics of weight updates and why they are far from low-rank. However, the text mainly focuses on empirically comparing the effectiveness of low-rank matrices and sketching techniques in terms of approximation error, which seems more like an outcome rather than a motivation. Since the authors prove that SketchTune is well-suited to approximate delta when eta is close to 0 and delta is nearly full-rank in Section 3, providing a more in-depth analysis of empirical observations on whether weight updates of LLMs are actually close to full-rank would better support the authors' theoretical claims. This would also clearly establish the motivation for why weight updates are far from low-rank.", "experimental_designs_or_analyses": "In LLM fine-tuning, instruction-following benchmark results are typically included (e.g., the Alpaca GPT-4 dataset with MT-Bench scores; refer to the S2FT paper as an example). Given the significant potential of the proposed method as an efficient LLM compression and fine-tuning technique, I am curious whether it performs well on instruction-following benchmarks. Additionally, why do the authors present GPR=4 only in Table 2, while Table 1 includes GPR=1, 2, 4, and 8?", "relation_to_broader_scientific_literature": "This paper introduces a new research direction that overcomes the limitations of existing PEFT methods based on low-rank or sparse assumptions.", "essential_references_not_discussed": "I think the authors included the necessary references to support their claims.", "other_strengths_and_weaknesses": "The authors develop a custom CUDA kernel optimized for the specific operations required by SketchTune.", "other_comments_or_suggestions": "What is the overhead of sketching time in Table 9 of the Appendix? It would be helpful to know how much memory and computation are required for learning to sketch LLM weights (described in Algorithm 1).", "questions_for_authors": "Please refer to my questions for authors in the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xlUtmP0baW", "reviewer": "Reviewer_5Qcf", "summary": "The paper introduces  a method that compresses pre-trained LLM weights row-by-row into a smaller, shared set of trainable “sketched” weights. They compress weights by approximately minimizing the reconstruction error for activations over a set of data. Experimentally, SketchTune shows advantages in terms of model size and efficient inference with minimal performance trade-offs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and thoughtful feedback. Below, we address your questions and concerns.\n\n## **[Theoretical Claims]**\n\nThe effectiveness of LoRA and SketchTune depends on the structure of the true update matrix $\\Delta$. If $\\Delta$ is low-rank, LoRA is favored; if $\\Delta$ aligns with the mapping matrix, SketchTune performs better.\n\nTheorem 3.1 addresses how to choose between methods without detailed knowledge of $\\Delta$. Assuming only the singular value distribution, the most neutral case is when $\\Delta$ is uncorrelated with the mapping. Under this assumption, Theorem 3.1 shows that as $\\Delta$ becomes less low-rank, SketchTune is increasingly likely to outperform LoRA.\n\nWe adopt the uncorrelated setting in Theorem 3.1 to avoid bias toward either method. Empirically, however, our analysis (Figure 1) shows that $\\Delta$ is correlated with the mapping in a way that favors SketchTune.\n\n## **[Experimental Designs 1 - More Data-Dependent Quantization + LoRA Baselines]**\n\nWe performed additional comparison with data-dependent AQLM [4] quantization + LoRA baselines. We fine-tuned LoRA adapters with a 2-bit AQLM Llama-2-7B model on GSM8K using a learning rate of 1e-5 and a batch size of 16 for 5 epochs. We report the results in the table below. We are actively trying other hyperparameters to improve performance.\n\n|Model|Method|Trainable Param (M)|GSM8K|\n|-|-|-|-|\n|Llama-2-7B|$\\text{AQLM}_{\\text{2-bit}}$|159.91|2.81|\n| |$\\text{SketchTune}_{GPR=4}$|21.75|**29.95**|\n\n## **[W1 – Additional Data for Calibration & Sketching Overhead]**\n\nThe sketching step is efficient, requires only a small calibration dataset, and is performed once per base model. Specifically, we use just 128 sequences of 2048 tokens, which is minimal compared to typical fine-tuning datasets. For further details and results, we kindly refer you to our response to Reviewer WZr9 under **[W1]**.\n\n## **[W2 - Sparse adapters are not thoroughly compared]**\n\nTables 1 and 2 in our paper include comparisons with a sparsity-based method, S2FT. Additionally, while we aimed to compare with Diff-Pruning[1], we encountered a challenge: the official implementation does not support LLMs such as Llama. In the table below, we instead provide additional comparison with SpIEL[2] and SMT[3], two recent sparsity-based PEFT methods, reporting average accuracy across eight commonsense reasoning tasks. \n\n|Model|Method|Base Model (GB)|Trainable Param (M)|Avg Acc.|\n|-|-|-|-|-|\n| Llama-2-7B|SpIEL|13.48|55.9|78.3|\n| |SMT(Best)|13.48|330.9|83.4|\n| |$\\text{SketchTune}_{GPR=4}$|4.05|87.0|**83.6**|\n| Llama-3-8B|SpIEL|16.06|47.2|82.7|\n| |SMT(Best)|16.06|202.8|**87.2**|\n| |$\\text{SketchTune}_{GPR=4}$|5.92|88.1|87.0|\n\n## **[Q1 - Time/Compute Overhead]**\n\nWe ran additional experiments to assess the impact of using smaller C4 subsets for calibration, evaluating model quality via WikiText2 perplexity and MMLU accuracy.\n\nAs shown in the table below, larger calibration sets improve model quality but **do not significantly increase sketching time**, since the dominant cost in sketching comes from clustering to obtain the $k$ centers (Equation 4).\n\nThese results highlight a trade-off: even a single sequence yields a usable sketch, but larger sets lead to better performance.\n\n|Model|# of Sequences|Sketch Time (min)|WikiText2 Perplexity|MMLU Accuracy|\n|-|-|-|-|-|\n|Llama-3-8B|128|39.88|6.52|60.56|\n| |32|36.62|6.59|60.36|\n| |8|32.88|6.65|60.40|\n| |1|31.70|7.00|58.58|\n\n## **[Q2 - Sparse Adapters]**\n\nYes, SketchTune can be combined with Diff-Pruning for greater parameter efficiency. This involves keeping the sketched parameters frozen and learning a sparse, task-specific update $\\delta_\\tau$, so that $w_{sketched, \\tau} = w_{sketched} + \\delta_\\tau$.\n\nFollowing Diff-Pruning, $\\delta_\\tau$ can be modeled as a gated update using a Hard-Concrete distribution to approximate a binary mask $z_\\tau$, with $\\delta_\\tau = z_\\tau \\odot w_\\tau$, where $w_\\tau$ is the dense task-specific update.\n\nImplementing this extension is non-trivial, and we defer it to future work.\n\n## **[Q3 - Dynamic Remapping]**\n\nSince each sketched parameter is shared by multiple model parameters, re-mapping columns or recalculating the Hessian mid-training may help better align the sketch with the evolving gradients, potentially leading to improved final performance. Yet such an approach may introduce non-trivial computational overhead, as it would require additional clustering steps and second-order estimation during training. We defer a thorough investigation of dynamic remapping strategies to future work.\n\n**References**\n1. Parameter-efficient transfer learning with diff pruning\n2. Scaling sparse fine-tuning to large language models\n3. Sparse Matrix in Large Language Model Fine-tuning\n4. Extreme compression of large language models via additive quantization", "claims_and_evidence": "Main Idea / Theoretical Analysis: SketchTune replaces a low-rank update assumption with a sketch-based approximation, where theoretical arguments (Section 3) show that if weight updates are not actually low-rank, a sketch can give a better approximation. Under some assumptions, the paper characterizes conditions (power-law decay of singular values) under which sketches outperform low-rank approximations.\nEmpirical Results: On math (e.g., GSM8K) and commonsense tasks, SketchTune achieves performance on par with or surpassing strong PEFT baselines (LoRA, DoRA, etc.), even though it uses a smaller “sketched” base model. The method also compares favorably to quantized PEFT approaches like LoftQ.\n\nHowever, one notable methodological detail is that SketchTune uses additional C4 data to compute the Hessian-based weighting for the sketch, effectively a calibration pass, whereas competing methods do not use additional data or have a previous data dependent compression step. Also, Table 9 in the appendix shows this “sketching” step can be a significant overhead relative to normal training times, as it involves an extra pass over substantial data and computes second-order derivatives.", "methods_and_evaluation_criteria": "The paper evaluates on standard math, commonsense, and language modeling tasks, reporting the usual metrics. The authors also benchmark training and inference time (time to first token, decoding latency), along with memory usage. These criteria align well with the stated aim of PEFT.", "theoretical_claims": "I didn't check it thoroughly, but it seems to be correct. I am not sure how mild is the assumption that the random mapping is uncorrelated with the true update matrix.", "experimental_designs_or_analyses": "The experiments are generally strong, spanning multiple tasks and baselines (low-rank, quantized). Performance improvements hold across tasks. The main concern is the use of additional data and overhead of sketch computation (Appendix I), which can be quite large compared to normal finetuning time. Specifically, Table 9 suggests that multi-hour overhead might be incurred on some GPUs. This overhead may offset the claimed efficiency advantages in certain settings. I would also like to see more quantization (or actually quantization+LoRA) baselines, especially data dependent quantization methods.", "relation_to_broader_scientific_literature": "SketchTune builds on prior compressive adaptation ideas (random hashing, count-sketch) and extends them with a layerwise, Hessian-based weighting. It competes directly with low-rank (LoRA, DoRA) and quantization-based (LoftQ, QLoRA) methods but has limited discussion or comparison with pure sparsity-based finetuning.", "essential_references_not_discussed": "The coverage of low-rank, quantization, and other compressive approaches is fairly thorough. However, methods focusing purely on model sparsity as an alternative – e.g. forcing adapter updates to remain in a sparse subset – only receive brief mention. A more direct empirical comparison would strengthen the position of SketchTune among all “competing” approaches.", "other_strengths_and_weaknesses": "Strengths:\nPrincipled approach that is complemetary to low-rank updates.\nStrong empirical results vs. both low-rank adapters and quantized finetuning.\nDedicated GPU kernel designs.\n\nWeaknesses:\nRequires a calibration pass on data to compute Hessians and sketch the pre-trained model, which can add substantial overhead.\nSparse adapters are not thoroughly compared.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Time/Compute Overhead: Have you explored sampling a smaller subset of C4 or using fewer Hessian blocks to reduce overhead, and what are the trade-offs in final accuracy?\nSparse Adapters: Could row-wise sketches be combined or compared directly with a learned sparse subset of weights (like Diff-Pruning)?\nDynamic Remapping: Would re-mapping columns or recalculating the Hessian mid-training yield significantly better final performance, or do diminishing returns make this impractical?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vuk4dG3SFT", "reviewer": "Reviewer_kWKb", "summary": "This paper is concerned with parameter-efficient fine-tuning. It is argued in this paper that previous solutions in parameter-efficient fine-tuning are either low-rank or quantized. They are limited in applicability due to restrictive assumptions. In this paper, a sketchtune approach is proposed with the inspiration of sketching. By rigorously experimenting with LLaMA-1/2/3 models on math problem-solving, commonsense reasoning, and instruction-following tasks, it is shown that the proposed sketchtune is way better than considered baselines ranging from LoRA, DoRA, to S2FT in terms of both model performance and parameter efficiency. Specifically, sketchtune can achieve the same performance with baselines with smaller base models and fewer trainable parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the soundness of our work and providing thoughtful feedback. We address the reviewer’s concerns and suggestions below:\n\n## **[Concern 1 - The sketch strategy is somehow hard to understand. Specifically, in Section 2.3, it is not very clear how the mapping matrix, i.e., M, is determined.]**\n\nThank you for pointing this out. We provide further clarification below on how the mapping matrix $M$ is learned.\n\nEach column of $M$ is a binary one-hot vector that maps an original parameter (from a row of size $c$) to one of the $k$ entries in the sketched parameter vector, where $k \\ll c$. This mapping inevitably introduces some error in the model output. To minimize this error, we learn the columns of $M$ sequentially and iteratively update the remaining unmapped parameters to compensate for the introduced error after each step.\n\nConcretely, for each original parameter, we identify the entry in the sketched parameter vector that is closest to it and assign the corresponding one-hot value in $M$. After fixing this column of $M$, we apply an update $\\boldsymbol\\delta$ (as defined in Equation 13) to the remaining unmapped parameters to absorb the approximation error. This process is repeated until all columns of $M$ are assigned.\n\nWe will clarify this procedure in the final version of the paper.\n\n## **[W1 - Misleading Equation 6]**\n\nWe thank the reviewer for pointing out this typo. We will correct this in the final paper.\n\n## **[W2 - Release CUDA kernel as supplementary material]**\n\nThank you for the great suggestion. We will include the CUDA kernel in the appendix and release our code and models for the final version.", "claims_and_evidence": "The claims in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods are mostly elaborated and the evaluation criteria is well justified. However, I still have several concerns:\n1) The sketch strategy is somehow hard to understand, specifically, in section 2.3, it is not very clear how the mapping matrix, i.e. M, is determined.", "theoretical_claims": "The proofs for theoretical claims are correctly provided.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and adequate.", "relation_to_broader_scientific_literature": "The contributions are built upon previous studies in parameter-efficient fine-tuning methods, e.g. LoRA and QLoRA. LoRA and QLoRA-like methods are limited in that they are either low-rank or quantized. So sketchtune proposes optimizations toward the mentioned limitations.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n1) Equation 6 is mis-leading, the formulation of derivative seems upside-down.\n2) The dedicated CUDA kernel would better be attached as supplementary material.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SketchTune, a novel parameter-efficient fine-tuning (PEFT) technique based on learning sketches of model weights. This approach provides an alternative to prevalent low-rank adaptation (LoRA) and quantization methods. The reviewers generally found the method well-motivated and the empirical results strong, demonstrating superior performance compared to several established PEFT baselines across various tasks (math, commonsense reasoning, instruction-following) while often using smaller base models or fewer trainable parameters. The authors successfully addressed most of the reviewers' concerns during the rebuttal phase, leading two reviewers (WZr9, v96N) to raise their scores.

**Consolidated Strong Points:**

* **Novelty and Strong Empirical Performance (All Reviewers):** SketchTune offers a principled alternative to existing PEFT methods, backed by strong empirical results showing consistent improvements over LoRA, DoRA, S2FT, LoftQ, and added sparsity baselines (SpIEL, SMT) on diverse benchmarks.
* **Parameter and Computational Efficiency (WZr9, v96N, 5Qcf):** The method achieves significant model compression and demonstrates efficiency benefits, supported by a custom CUDA kernel. It often matches baseline performance with smaller underlying models.
* **Theoretical Motivation (WZr9, v96N, 5Qcf):** The paper provides theoretical grounding, arguing SketchTune is particularly advantageous when weight updates deviate from low-rank structure, a claim supported by additional empirical analysis provided in the rebuttal.
* **Responsiveness to Feedback (All Reviewers):** Authors provided detailed clarifications, corrected minor issues, added requested experiments (instruction-following, new baselines), and offered further analysis during rebuttal.

**Consolidated Concerns (and how they were addressed):**

* **Initial Sketching Overhead (WZr9, v96N, 5Qcf):** Concerns about the one-time cost (time, compute, calibration data) of the sketching step. Addressed by authors providing benchmarks demonstrating manageable cost, clarifying minimal data requirements, showing cost/performance trade-offs, and promising pre-sketched model release. *Reviewers WZr9, v96N, 5Qcf accepted the response.*
* **Clarity and Minor Errors (WZr9, v96N, 5Qcf, kWKb):** Initial lack of clarity regarding Eq. 6 derivation, Section 2.1 motivation, mapping matrix M determination, and a theoretical assumption. Addressed via corrections, additional empirical analysis, detailed explanations, and justifications in the rebuttal. *All relevant reviewers acknowledged the clarifications.*
* **Initial Experimental Gaps (v96N, 5Qcf):** Lack of instruction-following results and limited comparison to certain quantization/sparsity methods. Addressed by adding MT-Bench results and comparisons to AQLM+LoRA, SpIEL, SMT. *Reviewers v96N, 5Qcf accepted the additions.*
* **Code/Kernel Availability (kWKb):** Request for the custom CUDA kernel. Addressed by authors promising code, model, and kernel release. *Reviewer kWKb acknowledged.*

Overall, the paper presents a compelling new PEFT method with strong results. While the initial sketching step adds a layer of complexity compared to methods like LoRA, the authors effectively argued its feasibility and benefits. The concerns raised were substantially addressed.
Decision: Accept (poster)