PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection
Authors: Junseo Hwang
Wonguk Cho
Taesup Kim
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Fine Tuning
Deep Learning
Primary area: generative models

Abstract: Fine-tuning large foundation models is essential for building expert models tailored to specialized tasks and domains, but fully updating billions of parameters is computationally prohibitive. Reducing the number of trainable parameters using Parameter-Efficient Fine-Tuning (PEFT), such as Low-Rank Adaptation (LoRA), is therefore crucial not only to reduce training costs but also to mitigate storage, caching, and serving overheads during deployment. Prior works, such as Singular Vectors-guided Fine-Tuning (SVFT), have shown that exploiting the geometry of pre-trained weights based on Singular Value Decomposition (SVD) can significantly improve parameter-efficiency, but they lack a solid theoretical foundation. In this paper, we introduce Parameter-Efficient Fine-Tuning with Column Space Projection (PiCa), a novel theoretically grounded PEFT method. We prove that projecting gradients onto the principal column space of pre-trained weights provides an effective inductive bias for adaptation and further enhance parameter efficiency through a novel weight-sharing strategy. Across diverse NLP and vision tasks, PiCa consistently outperforms state-of-the-art baselines under comparable or smaller parameter budgets, demonstrating both theoretical rigor and practical effectiveness.

Published as a conference paper at ICLR 2026
PICA: PARAMETER-EFFICIENT FINE-TUNING WITH COLUMN SPACE PROJECTION
FUNNG
Junseo Hwang Wonguk Cho Taesup Kim
Graduate School of Data Science, Scoul National University
ABSTRACT
Fine-tuning large foundation models is essential for building expert models tai- lored to specialized tasks and domains, but fully updating billions of parameters is computationally prohibitive. Reducing the number of trainable parameters using Carameter-Efficien Fine-Tuning (PEFT), such as Low- -awnkank&aa Adaptation (LoRA), is therefore erucial not only reduce training costs but also to mitigate storage, caching, and serving overheads during deployment. Prior works, such as Singular rectors-guided Fine-Tuning (SVFT), have shown that exploiting the geometry of pre-trained weights based on Singular Value Decomposition (SVD) can signifi- cantly improve parameter-efficienny, but they lack a solid theoretical foundation. In this paper, we introduce Parameter-Effice Projection (PiCa), a novel theoretically grounded PEFT method. We prove that pro- jecting gradients onto the principal column space of pre-trained weights provides effective inductive bias for adaptation and further enhance parameter efficiency through novel weight-sharing strategy. Across diverse NLP and vision tasks, PiCa consistently outperforms state-of-the-art baselines under comparable or smaller parameter budgets, demonstrating both theoretical rigor and practical effectiveness.
Fine- Fine-Tuning Tuning with Column Space
INTRODUCTION
Fine-tuning large foundation models is essential for building expert models tailored to special- ized tasks and domains. However, fully fine-tuning billions of parameters is often computation- ally prohibitive in terms of both training and deployment cost. Parameter-Efficiiet Fine- Fine-Tuning (PEFT) (Houlsby al. 2019) addresses this challenge by adapting models with only small number of trainable parameters while keeping the pre-trained backbone frozen. In particular, minimizing the number of trainable parameters is critical in practical scenarios where multiple adapters must be deployed simultaneously (Chen et al.) 2024). In such cases, numerous sets of fine-tuned parameters for different tasks, models, and checkpoints per user must be stored separately from the pre-traue pre-trained models, leading to significant storage, caching, and serving overheads.
prominent line of research Low-Rank Adaptation (LoRA) (Hu al. 2022p. known for its simplicity and strong empirical performance. While reducing its rank lowers the number of trainable parameters, inevitably causes significant performance degradation. To address this, DoRA (Liu) et al., 2024a) introduces weight decomposition into LoRA, achieving stronger performance at fixed rank and often matching or surpassing LoRA while requiring only half the trainable parameters. VeRA (Kopiczko et al.) 2023) further reduces parameter budgets by training small scaling vectors, demonstrating that comparable or superior performance to LoRA can obtained with up to 4x fewer trainable parameters. However, these LoRA-based methods typically rely randomly initialized low-rank matrices and thus do not explicitly leverage the geometry or prior knowledge encoded in the pre-trained weights.
Furthermore, recent studies (Lingam et al. 2024 Han et al. 2023) Mantri et al.) 2025) have shown that leveraging the geometry of pre-trained weights, particularly their spectral structure, can lead to further parameter-efficiency without performance degradation. For instance, Singular Vectors-guided Fine-Tuning (SVFT) (Lingam al. 2024) constructs sparse, weighted combination of model's
"quual contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
pre-trained singular vectors to achieve strong performance with fewer trainable parameters. However, despite their empirical success, these SVD-based approaches (Lingam et al.) 2024; Han et al. [023]] Mantri et al. 2025) lack theoretical foundation for their approaches and leave open why using the spectral structure of pre-trained weights constitutes an effective inductive bias for fine-tuning.
In this work, Efficient Fine- Tuning with Column Space Projection (PiCa), a new theoretically grounded PEFT method that leverages the geometry pre-traainee weights. Our theo-
propose Parameter-
70
Full FT (2500M)
Accuracy (%)
retical analysis demonstrates that project- ing gradients onto the principal column space spanned by pre-trained weights can lead to effective adaptation. This gradient projection is effectively paired with our novel weight-sharing method for further parameter efficiency. With this approach, we can significantly reduce the number of trainable parameters, even using less than the most parameter-effiiient config- urations of other methods (e.g.. rank-1 LoRA and DoRA), while achieving sig- nificantly better performance. Our exten- sive experiments across various models and datasets demonstrate that PiCa consis- tently outperforms all baseline methods under comparable parameter budgets, as shown in Fig
PiCa. PiCa,-25
LoRA, -1 LORA,
DoRA,
DoRA
SVFT" SVFT
50
VeRA-
BOFT
10
15
20
25
# Params (M)
Figure 1: Average accuracy function of the num- ber of trainable parameters on Commonsense Reasoning datasets using Gemma-28 -B. PiCa demonstrates superior performance compared to baseline methods with similar parameter budgets.
Our contributions can be summarized as follows:
We introduce PiCa, theoretically grounded PEFT method that explicitly exploits the geometry of pre-trained weights. We provide a theoretical foundation showing that projecting gradients onto the principal column space of pre-trained weights enables effective adaptation. For further parameter efficiency, PiCa also introduces a novel weight-sharing approach that can be paired with gradient projection.
PiCa consistently achieves competitive or superior performance with significantly fewer pa- rameters compared to other baselines. In particular, it outperforms state-of-the-ate baselines, SVFT and SVFT across all datasets and models under smaller parameter budgets.
Our experiments span wide range of NLP tasks including mathematical reasoning, common- sense reasoning, and natural language understanding with different language models, as well as diverse vision tasks such as visual adaptation on VTAB datasets with vision transformers and -ubject-driven generation DreamBooth with text-to-image diffusion models. We also conduct comprehensive ablation studies to better understand the individual components of our method and their effects.
RELATED WORK
Carameter-Efficien Fine-Tuning In adapting large foundation models for downstream tasks, while full fine-tuning often yields superior performance on these tasks, its prohibitive computational overheads have motivated the development of various PEFT methods that aim to achieve comparable performance with much smaller number of trainable parameters. Recently highlighted approaches include low rank approximation (Hu et al.] 2022 Liu et al. 2024a] Kopiezko al.) 2023), orthogonal reparametrization (Qiu al. 2023 Liu et al. 2024b), and Singular Value Decomposition (SVD)- based approaches (Lingam et al. [2024] Han et al. 2023 Mantri ct al. 2025).
In particular, LoRA and its variants (Hu al. 2022 Liu et al. 2024a: Kopiczko et al.) 2023 have significant attention due to its simplicity and efficiency, based low-rank decomposition. DoRA (Lu eet al.] 2024a) decomposes weights and achieves stronger performance at fixed rank, often matching or surpassing LoRA while requiring only half the trainable parameters. VeRA (Kopiczko et al. 2023 further reduces parameter budgets by training small scaling vectors.
2

---

Published as a conference paper at ICLR 2026
On the other hand, methods leveraging the structure of pre- traitrained weights, specifically through their SVD components, have been explored (Lingam et al. [024 Han et al.) [023] Mantri et al.) 2025 Meng et al., 2024). SVFT (Lingam al. 2024) utilizes the entire singular vectors of pre- trained weights as a basis and employs a sparse matrix for updates. SVDiff (Han et al. (023) has demonstrated fine-tuning only the singular values of pre-trained weight matrices is effective in
personalization of text-to-image diffusion models, Similarly, DiTASK (Mantri et al. 2025) has shown that preserving singular vectors and enabling task-specific adaptations through neural diffeomorphic transformations of the singular values can be effective for dense prediction tasks.
Although these SVD-based methods have shown empirical success, they often lack strong theoretical foundation that provides an analytical justification for their methods, and only few works has attempted to analyze the change in spectral structure after fine-tuning (Shuttleworth et al. 2024). In contrast, we develop a method based on theoretical proof that the optimal rank-r approximation of AW can be achieved by the singular vectors of the pre-trained weights, which aligns with our empirical findings. The effectiveness of this approach is validated through extensive experiments.
Weight-sharing Prior research has explored weight-sharing reduce the number of parameters in neural networks (Press & Wolf 2017]] Inan et al. 2016). More recently, this concept of weight-sharing
has been adapted within the LoRA framework (Kopickko et al. 2023 Renduchintala et al. 2023
Zhou et al., 2025 Shen et al. 2024, Song et al. 2024). For instance, VeRA (Kopiczko et al. 2023
introduces a frozen random projection matrix shared across all layers, combined with trainable scaling vectors. Furthermore, recent works (Renduchintala et al. 2023 Song et al.] 2024) explore different strategies of combining freezing, training, and sharing both projection matrices and scaling vectors. While demonstrating progress in parameter reduction, these prior approaches tend to be highly sensitive to randomly initialized projection matrices and often their performance is below that of standard LoRA. However, in PiCa, construct projection matrix based on structure of pre-trained weights for each layer and share trainable weights across layers with the same function role. This approach allows significant reduction of trainable parameters without performance degradation.
3 METHODOLOGGY
In this section, we introduce our novel PEFT method, PiCa. (1) We first discuss how fine-tuning relates to singular vectors and introduces Theorem [ which shows that the principal subspace of pre-trained weights offers an effective space for adaptation (Section 3.T). (2) We develop this idea in the context of PEFT settings, showing that sequentially projecting gradients onto this subspace offers a theoretically grounded way to perform fine-tuning under parameter constraints through Theorem (Section 3.2 (3) On top of these insights, we finally present our algorithm, PiCa, which integrates sequential projection with weight-sharing for further parameter-efficient adaptation (Section 3.3).
3.1 FINE-TUNING AND COLUMN SPACE PROJECTION
"Fine" "iine"turn -tunngg is, by definition, the process of making relatively small update from the pre- trained weights Wo the fine-tuned weights
Telquency
W', in order to adapt a model to a specific downstream task with much smaller dataset. As large foundation models are pre-trained on vast, diverse corpora, good optima tend to lie in small-update neighborhood of Wo. There- fore, in the context of munit of large foundation models, it is natural to assume that AW = Wo with ||Woll > ||AW]] Lemma 3.1 indicates that, when this change small, the leading singular structures W, and W remain closely aligned.
-09250-330023
Eg
Figure 2:
Distribution of Ef and E across all weight matrix elements using DeBERTa V3 Most values are tightly concentrated around zero, validat- ing that O(c) is negligible in practice.
3

---

Published as a conference paper at ICLR 2026
Lemma 3.1 Wedin (972)). Let Wo, W E R with W Wo + AW. Let Ur, U2 denote the top-r left singular-vector matrices of Wo and W Define the gap
= min or (Wo) - o C++1(W") o,(W)) - or+1(Wo))
Then for any unitarily invariant norm,
8 |
sin OU,,, <
Building this insight, Theorem expresses Txxprssseexpresses the relation between Wo and W in form that involves a small deviation E, and uses this to analyze how the update AW can be captured within the column space of U.. Empirical results in Fig. support this view, showing that the entries of E are tightly concentrated near zero.
Theorem 1 (Approximation emror of projection onto U.). Let Wo UEVT mmxn be the Singular Value Decomposition (SVD) W. Suppose the fine-tuned matrix W E mRxx has the form
W = =UPP)
where:
U UP and V VQ are the left and right singular vectors of W*, respectively,
Σ = do 1 (W*), ..... minn(m. (W*)),
P= mm + E., Q = In + EQ, with EET < e, LE <E
Lei AW V* Wo. and let U, E Ruxr be the top-r left singular vectors of Wo. Then, the approximation error incurred by projecting AW onto the subspace spanned by U, satisfies
min(m...)
||W -U,,
of(W) + O(c).
The complete proof of Theorem detailed in Appendix B
Theorem indicates that the update AW can be well captured within the principal column space of Wo. The first term on the right-hand side, Emin(m.x) (AW), corresponds to the rank-r approximation error AW given by the Yockart-Yungg theorem (Eckart & Young 1936). The additional Oee) term reflects the small deviation introduced through E" and E and empirical evidence Fig. 2 suggesss that the O(e) term is negligible in practice. Appendix C.2 provides further observations large-scale models, which is consistent with this view.
Theorem shows that the dominant directions of the resulting update AW are well captured within the pre- tre-trinnd column space U, of Wo, This implies that by keeping U, fixed and learning only small set coefficients that determine the task-specific choice of how to move inside this space, substantially reduce the number of trainable parameters, which is precisely the notion of parameter efficiency we target.
Theorem D is meant to show that U, projection is globally optimal that projection alone guarantees task-optimal performance. Other projection spaces may also reach good optima, which does not contradict our claims. Rather, our contribution is to provide theoretical support for why this particular projection can work well, whereas most prior methods are justified only empirically.
3.2 SEQUENTIAL GRADIENT PROJECTION
Theorem 2 shows that the principal column space in Theorem can be naturally incorporated into PEFT by projecting gradients onto the subspace at each step. This provides practical way exploit the same effective space throughout training, offering simple and theoretically supported view of how sequential updates operate within the projection framework.
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: Adam with PiCa
Input: rank r; learning rate nj; decay rates 31, S9g small 0.
Setup/ Notation. For each group f e F and layer i: compute SVD W = (VA
and set pfı +- U.f [.1:a]
// Layer-wise fixed projector
Set W W Rm
For each group f: set shared compact states Bi Ma Vo' E E + set 0; Elementvise ops: @ (Hadamard), @ (elementwise divide), v- (clementwise).
repeat
++++;;
foreach group f do
// (1) Project layer-wise gradients aggregate
R/ + E. (PAN)T Vwra& ((wi)(
11. (2) Adam update in compact space
Mf < B,M, (1-1++1111)1111 R
V + (1- @ R();
Mr < Mf/(1-B1) V/ + v 1 (3);
AB Mf 2 (Vv/ +e); B t B + AB/
// (3) Decompress shared update to each layer foreach layer do
Whn Whi +n pf AB(;
until convergence:
return (B/)
// Final shared compact parameters
Definition (L-smoothness for matrix-valued functions). A differentiable function f: gmxx R is L-smooth (w.t. || ||x) if
[[e((W)) ve(W2)llr s L [| Walle for all W1, W2 E mR
Theorem (Sequential projection approximates accumulated projection). f. Bxxx RR. be L-smooth with [|Ve(W) S
G. Define the unprojected gradient descent path
Zi+1 - V)())
Let the accumulated-proojcttion iterate be
W Wo 17 Il,
v((Z.))
and the sequential-projectin iterates
P Ilu V((P),
Po Wo.
where Uy, U,U.T is the fixed rank-r projector. Then, for any T, the difference satisfies
n 2 1)) | PrllF <
The proof is provided in Appendix B
5

---

Published as a conference paper at ICLR 2026
3.3 PICA: PEFT WITH COLUMN SPACE PROJECTION
Based on the preceding results, we propose PiCa that projects gradients onto the principal column space spanned by pre-trained weights for each update. This gradient projection is effectively paired with our novel weight-sharing method for further parameter efficiency. For clarity, we describe PiCa in Algorithm T Jusing Adam, though the approach is not limited to this optimizer.
In Algorithm each functional group f E F= (query.key.valuee value.... is associated with a single trainable matrix B 1=*=" which is shared across all layers i 1..... L of the same group. The projection matrices pf remain layer-specific, leveraging the geometry of each pre-trained weight W E |R=xx. The gradients of each layer i are first projected onto p defined by the top-r singular vectors of the corresponding pre-trained weight, U.it E mR" The updates are then accumulated in this compact space shared parameters B Momentum and variance statistics also updated in this compact space. The shared update is then mapped back to each layer through its layer-specific projector U.F..
This procedure can be implemented using the reparameterization
Whi = of + UUABB
where Bf is trainable matrix initialized with zero and U. remains fixed during fine-tuning, Under this parameterization, optimizing BF is mathematically equivalent to the update Algorithm
Unlike prior approaches (Kopiczko et al.) 2023 Renduchintala et al. 2023) that primarily rely on random projection matrices for weight-sharing our method leverages layer-specific projection matrices U derived from the structure of the pre-trained weights W for each layer i of group f. This allows to capture the distinct characteristics and pre-trained knowledge encoded in each Wo' Given the use of unique projection matrices per layer, we posit that the trainable parameter B can be effectively shared across layers with the same functionality, facilitating efficient adaptation to downstream tasks. Our extensive experiments demonstrate the effectiveness of weight-sharingg in PiCa. which reduces the number of trainable parameters by up to 7x without compromising performance (see Sec. 4.3 for details).
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We evaluate the effectiveness of PiCa across a diverse set of Natural Language Processing (NLP) tasks, covering Mathematical Reasoning, Commonsense Reasoning, and Natural Language Understanding (NLU). For Mathematical Reasoning tasks, fine-tune
model on the MetaMathQA--4K
dataset (Yu et al. (023) and assess its performance on the GSM-8K (Cobbe et al.) 202I) and MATH (Hendrycks al. 2021) datasets. Furthermore, we conduct evaluations eight commonsense reasoning benchmarks: BoolQ (Clark et al. 2019). PIQA (Bisk et al., 2020), SIQA (Sap et al. 2019). HellaSwag (Zellers et al. 2019). Winogrande (Sakaguchi et al. (019). ARC-Easy/ARC- Challenge (Clark et al., 2018). and Open (Mihaylov et al. 2018). For NLU tasks, we utilize the GLUE benchmark (Wang al. 2018). We report matched accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for all other tasks. We employ the Sammaa-2B/78 (Tea et al. 2024), and LLaMA-3-8P (Al 2024) models for Mathematical Reasoning tasks and adopt the DeBERTaV3-baas (He et al. 2023 model for NLU tasks.
Beyond NLP, we also evaluate PiCa on vision tasks. Specifically, we conduct experiments with visual adaptation using the ViT-B/16 (Dosovitskiy et al.] (021) on 19 different datasets of VTAB-1K (Zhai) ctt al. 2020), grouped into Natural, Specialized, and Structured categories. Performance is reported as the average accuracy across these groups. In addition, we evaluate subject-driven generation tasks with the Stable Diffusion v2.1 (Rombach et al. 2022) on the DreamBooth dataset (Ruiz et al.) 2023), which includes 30 subjects and 25 prompts per subject, totaling 750 different personalization tasks. Following prior work (Ruiz et al. 2023). we report results using DINO for subject fidelity and CLIP-T for text fidelity. To ensure fair comparison, hyperparameters and training protocols

---

Published as a conference paper at ICLR 2026
aligned with those outlined in
(Lingam al. 2024 Cho al. 2024). Further details are provided in the Appendix C
4.2 RESULTS
For fair comparison, we follow (Lingam al. 2024 Dosovitskiy et al. 2021 Cho et al. 2024 and evaluate the effectiveness of PiCa across three NLP tasks (Mathematical Reasoning, Commonsense Reasoning, and Natural Language Understanding) and two vision tasks (Visual Adaptation and Subject-Driven Generation). The baselines include LoRA (Hu et al. 2022), DoRA (Liu et al. 2024a BOFT (Liu al., 2024b), VeRA (Kopiczko et al. 2023). and SVFT (Lingam et al. experimental details are provided in Appendix
2024 Full
Mathematical Reasoning In Table [ we provide results on mathematical question answering, comparing our method against baseline PEFT methods across three different base models ranging from 2B to 8B parameters. Our experiments include with fewer trainable parameters than SVFT" and low-rank configuration with fewer trainable parameters than rank LoRA. As shown in Table our high-rank PiCa consistently achieves superior performance while using the fewest trainable parameters across all models and datasets. In the low-rank setting, PiCa achieves either the best or second-best performance.
configurations of PiCa: a high-rank setting
Table 1: Performance on Mathematical Reasoning benchmarks (GSM-8K and MATH). #Params indicates the number trainable parameters. The best and second-best PEFT methods are highlighted in bold and underlined, respectively. For Gemma-7B, parameters remains below that of rank-1 LoRA. For SVFT we use 16 for Gemma models and d 12 for LLaMA-3 models. In the high-rank setting, PiCa consistently achieves the best performance across all models and datasets, while using the fewest trainable parameters.
set r 16 to ensure the number of trainable
Gemma-2B
Gemma-7B
LLaMA-3-8B
Method
#Params GSM-8K MATH #Params GSM-8K MATH #Params GSM-8K MATH
Full-FT
2.5B
52.69
17.94
8.5B
78.09
30.98
8.0B
76.57
26.12
BOFT 8
1.22M
36.01
12.13
2.90M
71.79
28.98
4.35M
67.09
21.64
DoRA,
1.19M
35.35
13.04
3.26M
74.37
26.28
2.55M
68.30
21.96
LoRA,
0.82M
32.97
13.04
0.82M
72.40
26.28
1.77M
68.84
20.94
VeRA, 1024
0.63M
36.77
14.12
0.43M
71.11
27.04
0.98M
63.76
20.28
SVFT
0.19M
40.34
14.38
0.43M
73.50
27.30
0.48M
69.22
20.44
PiCa,
0.67M
41.32
15.22
0.64M
74.30
28.92
1.38M
73.54
24.14
LoRA
26.2M
43.06
15.50
68.8M
76.57
29.34
56.6M
75.89
24.74
DoRA,=
13.5M
44.27
16.18
35.5M
74.52
29.84
29.1M
75.66
24.72
SVFT 6.35M
50.03
15.56
19.8M
76.81
29.98
13.1M
75.90
24.22
PiCa.=256
5.37M
52.77
16.36
10.22M
78.39
30.16
11.01M
76.12
24.88
Commonsense Reasoning In Table we evaluate commonsense reasoning performance eight benchmark datasets using Gemma-78 7B, following the same experimental setup as in the Mathematical Reasoning task. We compare both high-rank and low-rank configurations of our method against PEFT baselines. In both settings, PiCa outperforms all baselines on average across the eight datasets. In the high-rank setting, our method achieves state-of-the-a performance on seven out of eight datasets while using over 13x fewer parameters than LoRA, and it consistently outperforms SVFT on all eight datasets with approximately half number of parameters. achieves the best average performance, surpassing rank 1 DoRA while using more than 5x fewer parameters. Compared to SVFT", our method delivers superior performance on seven out of eight datasets, with an average improvement of nearly two percentage points. Similar trends are observed with Gemma-2B (see Appendix C33)
the low-rank setting, PiCa also
Natural Language Understanding Table 3
presents the results on the GLUE benchmark using DeBERTaVas B Compared to LoRA with rank 8, our method achieves over one percentage point
'Te official implementation available http:://ghhhb...c//////e/eeeee

---

Published as a conference paper at ICLR 2026
Table 2: Performance on Commonsense Reasoning benchmarks. #Params refers to the number of trainable parameters. The best and second-best PEFT methods are highlighted in bold and underlined text, respectively. In the high-rank setting. PiCa achieves state-of t-t-heerr art performance datasets, using over 13x fewer parameters than LoRA and about half the parameters of SVFT.
out of
Method
#Params
BoolQ
PIQA
SIQA
HS
WG
ARC-e
ARC-c
OBQA
Avg.
Full-FT
8.5B
72.32
87.32
76.86
91.07
81.76
92.46
82.87
89.00
84.19
DoRA,=1
3.31M
68.22
86.72
75.23
91.14
78.13
91.87
83.19
86.20
82.59
VeRA, 2004
1.49M
64,25
86.28
74.04
86.96
69.00
92.76
82.33
82.00
79.70
LoRA,=1
0.82M
65.44
86.28
75.02
89.91
75.92
91.79
81.91
85.40
81.46
SVFTP
0.51M
67.92
86.45
75.47
86.92
74.03
91.80
81.21
83.00
80.85
PiCa.-16
0.64M
70.95
86.29
76.00
91,42
76.32
92.89
83.19
85.60
82.83
LoRA,
68.8M
71.55
87.95
77.27
91.80
79.71
92.67
82.16
86.40
83.69
DoRA,-18
35.5M
71.46
87.59
76.35
92.11
78.29
92.00
80.63
85.60
83.00
SVFT d=8
9.80M
71.90
76.28
91.55
78.76
92.80
83.11
85,40
83.35
P1Ca,=128
5.11M
72.84
87.98
77.79
92.82
79.40
93.14
83.62
88.20
84.47
Table Performance of DeBERTa Daase on the GLUE benchmark. #Params refers the number of trainable parameters. The best and second-best PEFT methods are highlighted in bold and underlined text, respectively. While using more than 2.5 x fewer parameters than SVFT d=2+ PiCa outperforms it on all datasets.
Method
#Params MNLI SST-2 MRPC CoLA QQP QNLI RTE STS-B Avg.
Full-FT
183.83M
89.90
95.63
89.46
69.19
92.40 94.03 83.75
91.60
88.25
LoRA,=S
1.33M
90.65
94.95
89.95
69.82
93.87 91.99 85.20
91.60
88.50
LoRA,
0.17M
90.12
95.64
86.43
69.13
91.43 94.18
87.36
91.52
88.23
DoRA.
0.75M
89.92
95.41
89.10
69.37
91.53 94.14
87.00
91.80
88.53
BOFT
0.75M
90.25
96.44
92.40
72.95
92.10 94.23
88.81
91.92
89.89
VeRA, 1024
0.09M
89.93
95.53
87.94
69.06
90.40 93.24
87.00
88.71
87.73
SVFT
0.06M
89.69
95.41
88.77
70.95
90.16 94.27
87.24
91.80
88.54
SVFT d=2
0.28M
89.97
95.99
88.99
72.61
91.50
93.90
88.09
91.73
89.10
PiCa,=16
0.11M
90.20
96.00
91.40
73.10
91.60 94.20
89.20
91.80
89.69
Table 4: Performance vision benchmarks. VTAB-IK (ViT-B/16) is averaged over 19 datasets grouped into Natural, Specialized, Structured. DreamBooth is evaluated with Stable Diffusion v2.1 using DINO (subject fidelity) and CLIP-T (text fidelity). The best and second-best results are highlighted in bold and underlined, respectively.
VTAB-1K (ViT-B/16)
DreamBooth (Stable Diffusion v2.1)
Method
#Params Natural Specialized Structured
All
Method
#Params
DINO CLIP-T
LoRA.-s
1.32M
0.823
0.851
0.508
0.696
3.37M
0.618
0.305
DoRA,
1.41M
0.827
0.846
0.505
0.695
DoRA,=16
3.42M
0.617
0.306
SVFT
0.93M
0.820
0.844
0.486
0.684
SVFT
d=
2.50M
0.622
0.307
VeRA
0.45M
0.813
0.845
0.474
0.677
VeRA,=13312
1.80M
0.613
0.305
PiCa. 0.44M
0.825
0.851
0.508
0.697
PiCa.=128
1.72M
0.634
0.306
higher average performance. While using more than 2.5x fewer parameters than SVFT our method outperforms it on all datasets. Furthermore, despite using over 7 7 fewer parameters than BOFT, method achieves comparable average performance.
Vision Experiments Table reports results on VTAB-IK and DreamBooth dataset. On the VTAB- 1K dataset, PiCa achieves the best overall score while using the fewest trainable parameters. In particular, PiCa achieves competitive results compared to other baselines while using 2 to 3x fewer trainable parameters in VTAB-1K. On the DreamBooth dataset, PiCa achieves higher DINO score
8

---

Published as a conference paper at ICLR 2026
while maintaining comparable CLIP-T score, demonstrating strong personalization with fewer parameters than other baselines. These results highlight that PiCa maintains strong performance on vision tasks under substantially reduced parameter budgets.
4.3 FURTHER ANALYSIS
Ablation Study of Column Space Pro- jection Table 5 we compare the ef- fect of using column space projection ver- sus random space projection. We use commonsense reasoning benchmarks with Gemma-28 2B. The results show that column space projection improves overall accu- racy by 4.42 points compared to random space projection, demonstrating the effec- tiveness of leveraging the spectral struc- ture of pre-trained weights, aligned with the results in Theorem
Table 5: Ablation study projection choice (rank 256). Average scores are reported across commonsense reasoning benchmarks using Gemma-2B.
Projection Method
#Params
Avg.
Random Space
5.37M
63.18
Column Space (Ours)
5.37M
67.60
Share 135
Full FT (2500M)
Share (5.140
Accuracy (%6)
Accuracy
70
40
66
50
35-
PKCa (w/o shordtr 16.
sharel: 128.
LoRK: 8,8.
Avg.
Datasets
Params (M)
(a) Accuracy across Commonsense Reasoning datasets with and without weight-sharing weight-sharing re- duces the number of trainable parameters by up to 7x without compromising performance.
Accuracy on GSM-8K under varying rank tings. weight-sharing consistently yields superior performance under similar parameter budgets.
Figure 3: Ablation study of weight-sharing across different datasets and rank configurations.
Ablation Study of Weight-sharing In Fig. 3a analyze the impact of weight-sharing in PiCa across eight Commonsense Reasoning datasets using Gemma-7B. By comparing PiCa with its standard configuration (rank 128 with weight-sharing: 5. trainable parameters) against a variant without sharing (rank 16; 35.8M parameters), we find that the default PiCa consistently achieves performance comparable its non-sharing variant while requiring about 7x fewer trainable parame- ters. These results indicate that weight-sharin substantially improves parameter efficiency without performance degradation.
Furthermore, we conduct an additional study on the effect of weight-sharing under varying rank settings using the GSM-8K benchmark with Gemma-2B. shown in Fig. [3b] PiCa consistently achieves superior performance under similar parameter budgets compared to both its no-sharing ablation and LoRA.
DISCUSSION
While PiCa significantly reduces the number of trainable parameters required, introduces a minor limitation during inference. Specifically, PiCa stores only small shared matrix B for each functional group f. but requires to perform an additional SVD on the pre-trained weights Wo at load time to recover the projection matrix pf. U This presents trade-off between storage cost and loading overhead. If the loading overhead a concern, one can optionally store Ufa Nonetheless, in scenarios where multiple task-specific adaptations required from single base model, PiCa

---

Published as a conference paper at ICLR 2026
offers greater scalability; shared set of task- U.A can be pre-computed and paired with multiple sets of lightweight task-specific Bi, enabling efficient adaptation across diverse tasks.
CONCLUSION
In this work, introduced PiCa, parameter-efficiett fine-tuning method that integrates gradi- ent projection onto the principal column space of pre-trained weights with a novel weight-sharing mechanism. Our theoretical analysis establishes that column space projection provides effective inductive bias for fine-tuning, while the addition of weight-sharing offers substantial reductions in trainable parameters without compromising performance. Through extensive experiments, we demon- strated that PiCa consistently achieves competitive or superior results compared to state-of-the--e arrt baselines across wide spectrum of NLP tasks (Mathematical Reasoning, Commonsense Reasoning. and Natural Language Understanding) as well as challenging vision tasks (Visual Adaptation and Subject-Driven Generation).
Taken together, results indicate that PiCa offers a theoretically grounded and empirically validated approach to parameter-efficien adaptation of large models. We hope this work motivates further exploration of theoretically guided approaches that unify geometry-awant design with practical efficiency in fine-tuning large-scale foundation models. In future work, we aim to extend PiCa to more dynamic and practical settings such multi-task adaptation and continual learning, where efficient and scalable fine-tuning is critical.
ACKNOWLEDGMENT
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00345889, Research on Al Robustness Against Distribution Shift in Real- World Scenarios; and No. RS RS-2023-00222666 Center for Optimizing Hyperscale AI Models and Platforms).
REFERENCES
Meta AI. Introducing meta llama The most capable openly available 11m to date. April 2024. URL
https: //ai meta. com/bbo/////ocoo/ooo com/blog/ wollo//on /meta Ilama 3/
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning
about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy, Punica: Multi-tenant lora serving. Proceedings of Machine Leurning and Systems, 6: 6-1, 2024.
Wonguk Cho, Scokeon Choi, Debasmit Das, Matthias Reisser, Taesup Kim, Sungrack Yun, and Fatih Porikli. Hollowed net for on-device personalization of text-to-image diffusion models. Advances in Neural Informatinn Processing Systems, 37:43058 43079, 2024.
Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint 100.6.110..4168, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale, 2021. URL TRRLhttps. //arxi www.vorg/abba.... 1111292000.0199999
10

---

Published as a conference paper at ICLR 2026
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 133)2111228,, 1936.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fin-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7323-7334, 2023.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv: 11..111111000 1.01462, 2016.
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint 20. 1/544, 2023.
Vijay Chandra Lingam, Atula Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parfmeter--fffcientt fine-tuning with singular vectors. Advances Neural Information Processing Systems, 37:41425-41444 2024.
Shih- Yang
Shih-Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decompon low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024a.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng. Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In The Twelfth International Conference Learning Representations, 2024b.
Krishna Sri Ipsit Mantri, Biroia-Bibiine Schönlieb, Bruno Ribeiro, Chaim Baskin, and Moshe Eliasof. Ditask: Multi-task fine-tuning with diffeomorphic transformations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25218-25229, 2025.
Fanxu Meng, Zhaohui Wang. and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances Neural Information Processing Systems. 37: 121038-121072. 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering, 2018.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Asxociation for Computational Linguistics: Volume 2, Short Papers, pp. 157-163, Valencia, Spain, April 2017. Association for Computational Linguistics. URL
org. E1 7 20255
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng. Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning, Advances in Neural Information Processing Systems, 16:79320-79366 2023.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhancing parameter efficiency of lora with weight tying. arXiv preprint arXiv:2311.09578 2023.

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL httpp::/aaxiv..orrrtt.
0iss/2112.10552
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023. URL
https://arxis: org/ abs 2208. 12242
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande:
adversarial winograd schema challenge at scale, 2019.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019.
Zheyu Shen, Guoheng Sun, Yexiao He, Ziyao Wang, Yuning Zhang, Souvik Kundu, Eric P. Xing, Hongyi Wang, and Ang Li. ShareloRA: Less tuning, more performance for loRA fine-tuning of LLMs, 2024. URL /ttp:p:://pppneeeeeett net wttp://eeee/tturneeeeeee
Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:24/0.21222, 2024.
Yurun Song. Junchen Zhao, Ian Harris, and Sangeetha Abdu Jyothi. Sharelora: Parameter efficient and robust large language model fine-tuning via shared low-rank adaptation. arXiv preprint arXiv:2406. 10785, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08290 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium. November 2018. Association for Computational Linguistics. doi: (4/533/1/W11-5444/ URL https::/p//. //acl///cclcc a antho ogy. wwg.oorg//e /W18 5446
Per-Åke Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics, 12(1):99-111, 1972.
Hermann Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialgle- ichungen (mit einer anwendung auf die theorie der hohlraumstrahlung)) Mathematische Annalen, 71:441-479, 1912.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael T'channen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
04867]
Yuhua Zhou, Ruifeng Li, Changhai Zhou, Fei Yang, and Aimin PAN. Bi-share loRA: Enhancing the parameter efficiency of loRA with llyer and inter-layer sharing, 2025. URL https //penneview wttt:/ffrmmiidddie T
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A PRELIMINARIES
A.I NOTATION
Notation The following notation used throughout this paper:
For any matrix E xm let (AA denote its i-th largest singular value, with 01 x1()) 2 (A) 2 ... 2 m) 2 0.
|A]| Frobenius norm of matrix A, defined as ||A]|e E A3..
||l2: Spectral norm of matrix A, defined as ||A||l2 = 01 (A).
Ag: Entry at the i-th row and j-th column of matrix A.
J:: Identity matrix of size k x k.
diag(a),...,. (.,,,.,,,))) Diagonal matrix with entries ai,...,am An.
sin (U,, U²): denotes the principal angles between the subspaces range(U,) and range(U;))
A.2 PRELIMINARY RESULTS
Lemma A.1 (Weyl's Inequality (Weyl 1912 For A, B E mm and all
|a B) (A) ||
Lemma A.2 (Invariance of Frobenius Norm). If mm and U, V are orthogonal matrices, then |UAV |||||r.
Lemma A.3 (Orthogonal projection is non-expansive Frobenius norm). Let U, E RRaxrr have orthonormal columns and let II; U,U,T be the orthogonal projector onto range(U,). Then, for all X mm
XIIF S |X
B PROOF OF THEOREMS
Theorem I (Approximation enoor of projection onto U,). Let Wo UEVT <R be the Singular Value Decomposition (SVD) Wg. Suppose the fine- tuned matrix W E mmmx has the form
W = (VQ)
where:
U = UP and V* VQ are the left and right singular vectors of W", respectively,
Σ. = diag( ( (W"), .,...,.. min(m, (W*)),
P Im + F.,. Q In + EQ, with |E <,, 1E <e
Let AW W* Wo. and let U, E guxr be the top-r left singular vectors of Wo. Then, the approximation error incurred by projecting AW onto the subspace spanned by U, satisfies
min(m.x)
|W -U,UU,, -U,
Σ
o?(AW) + +(())
13

---

Published as a conference paper at ICLR 2026
Proof. We derive the inequality through series of steps, decomposing the perturbation, analyzing the projection error, and bounding the terms using spectral and entrywise techniques.
The perturbed matrix has the form
W = U + E' + E
Subtracting Wo UEVT gives
AW U [4 + (L, - VT.
For notational clarity, define
== + (h+ -E,
so that AW = T.V Let us expand H explicitly. Multiplying out terms yields
(Im + F + ET)T = E' + + + (F9) + EBET(E
Thus
= E1 E3,
where
D= E, E E
E -E Eg (F9)
The diagonal matrix D captures the shifts in singular values: Da = o.(W)) (W))
The error of projecting AW onto U, is
|AW U,U, A
Since AW = T and U.U = [T, 0], we can write
I 0 HV
U,U, AW = U
0
0
Subtracting gives
- AW U.U,T AW = -HH-H
I,
where P,
By invariance of the Frobenius norm,
0
|AW - U.U. Aw = H P.H. ΣH.. H
imr+1
For i > r, each entry has the form
Hij = Dig + Ei,5 + E2,15 Es,úi-
For diagonal terms (j = i), we have
H (W") o. (Wo)+ E(o,(("") (W, E + Σ (
k
Using Lith. ( e, can bound each component:
Ei,al S (W:) |Eg,il I cW,),"), |E & e² min(m, n) Fmax (W*).
For off-diagonal terms (j / i), we have
Hij Effa (o')) EEhor("))
leading to analogous bounds
Fl.igl S (W)(W"), |F2 <o,(W"),, Faaiy 62 min(m, n) mmax (W*).
14

---

Published as a conference paper at ICLR 2026
We now square and sum these contributions. For diagonals,
H = (av(W") (.,)) -VV E2,4 Bs,u) + (Ei, + E2,1 + Ez,1s)
Cross term is bounded using Cauchy-Schwarz, and third quadratic term is bounded by 3
F2 + a. Therefore,
minfm.n)
min(m.n)
Σ
H S
Σ
(or(W*)- ow + 222
where
min(m,a)
C
Σ
2(o,(W") o: (WW) + + ma)n(m, (W*))
i=+11
min(m.n)
C2 =
Σ 3(20)(W*) + mminmmm n² y (W
Similar expansions apply for off-diagonal terms, where only E,, Ez, Ey contribute. For off-diagonal
terms:
H
Σ
ELay + Ba + Ez,is)²
Σ
3 + E2.4 + EZ,0)) S 2²
i=++j== f+i
3f1
where
2
C3
Σ
30o]
? min(m, n, n2 max (W*))
jæi
Collecting everything, the sum takes the form
m
n
m.))
Σ
EHS
H S Σ
(a:(W)) - o(:o)) +001 + 22 Cz).
i=+1-==
Recall the decomposition
H = D+ + E2 + E3,
AW = UHV
so that by orthogonal invariance of singular values
(AW) (1) for all i.
Since UP and VQ are the singular- vector matrices of W the factors P, Q are orthogonal. Hence
= D
(D( = jo,(W*) (:WW) (Vi).
Let Frot = Eg + F. + Eg. By Weyl's inequality applied to D + Frot.
(0,(1)) oD) (a,(W)
- BBot 1|2-
We now bound |Ese ll2 piecewise. Using submultiplicatiiity and | l|2 S ||P||r ||r S /mnc (and similarly for E get
E 1l2 1E"""1 |E"
ll2 || 1|2 S mn e F m'*))
| Eg l2 = [E 112 < |E*||
EE 112 S / /nne max (W*),
| = EE (E9) 12 S | E 2/ | 2|2 & mn e² m (W*).
Therefore
F2ooel22 S 2/mme maxx (W*) mn 2 mmase (W*).
15

---

Published as a conference paper at ICLR 2026
Define
6 == o,(AW) - lo:(W>* -w:))
[8:| x
Peot 2|
Then
- oi(AW) d,.
and squaring gives
(o,(AW) 0.)² S di()W) + 20,(AW) [| + |rot 2- Let :== min(m, n). Summing for i I,...,e.
( <
(AW) + 2 Boot ll Σ 1=++1
a(WA) + (e (e-r) Etot
1=+1
i=+1
With the bound on |iott|2 just obtained, this can be written as
Σ
(((W))oo:(We))) <
Σ
((WW)
+ cCa + Co.
2m7+1
where
C4 2(2/mm m (W*) + mn o
m (W*))
Σ (i(A))
i=7+1
Cs = (l r) (2 max (W*) + mi Tmax (W+))²
Finally, recalling the earlier analysis, finally combine the bounds to obtain
min(m.n)
|AW - U,U, AW
Σ
(AW) cCI + &., + &C + cC4 cC4+2Cs
i=++
min(m.n)
Σ
(AWW) +
im+l
where
C= (C, cC2 + C
Theorem 2 (Sequential projection approximates accumulated projection). Let l Bx - R be L-smooth with [[VE(W) [ S G. Define the unprojected gradient descent path
Z Z, v)()).
Let the cownumullate-rojjectiin iterate be
T-1
W, = Wn 11 Iy.
v((2+))
1=0
and the
iterates
- V((P),
Po = Wo.
where Iu. = U,U. is the fixed rank-r projector. Then, for any T, the difference satisfies
r²²
w - Prlle
<
2
16

---

Published as a conference paper at ICLR 2026
Proof. We now prove that the sequentially projected iterates closely approximate the delayed pro- jection iterate when both use the same fixed projector Ily, = U,U,". Throughout we work with the Frobenius norm, and recall from Lemma A.3 that Hy, is non-expansive in |!
The delayed projection iterate is defined by
T-1
Wrdelayod = Wo 17 Iu
VE(Z)))
Z411 2: -1vV2(2)))
t=
The sequentially projected iterates follow
Pt+1 n IT, V((P)),
Po Wo-
Subtracting the update rules yields
T-1
Pr WT Wedelayed
Σ
He. (VE(P)) vZ(Z.))
Taking Frobenius norms and using |[ ||= 1
T-1
Pr Wrdelayed
F 1 1
V(((,) V((Z)ll
1=
By Definition [ l is I-smooth w.r.t. || |F, so the gradient is I.-Lipschitz:
S LIPt ZellF.
-
Denoting D, || Z, we obtain
T-1
Pr
Wedelaydd
S nL Σ Dr.
t=
To bound D,, expand one step of the deviation:
D++1 PP++1 Zet1lle = Pt - --,,,, V((P)) (Z, - -V(())l2
P. Z, (Il, V((P:) - ve(Zo))le.
Applying the triangle inequality and splitting terms,
D++1 $ Dj + [ (V((P)) - vε(Z,)) + + nl([ -I,
Ic, (((Z))l
For the first term, by non-expansiveness of NT, and L-smoothness,
|[T (V((P:) VE(Z))) s [[V((P)) Ve(Z.)]l S LD.
For the second term, since [[((2))] by assumption.
[I(I - VeZiill & ||VE(Z)))l s G.
Hence the recurrence
D (1+nL)D, GG
With D, = 0, standard unrolling argument gives
G (11 DL)² -1) s L
G (erLe -1).
D, s I
Plugging back into Step 2.
T-1
T-
|| Pr - W
||
< nL Σ
Dr
< G
Σ
ge -1).
t=0
t=0
17

---

Published as a conference paper at ICLR 2026
For small nLT, we use the recond-order Taylor expansion of the exponential:
e" - -- o/2 + +(x²)
as x + 0.
Applying this with x ntLt yields
en -1 =ngl [(n ((TTa))))
and hence
T-1 D.
T-1
L2
L Σ
S G
Σ
(enta -1
T(T 1) + ((nLT)))
2
t=
1
Combining all estimates, we conclude
W Prile S L.G -(T-1) ((LLT)))
which shows that the sequential projection scheme faithfully tracks the delayed projection up to higher-order error in the learning rate and horizon.
C IMPLEMENTATION DETAILS AND ADDITIONAL EXPERIMENTS
To ensure a direct and unbiased comparison with existing baseline methods, we adopted the same experimental setup outlined in SVFT (Lingam et al. (024 for NLP tasks. For consistency, all baseline results in NLP tasks were also sourced from (Lingam et al. 2024), enabling a fair evaluation of our method's performance. For vision tasks, we follow Dosovitskiy et al. (202T) and Cho al.
(2024).
For Mathematical Reasoning tasks, reproduced Full Fine-Tuning (Full FT) experiments for Gemma-78 7B and LLaMA-3-8B using smaller learning rates. While previous results were reported in (Lingam et al. 2024), we found that with sufficiently low learning rates (1e-6/5e-6), Full FT achieves higher accuracy than PEFT methods. We suspect SVFT used higher learning rates (le-5/5e-5) for
Full FT. The comparison is summarized in Table
Table 6: Full Fine- Sine-Tunigg (Full FT) accuracy on GSM8K and MATH.
Setting
Gemma-77
LLaMA-3-SB
GSM8K
MATH
GSM8K
MATH
Reported in SVFT
74.67
25.70
64.13 76.57
16.24
Our Reproduction (Low LR)
78.09
30.98
26.12
C.I IMPLEMENTATION DETAILS
Mathematical Reasoning Table 7 presents the hyperparameter configurations employed for these experiments. For the Gemma model family, PiCa is applied to the Q, K, V. U, D matrices, while for the LLaMA-3-SB model, the Q, K, V, U, DDO.D.C. matrices are targeted. The experimental codebase and evaluation procedures git] and the fine-tuning dataset are sourced from Met haat:h//MMMMMaataaaaaaaaaa
adapted from hhtp::///aattub..c.////aaaaaaaip http://www.....coaaaaaaeeeeeeeee co/datasets///////
met tatta-math/Mettee
Commonsense Reasoning We follow the setting outlined in prior work (Lingam et al. 2024), fine-tuning on 15K examples. The hyperparameter configurations for these experiments are de- tailed in Table 8. We utilize the same set of matrices as in the Mathematical Reasoning tasks. The codebase, including training and evaluation data, is sourced from hntthhtss///// httpp:://ggttu..... com/ Vi jay..iggm955 http///Ft.it..
18

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameter setup used for fine-tuning on MetaMathQA-40K.
Hyperparameter
Gemma-2B
Gemma-7B
L.AMMA3-8B
Optimizer
AdamW
Warmup Ratio
0.1
LR Schedule
Cosine
Max Seq. Len.
512
# Epochs
Batch Size
64
Rank
32 1E-03
256
16
256
32
256
Learning Rate
9E-04
IE-04
5E-05
2E-04
2E-04
Table 8: Hyperparameter setup used for fine-tuning 1onmonsense-15K.
Hyperparameter
Gemma-2B
Gemma-7B
Optimizer
AdamW
Warmup Steps
100
LR Schedule
Linear
Max Seq. Len.
512
# Epochs
Batch Size
Rank
32
256
16
128
Learning Rate
1E-03
9E-04
3E-04
8E-05
Natural Language Understanding We fine-tune DeBERTaV3:
base (ee et al. 2023 applying PiCa
all linear layers within each transformer block. We constrain hyperparameter optimization to moderate adjustments of the learning rate and the number of training epochs. For rigorous comparison, we employ identical model sequence lengths those reported by (Lingam et al.) 2024 Liu 2024b). The precise hyperparameter settings utilized in these experiments specified in Table 9
Table 9: Hyperparameter setup used for DeBERTaV3 on the GLUE benchmark.
Method
Dataset
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Optimizer
AdamW
Warmup Ratio
0.1
LR Schedulc
Linear
Batch Size
Max Seq. Len.
256
128
320
64
512
320
320
128
Learning Rate
3E-04
1E-03
2E-03
8E-4
3E-04 5
1E-04
1E-03
3E-03
PiCay-16
Epochs
7
35
50
15
40
15
Vision Experiments For vision adaptation tasks, we fine-tune ViT-B/16 (Dosovitskiy et al. 2021 by updating all linear layers within each transformer block, using a learning rate of 0.004 for PiCa and LoRA, 0.005 for DoRA, and 0.05 for VeRA and SVFT. For all methods, the classifier learning rate fixed at 0.005. Fine-tuning is conducted for 10 epochs, and the checkpoint from the best validation epoch is used for testing. The same hyperparameter configurations are applied across all 19 datasets VTAB-1K (Zhai et al., 2020)). For subject-driven generation tasks, we follow training and evaluation protocols of previous works (Lingam al.) 2024 Cho et al. 2024). We use learning rate 0.0001 for LoRA and DoRA, 0.0005 for PiCa, 0.001 for SVFT. and 0.005 for VeRA. Other settings remain the same with Cho et al. (2024).
19

---

Published as a conference paper at ICLR 2026
C.2 EVIDENCE FROM LARGE- SCALE MODELS.
While Fig. provides visual evidence of subspace alignment in moderate-scale settings, here we empirically validate the assumptions underlying Theorem [ on a larger model. Specifically, analyze LLaMA3-8B fine-tuned on Commonsense Reasoning benchmarks.
For each pair of pre-trained and fine-tuned weight matrices, we computed the cosine similarity between their singular vectors and defined Diagonal Similarity the average of the diagonal entries of the similarity matrix, aggregated across layers of each module (query, key, and value). The consistently high Diagonal Similarity values reported in Table I0 demonstrate that the leading singular subspaces remain well aligned after fine-tuning, thus supporting the subspace stability assumption of Theorem
We also extend the analysis of Fig. by reporting the averaged entries of E" and E across layers. As shown in Table 10 these values are tightly concentrated around zero, empirically confirming that the additional (c)) term in Theorem is negligible in practice.
F
Table 10: Empirical validation of Theorem assumptions LLaMA3-SB fine-tuned for Common- sense Reasoning. Diagonal Similarity measures alignment of singular vectors between pre-trained and fine-tuned weights. The averaged values of E.j and Ei are tightly concentrated near zero, confirming that the (e)) term is negligible.
Layer
Diagonal Similarity
E
E
Query
0.927 0.047
-2.44c 4 4.27e- -
-2.44e 4.25c
Key
0.998 0.003
-9.66c
-
-9.66e
10--4-13 3.76c
Value
0.972 = 0.011
-9.69e-44 2.76c
-9.66e
2
C.3 COMMONSENSE REASONING WITH GEMMA-2B
We evaluate PiCa commonsense reasoning tasks with Gemma-2B. The results presented in Table TI PiCa achieves the highest average performance across both high- and low-rank settings, outperforming the second-best method by approximately 2-3 percentage points.
Table 11: Performance on Commonsense Reasoning benchmarks using Gemma-2B. #Params refers to the number of trainable parameters. The best and second-bes best PEFT methods are highlighted in bold and underlined text, respectively. PiCa achieves state-of- of-the-art average performance across both high- and low-rank settings, outperforming the second-best method by up to percentage points.
Method
#Params
BoolQ
PIQA
SIQA
HI
WG
ARC-e
ARC-c
OBQA
Avg.
Full-FT
2.5B
63.57
74.10
65.86
70.00
61.95
75.36
59.72
69.00
67.45
BOFT
m=2
1.22M
59.23
63.65
47.90
29.93
50.35
$9.04
42.66
41.00
49,22
2045 2048
0.66M
62.11
64.31
49.18
32.00
50.74
58.08
42.83
42.60
50.23
LoRA,
0.82M
62.20
69.31
56.24
32.47
51.53
69.52
48.80
56.40
55.81
DORA
1.19M
62.17
68.77
55.93
32.95
51.22
68.81
48.72
55.60
55,52
SVFT
0.19M
62.26
70.18
56.70
32.47
47.04
69.31
50.08
58.40
55.81
PiCa,=12
0.67M
62.11
71.76
60.13
36.49
50.59
73.74
52,56
63,20
58.82
LoRA,-12
26.2M
63.11
73.44
63.20
47.79
52.95
74.78
57.16
67.00
62,43
DoRA,
13.5M
62,87
73.93
65,34
53.16
55.51
76.43
59.55
68.40
64.40
SVFT! B
6.35M
63.42
73.72
63.86
71.21
59.58
73.69
54.77
66.60
65.86
PiCa 256
5.37M
63.91
75.57
64.38
71.75
60.62
77.44
58.70
68.40
67.60
C.4 RESOURCE AND EFFICIENCY ANALYSIS
We present a comparative analysis of training memory usage between PiCa and the state-of-the-ar baseline SVFT. Training memory is measured by peak GPU memory usage during fine-tuning. As
20

---

Published as a conference paper at ICLR 2026
Table 12: Trainable parameters and training memory consumption for different parameter-efficiet fine-tuning methods on Gemma-7B.
Method
LoRA,=32
DoRA,=16
BOFT m=2
VeRA,=1024
SVFT
SVFT
PiCa,-128
Params 68.8M
35.5M
2.90M
0.43M
0.43M
19.8M 71,421
5.11M
Memory
52,764
55,334
65,423
55,980
70,176
52,614
shown Table 12 although both SVFT and SVFT d=16 substantially reduce the number of trainable parameters, they require up to 36% higher GPU memory consumption compared to PiCa,=128.
This overhead arises from SVFT's factorization of weight updates as AW UMV where U' E Rmxm and V Rnx* denote the singular vectors of the pre-trained weight matrix Wo E IR Although U and V are not trainable, they must be retained throughout fine-tuning, leading to significant memory overhead. These results demonstrate that PiCa offers more memory-efficient alternative, particularly in resource environments where memory I/O constitutes a critical bottleneck.
C.5 COMPARISON UNDER WEIGHT-SHARING
We compare PiCa with PEFT baselines under the same weight-sharing, scheme. For SVFT, we report both Random and Banded variants with sharing. As shown Table [3] PiCa achieves the best performance while using fewer parameters.
Table 13: Results with weight-sharing on GSM-8K and MATH using Gemma-2B.
Method
#Params
GSM-8K
MATH
LoRA (r=32, B shared)
14.83M
44.28
15.08
DoRA (r=16, shared)
7.79M
41.02
14.92
SVFT R w/ sharing (d = 280)
5,48M
50.49
15.86
SVFT sharing (d 280)
5.48M
50.34
16.18
PiCa (r=256)
5.37M
52.77
16.36
D LLM USAGE
We used large language models only for minor tasks such as spell-checking, grammar correction, and formatting.
E REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. All models, datasets, training protocols, and hyperparameters required reproduce experimental results described in detail in Section and Appendix C
21


Reviews:
[{"review_id": "3Ec7DFhgxq", "reviewer": "Reviewer_e2mp", "summary": "This paper introduces PiCa, a new Parameter-Efficient Fine-Tuning (PEFT) method. The core idea is to improve parameter efficiency by providing a theoretically-grounded inductive bias. The method operates by projecting gradients onto the principal column space of the pre-trained weights, which the authors argue is the most effective subspace for adaptation. This gradient projection is then combined with a novel weight-sharing strategy, where layers with the same function (e.g., all \"query\" matrices) share a single set of trainable parameters, further reducing the parameter budget.\n\n\nThe authors provide a theoretical analysis to support this approach. Theorem 1 aims to justify that the principal column space captures the ideal update, based on the observation that fine-tuned weights are a small perturbation of the pre-trained weights . Theorem 2 argues that PiCa's practical \"sequential projection\" algorithm is a high-fidelity approximation of an \"ideal\" but computationally infeasible \"accumulated projection\" . Empirically, the paper demonstrates that PiCa achieves strong results on a wide array of NLP and vision tasks, often outperforming baselines like LoRA and SVFT with a significantly smaller parameter count", "questions": "Please see my comments above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles the problem of parameter-efficient fine-tuning, which is of critical importance for the practical adaptation and deployment of large-scale foundation models.\n\n2. The paper is well-written and clearly structured. The authors effectively articulate the components of their method (gradient projection and weight-sharing) and the high-level intuition behind their theoretical claims.\n\n3. The experimental evaluation is commendable for its broad coverage of foundation models. The method is tested across a diverse set of backbones, including both vision and language foundation models.", "weaknesses": "My main concerns lie with the theoretical justification, which feels less convincing than presented, and the comprehensiveness of the experimental baselines, which are insufficient to fully support the claims of state-of-the-art parameter efficiency.\n\n1. The paper claims to provide a strong theoretical foundation, contrasting with prior \"heuristic\" SVD-based works. However, I am not fully convinced by Theorem 1's argument for projecting onto the column space. The justification relies on the finding that Full Fine-Tuning (FFT) produces a small deviation (gap) from the pre-trained weights . This \"small deviation\" is often an artifact of the FFT setup itself (e.g., low learning rates, few epochs) which is intentionally designed to prevent catastrophic forgetting. Thus, the argument feels somewhat circular: it observes that a \"forgetting-constrained\" method produces a small change, and then builds a theory based on this small-change assumption, rather than proving that the optimal unconstrained solution must lie in this space.\n\n2. Both theorems provided upper bounds for the errors. These bounds look incomplete to me, in the sense that they are not proven to be optimal or tighter than bounds that could be derived for other projection (or even pruning) strategies. They do not, by themselves, demonstrate that this specific projection policy is inherently superior to any other. \n\n3. The entire theoretical analysis is based on matrix norms (e.g., Frobenius norm). A small approximation error in the matrix norm does not necessarily or directly translate to higher parameter efficiency or better final task performance. This represents a significant gap between the theoretical claims and the practical outcomes.\n\n4. The paper's core hypothesis—that the principal column space (the \"head\" of the weights) is the correct subspace for adaptation—is debatable. A growing body of work (e.g., [1], [2]) has suggested the opposite: that the tail singular vectors or the null space of the pre-trained weights may be more important for learning new task-specific knowledge while mitigating forgetting. The paper fails to acknowledge or address this contradictory line of research.\n\n\nIn addition, I feel that the experiments can be enhanced: While the backbone model diversity is a strength, the diversity of the PEFT baselines is a significant weakness. The paper claims state-of-the-art parameter efficiency but omits comparisons to many crucial, high-performance PEFT methods. To truly situate PiCa's efficiency, comparisons against methods like MiLoRA ([1]which focuses on the tail-space), PiSSA ([3] which also uses principal singular vectors), AdaLoRA ([4] which dynamically allocates rank), , and RoseLoRA ([5] which prunes LoRA entries in an unstructured way) are necessary. Without these baselines, the claim to superior efficiency is not fully substantiated.\n\n\n\n[1] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning.\n\n[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.\n\n[3] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models.\n\n[4] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning.\n\n[5] RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNsyPPmCXc", "reviewer": "Reviewer_UoTk", "summary": "This paper proposes a PEFT method which involves, for any given weight matrix, using the singular vector spaces of that matrix to determine which parameters to update. In particular, if a matrix W has SVD given by USV', and the top-r components are U_r and V_r, this paper proposes a low-rank update of the form U_r P V_r' - where P is a small square matrix of their trainable parameters. \n\nFurther, to decrease parameter count, they do weight-tying of these P across groups of layers.\n\nThey show improvements over a few other methods that do similar things for PEFT.", "questions": "What does specifically does Theorem 2 add to the method from an algorithmic perspective? It is a pretty standard derivation from Lipschitz analysis (which, also does not hold for these models which have ReLUs in them).\n\nHow is it that PEFT methods beat fullFT even in accuracy for the 7B and 8B models (Table 2)? Was the FullFT done properly?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method is clearly written. While not exactly theory, there is mathematical grounding of their method presented. They have compared against the correct set of competing methods.", "weaknesses": "The main issue with this paper is that it is a special case of the SVFT method, with weight tying. In particular (see the notation above in the \"Summary\" of this review), in SVFT the update is of the form U Q V' where Q is a matrix with <some> pre-determined sparsity pattern. The only difference in this paper is that this sparsity pattern is fixed to be the r x r \"top left\" square corresponding to the main singular values.\n\nThe other innovation is weight tying across layers, a standard parameter-saving technique. Indeed it is likely the gains over SVFT arise mainly from weight-tying, and not some fundamental algebraic structural difference of the update. A fairer comparison would be with weight-tying applied to other methods as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1rBUCAiWVi", "reviewer": "Reviewer_8LsH", "summary": "This paper introduces PiCa, a theoretically grounded PEFT method that exploits of the spectral geometry of the pre-trained weights. PiCa projects gradient updates onto the principal column space (top-rank singular vectors) of each pretrained weight matrix, supported by theoretical analysis showing this projection yields a near-optimal low-rank update. To further reduce trainable parameters, PiCa also shares the projected update matrix across layers of the same type. Experiments across multiple NLP and some vision tasks demonstrate consistent state-of-the-art performance compared to previous SoTA PEFT baselines with fewer trainable parameters.", "questions": "1. Could you apply the same weight-sharing scheme to SVFT so both methods use a comparable trainable-parameter budget? This would isolate the effect of PiCa’s rank-r projection versus SVFT’s full-rank update.\n\n2.  Have you assessed robustness to decoding hyperparameters (e.g., temperature, top-p) and reported mean ± std over multiple seeds? Re-running with varied sampling settings would strengthen claims of statistical reliability.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- By projecting gradients onto the top singular vectors’ subspace, PiCa leverages a mathematically principled inductive bias. The authors prove that this yields a near-optimal low-rank update (minimizing Frobenius error) up to small residual terms.\n\n- Extensive experiments validate that PiCa achieves competitive or superior performance while using fewer parameters than baselines.", "weaknesses": "- PiCa’s theoretical guarantee (Theorem 1) assumes that the transformation from pre-trained to fine-tuned singular vectors is almost identity, i.e. fine-tuning doesn’t drastically alter the singular directions. While empirically supported for conducted experiments, this might not universally hold. If a downstream task requires introducing entirely new features or directions that were not significant in the pre-trained model, limiting updates to the original top-r singular space could be restrictive. In cases where the task’s optimal solution exists outside the column space of the pretrained weights (e.g., OOD tasks), PiCa would then incur an approximation error equal to the omitted singular components.\n\n- Weight-sharing theoretically reduces flexibility; broader ablations on when sharing hurts would strengthen claims. Running PiCa on some OOD downstream tasks, or on less capable base LLM models would offer more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qlw4LtLJ0Z", "reviewer": "Reviewer_XBE4", "summary": "This paper introduces PiCa, a novel and theoretically grounded method for parameter-efficient fine-tuning (PEFT) of large foundation models. The core idea is to project gradients onto the principal column space of pre-trained weights, a space empirically and theoretically shown to preserve fine-tuning directions with minimal update error. To further reduce trainable parameters, PiCa integrates a weight-sharing mechanism, where shared compact parameters are used across layers within the same functional group. The proposed method is evaluated across a wide range of NLP (Mathematical Reasoning, Commonsense Reasoning, GLUE) and vision tasks (VTAB-1K, DreamBooth). PiCa outperforms state-of-the-art PEFT baselines, including LoRA, DoRA, SVFT, and VeRA, under equal or lower parameter budgets.", "questions": "- **How sensitive is PiCa to the choice of projection rank $r$?** Although some rank settings are explored, it would be helpful to know if PiCa is **robust across a range of $r$**, especially in lower-resource settings. Could adaptive rank selection (e.g., via energy thresholding) be incorporated?\n\n- **Can PiCa be combined with LoRA or orthogonal fine-tuning?** Given its theoretical grounding, could PiCa act as a drop-in enhancement to existing PEFT layers like LoRA by refining the directionality of updates?\n\n- Can the authors comment on scalability in terms of actual wall-clock latency or GPU memory?\n\n- The shared representation $\\theta_f$ and structured projection matrix offer intriguing possibilities for compositional fine-tuning. Could this architecture be extended to **task-conditioned adaptation**?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- **Innovative method with practical benefits:** The combination of gradient projection and parameter sharing is novel and effective. The layer-specific projection matrices allow for structural adaptivity, while shared compact parameters reduce redundancy, enabling PiCa to use up to 13× fewer parameters than LoRA without sacrificing performance.\n\n- **Comprehensive experiments:** PiCa is extensively evaluated across three NLP subfields and two vision domains, including strong baselines. The method shows superior average performance across tasks such as GLUE, GSM-8K, and VTAB-1K, highlighting its general applicability beyond a single modality.", "weaknesses": "- **Inference-time overhead not deeply addressed:** PiCa requires computing (or storing) the top singular vectors of each pre-trained layer for projection. Although the authors mention this as a minor load-time overhead, more discussion or benchmarking of real-world inference-time latency/storage trade-offs would be helpful.\n\n- **Lack of comparison on long-context or sequence-heavy tasks:** While the NLP evaluations are diverse, no experiments are shown on long-context reasoning, e.g., document-level QA or summarization. Since PiCa modifies fine-tuning dynamics, it remains unclear how it scales in such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)