IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
Authors: Yuan Yin
Shashanka Venkataramanan
Tuan-Hung Vu
Andrei Bursuc
Matthieu Cord
Venue: Accepted by TMLR ()

Abstract: Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA’s down-projection is randomly initialized and data-agnostic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving on average 1.5 points higher accuracy on commonsense reasoning and 2.3 points on VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https://github.com/valeoai/peft-ipa.

Published in Transactions on Machine Learning Research (12/2025)
IPA: An work for Efficient Foundation Model Adaptation
Input Projection Frame-
Yuan Yin Valeo.oi, Paris, France
yuon. guan.yin@valeo.com
Shashanka Venkataramanan
shashanka. Sankllmmkeeeekkettommmnnneeeeooe
Valeo.ai, Paris, France
Tuan-Hung Vu
Valeo.ai, Paris, France
Andrei Bursuc
ondrei, wwwrriibbusuuc@aallo.coo
Valeo.ai, Paris, France
Matthieu Cord
matthieu.corddávlle... com
Valeo.oi, Paris, France
Sorbonne Université, CNRS, ISIR, F-75005 Paris, France
Reviewed on OpenReview: https: openreview. net/ formm? id= almQeZa2pR
Abstract
Parameter--fficient fine-tuning (PEFT) methods, such as LoRA, reduce adaptation cost by injecting low-rank updates into pretrained weights. However, LoRA's down-projection is randomly initialized and dala-agnoslic, discarding potentially useful information. Prior analyses show that this projection changes little during training, while the up-projection carries most of the adaptation, making the random input compression a performance bottleneck. We propose IPA, a feature-aware projection framework that explicitly aims to reconstruct the original input within a reduced hidden space. In the linear case, we instantiate IPA with algorithms approximating top principal components, enabling efficient projector pretraining with negligible inference overhead. Across language and vision benchmarks, IPA consistently improves over LoRA and DoRA, achieving average 1.5 points higher accuracy on commonsense reasoning and 2.3 points VTAB-1k, while matching full LoRA performance with roughly half the trainable parameters when the projection is frozen. Code available at https: //github/ www.co////aaoooe cwm.valeeaippett-iae
1 Introduction
Foundation models have transformed machine learning research and applications with their broad capabilities in language and visual understanding. These capabilitics arisc from model scalc, since models with billions of parameters pre-trained massive corpora (Bommasani gerpral-purpose models to specialized tasks domains remains challenging: fine-tuning becomes computationally and financially expensive (Houlsby et al., 2019; Hu et al., 2022).
al., 2021). However, adapting such
model sizes grow, full
To address this bottleneck, the community has developed a range of parameter-efficient fine-tuning (PEFT) methods that reduce the number trainable parameters by an order of magnitude compared to the base model (sce surveys, c.g, Han al., 2024; Zhang al., 2025). Among these, Low-Rank Adaptation (LoRA; Hu et al., 2022) has gained significant traction due its simplicity and effectiveness in the large-language-ne community. Tn LoRA, each target weight matrix reparameterized as the sum of the original pre-trained

---

Published in Transactions on Machine Learning Research (12/2025)
weight W and a low-rank update AW = BA, where A (the "down" projection) maps imputs into a lower- dimensional space and B (the "up" projection) maps them back to the output dimension. This additional module can be merged into W without introducing any extra latency during inference.
Although there has been a fiurry of follow-up works around LoRA, most focus on alternative initializations (Meng et al., 2024; Yang et al., 2024) or extended structures (Liu et al., 2024; Huang et al., 2025; Albert et al., 2025) by restricting their analysis to the pretrained weight matrix, while paying little attention to the distribution of imput features. In contrast, we broaden the focus to explicitly account for the role of imput features.
In the original LoRA formulation, the down-projection matrix A is randomly initialized and thus data-agnostic. Analyses of LoRA's i inherent asymmelry show that during adaptation, this down-projection A remains close to its initialization, whereas the up-projection B adapts more effectively to the data (Tian et al., 2024; Hayou et al., 2024b; Zhu et al., 2024). This suggests that a data-agnostic input projection can become a performance bottleneck, motivating its replacement with a feature-aware, dala-dependent alternative that better aligns with the intrinsic structure of the inputs.
In this paper, we tackle this direction and introduce IPA, an input-featuuc-aware projection scheme that preserves optimally the imput information in the reduced hidden feature space. Our contributions are:
projeccioooo pretraining objective that favors information reconstruction in the bottleneck dimension through an encoder-decoder formulation. We instantiate this framework in the linear setting using efficient forward-only pretraining algorithms. We empirically validate IPA language and vision-language tasks, showing consistent improvements over random linear projections. On several architectures, IPA matches the performance of fully trained LoRA while requiring roughly half as many trainable parameters.
We formulate adaptation with a dedicated
2 Related Work
Parameter efficient adapters. PEFT techniques address the high computational cost of fine-tuning large foundational models by updating only a small set of parameters, rather than the full original network. A prominent class of PEFT methods is adapter-based: small trainable modules are added to a frozen model. Early work inserted bottleneck adapters between layers to enable task-specific tuning without altering original weights (Houlsby et al., 2019; Rebuffi et al., 2017); later designs placed adapters in parallel to existing layers for improved adaptation (Ie et al., 2022a). Recent work have explored structured parameterizations, e.g., Kromecker-factored matrices (Mahabadi et al., 2021). Li et al. (2024) employ block-specific adapter designs, dynamic parameter sharing, and mixtures of experts to improve efficiency and generalization. At the matrix level, LoRA and its variants constrain weight updates to a low-dimensional subspace for memory and compute-efficienl tuning (Hu et al., 2022; Liu et al., 2024). Indeed, He et al. (2022a) show that many PEFT methods can be viewed through a unified lens of adapter.
Beyond architectural modifications, other PEFT strategies focus on minimizing the number of updated weights directly. These include sparse update methods (Guo el al., 2021; Sung et al., 2021; He et al., 2022b), which identify and tune only the most critical parameters. Recent work has even explored extremely low-precision adapters through quantization (Jie et al., 2023), demonstrating that 1-bit adapters can rival or surpass other PEFT strategies in both parameter efficiency and performance.
LoRA methods and insights. Among PEFT techniques, LoRA-basod methods have emerged as particu- larly prominent due to their simplicity, inspiring a wide range of follow-up studies.
Several works aim to improve LoRA's design. Some focus on alternative initialization schemes. PiSSA (Meng et al., 2024) and CorDA (Yang et al., 2024) leverage spectral decompositions of the pretrained weights to initialize LoRA modules more effectively. Shuttleworth et al. (2024) observe that LoRA introduces novel singular directions absent in full fine-tuning. Building on this, LoRA-Null (Tang et al., 2025) initializes adapters in the nullspace of pretrained activations to reduce forgetting. Other approaches propose architoctural modifications. DoRA (Liu et al., 2024) decomposes pretrained weights into basis and scaling components and applies LoRA on the basis. VeRA (Kopiczko et al., 2024) further simplifies this by fixing both A and B to
2

---

Published in Transactions on Machine Learning Research (12/2025)
random bases and learning only scaling coefficients. RandLoRA (Albert et al., 2025) aggregates multiple VeRA-like components to achieve higher-rank updates. HiRA (Huang et al., 2025) follows different, route, applying element-wise wisee multiplication between the LoRA module and the pretrained weight. These methods are all motivated by structural properties of the pretrained weights.
A parallel line of work investigates LoRA's 's learning behavior. Hayou et al. (2024b;a) analyze how imbalanced initialization affects feature-level dynamics during training. Zhu et al. (2024) report an asymmetry between the down- and up-projection matrices induced by standard initialization, which motivates subsequent variants such HydraLoRA (Tian et al., 2024) and MALoRA (Wang et al., 2025). We refer the reader to Mao et al. (2025); Han et al. (2024) for more comprchensive overviews of LoRA and its many variants.
Our method differs from prior architectural improvements in that it also analyzes the input features to the target layers, rather than focusing solely on the pretrained weights. Drawing inspiration from studies on LoRA's learning behavior, our approach introduces a feature-aware projection objective that optimally preserves information in the input representation before applying the low-rank updale.
3 IPA: cotimaation-Reconstuutiiii Input Projection for Adaptation
In this section, we begin by motivating the framework through empirical observations of standard LoRA training (Sections 3.1 and 3.2). Next, we present the IPA framework (Section 3.3), and finally provide a concrete instantiation (Section 3.4), which we evaluate in Section 4.
3.1 Preliminaries: LoRA
LoRA adapts a pretrained transformer-batt model to a downstream task defined by a loss function L over a dataset D. The pretrained model contains K weight matrices denoted by {W()) omit. the layer index when not required. Each rreted weight maárix W E Rd Xdm defines linear map fw: Rdd + z = Wz Edou, which transforms an input feature vector æ into an output feature vector z. Here, di and dout denote the input and output dimensions of the layer, respectively.
fF=1- For simplicity, we
To adapt a target layer, this linear map fw is augmented with two additional learnable maps: SA: x E 1R + = Ax E R,, fB: 2n E IR' + BIn E Rdoat, where A E Rgd and B E RR x are low-rank matrices with a maximum rank < max(din dout). In is the hidden feature projected by the mapping Sa. Adaptation proceeds over T steps of gradient descent, with = 0 denoting the initial state. We denote the matrices at step by At and B, The modified forward pass of each target layer becomes
z == + ABd, (Ja, = Wx + AB,
The elements of Ao are drawn from a zero-mean Gaussian (or uniform) distribution and Bo = 0, ensuring W + ByAA = W while remaining trainable. The pretrained matrix W remains frozen, and the positive scalar À rescales the low-rank residual update. In the original LoRA formulation, X e, where a 0 is a tunable hyperparameter. Training LoRA thus implics computing gradients only for A, and Be:
vn,C= AT
VA = AB, TV V.LxT,
(2)
leaving the pretrained weight unchanged.
3.2
Asymmetric Behaviors in LoRA
While LoRA has been widely adopted for efficient fine-tuning of large pretrained models, we observe a notable asymmetry between its two projection matrices: the down-projection matrix A primarily serves to compress input features into low-dimensional subspace, whereas the up-projection matrix B plays the critical role of recombining those features to adapt the final model outputs. Notably, tuning B alone while keeping A fixed or randomly initialized often yields performance comparable to tuning both. This suggests that B is mainly responsible for adapting the output, whereas A serves as a feature projector.
To empirically illustrate this asymmetry, we conduct an adaptation experiment across multiple tasks. Following Huang et al. (2023), choose the fhw-shot adaptation setting on the BIG-Bench Hard benchmark (BBH;
3

---

Published in Transactions on Machine Learning Research (12/2025)
Bodleen Expressons
FormalFFallacie Geoomenncc Shapes
11: Logical Deduction (3 objects)
16: Penguins In Table
21: Sports Understanding
26: Web Lies
2: Causal Judgement
12: Movie Recommendation
17: TeaaionigAAhin About Colored Objects
22: Temporal Sequencee
22: Word Sorting
Date Unddrsranding
Hyperbaton
13: Multistep Arthhmtte
18: Ruin Names
23: Tracking Shuffled Objeects objectsl
4: S: Dyck Languages
Disamiiguation QA
9: Logical Deduction (5 abjerts)
14: Navigate
19: Salient Translation Eroor Detection
24. Tracking Shufflod Objects objects)
10: Logical Deduction (1 objects!
15: Object Counting
20: Snarks
25: Tr Tracking Shuflled Objects objects)
Inii
Task-Init, LoRA-A
Task
1.00
0.75
0.50
0.25
Covinne Siillrry
13
0.00
0.25
-0.75
-1.00
(b) Task-Tas LoRA-A
(c) Pask-Task, LoRA-E
(d) Task-Task, Full FT update
Figure 1: Cosine-similarity matrices for LoRA and full-fne-tune updates on 27 BIG-Bench Hard tasks. (a) shows the similarity between each trained LoRA-A vector and its initialization; panels (b)-(d) show pairwise mask- similarities for LoRA-A, LoRA-B, and full fine-tune updates, respectively. The LoRA-A vectors remain close to their shared initialization (a) and vary little across tasks in (b), while the dask-dependent patterns in LoRA-B (c) closely match those from full fine-t fne-tuning (d).
Suzgun al., 2023), which comprises 27 diverse tasks. We use Flan-T5 (Chung et al., 2024) as the base pretrained model. For each task J, we either fully fine-tune the pretrained model or learning LoRA adapters set of target layer A for fixed number of steps T, reaching zero training loss in both cases. All LoRA adapters are initialized with the same random seed across tasks, ensuring that A = A for every target layer E A. This facilitates comparison of the learned LoRA matrices across tasks.
To analyze inter-task similarity, across all target layers A, flatten and concatenate full-fine-tnne updates and the trained LoRA matrices, yielding vectors for each task j: OA.,- = lleGs vec(A On.j = llee veo vec( (B/)) and Adw, = llee vec vec (w//.) W()). Figure then presents simil--immllaity matrices for two cases: in panel (a) ("Task Init, LoRA-A") we compare each trained vector OA.j to their common LoRA-A initialization; panels (d)-(d) TaTa LoRA-A", LoRA-B and Full FT, respectively) show pairwise similarities cos(f OA.)), cos(0 OBJ), and cos( (OAAw,w, A0w,).
Remarkably, Figure shows that matrices are still pretty similar to their initialization, while Figure 1b is largely uniform across tasks. This indicates that the learned A matrices undergo little change during adaptation and capture minimal task-dependent variation. In contrast, Figures 1c and 1d reveal nearly identical block structures, suggesting that the task-specific information recovered by full fine-tuning is almost entirely absorbed by the B matrices.
Implications. These findings indicate that the down-projectio matrix A in vanilla LoRA operates primarily random feature projector, rather than encodes the task-specific distinctions. Recent theorctical and empirical studies of LoRA (Hayou et al., 2024b; Zhu el al., 2024; Tian el al., 2024; Zhang el al., 2025) arrive at similar conclusions, showing that standard LoRA initialization induces pronounced asymmetries in both learning dynamics and representational behavior. Consoquently, replacing this random feature projector with a more expressive, task-aware mapping could yield richer intermediate representations and improve adaptation performance.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Output Feature Updote
A
A
fB
fB
Hidden Feature
h
T.
fA
P
Input Feature
æ
Reconstruct
(a) LoRA
(b) IPA
Figure 2: Schematic of IPA vs. standard LoRA. The gray arrows denote mappings between two vector feature spaces. In standard LoRA, an input feature æ is projected to low-dimensional space by SA and then lifted back by fa to yield the update Az. IPA introduces two pretrained projectors, P and Q, enforcing that the hidden feature can reconstruct x; adaptation time only P retained.
3.3 The IPA Framework
We reinterpret the adaptation scheme by introducing general feature projection function P: Rd - Rdd and write
= fw) + Afa, (P(x)) = Wx XB,P(x), B, = 0.
where maps the input into a hidden feature 2n P(x) Rdh Therefore, full fine-tuning corresponds to P being the identity. It thus natural to seek closer approximation to full fine-tuning through more effective input compression.
input projection. When d₁ < dm, the projection P must compress x, which risks discarding task-relevant information if it is not chosen in an input-aware manner and the inputs in a subspace with larger intrinsic dimension. Standard LoRA initializes P as a random linear map, thus ignoring the input distribution. address this, we instead seek P (and a complementary decoder Q: dh pdo) that minimize the reconstruction error:
min P.O.
ExnnEep(()lll 12 E||², where ã = Q(P(x))
(3)
This objective encourages P to preserve as much information from the original imput as possible, as measured by the L reconstruction loss. Figure contrasts IPA with standard LoRA, highlighting the additional optimization of the projector.
Forward-only pretraining of projector. Eq. (3) corresponds preciscly the objective of an autoencoder. One could therefore imagine training it with either linear nonlinear functions for P and Q. However, doing so for each modulated layer via backpropagation is impractical: the loss is difficult to integrate into the adapter training pipeline and incurs significant computational overhead compared to LoRA. Instead, we propose to learn the projector in forward-only manner.
3.4 Instantiation: Linear Case
To instantiate the framework in practice, must specify (i) the distribution of features used to pretrain the projector P, (ii) the form of the projector P, and (iii) the algorithm used to pretrain it.
Pretraining distribution. We pretrain P using target-domain hidden representations. Concretely, we pass training tokens through the frozen pretrained model and collect the resulting hidden states, forming a pretraining set X [+1 R Nxdi, This ensures that X reflects both the model's internal feature and the farget-domain data distributions.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Projector architecture. To preserve LoRA's inferenco-time efficiency, restrict P and its decoder Q to linear maps defined by shared matrix U E xdu;
P(x) = Uz (x)a) = U Th- Th-
In this setting, solving Eq. (3) is equivalent to performing PCA (Hotelling, 1933), by extracting the top-dh cigenvectors of the empirical covariance E *
Pretraining algorithm. Full PCA over all hidden states is infeasible due to storage and compute costs. Instead, we adopt incremental PCA (IPCA; Ross et al., 2008), which processes feature mini-batches sequentially and updates low-rank approximation of E. Alternatives such as the generalized Hebbian algorithm (GHA; Sanger, 1989) also approximate principal components, but we found TPCA both more efficient and slightly more accurate in practice (see Section 4.4 for the ablation study).
Default Configuration. Unless otherwise specified, we use target-domain hidden representations as input, a linear projector, and IPCA for pretraining. All main experiments adopt TPCA, and TPA refers to this implementation unless noted otherwise. The projector U can optionally be refined by backpropagating the task loss, allowing it to adapt jointly with the residual weights and align more closely with the downstream objective. We analyze the effect of projector fine-tuning in Section 4.
4 Experiments
We evaluate IPA on both standard language- and vision-centric benchmarks, comparing it with leading PEFT baselines that rely on low-rank updates.
4.1
Experimental Setting
4.1.1 Tasks, Datasets, Base Models, and Evaluation Protocol
Language Tasks: Instruction Following We follow the adaptation protocol for instruction following proposed by Hu et al. (2023), where the adapted model must generate a response in the expected format from a set of given options. Each question is wrapped in a predefined natural language template with an instruction, and the training example is appended with the desired answer.
We use the commonsense-178 sense reasoning lasks. We study four recent. strong pretrained large language models of moderate size: LLAMA-2 7B (Touvron et al., 2023), LLAMA-3 8B (Grattafiori et al., 2024), QWEN-2.5 7B (Qwen et al., 2024), and GEMMA-3 4B (Gemma Team et al., 2025). We only use the language model part if the model is multi-modal. modal. All models are adapted once with the same dataset for 3 epochs and evaluated on the corresponding test splits for each reasoning task.
dataset, which contains 170,000 examples drawn from cight common-
Vision Tasks: Open- Vocabulary Image Classification We assess our method on open-vocabulary classification using the VTAB-1k benchmark (Zhai et al., 2019), which includes 19 tasks/ fasks/datssets categorized into three groups: NATURAL, SPECIALIZED, and STRUCTURED. VTAB-1k uses exactly 1000 labeled examples per task for adaplation; the original lest splits remain unchanged."
We use the base variant of SIGLIP-2 (Tschannen et al., 2025) as the pretrained vision-language model. For each class, we construct a text embedding by inscrting the class name into the prompt "a photo of a [CLASSNAME]* " and encoding it with the model's text encoder. All embeddings are padded to the model's maximum sequence length. This prompt formulation is fixed across methods and orthogonal to the PEFT approach.
We adapt only the vision encoder, training it with standard cross-entropy loss over logits computed with similarity scores between image and text embeddings. Logits are computed following the procedure used in
We regenerated VTAB-1k using the code provided in Zhang al. (2022), converting all examples lossless format instcad of the lossy JPEG compression used in the original work. After this change, our zero-shot performance for StGL.IP-2 matches the results reported in the model's original paper.

---

Published in Transactions on Machine Learning Research (12/2025)
SIGLIP-2, for each image-class pair. Adaptation and evaluation performed independently for each task. As in Zhang et al. (2022), we evaluate models every 10 epochs and report the best test performance observed over 100 training epochs.
4.1.2 Main Baselines
Our goal to isolate and quantify the effect of the input projector introduced in IPA, w.r.t. random input projector-basod mothods. We therefore compare against two strong basclines from the same reparamcterization PEFT lamily: LoRA (Hu al., 2022), which injects low-rank residual update with a randomly initialized down-projection fA. DoRA (Liu et al., 2024), which decomposes each pretrained weight into magnitude and direction: W ||W||
W [[W]] where ||] is the magnitude and [[] is the unit-norm direction. Standard LoRA is then applied only to the direction, while the magnitude |[W]| is fine-tuned jointly. For each method we evaluate two variants of the input projector: fixed (no gradient updates, denoted x) and trainable (finettuned during adaptation, denoted v), aligning with IPA's variants.
W
4.1.3 Hyperparameters
Optimization. To ensure a fair comparison across adaptation methods, we fixed the optimizer hyperparam- eters and learning rate scheduler in all experiments except the base learning rate. We employ Adam (Kingma & Ba, 2015) with a linear warm-up schedule: the learning rale is ramped up over the first steps, then decayed linearly. The base learning rate remains unchanged regardless of whether Seature-projectoo fine-tuning is enabled. The same batch size applied in each benchmark.
For each baseline, we align the recommended learning rates for baselines with those from Tiu et al. (2024) for LLAMA-2 and LLAMA-3. We only deviate from these defaults when a more effective rate is identified. We choose a performing learning rate for more recent QWEN-2.5 7B and GEMMA-3 4B.
We report the detailed hyperparameters in Appendix A.
Adapter configuration. All methods share identical essential adapter settings to isolate differences coming solely from feature projection. Concretely, for language benchmark, we use identical hidden dimension d, = 32 for the query, key, and value projections, as well as for the up/down projections in each MLP block, following Hu et al. (2023); Liu al. (2024). For vision benchmarks, we restrict adapters the query/ /vely//alue projections with d, 8, per standard practice in vision transformers in Zhang al. (2022). Therefore, the only methodological distinction lies in how feature projectors are trained. The scaling factor is fixed for all methods.
For Instruction-followee tasks, IPA uses random 10% subset of the training data to construct the projector pretraining feature set. An ablation study examining the impact of this choice is presented Figure 4b. For opea-vocabulary classification, we leverage the full training set for each task, as each contains only 1000 examples.
4.2 Main Results
IPA projector improves performance over random projection. Tables 1 and 2 summarize our accu- racy results on the cestruction-folloiing benchmark and the veceb-oocy classification tasks, respectively.
On the inslloig benchmark at hidden dimension d, = 32, IPA outperforms both LoRA and DoRA across most configurations and base models. For example, in Table 1, on LLAMA-3 8B without projector fine-tuning (25M trainable parameters, 0.31% of the model), IPA achieves average accuracy of 85.6%, outperforming LoRA (85.0%) by 0.6 points and DoRA (84.7%) by 0.9 points. Even with projector fine-tuning (57M parameters, 0.70%), IPA still leads with 85.9%, compared to 85.5% for LoRA and 85.1% for DoRA. Similar gains are observed across other base models, yielding an average gain of 1.5 points.
On the open-vocabulary classification benchmark (Table 2), at hidden dimension d, 8, IPA reaches 73.7% gevul-level macro average accuracy without projector fine-tuning, surpassing LoRA by 3.0 points and DoRA by 2.8 points. With projector fine-tuning, performance improves to 76.5%, 1.8-point gain over both baselines.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Comparison of instruction-folloiing answer accuracy (%) on 8 commonsense reasoning benchmarks across multiple base models. All methods are compared the configuration with (V) and without (X) projector finetuning. We highlight the best and the second best scores under the same projector finetuning setting.
SocialIQA
HellaSwng
WinoGrande
ARC-
ARC-challenge
OpenbookQA
Proj.
Trainable Params (%)
BoolQ
PIQA
Avg.
Base model
Method
FT
LoRA
OM. (0.41%)
60.5
78.7
74.5
76.3
75.1
82.8
66.1
76.8
73.8
DoRA
X
28.9M (0.43%)
58.0
82.0
33.5
12.8
42.1
64.9
43.9
68.4
50.7
IPA (Ours)
28.0M (0.41%)
71.7
83.2
80.0
89.0
82.0
84.8
70.1
79.0
80.0
LLAMA-2 7B
LoRA
56. 1M (0 (0.83%)
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
DoRA
/
57. 07.0M (0.84%)
71.8
83.7
76.0
89.1
82.6
83.7
68.2
82.4
79.7
IPA (Ours)
56.1M (0.83%)
71.1
84.4
80.9
90.5
82.7
85.6
71.5
81.4
81.1
LoRA
25. 25.2M (0.31%)
73.6
88,1
80.3
95.0
85.2
90.4
80.1
87.4 4
85.0
DoRA
x
26.0M (0.32%)
74.3
87.9
79.7
95.3
84.2
90.3
79.5
86.2
84.7
IPA (Ours)
25. 25 (0 (0.31%)
74.8
88.6
81.1
95.4
85.6
91.7
79.9
87.8
85.6
LLAMA-3 8B
LoRA
56.6M (0. 70%)
75.4
88.6
80.7
95.4
86.2
91.2
80.1
86.1
85.5
DoRA
57. 57.4M (0.71%)
75.3
89.3
80.8
95.3
85.8
89.9
79.3
85.6
85.1
IPA (Ours)
56.6M (0.70%)
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
LoRA
24.3M (0.32%)
62.8
89.3
79.9
94.6
83.1
95.9
88.6
91.4
85.7
DoRA
x
25.1M (0.33%)
62.0
89.8
78.6
94.6
83.0
96.1
88.9
89.8
85.3
IPA (Ours)
24.3M (0.33 (2.3%))
73.3
90.0
80.2
95.0
85.2
95.8
88.8
92.4 4
87.6
QwRN-2.5 7B
LoRA
54.1M (0.71%)
63.5
89.8
79.5
95.4
85.9
95.9
88.3
92.2
86.3
DoRA
/
54.9M (0.72%)
74.5
90.0
80.2
95.4
85.9
95.7
87.7
91.8
87.6
IPA (Ours)
54.1M (0.71%)
74.5
90.0
79.7
95.3
85.5
96.2
88.7
92.0
87.7
LoRA
21.4M (0 40%)
69.3
84.4
78.2
90.6
80.3
89.5
76.4
82.0
81.3
DoRA
x
22.0M (0.51%)
69.1
84.2
77.9
91.0
80.5
89.4
78.1
82.2
81.5
IPA (Ours)
21.4M (0 (0.4
68.7
85.0
78.5
90.0
81.5
90.3
78.0
84.4
82.0
GEMMA-3 4B
LoRA
46. GM.6MM (1.07%)
70.3
86.0
79.7
93.1
82.3
89.7
79.7
84.4
83.1
DoRA
47.3M (1.09%)
70.6
85.3
80.0
92.9
82.8
90.0
77.6
85.4
83.1
IPA (Ours)
46.6M (1.07%)
69.8
86.3
78.8
93.4
83.3
90.7
80.3
86.0
83.6
Liama-2 7B
Liama- 88
Qwen-2.5 78
Gemma-3 48
85
50.1%
86.5
87.5
83.5
80
35. params -0.4 pts
Average Score (N)
75
86.0
87.0
93.0
70
85.5
86.5
$2.5
65
60
86.0
82.0
LoRA IPA
85.5
81.5
50
45
84.0
85.0
81.0
0
20
40
60
20
40
60
0
20
40
60
20
40
60
Leamable Params (M)
Figure 3: Comparison of IPA with baselines in both settings, with (O) and without (8) finetunable feature projection on the commonsense benchmark. The dotted red line marks the highest baseline performance.
We also note that, across both benchmarks, LoRA remains competitive with DoRA, as the two methods yield similar scores most settings when the learning rates are aligned.
IPA suffers less from fixing input projectors. The performance degradation from fixing the input projectors appears be less pronounced for IPA in several cases. We showcase this Figure 3. For instance, with TLAMA-2 7B, fixing the IPA projectors results in only 1.1-point drop, compared to 3.8 points for
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Top-1 accuracy (%) on 19 VTAB-1k open-vocabulary classification tasks with the SIGLIP-2 base model. Methods are evaluated with (V) or without (X) projector finetuning. "Vision QV FT finetunes the query/value projection layers of the vision encoder, while "Vision Full FT finetunes all its parameters. We highlight the best and second best scores under the same setting. We report per-group averages (*G1/G2/Ga Avg."), the Maero Aug. (mean of group averages), and the Micro Aug. (mean over all tasks). Significonce: for each setting (w/o and w/ projector finetuning), we run paired sign tests (two sided, Ho: p ) comparing IPA against each baseline (LoRA, DoRA) over the 19 tasks, The "Sign p" column in row reports the p-value for IPA the method in that row (IPA rows use "_").
GrouD 1: NATURAL
Gooup SPECIALIZED
Group & STRECTURID
Poo F
Chitecch
CI 100
SHm1007
SWINN
Are
Camnloon
EmnSAT
Bestold
Resinogaai
02 Ave.
Cloon-Doorr
Clooe-Duee
48pe
ANORBB- Anin
&OORD-EW Elev
Macro Arr.
Mi3o Ang.
Methodl
Parame (%)
Zon-oliad
B (0.00%)
54.4
73.9
63.0
64.1
94.99
61.2
28.0
7000
50.9
00.11
62.88
5.0
30.7
27.9
20.0
17.0
4.2
6.4
4.8
5.2
10.4
12:0
40.5
86.7
Viston QV FT
14.231 3.85)
94.2
78.3
80.2
96.2
94.33
03.6
98.2
55.2
85.3
96.5
91.0
74.8
56.9
85.0
040.2
65.5
85.1
58.2
52.2
37.2
43.6
02.5
78.2
75.5
Viston Pall FT
92.93 (24.8%)
94.8
98.3
67.7
50.2
75.1
57.0
83.0
29.7
74.7
LoRt
0,133 (0.039%%
30.0
75.4
64.6
81.7
65.9
17.
86.0
DoltA
(0.04455)
89.5
05.4
90.3
72.5
34.5
IPA (Ours)
0.15M (0.03955)
93.3
51.7
77.7
95.3
05.1
663.2
90.7
86.5
81.5
05.7
87.3
73.3
64.5
53.5
50.7
29.2
81.4
75.0
25.1
15.8
38.6
51.40
13.7
69.5
LoRA
0.2901 (0.079%)
994.8
90.8
75.4
95.8
05.2
65.0
91.4
85.9
82.4
96.1
86.0
74.0
85.1
91.8
58.5
81.7
83.1
76.8
38.4
18:2
38.00
51.9
76.3
71.5
0.019
DOHA
12,8331 (0.0525%)
94.5
81.1
78.1
96.8
06.2
65.6
91.4
55.7
83.5
96.01
87.6
76.1
55.3
91.5
00.6
25.3
84.5
78.4
83.1
17.0
57.1
02.0
70,3
71.5
0.019
IPA (Oras)
0,2534 (0,07995)
94.8
81.3
63.6
74.4
85.6
70.5
LoRA and a substantial 29.0 points for DoRA. On QWEN-2.5 7B, the drop is only 0.1 points for IPA, versus 0.6 and 2.3 points for LoRA and DoRA, respectively.
In Figure 3, the red dashed lines indicate the average accuracy of the best bascline with projector finctuning. We observe that IPA without projector finetuning matches exceeds this baseline in 3 out of models, while using less than half the tunable parameters. For instance, on LLAMA-2 7B, it requires 50.1% fewer parameters yet surpasses LoRA and DoRA by 3.5 and 0.3 points, respectively. On LLAMA-3 8B, it achieves comparable gains with 55.5% fewer parameters, outperforming LoRA and DoRA by 0.1 and 0.5 points.
These results further underscore the benefit of our input projector's information-preeeriig property.
4.3
Extended Comparison with Additional Baselines
We further broadened our evaluation on the commonsense reasoning benchmark using LLAMA-3 8B (cf. Section 4.1).
Method
Avg.
Projection
Proj. FT
Scaling À
Avg.
PiSSA
84.6
Random orthogonal
0.25
81.7
OLoRA
84.4
Random orthogonal
0.25
V
d.n
du
82.2
CorDA
80.8
LoRA
X
2
85.0
RandLoRA
83.4
IPA (Ours)
0.25
85.6
VeRA (r=512)
80.1
LoRA
85.5
Random orthogonal
0.25
85.5
DoRA
85.1
Random orthogonal
0.25
V
dm
dn
83.1
IPA (Ours)
85.9
LoRA
2
85.5
IPA (Ours)
0.25
85.9
Table 3: Comparison with additional PEFT methods.
Table 4: Comparison of projection lypes commonsenise.
Comparison with additional baselines. In Table 3, we evaluated IPA against a broader set of baselines, including mothods that rely on random projections such as VeRA (Kopiczko et al., 2024) and RandLoRA (Al- bert al., 2025), as well as approaches based on spectral or low-rank matrix decompositions such PiSSA (Meng et al., 2024), OLoRA (Büyūkakyūzz 2024), and CorDA (Yang al., 2024). Within this
9

---

Published in Transactions on Machine Learning Research (12/2025)
expanded set of techniques, IPA attains the highest average performance. All models use hidden dimension da = 32, except VeRA. Additional per-task results and learning rate are provided in Table 6 in Appendix B.
Comparison with random orthogonal projection. In Table 4, compared IPA with random orthogonal projections under matched hidden dimension (d,, = 32) and scaling configurations. Since random orthogonal projections reduce the norm of projected activations, we introduce an additional scaling factor to compensate for this effect and ensure a fair comparison with the norm-preserving PCA-based projection. In both scaling setups, random projection remains consistently below the performance of LoRA and IPA. See Appendix B, Table for dctailed results.
4.4 Ablation Studies
The following ablations also use LLAMA-3 8B on the inslruction-following fine-tuning lask.
Projector pretraining algorithm. As introduced in Section 3.4, compare two online algorithms for estimating the top principal components: IPCA and GHA. Both optimize the same autoencoding objective Eq. (3). Table 5 reports results with and without projector fine-tuning. Across all settings, IPA-IPCA achioves higher downstream accuracy and converges more reliably than its GHA-based counterpart, making it our default choice.
Table 5: Comparison of instruction -orrlucttioooooooiinggg answer accuracy (%) commonsense reasoning benchmark.
between IPCA and GHA algorithms on
WinoGrande
ARC- chalchale
OpenbookQA
SocialIQA
HellaSwag
ARC-easy
BoolQ
PIQA
Avg.
Method
Proj. FT
IPA-IPCA
74.8
88.6
81.1
95.4
85.6
91.7
79.9
87.8
85.6
x
IPA-GHA
73.3
88.1
80.3
95.0
85.1
91.0
80.0
87.2
85.0
IPA-IPCA
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
IPA-GHA
74.9
89.3
81.3
95.8
86.3
90.4
80.1
86.2
85.6
Projector pretraining set size. The commonsense-170k dataset is large enough to investigate how the size of the projector pretraining set affects downstream performance. In Figure 4b, we pretrain the projector on randomly shuffled subsets ranging from 1% to 100% of the data, using a fixed seed for reproducibility. We select the first X% of examples from the shuffled split. Although performance generally improves up to around 10% of the data, observe miligated results beyond that point, which is likely due to variance in sample composition and/or randomized version of IPCA. Pretraining the feature projector on the full feature set takes roughly 1.7 hours on a NVIDIA H100 GPU, which is about ten times longer than using a 10% subset (=:10 minutes). Note that adapter tuning on the full dataset requires about 5 hours for 3 epochs. Despite the substantially lower cost, the full dataset yields negligible or no accuracy improvement (and occasionally slight, degradation due to variance), so we conclude that 10% is a practical sweet spot for efficient pretraining on commonsense-1700 dataset without sacrificing downstream performance.
Wall-clock time and memory consumption. With the same setup described above, pretraining the projector 10% subset adds only about 10 minutes to the total fine-tuning time of roughly 5 hours for 3 epochs single NVIDIA H100 GPU, resulting in runtime comparable to LoRA. In contrast, DoRA requires around 10 hours under the same configuration. Peak memory usage is dominated by the fine-tuning phase and remains similar across methods (82 GiB for LoRA and IPA, 81 GiB for DoRA). To fairly compare IPA and LoRA with similar computational budget, we also compared IPA (with 10%-data pretraining) to LoRA trained for 3% more steps, yielding average scores of 85.9 and 85.0 respectively, showing that the small pretraining cost consistently translates into a performance gain. Full ablation results are given in Appendix B.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Projected feature dimension. In ablation study, vary the hidden dimension dN for IPA, LoRA, and DoRA, while keeping the learning rate, pretraining set size, and scaling ratio fixed. Figure 4a shows a characteristic bell-shaped curve for both IPA and LoRA: accuracy falls off steeply at very low dimensions, reaches a maximum over an intermediate range, then gradually declines as d, increases further. Importantly, d₁ = 16, whereas LoRA's 's
IPA is more robust than LoRA: d, = 8, it matches LoRA's performance accuracy drops sharply. DoRA maintains a relatively flat performance profile across all tested dimensions but underperforms IPA once d, 8. For intermediate dimensions (d4 16, 32, 64), LoRA still outperforms DoRA. Detailed task results provided in Appendix B.
86.5
87.5
(PA proj. FT) IPA/ FT)
86.0
87.0
86.5
Accuracy (%6)
85.0
Accuracy (9g)
86.0
84.5
85.5
85.0
83.5
LoRA (w/ Proj. FT)
84.5
IPA (w/ Proj. FT)
84.0
82.5
DORA (w/ Proj.
63.5
16
32 64
96
10
50
100
Hidden Dimension dp (a)
Projector Pretraining Size (%)
(b)
Figure 4: Average accuracy of LLAMA-3 8B models fine-tuned on commonsense benchmark with (a) varying hidden dimension d, for IPA, compared to LoRA and DoRA, both with input projection fine-tuning and (b) IPA (with projection fine-tuning or without x) with varying percentage of the training dataset obtain the projection pretraining feature set.
5 Discussion and Conclusion
In this paper, we presented the TPA framework, which addressos parameter-effiiiet adaptation by focusing on improving input-feature projection. We augment the feature projector with an input-feature-awaat optimization objective, allowing it to learn meaningful projections without requiring gradient backpropagation. In our instantiation, employ batched PCA algorithm for unsupervised feature projection, demonstrating that this simple approach can be trained efficiently and effectively.
Through experiments on both language and vision-language benchmarks, we showed that IPA consistently outperforms existing PEFT methods that rely on random feature projections when the setting is aligned. These results confirm that incorporating data-driven projections yields more expressive and adaptable models while maintaining low additional parameter cost.
Future work could investigate alternative projection techniques, novel optimization objectives for the projector and the incorporation of other backpropaggtiin--in -free unsupervised or self-supervised learning methods. Such enhancements may further boost adaptation performance while retaining the computational efficiency of our framework.
Acknowledgment
This work was granted access the HPC resources IDRIS under the allocation A20-10001111 made by GENCI.
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez Opazo, Anton van den Hengel, and Ehsan Abbasnejad. RandLoRA: Full rank parameter-effiiien fine-tuning of large models. In The Thirleenth International Conference on Learning Represenialions, ICLR 2025, Singapore, April 24-28, 2, 3,
2025. OpenReview.net, 2025. URL https////ppppaeaaaaa///oooaaaaaa set//forumIdd-thheooTe
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji et al. On the Opportunities and Risks of Foundation Models. arXiv e-prints, art. arXiv:2108.07255
August 2021. doi: 10.48550 10.48550/arXiv.
Kerim Büyükakyū... OLoRA: Orthonormal low-rank adaptation of large language models, 2024. URL
attps://arxivv.rr//aae http////a//200.../// 01775. 9
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen et al. Scaling eestruction-fietuuee language models. J. Mach. Learn. Res., 25: 70:1-70:33, 2024. URL https:// http::///..rorrr/././/o//oo org/ p2ppps///s////23 000008800..00. html. 4
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Gcoffrey Cideron, Jean-basticn Grill, Sabela Ramos ct al. Gemma 3 Tochnical Report. arXiv e-prints, art. arXiv:2503.1977 19786, March 2025. doi: 200.500//Xii..53..11.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Authony Hartshorn, Aobo Yang et al. The Llama 3 Herd of Models. arXiv e-prints, art arXiv:240.221783, July 2024. doi: 10.48550/are /ar550////iv../. 2407.21783
Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efffiient transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLL 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 4884 4896. Association for Computational Linguistics, 2021. doi: 10. 111188//V///21AAALLLOOCCCC URL https://doi 0rg/10.0/ 18653/v1/2021 acl 378. 2
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang, Parameter-effiiient fine-tuning for large models: A comprehensive survey, Trans. Mach. Learn. Res., 2024, 2024. URL https://openreview..eett forumiid-1sscS8b6e) 1,
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization LoRA finetuning dynam- ics. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouser, BC, Canada, December 10 15, 2024, 2024a. URL http://papers: wiw..nin..... cwtppper.ppppppp.eeeeee htc//aperffille/ppeeeeeeeeeeeeee hash/d438) www.7770000006666600eeeeeoo. www.:=8229-bbractee Cecsstrcct-Coffreeeee html. 3
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. Tn Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL
2, 3, 4
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor unified view of parameter-cfficiont transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL https: //oppp/reieee..attf//rmm/m/////a www.nneet/oormmddddddddeede 2
and Graham Neubig. Towards a
12

---

Published in Transactions on Machine Learning Research (12/2025)
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. SparscAdapter: An easy approach for improving the parameter--ficiencc of adapters. In Yoar Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2184-2190. Association for Computational Linguistics, 2022b. doi: 10.18653/VV /V1/ 2022. FIN.ININNNNGSSI BENIGGS-EMMP..IIE 160. URL https://doi....
11653/111/2122 http:///www2222...innnn-eeeeee wiwdings-ennlp...... 160.
Harold Hotelling. Analysis of complex of statistical variables into principal components. J. Ed. Psych., 24: 417-441, 1933. 6
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-cfficiett transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790 2799. PMLR, 2019. URI. http:://pproceeddgggeg mlr. wtt.:rerr...eee.ee...m.mmeereeee hht/hhoblbba..... html. 1,2
Machine
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Represenlations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.ned 2022. URL
https:///penreee http::///ppprreeeeeeeetttt////e/ (tw.rruummdddn2eeeeaFF... 2,
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lec. LLM-Adapters: An adapter family for parameter-cffciient fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp- 5254 5276. Association for Computational Linguistics, 2023. doi: 10.18653/ 20/5533/1/222../22/ URL attps://doi. http:::///oo.o/1.////// 100 18653/v1/2028 http://222...mm.mppmmmmmmmme main. 319. 6, 7, 16
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. LoraHub: Efficient 2023. doi: 10.48550/
cross-lask generalization via dynamic LoRA composition. CoRR, abss/330.118 ARXIV 132.V,, URL https: /tt.///o..ioo/// org/10. 48550/ /ari0//r/iv..// 2307 13269.
Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-elficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Represenlations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.ne,, 2025. URL https:// 2,
spenreriew.net/forummP wtt.://ooooumm.ddddat........
Shibo Jie, Haoqing Wang, and Zhi-Hong Deng. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In IEFE/CVE International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 17171 17180. IEEE, 2023. doi: 00.110/1CCCV510000022333333333 URL attps://dde org/10 11109/CCV51000 2023. 01579. 2
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Leurning Represenlations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.ott org/ haaaaba///// /4aa.. 6980. 7
Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki M. Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
wwwpteeuumttieeeee OpenReview.nee 2024. URL https: ptttps:/oppereviiie http://parrrreeeeeeeeee/aaddddd/ htt.:/foorummdddddjjf.dddddddd 2, 9
Minglei Li, Peng Yc, Yongqi Huang, Lin Zhang, Tao Chen, Tong Hc, Jiayuan Fan, and Wanli Ouyang. Adapter-X: X: A novel general parameter-efficien fine-tuning framework for vision. CoRR, abs abs/2406.00 2436003051 2024. doi: 10.48550/ BAB.0//ARXVVV ARXIV.240..0300 URL https: org/ 10. (8550/arX111 http//.wwvvvv......... 03051. 2
Shih- Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Tine Cheng, and Min-Hung Chen. DoRA: Woight-dcoomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nee net, 2024. URL https:/ /ttp:r/////peeevieee mttf:/oorruuti1-aa3cc1111...2 2, 7,
13

---

Published in Transactions on Machine Learning Research (12/2025)
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient, multi-task fine-tuning for transformers via shared hypernetworks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI./IJNNLP 2021, (Volume 1: Long Pupers), Virtual Evend, August 1-6, 2021, pp. 565- 576. Association for Computational Linguistics, 2021. doi: 10. N0.18653/V1/20 2.55://V//222..ALLLLON.. https://doi.oom org/ wtwpr//...1653/111// 116538816/222//// attp:///111211..111111a.aa.. long--log--. 47.
URL
Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on LoRA of large language models. Frontiers Comput. Sci., 19(7):197005, 2025. doi: 10.1007/111744-02--00000
URL https: / //ooii//ooo//oo www.aoii.oooooooeeeeoeooooooeeee org/10.//////// 000007/111000044-00000000
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PISSA: Principal singular values and singular vectors adaptation of large language models. In Amir Globersons, Tester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Newral Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Conoda, December 10 15, 2024, 2024. URL http: sttt::/paperrs nips httppapp__ffiie fol_ffieespppppe wwt.fffeeeeeeeeeaeeeeeee http///pppp///////// htttaaaaaattttteeooeoeaattttttte Abstract-Conte CeaettCCooffeeeee erence html. 2,
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang al. Qwen2.5 Technical Report. arXiv e-prinls, art arXiv:2412.15115, December 2024. doi:
6
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with resid- ual adapters. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Newral Information Processing Sys- tems 30: Anrual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 506-50 516, 2017. URL https:/ hitee://proceeeee ptwp/eedeeggg.../////////o www.ggp...rrp...oo cc/paper/2017/ (apper//01//hshhh haahh/ Abstract. 572201122440000099199966662 abstraattttttccccaaaaataattt... html. 2
David A. Ross, Jongwoo Lim, Ruei-Sung Lin, and Ming- Hsuan Yang. Incremental learning for robust. visual tracking. Int. J. Comput. Vis., 77(1-3):125 -41, 2008. doi: 10.1007/S11223-007000775 URL
https: httrs::/ddoiorr/111 1ttg/10.10 10..100//1122330000000555
Terence D. Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2(6):459-473, 1989. doi: 10.1016/0880 10011/00000000000))9900000000 URL https://dos 1ttg/10.1111
0893-6080 (89) 90044-0. 6
Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. LoRA vs full fine-tuning: An illusion of equivalence. CoRR, abs/2410.21228,, 212228, 2024. doi: 10.48550/ ARXIV 2410..1122 URL
https://doi. org/ 10. attp://rXiv..4111 24....2222........... 2
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Newral Information Pro- cessing Systems 2021, NearIPS 2021, December 6-14, 2021, virtual, pp. 24193 24205,, 2021. URL https: // proceedings.
cc/ wwtpcc//ppppe//////////
Abstracttt..........o. html. 2
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan L. Boyd- Grabbrrr,rrr and Naoaki Okazaki (eds.), Findings of the Association for Compulational Linguistics: ACI 2023, Toronto, Canadaa, July 9- 14, 2023, pp. 13003 13051. Association for Computational Linguistics, 2023. doi: URL https: //doi. org/ 10. 18653/v1/2023 findiiggnnnase 8244. 4
/ttp////2223///1/D// ACL.824.
Pengwei Tang, Yong Liu, Dongjic Zhang, Xing Wu, and Debing Zhang. LoRA-Null: Low-rank adaptation via mull space for large language models. CoRR, abs/ hals//253..0200 020639, 2025. doi: 10.48550/AA AAR.II 2503.02659. URL https: /ttp:://ddoiioeee http:://ooooooo///ee org/10 48550/arXiv 2503 02659. 2
14

---

Published in Transactions on Machine Learning Research (12/2025)
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. HydraLoRA: An asymmetric LoRA architecture for efficient fine-tuning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Newral Information Processing Systems 2024, NeurIPS 2024, Vancouser, BC, Canada, December 15, 2024, 2024. URL http: / //tp::///pppaaaae/ee nips. ww/paper.ppppp./.. files/ hallee//pper///2A// http://2..555551192233326282 htwp://deee.oeoaa-a--aaat-----a-
html. 2,3,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blocher, Cristian Canton- Perrer, Moya Chen, Guillem Cucurull et al. Llama 2: Open loundation and fine-tuned chat models. CoRR, abs/22307.0928, 2023. doi: 10.48550/ (0.48550/ARXIV. ARXIV NEKTV.2307.00 0.288.. URL https: /ttps:://do.... http://do...og//////// /t////0.4500 (8550/aaXi11 2307. 09288.
Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. CoRR, abs Abs/2500.14486, 2025. doi: 10.48550AARXIV.00
URL https: pttps://doi.. org/ 10. 48550/arXiv 2502 14786. 6
Xujia Wang, Haiyan Zhao, Shuo Wang, Hanqing Wang, and Zhiyuan Liu. MALoRA: Mixture of asymmetric
low-rank adaptation for enhanced multi-task learning. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), ), Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mezico, USA, April 29 May 4, 2025, pp. 5609-56266 Association for Computational Linguisties, 2025. doi: 10.18653/ V1/2025. WWW.SFFIIIIIIGAAACICOO
URL ntttps://dlt org/10. 100.3/8663vvv v12//221/2222.2///// findings-naacl. 312. 3
Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-oriented dccomposition adaptation of large language models for task-aware parameter- efficient fine-tuning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advonces in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Voncoweer, BC, Canada, December 10 15, 2024, 2024. URL http::/ nips wc/ppcer_pttceeeeeeeeee (iiles/paper/22222 CenstracttCooffeee
html. 2, 9
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexcy Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet et al. The visual task adaptation benchmark.
CoRR, 0bs/1910.04867, 2019. URL http://arxiss wwwpsiii.orraaaae ha/aabb//9999 19100.90...0 00800.0486... 6
Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, and Jie Tang. Carameter-Effiieen Fine-Tuning for Foundation Models. arXiv e-prints, art. arXiv:2501.13787 Jamuary 2025. doi: 10.48550/
1
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. CoRR, abs/a22/0000 22//222060467 2022. doi: 0044855/ARXIT/ 2206.04673. URL https: //doi. 10g/10. 48550/arXiv 2206 04673. 6, 7
Yuanhe Zhang, Fanghui Liu, and Yudong Chen. LoRA-One: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. In Forby-second International Conference on Machine Learning, 2025. URL https://openrer///
Jiacheng Zhu, Kristjan H. Greenewald, Kimia Nadjahi, Haitz Séez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Forty-forst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nng net, 2024. URL https:/ sttps://openrevieee /pporeeeeeveeeeeeeeeeeeeeeeeeeee net/forum?id txRZBD8tBV. 3,
15

---

Published in Transactions on Machine Learning Research (12/2025)
A Experimental Detail
The hyperparameters used across all models are summarized as follows. For instruction- following tasks, we adopt a batch size of 16, aligning with Hu et al. (2023) and Liu et al, (2024). For open-vocabulary image classification, we use batch size of 64.
Hyperparameters.
We use a learning rate of 3 x 10-1 for LLAMA-2 7B, and 1 x 10-1 for LLAMA-3 8B, QWEN-2.5 7B, and GEMMA-3 4B. For all, LoRA and DoRA use a scaling factor (A = di of 2, while IPA uses 0.25, except for GEMMA-3 4B, where it 0.4. For SIGLIP 2, we apply learning rate of 1 10-3, scaling factors of 2 (LoRA/DoRA) and 0.5 (IPA), with dropout rate of 0.1 across all variants.
B Additional Illustration and Experimental Results
Feature-wise similarity. To complement the global similarity shown Figure la through the cosine similarity of LoRA's matrices with respect to their initialization, we present in Figure 5 a comparison based on feature-wise similarity. We compute the expected cosine similarity of the projected features follows: using the pretraining features & introduced in Section 3.4, we calculate, for each task j, the cosine similarity across the layers A where LoRA is applied:
EvEr Exco ((A)
s(0)) A41))000)
V EeEA Exco (A!0)(00) 1 (1) v ICes Exso (109) T 199,000
On average, the projected features remain moderately similar to the original projected features across tasks.
Average cosine similarity to init
Projection Weight
Projected Features
0.2
0.0
Boolean Expressions
Causal Judgement
Date Understanding
disambiguation QA
Dyck Languages
Formal Fallacies
Geometric Shapes
Logical Deduction (5 objects)
Logical Deduction (7 abjects)
Hyperbaton
Logical Deduction (3 abjects)
Movie recommendation
Multistep Arithmetic
Navigate
Object Counting
Reasoning About Colored Objects
Penguins in A Table
Salient Translation Error Detection
Ruin Names
Sports Understanding nn
Tracking Shuffled Objects (5 abjects)
Snarks
Temporal Sequences
Tracking Shuffled Objects objects)
Tracking Shuffled objects objects)
Web of Lies
Word Sorting
Figure 5: Cosine similarity of LoRA A projections after finc-tuning on 27 BBH tasks, compared to the initial projection, versus the similarity of projected features. To capture the local behavior of the projection, we complement Figure with a measure of feature-wise similarity between the projected representations.
Detailed results of additional baselines. of baselines introduced in Table 3. Table 6 reports the full per-task results for the extended set
16

---

Published in Transactions on Machine Learning Research (12/2025)
Learning Rate
WinoGrande
ARC-challenge
Openbook
SocialIQA
HellaSwag
ARC-easy
BoolQ
PIQA
Avg.
Method
LoRA
1 1 10-4
75.4
88.6
80.7
95.4
86.2
91.2
80.1
86.1
85.5
DoRA
1 1 10-4
75.3
89.3
80.8
95.3
85.8
89.9
79.3
85.6
85.1
PISSA
2 x 10-5
75.2
88.6
81.1
94.8
85.5
89.8
77.1
84.6
84.6
OLoRA
2 x 10-6
73.9
87.6
79.3
94.7
84.9
90.0
79.4
85.0
84.4
CorDA
2 x 10-6
71.6
85.0
77.0
91.6
82.6
85.4
72.0
81.4
80.8
RandLoRA
1 1 10
72.0
86.8
79.7
94.6
85.2
89.3
77.0
82.8
83.4
VeRA (r=512)
4 x 10-3
62.2
85.2
77.8
91.7
80.3
87.2
75.2
81.6
80.1
IPA
1 10
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
Table 6: Detailed results comparison of with additional PEFT methods.
WinoGrande
ARC-challenge OpenbookQAA Avg.
SocialIQA
HellaSwag
ARC-easy
Proj. FT
BoolQ
PIQA
Projection
Scaling
Random ortho.
0.25
62.5
85.9
79.3
93.3
82.7
89.6
78.2
82.2
81.7
Random ortho.
0.25 dm
65.9
86.9
79.5
94.1
84.1
89.1
75.8
81.6
82.2
LoRA
a/r- = 0.25
73.6
88.1
80.3
95.0
85.2
90.4
80.1
87.4
85.0
IPA (Ours)
74.8
88.6
81.1
95.4
85.6
91.7
79.9
87.8
85.6
Random ortho.
0.25
75.1
88.8
81.5
95.60
86.1
90.5
80.0
86.8
85.5
Random ortho.
0.25 de
63.3
87.3
80.5
94.0
85.2
90.0
78.8
85.8
83.1
LoRA IPA (Ours)
a/r =
75.4
88.6
80.7
95.4
86.2
91.2
80.1
86.1
85.3
0.25
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
Table 7: Comparison of projection types on commonsense.
Detailed results of ablation study. Tables 8 and 9 show the detailed results of the ablation studies in Figures 4a and 4b in Section 4.4. Table 10 shows the detailed results on the ablation study about compute budget in Section 4.4.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Detailed results the ablation study on different hidden dimensions.
WranGr
ARC- ARC-challenge
Openbook
SocialIQA
HellaSwag
ARC-easy
Proj. FT
Hidden Dim.
BoolQ
PIQA
Avg.
Method
62.1
87.9
78.9
91.3
84.0
89.9
79.4
86.6
82.5
8
62.1
88.8
80.5
92.3
83.0
90.2
80.7
84.8
82.8
16
74.7
87.4
80.9
95.4
86.7
90.0
79.4
87.2
85.2
LoRA
32
75.4
88.6
80.7
95.4
86.2
91.2
80.1
86.1
85.5
64
75.1
88.4
81.0
93.0
86.9
90.4
79.7
86.8
85.1
96
74.9
88.4
79.8
94.6
86.3
89.6
78.8
85.4
84.7
4
73.6
88.6
79.8
95.5
85.1
90.2
80.3
86.2
84.9
8
75.6
89.1
80.7
95.6
85.2
90.9
78.7
85.8
85.2
16
73.5
88.9
80.2
95.3
86.1
90.5
78.6
85.6
84.8
DoRA
32
75.3
89.3
80.8
95.3
85.8
89.9
79.3
85.6
85.1
64
74.8
88.6
80.9
94.9
85.3
89.4
79.9
86.2
85.0
96
74.6
89.0
80.0
95.3
85.9
90.4
79.0
88.8
85.4
4
73.7
88.0
79.2
95.0
84.0
89.9
79.7
85.8
84.4
8
73.7
89.0
81.1
95.6
86.3
91.0
80.1
85.2
85.2
74.6
88.9
80.6
96.0
85.1
91.0
80.3
88.6
85.6
IPA
32
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
64
75.9
88.4
80.4
95.9
87.5
91.3
81.0
87.8
86.1
96
75.6
88.2
81.4
95.9
86.6
91.0
80.5
86.8
85.7
Table Detailed results the ablation study on projector pretraining size.
Proj. Pre- training Set
WinoGrande
ARC-challenge
Openbook
SocialIQA
HellaSwag
ARC-easy
Proj. FT
BoolQ
PIQA
Avg.
Method
1%
74.1
88.5
80.9
95.3
86.1
91.4
80.8
87.6
85.6
10%
74.9
88.5
81.0
95.7
85.6
91.0
80.0
88.2
85.6
X
25%
73.6
88.2
80.5
95.5
85.8
91.0
80.1
88.4
85.4
50%
74.3
88.2
80.7
95.3
85.4
90.2
80.4
88.2
85.3
100%
73.7
88.0
81.1
95.2
86.6
90.7
80.1
88.8
85.5
IPA
1%
75.2
88.8
81.0
95.6
86.5
91.3
79.6
87.2
85.7
10%
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
25%
75.4
89.4
81.8
96.0
88.1
91.1
79.9
87.4
86.1
50%
74.9
89.2
81.5
95.9
87.6
91.1
80.7
86.6
85.9
100%
75.1
88.8
80.8
96.1
86.9
90.9
79.9
87.6
85.7
Table 10: Comparison between LoRA and IPA under matched computational budgets. LoRA is trained for about 3% more steps to match total training time of IPA pretrained with 10% of training data.
WinoGrande
chC-enal
OpenbookQA
SocialIQA
HellaSwag
ARC-easy
Proj- FT
FT Steps
BoolQ
PIQA
Avg.
Method
LoRA
/
32,887 31,926
76.2
88.0
80.1
95.0
86.7
90.6
78.5
84.4
85.0
IPA
/
75.0
89.9
81.2
96.0
85.9
91.2
79.6
88.4
85.9
18


Reviews:
[{"review_id": "Tz2sEaG7w0", "reviewer": "Reviewer_4uE7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper argues that LoRA’s down‑projection (A) behaves largely as a random compressor while the up‑projection (B) carries the learning signal. It proposes IPA, which replaces A with a feature‑aware projector trained to preserve information (minimize reconstruction error) using forward‑only incremental PCA over hidden states from the target domain; only the encoder P is kept for adaptation (decoder Q is discarded). Across commonsense instruction‑following (Llama‑2/3, Qwen‑2.5, Gemma‑3), IPA improves over LoRA/DoRA and, when projectors are frozen, reaches LoRA‑level accuracy with roughly half the trainable parameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Compelling diagnosis of LoRA asymmetry. The cosine‑similarity study over 27 BBH tasks shows LoRA‑A stays near initialization while LoRA‑B mirrors full‑FT structure (Figure 1), crisply motivating an input‑aware projector。\n\n2. Simple, practical formulation. IPA reduces to a linear autoencoder objective; with IPCA the projector is learned via forward passes only (Sec. 3.4; Fig. 2), preserving LoRA’s inference cost.\n\n3. Consistent empirical gains and thoughtful ablations.", "audience_yn": "Yes", "audience_explanation": "Parameter efficient fine-tuning is of great interest to the foundation model community.", "requested_changes": "1. stronger baselines. Include PiSSA/CorDA (spectral init), VeRA/RandLoRA (fixed/random bases), and LoRA‑Null; even a subset of tasks would calibrate IPA against the most relevant contenders. \n\n2. Limited theory. The paper motivates PCA empirically but offers no analysis linking PCA’s MSE optimality to downstream task risk; even a linear‑Gaussian argument would strengthen claims.\n\n3. Scope of projection learning. Only linear projectors are tested. What about mild-linear projectors (e.g., tied two‑layer linear + activation)?\n\n4. Generalization and reuse. It remains unclear how projectors transfer across tasks or share across layers/blocks; the paper notes optional fine‑tuning of P but does not study cross‑task robustness or projector sharing. Of course, it could be beyond scopes of this paper.\n\n5. Compute and memory accounting. While pretraining time for IPCA is sketched, a fuller wall‑clock and memory profile vs. LoRA/DoRA (pretraining + tuning) would make the efficiency claim more concrete.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UTCFyS6DXN", "reviewer": "Reviewer_VdwZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work the authors seek to improve the LORA finetuning method which updates weight matrices with a low-rank factorized version. They make the observation that in the LORA decomposition $W = W_{0}+BA$, the matrix $A$ often does not change much from its initialized value. They hypothesize that the initial value of $A$, representing a random feature map, may not be the best space in which to make changes.\n\nThe authors then propose IPA, a method which uses iterative SVD to choose an initialization for $A$ which preserves as much input information as possible. They then provide evidence on a variety of finetuning settings that IPA outperforms LORA and DORA more often than not, and that IPA suffers less degradation than the other methods when $A$ is frozen. They suggest that methods like IPA which more faithfully preserve information in the inputs are a good feature to have for low rank finetuning methods, that don't require much compute overhead.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper clearly presents the background of finetuning, and motivates the approach well using Figure 1. The proposed algorithm is well motivated, easy to implement, and easy to run. The experiments to the best of my knowledge seem compelling. I am not a subject matter expert in finetuning, so I will defer to the other reviewers on questions of appropriate datasets and tasks.\n\nTo the best of my knowledge this approach is new; but again I will defer to reviewers more versed in this area on this point.", "audience_yn": "Yes", "audience_explanation": "Finetuning large models is an area of interest to the community. This work provides an clear explanation about the key phenomenology motivating their approach, which some readers may find helpful. The method itself seems interesting and promising which will also be of interest to the readership.", "requested_changes": "The main methodological issue I have is that, if I understand Appendix A.1 correctly, the same learning rate was used for all methods. It's not clear to me that this is the best thing to do; for example, if the chosen learning rate is better adapted to IPA than to LORA, LORA may perform worse while actually not being worse. I ask that the authors provide experimental evidence that tuning the learning rate better for each method does not destroy the gains of IPA. I think this can be done on a subset of the tasks (in the setting with feature map finetuning) to be convincing. Adding such an experiment would be crucial to secure my recommendation for acceptance.\n\nThe remaining are comments about presentation.\n\nFigure 1 took a while for me to parse; in particular I almost missed figure 1 (a) because of its size relative to the other figures. Is there some alternative way to display that information? I also would love the equivalent of 1 (a) but for B.\n\nOn the topic of Figure 1: something that was not clear to me is how much one should care about the absolute change in A, versus the change in the projections $Ax$ for vectors $x$ coming into the layer. Is there a way to display information on this? Something like:\n\n$$C(0, t) = \\frac{\\mathbb{E}[x^{\\top}A^{\\top}\\_{t} A\\_{0} x]}{\\sqrt{\\mathbb{E}[x^{\\top}A^{\\top}\\_{0} A\\_{0} x]}\\sqrt{\\mathbb{E}[x^{\\top}A^{\\top}\\_{t} A\\_{t} x]}}$$\n\nThis more directly ties into the motivation of the approach as well.\n\nFor the table in Figure 2, it would be helpful to report what fraction of times IPA is the best (separately for projection FT/no projection FT), and maybe a basic statistical test showing that IPA wins more than expected by random chance (using either a Sign Test or Wilcoxon signed rank test, with null hypothesis of success $p=1/3$, or $p=1/2$ to be more stringent if you believe that LORA and DORA are very similar).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KTp6aEbs98", "reviewer": "Reviewer_rN61", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors pose a problem with the random-projection behavior of the LoRA down-projection weight matrix. They propose an alternative down-projection matrix construction by pretraining the projector matrix on the reconstruction task with downstream data. Empirical validation results show that the proposed framework somehow improves the alignment of large language models on the instruction following tasks.\n\n# Strengths\n- The authors deal with an important yet underexplored problem, i.e., the problem of better initialization for LoRA.\n- The proposed framework is conceptually simple and makes sense.\n\n# Weaknesses\n- Lack of rigorous evidence and justification on the key claim (See the next sections).\n- Insignificant performance improvements compared to the baseline.\n- Limited volume of empirical studies in terms of baseline methods.", "claims_and_evidence_yn": "No", "claims_explanation": "Lack of justification for the claim \"data-dependent projection would be better than the random projection for downstream adaptation\".\n  * There is much evidence that random projection quite well preserves the information of the original high-dimensional vector [1, 2, 3].\n  * Given this knowledge, the authors' major claim on the effectiveness of data-dependent initialization over random initialization should be rigorously validated.\n  * While the authors provide empirical results by comparing their method with LoRA, it is still unclear whether some of the performance gains are truly due to better initialization or just driven by longer training with more computation.\n\n\n\n\n> Reference\n1. Random projection in dimensionality reduction: Applications to image and text data, Bingham and Mannila 2001\n2. MEASURING THE INTRINSIC DIMENSION OF OBJECTIVE LANDSCAPES, Li et al. 2018\n3. Black-Box Tuning for Language-Model-as-a-Service, Sun et al. 2022", "audience_yn": "Yes", "audience_explanation": "The authors nicely pose a phenomenon -- asymmetric weight learning -- in LoRA, and show how enhanced weight initialization brings better downstream adaptation. This observation offers plenty of insights, and some audiences in TMLR may enjoy this insight.", "requested_changes": "## critical to securing my recommendation for acceptance \n- Please add more explicit evidence on whether the data-dependent projection is genuinely better than random projection.\n- Please add a controlled experiment on LoRA with an extended computation budget, so that a fair comparison with the proposed IPA is possible.\n- Please add more discussion on the trade-off between the computational overhead and performance gain of IPA compared to the vanilla LoRA.\n\n## simply strengthen the work in my view\n- On page 6, there is a little jump in the 'pretraining algorithm' paragraph.\n  - The authors directly introduce the principal component analysis (PCA) as a learning objective of the proposed projector.\n  - Adding a brief cushion sentence with a reference that explicitly mentions the equivalence between the linear autoencoder reconstruction learning with paired weight and the PCA would be helpful for broader readers who may be less familiar with this.\n- Although there exist tons of LoRA variants, the authors only provide DoRA as a non-LoRA baseline. Could the authors add more baseline to enhance the completeness of this work?", "broader_impact_concerns": "No concerns on the ethical side.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)