Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
Authors: Fei Wu
Jia Hu
Geyong Min
Shiqiang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Parameter-Efficient Fine-Tuning
Efficiency
Orthogonal Fine-Tuning
Large Language Models
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Driven by the rapid growth of model parameters, parameter-efficient fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained computational resources. Within this paradigm, orthogonal fine-tuning and its variants preserve semantic representations of pre-trained models, but struggle to achieve both expressiveness and efficiency in terms of parameter counts, memory, and computation. To overcome this limitation, we propose efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights. Specifically, PSOFT constructs this subspace via matrix decomposition to enable compatible transformations with higher rank, establishes a theoretical condition that strictly maintains the geometry of this subspace for essential semantic preservation, and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability. Extensive experiments on 35 NLP and CV tasks across four representative models demonstrate that PSOFT offers a practical and scalable solution to simultaneously achieve semantic preservation, expressiveness, and multi-dimensional efficiency in PEFT.

Published as a conference paper at ICLR 2026
EFFICIENT ORTHOGONAL FINE-TUNING WITH PRINCIPAL SUBSPACE ADAPTATION
Fei Wu, Jia Hu, Geyong Min; Shiqiang Wang Department of Computer Science, University of Exeter, UK (fw407, j h-u,, g mim,,, sns,, wang w
ABSTRACT
Driven by the rapid growth of model parameters, parameter-efficien fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained computational resources. Within this paradigm, orthogonal fine-tuning and its variants preserve semantic representations of pre-trained models, but struggle to achieve both expressiveness and efficiency in terms of parame- ter counts, memory, and computation. To overcome this limitation, efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre pre-trained weights. Specifically, PSOFT constructs this subspace via matrix decomposition to enable compatible transformations, establishes theoretical condition that strictly maintains the geometry of this subspace for essential semantic preservation, and introduces efficient tunable vectors that gradually relax orthogonality during train- ing to enhance adaptability. Extensive experiments on 35 NLP and CV tasks across four representative models demonstrate that PSOFT offers practical and scalable solution to simultaneously achieve semantic preservation, expressive- ness, and multi mullii-dimensoonall efficiency in PEFT. The code is publicly available https:/ /ttpps//giituse wwtpbbtttttu....o..g. com http://oooo/ee0//PP0000000
propose
INTRODUCTION
Pre- tettrrined foundation models including large language models (LLMs) (Grattafiori et al. 2024) and vision transformers (ViT) (Dosovitskly al., 2021) have transformed natural language pro- cessing (NLP) (Qin et af.) 2023) and computer vision (CV) (Liu et al. 2023). This success is attributed to emergent abilities (Wei et al. 2022) that arise these models scaled up. However, their ever-growing scale poses practical barrier to efficiently tailoring (i.e., fine- tuning) these sophisticated foundation models to specific downstream tasks. To address this challenge, serrameter-efficicn
fine-tur fine-tuning (PEFT) has emerged as promising paradigm that adapts models by updating only a minimal subset of parameters (Houlsby et al. 2019 Lester al.) 2021 Li & Liang [021] Hu et al.) 2022 Meng et al. 2024 yang Liu et al. 2024).
Among PEFT studies, reparameterivation- methods (Hu et al. [2022, Qiu et al. (023) are
widely adopted because they seamlessly inte-
grate with pre-trained weights without adding inference latency.
Prefrained
Weights clicfh
Prindial
As illustrated in the left panel of Figure reparameterization-- based methods include Low- Rank Adaptation (LoRA) (Hu al. 2022 and Orthogonal Tun-iuning (OFT) (Liu et al. 2021 Qiu et al. 2023). LoRA has been widely adopted for its efficient low-rank structure, but it may distort semantic representations embedded in the pre- preitrained weights. These semantic rep- resentations can be understood as the geometric
Residual Weights
LORA
W.,Fll
Rg ERR
Orthagonal Weights dae
Pretrained Weights Wer de
LFFT
Figure 1: Overview of the architectures of LoRA, OFT, and the proposed PSOFT.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of LoRA, OFT variants, and the proposed PSOFT. The table summarizes the trade-off' among semantic preservation, multi-dimensional efficiency, and expressiveness (as reflected in performance) across PEFT methods.
Semantic Representations Parameter-efliienee Memory Usage
Computational Overhead
Methods
(explicitly preserved)
Mechanism
Performance
LoRA (Hu et al.
2022
X Full space Full space
Low-rank
Low
Low
Medium
Full (Luu et
X
Very High
Very High
High
Block diagk-aa DFT (Qu
2023
Block-diagond didgggannall
Medium
Medium
Medium High
BOFT (Liu al 2024
Butterfly factorization
Full space
High
High
High
4GOFF eM
Givens rotation
PSOFT (Ours)
Principal suhspace
I Low-rant wwwrran
Low
Low
High T
structure of weight vectors, specifically the pairwise angles and norms among columns, which en- code relational information learned during pre-training, Distorting this structure may weaken the model's ability transfer knowledge to downstream tasks (Wang al., 2023). In contrast, OFT applies isometric orthogonal transformations, which strictly maintain this geometric structure and thereby preserve semantic representations. However, full-dimensional orthogonal transformations are inefficient in terms of parameter counts, memory, and computation, rendering them impractical for large-scale applications.
This contrast leaves a gap in PEFT between the efficiency of LoRA and the semantic preservation of OFT. Building on OFT's advantages, several studies have explored ways to improve its efficiency while retaining its core strength. Early attempts such block-diagonal OFT (Qiu al.) 2023 reduced parameter counts and partially alleviated computational and memory overhead through block-diagonal sparsity. However, the rigid block structure restricts the model's expressiveness (its ability to capture diverse transformations) and consequently limits the performance that can be empirically attained. To address this limitation, later variants such as BOFT (Liu et al. 2024) and gGOFT (Ma et al., 2024) have sought to restore expressiveness while maintaining parameter efficiency by composing multiple sparse orthogonal matrices sequence. Yet this design incurs a new drawback: chaining multiple sparse matrices introduces substantial intermediate states that dominate runtime and memory consumption. Empirically, qGOFT has been reported to run nearly 6x slower than LoRA A during training (Ma et al. 2024)), while BOFT and qGOFT frequently consume more than 80 GB of memory in large-scale model settings. Such overhead inflates training costs and undermines their practicality. Thus, pparsity-driven OFT variants struggle to achieve both expressiveness and efficiency across multiple dimensions. This tension underlies the central challenge of our work:
How to design PEFT method that simultaneously achieves semantic preservation, expressiveness, and multi dimensional efficiency (parameter counts, memory, and computation)?
To address this challenge. motivated by evidence that both pre-trained models and their task- adaptations reside in a low intrinsic rank (Li et al. 2018 Aghajanyan et al., 2021 Hu et al. 2022). we propose efficient Orthogonal Fine-Tuning Tuning with Principal Subspace adaptation (PSOFT), as illustrated in the right panel of Figure []] The key idea is to confine orthogonal transformations to the low-rank principal subspace of pre-trained weights, thereby overcoming the limitations of conventional OFT operating in the full parameter space and simultaneously achieving semantic preservation, expressiveness, and multi-dimensional efficiency.
However, realizing this idea is non-trivial, as it entails overcoming several technical difficulties: 1) Compatibility. A now-dimensional orthogonal transformation cannot be directly applied to the high-dimensional weight matrix, leading to dimensional incompatibility with the pre-trained model. 2) Geometry preservation. Naively applying low-rank orthogonal transformations may distort the geometry of the subspace, thereby undermining the strict preservation of essential semantic representations. 3) Adaptability. Striet orthogonality constraints may hinder adaptation to slight task-specific drifts, resulting in suboptimal performance on downstream tasks.
PSOFT resolves these difficulties through principled designs. First, constructs a principal subspace of pre-trained weights through matrix decomposition, enabling compatible orthogonal transformations and yielding a higher rank that enhances expressiveness. Next, it establishes a theoretical condition to strictly maintain the geometry of the subspace, thereby ensuring essential semantic preservation. Finally, it introduces efficient tunable vectors to gradually relax orthogonality during training at negligible cost, improving adaptability across diverse downstream tasks.
We evaluate PSOFT through extensive experiments on 35 NLP and CV tasks with four representative pre-trained models. Compared with OFT variants, PSOFT consistently avoids oft-of- memory (OOM)
2

---

Published as a conference paper at ICLR 2026
failures and accelerates training. On small-scale models, it achieves up to 18x higher parameter efficiency with the lowest memory footprint among baselines, without compromising average per- formance. On larger models, PSOFT lowers the memory footprint of OFT to level comparable with LoRA-lik A-like methods while outperforming LoRA on GSM-8K (+2.3%) and Commonsense Rea- soning (+1 1.4%) with comparable parameter counts. As summarized in Table [ PSOFT preserves semantic representation in the principal subspace while minimizing parameter counts, memory, and computation overhead, and simultaneously maintains expressiveness as reflected in high performance.
The main contributions of this work are summarized as follows:
We introduce new low-rank perspective that unifies efficiency and expressiveness in OFT, bridging the gap between low-rank adaptation and orthogonal fine-tuning,
We establish a theoretical condition under which low-dimensional orthogonal fine-tuning strictly preserves the geometric structure of the subspace.
We propose PSOFT, a framework that confines OFT to the principal subspace with theoretical guarantees and practical adaptability.
We validate PSOFT through extensive experiments, establishing a practical and scalable solution to efficiency.
simultaneously achieve semantic preservation, expressiveness, and
2 RELATED WORK
Srrameter-Efficiees Fine-Tuning (PEFT). PEFT adapts pre-trained models to diverse down- stream tasks by fine-tuning only a small subset of parameters. Specifically, existing PEFT meth- ods fall into three categories: 1) Selection- based methods select specific components of the pre- trained model without altering its architecture (Zaken al. 2022 Song et al., 2024 Xu & Zhang
2024). 2) Addition- based methods insert prompts or adapters at the input or within Transformer
blocks (Houlsby et al. 2019 Pfeifter et al., 2021 Lester al. 2021 Li & Liang 2021 Liu et al.
2022). 3) Reparameterization-aasse
methods reparameterize weights in paral lel with minimal
parameters (Hu et al.
2022 Azizi et al. 2024 (alazy et al.) [0224] Gao et al. 2024 Kopiczko et al.
2024; Lingam et al. 2024; yang Liu et al.) [2224] Meng et al. (2024). Reparameterization-
methods are particularly appealing since they incur no additional inference latency, with represen-
tative examples including LoRA (Hu et al. 2022) and OFT (Qiu et al. 2003). LoRA's variants, such as PiSSA (Meng al.] (024) and DoRA (yang Liu al. 2024). improve convergence through re-initialization and enhance performance via weight decomposition, respectively. DoR decomposes the low-rank update into direction and magnitude components, but it may introduce additional mem- ory and computational overhead for computing these components. In addition, LaMDA (Azizi et al.) 2024) and LoRA-XS (Balazy et al. 2024 reduce the parameter count and resource usage of LoRA by employing more compact matrices. In LoRA-XS, the learnable square matrix is constrained by the fixed LoRA matrices, which may limit its expressiveness. However, these LoRA- methods may induce semantic drift from the pre- trained representations (Wang et al., 2023), which may degrade output quality in generative tasks.
Orthogonal Fine-Tume Fine-Tuning (OFT). Unlike additive methods such as LoRA, multiplicative OFT preserves semantic representations of pre-trained models through orthogonal transformations, which maintains the hyperspherical energy among neurons (Liu et al. 2021 Qiu et al. (202)). To mitigate the prohibitive cost of applying orthogonal transformations over the full parameter space, prior studies typically introduce sparsity constraints. For instance, block-diagonal OFT (Qiu et al. 2023) adopts block- Bloc-ddiagonal sparse structure to reduce parameter counts, though at the risk of undesired inductive biases (Liu et al. 2024). BOFT (Liu al., 2024) and gGOFT (Ma et al. 2024) address this issue by replacing dense matrices with sequences of sparse multiplications, thereby improving parameter efficiency while restoring expressiveness. Nevertheless, these variants remain less efficient in memory and computation than LoRA and its variants. In parallel, Adapter (Zhang & Pilanci (022)) rotates the top spectral space using orthogonal transformations to preserve spectral characteristics of pretrained weights, in contrast to the geometric structure emphasized in OFT. Overall, existing OFT variants struggle to achieve both expressiveness and efficiency across multiple dimensions.
These limitations motivate our PSOFT algorithm, which confines orthogonal transformations to the principal subspace with a theoretical guarantee of preserving essential semantic representations, followed by relaxation of strict orthogonality negligible cost enhance adaptability.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
In this section, we formalize LoRA and OFT variants in mathematical notation, providing a unified view of their parameterization strategies.
Conventional full fine-tuning (FFT) updates the entire pre-trained weight matrix Wpe pre Rd to obtain W, whereas PEFT methods freeze pue and introduce only a small set of trainable parameters. For LoRA (Hu et al. 2022). the update is parameterized by a low-rank decomposition:
h W (W pre AB) x, s.t. rank(AB) r,
where RRdx and R are trainable matrices. Following standard practice, A is initialized with Kaiming initialization (He et al. 2015) and B with zeros, training begins from W pre.
For OFT (Liu et al. 2021 Qiu al. 2023). the update parameterized by an orthogonal matrix R. which fine-tunes W pre in the full parameter space, i.e., Wi-amed = R.W pre. The forward pass is given by:
fs-uned (RW pre)
S.I. R RR Ia-
where pdddd is initialized the identity matrix that training begins from W By construc- tion, orthogonal transformations in the full parameter space preserve both angles and norms, thereby maintaining the geometric structure pre.
To reduce parameter overhead, block diagonal OFT (Qiu et al. 2023) constrains R to block- diagonal form diag(Rs, R..... R...... Ra/r), where each R. O(d/r). Although efficient, this structure may introduce undesirable inductive bias. BOFT (Liu et al.
2024) and qGOFT (Ma
al. 2024) mitigate this by factorizing R into sparse matrices, R Rm Raxd sparse. Assuming is a power of two, log d integral, ensuring a valid factorization.
log I= Rm, with each
This construction restores the expressiveness of dense rotations with reduced parameters.
4 METHODOLOGY
As discussed Section existing OFT variants such as BOFT and qGOFT still incur substantial computational and memory overhead. Prior studies (Li al. 2018 Aghajanyan et al. 2021 Hu al. 2022) further suggest that both pre-trained models and their task-specific adaptations lie in a low-rank intrinsic subspace. Motivated by this insight, we propose Orthogonal Vine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the low-rank principal subspace of W pre. The complete algorithm given Appendix and the remainder of this section details its design.
4.1 DIMENSION-COPPATIBE. ORTIIOGONAL TRANSFORMS
Realizing orthogonal fine-tuning in the subspace requires a projection of high-dimensional weights onto low-dimensional subspace, since directly applying the orthogonal matrix R E Rrxr to Wfr dx infeasible due to dimensional incompatibility. To construct this projection, we perform Singular Value Decomposition (SVD), Wpr = UEV and decompose it into Wp and Wes. such that W pre Wp + W Here, the subscript "pri" denotes the principal component reconstructed from the top-r singular values and vectors, while "res" denotes the residual component. The principal component W
pri then used to derive symmetric low-rank matrices A and B
W pri = U Rarry 2 [arar V (
Bar) E Bdd (Symmetric),
AERdxy
BeR
where A projects weights into the r-dimensional principal subspace, while B reconstructs them back. The residual component W.. then obtained from the remaining singular values and vectors:
W
W pre
Wpri = Up.rs D [n:r:] [ar E _d
Building on this, regard W pu AB representing the initial principal subspace of W pre- This subspace enables dimension-compar orthogonal transformations, yielding W Wnsttuee ARB, where the subscript "ps-tuned" denotes the fine-tuned weights in the principal subspace for PSOFT.
4

---

Published as a conference paper at ICLR 2026
Orthogonal Weights R
x
Pre-tralned Weights W
Pre-trained
Weights W
n
Decompasition
Principal
X
X
B
Residual Weights Wes
BOFT,
Weights Wpn
Wpr= A-B=
Sparse Orthogonal Fine-Tuning (OFT variants)
Orthogonal Fine-Tuning in the Principal Subspace (PSOFT)
C2
Preserve angles and norms the space
B
Preserve angles and norms in the principal subspace Wpssunes A A'ddia((a) R R-diag(B)-
Wund = R- W
C5'
C2
e
x
Higher computational cost
computational
C, 0
C2
Larger memory footprint
Smaller GPU memory footprint
C
Col Vectors in Was
C in Wresueed
Col Vectors
Vectors In Wat
Vectors Wiraaee (initial of PSOFT)
Warazed (PSOFT)
(OFT variants)
Figure Our proposed method: PSOFT. The left panel illustrates principles of OFT variants. the right, PSOFT preserves the angles and norms of W pai (blue) in the fine-tuned W ps-tumed (orange), while allowing adjustable angles and scalable norms in the sector.
Unlike LoRA (Hu et al. 2022) and PiSSA (Meng et al. 2024), which train both A and B, PSOFT freezes them and fine-tunes only the orthogonal matrix R. LoRA produces updates A AW AAB that span the low-rank manifold fAW W rank (W) s r} } of dimension r(d+n-r). In contrast, PSOFT generates updates AW A(R B) parameterized solely by an orthogonal matrix R O(r). where (r)) denotes the r(r 1) 1)/2-dimmnsional orthogonal group. Because the variability of AW arises only through R, all updates remain confined to the fixed row and column subspaces defined by A and B. Consequently, LoRA and PSOFT operate fundamentally different geometric families updates (low-rank vs. orthogonal), and their expressiveness is therefore not directly comparable. The same structural distinction also determines different feasible ranks under equal trainable- trainable-parameter parameter budget M. LoRA trains two matrices, giving M = (d + n) FLORA and thus "LoRA M/(d 12), whereas PSOFT trains only orthogonal matrix, yielding M rPSOrT and hence TrsoFT VM. Since typically v VM < (d + n). we obtain rPsoFT > FLORA, which explains why PSOFT empirically operates with much larger ranks under the same parameter budget.
4.2 GUARANTEED GEOMETRY PRESERVATION IN THE PRINCIPAL SUBSPACE
Orthogonal transformations within the constructed principal subspace in Section 4.1 merely ensure dimensional compatibility but do not strictly preserve subspace geometry. In particular, applying low-dimensional orthogonal matrix R to the subspace spanned by symmetric A and B in Eq. may distort the pairwise angles and norms among the column vectors of W To address this issue, we analyze the conditions under which orthogonal fine-tuning preserves the geometry of the principal subspace, and present an informal Theorem 4.1 with the formal theorem and proof in Appendix
Theorem 4.1 (Informal: Angle and norm preservation in the principal subspace). Let W AB denote the principal weights and Wps-tuned = ARB denote the fine-tuned weights. For Wps-tuned to preserve (i) pairwise angles between columns, and (ii) column norms W pri the following condition must hold:
R A AR = ATA.
(5)

---

Published as a conference paper at ICLR 2026
We provide intuitive explanation of Theorem 4.1 The geometry of the principal subspace is determined by the relative angles and lengths of its column vectors, which are encoded in the Gram matrix G' A A. Any R satisfying R GR G' can be viewed as symmetry of this geometry, similar to a rotation or reflection. In other words, if we first apply R to the columns of B and then project them using A, their angles and lengths in the high-dimensional space remain unchanged.
In practice, normalizing A so that A A- = I, simplifies the condition, in which case R reduces to a standard orthogonal matrix. Accordingly, Eq. is modified in PSOFT as:
W pri = U B.ar
(ar)
dd
IR
(Asymmetric),
(6)
A'eRex
B'ERex"
where asymmetric A' and B' are derived from the top-r principal components of the SVD. The residual Wes remains as in Eq. and the forward computation becomes:
h W ps-tumed Wes) (A'RB' Wres)
(7)
where A', B', and W are frozen, and only R E Rexr is trainable, initialized as the identity matrix.
To satisfy Eq. 19 during training, it is ssential to maintain the orthogonality R. Enforcing orthogo-
orthogonalization) is computationally expensive. To reduce this cost, following prior studies (Qiu et al. 2023; 2025), we adopt the Cayley parameterization (Cayley 1894 to enforce the strict orthogonality of R, where R (I Q) + Q)- and Q Q is
nality of R (e.g.. via
skew-symen symmetrii matrix. Further details on the Cayley parameterization are provided Appendix C
4.3 EFFICIENT RELAXATIONS OF ORTIIOGONALITY
Eqs. and guarantee geometry preservation in the principal subspace, but strict orthogonality con- straints may hinder adaptation to task-specific drifts, leading to suboptimal performance. Empirical evidence shows that moderate relaxation improves results (Ma al., 2024). Yet existing methods sacrifice efficiency: gGOFT relaxes constraints more flexibly but requires four times the parameters of GOFT (Ma et al., 2024). while BOFT relaxes them through additional scaling vectors the output dimension, whose size grows linearly with model scale (Liu et al.) 2024). To overcome these issues, we propose efficient relaxations of PSOFT that enhance adaptability with minimal overhead.
Specifically, we introduce two tunable vectors that modulate the input and output norms around the orthogonal matrix, modifying Eq. to yield the following forward computation:
h- = (A' diag (a) R (3i) + Wre.) Tx (PSOFT),
(8)
where A', B', and W.e. remain fixed, while only R and the tunable vectors a and ,3 are trained. Both vectors are initialized all-one vectors to ensure strict orthogonality at the start of training. As illustrated in Figure PSOFT relaxes this constraint during training, enabling adjustable angles and scalable norms that adapt task objectives. As these additional vectors inserted within the subspace, the overhead is limited to 2r parameters (2r < where is the output dimension), enhancing adaptability with minimal cost and without significantly affecting the geometric structure.
To avoid excessive deviation from orthogonality, an explicit constraint can be imposed: |CTC -| S, c, where C diag(a:) R diag(G). Deviation arises when either diag(a: or diag(B) deviates from scalar multiple of the identity. In the special case where diag(a) = I and (ia \2 angular relationships are preserved, and magnitudes are uniformly scaled.
In summary, PSOFT performs orthogonal fine-tuning to the low-rank principal subspace, enabling dimension-compatibbe transformations with theoretical guarantees on subspace geometry, while relaxing strict orthogonality at negligible cost to enhance adaptability. It requires only r(r (1-11/22+3 trainable parameters by combining the Cayley parameterization with two efficient tunable vectors. Moreover, it reduces both the number and size of additional matrices (from min(d, n)) to r, with < min(d, n))), thereby yielding substantially lower activation memory than other OFT variants under the same batch size and sequence length. Detailed comparisons parameter counts and activation memory analysis across different PEFT methods are provided in Appendices D Dnd

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
To evaluate PSOFT, we conduct experiments on 35 tasks spanning language and vision domains, using encoder-only models (DeBERTaV3-baas) (He et al. 2021), ViT-B/16 (Dosovitskiy et al. 2021
and decoder-only models (LLaMA-3.2-3B (Meta All 2024). LLaMA-3. 1-8B (Grattafiori et al. 2024)
These models are fine-tuned on downstream tasks, covering natural language understanding (Wang
et al. 2019). visual classification (Zhai et al. 2019), mathematical QA (Yu et al. 2024). and commonsense reasoning (Hu et al., 2023). We evaluate key metries such as parameter counts, peak memory usage, and accuracy in the main experiments, and assess training speed separately in the efficiency analysis. Following OFTv2 (Qiu al.
2025), we implement the Cayley parameterization
by approximating (I+ Q)= with a truncated Neumann series, - (-Q² using K = terms in practice. All experiments are performed on single GPU with FP32, using an NVIDIA RTX 4090 (24 GB) for encoder-only models and an NVIDIA H100-SXM (80 GB) for decoder-only models.
K
Baselines. We employ state-of-the-art OFT variants with other advanced PEFT methods baselines:
FFT (Howard & Ruder 2018) updates all model weights during fine-tuning.
GOFTv2 & qGOFTv2
Ma al. 2024) replace full-space OFT with Givens rotations. The latest
implementation uses Hadamard products instead of sparse multiplication.
BOFT (Liu et al.
2024
substitutes full-space OFT with butterfly factorization.
OFTv2 QQu et al.
2023
2025) employs a block-diagonal structure for OFT, with the latest version
adopting an imput-
centric computation and Cayley-Neumann parameterization.
LoRA (Hu et al.
2022)
freezes pre-treennntt traitreined weights and adjusts only two low-rank matrices.
PiSSA (Meng et al.
2024) improves LoRA initialization to fine-tune principal weights.
DoRA (yang Liu et al.
2024) decomposes low-rank adaptation into direction and magnitude.
LoRA-XS (Balazy et al.
2024 injects and tunes a single square matrix between LoRA's matrices.
Encoder-only Models. We evalu- ate PSOFT by fine-tuning DeBER- TaV3 (He al.] 2021) on sev-
Table 2:
Experimental results of fine-tuned DeBERTa V3- base. Results are averaged over 5 random seeds. Memory (GB) denotes peak memory with sequence length 64.
eral datasets from the GLUE bench- mark (Wang et al.] 2019). Following prior work Wu et al.) 2024a b Bini al. 2025), we split the original vali- dation set into new validation/test sets with fixed seed, and report test accu- racy from the best validation check- point to ensure rigorous evaluation. Details are in Appendix
Memory (GB)
Methods
#Params
CoLA STS-B RTE MRPC QNLI
Avg.
FFT
184M
5.9
67.56
91.46
82.88 90.69 94.13 93.37 86.68
GOFTV2
0.08M
18.5
65.45
N/A. (OOM)
QGOFTV2
0.33M
18.5
68.03
N/A. (0OM)
BOFT
141M
6.3
68.85
91.09
83.60
88.40 95.28 93.78
86.83
OFTv2
1.29M
4.5
66.79
91.22 84.03
89.61
93.72 92.64
86.34
LoRA,es
1.3M
4,5
67.98
91.60 84.87
90.20
95.28 93.89
87.30
PISSA
1.35M
4,5
66.50
91.40
83.77
89.90
93.17 92.72
86.24
DoRA,.S
L4IM
5.8
67.06
91.60
87.19
90.49
95.23
94.09
87.61
As shown in Table 2 GOFTv2 and qGOFTv2 have non-tunable parame- ters and often encounter OOM failures
LoRA-XS..1100 136
1.33M
4.2
64.67
91.48
84.17
91.27
93.85
93.14
86.43
PSOFT
0.08M
4.1
70.42
91.56
86.74
90.49
95.55
93.47 88.04
as the sequence length increases. PSOFT improves parameter and memory efficiency without com- promising performance. Although GOFT and PSOFT have the same parameter counts, PSOFT reduces memory usage by about 80% and avoids OOM issues. further achieves up to an 18x improvement in parameter efficiency over BOFT, OFTv2, and LoRA variants, attaining the best average performance across all baselines with the lowest memory footprint. Compared with LoRA variants that do not rely on weight decomposition, DoRA introduces additional memory overhead. For L-RA-XS, the update is constrained by the initialization of its low-rank matrices, which limits its expressiveness and consequently leads to degraded performance. These results highlight PSOFT's ability to achieve both efficiency and performance.
We also evaluate PSOFT by fine-tuning ViT-B/16 (Dosovitskiy et al. 202 1)
the VTAB-IK benchmark (Zhai et al. 2019). Further details are provided in Appendix G As shown in Table 3 PSOFT extends its fffccency--erformman to vision tasks. Beyond avoiding the heavy memory demands of GOFTv2 and gGOFTv2, PSOFT consistently reduces the memory overhead of BOFT and OFTv2. Compared to LoRA and its variants,
advantages the small-scale model from language tasks
7

---

Published as a conference paper at ICLR 2026
Table Experimental results of fine-tuned ViT-B/16 on the VTAB-IK benchmark. Reported values
(top-1 accuracy %) are the mean of 5 runs with different random seeds.
Natural
Specialized
Structured
Methods
Alaarmn
Mem (GB)
Cifar 100
Caltech
DTD102
Flower102
SVHN
Sun
Camelyon
EBroAAT
Reshe45
Redioopathy
Cleve-Count Count
Cleve
Cleve-Dist
DMLab
KITTI-DiS Dist
dSpr-Loc Loc
dSpr-Ori
ANORB-Azin
sNORB-EK
AY8
Pets
85.9M
69.5
99.0 90.4
81.7
54.9
83.8 74.5
58.3 51.5 43.2
30.0
GOFTv2
0.08M
00M
N/A.
@GOFTv2
0.33M
0OM
N/A.
BOFT
1.4MM
10.9
70.6
88.2
69.8
99.0
91.4
77.4
55.1
85.1 93.6 82.3 74,9 61.8
50.4
42.9
76.1
73.7
48.8
15.7
30.8
70.9
OFTv2
1.29M
7.7
88.9
67.5
98.4
89.5
84.2
74.6
58.7
56.4
46.7
48
17.3
32.5
LoRA.
1.33M
70.1
99.0
91.4
83.3
74.1
72.0
54.3
43.0
31.8
PISSA,-S
68.9
99.2
91. 81.9
74.0
71.0
60.2
33.1
DoRA.
17.8
91.0
67.3
54.2
16.9
31.5
SURA-XS.=10
1.33M
68.4
90.9
60.0
57.7
45.8
30.8
PSOIT 66
91.8
75.2
71.2
59.9
32.9
achieves the best average aceuracy with about 94% fewer parameters and the lowest peak memory footprint. Interestingly, we also observe that parameter counts and memory overheads of different PEFT methods do not necessarily correlate. For example, the weight decomposition introduces substantial memory overhead on the ViT-B/16 model compared with other LoRA variants, even when the number of trainable parameters is similar. This suggests that PEFT design should consider multi-dimensional efficiency beyond parameter efficiency alone.
DoRA
Decoder-only Models.
Following prior
Table 4: Experimental results of fine-tuned LLaMA-3.2-3B on GSM-8K and MATH.
work (Lingam al. 2024 Liu et al. 2024). we
fine-tune the LLaMA-3.2-33 (Meta Al 2024)
model on MetaMathhQA-400
(Yu et al. 2024
Memory (GB)
Methods
#Params
GSM-8K MATH
and evaluate on GSM-8K (Cobbe et al., 2021)
and MATH (Hendrycks et al., 2001). For large-
FFT
3.21B
69.0
63.00
16.84
scale models and complex tasks, where perfor- mance more sensitive parameter counts, align trainable parameters by setting the LoRA rank to 8 to ensure a fair comparison. PEFT modules are applied to all linear layers, with ad- ditional hyperparameter details Appendix
GOFTv2
0.75M
OOM
N/A.
QGOFTv2
2.98M
OOM
N/A.
BOFT mm2
3.76M
OOM
N/A.
OFTV2
11.6M
35.2
61.03
15.70
LoRA
12.2M
32.2
60.80
15.76
PISSA
12.2M
32.2
61.26
14.96
DoRA
12.9M
43.4
62.62
15.48
LoRA-XS,-244
12.1M
34.4
61.56
15.02
As shown in Table 4 as models scale up, BOFT suffers from OOM failures like GOFTv2 and qGOFTv2, whereas PSOFT avoids this issue PSOFT reduces the peak memory footprint of OFT variants to level comparable with LoRA-like methods, while delivering superior performance under similar parameter counts. Against advanced PEFT methods, it outperforms LoRA (+2.28%) on GSM-8K and PiSSA (+1.02%) on MATH, while maintaining memory usage comparable to LoRA-like baselines. Compared the sparsity-based OFTv2, PSOFT achieves higher performance at comparable cost. When scaling to large models and complex reasoning tasks, PSOFT adapts by employing a higher rank r ensure sufficient expressiveness, yet still maintains efficiency and clear memory advantages over BOFT, GOFTv2, qGOFTv2, and DoRA. Although increasing the rank may enhance the expressiveness of LoRA-XS, its performance remains fundamentally constrained by the initialization: the inserted square matrix is trainable only as linear combination within the original low-rank subspace. Even under restricted module insertion and tighter parameter budgets, PSOFT still reduces memory overhead relative to qGOFTv2 and BOFT [33 in Appendix H.. demonstrating strong scalability to large models and complex mathematical tasks.
PSOFT
12.2M
36.2
63.08
15.98
Following prior work (Hu et al. 2023 Lingam et al., 2024 yang Liu et al. 2024), we further fine- tune LLaMA-3.1 1-B (Grattatiori al. 2024) on the Commonsense-133 15K dataset (Hu et al.) 2023) and evaluate it on eight commonsense reasoning benchmarks. PEFT modules are applied to the Q.K. D linear layers. popendix([dtaailn the hyperparameter settings. As shown in Table PSOFT mitigates the frequent 00M issues of OFT on larger models while achieving the best average performance. In practice, GOFTv2, gGOFTv2, and BOFT suffer from OOM failures without
8

---

Published as a conference paper at ICLR 2026
Table 5: Experimental results of fine-tuned LLaMA-3.1-88 on commonsense reasoning benchmarks.
Memory (GB)
Methods
#Params
BoolQ PIQA SIQA
HS
WG ARC-e ARC-c
Avg.
FFT
8.03B
OOM
N/A
GOFTv2
0,98M
OOM
N/A.
qGOFTv2
3,93M
OOM
N/A.
BOFT
4.72M
OOM
N/A.
OF
14.3M
55.5
70.83
84.44
73.34
90.63
74.11
90.87
80.12
81.80
80.77
LoRA
14.2M
73.18
85.31
74.36
86.57
74.19
90.95
80.29
84.00
81.11
PISSA
14,2M
54.1
71.22
86.02
75.38
90.27 74.19
89.90
79.44
84,00
81.30
DoRA..
14.9M
65.6
73.09
85.96
75.08
90.48
75.53
90.74
81.40
84.40
82.09
LoRA LoRA-XS
14.2M
56.2
72.35
86.51
75.18
91.73
74.98
90.74
79.52
84.00
81.88
PSOFT
14.5M
58.4
72.17
86.51
75.79 91.28 75.61 91.46
81.48
86,00
82.54
inserting modules into all linear layers, severely limiting their use in large-scale fine-tuning, whereas PSOFT provides a more memory-friendly alternative. Under comparable costs, it surpasses OFTv2 by 1.77% average accuracy, matches the memory efficiency of LoRA-like baselines while delivering higher accuracy, and reduces memory usage by about 7 GB relative to DoRA. As the model size increases, DoRA attains performance that surpassed only by PSOFT, but its memory overhead becomes noticeably higher than that of other LoRA variants. PSOFT further remains effective under reduced parameter budgets and restricted module insertion (Table 15 in Appendix I]. underscoring its practicality in balancing efficiency and performance across diverse settings.
Table 6: Effect orthogonality of on LLaMA-3.2-3B.
52.0
51.63
Methods
#Params GSM-8K MATH
GSM 8K Accuracy
51.5
LLSSA+LORA-XS (y=0.0)
12.1M
61.26
14.72
51.0
50.72
PISSA=LORA-XS (y=0.01)
12.1M
61.26
14.80
50.57
50.5
PISSA+LORA-XS (y=0.1)
12.1M
59.89
14.90
50.19
RSSA=LORA-XS (y=1.0)
12.1M
59.36
14.44
50.0
PSOFT. (strict orthogonality)
6.0M
61.18
14,80
only only PSOFT with tunable vectors
and
PSOFT, (strict orthogonality)
12.1M
62.77
15.74
Figure 3: Effect of tunable vectors.
Ablation Studies. To study the effect orthogonality R, we follow AdaLoRA (Zhang et al. 2023 and add orthogonality regularizer Lonh = || R R RR-Ill F,, resulting in the objective L L Loeth with weight y. Setting y = recovers PISSA+LORA-X0 with unconstrained R. As shown in Table 6 this regularization avoids Cayley inversion but demands careful tuning. Under equal rank, PSOFT with strict orthogonality matches the unconstrained variant with half the parameters, and achieves clear gains once parameter counts are aligned. Therefore, Cayley parametrization in PSOFT not only enforces orthogonality but also exploits its skew-symmetric structure to improve parameter efficiency.
To study the effect of tunable vectors a and 8, we fine-tune LLaMA-3.2-3B with rank 64, inserting PSOFT into all linear layers and evaluating on GSM-8K and MATH. As shown in 3 enabling both vectors achieves the best performance, while single-sided insertion provides smaller gains. This suggests that tuning only one side lacks sufficient capacity to capture task-specitt variations.
To study the effect of initialization, we compare three vari- ants: Runth B, ARRuh Banh. and A R, where A and follow PiSSA (Meng et al. 2024) and Auth: Buth
Table 7: Effect of initialization.
Methods
RTE
CoLA
use orthogonal initialization with rank 64. As shown in Ta- ble 7 Aorh Ronh yields the best results, outperforming PiSSA without constraining A and B, whereas enforcing orthogonality on B reduces model expressiveness.
Auch NothRem. B
85.92
70.63
AR.m Banh
52.71
67.97
A AR
71.11
69.23
Memory and Computational Efficiency. We evaluate memory usage among different batch sizes by fine-tuning ViT-B/16 on VTAB-IK with PEFT modules in linear layers. As shown Figure 4a PSOFT consistently requires less memory than advanced OFT variants across batch sizes, maintaining a peak footprint below GB even at batch size 32, which highlights its suitability for resource- constrained settings. Further detailed memory analysis and experiments are provided in Appendix M
We also evaluate the computational cost under the same experimental settings on a single H100 GPU in Tables Fables|ando 5 As shown in Figure 4b, on LLaMA-3. 3.2-BB, PSOFT (Q.K, V) trains in 57 minutes,

---

Published as a conference paper at ICLR 2026
LLaMA:3.2-30 (Q.KV)
IO.KKNUUDD.OGG
00M
00M
0OM
Training Speed
2,x
Peak GPU Memory (GB)
NUA.
1.0k
GOFFV2
1.0x
16-
PSOF Ta [ours]
coFTa
DOFT
PS3rT
GOFTH
BOFT
Bola
PSOF7
LLaMA-3,1-88
LLaMA
VQ..V.U.)
Training Speed
21x
2.2x
177x
IODMI
(0OM)
NUA.
1,x
16 32
128
coma
PSoPT
9oom
RoFT
Dolka
PSDPT
Batch
(a)
Figure 4: (a) Memory usage across batch sizes, (b) Training speed across different models.
yielding 3.5x and 2.1x speedups over GOFTV2/4GOFTV2 and BOFT, respectively, while its full configuration (Q.K, K.K, WUU.DO,G) requires hour 31 minutes and achieves 1.3x speedup over DoRA. On LLaMA-3. 18B,, PSOFT (Q.V V) completes training in 29 minutes with a 3.2x speedup over BOFT, and PSOFT (Q. K, V,U,D) finishes in 53 minutes, running 1.7x faster than DoRA. Compared with other PEFT methods, its computational efficiency falls between that of DoRA and LoRA.
DISCUSSION ON SCALING TO LARGER MODELS
Due to hardware resource constraints, our empirical evaluation is limited to models of up to 8B parameters. Nevertheless, we further discuss the potential limitations and stability considerations when extending PSOFT larger-scale models. From methodological perspective, PSOFT scales favorably as model size increases. Because the orthogonal transformation operates in an r-dimensional principal subspace rather than the full d-dimensional weight space, both computational and activation- memory costs grow with the controllable rank r instead of the expanding dimension required by many PEFT methods (a detailed analysis is provided in Appendix EE. As shown in Appendix (Tables 17 and T8), memory usage and training time remain stable as r increases. The subspace- based update also avoids the long chains of full-dimensional multiplications used in GOFT and BOFT, which become increasingly expensive at larger scales. Moreover, the number of trainable parameters in PSOFT decoupled from the hidden dimension, enabling fine-graines parameter control and preventing the minimum parameter budget from being tied to layer width. Collectively, these properties indicate that PSOFT can extend effectively to larger architectures while maintaining stable optimization behavior.
However, when applying PSOFT to models larger than 8B, several practical factors may need to be considered. Large models often exhibit higher sensitivity to hyperparameters, including learning-rate settings for structured updates such as orthogonal transformations. While PSOFT does not rely on full-dimensional orthogonal matrices, stable training at very large scales may still require careful pepraaaraeeter tuning. Moreover, although the activation memior growth of PSOFT slower than that of some OFT approaches, the activations of the underlying backbone (e.g., attention and cord-forward layers) can become the dominant source memory usage at large scales, which may constrain the choice of batch size or sequence length. Finally, as shown in the main experiments and in the additional rank-sensitivity analyses in Appendix ! larger models tend to benefit from higher ranks capture task-specific variations. Very small ranks may lead underfitting on complex tasks, whereas larger ranks improve expressiveness but also increase the trainable parameter budget.
CONCLUSION
In this work, we have proposed PSOFT, a novel PEFT framework that confines OFT to the principal subspace with theoretical guarantees, while enhancing practical adaptability through two tunable scaling vectors. Extensive experiments demonstrate that PSOFT introduces a low-rank perspective that resolves the tension between expressiveness and multi-din mulli--iiensionall efficiency in OFT, bridges the gap between orthogonal fine-tuning and low-rank adaptation within the broader PEFT landscape, and offers solution with superior scalability and practicality for adapting future foundation models.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work and have taken the following steps. For the proposed method, we provide source code at/ttps::/g/thui thub. Fooo//ee40//PSFFFF For theoretical results, we include formal statements and complete mathematical proofs in Appendix For datasets and experimental settings, we offer detailed descriptions and full hyperparameter configurations in Appendices
and
ACKNOWLEDGMENTS
This work was supported in part by UK Research and Innovation (UKRI) Grant No. EP/X038866/ and Horizon Europe Grant No. 101086159.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume I: long papers), pp. 7319-7322 2021.
Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram. LaMDA: Large model fine-tuning via spec- trally decomposed owwdimensional adaptation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9635-9646, 2024.
Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604, 2024.
Massimo Bini, Leander Girrbach, and Zeynep Akata. Decoupling angles and strength in low-rank adaptation. In The Thirteenth International Conference Learning Representations, 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Arthur Cayley. The collected mathematical papers of Arthur Cayley, volume 7. University of Michigan Library, 1894.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-292222 2936. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1r03vv1803.05443 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.11 141668 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In The Ninth International Conference on Learning Representations, 2021.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform. In Forty-first International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXIv:2407.21783, 2024.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2): 217-288, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Dehertav3: Improving deberta using electra-style embedding sharing. arXiv preprint arXiv:21 1100543,
pre-training with 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 228-339, 2018.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LL -ddattr:: An adapter family for parameter-efficien fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, 2023.
Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341-353, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In The Sixth International Conference on Learning Representations, 2018
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, 2021.
Vijay Chandra Lingam, Atula Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-efficient fine-tuning with singular vectors. Advances in Neural Information Processing Systems, 37:41425-4 41446, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
12

---

Published as a conference paper at ICLR 2026
Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg. Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal over parameterrzze training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7251-7260, 2021.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue. Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In The Twelfth International Conference on Learning Representations, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 61 68, 2022.
Xinyu Ma Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, and Junfeng Zhao. Parameter efficient quasi- orthogonal fine-tumin the-ttnnngg via givens rotation. In Forty-first International Conference on Machine Learning, 2024.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficien fine-tuning methods. https: //tp://ggtthb// com/ com//hagginggfcce/eeet 2022. Accessed: March, 2025.
Fanxu Meng, Zhaohui Wang. and Muhan Zhang, Pissa: Principal singular values and singular vectors adaptation of large language models. Advances Neural Information Processing Systems, 37:
121038-121072, 2024.
Meta
AL.
Llama 3.2:
Revolutionizing edge AI
and vision with https: meta com/blog
open, 11ama customizable models.
lese-mombieeeeiieee.. 2024. Accessed:
March, 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? ? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, gigh-performance deep learning library. Advances in neural information processing systems, 32. 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych, Adapter- Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, 2021.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatGPT general- Empirical Methods Natural Language Processing. 2023.
ceneral-purpose purpose natural language processing task solver? In The 2023 Conference on
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning, Advances in Neural Information Processing Systems, (1579320-77962.. 2023.
Zeju Qiu, Weiyang Liu, Adrian Weller, and Bernhard Schölkopf. Orthogonal finetuning made scalable. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 31 31946-31963, 2025.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
13

---

Published as a conference paper at ICLR 2026
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI,P-JCNILP), pp. 4463-4475 2019.
Weixi Song, Zuchao Li, Lefei Zhang, hai zhao, and Bo Du. Sparse is enough in fine-tuning pre-trained large language models. In Forty-first International Conference on Machine Learning, 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-10671, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Zhu JianHao, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine- tuning via representation editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13445-13464, 2024a.
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. Advances in Neural Information Processing Systems, 0006663988669666 2024b.
Jing Xu and Jingzhao Zhang. Random masking finds winning tickets for parameter efficient fine- tuning. In Forty-first International Conference on Machine Learning, 2024.
Shih yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 1-9, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 111.00111.0 2019.
Fangzhao Zhang and Mert Pilanci. Spectral adapter: Fine-tuning in spectral space. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference on Learning Representations, 2023.
14

---

Published as a conference paper at ICLR 2026
ORGANIZATION OF THE APPENDIX
The appendix is organized as follows:
Appendix
introduces the algorithm of the proposed PSOFT.
Appendix
provides the theoretical proof for the column-wise angle and norm preservation
theorem.
Appendix
presents theoretical details of the Cayley parameterization.
Appendix
compares the number of trainable parameters across popular PEFT methods.
Appendix
analyzes activation memory statistics for different PEFT methods.
Appendix
outlines experimental details for natural language understanding on GLUE.
Appendix
covers experimental details for visual classification on VTAB-IK.
Appendix reports experimental details for mathematical question answering on MetaMathQA-
40K.
Appendix describes experimental details for commonsense reasoning Commonsense- 155K.
Appendix details extended experiments on the effects of SVD initialization, different rank settings, inserted modules, and Neumann terms.
Appendix K illustrates the angular structure of the weight changes before and after fine-tuning.
Appendix analyzes the difference between PSOFT and full-space OFT in terms of their optimiza- tion dynamics and training loss trajectories.
Appendix M provides the additional memory usage experiments covering single linear layer, Transformer block, and end-to-end models.
Appendix explains the use large language models this paper.
A ALGORITHM OF THE PROPOSED PSOFT
For completeness, provide detailed description of the proposed PSOFT framework, which corresponds to Algorithm [ For initialization, the orthogonal matrix R is set to the identity matrix I,,, while PSOFT further introduces two additional vectors, a and [3, both initialized Before training begins. singular value decomposition (SVD) is performed once to extract the top-r singular values and vectors, which are then used construct the matrices A', B', and the residual weights W During training, the forward computation follows Eq. 8 and the gradients of both R and the vectors a and B3 are updated jointly to obtain the final weights Winal-
all ones.
Algorithm PSOFT: orthogonal fine-tuning in the principal subspace
Input: Pre-trained weight matrix pee E _d rank r, input æ, and number of epochs E
2: Output: Fine-tuned orthogonal matrix R, two vectors a and B3, and final weight matrix W
Winal
3: Initialize: Orthogonal matrix: I,, two vectors: +- 1,, 1,
4: Pre-compute:
Wiss +- Str.r) (ur) 5: Wpe pme = USVT A' Uprr B' Sprar) VT bun)
6: for epoch 1 to E
7: for each mini-batch do
8:
= (A' diag(a) R diag B' + W.S) Wes) x,
9: compute 2R da ag.
0C DC ac then update R-n R
DC
a t- a -
30
0c
B
8g
10: end for
end for
12:
Reconstruct: W Winal f A' A'diag(o) Rdaag W
B PROOF FOR THE ANGLE AND NORM PRESERVATION THEOREM
Theorem B.1 (Formal: Column wise angle and norm preservation the low-rank subspace). Let
W pri AB dd and W prinnned ARB E d with A E Rdxr, B E Brxn
Assume
15

---

Published as a conference paper at ICLR 2026
rank(A) rank(B)) and every column b; (so all angles well-defined). Let G == A
G is symmetric positive definite, w.. Ab, w w.ttiuueed A ARb, and denote by Ohg (resp. Opg-tuned
the angle between w. ,h." (resp. w. w.ttmedd = w. Then
G = (Vi 033 gps-tumed (Pgg)) and (Vi, |uug |
pu-tuedd ||
| (1)
Proof. For any pair of column indices + j. the cosines of the angles between the vectors in principal weights (w,,", w and the vectors in fine-tuned weights (wo ps-tuned
pa-tuned, are
b. Gb,
b R GRb,
cos 0 Moreover, for any
cos Ope-tuned
Vb. Gb. V GGb,,
b R GRb, 0J GRG;
b; Gb;
[ 1 = by. GRb
Sufficiency. If R GR G, then the two cosine expressions coincide for every + j. hence
Oig Ops-tumed cos 0.00 Since all angles lie in [0, æ] where the cosine is strictly decreasing, obtain
gps-tumed 0.5 Similarly, || || by Gb: ||,,,"11 || so | ||,,5"||
Necessity, Conversely, assume that 0g) 0pe-tuned o for all i * j and || = || for all i.
Define M R GR G. From norm preservation we obtain
6, Mb; = O, Vi,
Since b; * 0 and G 0, both denominators in the cosine formulas equal and positive; hence angle preservation implies
b._ Mb, = :
Thus B = 0 with rank B) = r. Because B has full row rank, it admits a right inverse CE Raxr (e.g.. C B BBBBB
such that BC I,. Multiplying gives
M - - MB)C=
hence RTGRE. GR G.
C CAYLEY PARAMETERIZATION
The Cayley parameterization (Cayley 1894) is mapping that converts real skew-symmetric matrices into orthogonal matrices. For real -kem-symmettic matrix Q (i.e. Q" -Q),, the Cayley transform is defined as:
R QI QQ)1
where I is the identity matrix of the same size Q and matrix R does not have -1 an eigenvalue.
The Cayley transform provides way parameterize orthogonal matrices near the identity matrix using skew-symmetric matrices. The orthogonality of the Cayley transform is proved as follows.
Theorem C.1. If Q is a real skew-symmetric matrix and (I + Q) is invertible, then the Cayley transform R (I (I Q) is an orthogonal matrix.
Proof. We aim to proof that the matrix R after Cayley transform satisfies R R= RR = =I.
To compute R_T R:
RJ R = -
=((+ (1-)))0))110)0--1)1)0--00000000-
-[ (I--))++++""
16

---

Published as a conference paper at ICLR 2026
By the definition of skew-symmetry, Q" =Q
(I (4-Q-14-044-0(+++
Since (I Q) and (I Q) are commute, we can switch the order of the factors:
= -1-0)"140)(++0
Similarly, it can be proven that RR I. Therefore, the result of Cayley transform R (I (I-Q)(1 + Q)- an orthogonal matrix.
In this paper, PSOFT leverages the Cayley parameterization to construct orthogonal matrices with approximately half the number of trainable parameters compared full orthogonal matrix, while rigorously preserving orthogonality.
D COMPARISON OF TRAINABLE PARAMETERS FOR PEFT METHODS
Table reports the number of trainable parameters across representative PEFT methods. Most existing approaches scale their parameter counts with hidden layer dimensions, which constrains their applicability to larger models. In contrast, PSOFT and LORA-XS decouple the number of trainable parameters from layer width. PSOFT further reduces parameter complexity through the Cayley parameterization, which requires only r(r 1)/2 parameters represent orthogonal matrix. Consequently, the total number of trainable parameters in PSOFT remains fixed for a given rank r, allowing fine-graine one-grained control over parameter budgets. Moreover, PSOFT introduces two learnable scaling vectors within the subspace, contributing merely 2r additional parameters, which is negligible compared with other methods.
Table Comparison of trainable parameters for different PEFT methods within a single linear layer, assuming input/output dimensions d and n, respectively. Here, denotes the low-rank dimension, m the number of butterfly factors in BOFT, the block size in BOFT, d'min min(d, n), and the number of additional off-diagonals in SVFT. All statistics are based on implementations from the HuggingFace' ' PEFT library (Mangrulkar et al. 2022).
Method
Number of Trainable Parameters
LoRA
dxr+rxn x r
DoRA
VeRA
r F x (d/r) ) (d/r) +
OFT
BOFT
m X (d/b) x b² +
SVFT
dmin x k: + (dmin - k): +
LoRA-XS
PSOFT (Ours)
r(r 1) 2r
E THE ACTIVATION MEMORY STATISTICS ACROSS DIFFERENT PEFT METHODS
In this section, we analyze the activation memory requirements of various PEFT methods during fine-tuning. In transformer-basel networks, memory usage primarily arises from three sources: pre-trained weight storage, activation storage, and gradient/optimizer state storage. Activation storage refers to intermediate values created during the forward pass and retained for gradient computation during backpropagation. Different PEFT methods consume comparable amounts of memory for weights, gradients, and optimizer states, as they all involve substantially reduced number of trainable parameters (Hu al. 2022 Batazy et al. 2024 Kopiczko et al., 2024). In contrast, their activation memory consumption exhibits clear differences. As the batch size increases. activation storage gradually becomes the dominant memory bottleneck, illustrated in Figure 4a
17

---

Published as a conference paper at ICLR 2026
Notably, activation memory in transformer layers accounts for over 99.9% of the total activation
memory across all layers (Korthikanti et al. 2023). We therefore focus our analysis on the activation storage of transformer layers.
Input
Query
M
MatMul
So
Goop
MatMuu
Output
Dropout
LayerNorm
FFN1
GELU
FFN2
Dropout
LayerNorm
Output
Key
Value
Self Attention
FFN
Attention
Transformer Layer
Figure 5: The architecture of a single transformer layer, including the attention layer and the feed
forward network layer and self attention layer.
In this study, we consider the transformer layers within an encoder or decoder, where the input has dimensions b X x h, where b denotes the micro-batch size, represents the maximum sequence length, and indicates the hidden dimension size. Each transformer layer consists a self-attention layer with attention heads, and in the feed-forward network (FFN) layer, the hidden dimension is expended to 4h before being projected back to h. We assume that activations are stored in 32-bit floating- boating--pintt format, requiring 4 bytes of memory. All results in this section reported in bytes unless otherwise specified.
4bsh
8bsh
4abs
obs
4bsh
4bsh 4bsh
16bsh
16bsh
46sh
Query
Mat
Softmax
Croport
4obs
LayerNorm
LayerNorm
Input
M MatMun
Output
Dropoul
FFN1
GELU
FFN2
Dropout
Output
Key
Value
4bsh
bsh
bsh
Figure Activation memory statistics in a single transformer layer for full fine-tuning.
As illustrated in Figure each transformer layer consists of a self-attention block (including Query, Key, and Value matrices) combined with output linear layer form the attention block. Addi- tionally. it includes two FFN layers, two normalization layers, and three dropout layers. Building on prior work (Korthikanti et al. )023), we derive an approximate formula for the activation memory required during the forward pass of single transformer layer. For backpropagation, we consider the input of each module (which serves as the output for the subsequent module) as activations. As illustrated in Figure 6 the activation memory includes the following components:
Self-Attention:
Query (Q), Key (K), and Value (V) matrices: Require 4bsh for their shared inputs.
First MatMul: Requires 8hsh as input to the module.
Softmax: Requires 4abs² for activation storage.
Self-attention dropout: Only the mask is stored, with size of abs?.
Second MatMul: Requires activations from the output of dropout (4abs2) and linear layer Value (4bsh), totaling 4abs² 4bsh.
Attention:
Output linear layer: Requires 4bsh as input.
Attention dropout: Only the mask stored, with a size of bsh.
First layer normalization: Requires 4bsh for activation storage.
FFN:
FFN1: Requires 4bsh input.
18

---

Published as a conference paper at ICLR 2026
Absr
4bsh
4bsr
4bsh
4bsh
40sh
LORA A
LORA
DoRA A
DoRA B
DoRA Mag
VeRA A
VeRA D
VeRA B
VeRA
OFT_R
OFT
BOFT R1
BOFT Rm
BOFT_S
Query
Query
Query
Query
Query
LoRA
DoRA
VeRA
OFT
BOFT
4bsh
4bsr
4bst
GOFT
GOFT
GOFT
SV
SVFT
SVFT B
LORA-XS
LORA-XS
LORA-XS B
PSOFT_
PSOFT Alpha
PSOFT
PSOFT Beta
PSOFT B
Query
Query
Query
Query-Res
GOFT
SVFT
LoRA-XS
PSOFT
Figure 7: Activation memory statistics in single linear layer (Query) across different PEFT methods.
GELU activation: Requires 16bsh for activation storage.
FFN2: Requires 16bsh as input.
FFN dropout: Only the mask is stored, with size of bsh.
Second layer normalization: Requires 4bsh for activation storage.
Summing these sub-layers, the total activation storage for a single transformer layer is:
ACT base = 66bsh + 9abs
(10)
The six linear layers within transformer layer undergo changes in activation memory storage when different PEFT methods are applied, as reflected by modifications to the base formula (ACTbase) For example, the LoRA method introduces set of low-rank matrices B and A in parallel. The activation memory requirements for various PEFT methods in a single linear layer are summarized in Figure 7 with the Query matrix representative example. The specific details of these changes are as follows:
LoRA: Adds 4bsr to the original activation storage for gradient computation during back- propagation.
DoRA: Adds 4bsr + 4bsh to the original activation storage.
VeRA: Replaces the original input 4bsh with 4bsr and adds 4bsh for activation storage.
OFT: Adds 4bsh to original activation storage.
BOFT: Requires an additional 4mbsh, where m the number of sparse matrices.
GOFT: Adds 4bsh log where h is the hidden layer dimension.
SVFT: Removes the original input activation storage and adds 4bsh.
LoRA-XS: Removes the original input activation storage and adds 4bsr.
PSOFT: Removes the original input activation storage and adds 12bsr.
The activation memory requirements various PEFT methods for single transformer layer are summarized in Table 9 Notably, PSOFT incurs significantly lower activation memory than all other methods except LoRA-XS. Its activation memory is comparable to that of LoRA-XS, as the rank r is much smaller than the hidden dimension h (r < ht). A key observation is that PSOFT employs scale vectors to enhance -ask-specific flexibility, similar to other orthogonal fine-tuning methods (Qiu et al. 2023 Liu al. 2024 Ma et al. 2024). However, unlike these methods, PSOFT applies the scale vectors within a principal subspace, effectively preventing a substantial increase in activation memory usage.
19

---

Published as a conference paper at ICLR 2026
Table Total activation memory statistics in single transformer layer for different PEFT methods and FFT. In BOFT, m denotes the number of sparse matrices.
Methods
Activation memory (Relative)
Activation memory (Absolute)
FFT
ACTbuse
66bsh 9abs²
LoRA
ACTbuse
24bsr
66bsh + 24bsr 9abs
DoRA
A CTaase 24bsr + 36bsh
102bsh + 24bsr + 9ahs
VeRA
ACTbase 28bsh 16bsr + 36bsh
74bsh 16bsr 9abs?
OFT
A ACTbuse 36bsh
102bsh + 9abss
BOFT
AC Coaaa 36mbss
66bsh + 336mbsh + 9abs
GOFT
ACTbasn base 36bsh log
66bsh 36bsh log h 9abs
SVET
A ACTbase 28bsh + 24bsh
62bsh + 9abss
LoRA-XS PSOFT
AC Aase
28bsh + 24bsr
38bsh + 24bsr 9abs²
ACTbase
28bsh 72bsr
38bsh 72bsr 9abs
F
NATURAL LANGUAGE UNDERSTANDING ON GLUE
DATASETS
The General Language Understanding Evaluation (GLUE) (Wang al. 2019) comprehensive
benchmark for evaluating the performance of natural language understanding (NLU) models across diverse tasks. It includes one similarity task (SST-B), five pairwise text classification tasks (MNLI, RTE, QQP, MRPC, and QNLI), and two single-sentence classification tasks (CoLA and SST).
Table 10: Hyperparameter settings for fine-tuning DeBERTaV3-base GLUE
Hyperparameter
CoLA
STS-B
MRPC
RTE
SST-2
QNLI
Optimizer
AdamW
Warmup Ratio
0.1
LR Schedule
Linear
Learning Rate (Head)
5E-04
Batch Size
32
Max Seq. Len.
64
128
256
256
128
256
#Epochs
20
20
30
30
10
LR PSOFT,216
6E-04
4E-04
4E-04
4E-04
2E-04
4E-04
F.2
IMPLEMENTATION DETAILS
While it is common in prior PEFT studies (Hu et al., 2022 Lingam et al. 2024 yang Liu al. 2024 Meng et al. 2024) to report results on the GLUE val validation set, concerns have been raised regarding the reliability of this protocol (Wu al. 2024a Bini al. 2025). To ensure a more rigorous evaluation, we evenly split the original validation set into new validation and test subsets using a fixed random seed. All reported results are based on the set, with checkpoints selected according to the best accuracy on the new validation set. Given the prohibitive computational cost of evaluating every baseline across all GLUE datasets, omit the two largest subsets MNLI and QQP) from our experiments. The peak memory usage during training is measured using torch. cuda max_memory_al
loaated ().
All experiments are implemented on top of the open-source LoRA framework (Hu et al. 2022). using PyTorch (Paszke et al.) 2019) and Huggingface's s PEFT library (Mangrulkar et al. 2022) Following Liu et al. (2024), training epochs. Due to resource constraints, we set the maximum sequence length to 256. PSOFT is applied to all linear layers of the DeBERTaV3-bars model. Evaluation metrics include Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for the other GLUE sub-tasks. Detailed hyperparameter configurations provided in Table T0
tune only model-agnostic hyperparameters such as learning rate and
20

---

Published as a conference paper at ICLR 2026
G VISUAL CLASSIFICATION ON VTAB-1K
G.1 DATASETS
The Visual Task Adaptation Benchmark (VTAB-IK) (Zhai et al. 2019) comprises 19 image classifi- cation tasks grouped into three categories: natural, specialized, and structured.
Natural tasks involve images captured with standard cameras, depicting scenes from the natural environment, generic objects, fine-grained categories, or abstract concepts.
Specialized tasks use images obtained through specialized equipment, such medical imaging devices or remote sensing technologies.
Structured tasks focus on artificially designed scenarios to analyze specific relationships or changes between images, such as estimating object distances in 3D scenes (e.g.. DMLab), counting objects (e.g., CLEVR), or detecting orientations (e.g.. dSprites for disentangled representations).
In VTAB-IK, IK, each dataset provides 800 labeled samples from its original training set, which are used to fine-tune the base model. Additionally. 200 labeled samples in the validation set adjust hyperparameters during fine-tuning. The performance is evaluated using Top-1 classification accuracy on the respective original test set.
Table 11: Hyperparameter settings for fine-tuning ViT-B/16 on VTAB-IK
Hyperparameter
ViT-B/16
Optimizer
Adam W
Warmup Ratio
0.1
LR Schedule
Cosine
Learning Rate (Head)
5E-03
Batch Size
64
Weight Decay
IE-03
Dropout
IE-01
#Epochs
50
LR PSOFT,
(5E-04, 1E-03, SE-033
G.2 IMPLEMENTATION DETAILS
Our experiments are conducted in PyTorch (Paszke et al. 2019) using HuggingFace' Datasets, Transformed and PEFT (Mangrulkar al.] 2022) libraries. Unlike prior works that rely on the Timm framework with custom preprocessing and training loops (Liu et al. 2024 Ma et al. and Trainer, eliminating manual dataset/model handling and enabling fast integration of advanced methods (e.g..
2024), our framework leverages standardized APIs such as Aut o www.ammeaaeeeseee.oe.oeeeee
DoRA (yang Liu et al.) 2024). SVFT (Lingam al. 2024), BOFT (Liu et 2024)).
We adopt the experimental settings from (Liu et al. 2024, Ma et al. 2024). adjusting learning rates, weight decay, and training epochs accordingly. Following (Balazy et al. 2024 Kopiczko et al. 2024 Lingam et al. 2024), we separate learning rates for the classification head and PEFT modules, with fixed learning rate applied to the head across all methods. Complete hyperparameter configurations are listed in Table []]
H MATHEMATICAL QUESTION ANSWERING ON METAMATHQA-40K
H.1 DATASETS
For mathematical question answering tasks, fine-tune baselines using the MetaMathQAA-00K dataset (Yu et al. 2024) and evaluate their performance on the two challenge benchmarks: GSM-8K (Cobbe et al. 2021 and MATH (Hendrycks et al. 2021).
21

---

Published as a conference paper at ICLR 2026
Table 12: Hyperparameter settings for fine-tuning on MetaMathQA-40K
Hyperparameter
LLaMA-3.2-3B
Optimizer
AdamW
Warmup Ratio
0.1
LR Schedule
Cosine
Max Seq. Len.
512
Batch Size
64
# Epochs
2
LR 150
4E-04
LR PS0FT
2E-04
H.2 IMPLEMENTATION DETAILS
Our experiments follow prior work (Liu et al. 2024 Lingam et al.) 2024) and are implemented
in PyTorch (Paszke et al. 2019) using HuggingFace's PEFT library (Mangrulkar et al. 2022). Consistent with (Lingam et al. 20244, we tune only learning rates for different models, with full hyperparameters listed in Table 12 We adopt gradient accumulation with small batch sizes (S to approximate large-batch training across all baselines.
Table 13: Experimental results of fine-tuned LLaMA-3.2-30 GSM-8K and MATH with extremely low parameter counts. The best result for each dataset is marked in bold. Accuracy (%) is reported for both GSM-8K and MATH datasets.
Methods
#Params
Inserted Modules
Mem (GB)
GSM-8K
MATH
GOFTv2
0.26M
Q.K.V
75.3
41.02
9.22
gGOFTv2
1.03M
Q.K.V
75.3
42.46
9.32
BOFT
1.8MM
Q.K.V
48.2
52.46
10.78
PSOFT
1.20M
Q.K.V
29.8
52.84
12.24
LoRA-=I
0.40M
Q,K.V
30.1
47.23
10.36
SVFT
0.49M
O.K.V.UDDDO.
41.1
52.01
12.18
Lo
0.45M
O.K.V.UDD.OG
32.3
$1.86
9.80
PSOFT,
0.53M
Q.K.VUUDDD..
32.7
52.01
12.44
LoRA
1.52M
O.K.VUUDD.OG
32.0
$7.32
12.88
PiSSA
1.52M
O.K.VUUD...G
32.0
56.48
13.18
LORA-XS,=88
1.52M
O.K.V.UDD.OG
32.8
$4.66
12.70
PSOFT,
1.54M
O.K.V.U.D.OGG
33.2
$7.47
13.26
DoRA,
2.29M
O.K.V.U.D.OG
43,2
57.54
13.60
PSOFT, 1552
2.31M
Q.K.V.U...OGG
33.5
58.23
13.66
Beyond the main experiments, provide additional evaluations of PEFT methods under constrained parameter budgets, as summarized in Table [3 When fine-tuned on the Q. K, and modules, PSOFT achieves 10% and 3% higher accuracy than GOFTv2 and qGOFTv2 on GSM-8K and MATH, respectively, while using only 40% of their memory. On MATH, PSOFT also exceeds BOFT by 0.82%/1.46% with just 60% of its memory usage.
PSOFT allows flexible control of parameter counts by adjusting the rank r. whereas LoRA is restricted to minimum rank of 1, inherently tying its parameter count to hidden dimension size. Under stricter parameter budgets, LoRA must reduce the scope of inserted modules, often leading to performance degradation. In contrast, PSOFT consistently achieves superior performance even at extremely low parameter configurations. In terms of memory efficiency, PSOFT matches LoRA while outperforming DoRA and SVFT.
22

---

Published as a conference paper at ICLR 2026
1 COMMONSENSE REASONING ON COMMONSENSE- 15K
I.1 DATASETS
Commonsense reasoning benchmarks encompass eight distinct sub-tasks: BoolQ (Clark et al. 2019
PIQA (Bisk et al., 2020), SIQA (Sap et al. 2019). HellaSwa wag (Zellers et al., 2019), Winogrande Clark et al. 2018), and OpenBookQA (Mihaylov
(Sakaguchi al. 2.21), ARC-easy/RRC-challeee
et al. 2018). Following the approach described in (Hu et al. 2023 Lingam et et. [
(202)) yang Liu et al.
2024). we also combine the training datasets from all eight tasks to construct unified fine-tuning
dataset, Commonsense- 15K tailored for each task.
Table 14: Hyperparameter settings for fine-tuning on Commonsense-lle
Hyperparameter
LLaMA-3.1-BB
Optimizer
AdamW
Warmup Steps
100
LR Schedule
Linear
Max Seq. Len.
512
Batch Size
64
# Epochs
3
LR PSOFT,
4E-04
LR PSOFT,=
IE-04
Table 15: Experimental results of fine-tuned LLaMA-3.1-88 on eight commonsense reasoning benchmarks with extremely low parameter counts. The best average result is highlighted in bold. Accuracy (%) is reported for all sub-datasets.
Inserted Modules
Mem BoolQ PIQA SIQA HS WG ARC- OBQA
Methods
#Params
Avg.
(GB)
GOFTv2
0.26M
Q.V
OOM
N/A.
qGOFTv2
1.05M
Q
OOM
N/A.
BOFT
1,21M
Q.V
79.4
69.66 83.95
71.65 80.87 70.01
90.40
77.82
79.00
77.92
PSOFT 194
1.22M
Q
52.6
68.87
84.17 71.44 86.46
67.56
90.45
77.73
81.20
78.49
LoRA
0.59M
Q.K.V
52.8
66.97 83.08 71.03 77.06 64.01
90.70
77.39
78.80
76.13
SVFTP
0.46M
Q.K.V.U.D
65.8
65.08 81.07 69.40 85.69 68.82
88.47
77.05
76.00
76.45
LORA-XS,=48
0.37M
Q.K.V.U,D
53.4
69.30
84.82 71.29 87.44 67.01
89.39
77.22
82.60
78.63
PSOFT
0.43M
Q.K.V.U.D
53.7
69.72
84.39 72.01 87.99 68.67
90.19
78.16
81.00
79.02
LoRA,=1
1.77M
Q.K.V.UDD
53.9 71.13 85.31 74.67 89.08 72.61
90.24
78.16
82.40
80.45
PISSA
1.77M
Q.K.V.U.D
53.9 72.05 84.60 74.21 89.93 70.88
90.15
79.01
82.00
80.35
LORA-XS, XS,
1.73M
Q.K.V.U.D
54.0 71.04 85.47 72.67 71.74 90.82
79.61
83.20
80.48
PSOFT 1.48
1.74M
Q.K.V.U.D
54.5 71.31 85.69 73.18 89.38 72.38 90.91
80.03
83.00
80.74
DoRA
2.56M
Q.K.V.UD
65.4
71.05 85.29
73.25 90.09
73.32
90.74
79.75
81.87
80.67
PSOFT
2.52M
Q.K.V.UDD 55.0
71.47
86.02 75.33 90.81 72.69
90.45
78.75
84.00
81.19
I.2
IMPLEMENTATION DETAILS
The experiments are conducted following the frameworks Hu et al. (2023); yang Liu al. (2024). J implemented in PyTorch (Paszke et al. 2019) with HuggingFace PEFT library (Mangrulkar et al., 2022). Consistent with Lingam et al. (2024), we tune only the learning rates for different models. Detailed hyperparameter configurations
provided in Table 14
As shown in Table 15 when fine-tuning the Q and modules, PSOFT avoids the OOM failures observed in GOFT and gGOFT, and surpasses BOFT by 0.33% in average accuracy while using only 66% of its peak memory. We further evaluate under more constrained parameter budgets, where PSOFT continues to deliver superior average accuracy across eight commonsense reasoning
23

---

Published as a conference paper at ICLR 2026
benchmarks. In terms of memory efficiency, PSOFT requires only about 80% of the memory of DoRA and SVFT, while remaining comparable to LoRA.
EXTENSION EXPERIMENTS
J.1 EFFECT OF SVD INITIALIZATION
Table 16: The effect of SVD Initialization on the Commonsense-155 15K Dataset using the LLaMA-3.2
3B model.
Methods
SVD niiter
SVD Init Time
Validation Loss
2.79
0,9343
10
3.74
0,9328
PSOFT 5=
20
4.84
0,9283
89.68
0.9276
4.11
0,9174
10
5.13
0.9134
PSOFT,
7.51
0.9157
89.48
0.9147
6.33
0.9092
10
8.38
0.9028
PSOFT,=128
20
13.01
0.9029
90.50
0.8992
PSOFT constructs the principal subspace via SVD, where the initialization time and accuracy of fast SVD depend the n-iter parameter (Halko et al. M et al.) 2024). We evaluate this the Commonsense- 15K dataset (Hu et 2003) using the LLaMA-3.2-33 model (Meta Al 2024), reporting both initialization time and validation loss. As shown in Table 16 smaller n_iter values yield faster initialization, while larger values improve accuracy. With n iierrr- = 20, the loss is nearly identical to that of full SVD (n (n-iter - oo). These results show that fast SVD initializes PSOFT within seconds, and even full SVD introduces negligible overhead relative to the total fine-tuning time.
Table 17: Effects of different ranks fine-tuned the CoLA Dataset using the DeBERTA-V3-bas model (on a single RTX5090).
Matthew's Correlation %)
Peak GPU
Methods
Ranks
#Params
Runtime
Memory (GB)
144
59.20
4,0
17m34s
2
360
68.80
4,0
18m32s
4
1,008
70.08
4.0
19m17s
3,168
70.93
4,0
19m08s
10,944
68.36
4.0
19m32s
PSOFT
32
40,320
72.09
4.0
19m41s
64
154,368
69.16
4.1
21m29s
128
603,648
72.46
4.2
20m42s
2,386.944
74.09
4.6
24m35s
512
9,492,480
71.04
5.8
27m20s
J.2 EFFECT OF RANKS
To provide guidance on rank selection, we evaluate PSOFT with ranks ranging from 1 to 512 on the CoLA and the Commonsense- 15K dataset (Hu et al. 2023 using DeBERTA- V3-base (He et al.) (Meta AI 2024). As shown Table 17 and Table 18 PSOFT exhibits wide range of usable ranks: as r increases, the number of trainable parameters grows according to the formula in g r(r 1)/2 2r, and performance improves correspondingly. though with diminishing returns. Memory usage increases with r. but remains nearly flat when r small. Since we adopt the truncated Neumann-series approximation, training time does increase noticeably with larger
2021) and LLaMA-3.2-23
24

---

Published as a conference paper at ICLR 2026
Table 18: Effects of different ranks fine-tuned on the Commonsense-155 15K Dataset using the LLaMA-
3.2-3B model (on single H100).
Peak GPU Memory (GB)
Methods
Ranks
#Params
Avg. (%)
Runtime
392
27.07
31.5
50m13s
2
980
32,45
31.5
46m37s
2,744
36.16
31.5
48m30s
8,624
38,21
31.5
46m18s
29,792
57.12
31.6
48m52s
PSOFT
32
109,760
62.94
31.8
51m12s
64
420,244
70.95
32.1
48m47s
128
1,643,264
73.90
32.8
46ml1s
256
6,497.792
74.95
34.5
47m29s
512
25,840,640
75.05
38.4
49m49s
The results further reveal consistent pattern across models and tasks. For smaller models and simpler tasks, PSOFT is highly parameter-efficient even very small ranks achieve strong performance, indicating that the low-dimensional subspace is already sufficient specific transformations. In contrast, for larger models and more complex tasks, performance tends to increase with larger ranks, reflecting the greater capacity required capture task-specific transformations. In such cases, the main trade-off is between the performance gains from increasing r and the corresponding growth in trainable parameters.
capture the necessary task-
Based these observations, we provide the following practical guidance for choosing the rank. For simpler tasks, we recommend using small to moderate ranks (e.g.. 32-128), as they provide good parameter efficiency with little performance loss. For more complex tasks, larger ranks generally lead to higher performance, while extremely small ranks (e.g, below 16) may hurt results. In such cases, moderate large ranks (e.g., 64-256) offer better balance between performance and efficiency.
J.3 EFFECT OF INSERTED MODULES
We fine-tune LLaMA-3.2-3B with PSOFT and evaluate on GSM-8K under different insertion schemes, with results shown in Figure Sa Overall, performance improves as more modules are inserted and as the rank increases, showing that complex mathematical tasks benefit directly from higher model capacity under PSOFT. For fixed rank r, applying PSOFT to the Q, K, V, U, and D modules generally provides the best trade-off between performance and parameter efficiency. When the parameter budget permits, inserting PSOFT into all linear layers yields the strongest results.
60
91.6
1.4
Pearson correlation
Neumann
1.3
91.5
100
G.V.G.
1.2
65
91.4
1.1
Training
Rork meo
w/o Neumann
1.0
91.3
0.9
35
S0K
200K
10
Number ameters
PSOFT with different Neumann Terms
(a)
(b)
Figure 8: (a) Effect inserted modules on GSM-8K using LLaMA-3.2-3B. (b) Effect of Neumann terms on STS-B using DeBERTaV3-base
J.4 EFFECT OF NEUMANN TERMS
To assess the effect of different Neumann terms on training speed and performance, we fine-tune DeBERTaV3.. STS-B with rank 46. As shown in Figure 3b the Neumann series approxima- tion substantially accelerates training while maintaining performance close to the original Cayley parameterization. Training speed decreases the number of terms increases, gradually approaching
25

---

Published as a conference paper at ICLR 2026
that of Cayley, whereas performance improves with more terms and eventually converges to the Cayley result.
K PAIRWISE ANGLES OF WEIGHTS
We fine-tune DeBERTa-V3-bonn on the CoLA dataset using the same setup as in the main paper. We then extract the query matrix from layer 6 and compute the pairwise angles among the first eight column vectors of Wp and Wre. as well as those of W pe-tuned and Winal= = Wps-tuedd + Wies- Figures 9a and 10a show that, before fine-tuning, the angles in W pri and W pre follow clear and stable pattern. Figures 95 and show that PSOFT with strict orthogonality keeps this pattern: W ps-tuned preserves the angles in W pei. and W preserves those in W Wre As shown in Figures TOb and [0c| PSOFT with relaxed orthogonality also keeps the main angular structure, but introduces small and controlled changes. These changes help improve task adaptation while keeping the key structure intact.
120
120
120
100
7767
100
100
2
101.111 99
53.1
1
95.3
055
80
101.1 96 09
F3.1 67
65.3
05
80
2
1017705.
0.
93.0 867
000
95.
107
80
3 -20.0 25:2 33.1
5.
420
52.5
112.1
$4.5
3 20.0
26 25.1
50
420
82.8
12:1
34.5
-90.0
28.5 93.9
L20
83
211.7
85.0
60
60
60
87.3
92.5
-204
34.9 87.310558
440
40
40
08.9 0..9 24:1
05.3212.155
07.8
20
88.0 14.1
25.3112.183
25.0
03.3
20
07.4
24.3 95.7111.754
54.2
07.2
20
7
72.8
80
105.7
22.4 05. 65.1
12.0
10.6100 105.7 BA.
92.4
65.3
80
61.7 79 107.3
85.0 22 615
61.7
00
0
1
5
6
2
3
0
5
6
(a) pn
(b) Wp potteeeed (strict orth.)
(c) W puttumed (relaxed orth.)
Figure 9: Angle structures of pri (the query matrix in layer 6) before fine-tuning (a), and W Wps-tuned after PSOFT fine-tuning under strict (b) and relaxed (c) orthogonality.
120
120
120
87.5
50.7
20.0
01.8
01.0
100
87.5
93.5 50.7
05.0
05.0
04.3
100
07.2
93.9
50.7
90.8
0 03.5 03.0
100
2
$6.3 93.5
$1.8 99
966
61.6
98.3
80
66.3 93.6
F1.8 59.8
06
88.6
80
2
96.6
93
$1.8 89.5
07.2
08.6 99
80
3
$5.6
91
55.8
40.7 91.8
65.7
3
85.6
25.8
60
60
60
15
30 89.8
9
40
40
40
67.9
88.
56.7
52.0
89.3
20
67.9 20 88.8 56.7 52.3
02.2
-0
00.33
82.9 88.8
16.5 52.5 09.1
-20
7
18.2
54.3 08.3
80.0
57.5
79.3
83.5
0.
7 18.2 54.5
18.3 80.0
57.5
19.3
85.5
0.
7
77.6
53.9
19.1
88.1
57.5
78.2 63.4
0
1
4
5
6
7
2
3
4
5
6
7
0
1
3 4
5 6
(a) Wire
(b) W imal (strict orth.)
(c) Wina (relaxed orth.)
Figure 10: Angle structures of Wpr (the query matrix in layer 6) before fine-tuning (a), and of W final after PSOFT fine-tuning under strict (b) and relaxed (c) orthogonality.
L LOSS AND CONVERGENCE COMPARISON
PSOFT can be viewed as a specialized form of orthogonal fine-tuning, where iinal R Wpee with Riull diag Iar)), meaning that the orthogonal transformation is applied only to the principal (low-rank) subspace of the pre-trained weight matrix, while an identity mapping is imposed on its orthogonal complement. This formulation implies that the optimization behavior of PSOFT gradually approaches that of full-space OFT methods as the rank r increases.
Therefore, PSOFT induces principled modification of the optimization geometry: Full-space OFT optimizes over the Stiefel manifold St,d, d), whose tangent space consists all skew-symmetric directions in the full d-dimensional parameter space. In contrast, PSOFT restricts optimization to
26

---

Published as a conference paper at ICLR 2026
the tangent space of block-diagonal submanifold St(r, @ R(d-r), As result, only the principal subspace receives curvature-awart updates, while the orthogonal complement experiences zero curvature (identity block).
PSOFT
0.5
PSOFT,
PSDFT,
0.4
--0
P5OFT
GOFT-V2
Loss
0.3
qGOFT-v2
OFTvL,
OFTV2: -22
0.2
BOFT
0.1
0.0
Epoch
Figure 11: Comparison of loss curves for different PSOFT ranks and various orthogonal fine-tuning methods.
Building upon this geometric distinction, PSOFT exhibits three complementary behaviors that characterize its optimization dynamics. First, the low-rank orthogonal constraint simplifies the optimization landscape by preventing large full-space orthogonal transformations. This restriction reduces the effective curvature of the optimization path, yielding more stable and predictable gradient updates, while at the same time limiting expressiveness when is very small. Second, because PSOFT applies orthogonal transformations only within the principal subspace, stochastic noise is confined to this lower-dimensional region rather than being amplified across all d dimensions in full-space OFT, leading to more robust and less destructive updates. Third, as r increases, the PSOFT tangent space increasingly approximates that of full-space OFT, supporting richer expressiveness and convergence trajectories that gradually approach full-space OFT, yet without the severe overfitting that may arise in full-space OFT. Collectively, these properties illustrate how PSOFT navigates the trade-off between stability, expressiveness, and generalization.
We conduct additional experiments the CoLA dataset using DeBERTa- V3 and report the training loss curves of different OFT variants. As shown in Figure 11 the green curves correspond to PSOFT, with darker colors indicating larger ranks. We observe that as r increases, the PSOFT loss curves progressively approach those of full-space OFT methods such as BOFT and OFTv2, reflecting the improved convergence speed and expressiveness of higher-rank subspaces. PSOFT with very small ranks constrains the update space too aggressively, which may lead to underfitting and slower loss reduction. contrast, full-space OFT methods such as BOFT BOFTadispla the fastest initial convergence, but their full-rank orthogonal updates raise the risk of overfitting. This phenomenon is evident in our main GLUE experiments, where BOFT achieves the lowest training loss yet fails to obtain the best generalization performance.
These trends are consistent with the geometric properties of PSOFT discussed above: by con- straining orthogonal updates to lower-dimensional principal subspace, PSOFT naturally balances expressiveness and generalization. Unlike full-space OFT, PSOFT enables explicit capacity control through r, allowing moderate ranks to achieve more favorable bias variine trade-off and stronger generalization.
M ADDITIONAL EXPERIMENTS ON MEMORY USAGE
we additionally conducted memory experiments single NVIDIA H100 80GB, covering:
the forward/backward (FP/BP) peak memory usage on a single custom linear layer, and
the forward/backwwrd (FP/BP) peak memory usage on a Transformer block, and the peak memory usage the DeBERTaV3-bass and ViT-B/16 models during training.
27

---

Published as a conference paper at ICLR 2026
For the single-layer analysis, implemented Python-basee evaluation framework that separately measures peak memory usage and runtime for the forward and backward passes. The implementation of GOFTv2 uses the latest available code, while BOFT taken from the PEFT library (version 0.17.0). We track peak memory consumption (in GB) and runtime (in milliseconds, ms), as peak memory is the primary factor limiting memory-constraing hardware. The linear layer input is configured with batch size b = 64, sequence length = 512, and hidden dimension h = 4096. Runtime results averaged over 100 forward/backward runs. The results are summarized as follows:
Table 19: Peak memory usage (GB) and runtime (ms) statistics for different methods single custom linear layer.
Methods
Peak Memory (FP)
Peak Memory (BP)
Runtime (FP)
Runtime (BP)
GOFTv2 gGOFTv2
13.6
14.3
5.2
129.3
13.6
14.3
5.4
129.6
BOFT m=
1.8
2.6
102.9
2.1
BOFT m
2.3
3.0
139.6
2.5
PSOFT
2.1
2.6
43.4
4.3
PSOFT
2.1
2.6
43.8
4.8
PSOFT 128
2.1
2.6
22.9
25.9
PSOFT 256
2.2
2.6
4.0
48.8
PSOFT r0012
2.2
2.7
5.6
53.1
As shown in although GOFTv2 benefits from the Hadamard-produtm optimization and achieves reduced forward-pass computation time, it still consumes substantially more activation memory than both BOFT and PSOFT. Importantly, the single-layer activation- memory measurement slightly underrepresents PSOFT's true advantage: discussed in the theoretical analysis, PSOFT reduces activation memory across multiple layers, but when evaluating single layer in isolation, it should still store the full input and output activations, which partially diminishes its advantage. Nevertheless, even under this conservative setting, PSOFT achieves lower activation-ment memmoy usage and faster computation compared with BOFT and GOFTv2, and its advantages become increasingly pronounced when moving from single linear layer to a Transformer block or end-to-end models.
aetivation-memory
Table 20: Peak memory usage (GB) and runtime (ms) statistics for different methods on Transformer block.
Methods
Peak Memory (FP)
Peak Memory (BP)
Runtime (FP)
Runtime (BP)
GOFTv2 gGOFTv2
65.4
65.4
49.5
667.1
65.4
65,4
49.5
671.2
BOFT pm
19.0
19.0
2813.9
7.5
BOFT m-
28.9
28.9
5427.9
8.7
PSOFT 32
7.2
7.2
162.7
134.4
PSOFT 64
7.2
7.2
166.0
134.2
PSOFT 128
7.2
7.3
137.4
170.3
PSOFT 256
7.3
7.4
122.2
197.7
PSOFT,=51 r512
7.6
7.6
130.3
215.3
To validate this, we extend the single-layer setup to a complete Transformer block, configured with 8 attention heads and with all PEFT modules inserted into all linear layers. The input configured with batch size b = 32, sequence length 512, and hidden dimension h 4096, and runtime results are averaged over 100 forward and backward runs. We report peak memory consumption (in GB) and runtime (in milliseconds, ms). As shown in 20, these block-level experiments confirm that PSOFT further reduces both memory usage and runtime by avoiding full-dimensional chained multiplications and performing orthogonal transformations only within much smaller subspace.
We then conduct full-layer experiments following the same configuration as the main paper. For DeBERTaV V3-base we use fixed batch size b 64 and and task-dependent sequence length s E 64, 128, 256. For ViT-B/16, we follow the original setup with a fixed sequence length s 197 and batch size of b = 64. Additionally, we include results with smaller batch sizes E 16, 32 for a more comprehensive comparison. PSOFT uses the same rank r = 46 as reported in the original paper, and all PEFT modules are inserted into all linear layers. The results are presented follows:
28

---

Published as a conference paper at ICLR 2026
Table 21: Peak memory usage (GB) of different methods on DeBERTaV3-base
Methods
Peak Memory (s=64)
Peak Memory (s=128)
Peak Memory (s=256)
GOFTv2 18.5
34.4
67.5
qGOFTv2 18.5
34.4
67.5
BOFT
6,3
9.4
17.5
PSOFT
4.1
6.8
14.0
Table 22: Peak memory usage (GB) of different methods on ViT-B/16.
Methods
Peak Memory (b=16)
Peak Memory (b=32)
Peak Memory (b=64)
GOFTv2
22.5
44.7
0OM
qGOFTv2
22.5
44.7
OOM
BOFT m=2
5,4
7.3
10.9
PSOFT
2.4
2.9
6.2
As shown in 21 and 22 Remarkably, even size b 64. This behavior stems from its activation-mennnn memory scaling of Obshh log h). which grows rapidly at larger batch sizes and ultimately limits its applicability contrast, PSOFT consistently avoids such OOM issues: by restricting OFT to the principal subspace, it preserves the essential semantic representations while simultaneously improving multi-dimensional efficiency (parameter counts, memory, and computation) for OFT.
PSOFT achieves the lowest peak memory usage across different settings. H100 GPU, GOFT still encounters OOM failures for ViT-B/16 with a batch
memory-constrained hardware. In
N THE USE OF LARGE LANGUAGE MODELS (LLMS)
this work, large language models (LLMs) are used solely as general-purpose tools to assist with writing polish. Specifically, LLMs are employed to refine grammar, improve readability, and ensure that the overall writing style conforms to academic conventions. LLMs are not involved research ideation, experimental design, data amalysis, conclusion formulation. All technical contributions, theoretical analyses, and experimental results are entirely original work by the authors.
29


Reviews:
[{"review_id": "OwCpCxTbP9", "reviewer": "Reviewer_rmrW", "summary": "This paper introduces PSOFT, a parameter-efficient finetuning (PEFT) approach that combines the low-rank decomposition and orthogonality principles from LoRA and OFT-style methods. Specifically, PSOFT first applies an SVD decomposition to the pretrained weight matrices, and then inserts a learnable rotation matrix between the decomposed components. The SVD decomposition enforces low-rank structure and orthogonality, while the rotation matrix provides flexibility for task-specific adaptation. Experimental results on both encoder-only and decoder-only models across multiple benchmarks demonstrate the effectiveness of PSOFT.", "questions": "The number of tunable parameters changes significantly from 0.08M to 12.2M while other methods like OFT do not change so much (from 1.4M to 11.6M).\nWhether it stems from differences in model architecture, rank choice, or implementation details?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea is conceptually elegant and well-motivated. Combining SVD-based low-rank decomposition with a learnable rotation matrix is a natural way to balance efficiency and expressiveness. The orthogonal projection helps preserve the representational capacity of the subspace, while the rotation matrix allows fine-grained adaptation to downstream tasks.\n\nThe paper is clearly written and easy to follow. The method is presented in a straightforward manner with comprehensive experimental validation across various settings.", "weaknesses": "1. The related work section lacks a clear comparison and differentiation between PSOFT and closely related methods such as **DoRA** and **LoRA-XS**. A more detailed discussion highlighting conceptual and empirical differences would strengthen the contribution.\n\n2. The reported ranks $r$ (e.g., 46, 354, 424) are irregular and seem to vary considerably. This suggests possible **sensitivity to hyperparameter tuning**, which should be verified through an ablation study on $r$.\n\n3. It appears that all results are produced via **re-implementations** rather than directly using reported numbers from prior works. While this ensures consistency, it would also be useful to include **direct comparisons under standardized settings** from previously published benchmarks to contextualize the gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8aliD4YS2C", "reviewer": "Reviewer_o81u", "summary": "This paper presents PSOFT, a novel parameter-efficient fine-tuning (PEFT) method that confines orthogonal fine-tuning to the principal subspace of pre-trained weights, identified via SVD. The goal is to merge the efficiency of low-rank adaptation (LoRA) with the semantic preservation of orthogonal fine-tuning (OFT). The method demonstrates a strong balance of accuracy, parameter efficiency, and memory usage across a wide range of NLP and CV tasks, often outperforming existing methods.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The core idea of applying orthogonal transformations within a low-rank principal subspace is an intuitive and effective way to bridge the gap between LoRA and OFT.\n* The comprehensive evaluation across 35 NLP and CV tasks shows strong performance. PSOFT is not only accurate but also highly parameter- and memory-efficient, crucially avoiding the out-of-memory (OOM) errors that plague other OFT variants.\n* The paper successfully highlights that efficiency is multi-dimensional. PSOFT shows clear advantages in memory footprint and training speed, making it a practical and scalable solution for large models.", "weaknesses": "1.  **Theory vs. Practice:** The paper emphasizes \"strict\" geometry preservation as a key benefit, yet the best-performing algorithm intentionally relaxes this condition with tunable vectors to improve results. Could you discuss the trade-off here and the impact of this relaxation on the semantic preservation you aim for?\n2.  **\"Effective Rank\" Definition:** The claim of a \"higher effective rank\" is based on a non-standard definition (`r_PSOFT = √M`). This is confusing and weakens the claim of higher expressiveness. Could you please clarify this definition and provide a more rigorous justification?\n3.  **Guidance on Hyperparameter `r`:** The method's performance is tied to the choice of rank `r`, but the paper lacks an analysis of its sensitivity or guidance on how to select it for different tasks or models.\n4.  **Implementation Clarity:**\n    *   The choice of K=5 for the Neumann series approximation is stated without justification. What is the impact of this choice on orthogonality and training speed?\n    *   Reporting performance in relative speedup (\"2.1x\") without absolute wall-clock times makes it difficult to assess the true computational overhead. Could you provide absolute training time comparisons?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15sPagSJko", "reviewer": "Reviewer_tLhH", "summary": "This paper introduces PSOFT, a PEFT algorithm for pretrained models, incorporating orthogonal fine-tuning strategy into additive fine-tuning framworks. Extensive experiments on several tasks validates the effectiveness of this method.", "questions": "See Weaknesses.\n\nSome further questions:\n1. Following on Weakness 2, is the original principal subspace always sufficient for downstream tasks? Can we reach a conclusion that the principal space is a better regularization of the entire low-rank subspaces? Is it possible that the diminishing returns at higher ranks (Table 17) are a symptom of this fixed-subspace limitation?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper rethinks the updates of fine-tuning within the principal space of the original weight matrix.\n\n2. The method demonstrates strong results, consistently outperforming LoRA, PiSSA, and other OFT variants (BOFT, GOFT) on a wide range of benchmarks, including GLUE, VTAB-1K, GSM-8K, and commonsense reasoning.\n\n3. PSOFT achieves its strong performance while using significantly fewer parameters than competitors. In Table 2, PSOFT and GOFT (0.08M) achieves the best average performance while being much more parameter-efficient than BOFT (1.41M) and LoRA (1.33M).", "weaknesses": "1. A misclaim of the core idea of the proposed algorithm: The key idea of \"orthogonal fine-tuning\" is keeping correlations between neurons (say inner products) unchanged after fine-tuning, thus providing semantic-preservation. However, this does not hold for PSOFT. In fact, PSOFT is actually an additive fine-tuning method similar to LoRA, $\\Delta W = W_{final} - W_{pre} = (A'RB' + W_{res}) - (A'B' + W_{res}) = A'(R-I)B'$, where the updates are restricted to principal subspace and enjoy a natural low-rank property. Yet, $(W_{pre} + \\Delta W)^\\top (W_{pre} + \\Delta W)$ is not equal to $W_{pre} ^ \\top W_{pre}$. Therefore, it is not correct to claim that this is an orthogonal fine-tuning method, and does not achieve the so-called semantic-preservation in Intro.\n\n2. What has PSOFT done?: Keep the orthogonal basis of the original pretrained weights unchanged, and adjust the main ranks of the entire space by adjusting the coordinates using orthogonal transformation. So the expressiveness is actually bounded by the identical orthogonal bases of the pretrained weights. Is this expressive enough? It seems that LoRA could span a larger subspace of the updates.\n\n3. Some doubts of the experimental results: The experiment results seem to show great improvements. But the OOM phenomena of GOFT seems weird and not convincing, as they incorporate only vector hadamard multiplications on activations to implement their forward, this also means the backward will not consume large matrix storage in GPUs. Moreover, BOFT incorporates matrix multiplications instead, and I think GOFT should be more memory-efficient than BOFT. I suggest conducting analyses on memory consumptions of their forward and backwards of a single layer through rigorous calculations and deductions (better with experimental evidence) to support your results. At the very least, as you indeed have NVIDIA H100 80GB for decoder experiments, why not conduct the OOM experiments on those chips?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ti12q13wt3", "reviewer": "Reviewer_UR1Z", "summary": "The paper proposes PSOFT (Principal Subspace Orthogonal Fine-Tuning), a novel and efficient orthogonal fine-tuning framework for large pretrained models. By constraining orthogonal transformations within the principal subspace of pretrained weights, PSOFT effectively balances geometric preservation, expressive capacity, and computational efficiency. The method employs SVD-based subspace extraction, Cayley parameterization for strict orthogonality, and learnable scaling factors for task adaptability.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. PSOFT introduces an elegant combination of orthogonal fine-tuning and low-rank adaptation through principal subspace projection, offering a fresh perspective on improving scaling efficiency.\n2. The paper provides a clear mathematical justification for subspace-constrained orthogonality and uses Cayley parameterization to guarantee exact geometric preservation.\n3.  Extensive experiments across diverse NLP and CV tasks show consistent improvements over LoRA and BOFT, with significantly lower memory consumption and stable performance on large models.", "weaknesses": "1.  The paper could analyze more systematically how the choice of principal subspace rank affects performance and efficiency.\n2.  While results on up to 8B models are shown, the paper lacks discussion on potential limitations or stability when scaling beyond that range.\n3. Are the principal subspaces learned independently for each layer, or do they share correlated bases across layers? If the latter, does PSOFT exploit inter-layer subspace alignment to further improve transferability?\n4. Beyond downstream accuracy, have the authors analyzed how PSOFT affects representation geometry—for instance, by measuring canonical correlation alignment (CCA) or subspace angles before and after fine-tuning?\n5. Orthogonal fine-tuning implicitly constrains updates on the Stiefel manifold. How does PSOFT’s subspace constraint modify the tangent space of optimization compared to full-space OFT, and what implications does it have for curvature and convergence speed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)