CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning
Authors: Yifei Zhang
Hao Zhu
Junhao Dong
Haoran Shi
Ziqiao Meng
Piotr Koniusz
Han Yu
Venue: NeurIPS 2025 poster ()
Keywords: Parameter-efficient fine-tuning (PEFT)
Primary area: deep_learning
TL;DR: How to employ the coherence between layers to design a PEFT method.

Abstract: Parameter-efficient fine-tuning (PEFT) is essential for adapting large foundation models without excessive storage cost. However, current approaches such as LoRA treat each layer’s adaptation independently, overlooking correlations across layers. This independence causes the number of trainable parameters to grow linearly with model depth. We provide theoretical and empirical evidence that skip connections in transformers create smooth gradient propagation across layers. This smoothness leads to weight adaptations that concentrate most of their energy in low-frequency spectral components, especially along the layer dimension. Empirical analysis confirms this effect, showing that most of adaptation energy lies in low frequencies. Building on this insight, we propose CrossSpectra, which parameterizes all attention-weight adaptations $(Q, K, V)$ across layers as a single 3D tensor and represents them with sparse spectral coefficients ($\kappa_1, \kappa_2$). Using $\kappa_{1}$ non-zero coefficients within each layer’s frequency space and truncating to $\kappa_{2}$ frequencies across layers, CrossSpectra requires $\mathcal{O}(\kappa_{1}\kappa_{2})$ parameters instead of LoRA’s $\mathcal{O}(Lrd)$, where $L$ is the number of layers and $r$ the rank. Across natural-language and vision benchmarks, \methodname{} matches or surpasses baseline performance while using fewer parameters than LoRA, achieving only $0.36\%$ of LoRA’s parameter count when fine-tuning LLaMA-7B on instruction-following tasks. These results show that exploiting the \textbf{architectural smoothness of transformers} through spectral analysis yields major efficiency gains in PEFT.

CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficiet Fine-Tuning
Yifei Zhang 1.2; Hao Zhu Junhao Dong", Haoran Shi?, Ziqiao Meng', Piotr Koniusz4.5.6 and Han Yu²t
Sccooo of Computer Science, Northwestern Polytechnical University; PCCDS, Nanyang Technological University; Nattional University of Singapore; 1Dtt669CSSROO; University of New South Wales; Australian National University y yifeiaccdgmail. com
Abstract
Parameter-efficient fine-tuning (PEFT) is essential for adapting large foundation models without excessive storage cost. However, current approaches such as LoRA treat each layer's adaptation independently, overlooking correlations across layers. This independence causes the number of trainable parameters to grow linearly with model depth. We provide theoretical and empirical evidence that skip con- nections in transformers create smooth gradient propagation across layers. This smoothness leads to weight adaptations that concentrate most of their energy in low-frequency spectral components, especially along the layer dimension. Em- pirical analysis confirms this effect, showing that most of adaptation energy lies in low frequencies. Building on this insight, we propose CrossSpectra, which parameterizes all attention-weight adaptations (Q, K, V) across layers as a single 3D tensor and represents them with sparse spectral coefficients (N1, 82). Using K1 non-zero coefficients within each layer's frequency space and truncating to K2 frequencies across layers, CrossSpectra requires (K))g) parameters instead LoRA's ((rr)), where the number of layers and r is the rank. Across natural language and vision benchmarks, CrossSpectra matches surpasses baseline performance while using fewer parameters than LoRA, achieving only 0.36% of LoRA's parameter count when fine-tuning LLaMA-7B instruction- following tasks. These results show that exploiting the architectural smoothness of transformers through spectral analysis yields major efficiency gains in PEFT.
Introduction
Large Language models (LLMs) have demonstrated exceptional capabilities across domains ranging from natural language processing [Liu al., 2019, He al., 2020, Radford et al., 2019, Brown et al., 2020] to computer vision [Liu et al., 2023b,a, Singh et al., 2022, Zhang et al., 2024c] to multi- modal problems [Radford al., 2021, Dong et al., 20,5a,b,c,d, Zhang al., 2025a,b)) and other domains [Fan et al., 2025, Zhang et al., 2024a, Guo et al., 2024]. However, with model parameters now reaching hundreds of billions or even trillions, adapting these models specific downstream tasks through conventional fine-tuning has become increasingly impractical. Each customized model typically requires storing as many parameters as the original model [Qiu al., 2020, Raffel et al., 2020a], leading to substantial storage and deployment challenges customization needs expand.
Parameter-efffiient fine-tuning (PEFT) methods have emerged promising solutions this chal- lenge Yang et al. [2024]. These approaches adapt pre-trained models using only a small subset of
'Equal Contribution. Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

trainable parameters, significantly reducing storage requirements while maintaining performance. Among these methods, Low-Rank Adaptation (LoRA) Hu et [2021a] and its variants [Liu et al., 2024, Meng et al., 2024, Wang et al., 2024b, Kalajdzievski, 2023, Si et al., 2024, Zhong et al., 2024, Wang al., 2024a, Ni al., 2024J) have gained widespread adoption by representing weight changes through low-rank matrices, achieving impressive results with fraction of the parameters required for full fine-tuning, Despite these advances, current PEFT methods face a fundamental limitation: they treat each layer's adaptation independently, overlooking potential structural relationships across layers. This independence assumption results in parameter counts that scale linearly with model depth, limiting efficiency gains for increasingly deeper architectures.
fundamental insight from transformer architecture analysis illuminates a path toward more efficient adaptation. Skip connections, widely used in architectures including ResNets [He et al., 2016] and Transformers [Vaswani et al., 2017], facilitate stable and smooth gradient propagation through depth. This architectural property, combined with the natural spectral bias of neural networks [Rahaman et al., 2019], suggests that weight adaptations during fine-tuning should exhibit exploitable smoothness patterns, particularly across the layer dimension. Empirical analysis confirms this intuition. Figure I demonstrates that attention weight adaptations exhibit dramatic spectral concentration across layers, with nearly 70% of adaptation energy concentrated in low-frequency components. This observation motivates a spectral approach to parameter-effie efficceent fine-tuning.
We formalize this intuition through theoretical analysis showing that attention weight adaptations in transformers naturally concentrate in low-frequency components. Our key theoretical find- ings are: (1) skip connections create Lipschitz-continuuus gradient fields across layers, ensuring similar gradients in adjacent layers: (2) these smooth gradients accumulate into smooth weight adaptations during fine-tuning; and (3) the resulting spectral structure shows strongest decay in the cross-layer dimension, enabling aggressive frequency truncation. In this paper, we propose CrossSpectra, novel PEFT method that exploits these cross-layer spectral properties. Instead of parameterizing each layer's adaptations independently, we construct a unified 3D tensor repre- sentation of all attention (Query, Key, Value) weight changes and decompose it in the frequency domain. By working directly with spectral coefficients and leveraging 3D inverse FFT for effi- cient computation, achieve parameter reductions that scale sub-lineam ainelinerrly with model dimensions. Our key contributions are:
i. We establish a theoretical framework connecting skip connection-indueee gradient smoothness to spectral properties of weight adaptations. providing a princi- pled foundation for cross- cross-layer parameter sharing in attention mechanisms.
0.35
Low Freq 69.7%
High Freq 30.3%
0.30
peggniiin of Liverrr
0.25
Freq 169. 7%, ener gy) High Freq (30.3% of energy) Energy Proportion #1 Std Dev
ii We introduce CrossSpectra, unified tensor formu- lation that represents all QKV adaptations across layers using sparse spectral decomposition with Kn coeffi- cients per layer and l2 cross cross-layer frequencies, achiey- ing (2) parameter complexity.
0.20
0.15
0.10
iii. We demonstrate that our approach achieves 275x pa- rameter reduction compared to LoRA and 5,250x com- pared to full fine-tuning, requiring only 8 KB of mem- ory versus LoRA's 2.2 MB for typical transformer configurations.
0.05
0.00
Total Energy: 100.0%
0.1
0.2
0.3
0.4
Frequency
iv. Through extensive experiments across natural language understanding, instruction tuning, and image classifi- cation tasks, we show that CrossSpectra matches or exceeds baseline performance while using fraction of the parameters.
Figure 1: Distribution of spectral en- ergy. Most of adaptation energy concen- trates in low frequencies, confirming that skip connections induce smooth, low- frequency dominated weight changes across layer.
2 Related Work
Parameter-efficient Fine-tuning (PEFT) Methods. As foundation models have grown in size and
computational requirements, efficiently adapting them has become essential. PEFT methods fall into two primary categories: non non-weight-basee [He et al., 2021, Rebuffi et al., 2017, Lester et al.,

---

2021. Gao et al., 2020, Li and Liang, 2021] and weight- based approaches [Zhu et al., 2025, Hu et al., 2021b, Zhang et al., 2025c, 2024b]. Weight-base based methods directly learn modifications pre-trained weights. LoRA [Hu et al., 2021b] represents weight changes through low-rank matrix decomposition. LoRA has gained widespread adoption due to its simplicity, effectiveness, and ability to merge adapted weights during inference to avoid latency increases. Several extensions have been proposed, including AdALoRA IZhang et al., 2023]. which adaptively allocates parameter budgets across weight matrices, and FourierFT [Gao et al., 2024] and BiLoRA [Zhu et al., 2025] learn in frequency domain. PACE [Ni et al., 2024] uses noise modulation for generalization guarantees. Despite these advances, all existing methods treat each layer's adaptation independently. failing to exploit potential structural relationships across layers. This independent treatment results in parameter counts that scale linearly with model depth, limiting efficiency gains for increasingly deeper architectures.
Skip Connections in Deep Learning. Skip connections, first popularized in ResNet [He et al., 2016] and now fundamental to modern architectures like Transformers [Vaswani et al., 2017], allow information to bypass one or more layers by adding the input of a layer block its output. Their empirical success in enabling training of very deep networks has been well-documented, but the theoretical understanding of their properties continues to evolve. A pivotal insight came from Chen et al. [2018], who established connections between networks with skip connections and ordinary differential equations (ODEs). This perspective views ResNets as discretizations of continuous dynamic systems, where each layer represents step in numerical ODE solver. The key implication is that skip connections induce smoothness in representations between consecutive layers, constraining how drastically the network can transform its inputs at each step. Further work has explored the spectral properties of networks with skip connections. Fourier-based, analyses [Rahaman et al., 2019, Tancik et al., 2020] have shown that these networks tend prioritize low-frequency components, which correspond to smoother transformations. This provides additional evidence for the smoothness-inducing nature of skip connections. Several studies have also investigated the optimization advantages provided by skip connections. The gradient stability properties of these architectures [Miyato et al., 2018] explain their trainability at extreme depths, where traditional networks struggle with vanishing or exploding gradients. However, despite these theoretical advances, the implications of skip efficient adaptation remain unexplored. Our work bridges this gap by connecting the smoothness properties of representations the structure of weight adaptations, enabling more efficient parameter sharing across layers.
perspective
cnnnectioo-nnnuced smoothness for parameter-
3 Theoretical Foundation for CrossSpectra
We develop a comprehensive theoretical framework explaining why neural network adaptations exhibit exploitable spectral structure across layers. Our analysis focuses on attention mechanisms in transformer architectures, revealing how skip connections induce smooth gradient fields that manifest as low-frequency patterns in weight updates. This spectral structure enables dramatic parameter reduction through frequency-domain methods. Our theoretical analysis yields three fundamental insights that directly inform CrossSpectra's design:
Cross-layer gradient smoothness: Skip connections create smooth gradient propagation across transformer layers, with attention weights (Query, Key, Values) exhibiting particularly strong cross-layer correlation.
ii Dimension-specife spectral decay: The spectral energy of adaptation patterns decays at different rates across dimensions; the decay is strongest along the layer axis, while spatial dimensions within each layer show slower attenuation of high-frequenge components.
ii. Joint QKV structure: All attention matrices (Q, K, V) can be efficiently represented in a unified spectral framework due to their shared architectural context.
These insights motivate our unified spectral approach, where we exploit cross-layer patterns through joint parameterization of all QKV matrices.
3.1 Gradient Structure in Transformers with Skip Connections
To understand the spectral properties of attention adaptations, we first analyze how skip connections shape gradient flow through transformer architectures.
3

---

Architecture Setup. Consider a transformer with L layers. Each layer I E fl...., L} contains multi-head attention with weight matrices: Query w. idddd, Key WiK I2dx and Value Wy Ru The attention mechanism computes:
H H
(H
Attention(HHi-) softmax
H W.
(I)
Vd
Crucially, skip connections define the layer-wise evolution of hidden representations:
Hi Hi-1 + HaaiHHHaddAAtnntoonnnHHRR
+ FFN, (H,1))
(2)
This residual structure is key to understanding gradient smoothness-in ensures that gradient informa- tion flows directly across layers while being modulated by local computations.
Cross-Layer Gradient Analysis. To quantify gradient behavior across layers, we examine the gradient structure at each layer:
Definition 3.1 (Layer-wise Gradient). For loss function L and attention matrices (WM)) M E {Q, K, V), the gradient at layer I is:
li=1 where
Oc
G
OWM
where GM E Raxd has the same shape W.M.
Skip connections promote smooth gradient propagation across layers. Our first key result formalizes this effect under the standard residual- ODE scaling:
Theorem 3.2 (Gradient Smoothness for Attention Weights). Under residual scaling Hi Hi-1
I Ri Hi Wi) and bounded Jacobians of Rí, the gradients of attention weights satisfy
CM L
G 141
G.
where Cas is a depth-independers constant.
Thus, adjacent layers exhibit gradually varying gradients, and skip connections ensure this variation decays with depth, avoiding the abrupt changes typical of purely feed-forward architectures. Intu- itively, the residual scaling Т constrains each layer's s transformation to be a small perturbation of its predecessor, gradient changes accumulate continuously along depth rather than discretely.
3.2 From Gradient Smoothness to Spectral Properties
The gradient smoothness directly translates spectral properties of weight adaptations during fine-tuning. When gradients vary smoothly across layers, the accumulated weight changes exhibit low-frequency dominance in the layer dimension.
Adaptation Accumulation Process. During fine-tuning over 7 steps with learning rate y,, attention weights accumulate updates:
T
AWM M = =-1 Σ WwyC
M (Q.
(5)
t=1
To analyze the spectral structure, we organize all weight adaptations into a 3D tensor and examine its frequency characteristics.
Theorem 3.3 (Spectral Concentration for Attention Matrices). For attention adaptations (AWM) M M where M (Q, V), define the 3D adaptation tensor WM Rxxxx with MWM AWM Let WM n22 n3) denote its 3D discrete Fourier transform, where E {O,..., 1d1--
n1 E (0....d-11 frequency index for input dimension,
1} is frequency in- dex for output dimension, and nh E (O...., L 1-1) 1} is frequency index across layers. The Fourier coefficients exhibit dimension-specife decay with frequency:
CM
W M (nz, n22, n2), na)l (n1 +1DL.M (nz + 1) (ng + 1) 33.as
(6)
where the decay rates satisfy: Ba.M 31,M, B2, for all M (Q, K,V).

---

Low Freg.
Height Dim.
Skip Connection
Smooth Gradient
Wg
W.
Layer Dim.
Width Dim.
sporse coeffiicions with truncated layer dim. froy. Pior)
Adapters
(aw
3D IFFT
The overall model: Skip connections create smooth gradient flow across transformer layers, resulting in weight adaptations that are dominated by low-frequency patterns. CrossSpectra exploits this by representing all layers' adaptations a 3D tensor and using sparse spectral coefficients via 3D inverse FFT, with truncation in the cross-layer frequency dimension.
Figure 2:
The stronger decay in the cross-layer dimension (33.M > $1.M 32.M) implies that adaptations are dominated by low-frequency patterns across layers. This is a direct consequence of the gradient smoothness induced by skip connections-smoot gradients accumulate into smooth adaptations.
CrossSpectra: From Spectral Properties to Efficient Adaptation
Building on our theoretical analysis, we now present CrossSpectra, a parameter-efficient fine- tuning method that directly exploits the cross-layer spectral structure in attention weight adaptations. As illustrated in Figure 2, skip connections induce smooth gradient propagation, which leads to low-frequency dominance in the learned adaptations.
From Theory to Design. Our theoretical findings directly inform three key design principles for CrossSpectra:
i. Unified 3D representation: Since gradients vary smoothly across layers, parameterize all QKV matrices jointly rather than treating each layer independently.
ii Fourier decomposition with selective truncation: The strongest spectral decay occurs in the cross-layer dimension (Bs > $1, 32), so we aggressively truncate high frequencies in this dimension while preserving more frequencies within layers.
iii. Implicit spectral regularization: trequency-domait parameterization naturally enforces the smoothness constraints revealed by our theoretical analysis.
These principles guide implementation choices in the following subsections.
4.1 Unified QKV Tensor Representation
Traditional parameter-efficient methods decompose each layer's adaptations independently. For example, LoRA parameterizes:
AWM A BK A e Rdrr BBM e Rrdd
(7)
This layer-wise approach cannot exploit the cross-layer patterns identified in our theoretical analysis. CrossSpectra constructs a unified 3D tensor that stacks all QKV matrices across layers:
ToKv E 3dxdx3L
(8)
where the dimensions represent d for both input and output dimensions (for simplicity), and 3L extended layer dimension containing all Q, K, V matrices across I. layers. This unified representation enables us to capture cross-layer patterns single spectral decomposition, dramatically reducing parameters compared to layer-wise methods.
5

---

4.2 Spectral Decomposition via 3D Fourier Transform
We decompose the unified QKV tensor using 3D inverse Fourier transform:
TQKV iFFT3D(C).
The coefficient tensor C E Cdxdx3L is highly sparse, with non-zero entries denoted by the index set Q2 We control this sparsity through two key parameters: K1 /52/((3)) is number of non-zero coefficients sampled within each x d layer's frequency space. #2 is number of frequencies retained in the cross-layer dimension (K2 < 3L). This parameterization yields a total parameter count of [|] = K1 f2, resulting in a sparsity ratio of [ 3L). For typical transformer configurations, achieve dramatic parameter reduction by setting [ 0.1% of the full tensor size. The key insight from our theory is that cross-layer spectral decay is strongest (53 > B1, B2). We exploit this by aggressive truncation in the layer dimension (x2 3L) while employing sparse sampling within each layer (K) E ) targeting the most important frequency components.
Complex Fourier Bases with Real- Valued Output. We use crmplex-vvluese inverse 3D Fourier trans- forms for computational efficiency. To ensure that the reconstructed tensor Tokv = iFFT3D(()) is real-valued, the sparse coefficient tensor C' must satisfy discrete Hermitian symmetry:
C u,,0au = CC] (-) mod (-v) mod d, (-w) mod (3L); V(u, v, w) E f.
(10)
Here (u, v, w) are frequency indices along the input, output, and layer dimensions, and (-) denotes complex conjugation. This condition guarantees that (PFTTD((C) yields a real-number tensor. Consequently, only one half-space of frequency coefficients needs to be parameterized independently, effectively halving the number of learnable spectral parameters and reducing storage cost.
Efficient Implementation via 3D iFFT. CrossSpectra leverages highly optimized FFT implemen- tations for computational efficiency. The core operations are:
Forward Pass (Algorithm 1): We first convert the sparse spectral coefficients to the spatial domain using 3D inverse FFT, yielding the full adaptation tensor Tokv. This operation is performed only once per forward pass. We then extract the appropriate slice for each layer and attention matrix (Q, K, V) and add it to the pre-trained weights.
ii Backward Pass (Algorithm 2): We collect all gradient updates into a single tensor, transform it into the frequency domain using 3D FFT, and keep only the gradients corresponding to the sparse indices in Ω. This effectively projects the gradient into our low- wiw-dimensional spectral subspace.
This implementation ensures that computation scales with the full tensor dimensions only for the FFT operations, while parameter storage and updates remain proportional to |S2].
Implicit Gradient Regularization. Algorithm 2 reveals that the backward pass acts as a gradient filter. When we compute Vc = FFT3D (TT3D(VVrRr)) ) and retain only к₁ sparse coefficients within each layer's frequency space and K2 frequencies across layers, we project gradients into a subspace with low-frequency basis. This filtering prevents the accumulation of rapid variations, with the strongest regularization occurring in the cross-layer dimension where truncation is the most aggressive. The optimization process is thus biased toward discovering smooth adaptation patterns that align with the natural spectral structure revealed by our theoretical analysis.
4.3
Complexity Analysis
Table 1 provides comprehensive comparison of memory requirements and computational costs.
Method
Memory
Forward Pass
Backward Pass
Full Fine-tuning
(3Ld))
(Lnd²) ((2drrd) Ond + D)
(Lndd) ((2Lnrd) Ond + +D)
LoRA (rank r)
((Lrrd)
CrossSpectra
(K1K2)
Table 1: Complexity comparison. We denote as sequence length, T. is number of layers, and
D = 3l. (2 log + log(3I.)) is the 3D FFT cost.

---

Algorithm CrossSpectra Forward Pass
Algorithm 2 CrossSpectra Backward Pass
1: Input: Sparse coefficients C with
1: Input: Gradients {V W.r C}.
2: K1 non-zeros per layer, K2 layers.
2: Initialize VTgKY E RudxdxLL
3: Crult <
3: for I = to do
4: TQKV < iFFT3D(C fult)
4: for M E {Q, K, K. do
5: for = 1 to L do
5: Place V Ww L into
6: for M [Q, K, V} do
7: AWA M + Extract(Tovv. M.
6:
VTQRV at (M,1)
8:
WM < w + AWA
7: end for
8: end for
9: end for
Veran FFT3D(V TOKY)
10:
end for
11:
Return: (WM))
10: Vc f- SparseSample V Crau K1, K2)
11: Return: V
Table 2: Performance comparison of LLaMA2 7B with different methods on eight commonsense reasoning datasets. The symbol f indicates that the results are taken from [Wang et al., 2024a, Zhong et al., 2024, Si et al., 2024].
Method
Paramss %)
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-
ARC-c
OBQA
Average
ChatGPT
73.10
85.40
68.50
78.50
66.10
89.80
79.90
74,80
77.01
LoRÁ
0.84
69.80
79.90
79.50
83.60
82.60
79.80
64.70
81.00
77.61
DoRAi
0.84
71.80
83.10
79.90
89.10
83.00
84.50
71.00
81.20
80.45
PISSA
0.84
67.60
78.10
78.40
76.60
78.00
75.80
60.20
75.60
73.78
MiLoRAt
0.84
67.60
83.80
80.10
88. $8.20
82.00
82.80
68.80
80.60
79.24
LoRA-Dashf
71.00
79.30
NEAT!
0.84
71.70
83,90
80,20
88. 90
84,30
86.30
71.40
83.00
81.21
KaSA
73.60
84.40
80.20
91.50
84,50
84.70
72.10
81.20
81.53
MoLoRA
0.96
73:15
83.68
80.09
74.57
85.95
87,33
72.53
86,20
80.43
HydraLoRA
79.68
86.66
CroseSpectra
0.02
73.69
83,95
80.34
88.42
87.24
87.75
76.88
87.21
82.73
The memory efficiency of CrossSpectra stems from sparse spectral parameterization. For typical transformer configurations with L 24 layers and d 768, full fine-tuning requires 42 MB, while LoRA with rank 8 needs 2.2 MB. CrossSpectra with #ı = 1024 (sparse within layers) and R2 = 8 (cross-layer truncation) requires only 8 KB achieving 275x reduction compared LoRA and 5,250x reduction compared to full fine-tuning.
Computationally, Cross$peccra's forward pass consists of single 3D iFFT operation with com- plexity (2 log d + log(3)))), followed by standard matrix multiplications. For sequences of length n, the dominant cost remains the attention mechanism's Onn2d) complexity, making CrossSpectra's 's FFT overhead negligible in practice. The backward pass similarly requires a 3D FFT for gradient computation with the same complexity. Modern FFT implementations on GPU further reduce this overhead through optimized memory access patterns and parallelization.
(2
Experiments
Baselines. To validate the effectiveness of CrossSpectra, we compare against three categories of baselines: 1) Full FT: Full fine-tuning of all parameters; 2) Single-LoRA variants: LoRA [Hu et al., 2021c], DoRA [L_u al., 2024], PiSSA [Meng al., 2024], MiLoRA [Wang et al., 2024b], rsLoRA (Kalajdzievski, 2023), LoRA-Dash [Si et al., 2024], NEAT [Zhong et al., 2024], and KaSA [Wang al., 2024a]: ; LoRA MoE methods: MoLoRA [Zadouri et al., 2024], AdaMoLE [Liu and Luo, 2024], and HydraLoRA [Tian et al., 2024]. These baselines represent the current state of the art in Darameter-efficient fine-tuning, both for standard and mixture-of-experts variants.
Benchmarks. demonstrate the cross-modal effectiveness our spectral approach, we evaluate CrossSpectra on four diverse tasks. Image Classification (IC): We fine-tune CLIP ViT-B/32 [Radford et al., 2021] on standard image datasets including StanfordCars, DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN [Ilharco et al., 2023]. This evaluates CrossSpectra's ability
7

---

Table 3: Performance comparison of RoBERTa-large with different methods 7 GLUE tasks. Total rank is set to 32.
Method
A Params (%)
CoL.A
SST-2
MRPC
QQP
MNLI
QNLI
RTE
Average
Full
100
95.98
85.29
94.49
LoRA
4.00
83.41
95.64
81.33
90.06
89:00
93.28
84.47
88.46
DoRA
95.99
91.24
PISSA
4,00
69.12
95.98
82.84
91.24
88.94
93.59
73.29
85,00
91.33
89.51
94.12
LSLoRA
4.00
83.51
95.98
86,02
90.75
88.97
93.84
84,12
89.03
MoLoRA
4.50
96.10
87.75
93.90
AdaMoLEE
4.56
83.99
95.76
86.03
91.48
89.21
93.64
83.75
89.12
HydraT.oRA
2.75
95.52
93.87
88.56
CroasSpectza
0.01
86.86
96.21
84.55
91:40
89.55
94.19
85.56
89.76
to capture visual adaptation patterns. Natural Language Understanding (NLU); We fine-tune RoBERTa-large [Liu, 2019] on the GLUE benchmark [Raffel et al., 2020b], which comprises diverse language tasks including grammatical acceptance (CoLA), sentiment analysis (SST-2), paraphrase detection (MRPC, QQP), and natural language inference (MNLI, QNLI, RTE). Commonsense Reasoning (CR): We use LLaMA2-7B [Touvron al., 2023] 8 reasoning benchmarks: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA. Following Hu et al. [2023], combine training datasets from all tasks and evaluate each test set separately. Arithmetic Reasoning (AR): Using LLaMA2-7B, we evaluate mathematical reasoning capabilities on GSM8K [Cobbe al., 2021], MAWPS (Koncel-Kedziorstt AQuA [Ling et al., 2017] benchmarks. The training data combines these sources with step-by-step rationales. These diverse tasks let verify that the cross-layer spectral structure exploit exists across various model architectures and domains.
al., 2016], SVAMP [Patel al., 2021], and
Implementation Details. For CrossSpectra, adapt all query, key, and value projection matrices in transformer attention blocks, except in image classification tasks where we only adapt query and key matrices following standard practice. For frequency sparsity,
set the number of non-zero coefficients [02] = 3000 (corresponding to approximately b = 1000 samples per layer slice and l = frequencies in the layer dimension). This represents just 0. 0.1-0.5% of the full parameter space depending on model size. For baseline comparisons, we use LoRA with rank r = 16 and = 32. All models are trained using Adam optimizer [Kingma and Ba, 2014] with batch size 64 and cosine learning rate scheduling. For image classification, we use separate learning rates: 1e-3 for the classification layer and le for adaptation parameters.
5.1 Main Results
Table 4: We evaluate CLIP ViT-B/32 with full fine-tuning and LoRA variants with total rank 8 across StanfordCars, DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN datasets. Bold indicates the highest results.
Method
# Params (%)
Cars
DTD
EarSSAT
GTSRB
RESISC45
SUN397
SVHN
Average
Full FT
100
60.33
73.88
98.96
98.30
93.65
53.84
96.78
82.25
1.49
LoRA (rank16)
2,99
46.51
72.07
98.74
98.04
92.08
51.63
96.00
79.30
LoRA (rank32)
5.98
50.13
72.87
98.85
98.13
92.87
53.65
96.55
80.44
DoRA
1.49
40.75
71.91
98.89
97.71
90.19
47.54
95.46
77.49
PISSA
1.49
40.41
69.62
98. 48
95.84
90.58
47.21
95.84
76.85
MiLoRA
1.49
70:48
97.52
45.38
76.68
MoLoRA
2.24
50.83
73.51
98.63
97.72
92.58
$2.55
96.00
80,26
AdaMoL.E
2.33
49.47
71.65
98.52
97.73
91.95
52.29
95.82
79.63
HydraLoRA
1.58
48.42
72.18
98.40
97.28
92.93
51.80
96.06
79.58
CroesSpectre
0.03
53.50
75.32
98.82
98.17
93.46
54.53
96.62
81.49
The experimental results across multiple modalities in Tables 2,3,4 and 6 demonstrate that CrossSpectra's superior effectiveness parameter-efficient fine-tuning. On commonsense reason- ing tasks with LLaMA2-7B, CrossSpectra achieves 82.73% average accuracy, outperforming methods

---

Table Performance comparison under different frequency sparsity levels. CrossSpectra achieves optimal performance with only 3.2%o of the frequency space.
CSR Task
IC Task
Freq. [[]]
10000
30000
60000
120000
1000
3000
6000
12000
Sparsity (21/(d²)
0.9%.
1.8%
3.6%
7.2%
2.5%
5.0%
10.1%
20.2%
CrossSpectra
82.15
82.73
82.33
82.65
79.18
80.34
81.49
81.32
Table Accuracy comparison of various LLMs using PEFT methods on arithmetic reasoning tasks. Results marked with an asterisk (*) are sourced from Hu et al. [Hu et al., 2023]. (f)
90.0
87.5
Freq.
High
denotes our reproduced results on LoRA.
62.5
Model
Methods
MAWPS
SVAMP
GSMBK
AQuA
Avg.
80.0
Base
51.7
32.4
15.7
16.9
24.8
77.5
LeRA*
52.1
DoRA
79.2
52.0
37.3
17.4
43.1
75.0
LLaMA2-7B
-4- CR Task
PISSA
78.6
52.5
17.7
44.7
72.5
NUU Task
MiLoRA
79.0
52.1
37.5
18.9
44.6
CrossSpectra
81.4
51.8
38.2
18.1
43.7
70.0
Parcontage Specturm (%6)
Figure 3: Left: Performance vs. spectrum coverage across modalities. Task performance rapidly increases with low frequencies (0-20%) and shows diminishing returns beyond 60%, validating our theoretical prediction of low-frequency dominance.
such as KaSA (81.53%) and NEAT (81.21%). For NLU tasks using RoBERTa-large, scores 89.76% across GLUE benchmarks, excelling particularly complex tasks such CoLA (86.86%) The cross-modality effectiveness is further validated in vision tasks with CLIP ViT-B/32, CrossSpectra reaches 81.49% average accuracy, surpassing even high-rank LoRA variants (80.44%). For arithmetic reasoming, it outperforms standard approaches on benchmarks such as GSM8K and MAWPS. These consistent results across diverse tasks validate theoretical framework that exploiting cross-layer smoothness via tensor-based Fourier parameterization enables substantial parameter reduction while maintaining improving performance, confirming that weight adaptations exhibit the spectral bias predicted by our gradient analysis.
5.2 Analysis of Spectral Properties in Weight Adaptations.
The experimental analysis of Figure and Figure provide compelling evidence for our theoretical claims about the spectral properties of weight adaptations neural networks with skip connections.
Low-Frequency Dominance in Adaptation Energy. Figure I illustrates the distribution of energy across different frequency components in weight of LLaMA2-7B (we use value projection in the attention layer for illustration). The analysis reveals that significant portion of the total adaptation energy (69.7%) is concentrated the low-frequency region of the spectrum, while only 30.3% is distributed across high-frequency components. This striking imbalance confirms weight adaptations should inherit the spectral properties with energy concentrated in low-frequency components.
Performance Correlation with Spectral Components. Figure 3 demonstrates how model perfor- mance across different tasks (Commonsense Reasoning, Natural Language Understanding, and Image Classification) varies as we progressively increase the sample portions of the frequency spectrum. (i.e., the x-axis indicate limit the selected frequency basis reside the first % part spectrum). It demonstrates that across all three modalities (Commonsense Reasoning, Natural Language Under- standing, and Image Classification), performance increases rapidly with just the initial low-frequency components (0 20% of the spectrum), stabilizes in the middle range (20 60%), and shows diminishing returns in the high-frequency range (beyond 60%). The consistent pattern across diverse modalities confirms that the smoothness induced by skip connections creates adaptation patterns that are efficiently representable in the low frequency domain along the layer dimension, allowing our approach to achieve strong performance while using dramatically fewer parameters.

---

Table 7: Spectral energy model scale. Low-
Table 8: LLaMA2-13B commonsense results. CrossSpectra surpasses LoRA with far fewer parameters.
frequency dominance persists across sizes.
Model
Layers
Params
Low Freq (%)
High Freg (8)
LLLMM2-13B
Mothod
BoolQ
PIQA
SIQA
HellaSwag
Avg.
LLaMA2-7B
32
7B
69.7
30.3
LoRA
75.2
83.4
82.1
89.8
82.62
ROBERTE-L
24
355M
68.4
31.6
CroseSpectre
76.1
83.2
82.7
92.1
83,52
V/TB/32
86M
70.9
29.1
Table Computational efficiency. Sub-linear time scaling with model size.
Table 10: FFT overhead. 3D iFFT
cost is negligible vs. attention.
Model
Layers Time/Eppch (8) Params( Avg.
Tenser Size
FFT Time
LLaMA2-13B (CroesSpoctra)
40
1.82
0,02
83.52
(4096, 4096, 3x10)
0.05
1.MMA27BB (CrosaSpactra)
32
0.79
0.02
K2.73
(4096, 4096, 2x)
0.12
LLaMA2-7B (LoRA)
32
0.61
0.84
77.61
(4096, 4096, 3x30)
0.28
(1096, 1096, 3x
0.41
Sparsity along the Weight Dimension. Table 6 presents a comprehensive comparison of various PEFT methods' performance specifically on arithmetic reasoning tasks across different LLM archi- tectures LLaMA2-7B). The evaluation spans four key mathematical benchmarks: MAWPS (math word problems), SVAMP (simple variations on arithmetic problems), GSM8K (grade school math), and AQuA (arithmetic questions and answers). The results demonstrate that CrossSpectra achieves comparable or superior performance to standard LoRA and DoRA methods in solving these complex mathematical problems.
Impact of Frequency Sparsity, Table 5 demonstrates how CrossSpectra's performance varies with different levels of frequency sparsity across modalities. For commonsense reasoning (CSR) tasks, we tested sparsity levels from 0.9%o to 7.2%o of the full frequency space (10,000 to 120,000 non-zero frequencies). Similarly, for image classification (IC) tasks, explored sparsity levels from 2.5% to 20.2% (1,000 to 12,000 frequencies). Remarkably, CrossSpectra achieves optimal performance with extremely sparse parameterization-ooni and 5.0% (3,000 frequencies) for IC tasks. This confirms our theoretical prediction that weight adap- tations naturally concentrate small subset frequency components. The consistent performance across wide sparsity ranges further validates that our approach captures the essential adaptation information while eliminating redundancy. Even at the sparsest setting (0.9%o., Cr CroseSpect tra outperforms traditional methods that use orders of magnitude more parameters, demonstrating the practical impact of our theoretical insights about spectral concentration in transformer adaptations.
1.8%c (30,000 frequencies) for CSR tasks
Scaling to Larger Models. We examine whether the low-frequency concentration holds as model size increases. Table 7 shows that the low-frequency share of adaptation energy remains high (61- 71%) from ViT-B/32 (86M) up to LLaMA2-13B (13B), supporting our scale-agnostic hypothesis. We further fine-tune LLaMA2- 13B on commonsense reasoning. CrossSpectra surpasses LoRA with far fewer parameters (Table 8), indicating that our cross-layer spectral parameterization scales favorably.
Computational Efficiency. Training time scales sub-linearly with model size due to sparse spectral coefficients (Table 9). LoRA is faster per epoch but yields lower accuracy, illustrating effi- accuraac trade-off. Using torch. fft. ifftn, 3D FFT runtime grows as O(N log N) and remains negligible relative attention (Table 10).
6 Conclusions
This work connects neural network theory with efficient adaptation methods, revealing how architec- tural properties determine optimal weight adaptation parameterization. CrossSpectra demonstrates that skip connections create structured adaptation patterns across layers an insight previously overlooked. Future research could explore dynamic spectral bases, adaptation for emerging ar- chitectures, or iiormatiion-thooree compression limits. Our findings suggest the most efficient neural network parameterizations align with their intrinsic dynamics rather than treating parameters independently. As foundation models grow, leveraging these inherent structures will be essential for making specialized adaptation accessible.
10

---

Acknowledgments
The research supported, in part, by the Ministry of Education, Singapore, under its Academic Research Fund Tier 1 (RG101/24); the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award 12301E0026), administered by A*STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL). Piotr Koniusz and Hao Zhu are supported by CSIRO's Science Digital.
References
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners, Advances in neural information processing systems, 33: 1877-190,, 2020.
R. T. Chen, Y. Rubamova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations.
Advances in neural information processing systems, 31, 2018. 3
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021.
J. Dong, P. Koniusz, L. Feng. Y. Zhang, H. Zhu, W. Liu, X. Qu, and Y.-S. Ong. Robustifying zero-shot vision language models by subspaces alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2 21037-21047, 2025a.
J. Dong, P. Koniusz, X. Qu, and Y.-S. Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I, KDD '25, page 236-247, New York, NY. USA, 2025b. Association for Computing Machinery. ISBN 9798400712444 12456.. doi: 10.1145/ 3690624.3702296..
J. Dong, P. Koniusz, Y. Zhang. H. Zhu, W. Liu, X. Qu, and Y-S. Ong. Improving zero-shot adversarial robustness in vision- languge models by closed-form alignment of adversarial path simplices. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 14061-14078. PMLR, 13-19 Jul 2025c. 1
J. Dong. H. Zhu, Y. Zhang, X. Qu, Y.-. Ong, and P. Koniusz. Machine unlearning via task simplex arithmetic. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025d.
T. Fan, H. Gu, X. Cao, C. S. Chan, Q. Chen, Y. Chen, Y. Feng, Y. Gu, J. Geng, B. Luo, et al. Ten challenging problems in federated foundation models. IEEE Transactions on Knowledge and Data Engineering, 2025.
T. Gao, A. Fisch, and D. Chen. Making pre-traing pre-trained language models better few-shot learners. arXiv preprint ar 1r2V 2020. 3
Z. Gao, Q. Wang, A. Chen, Z. Liu, B. Wu, L. Chen, and J. Li. Parameter-efficient fine-tuning with discrete fourier transform. In Proceedings of the 41st International Conference Machine Learning, pages 14884-1495 14901, 2024. 3
Z. Guo, Y. Zhang, Z. Zhang. Z. Xu, and I. King. Fedlfe: Towards efficient federated multilingual mod- eling with lora-based language family clustering. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1519-1528, 2024. 1
J. He. C. Zhou, X. Ma, T. Kink-KKikkatrckk and G. Neubig. Towards a unified view of parameter- efficient transfer learning. arXiv preprint 1rXiv:2110.04336 2021. 2
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 2,
He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding- eeaedncge bert with disentangled attention.
arXiv preprint arXiv:2006.03653 2020. 1
E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021a. 2
F. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L.. Wang, and W. Chen, Lora: Low-rank 2021b. 3
adaptation of large language models. arXiv preprint
11

---

E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021c. 7
Z. Hu, L.. Wang. Y. Lan, W. Xu, E.-P. Lim, L.. Bing, X. Xu, S. Poria, and R. K.-W. Lee. Llm-adapters: An adapter family for parameter-effiiien fine-tuning of large language models. arXiv preprint arXiv:2304.00 01933, 2023. 8, 9
G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, I.. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic,, 2023, URL https / //trxaiivv///////////a///aaaaala org/ abs 22122 04089. 7
D. Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023. URL https: //arxiv. , org/ 2212. 03732. 2, 7
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arriiv:142..600 2014. 8
R. Kedziorkk,,i,,,ki, S. Roy, A. Amini, N. Kushman, and H. Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguisties: human language technologies, pages 1152-1157, 2016.
8
B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0888 2021. 2
X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021. 3
W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: 04000004446,,... 2017.
H. Liu, C. Li. Y. Li, and Y. J. Lee. Improved basclines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. arXiv preprint 33ii::3304004885. 2023b.
L. Liu, X. Liu, Gao, W. Chen, and J. Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747-5763, 2020. 15
S.yy Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang. K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference Machine Learning, 2024. 2. 7
Y. Liu, Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. 2019. 8
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoy- anov. Roberta: A robustly optimized hert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019. 1
Z. Liu and J. Luo. AdamoLE: Fine-tuning large language models with adaptive mixture of low- rank adaptation experts. In First Conference on Language Modeling, 2024. URL https:// openreview wwt. dddmmmddd--dd
Meng, Z. Wang, and M. Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:://// http::///ppeeeviiieeeeee
2, 7
T. Miyato, Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. 3
Y. Ni, S. Zhang, and P. Koniusz. PACE:
marrying the generalization of MArameter-efficien fine-tuning with consistency regularization. In The Thirty-eighth Annual Conference on Neural Information
Processing Systems, 2024. 2,3
A. Patel, S. Bhattamishra, and N. Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint 1 2021. 8
X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang. Pre-trained models for natural language
processing: A survey, Science China Technological Sciences, 63(10))1872-1897 2020.
12

---

A. Radford, J. Wu. R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and L. Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https: //tp//aavvvvvvv.////// org/ httpaabs/21...0000
1,7
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 55555, 2020a. 1
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-55 1,,,,, 2020b. 8
N. Rahaman, A. Baratin, D. Arpit, Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville. On the spectral bias of neural networks. In International conference on machine learning, pages 5301-5310. PMLR, 2019. 3, 15, 16
A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017. 2
C. Si, Z. Shi, S. Zhang, X. Yang, H. Pfister, and W. Shen. Unleashing the power of task-specific directions in parameter efficient fine-tuning, 2024. URL htttt://taaaaatt www..ogg////a.o.eooooo..ooooee/a abb9 01035. 7.7
A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages (6638-16550, 2022. 1
M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoor- thi, J. Barron, and R. Ng. Fourier features let networks leam high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020. 3
C. Tian, Z. Shi, Z. Guo, L. Li, and C. Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024. URL https: : / // //taxiivvv.//////////e///aaaaa.a org/ absa 2204/2444.. 19245. 7
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint
arXiv:2302. 139711 2023. 8
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017. 2, 3
F. Wang, J. Jiang, C. Park, S. Kim, and J. Tang. Kasa: Knowledge-aware singular-value adaptation of large language models, 2024a. URL h /arxivixvvvv//////////////aai/// org/ abs haabb///42..... 06071. 7,7
H. Wang, Y. Li, S. Wang, G. Chen, and Y. Chen. Milora: Harnessing minor singular components for parameter-efficient IIm finetuning, 2024b. AtLhttps:: /wwxxv..ivvvv org/ abs/ /406 09044. 2, 7
R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. In International conference on machine learning, pages 10524-10533. PMLR, 2020. 15
M. Yang, J. Chen, Y. Zhang, J. Liu, Zhang, Q. Ma, H. Verma, Q. Zhang, M. Zhou, I. King, et al. Low-rank adaptation for foundation models: A comprehensive review. arXiv preprint arXiv:2501.0000 00365, 2024. 1
T. Zadouri, A. Üstün, A. Ahmadian, B. Ermis, A. Locatelli, and S. Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, 2024, URL httpp:://pperreeee
net/
7
Q Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303. 105/2, 2023. 3
S. Zhang, A. Chen, Y. Sun, J. Gu, Y.-Y. Zheng, P. Koniusz, K. Zou, A. van den Hengel, and Y. Xue. Primitive vision: Improving diagram understanding in MI.LMs. In Forty-second International Conference Machine Learning, 2025a. 1
13

---

S. Zhang, J. Du, Y. Xue, P. Torr, P. Koniusz, and A. van den Hengel. Open-world objectness modeling unifies novel object detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 30332-30342, June 2025b.
Y. Zhang, D. Zeng, J. Luo, X. Fu, G. Chen, Z. Xu, and I. King. A survey of trustworthy feder- ated learning: Issues, solutions, and challenges. ACM Transactions Intelligent Systems and Technology, 15(6):1-47, 2024a.
Y. Zhang, H. Zhu, A. Liu, H. Yu, Koniusz, and I. King. Less is more: Extreme gradient boost rank-1 adaption for efficient finetuning of Ilms. arXiv preprint WVXiv:2410.19600 2024b.
Y. Zhang, H. Zhu, Z. Song, Y. Chen, X. Fu, Z. Meng. P. Koniusz, and I. King. Geometric view of soft decorrelation in self-supervised learning. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4338 4349, 2024c.
Y. Zhang. H. Zhu, A. Z. Tan, D. Yu, L. Huang, and H. Yu. pfedmxf: Personalized federated class- incremental learning with mixture of frequency aggregation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 30640-30650, 2025c. 3
Y. Zhong, H. Jiang, L. Li, R. Nakada, T. Liu, L.. Zhang, H. Yao, and H. Wang. Neat: Nonlinear parameter-efficient adaptation of pre-trained models, 2024. URL https://arrs: www.vvorg/aaao.. abs// 01870. 7
H. Zhu, Zhang, J. Dong, and P. Koniusz. BiLoRA: almost-orthogonal parameter spaces for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 25613-25622, June 2025. 3
14

---

A Proofs
Our analysis builds on standard stability and smoothness properties of residual architectures.
Assumption A.1 (Residual stability and ODE scaling). Each residual block 1 Wi) is Lipschitz continuous with ||aR// 11 S си and |@RR/äw/ |2 S CM, where CH: CM are depth-inden ieppe-inddeendeett constants. We assume residual-ODI scaling Hi = H R- W1), and bounded loss gradients |aC/0Hr.||= G (Xiong al., 2020, Liu et al., 2020). Layer normalization and residual scaling guarantee such bounded Jacobians.
Assumption A.2 (Spatial regularity and weak separability). Within-layer adaptation maps possess bounded Sobolev norms. implying decay of frequenyy spectral components in each spatial dimension. Such spectral bias is well documented in neural networks (Rahaman et al., 2019). Moreover, the 3-D adaptation tensor is approximately separable across spatial and layer dimensions, as suggested by empirical analyses of attention weight spectra.
These assumptions are mild and supported by empirical evidence in prior work. They ensure that the subsequent proof' yield depth-normalized gradient smoothness and power-law spectral decay.
A.1 Proof of Theorem 3.2
Proof. Step 1: Residual formulation. We express each transformer block in residual- form H; H + 1 (H Wi), where RS denotes the combined attention and feed-forward submodules. Let J, = 0R,/0H,-1 and K2 = 0 with ||,ر l2 S си and | S cM-
Step 2: Gradient decomposition. By the chain rule,
ac OHL WWM HHg OW,.
HH, 1 aW. Σ P
where the propagation operator P IT 11= (T JJ) satisfies || con(t-t)/L
Step 3: Bounding cross-layer differences. Successive gradients differ by
OH, OH, 1
Σ
( @1+10441 1+t1+)).
OW
OW
L
1=++1
Using the series expansion P141+141 1 O and | I ((((t=))/ gives
0H,
0H,
CE 1)eM
aWM 1+1
OW. 2
Multiplying by ||8//HH, < G yields
CM
IG GM || <
Cm = GGM((En-
T
A.2 Proof of Theorem 3.3
Proof. Step Constructing the adaptation tensor: Let A AWA denote the accumulated weight = =AWA Define its 3D discrete Fourier transform:
updates, and form AM E IRdxdxL with A
A M (nj na) = Σ
AMM / i
Step 2: Smoothness implies spectral decay. From Theorem 3.2,
C
| 4
AW. ||r
S
15

---

A bounded first finite difference yields Fourier magnitude
C
A Ming
||
S
+ n33
If a stronger second-difference bound | AWA. M
22W AW 1-1 PS S C2/L2 holds, then
C2
M Siing
(1 + nz)
This confirms power-law decay in the cross-layer dimension. Step 3:
Combined spatial-layer spectral structure. Empirically, neural networks exhibit spectral bias toward low within-layer frequencies [Rahaman et al., 2019], so that
Cas
A M (n1, R2, ns)l Σ
(1 + n1) (1 + n (1+ [1+ng)) n2
where 83 31, B2 due to stronger smoothness along depth. This establishes the claimed dimension- specific spectral decay.
B Limitations and Future Work
While CrossSpectra demonstrates significant parameter efficiency across diverse tasks, there several limitations worth noting. First, current approach focuses exclusively adapting attention weights (Q, K, V), potentially missing optimization opportunities in other components like feed-forward networks. This design choice has the benefit of reducing overfitting risk by targeting the most information-dense parameters identified by our gradient analysis, but extending the spectral approach other transformer components in a principled way could yield further improvements. Second, the computational overhead of 3D FFT operations. while well-optimized on modern hardware, might become a bottleneck for extremely large models (trillions of parameters) or resource- constriinnd deployment environments. Third, our method requires careful selection of frequency sparsity patterns currently uniform sampling within layers-which -which may not be optimal for all model architectures tasks. A promising direction for future work is to develop adaptive frequency sampling strategies that automatically identify the most important spectral components for specific task. Additionally, exploring theoretical connections between CrossSpectra and other parameter-efficient methods like prompt tuning could lead to hybrid approaches that combine their complementary strengths. Finally, while we observed consistent cross-layer spectral patterns across model sizes up to 7B parameters, verifying that these properties scale efficiently to models with hundreds of billions of parameters remains an important direction for future research.
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes],
Justification:
We introduce contributions and scope on introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
ii. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work on the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
ii. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide the full set of assumptions and a roof
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
iv. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We write all the main experimental results
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification:
We promise code will be public available when paper get accepted
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
Soalli/gguide/CodeeeSiie son.eeeooso for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Co/eeeodiissseeeeeeee
/nips. wwtppcccpp/caceeccccccccccccccae Sttaaiigdddee/eeeSumiiiieee wwwsssooliigooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
vi. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The results and experiment are displayed in experiment part
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
vii. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We define the standard deviation over five times training, thereby conveying statistical significance and variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
viii. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of this experiment are displayed in main paper
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
ix. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //tne///nnuuppeeeeee httpp/pllcc/thiccaiiiieeeeeceeee
Answer: [Yes]
Justification: This paper conducted in the paper conform with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
X. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: paper discusses both potential positive societal impacts and negative societal impacts of the work performed
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
xi. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA],
Justification:
This work does not have any danger.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
xii. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use all public datasets without private data.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
Xiii. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
We use all public datasets without private data.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/cood/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
xiv.
Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
We do not have crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
xv. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
We do not have crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
xvi.
Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA].
Justification:
We use LLM for grammar correction.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttpas/:///eu.../eeeee //tps/////eeeeece///22////////// for what should or should not be described.
23


Reviews:
[{"review_id": "aPyZ1p1kT7", "reviewer": "Reviewer_puJh", "summary": "This paper introduces CrossSpectra, a parameter–efficient fine-tuning (PEFT) method that exploits cross-layer spectral smoothness in transformer attention weights. The authors show that skip connections make gradients smooth across layers, implying that weight updates mainly occur in low frequencies. Then they stack all Q,K,V adaptations into a single $d{}d{}3L$ tensor and learn only $k_1$ sparse spatial frequencies and $k_2$ low cross-layer frequencies. By a 3-D iFFT,  this yields $O(k_1k_2)$ trainable parameters, instead of traditional $O(Lrd)$. On LLaMA-7B instruction tuning, CrossSpectra matches or exceeds LoRA while using only 0.36% of its parameters. It also matches state-of-the-art PEFT baselines with very small expense. Overall, CrossSpectra shows that leveraging transformer spectral smoothness can reduce storage without sacrificing too much of the accuracy.", "questions": "-  The $(k_1,k_2)$ frequency budgets are tested on LLaMA-7B. How should people choose these values for deeper models with more parameters? I suggest that related experiments should be taken.\n  -  What is the motivation of this paper? It seems that the story line in not clear, and I would like to know more about how this method was developed by inspirations.\n  -  Have the authors tested CrossSpectra on models with more layers?  Does the low-frequency assumption still hold at that scale? I concern that although this holds for the case in the paper, it may have its flaw in some domains\n  -  Does updates to low cross-layer frequencies reduce adaptation on generation-heavy tasks where later layers may have larger shifts instead of smooth change? I think this might be the case, so this impact should also be evaluated", "limitations": "Apart from the limitations that have been mentioned in the appendix, CrossSpectra still have several potential limitations to improve. First, all results come from single experiments with no analysis in variance across several experiments, so the reported effectiveness might at a risk of statistical noise. Also, it seems that memory usage during training is not recorded, and since the choose of hyperparameter is not deterministic, it would be better to keep track of the memory. Finally, its effectiveness on more domains and architecture with more layers are not yet checked, so it would be better if more experiments relating to different models are tested.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n  -  *Conceptual novelty.*  This paper utilizes cross-layer spectral smoothness to collapse all Q/K/V parameters into a single 3-D Fourier space, which slash PEFT parameters.\n  -  *Theory  and evidence.*  It provides a gradient-smoothness bound and empirical spectral-energy plots that explain why low cross-layer frequencies matter most.\n  -  *Logical clear proof.*  The proof of the theorem seems to be clear, with reasonable conditions and logical proof.\n\n**Weaknesses**\n\n  -  *Limited scale.*  The result has only been operated on a 7 B-parameter LLaMA; no evidence shows that the approach holds for the model with more parameters.\n  -  *Lack preliminary.*  The motivation has not been clearly stated, for example, why this algorithm is applied, and the related preliminaries, such as fast Fourier transformation, are not explicitly introduced.\n  -  *Single report.*  All scores come from one run, and variance between different runs are not recorded or analyzed.\n  -  *Memory not measured.*  Reconstructing full $dd$ updates via iFFT each step could may potentially use memory,  but experiments relating to memory usage is not measured.\n  -  *Task range limited.*  There is no evaluation on generation-heavy tasks; robustness across different domains (setups) is not tested, which has been mentioned in the limitation.\n  -  *Hyperparameter sensitivity.*  The choice of hyperparameters to train is stochastic and the training process is sensitive to the selection, which has been mentioned in the limitation.", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: Show evidence that the approach holds for the model with more parameters. Does the low-frequency assumption still hold at that scale?**\n\n\nThank you for this important question regarding scalability.\n\n**1.1 Scaling Analysis**\nOur theoretical framework is scale-agnostic, as the spectral properties we exploit (cross-layer gradient smoothness from skip connections) are fundamental to transformer architectures regardless of size. We have validated this across diverse model scales and modalities:\n- **Vision**: ViT-B/32 (86M parameters) [included in paper]\n- **Language**: RoBERTa-large (355M), LLaMA2-7B (7B), [included in paper], LLaMA-13B (13B) [new]\n\n**1.2 Experimental Validation**\nTo directly address scalability concerns, we conducted additional experiments on LLaMA2-13B. Due to resource constraints, we were unable to test larger models, but the results strongly support our approach's scalability.\n\n**Spectral Energy Distribution Across Model Sizes:**\n| Model      | Layers | Parameters | Low-Freq Energy (%) | High-Freq Energy (%) | Assumption Holds |\n|------------|--------|------------|-------------------|---------------------|------------------|\n| LLaMA2-13B | 40     | 13B        | 61.2%             | 38.8%               | ✓                |\n| LLaMA2-7B  | 32     | 7B         | 69.7%             | 30.3%               | ✓                |\n| RoBERTa-L  | 24     | 355M       | 68.4%             | 31.6%               | ✓                |\n| ViT-B/32   | 12     | 86M        | 70.9%             | 29.1%               | ✓                |\n\n**Performance on LLaMA2-13B**\n| Method       | BoolQ | PIQA | SIQA | HellaSwag | Average |\n|--------------|----------------|-------|------|------|-----------|\n| LoRA         | 75.2  | 83.4 | 82.1 | 89.8      | 82.62    |\n| CrossSpectra | 76.1  | 83.2 | 82.7 | 92.1      | 83.52    |\n\n\n**Key Findings:**\n- The low-frequency spectral bias consistently holds across all model sizes, with 60-70% of adaptation energy concentrated in low frequencies.\n- The slight reduction in spectral bias for larger models (61.2% vs. 69.7%) is expected due to vanishing gradients in deeper networks, but remains sufficient for effective parameter reduction.\n\nThese results demonstrate that our approach scales effectively to larger models while maintaining the fundamental spectral properties that enable dramatic parameter efficiency.\n\n\n\n## **Q2： The motivation has not been clearly stated, for example, why this algorithm is applied, and the related preliminaries (e.g. FFT)**.\n\n**Motivation.** Existing PEFT methods such as LoRA treat each layer independently, ignoring the architectural characteristics of Transformers and resulting in parameter counts that scale linearly with model depth. This layer-independent assumption fails to exploit the architectural properties that should be preserved during fine-tuning, resulting in parameter counts that scale linearly with model depth and limiting efficiency gains for increasingly deeper architectures.\n\n**Key Insight.** Skip connections in Transformers ($H_l = H_{l-1} + f(H_{l-1})$) create smooth gradient fields across layers, causing weight adaptations to concentrate ~70% of energy in low-frequency spectral components (Figure 1).\n\n**Solution.**: Since smooth functions have sparse frequency representations, we exploit this cross-layer spectral structure using 3D inverse FFT to:\n\nReconstruct full adaptation tensors from sparse low-frequency coefficients\nEnable joint parameterization across all layers rather than independent treatment\nAchieve 275× parameter reduction vs. LoRA while preserving architectural characteristics\n\n**Why FFT**: The spectral concentration in low frequencies makes FFT ideal for efficient reconstruction with minimal parameters, leveraging modern optimized implementations for computational efficiency. This approach preserves Transformer characteristics during fine-tuning, achieving comparable or better performance with dramatically fewer parameters by exploiting architectural properties rather than ignoring them.\n\n## **Q3: Single Run**\n\nWe fix results by conducting five independent runs for each experiment with different random seeds and report the average results:\n\n| Method | NLU | CRS | IC |\n|--------|-----|-----|-----|\n| LoRA         | 87.92±0.31 | 76.84±0.28 | 76.53±0.42 |\n| CrossSpectra | 89.76±0.24 | 82.69±0.35 | 81.43±0.38 |\n\n\n\n## **Q4: Reconstructing full updates via iFFT each step could may potentially use memory**\n\n**4.1 Memory Complexity Analysis**\n\nWe acknowledge that reconstructing full updates via iFFT could temporarily require the same memory as the full adaptation tensor.  During the forward pass, CrossSpectra requires reconstructing the full adaptation tensor  $T_{QKV}\\in \\mathbb{R}^{(d×d×3L)}$  via `T_QKV ← torch.fft.ifftn(C_full)`, which temporarily uses **$O(3Ld^2)$**. \n\n\n**4.2 Chunked reconstruction strategy**\n\n However, we could apply the **Chunked reconstruction strategy** for controlling peak memory usage:\n\n**Memory-Efficient Forward Pass**\n```\n1. Divide layers into n chunks: {1,...,L} → {C₁, C₂, ..., Cₙ}\n2. For each chunk Cᵢ containing layers [l_start, l_end]:\n   -  Extract spectral coefficients: C_chunk ← C[:,:,3l_start:3l_end]\n   - Reconstruct chunk: T_chunk ← iFFT3D(C_chunk)  \n   - Apply adaptations: W̃ˡᴹ ← Wˡᴹ + ΔWˡᴹ for l ∈ Cᵢ, M ∈ {Q,K,V}\n   - Deallocate T_chunk\n```\n\n**4.3 Theoretical Justification**\n\nThis approach is theoretically sound because:\n\n1. **Spectral Structure Preservation**: Our theoretical analysis (Theorem 3.3) establishes that adaptation patterns exhibit the strongest decay in the cross-layer dimension (β₃ > β₁, β₂). The chunked approach preserves this structure within each chunk.\n\n2. **Gradient Smoothness Continuity**: The cross-layer gradient smoothness property (Theorem 3.2) ensures that adjacent layers have similar adaptations. Chunking maintains this local smoothness while reducing global memory requirements.\n\n**4.4 Complexity Analysis**\n\n- **Parameter Storage**: Remains **O(k₁k₂)** (unchanged)\n- **Peak Memory**: Reduced from **O(3Ld²)** to **O(3Ld²/n)** where n is the number of chunks\n\n\n## **Q5: Task range limited**\n\n**5.1 Robustness across different domains**\nThanks for the question. We have evaluated our method on a wide range of tasks, demonstrating robustness across different settings. Specifically, our experimental evaluation (Section 5) focuses on four specific task categories:\n- **Image Classification (IC)**: CLIP ViT-B/32 on 7 datasets\n- **Natural Language Understanding (NLU)**: RoBERTa-large on GLUE benchmark  \n- **Commonsense Reasoning (CR)**: LLaMA2-7B on 8 reasoning benchmarks\n- **Arithmetic Reasoning (AR)**: LLaMA2-7B on GSM8K, MAWPS, SVAMP, AQuA\n\n\n**5.2 Missing Evaluation**: As noted, we did not evaluate on **generation-heavy tasks** such as:\n- Text generation and completion\n- Machine translation\n- Dialogue systems\n- Creative writing tasks\n\nThis is because our evaluation design follows established practices for objective, quantifiable assessment of parameter-efficient fine-tuning methods.\n\n- **Commonsense Reasoning**: Yes/No and multiple-choice questions (BoolQ, PIQA, HellaSwag, etc.)\n- **Natural Language Understanding**: Classification tasks with definitive labels (GLUE benchmark)\n- **Arithmetic Reasoning**: Mathematical problems with exact numerical answers\n- **Image Classification**: Single correct class labels\n\nThis task approach enables deterministic performance measurement and fair comparison across methods.\nin contrast, generation-heavy tasks may be problematic, as it have \n\n- **Subjective Assessment:** Text generation quality relies on human judgment or complex automated metrics (BLEU, ROUGE) that may not reflect true quality differences between parameter-efficient methods.\n- **Multiple Valid Outputs:** Unlike classification tasks where there's one correct answer, generation tasks have numerous valid responses, making it difficult to isolate the impact of the parameter-efficient method from other factors.\n\n\n\n## **Q6: Hyperparameter sensitivity, (k_1, k_2)**\n\n**6.1 Parameter Selection Strategy:**\n- Optimal k₁ and k₂ values are determined through cross-validation on each dataset to ensure robust performance across tasks.\n- We recommend maintaining the layer sparsity ratio (k₁×k₂)/(d²×3L) around 2×10⁻³ for optimal parameter efficiency.\n- The cross-layer frequency range should cover approximately 20% of the spectrum (k₂/L), consistent with our theoretical analysis showing low-frequency dominance in adaptation energy (Figure 2).\n\n**6.2 Grid Search Results on Image Classification Tasks:**\n\n| k₁\\k₂ |  3   |  6   |  9   |\n|-------|------|------|------|\n| 1000  | 79.2 | 80.1 | 80.4 |\n| 2000  | 80.2 | 80.4 | 80.3 |\n| 3000  | 80.7 | 80.8 | 80.7 |\n| 4000  | 81.5 | 81.3 | 81.0 |\n\n**Key Observations:**\n- Performance improves with k₁ up to 4000, demonstrating the benefits of capturing more within-layer frequency components.\n- k₂ shows diminishing returns beyond 6, consistent with stronger spectral decay in the cross-layer dimension.\n- Optimal configuration: k₁=4000, k₂=3 (81.5% accuracy) with minimal parameter overhead."}, {"review_id": "dyo4hQ7MBT", "reviewer": "Reviewer_RbH6", "summary": "This paper introduces CrossSpectra, a parameter-efficient fine-tuning (PEFT) method that achieves extensive parameter reduction by exploiting the structural properties of transformer architectures. The authors posit that skip connections induce \"cross-layer smoothness,\" causing weight adaptations during fine-tuning to be highly correlated across layers and concentrated in low-frequency spectral components. Instead of tuning each layer independently like LoRA, this paper parameterizes all attention weight adaptations (Q, K, V) as a single 3D tensor, decomposes it using a 3D Fourier transform, and aggressively truncates the high-frequency components. Experiments show that CrossSpectra can match or exceed the performance of state-of-the-art PEFT methods while using orders of magnitude fewer parameters, notably demonstrating a 275x reduction compared to LoRA on LLaMA-7B.", "questions": "1. How were the values for k₁=1000 and k₂=3 determined? \n2. The theoretical motivation relies heavily on skip connections in transformers. How would this method apply to architectures with different residual connection patterns or to non-transformer models?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The core idea of linking skip connections to cross-layer gradient smoothness and then to spectral concentration is a novel contribution. It provides an interesting theoretical justification for why adaptations can be parameterized jointly across layers, moving beyond the layer-independent assumption of methods like LoRA.\n\n2. The empirical results are quite compelling. The authors validate CrossSpectra across a diverse set of tasks (NLU, commonsense reasoning, image classification, arithmetic reasoning) and model architectures (LLaMA, RoBERTa, CLIP), demonstrating the generalizability of their approach.\n\n3. The theoretical analysis presented in the paper, particularly the derivation from gradient smoothness to spectral concentration, appears sound.\n\nWeaknesses:\n\n1. As acknowledged in the paper, the method introduces computational overhead from the 3D FFT/iFFT operations in the forward and backward passes. The paper claims this is \"negligible in practice\" for typical sequence lengths, but it could become a bottleneck for extremely large models or resource-constrained environments. An empirical study on the scaling of computational load with model size would be appreciated.\n\n2. The current method exclusively adapts the Q, K, and V attention matrices. While these are information-dense, it potentially misses optimization opportunities in other components like the feed-forward layers, which also contain a large number of parameters.\n\n3. The method's performance depends on the choice of k1 and k2. The process for selecting these hyperparameters for new tasks or models may require careful tuning.", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1:  An empirical study on the scaling of computational load with model size would be appreciated.**\n\nThank you for the question. We provide a comprehensive analysis of computational scaling across different model sizes and configurations. For the memory analysis, please kindly see our response to Q4 from reviewer puJh, which details our approach to peak memory management.\n\n**Model Performance and Training Time Comparison**\n\nWe measure computational speed based on one forward pass and average performance on commonsense reasoning tasks:\n\n| Model Size | Layers | Parameters | Time per Epoch | Avg. Performance |\n|------------|--------|------------|----------------|------------------|\n| LLaMA2-13B | 40     | 13B        | 1.82s          | 83.52            |\n| LLaMA2-7B  | 32     | 7B         | 0.79s          | 82.73            |\n| LLaMA2-7B (LoRA) | 32 | 7B       | 0.61s          | 77.61            |\n\n**Key Observations:**\n- Training time scales sub-linearly with model size due to sparse parameterization\n- LoRA shows faster training but lower performance, highlighting the efficiency-accuracy trade-off\n\n**FFT Runtime Benchmarking**\n\nSince our implementation relies on `torch.fft.ifftn`, we provide runtime analysis for different 3D tensor sizes:\n\n| Tensor Size        | FFT Computation Time |\n|--------------------|----------------------|\n| (4096, 4096, 3×10) | 0.05s               |\n| (4096, 4096, 3×20) | 0.12s               |\n| (4096, 4096, 3×30) | 0.28s               |\n| (4096, 4096, 3×40) | 0.41s               |\n\n**Scaling Analysis:**\n- FFT computation scales approximately O(N log N) with tensor size, as expected\n- The 3D FFT overhead remains negligible compared to the attention mechanism's O(n²d) complexity\n- Modern GPU implementations ensure efficient parallel computation of frequency transformations\n\n\n## **Q2: While QKV are information-dense, it potentially misses optimization opportunities in other components like the feed-forward layers, which also contain a large number of parameters.**\n\nThank you for this insightful question. While adapting QKV layers follows established conventions in the LoRA literature, we conducted additional experiments to explore the impact of including feed-forward (FF) layers in our adaptation strategy.\n\n**1.1 Experimental Results:**\n\n**Commonsense Reasoning Tasks (LLaMA2-7B)**\n| Method | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average |\n|--------|-------|------|------|-----------|------------|-------|-------|------|---------|\n| CrossSpectra (QKV)    | 73.69 | 83.95 | 80.34 | 88.42 | 87.24 | 87.75 | 76.88 | 87.21 | **82.73** |\n| CrossSpectra (QKV+FF) | 72.85 | 83.81 | 79.67 | 87.89 | 86.51 | 87.72 | 75.94 | 86.33 | 82.59 |\n\n**Natural Language Understanding Tasks (RoBERTa-large)**\n| Method | CoLA | SST-2 | MRPC | QQP | MNLI | QNLI | RTE | Average |\n|--------|------|-------|------|-----|------|------|-----|---------|\n| CrossSpectra (QKV)    | 86.86 | 96.21 | 84.55 | 91.40 | 89.55 | 94.19 | 85.56 | **89.76** |\n| CrossSpectra (QKV+FF) | 85.92 | 96.11 | 83.78 | 91.35 | 88.73 | 93.42 | 84.21 | 89.07 |\n\n**Image Classification Tasks (ViT-B/32)**\n| Method | Cars | DTD | EuroSAT | GTSRB | RESISC45 | SUN397 | SVHN | Average |\n|--------|------|-----|---------|-------|----------|--------|------|---------|\n| CrossSpectra (QK)     | 53.50 | 75.32 | 98.82 | 98.17 | 93.46 | 54.53 | 96.62 | **81.49** |\n| CrossSpectra (QK+FF)  | 52.84 | 75.32 | 97.98 | 97.34 | 92.17 | 54.53 | 95.73 | 80.84 |\n\n**1.2 Analysis:**\nOur experiments reveal that including feed-forward layers consistently leads to performance degradation across all modalities. This counterintuitive result can be attributed to:\n\n1. **Overfitting Risk**: Adding more adaptable parameters increases the likelihood of overfitting, particularly given our extremely sparse parameterization\n2. **Spectral Mismatch**: Feed-forward layers may not exhibit the same cross-layer spectral properties as attention mechanisms, making them less suitable for our frequency-domain approach\n3. **Information Density**: QKV matrices capture the core relational information in transformers, while FF layers primarily perform non-linear transformations that may not benefit from cross-layer frequency sharing\n\nThese findings validate our design choice to focus on attention mechanisms, where the cross-layer smoothness induced by skip connections is most pronounced and beneficial for spectral parameterization.\n\n\n\n## **Q3: Hyperparameter Tuning for $k_1$ and $k_2$**\n\n**3.1 Parameter Selection Strategy:**\n- Optimal k₁ and k₂ values are determined through cross-validation on each dataset to ensure robust performance across tasks.\n- We recommend maintaining the layer sparsity ratio (k₁×k₂)/(d²×3L) around 2×10⁻³ for optimal parameter efficiency.\n- The cross-layer frequency range should cover approximately 20% of the spectrum (k₂/L), consistent with our theoretical analysis showing low-frequency dominance in adaptation energy (Figure 2).\n\n**3.2 Grid Search Results on Image Classification Tasks:**\n\n| k₁\\k₂ |  3   |  6   |  9   |\n|-------|------|------|------|\n| 1000  | 79.2 | 80.1 | 80.4 |\n| 2000  | 80.2 | 80.4 | 80.3 |\n| 3000  | 80.7 | 80.8 | 80.7 |\n| 4000  | 81.5 | 81.3 | 81.0 |\n\n**Key Observations:**\n- Performance improves with k₁ up to 4000, demonstrating the benefits of capturing more within-layer frequency components.\n- k₂ shows diminishing returns beyond 6, consistent with stronger spectral decay in the cross-layer dimension.\n- Optimal configuration: k₁=4000, k₂=3 (81.5% accuracy) with minimal parameter overhead."}, {"review_id": "XBK2IyEaZI", "reviewer": "Reviewer_obhA", "summary": "This paper introduces a new PEFT method CrossSpectra. The key innovation lies in leveraging the cross-layer smoothness induced by skip connections in transformers, which results in weight adaptations concentrated in low-frequency spectral components. The method is validated across diverse tasks, including natural language understanding, image classification, and arithmetic reasoning, and shows strong results.", "questions": "1. In Eq (2), why is the input of the FFN layer $H_{l−1}$ instead of the $H_{l−1} + MultiHeadAttention_l(H_{l−1})$? Is this consistent with standard transformer implementations?\n2. How are $k1$ and $k2$ chosen? Is there a theoretical or empirical guideline?\n3. How does CrossSpectra’s spectral approach differ from low-rank tensor factorization (e.g., Tucker/CP decomposition) in terms of efficiency and performance?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Evaluated across diverse tasks (NLU, vision, reasoning) and model architectures (LLaMA-7B, RoBERTa, CLIP). The paper demonstrates 275× parameter reduction compared to LoRA and 5,250× compared to full fine-tuning.\n\nWeaknesses\n1. The paper overlooks recent work of low-rank tensor decomposition (e.g., FaCT[1]) and layer-shared LoRA variants (e.g., VB-LoRA[2]). A comparison with these methods would better contextualize CrossSpectra’s contributions. Please consider adding a discussion comparing CrossSpectra’s spectral approach with tensor factorization alternatives.\n\n2. While the authors claim FFT overhead is \"negligible,\" no concrete training time or memory footprint comparisons are provided. For large-scale models, this could be a bottleneck. Including metrics like wall-clock time per epoch or GPU memory usage versus LoRA would be appreciated. \n\n- [1] Jie, S., & Deng, Z. H. (2023, June). Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 1, pp. 1060-1068).\n- [2] Li, Y., Han, S., & Ji, S. (2024). VB-LoRA: extreme parameter efficient fine-tuning with vector banks. NeurIPS 2024.", "was_revised": true, "final_justification": "The paper introduces a novel PEFT method, validated across a diverse set of tasks including natural language understanding, image classification, and arithmetic reasoning, where it demonstrates strong performance. During the rebuttal period, the authors provided important additional baselines and time consumption analysis. Overall, I believe this paper merits acceptance.", "rebuttal": "## **Q1 Add baseline FaCT[1] and VB-LoRA[2]**\n\n\nThank you for the suggestion. We have included comparisons with these important baseline methods (our reproduced result).\n\n **1.1 Experiment result**\n**Commonsense Reasoning (LLaMA2-7B):**\n| Method           | BoolQ | PIQA  | SIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|------------------|-------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| CrossSpectra     | 73.69 | 83.95 | 80.34 | 88.42     | 87.24      | 87.75 | 76.88 | 87.21 | 82.73   |\n| VB-LoRA          | 72.31 | 83.41 | 79.12 | 87.25     | 85.93      | 87.21 | 75.26 | 85.78 | 82.03   |\n\n**Image Classification (CLIP ViT-B/32):**\n| Method           | Cars  | DTD   | EuroSAT | GTSRB | RESISC45 | SUN397 | SVHN  | Average |\n|------------------|-------|-------|---------|-------|----------|--------|-------|---------|\n| CrossSpectra     | 53.50 | 75.32 | 98.82   | 98.17 | 93.46    | 54.53  | 96.62 | 81.49   |\n| FaCT             | 51.92 | 74.65 | 97.14   | 96.58 | 91.33    | 53.87  | 94.89 | 80.05   |\n\n**1.2 Discussion on CrossSpectra and Tensor factorization methods**\n\n- **Architectural Awareness**: Unlike other tensor factorization methods, CrossSpectra explicitly leverages the cross-layer smoothness property induced by skip connections in modern Transformers, aligning adaptations with the model's inherent architectural characteristics.\n\n- **Spectral Structure Exploitation**: While existing methods perform generic low-rank decomposition, CrossSpectra exploits the frequency-domain structure of weight adaptations, concentrating parameters in the most informative low-frequency components.\n\n- **Dual Benefits**: CrossSpectra simultaneously reduces trainable parameters and preserves the network's inherent layer smoothness property, leading to both efficiency gains and performance improvements.\n\nThis architecture-aware approach explains why CrossSpectra consistently outperforms both VB-LoRA (+1.2% average) and FaCT (+1.44% average) while using comparable or fewer parameters.\n\n\n\n\n## **Q2 Including metrics like wall-clock time per epoch or GPU memory usage versus LoRA would be appreciated.**\n\nThank you for the question. We provide a comprehensive analysis of computational scaling across different model sizes and configurations. For the memory analysis, please see our response to Q4 from reviewer puJh, which details our approach to peak memory management.\n\n**Model Performance and Training Time Comparison**\n\nWe measure computational speed based on one forward pass and average performance on commonsense reasoning tasks:\n\n| Model Size | Layers | Parameters | Time per Epoch | Avg. Performance |\n|------------|--------|------------|----------------|------------------|\n| LLaMA2-13B | 40     | 13B        | 1.82s          | 83.52            |\n| LLaMA2-7B  | 32     | 7B         | 0.79s          | 82.73            |\n| LLaMA2-7B (LoRA) | 32 | 7B       | 0.61s          | 77.61            |\n\n**Key Observations:**\n- Training time scales sub-linearly with model size due to sparse parameterization\n- LoRA shows faster training but lower performance, highlighting the efficiency-accuracy trade-off\n\n**FFT Runtime Benchmarking**\n\nSince our implementation relies on `torch.fft.ifftn`, we provide runtime analysis for different 3D tensor sizes:\n\n| Tensor Size        | FFT Computation Time |\n|--------------------|----------------------|\n| (4096, 4096, 3×10) | 0.05s               |\n| (4096, 4096, 3×20) | 0.12s               |\n| (4096, 4096, 3×30) | 0.28s               |\n| (4096, 4096, 3×40) | 0.41s               |\n\n**Scaling Analysis:**\n- FFT computation scales approximately O(N log N) with tensor size, as expected\n- The 3D FFT overhead remains negligible compared to the attention mechanism's O(n²d) complexity\n- Modern GPU implementations ensure efficient parallel computation of frequency transformations"}, {"review_id": "mui5ktmCKj", "reviewer": "Reviewer_WouS", "summary": "This paper proposes a novel PEFT algorithm based on skip connections, which saves 275 times the parameters. The proposed architecture achieves seemingly promising results on both text and vision tasks.", "questions": "See above", "limitations": "See above", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n(1) The motivation is interesting, as it hypothesizes the inefficiency of existing PEFT methods in deep model adaptation based on an analysis of energy distribution.\n(2) The results appear robust, with tests conducted on multiple tasks demonstrating advanced performance.\n\nWeaknesses:\n(1) The baseline methods on vision tasks do not seem state-of-the-art. Recent works [1-4] have focused on vision PEFT methods and achieved more advanced performance. It is recommended to include discussions of these methods in the related work section and possibly conduct comparative experiments with some of them.\n\n[1] 5%> 100%: Breaking performance shackles of full fine-tuning on visual recognition tasks, CVPR'25.\n\n[2] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions, CVPR'23.\n\n[3] Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks, CVPR'22.\n\n[4] Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions, ACMMM'24.\n\n(2) The methodology appears to be described in a somewhat complex manner. The authors are advised to present the core ideas and advantages of the new method more intuitively, possibly through diagrams. Additionally, the mathematical formulations could be simplified where necessary.", "was_revised": true, "final_justification": "The author partially resolved my issue, and I will increase my rating.", "rebuttal": "## **Q1: Add baseline**\nThanks for the related work. We will cite **ALL** the suggested papers and discuss them in the revision. \n\nWe compare with MONA[1] and LORAND[2] as they are most closely related to our Work. We add the MONA and LORAND adapters before the layer normalization of ViT-B/32, following the same experimental setup as Table 4.\n\n**Image Classification Results (ViT-B/32)**\n| Method | Cars | DTD | EuroSAT | GTSRB | RESISC45 | SUN397 | SVHN | Average |\n|--------|------|-----|---------|-------|----------|--------|------|---------|\n| CrossSpectra      | 53.50 | 75.32 | 98.82 | 98.17 | 93.46 | 54.53 | 96.62 | **81.49** |\n| MONA[1]           | 52.15 | 74.68 | 98.24 | 97.51 | 92.83 | 53.89 | 95.94 | 80.75 |\n| LORAND[2]         | 51.89 | 74.45 | 98.08 | 97.28 | 92.65 | 53.72 | 95.81 | 80.55 |\n\n**Key Differences and Advantages:**\n\nThe fundamental difference lies in the adaptation mechanism:\n\n- **CrossSpectra**: Follows the LoRA-style approach where adaptation is achieved through direct weight modification: **W = W₀ + ΔW**. This enables seamless integration without altering the original architecture.\n\n- **MONA & LORAND**: Inject additional trainable layers g(x) with parameters ΔW into the main computational branch, transforming the forward pass from f(x) to f(g(x)).\n\n**Advantages of Our Approach:**\n- **Plug-and-Play Nature**: CrossSpectra maintains the original architecture integrity, allowing easy deployment without structural modifications.\n- **Inference Efficiency**: No additional computational overhead during inference since adaptations are merged into existing weights.\n- **Superior Performance**: Achieves 0.74% and 0.94% higher average accuracy compared to MONA and LORAND, respectively.\n\n\n\n## **Q2 The methodology is described in a somewhat complex manner. Present the core ideas and advantages of the new method more intuitively, possibly through diagrams. Additionally, the mathematical formulations could be simplified where necessary.**\n\nThank you for this valuable suggestion. We will add more intuitive diagrams and simplify the mathematical formulations in the revision. Since we cannot provide the updated PDF in this rebuttal, we explain the core ideas and advantages more intuitively below:\n\n**2.1 Core Idea in Simple Terms:**\nThink of weight adaptations across transformer layers like a smooth curve rather than independent points. Just as a smooth curve can be efficiently represented using only its low-frequency components, we can represent all layer adaptations using sparse spectral coefficients.\n\n**2.2 Three Key Steps:**\n\n1. **Skip Connections Create Smoothness**: Skip connections ($H_l = H_{l-1} + f(H_{l-1})$) make adjacent layers similar, producing smooth gradient changes during fine-tuning rather than abrupt variations.\n\n2. **Smoothness Concentrates Energy**: This smoothness causes ~70% of adaptation energy to concentrate in low-frequency spectral components (Figure 1), with the strongest concentration across the layer dimension.\n\n3. **Sparse Representation Enables Efficiency**: Since most energy resides in low frequencies, we only store these essential coefficients and use 3D inverse FFT to reconstruct the full adaptation tensor, achieving 275× parameter reduction compared to LoRA.\n\n**2.3 Key Advantage:** Unlike LoRA, which treats each layer independently, CrossSpectra exploits the natural architectural properties of Transformers, enabling joint parameterization across all layers with dramatically fewer parameters while maintaining comparable or better performance.\n\n**2.4 Intuitive Analogy:** Imagine compressing a smooth signal—you need far fewer frequency components than for a noisy signal. Similarly, the smooth cross-layer adaptations in Transformers can be compressed much more efficiently than independent layer-wise adaptations."}]

Author rebuttal:

Meta-review: All four reviewers rated the paper positively, with concerns addressed convincingly in the rebuttal. CrossSpectra’s ​novel exploitation of cross-layer spectral smoothness​ achieves ​275× parameter reduction​ vs. LoRA while matching/exceeding SOTA performance across NLU, vision, and reasoning tasks. Key concerns (scalability, motivation, baselines) were resolved: scalability validated on LLaMA-13B, statistical significance confirmed via multi-run experiments, and new baselines (FaCT/VB-LoRA) outperformed. The ​method’s originality, efficiency gains, and rigorous validation​ outweigh minor limitations, making this a valid contribution to PEFT.
Decision: Accept (poster)