Title: CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Parameter-efficient fine-tuning (PEFT)
Abstract: Parameter-efficient fine-tuning (PEFT) is essential for adapting large foundation models without excessive storage cost. However, current approaches such as LoRA treat each layer’s adaptation independently, overlooking correlations across layers. This independence causes the number of trainable parameters to grow linearly with model depth. We provide theoretical and empirical evidence that skip connections in transformers create smooth gradient propagation across layers. This smoothness leads to weight adaptations that concentrate most of their energy in low-frequency spectral components, especially along the layer dimension. Empirical analysis confirms this effect, showing that most of adaptation energy lies in low frequencies. Building on this insight, we propose CrossSpectra, which parameterizes all attention-weight adaptations $(Q, K, V)$ across layers as a single 3D tensor and represents them with sparse spectral coefficients ($\kappa_1, \kappa_2$). Using $\kappa_{1}$ non-zero coefficients within each layer’s frequency space and truncating to $\kappa_{2}$ frequencies across layers, CrossSpectra requires $\mathcal{O}(\kappa_{1}\kappa_{2})$ parameters instead of LoRA’s $\mathcal{O}(Lrd)$, where $L$ is the number of layers and $r$ the rank. Across natural-language and vision benchmarks, \methodname{} matches or surpasses baseline performance while using fewer parameters than LoRA, achieving only $0.36\%$ of LoRA’s parameter count when fine-tuning LLaMA-7B on instruction-following tasks. These results show that exploiting the \textbf{architectural smoothness of transformers} through spectral analysis yields major efficiency gains in PEFT.


Peer reviews:
comments: SUMMARY:
This paper proposes a novel PEFT algorithm based on skip connections, which saves 275 times the parameters. The proposed architecture achieves seemingly promising results on both text and vision tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
(1) The motivation is interesting, as it hypothesizes the inefficiency of existing PEFT methods in deep model adaptation based on an analysis of energy distribution.
(2) The results appear robust, with tests conducted on multiple tasks demonstrating advanced performance.

Weaknesses:
(1) The baseline methods on vision tasks do not seem state-of-the-art. Recent works [1-4] have focused on vision PEFT methods and achieved more advanced performance. It is recommended to include discussions of these methods in the related work section and possibly conduct comparative experiments with some of them.

[1] 5%> 100%: Breaking performance shackles of full fine-tuning on visual recognition tasks, CVPR'25.

[2] 1% vs 100%: Parameter-efficient low rank adapter for dense predictions, CVPR'23.

[3] Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks, CVPR'22.

[4] Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions, ACMMM'24.

(2) The methodology appears to be described in a somewhat complex manner. The authors are advised to present the core ideas and advantages of the new method more intuitively, possibly through diagrams. Additionally, the mathematical formulations could be simplified where necessary.

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new PEFT method CrossSpectra. The key innovation lies in leveraging the cross-layer smoothness induced by skip connections in transformers, which results in weight adaptations concentrated in low-frequency spectral components. The method is validated across diverse tasks, including natural language understanding, image classification, and arithmetic reasoning, and shows strong results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Evaluated across diverse tasks (NLU, vision, reasoning) and model architectures (LLaMA-7B, RoBERTa, CLIP). The paper demonstrates 275× parameter reduction compared to LoRA and 5,250× compared to full fine-tuning.\n\nWeaknesses\n1. The paper overlooks recent work of low-rank tensor decomposition (e.g., FaCT[1]) and layer-shared LoRA variants (e.g., VB-LoRA[2]). A comparison with these methods would better contextualize CrossSpectra’s contributions. Please consider adding a discussion comparing CrossSpectra’s spectral approach with tensor factorization alternatives.\n\n2. While the authors claim FFT overhead is \"negligible,\" no concrete training time or memory footprint comparisons are provided. For large-scale models, this could be a bottleneck. Including metrics like wall-clock time per epoch or GPU memory usage versus LoRA would be appreciated. \n\n- [1] Jie, S., & Deng, Z. H. (2023, June). Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 1, pp. 1060-1068).\n- [2] Li, Y., Han, S., & Ji, S. (2024). VB-LoRA: extreme parameter efficient fine-tuning with vector banks. NeurIPS 2024.\n\nQUESTIONS:\n1. In Eq (2), why is the input of the FFN layer $H_{l−1}$ instead of the $H_{l−1} + MultiHeadAttention_l(H_{l−1})$? Is this consistent with standard transformer implementations?\n2. How are $k1$ and $k2$ chosen? Is there a theoretical or empirical guideline?\n3. How does CrossSpectra’s spectral approach differ from low-rank tensor factorization (e.g., Tucker/CP decomposition) in terms of efficiency and performance?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CrossSpectra, a parameter-efficient fine-tuning (PEFT) method that achieves extensive parameter reduction by exploiting the structural properties of transformer architectures. The authors posit that skip connections induce \"cross-layer smoothness,\" causing weight adaptations during fine-tuning to be highly correlated across layers and concentrated in low-frequency spectral components. Instead of tuning each layer independently like LoRA, this paper parameterizes all attention weight adaptations (Q, K, V) as a single 3D tensor, decomposes it using a 3D Fourier transform, and aggressively truncates the high-frequency components. Experiments show that CrossSpectra can match or exceed the performance of state-of-the-art PEFT methods while using orders of magnitude fewer parameters, notably demonstrating a 275x reduction compared to LoRA on LLaMA-7B.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The core idea of linking skip connections to cross-layer gradient smoothness and then to spectral concentration is a novel contribution. It provides an interesting theoretical justification for why adaptations can be parameterized jointly across layers, moving beyond the layer-independent assumption of methods like LoRA.\n\n2. The empirical results are quite compelling. The authors validate CrossSpectra across a diverse set of tasks (NLU, commonsense reasoning, image classification, arithmetic reasoning) and model architectures (LLaMA, RoBERTa, CLIP), demonstrating the generalizability of their approach.\n\n3. The theoretical analysis presented in the paper, particularly the derivation from gradient smoothness to spectral concentration, appears sound.\n\nWeaknesses:\n\n1. As acknowledged in the paper, the method introduces computational overhead from the 3D FFT/iFFT operations in the forward and backward passes. The paper claims this is \"negligible in practice\" for typical sequence lengths, but it could become a bottleneck for extremely large models or resource-constrained environments. An empirical study on the scaling of computational load with model size would be appreciated.\n\n2. The current method exclusively adapts the Q, K, and V attention matrices. While these are information-dense, it potentially misses optimization opportunities in other components like the feed-forward layers, which also contain a large number of parameters.\n\n3. The method's performance depends on the choice of k1 and k2. The process for selecting these hyperparameters for new tasks or models may require careful tuning.\n\nQUESTIONS:\n1. How were the values for k₁=1000 and k₂=3 determined? \n2. The theoretical motivation relies heavily on skip connections in transformers. How would this method apply to architectures with different residual connection patterns or to non-transformer models?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CrossSpectra, a parameter–efficient fine-tuning (PEFT) method that exploits cross-layer spectral smoothness in transformer attention weights. The authors show that skip connections make gradients smooth across layers, implying that weight updates mainly occur in low frequencies. Then they stack all Q,K,V adaptations into a single $d{}d{}3L$ tensor and learn only $k_1$ sparse spatial frequencies and $k_2$ low cross-layer frequencies. By a 3-D iFFT,  this yields $O(k_1k_2)$ trainable parameters, instead of traditional $O(Lrd)$. On LLaMA-7B instruction tuning, CrossSpectra matches or exceeds LoRA while using only 0.36% of its parameters. It also matches state-of-the-art PEFT baselines with very small expense. Overall, CrossSpectra shows that leveraging transformer spectral smoothness can reduce storage without sacrificing too much of the accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n  -  *Conceptual novelty.*  This paper utilizes cross-layer spectral smoothness to collapse all Q/K/V parameters into a single 3-D Fourier space, which slash PEFT parameters.\n  -  *Theory  and evidence.*  It provides a gradient-smoothness bound and empirical spectral-energy plots that explain why low cross-layer frequencies matter most.\n  -  *Logical clear proof.*  The proof of the theorem seems to be clear, with reasonable conditions and logical proof.\n\n**Weaknesses**\n\n  -  *Limited scale.*  The result has only been operated on a 7 B-parameter LLaMA; no evidence shows that the approach holds for the model with more parameters.\n  -  *Lack preliminary.*  The motivation has not been clearly stated, for example, why this algorithm is applied, and the related preliminaries, such as fast Fourier transformation, are not explicitly introduced.\n  -  *Single report.*  All scores come from one run, and variance between different runs are not recorded or analyzed.\n  -  *Memory not measured.*  Reconstructing full $dd$ updates via iFFT each step could may potentially use memory,  but experiments relating to memory usage is not measured.\n  -  *Task range limited.*  There is no evaluation on generation-heavy tasks; robustness across different domains (setups) is not tested, which has been mentioned in the limitation.\n  -  *Hyperparameter sensitivity.*  The choice of hyperparameters to train is stochastic and the training process is sensitive to the selection, which has been mentioned in the limitation.\n\nQUESTIONS:\n-  The $(k_1,k_2)$ frequency budgets are tested on LLaMA-7B. How should people choose these values for deeper models with more parameters? I suggest that related experiments should be taken.\n  -  What is the motivation of this paper? It seems that the story line in not clear, and I would like to know more about how this method was developed by inspirations.\n  -  Have the authors tested CrossSpectra on models with more layers?  Does the low-frequency assumption still hold at that scale? I concern that although this holds for the case in the paper, it may have its flaw in some domains\n  -  Does updates to low cross-layer frequencies reduce adaptation on generation-heavy tasks where later layers may have larger shifts instead of smooth change? I think this might be the case, so this impact should also be evaluated\n\nLIMITATIONS:\nApart from the limitations that have been mentioned in the appendix, CrossSpectra still have several potential limitations to improve. First, all results come from single experiments with no analysis in variance across several experiments, so the reported effectiveness might at a risk of statistical noise. Also, it seems that memory usage during training is not recorded, and since the choose of hyperparameter is not deterministic, it would be better to keep track of the memory. Finally, its effectiveness on more domains and architecture with more layers are not yet checked, so it would be better if more experiments relating to different models are tested.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}