SAN: Hypothesizing Long-Term Synaptic Development and Neural Engram Mechanism in Scalable Model's Parameter-Efficient Fine-Tuning
Authors: Gaole Dai
Chun-Kai Fan
Yiming Tang
Zhi Zhang
Yuan Zhang
Yulu Gan
Qizhe Zhang
Cheng-Ching Tseng
Shanghang Zhang
Tiejun Huang
Venue: ICML 2025 poster (2025)
Keywords: Parameter-efficient Fine-tuning
Foundation Models
Neuroscience
Neural Engrams
Long-term Synaptic Development
Primary area: deep_learning->everything_else

Abstract: Advances in Parameter-efficient Fine-tuning (PEFT) bridged the performance gap with Full Fine-Tuning (FFT) through sophisticated analysis of pre-trained parameter spaces. Starting from drawing insights from Neural Engrams (NE) in Biological Neural Networks (BNNs), we establish a connection between the low-rank property observed during PEFT's parameter space shifting and neurobiological mechanisms. This observation leads to our proposed method, **S**ynapse and **N**euron (**SAN**), which decomposes and propagates the scaling component from anterior feature adjustment vectors towards posterior weight matrices. Our approach is theoretically grounded in Long-Term Potentiation/Depression (LTP/D) phenomena, which govern synapse development through neurotransmitter release modulation. 
Extensive experiments demonstrate its effectiveness: on **vision tasks** across VTAB, FGVC, and GIC (25 datasets) using ViT, Swin-T and ConvNeXt architectures, SAN outperforms FFT up to *8.7%* and LoRA by *3.2%*; on **language tasks** using Commonsense Reasoning (8 datasets) with LLaMA models (all generations), surpassing ChatGPT up to *8.5%* and LoRA by *4.7%*; on **vision-language tasks** using Visual Instruction Tuning (7 datasets) with LLaVA models, it exceeds FFT up to *2.4%* and LoRA by *1.9%*. Our code and W&B log will be released

SAN: Hypothesizing Long-Term Synaptic Development and Neural Engram Mechanism in Scalable Model's Parameter-Eficiene Fine-Tuning
Gaole Dai
Chun-Kai Fan
Yiming Tang² Zhi Zhang'
Yuan Zhang' Yulu Gan Qizhe Zhang'
Cheng-Ching Tseng 15 Shanghang Zhang =1 Tiejun Huang =1
Abstract
LoRA
Adjust
Advances in Garameter-Efficien Fine-Tuning (PEFT) bridged the performance gap with Full Fine-Tuning (FFT) through sophisticated analysis of pre-trained parameter spaces. Starting from drawing insights from Neural Engrams (NE) in Biological Neural Networks (BNNs), estab- lish connection between the low-rank property observed during PEFT's parameter space shifting and neurobiological mechanisms. This observa- tion Ieads to our proposed method, Synapse and Neuron (SAN), which decomposes and propa- gates scaling components from anterior feature adjusting vectors towards posterior weight trices. Our approach is theoretically grounded in Term Potentiation/Deprssse phenomena, which govern synapse development through neurotransmitter release modulation. Ex- tensive experiments demonstrate effectiveness: on vision fasks across VTAB, FGVC, and GIC (25 datasets) using VIT, Swin'T and ConvNeXt, SAN outperforms FFT up 8.7% and LoRA by 3. 3.2%; on language tasks using Commonsense Reasoning (8 datasets) with LLaMA models (all generations). surpassing ChatGPT up to 8.5% and LoRA by 4.7%; on visual-language tasks using Mixed Visual Instruction (7 datasets) with LLaVVA models, exceeds FFT up 2.4% and LoRA by 1.9%. Our code and W&B log will be released.
Pre-trained W
Features
Dutputs
Fine-tuned
Inputs
Adjust Vector (current layer)
B
Adjust Vector (from anterior)
Pre-trained
decompose
propagate
Features
(to postertor)
ti W' [SAN)
Scaling Vector
Outputs
shift Complexity
Efficiency
SAN
(LTP/D)
Scaling Adding
Reparameterization
Tuning Difficultty
Figure I. Plug-and- SAN pipeline. core design SAN
involves extracting the scaling component by the decomposition of adjusting vectors from preceding layers and reapplying to subsequent layers. This approach simplifies learning objectives and enhances expressiveness hy providing scaling priors without introducing additional trainable parameters. SAN compatible with various PEFT techniques, such as LoRA.
1. Introduction
Transfer Learning attempts shift the pre-trained param- eter space to the downstream task's optimal parameter space. Proper Full Fine-Tuning (FFT) can accomplish this task (Shuttleworth et al., 2024; Biderman et al., 2024). How- ever, as models grow larger and tasks become more diverse, finding optimal FFT hyperparameters becomes impracti- cal (Liu et al., 2024c). Thus, Parameter- Efficieen Fine- Tuning (PEFT) becomes a better choice can approxi- mately simulate FFT's functionality and find near-optimal hyperparameters through multiple training rounds at low cost (Hu et al., 2022; Liu et al., 2024b; Lian et al., 2022)
"Equal contribution 'State Key Laboratory of Multimedia In- formation Processing, School of Computer Science, Peking Uni- versity NNational University of Singapore 'Institute for Logic, Lan- guage and Computation, University of Amsterdam "Massachusetts Institute of Technology 'MeiTuan (This work was done when Cheng-Ching Tseng
master student Peking Univer- sity). Correspondence to: Tiejun Huang Shanghang Zhang <shanghang @ pku.edu.en>>.
@ww..cdu.cn>.
PEFT evolved from Linear Probing (LP), which approxi- mates the shift of entire network parameters through shifting network output, to sparse tuning techniques like Bias Fine-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SAN-PEFT
Engram differ from task to task
The adjust vectors from different layers
Scaling Features (SSF)
Scaling Features Propagate (SFN
Blocko
Task Ca1
Block
Task B- B-Cifar100
Block
QKV
Block
QKV
Block
Block 11
Block
Block
T
Block
FFN-L1
Block
FFN-L1
LTP/D ease engram formation by propagation
Block
Block
Propagate Direction
Block:0
Block
Stimulus
Variance:
Variance: O 100x
Block
FFN-L2
O 10x
Blocks
FFN-L2
Accuracy:
Accuracy:
Block
1.2% (from 92%)
5.3% (from 69%)
0.5
1.0
1.4
0.6
1.0
1.4
Figure Motivations and observations decompose & propagate in SAN. Left panels: illustrate the concepts of Neural Engram (NE) and Long-term Potentiation/Depresse (LTP/D). Right panel: we applied PEFT pre-trained VIT-B to two VTAB subsets. This involved scaling the features using layer-wise trainable scaling vectors, akin to SSF QKV and FFN-L1/L2 represent the Attention layer and Feed Forward Network layer 1 &2, respectively. The dotted and solid lines indicate the histogram of scaling vectors from different layers with the presence or absence of SAN, respectively. Analysis of those vectors reveals significant intra-task similarities hut marked inter-task differences, consistent with the principles of NE. Moreover. SAN effectively controls the variance (a²) of the scaling vectors, thereby allowing for more nuanced adjustments and mitigating limitations in expressiveness, which aligns with the mechanisms of LTP/D.
Tuning (BitFit) (Zaken et al., 2021) and Gradient Parameter Selection (GPS) (Zhang et al., 2024b) which more explicitly covers different network parts for parameter space shifting. Later, techniques like Low-Rank Adaptation (LoRA) (Hu al., 2022) emerged, using additional trainable low-rank parameters model shifting rules in the original parameter space, addressing the insufficient expressiveness of sparse tuning. Recent works like Scale and Shift (SSF) (Lian et al., 2022) and Weight--eecomposed LoRA (DoRA) (Liu et al., 2024b) systematically analyzed PEFT's shifting properties in the original parameter space, independently concluding these shifts' low-rank (approximately linear) nature. Specif- ically, SSF directly applies linear shifts to features, equiv- alent to applying linear transformation toward pre-trained parameters. The state-of-the-art performance of SSF in vi- sion tasks unravels the minimum (i.e.. linear) requirements to adapt the pre-trained model to new tasks. On the other hand, A builds upon LoRA by decomposing pre-trained weight matrices into direction and magnitude components and updating the direction component by LoRA. This de- composition seems to constrain update, however, extensive experiments in scaled-up language tasks show its validity. Further demonstrated optimizing shifting strategies is more crucial than introducing more trainable parameters (e.g., increasing the rank in LoRA naively), considering the low- rank nature of both pre-trained and fine-tuned parameters.
certain patterns, forming topologically invariant but shifted "new networks". This helps preserve energy and time costs for developing new synapses and enables rapid learning (see Figure 2, top left). Here, we considered another important BNN characteristic closely related to the formation of NE the Long-Ter Potentiationn/eeprssiin (LTP/DD (Malenka & Bear, 2004) phenomenon. These phenomena extend Hebb's 's rule (Hebb, 1949), stating that frequent strengthen- ing/weakening of preceding related neurons directly affects subsequent neuronal development, showing a strong positive correlation (see Figure 2, bottom left). This occurs because the brain tends to simplify signal transmission mechanisms for efficiency and energy conservation. In PEFT settings, we believe this efficiency mechanism should be borrowed. Our approach comprises two parts: (1) we linearly decompose the shift between original and current parameter outputs (i.e. feature adjusting vectors) at the preceding layers, repre- senting the strengthening/weakkeiin
of anterior neurons in BNN. (2) we reapply this decomposed sealing vector to pa- rameters in subsequent layers, pre-scaling these pre-trained parameters (see Figure 1). By explicitly propagating these scaling vectors, similar phenomena observed in BNNs can also be identified in ANNs. In the right panel of Figure 2, we demonstrated that applying a simple feature scaling strategy in SSF for PEFT is feasible for certain tasks. However, with more pronounced domain gap, these one-dimensi seal- ing vectors tend to stretch further and struggle to shift the original parameter space effectively. Our method, Synapse and Neuron (SAN) addresses this issue by alleviating the burden on subsequent trainable parameters. This is achieved through the decomposition and propagation of scaling vec-
This aligns with the Neural Engram (NE) (Tonegawa al., 2015) phenomenon observed in Biological Neural Networks (BNNs), where the brain processes new knowledge by strengthening//eaakening existing connections (synapses) in
2

---

SAN-PEFT
tors, which reduces the complexity of remodelling shift quantities and mitigates learning difficulties with limited trainable parameters. We would further analyze those prop- erties in Section 3.2
decomposes low-rank adjustments into magnitude and di- rection components, revealing the nuanced dynamics of parameter shifts in fine-tuning. These methods demonstrate that strategically optimizing parameter updates, rather than naively increasing model complexity, leads to substantial improvements in transfer learning.
To summarize, our contributions can be outlined as follows:
We investigated the low-rank characteristics of adjust- ment values during PEFT in ANNs and discovered a strong analogy to well-documented phenomena in BNNs, such as Neural Engram (NE) and Long-Term Potentiation/DDeprssiio (LTP/D). Our analysis of the Engram pattern in ANNs revealed significant intra-task similarities but marked inter-task differences, suggest- ing the potential for sharing cross-layer adjustment vectors during PEFT.
Neural Engram and Long-Term Depresion/Potentiation In Biological Neural Networks (BNNs), the Neural Engram refers the physical and functional changes in the brain associated with the encoding, storage, and retrieval of mem- ories (Guskjolen & Cembrowski, 2023). These patterns are hypothesized to manifest as alterations in synaptic con- nections and activity across networks of neurons, forming topologically invariant but functionally adaptive "new net- works" (Tonegawa et al., 2015). This process facilitates the efficient acquisition and integration of new information while preserving existing knowledge. The concept of the Neural Engram is particularly relevant to transfer learning, as it parallels the optimization observed in pre-trained Arti- ficial Neural Networks (ANNs). A key mechanism underly- ing the Neural Engram is Long-Tern Potentiation (LTP) and Long-Teme Term Depression (LTD) (Malenka Bear, 2004; Bear et al., 2007), which describe the strengthening and weaken- ing of synaptic connections, respectively. These processes align with Hebbian principles (Hebb, 1949), often summa- rized as "neurons that fire together, wire together." LTP/D mechanisms enable efficient signal transmission by modu- lating synaptic weights based on activity patterns, thereby conserving energy while optimizing learning processes.
We developed Plg--andPPlay sharing method named SAN. This approach integrates individual tuning com- ponents from different layers into a cohesive system, thereby enhancing the performance of existing PEFT methods without incurring additional trainable costs.
Extensive experimental results demonstrate that our methods are glug-and-playable to various PEFT meth- ods on diverse scalable pre-trained models across dif- ferent tasks, including mainstream vision, language, and multi-modality benchmarks.
2. Related works
Parameter-Efficient Tine-Tuning Parameter- Efficient Fine-Tuning (PEFT) has emerged as a compelling alter- native Full Fine-Tuning (FFT), enabling the adaptation of large pre-trained models downstream tasks with mini- mal additional trainable parameters. Early approaches, such as Linear Probing (LP) (Alain & Bengio, 2016), proposed adding a simple, trainable classification head to pre-trained models. While effective, this method often suffered from limited expressiveness. Subsequent advancements intro- duced sparse tuming techniques such as BitFit (Zaken al., 2021), which focused on tuning only the bias terms of model. Similarly, prompt tuning (Liu et al., 202 Jia et al., 2022) adjusted model inputs rather than internal parame- ters. These techniques leveraged the pre-trained model's latent knowledge while significantly reducing computational costs. One of the most notable PEFT advancements is the Low-Rank Adapter (LoRA) (Hu et al., 2022), which uses low-rank matrix approximations to fine-tune large models with minimal parameter overhead. LoRA's success has in- spired further innovations, including quantized variations like Q-LoRA (Dettmers et al.. 2024), and methods such as Tuning (SSF) (Lian et al., 2022), which highlight the low-rank (approximately linear) properties of the adjustment space. DoRA (Liu et al., 2024b) further
3. Methods
In this section, we present SAN, encompassing several pre- liminary methods and their corresponding adjustments.
3.1. Preliminaries
LoRA & Adapter: These PEFT methods use two low- rank learnable matrices (Wdowwn E Rdxr(rr << d)) and W E xx) to simulate the full-rank dense layers.
y y of aydown=x))?
(I)
where x, y. and represent the inputs, outputs and linear/non-lineal function, respectively.
Scale & Shift Features (SSF): This PEFT method applies learnable linear transformation to each layer's is output y'
7 y + 8 E Raxd, where y, B E Rd are the scaling and shifting vectors, respectively, and @ is the element-wise product. The reparameterize formula is:
Scale and Shift Fine-Tuning
W O W
(2)
where W is the original weight of this layer.
3

---

-EET
DoRA: DoRA decomposes the pre-trained weight W into
Explicit Propagation: The key innovation of our SAN method lies in explicitly propagating the scaling vectors of the current layer to the parameters of the subsequent layer. This approach is motivated by fundamental insight into the nature of transformations in neural networks: any trans- formation applied to current features for PEFT implicitly affects the subsequent layer's parameters.
magnitude and direction components for fine-tuning:
W
W
m M
W
le
(3)
where m Rl*k x is the magnitude vector, V E Raxk is the directional matrix, with || lle being the vector-wise norm across each column.
To elaborate, consider a linear transformation T applied to the features yı of layer 1, y T(y)). The output of the subsequent layer 1 + 1 with weight W141 and bias b can be expressed as, Y1+1 = Ww+12(y) + b This is equivalent to:
During fine-tuning, DoRA keeps W frozen and makes m and V (using LoRA) trainable. After tuning, LoRA-style reparamcterization is adopted:
Y1+1 = T(W ((W411y Jy: + br4 = W4+1 + b
(9)
V+AV I + AVI
W =
(4)
where W is the adjusted weight matrix.
This equivalence reveals that the transformation of features in the layer / can be equalized as an adjustment to the layer's weights + 1, assuming no non-linear activations are ap- plied between these operations. Methods that apply linear transformations to features implicitly learn an adjustment matrix for the subsequent layer's weights.
where AV is the directional update multiplying two low- rank matrices Wdown E Rddrr and Wup E Reok with rank r < min(d, k))
3.2. Synapse and Neuron (SAN)
While this principle is straightforward in purely linear sce- narios, real-world neural networks incorporate non-linear activations and normalization layers. However, we argue that our approach remains approximately valid even in these more complex settings. This is based on two observations:
Basic Formula: Our SAN pipeline is depicted in Figure 1. Considering the simplest combination to SSF, the scaled features y/ of layer I can be described as y/ = 7 3 Yt. where "/ is the scaling vector (we initialize it to one) and yı is the original output features.
When combined to LoRA layers, we first compute the output following standard LoRA:
1. Near-linear behaviour of modern activation func- tions: Popular activation functions like ReLU and its variants exhibit approximately linear behaviour in their active regions. This near- cear-iine helps preserve seal- ing relationships, making it unlikely for scaling effects to be arbitrarily reversed.
y yi + [W (Widown x TT
(5)
where [W (W x )² represents the low-rank up- date. Then, we obtain the scaling vector by computing the element-wise ratio between y/ and yı:
2. Optimization stability: Modern optimization methods tend to avoid unstable paths that first reverse and then restore scaling effects. Given a fixed optimal output target, it would be inefficient and potentially destabiliz- ing for the optimization process to first counter-adjust the model parameters and then readjust them back.
n- Pool(y yy)
(6)
where @ denotes element-wiss wise division and Pool(-) is a dimension reduction operation. This scaling vector then explicitly propagated to scale the pre-trained weights of subsequent layers:
Therefore, rather than letting these scaling effects propa- gate implicitly, SAN makes this propagation explicitly, this explicit propagation allows for more efficient parameter adaptation and provides better optimization stability.
W441 = 7 C W441
The output goes through operations such as activation func- tion normalization, denoted o-)). The output for the next layer becomes:
Expressiveness & Self-regulation: SAN also achieve more fine-grained adjustment by explicit propagation. Con- sider the reparameterization introduced in Equation. 2, SSF implies a strong assumption: for each row of the current layer's weight matrix, the scaling & shifting vectors would be the same (similar to LoRA with r=1). However, explicit
Vim1 -1 7- (Win + bl+1)
(8)
Here 1 is the bias of the next layer.
4

---

SAN-PEFT
Train
Froze
Reparameterization
Rearranging this equation, we get:
SSF
8/41 (74+1 T G W4+1)x41 N41 O b(+1 (12)
Features
(less expressiveness)
The presence of (7x)² this formulation reveals crucial property: the effect of the scaling vectors is essentially squared when propagated through layers. This quadratic influence acts soft constraint the magnitude of n, discouraging extreme values and promoting stability. To formalize this regularization effect, we can express it as an implicit regularization term R(y)) added to the loss function:
WL
MZ
Weights
Mjuut layer)
Fine-tuned Weights
R(y) = A Σ In 11?
(13)
pposterior
where À is hyperparameter controlling the strength of regu- larization, this regularization term penalizes large deviations "i from its initial value of I, effectively limiting the model's capacity to make extreme adaptations.
|>>
(more expressiveness)
Adjust
(anterior
4. Experiments
SAN
4.1. Experiment Setups
Figure SAN's explicit propagation mechanism. Unlike tradi- tional PEFT methods (e.g., SSF) that only model the shifting of the current layer (top), SAN leverages propagation effectively adapt pre-trained parameters across layers (bottom) without introducing additional trainable parameters. The scaling vectors (?) learned in the anterior layer explicitly propagated influence the pre- trained weights of posterior layers, enabling more comprehensive parameter adaptation while maintaining parameter efficiency. Gray blocks represent pre- trained frozen weights, Color blocks indicate fine-tur fine-tuned trainable weights, and arrows show the adjusting direc- tion (i.c. reparameterization direction).
This section outlines our experimental framework, including the datasets, the backbone architectures employed, and the baseline methods compared against. For more details, please refer to the Appendix A.
Datasets:
Vision Tasks: We evaluate on major categories Fine-Grained Visual Classification (FGVC): 5 spe- cialized tasks using CUB-Birds (Van Horn et al., 2015), NA-Birds (Wah et al.). Oxford Flowers (Nilsback & Zisserman, 2008a), Stanford Dogs (Khosla et al., 2011), and Stanford Cars (Gebru et al., 2017). Visual Task Adaptation Benchmark (VTAB-1k) (Zhai et al., 2019): 19 diverse visual classification tasks across Nat- ural (7 datasets), Specialized (4 datasets), and Struc- tured (8 datasets) categories. VTAB- 1k emphasizes challenging data efficiency, resulting in each dataset with a limited 1000 images for training and full test sets. General Image Classification (GIC); Full train- ing and testing sets CIFAR- (Krizhevsky et al., 2009) and ImageNet- 1k (Deng et al., 2009).
propagation allows us to achieve unique adjustment value for every parameter without incurring any extra training bur- den, in such scenario. (see Figure 3) By propagating the scaling vectors from the current layer to the poste- rior layer's weight, we can overcome the strong assumption of SSF and achieve a more fine-grained adjustment. The reparameterization formula of SAN can be expressed as:
W 1 @ (cr ④ W
(10)
where 7 is the scaling vectors of the current layer, 744 the scaling vectors for the next layer.
Besides, the explicit propagation of scaling vectors in SAN introduces an implicit regularization effect to prevent over- fitting. This regularization emerges from the approximate quadratic nature of the scaling vector's influence when prop- agated through layers. To illustrate this, let's consider simplified two-layer linear network scenario without any activation and normalization:
Language Tasks:
Focus on Commonsense Reasoning 170k (CSR-170K) (Talmor et al., 2019) with datasets: BoolQ, PIQA. SIQA, WinoGrande, HellaSwag, ARC- Easy/Challenge, and OpenBookQA (QBQA).
Visual-Language Tasks: Evaluation on Mixed Vi- sual Instruction Tuning 665k (MVIT-665K) (Liu et al., 2023a) with datasets: VQA-v2, GQA, VisWiz, Sci- enceQA, TextVQA, POPE, and MMBench.
Mi4 = 1 @ ((n & W4 @ x41) + b4z) (11)
5

---

SAN-PEFT
Architecture: For vision tasks, we use Vision Trans- formers (ViT) (Dosovitskiy et al., 2020), Swin Transformers (SwinT) (Liu et al., 2021b), and ConvNeXt (Liu et al., 2022). For language tasks, we focus on Large Language Model Meta AT (LLaMA) (Touvron et al., 2023a) family, including models from all LLaMA generations from 1-3 (LLaMA- 7B, LLLMA-113, LLaMA2-7B, and LLaMA3-8B). For visual-language tasks, Large Language and Vision Assis- tants (LLaVAs) (Liu et al., 2023a) was adopted. We use LLaVAI.5-7B (Liu et al., 2023a) and LLa (Liu et al., 2024a), to be specific.
tasks like FGVC. Our results in Figure 4 demonstrate the ro- bustness of SAN. When FFT surpasses SSF using newer backbones, SAN compensates for the expressiveness lim- itations of SSF and outperforms FFT.
93.00%
92.40%
88.50%
85.0%
Baselines: Our baselines can be divided into 3 types Ba- sies: Full Fine-Tuning (FFT), Linear Probing (LP) (Alain & Bengio, 2016), and ChatGPT with Chain of Thought (Wei et al., 2022) (in language tasks). These methods are essen- tial to reflect the absolute performance level using other PEFT methods. Prompt Tuning: P-Tuning v2 (Liu et al., 2021a), Visual Prompt Tuning (VPT) (Jia et al., 2022), Neural Prompt Search (NOAH) (Zhang et al., 2024a), and Scale and Shift Features (SSF) (Lian al., 2022). These methods derive from prompt tuning, emphasizing adjusting the inputs/features to achieve PEFT. Model Tuning: Bias Fine-Tuming (Bitfit) (Zaken et al., 2021), Low-Rank Adap- tation (LoRA) (Hu et al., 2022), Sensitivity- LoRA (SPT-LORA) (He et al., 2023), Weight Decomposee LoRA (DoRA) (Liu et al., 2024d), Adapter (Series/Parallel) (Zhang et al., 2020), and Adaptformer (Chen et al., 2022). These methods aim to approximate the parameter-shifting pro- cess of FFT by using limited trainable parameters, includ- ing sparse tuning techniques (e.g.. Bitfit), Adapters (e.g.. LoRA), and hybrid methods (e.g., SPT-LoRA).
80000
SAN+SSF
93.30% 92.80%
88.00%
760099
91.60%
79.30%
87.90%
91.50%
90.70%
92.50%
VIT
Swin-T
ConvNeXt
Figure 4. Performance comparison on FGVC with different backbones. Results show accuracy (%) for SAN and various bascline methods across diffcrent vision backbones
4.3. Language Task Results
Compared to the foundation model for pure vision tasks in Section 4.2, the importance and necessity of PEFT in large language models (LLMs) are more pronounced. Table 2 compares our proposed SAN method, integrated with LoRA and DoRA, against other PEFT approaches using LLaMA (1-3) as the backbone. The results for LLaMA-7B and 13B indicate that while prompt tuning (c.g.. P-Tuning v2) is highly efficient, they exhibit limitations in terms of perfor- mance. However, increasing the trainable parameters, such as in series and parallel adapters, does not yield significant improvements. Therefore, recent advancements focus on optimizing the utilization of limited trainable parameters (e.g., DoRA derived from LoRA).
4.2. Vision Task Results
Table compares our proposed SAN method, integrated with SSF, against other otate-of-the-art fine-tuning ap- proaches using Vision Transformer (ViT-B) as the back- bone. The results highlight the effectiveness and efficiency of SAN across a wide range of tasks and datasets. As a plug-and-play method, SAN maintains the same parameter efficiency as SSF while delivering significant improvements. On FGVC, where SSF is already considered the modern state-of-the-art (SOTA) method, SAN further enhances SSF by 0.9%. On VTAB-1K, which has limited training set size and diversity, SSF begins to show its limitations as a trade-off for efficiency. However, SAN addresses this issue and achieves an overall improvement of 1.6%. On GIC, where the training set size and diversity are sufficient, the performance gap between baselines is narrow; nevertheless, the benefits of adopting SAN remain evident.
Our approach, SAN, further explores this trend. SAN demonstrates plug-and-play capability not only for LoRA but also for prompt tuning methods like SSE It can even be integrated with DoRA itself. When com- bined with LoRA, our method outperforms DoRA, achiev- ing accuracy gains ranging from 2.4% (LLaMA2-7B) to 4.6% (LLaMA3-8B). When integrated with DoRA, we still observe improvements up to 0.8%, underscoring SAN's flexibility and broader potential.
We further validate SAN on more advanced backbones, such as Swin Transformers and ConvNeXt, which inherit the win- dow mechanism and are better suited for fine- grained vision
6

---

SAN-PEFT
Table Performance comparison using ImageNet-22k pre-trained VIT-B, Results show accuracy (%) for SAN and various baseline methods across different vision benchmarks. We colour-coded the results red (Ist) and blue (2nd) and the column colour reflects the baseline type. We also demonstrated relative improvement compared with SSF.
Basic
Prompt Tuning
Model Tuning
Plug & Play
Baselines
PFT
LP
VPT-S
VPT-D
NOAH
SSF
Bittit
Adapter-8
Adapter- 16
Adaptiormer
LoRA-8
LoRA-16
SPT-LORA
SAN+SSF
Fine-Grained Visual Clansiftcation (PGVC)
Mean Param.55 % L.
100.00
0.12
0.31
0.98
0.61
0.45
0.13
0.55
0.90
0.44
0.55
0.90
0.60
0.45 91.6 (+0.9)
Mean Acc.% 1
88.5
79.3
84.6
89.1
85.2
90.7
88.4
85.5
85.5
85.1
86.0
84.8
90.1
Visual Task Adaptation Benchmark (VTAB-IK)
Param.5 4
100.00
1.14
0.52
0.13
0.23
0.23
0.63
72.0
75.5
76.1
65.2
73.9
74.0
74.9
74.9
76.4
(+1.6)
Natural
Mcan Acc. $
75.9
68.9
76.8
78.5
80.2
81.6
73.3
79.0
79.6
80.6
79.5
79.8
81.9
83.2 (+1.6)
Specialized
Mean Acc. %
83.4
77.2
79.7
82.4
84.9
86.6
78.3
84.0
84.0
85.4
84.6
85.0
85.9
88.6 (+2.0)
Structure
Mcan Acc.
1
47.6
26.8
47.0
55.0
61.3
59.9
44.1
58.5
58.3
58.5
60.5
60.2
61.4
61.3 (+1.4)
General Image Classification (GIC)
1.18
CIFAR100
93.8
88.7
90.4
93.2
93.0
94.0
93.4
93.1
93.3
93.3
93.8
93.5
93.7
94.2 (+0.2)
ImageNet-1k
83.6
82.0
82.1
82,5
82.6
83.1
82.7
83.3
82.7
83.5
82.6
82.4
83.7
83.8 (+0.7)
4.4. Visual-Language Task Results
vantages in few scenarios, it lacks robustness compared logically determined propagation. We hypothesize that this arises from the divergence of scaling components between layers far apart in the network.
After evaluating pure vision and language tasks, be- came particularly interested in multi-modal tasks due to their closer resemblance to how the human brain acquires new knowledge. Over the past year, LLaVA models have demonstrated the effectiveness of instruction tuning for visual-language tasks without relying on complex connector designs (e.g.., Q-former). Consequently, we chose LLaVA models as backbone for testing.
To random layers
LLuMAS-88
LL
Table 3 compares our proposed SAN method, integrated with LoRA and DoRA, against other tuning methods. It is worth noting that the official I.LaVA papers, FFT was used for transfer learning, and meticulous hyperparameter searches were conducted by the developers. However, we found that PEFT can perform comparably or even better than FFT in this scenario. Specifically, compared to FFT on the 7B model, SAN improved performance by 1.5% and 2.4% when integrated with I LoRA and DoRA, respec- tively. Surprisingly, these scores are even higher than those achieved using a larger backbone (i.e., 1-1BBI.5-13B) with FFT (67.8%) on average).
LLaMA-78
To close layers (ours)
LLaMAS-88
LLaM-170
LLaVA1S-7B
Relative Gain over LoRA
4.5. Ablation Study: Is Topological Reasonable Propagation Necessary?
Figure S. Unstable performance when ignoring propagation order. Blue and red bars represent positive and negative effects. respectively. Colour saturation indicates the relative gain intensity.
In Figure we present the relative improvement achieved by incorporating SAN into LoRA with propagation logi- cally determined posterior layers (e.g., layer-wise applica- tion to related layer) compared to random posterior layers. The results indicate that while random propagation offers ad-
In our view, closely related layers are typically locational proximity(c.gg within the same block, next layer) or func- tional proximity (e.g., within the next block, same layer).

---

SAN-PEFT
Table Performance comparison on Commonsense Reasoning using LLaMA Models. Results show accuracy (%) for SAN and various baseline methods across CRS language benchmarks. We codeur-coded the results red (Ist) and blue (2nd) and the row colour reflects baseline type (same Table 1). demonstrated relative improvement compared with LoRA and DoRA.
Datasets
Mean Param,
BoalQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC
ARC
QBQA
Mean Acc.Se
CharGPT (CoT)
73.1
68.5
78.5
66.1
89.8
79.9
74.8
77.0
LLaMA-7B
P-Tuning v2
0.11
64.3
76.8
73.9
42.1
72.1
72.9
$4.0
60.6
64.6
S Adaaper
1.95
63.0
19.2
76.3
67.9
15.7
74.5
57.1
72.4
70.8
P-Adapter
3.54
67.9
76.4
78.8
69.8
78.9
73.7
57.
72.4
71.9
LoRA-32
0.83
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
DoRA-32
0.84
69.4
82.4
78 6
85.3
81.0
81.9
66.2
79.2
78.0
SAN+LoRA
70.1
(+3.2)
SAN+DoRA
71.6
84,9
81.0
81.8
78.8 (+0.8)
LLaMA-13B
P-Tuning
0.03
75.4
72.1
55.2
68.6
79.5
68.0
68.4
S-Adapter
1.59
71.8
83.0
79
88.
82.4
82.5
67.3
81.8
79.5
P-Adapter
2.89
72.5
84.9
79.8
92.1
84.7
84.2
71.2
82.4
$1.4
LoRA-32
0.67
72.1
83.5
80.5
90.5
83.7
82.8
68.3
82.4
80.5
DoRA-32
0.68
72.4
94.9
81.5
92.4
84.2
84.2
69.6
82.8
81.5
SAN+LORA
0.67
71.9
84.8
80.0
91.7
84.5
84.8
72.1
83.8
81.7 (+1.2)
SAN+DoRA
0.68
72.8
92.6
84.2
83.8
86.0
82.0 (+0.5)
LLaMA2-7B
LoRA-32
0.83
69.8
83.6
82.6
79.8
81.0
77.6
DoRA-32
71.8
83.7
76.0
82.6
83.7
82.4
79.7
SAN+LoRA
0.83
72.8
79.6
89.8
81.9
83.2
70.0
80.4
80.0 (+2
SAN+DoRA
0.84
70.4
82.8
79.6
89.8
83.3
83.5
72.2
79.4
80..2 (+0.5)
LLaMA3-8B
LoRA-32
0.83
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
DoRA-32
0.84
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
SAN-LORA
0.83
75.0
88.5
80.0
95.5
87.5
90.3
79.6
86.4
65-466+
SAN+DoRA
0.84
75.0
89.0
81.2
95.4
86.4
90.2
80.2
86.4
85.5 (+0.3)
Table Performance comparison on Visual Instruction using the LLaVA Model. Results show accuracy (%) for SAN and various baseline methods the MVIT visual-language benchmarks. We colour-coded the results red (Ist) and blue (2nd) and the colour reflects the baseline type (same as in Table 1). We also demonstrated relative improvement compared with LoRA and DoRA.
Datasets
Mean Param
VQA-v2
GQA
VisWiz
SQA
TVQA
POPE
MMBench
Mean Acc. %
LLaVAI.5-13B (FFT)
100
80.0
63.3
56.7
71.6
48.7
85.9
68.7
67.8
LLaVAL.5-BB
FFT
78.5
58.2
64.3
66.5
LoRA-128
4.61
79.1
62.9
47.8
68.4
58.2
86.4
66.1
67.0
DoRA-128
4.63
78.6
62.9
52.2
69.9
57.0
87.2
66.1
67.7
SAN+LORA
4.61
79.2
63.4
51.9
69.8
58.3
66.6
68.0
SAN+DoRA
4.63
79.2
64.4
54.5
71.1
58.3
88.0
67.4
68.9 (+1.2)
This aligns with how LTP/D forms in BNNs, where unre- lated (neither locational nor functional) neurons are less likely to influence each other (Bailey et al., 2000). Please refer to Section C for more details about locational and functional proximity.
of subsequent layers. We have observed the self-regulation and plug-and-play performance-enhaniin capabilities of SAN. We hypothesize that these effects are a consequence of explicit propagation. Extensive experiments validate our method, and we believe future research should focus discovering more NE mechanisms in scalable A ANNs, particularly those related to transfer, continual, and mul- titask learning.
5. Conclusions and Future Works
Our approach Synapse and Neuron (SAN) is inspired by the Long-Term Dotentiation/DDpprssiion (LTP/DD mechanism observed in Biological Neural Networks (BNNs). BNNs, LTP/D facilitates the formation of Neural Engrams (NE). In Artificial Neural Networks (ANNs), introduce the concept of decomposing feature adjustment values and prop- agating the scaling components forward to the parameters
Acknowledgment
Gaole Dai was supported by the National Natural Science Foundation of China (Grant No. W2442028), Shanghang Zhang supported by the National Science and Technol- ogy Major Project (Grant No. 0022DD0117800)
8

---

LAN-PEFT
Impact Statement
Eva: An embodied world model for future video anticipa- tion, 2024. https::: /ttppps:///avviv///. org/ abs/ 2410. 15461.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Cimpoi, M., Maji, S., and Vedaldi, A. Deep filter banks for texture recognition, description, and segmentation. https://arxiie wttp:/ii.orr//b//////000 5456, 6, 2015.
References
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 1904, 2019.
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprinl arXiv:1610.01644, 2016.
Bailey, C. H., Giustetto, M., Huang, Y.-Y., Hawkins, R. D., and Kandel, E. R. Heterosynaptic plasticity in the hip- pocampus. Nature Reviews Neuroscience, 1(1):11-20,, 2000.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 0rx3v:1803.0555vv1 2018.
Bear, M. F., Connors, B. W. and Paradiso, M. A. Neu- roscience: Exploring the brain. Lippincott Williams & Wilkins, 2007.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 288.255.222 Icce. 2009.
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Zambaldi, M., Sadik, S., Runesson, A., Freitas, N., Lanctot, M., Degris, T., et al. Deepmind lab. https://arxis
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
org/abs/ 21..671....70 07420, 2016.
Biderman, D.. Ortiz, J. G., Portes, J., Paul, M., Green- gard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., and Cunningham. J. P. Lora learns less and forgets less. ArXiv, 0bs/2405.09663, 2024. URL. https: http://ppppaaaeeee .emantt.emnt...mmm isemanticscholass cscholar. htg/CorrpuSI::225990
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1102000...1199,....11 2020.
Bigham, J. P., Jayant, C., H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 333 342, 2010.
Fei-Fei, L., Fergus, R., and Perona, P. A quanti- tative evaluation of single-layer visual object recog- nition. http: vision..iiiii.....o caltech. edu/ Image_Dataset wwwpaaaattteetaaeettoooo. 2004.
Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., and Fei-Fei, Fine-grained car detection for visual census estimation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fonn Fourth AAAI Conference on Artificial Intelligence, 2020.
Geiger, A., Lenz, P., and Urtasun, R. Are ready for autonomous driving? the kitti vision benchmark suite, wwwpiiioorgg/bba wws.//a///////eeeee//e/eeeeeeaee http//as///24.. 1204 1.22443360 3632. 2012.
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 1116666666755 2022.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Cheng, G., Han, J., Lu, X., Shi, X., Liao, X., Li, H., Wang, J., Peng, Y., Wang, R., Li, B., et al. Remote sensing image scene classification: Benchmark and state of the art. https: vawivv.vvvv..////v///iv////ivvvv org/ abs// 710. 0. 04 049 2017.
Guskjolen, A. and Cembrowski, M. S. Engram neurons: En- coding, consolidation, retrieval, and forgetting of memory. Molecular psychiatry, 28(8):3207-3219 2023.
Chi, X., Zhang, H., Fan, C...., Qi, X., Zhang, R., Chen, A., min Chan, C., Xue, W., Luo, W., Zhang, S., and Guo, Y.

---

LAN-PEFT
He, X. al. Sensitivity-aware visual parameter-effiient
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
fine-tuning. ICCV, (30), 2023. URL https:
plopenaccess.theevve htt.:ttecccccccccooooocoeccoc htmm//onent///////
Visual waaaalaaaatteerrafaiiii Fine-Tuning ptt.:o23ppppppppppp
LeCun, Y., Huang, F. J., and Bottou, L. Learning methods for generic object recognition with invariance to pose and lighting. https: //t::///cc// nyu. www.nnu..uuuuuunne edu/-ylclab> www.wvl..v00 2004.
Hebb, O. The organization of behavior: A neuropsycho- logical theory. Wiley, 1949.
Helber, P., Bischke, B., and Dengel, A. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
LeCun, Y., Huang, F. J., and Bottou, L. Learning methods for generic object recognition with invariance to pose and lighting. https: cs nyu, www.:nnuuuuuuu... data aaraaanoobvvveee 0- small/, 2004b.
abs/1904 02191. 2019.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /wppnreveeeeeeeeeeeeeeeeee net/ forum?
Hu, et al. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2201.00000 (34), 2022. URL https:://// http///axvvvvee www.t.ogv...oo..ooooooooooooer httpg//bs/220000 00000.
Hudson, and Manning, C. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
/ id=xozJwOx2xF
Lian, D., Zhou, D., Feng, J., and Wang. ing your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Sys- tems, 35:109-123, 2022.
X. Scaling & shift-
Jia, M., Tang, L.., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A., van Ginneken, B., and Sánchez, C. I. A survey on deep learning in medical image analysis. https://arxiv: org/ abs /17071 09900, 2017.
Johnson, J., Hariharan, B., van der Maaten, I.,, Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional visual understanding. https: /ttp//aavvvvv..///////eee/ea.ava org/ wtgg/bs////eeeeeoooe/ 1112.. 6890. 2017a.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning, 2023a.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zimick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional visual understanding. 06890, 2017b.
Liu, H. Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https: ww..vvl.iiii..... wwtggg.itthoobbeeo 124401130-11aaw
Kaggle. tion.
Diabetic retinopathy https: //www//wwwwww/ www.kll....oooo....o com/c/ 2015.
detec-
Liu, S.-Y.. Wang, C.-Y., Yin. H., Molchanov, P., Wang. Y-C. F., Cheng, K.-T,, and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. ArXiv, abs/2402.09353, 2024b. URL https: //api. cemmanticschol/ org/ hts/CorpusSDp 267657886.
diabetic- iebbeiic-rttt htt.-rrree.nootthh-eeeeettttetee
Khosla, A., Jayadevaprakash. N., Yao, B., and Li, F-F.
Novel dataset for fine-grained image categorization: Stan- ford dogs. In Proc. CVPR workshop on sual categorization (FGVC), volume 2:1. Citeseer, 2011.
vi-
Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., and Tang. J. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint
Krizhevsky, A. Learning multiple layers of features from tiny images. https:/ / www. cs. toronto. "kriz/cifar, htm., 2009.
2021a.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W. Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., and Lin, D. Mmbench: Is your multi modal model an all-around player? N.0222307.06281, 2023b.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 1(4)77, 2009.
10

---

SAN-PEFT
Liu, Y., Zhang, R., Yang, H., Keutzer, K., Du, Y., Du, L.,, and Zhang, S. Intuition-aware mixture-of-rank-tt experts for parameter efficient finetuning, arXiv preprint orXiv:2404.08983, 2024c.
Patterson, D., Gonzalez, J., Hölzle, U., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., and Dean, J. The carbon footprint of machine learning training will plateau, then shrink, 2022. URL https: / //arxiv.org org/ abs / 2204 05149.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2012. 12/26, 2021b.
Sakaguchi, K., Bras, L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. arXiv preprint arXiv:2201.03545, 2022.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Socialiqa: Commonsense reasoning about social interac- tions. arXiv preprint arXiv: 09728,, 2019.
Liu, Z. et al. Dora: Weight-decompoosed low-rank adapta- tion. arXiv preprini arXivv:4422.09333, (33), 2024d. URL https: www.i.org./.....a.a.........t.at wwtp://ggppppppteeee.. http://f//2/// /2///////2..////// 09353.
Sermanet, P., Ding, T., Zhao, J., Xia, F,, Dwibedi, D., Gopalakrishnan, K., Chan, C., Drlac-Aroold, G., Maddi- neni, S., Joshi, N. J., Florence, P., Han, W., Baruch, R., Lu, Y., Mirehandani, S., Xu, P., Sanketi, P., Hausman, K., Shafran, I., Ichter, B., and Cao, Y. Robovqa: Multi- modal long-horizon reasoning for robotics, 2023. URL https: / awwaaiivvvooo.... abs /2los/2/11 2211. 00899.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Malenka, R. C. and Bear, M. F. Lip and Itd: an embarrass- ment of riches. Neuron, 44(1):5-21, 2004.
Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. Lora vs full fine-tuning: An lusion of equivalence. ArXiv, abs/2410.21228, 2024. URL https://api semanticscholar. org/ CorpusID:27365444
Matthey, L., Higgins, I. Hassabis, D., and Lerch- ner, A. dsprites: Disentanglement testing sprites dataset. https: https://githan //tpt/ggttb///////// com/ deepmind/ peppritesddatasee 2017a.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Matthey, L., Higgins, I., Hassabis, D., and Lerch- Disentanglement testing sprites dataset. https: / / g thub. com deemmdeemmii nd/ dsprites-die - dattestt, 2017b.
ner, A. dsprites:
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Common- senseQA: A question answering challenge targeting com- monsense knowledge. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/11/119-110 URL https:://cclaattat http///aaaaattoooooo org/N1 9-1421.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsuper- vised feature learning. http: /ttf::///fll... stanford. edu/housenumbers// 2011.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Im- age Processing, pp. 722-729. IEEE, 2008a.
Tonegawa, S., Liu, X., Ramirez, S., and Redondo, R. Mem- ory engram cells have come of age. Neuron, 87(5)9918- 931, 2015.
Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. http: //ww:.//wwwww http://wwwwwbbb.ooooooooo robots. ac. wt.p uk//vgg/daaa/ flowers/102/: 2008b.
Touvron, H., Lachaux, T., Blache, T., Schug, M., Hay, M., Lacroix, E.,, Hambro, F., Beaumont, J.-B.. Drame, O., Kharitonov, E., et al. Llama: Open and efficient founda- tion language models. arXiv preprint 102,11130..13971, 2023a.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. https:/ www.roobos.coo ox. ac. ac.uk/ wwwuuu////////ueeee
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
wuukk//ggg/dta/ptttst// data/pets/, 2012.

---

SAN-PEFT
Bhosale, S., Bikel, D., Blecher, L.,, Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W. Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y. Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., 7arov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https::/ /ttps///axxvvoooooee abs abs/2307 09288.
Zhang, J. O., Sax, A., Zamir, A., Guibas, L., and Malik, J. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow, UK. August 23-28. 2020, Proceedings, Part III 16, pp. 698-714. Springer, 2020.
Zhang. Y., Zhou, K., and Liu, Z. Neural prompt search. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2024a.
Zhang, Z., Zhang, Q., Gao, Z., Zhang, R., Shutova, E., Zhou, S., and Zhang, S. Gradient-basee parameter selection for efficient fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28566-28555555 2024b.
Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, S. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine- fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 595-604, 2015.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd hirds-200-201 dataset. 2011. Cali- fornia Institute of Technology.
Wei, J.. Wang, X., Schuurmans,
D.,, Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
prompting
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey zoo. http: /vtt: on pr prrppriineee www.eninnnnn.ne edu/ projects/2c100 SUN/ 2010.
Xing, J., Xia, M., Zhang, Y., Chen, H., Yu, W., Liu, H., Wang, X., Wong, T.-T., and Shan, Y. Dynamicrafter: Animating open-domain images with video diffusion pri- ors, 2023. URL https: wwwaaivvvvoooo org.vv.orrraaaae abs/ 2310. 12190.
Zaken, E. B., Ravfogel, S., and Goldberg, Y. Bitfit: Simple parameter-efffiienn fine-tuning for transformer- based masked monduage-models. arXiv preprint arXiv:2106.10199 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? 2019.
arXiv preprint
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neu- mann, M., Dosovitskiy, A., et al. large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 190.04477 2019.
12

---

Appendix SAN: Hypothesizing Long-Term Synaptic Development and Neural Engram Mechanism in Scalable Model's Parameter-Effiient Fine-Tuning
A. Experiment Setups
Datasets: Our comprehensive evaluation spans across vision, language, and visual-language domains:
Vision Tasks:
We evaluate on 3 major categories:
Fine-Grained Visual Classification (FGVC): 5 specialized tasks using
CUB-Birds (Van Horn et al., 2015)
NA-Birds (Wah et al.)
Oxford Flowers (Nilsback & Zisserman, 2008a)
Stanford Dogs (Khosla et al., 2011)
Stanford Cars (Gebru et al., 2017)
FGVC focuses on fine-graines image data classification, the definition of fine-grain in FGVC includes high-
images and fine-class separation. All of the selected datasets have moderate numbers of training sets.
Visual Task Adaptation Benchmark (VTAB-IK) (Zhai et al., 2019): 19 diverse visual classification tasks across
Natural (7 datasets):
CIFAR 100 (Krizhevsky. 2009)
Caltech101 (Fei-Fei et al., 2004)
Oxford Flowers (Nilsback & Zisserman, 2008b)
Oxford Pets (Parkhi et al., 2012)
Describable Textures Data) (Cimpoi et al., 2015)
Sun397 (Xiao et al., 2010)
SVHN (Netzer et al., 2011)
Specialized (4 datasets):
EuroSAT (Helber et al., 2019)
Patch Camelyon (Litjens et al., 2017)
Diabetic Retinopathy (Kaggle, 2015)
Resise45 (Cheng et al., 2017)
Structured (8 datasets):
Clevr Counting (Clevr-Count) (Johnson et al., 2017a)
Clevr Distance Prediction (Clevr-Dist) (Johnson et al., 2017b)
Dsprites Location Prediction (Dsprites-Loc) (Matthey et al., 2017a)
Dsprites Orientation Prediction (Dsprites-Ori) (Matthey et al., 201
Smallnorb Azimuth Prediction (Smallnorb-Azi) (LeCun et al., 2004a)
Smallnorb Elevation Prediction (Smallnorb-El (LeCun et al., 2004b)
Dmlab Frames (DMLab) (Beattie al., 2016)
Kitti Distance Prediction (Kitti-Dist) (Geiger et al., 2012)
VTAB-1K 1k emphasizes challenging data efficiency, Each dataset with limited 1000 images for training and full
test sets.
General Image Classification (GIC); Full training and testing sets of
CIFAR100 (Krizhevsky et al., 2009)
ImageNet- 1k (Deng et al., 2009)
13

---

SAN-PEFT
Language Tasks: Focus Commonsense Reasoning 170k (CSR-170K) (Talmor al., 2019) with 8 datasets:
BoolQ (Clark et al., 2019)
PIQA (Bisk al., 2020)
SIQA (Sap al., 2019)
WinoGrande (Sakaguchi et al., 2021)
HellaSwag (Zellers et al., 2019)
ARC-Easy (ARC-E) (Clark et al., 2018)
ARRC-Challenge (ARC-C) (Clark et al., 2018)
-OpenBooQQA (OBQA) (Mihaylov al., 2018)
Visual-language Tasks: Evaluation on Mixed Visual Instruction Tuning 665k (MVIT-665K) (Liu et al., 2023a) with
datasets:
VQA-v2 (Goyal al., 2017)
GQA (Hudson & Manning, 2019)
VisWiz (Bigham al., 2010)
ScienceQA (SQA) (Lu et al., 2022)
TeXIVQA (TVQA) (Singh et al., 2019)
POPE (Li et al., 2023)
MMBench (Liu et al., 2023b)
Architecture::
vision tasks: we use
Vision Transformers-Base Base (ViT-B) (Dosovitskiy et al., 2020) Swin Transformers-Be (SwinT-B) (Liu et al., 2021b)
ConvNeXt-Base (Liu et al., 2022)
language tasks: we focus on Large Language Model Meta (LLaMA) (Touvron et al., 2023a) family:
-LLMM-7BB (Touvron al., 2023a)
LLaMA-13B (Touvron et al., 2023a)
-LLaMA2-7B (Touvron et al., 2023b)
LLaMA3-8B (Patterson et al., 2022)
visual-language tasks: Large Language and Vision Assistants (LLaVAs) (Liu et al., 2023a) was adopted. We use:
- LLLVAI.5-7B (Liu et al., 2023a)
LLaVAI.5-13B (Liu et al., 2023a)
Baselines:
Our baselines can be divided into 3 types:
Basics:
Full Fine-Tuning (FFT)
Linear Probing (LP) (Alain Bengio, 2016)
ChatGPT with Chain of Thought (Wei al., 2022) (in language tasks)
These methods are essential to reflect the absolute performance level using other PEFT methods.
Prompt Tuning:
P-TTuning v2 (Liu et al., 2021a)
Visual Prompt Tuning (VPT) (Jia et al., 2022)
14

---

SAN-PEFT
Neural Prompt Search (NOAH) (Zhang et al., 2024a)
Scale and Shift Features (SSF) (Lian et al., 2022)
These methods derive from the prompt tuning technique, emphasizing shifting the inputs (or features) and achieving PEFT.
Model Tuning:
Bias Fine-Tuning (Bitfit) (Zaken et al., 2021)
Low-Rank Adaptation (LoRA) (Hu et al., 2022)
Sensitivity-Awaree LoRA (SPT-LORA) (He et al., 2023)
Weight-Decomppsed LoRA (DoRA) (Liu et al., 2024d)
Series Adapter (S-Adapter) (Zhang et al., 2020)
Parallel Adapter (P-Adapter) (Zhang al., 2020)
- Adaptformerr (Chen et al., 2022)
These methods aim to approximate the parameter-shifting process of FFT by using limited trainable parameters, including sparse tuning techniques (e.g.. Bitfit), Adapters (e.g., Adapter, LoRA), and hybrid methods (e.g., SPT-LoRA).
Code-Bases: Our implementation is based on the code of SSF https: / gighhu....... com wwwgzeliannnaiie for vision tasks and DoRA https:/ //gtttttb com/ com/NV /Vlabs/ /DORA for language and visual-language tasks. We followed most of the hyperparameter setups from these two works as aalug-nnd-plas method. However, we searched hyperparameters such the learning and drop-path for some tasks (mainly for the VTAB-IK benchmark). More detail for hyperparameters can be found in Section B and our W&B logs
15

---

ANN-PETT
B. Experiment Configurations
crop_path
weight_decay
result
65
0.20
0.0070
74
0.000070
0.0g1
300065
0.0060
6-10
Comnada
0.00ss
66
G000050
40
0.05
p.80ne
p.000030
56
20
0.15
0.0030
D 0.papo2
15
-0:00
0.0025
0 0000000
52
Figure 6. W&B auto-sweep example CIFARI0O with ViT-B
We search for hyperparameters in two ways:
Manual Search: For FGVC, GIC and all the language, visual-language tasks, search for hyperparameters (mainly learning rate) manually. Notice for FGVC and GIC, each dataset is trained separately and could have different optimal hyperparameters. However, all the datasets in CRS/MVIT, are trained together and share the same hyperparameters.
W&B Auto-Sweep: For VTAB-IK, since all 19 datasets need to search hyperparameters separately, we conduct W&B Auto-Sweep to narrow down the optimal range. A sample Auto-Sweep outcome is shown as Figure
As result, we will list the core hyperparameter for each dataset in the upcoming paragraph.
B.1. Vision Tasks
Basic Informations:
Optimizer: AdamW
Scheduler: Cosine Annealing with Warm-Up
Total Epoch Number: 100
Warm-Up Epoch Number: 10
Device: NVIDIA RTX 3090 Ti x
Hyperparmeters: For more details, please refer to our W&B log, some core configurations are listed in Table
16

---

SAN-PEFT
Table 4. Dataset-Specifie hyperparameters
Dataset Name
Learning Rate
Drop Path Rate
Search Type
CUB-Birds
0.01
0.0
Manual
NA-Birds
0.0001
0.1
Manual
Oxford Flowers
0.01
0.1
Manual
Stanford Dogs
0.00025
0.0
Manual
Stanford Cars
0.01
0.0
Manual
CIFAR100
0.005
0.0
Auto
Caltech101
0.0015
0.3
Auto
Oxford Flowers
0.005
0.0
Manual
Oxford Pets
0.005
0.0
Auto
DTD
0.0025
0.0
Manual
Sun397
0.005
0.0
Auto
SVHN
0.008
0.1
Auto
EuroSAT
0.005
0.1
Auto
Patch Camelyon
0.005
0.0
Auto
Diabetic Retinopathy
0.005
0.0
Manual
Resisc45
0.001
0.1
Auto
Clevr-Count
0.002
0.3
Auto
Clevr-Dist
0.05
0.1
Auto
Dsprites-Loc
0.01
0.2
Manual
Dsprites-Ori
0.005
0.15
Auto
Smallnorb-Azi
0.015
0.05
Auto
Smallnorb-Ele
0.005
0.3
Auto
DMLab
0.005
0.0
Manual
Kitti-Dist
0.01
0.0
Manual
CIFAR 100
0.001
0.0
Manual
ImageNet-1k 1k
0.001
0.1
Manual
Table 5. Model training hyperparameters
Model Name
Base Method
Learning Rate
Batch Size
LoRA
0.0002
64
LLaMA-7B
DoRA
0.0002
64
LoRA
0.0002
32
LLaMA-13B
D
0.0002
32
LoRA
0.0002
64
LLaMA2-7B
DoRA
0.0002
64
LoRA
0.0001
64
LLaMA3-8B
DoRA
0.0002
64
LoRA
0.0002
16
LLaVAL.5-BB
DoRA
0.0001
16
17

---

SAN-PEFT
Table Performance comparison on Commonsense Reasoning using LLaMA-7B Model. Results show accuracy (%) SAN using
different setups across CRS language benchmarks.
We colour-coded the row colour to reflect the baseline type (same as in Table 1).
Datasets
Mcan Param % 1
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
QBQA
Mean Ace. %
ChaiGPT (CoT)
73.1
85.4
68.5
78.5
66.
89.8
79.9
74.8
77.0
L-RA-32
0.83
77.4
78.8
77.8
61.3
74.8
74.7
Applied Layers Q. UP, Down
0.83
63.2
75.9
79.0
82.5
81.3
80.8
64.5
79.0
75.8
Learning Rate
0.83
69.2
81.7
79.9
83.0
81.0
79.0
62.7
77.0
76.7
Applicd Type Layerwise
0.83
65.0
81.8
78.8
84.7
78.8
81.0
67.1
79.8
77.2
Applied Layers = Q. K, UP, Down Applied Type Blockwise Learning Rate 0.0002
Adopted Setups
0.83
70.1
82.1
78.6
85.3
81.1
81.5
66.3
78.6
78.0
B.2. Language & Visual-Languaag Tasks
Basic Informations:
Optimizer: AdamW
Scheduler: Cosine Annealing with Warm-Up
Total Epoch Number: (language), (visual-language)
Warm-Up Epoch Number: 0.03
Device: NVIDIA A800 x x 2
Hyperparmeters: In this section, we present our methodology for conducting manual hyperparameter searches for language and visual-language tasks, detailed in Table 6. Using LLaMA-7B layers significantly influences performance. This finding underscores the importance of adjusting more original weight regions for effective parameter-efficient fine-tuning (PEFT) in scaled-up models. In contrast, layerwise SAN and blockwise SAN exhibit minimal impact, highlighting the robustness SAN structures. comparison between these two variants of SAN will be provided in Section C.
study, observe that the number of applied
18

---

SAN-PEFT
C. Variations of SAN
Train
Troze
| Zeparamabeniaatoon
Block N
Block
Block
Block N+1
Adjust (pesterior layer)
Pre trannd
Weights
95
Adjest Interloriayer)
Aduut (onterior layer)
Adjut conterier layer
Propegation
Propagate layer location
Propagate layer function (Next block, same layer)
(Same block, next nextlayer)
Figure 7. SAN variants
Recall Basic Formula of SAN
Our SAN pipeline is depicted in Figure Considering the simplest combination SSF, the scaled features y of layer 1 can be described y ? yt, where " is the scaling vector (we initialize it one) and yı is the original output features.
When combined to LoRA layers, we first compute the output following standard LoRA:
y/ y: + W'P (Wdowwn x" )²'
(14)
where WyuPP(Woww,,a )"" represents the low-rank update. Then, we obtain the scaling vector by computing the element-wiss ratio between y and yı:
= Pool(y) yi)
(15)
where Ø denotes element-wise division and Pool(.) a dimension reduction operation.
Locational Proximity Functional Proximity: As shown in Figure propose two variants for propagating the scaling vectors: locational proximity and functional proximity. In locational proximity, as described above, the scaling vector is propagated to the subsequent layer within the same block:
W = " O W1+1
(16)
In functional proximity, instead of propagating to the next layer, we apply the scaling vector the corresponding layer in the next block:
W 7 O W445
(17)
where is the number of layers block, this design motivated by our empirical observation that corresponding layers across different blocks exhibit high functional similarity. This be attributed the inherent functional specificity of different layer types within transformer blocks (e.g., attention layers performing QKV mapping, FFN layers handling dimension up/down scaling). As an example, in Figure 8, we observe that the perplexity remains consistently low for functionally proximate layers, even when they are several layers apart. However, this trend becomes less pronounced for extremely distant layers, which can lead to instability in the random application strategy (see Section 4.5).
While functional proximity may seem less theoretically rigorous than locational proximity, can be approximately reduced to locational proximity formulation if we consider each transformer block complex non non-linear layer. Under this interpretation, reapplying scaling vectors functionally equivalent layers becomes natural extension of the locational proximity principle.
The final output for both variants goes through operations such as activation function or normalization, denoted a(-):
Yi41 = 7++1 ④ (W) (y)) + b(41)
(18)
19

---

SAN-PEFT
for locational proximity, or for functional proximity, where represents the layer bias.
Y44 *++ (W (y b(+1)
(19)
Distribution Distance Matrix (mlp.fc1)
0.014
0.021
0.020
0.024
0.017
0.018
0.023
0.013
0.035
0.019
0.000
0.009
0.007
0.009
0.014
0.010
0.012
0.016
0.023
0.030
0.034
-0.030
0.021
0.007
0.008
6.000
0.008
0.012
0.011
0.011
0.013
0.019
0.026
0.015
00.255
0.020
0.008
0.008
0.010
0.008
0.010
0.012
0.017
0.024
0.013
0.024
-0.020
0.017
0.010
0.007
0.011
0.008
0.011
0.000
0.006
0.010
0.018
0.025
0.011
00.15
0.018
0.023
0.016
0.013
0.013
0.012
0.009
0.010
0.009
0.000
0.010
0.016
0.016
0.032
0.023
0.021
0.019
0.017
0.012
0.018
0.015
0.010
0.000
0.011
0.023
0.005
0.013
0.014
0.000
0.015
0.013
0.017
0.011
0.012
0.016
0.023
0.030
6.00
3
5
8
10
11
Pigure 8. Perplexity matrix of cross-block functional proximate layers
We visualized the perplexity matrix of FFN first layers in ViT-B after using SSF to learn the CIFAR 100 dataset.
20

---

SAN-PEFT
D. Self-regulation & Inter-task Difference/Intra-tar task Similarity in Neural Engram
30
CPAR100 Adstaaiinn Seales
Ong of Hooe Wen
Figure Distribution adjusting vectors different task
We have provided detailed values of the adjusting vector illustrated in Figure 2. These values substantiate our approach to self-regulation and inter-task differeences/ntra-ttae similarities during the PEFT of ANNs, drawing parallels with Neural Engrams in BNNs.
21

---

SAN-PEFT
E. Full Results of Vision Tasks
Table 7. Performance comparison using imagenet-21k pretrained VIT-B as the backbone. Results show accuracy (%) for various fine-tuning methods across different datasets. The best results are highlighted in red (1st) and blue (2nd).
Basic
Model Tuning
Prompt Tuning VPT-S VPT-D SSF
Plug&PPe SAN+SSF
Dataset FFT LP
Bitfit LoRA
Fine-Grained Visual Classification (FGVC)
CUB-Brids
87.3
85.3
87.1
86.7
86.7
88.5
89.5
90.6
NA-Brids
82.7
75.9
84.3
78.8
78.8
84.2
85.7
86.3
Oxford Flowers
98.8
97.9
98.5
98.4
98.4
99.0
99.6
99.7
Stanford Dogs
89.4
86.2
89.8
90.7
90.7
90.2
89.6
91.1 (+0.5)
Stanford Cars
84.5
51.3
68.6
68.7
68.7
83.6
89.2
90.4 (+1.2)
Visual Task Adaptation Benchmark (VTAB-1K -k)
CIFAR 100
68.9
63.4
72.8
68.1
77.7
78.8
69.0
74.3 (45.3)
Caltech101
87.7
85.0
87.0
91.4
86.9
90.8
92.6
93.8 (+1.2)
Oxford Flowers
97.9
97.2
97.5
99.0
97.5
98.0
99.4
99.7 (40.3))
Oxford Pets
86.9
86.3
85.3
90.5
87.3
88.3
91.8
93.0
DTD
64.3
64.1
59.2
69.8
62.6
65.8
75.1
76.4
Sun397
38.8
51.0
51.4
53.1
51.2
49.6
52.9
53.4 (+0.5)
SVHN
87.4
36.6
60.0
86.4
74.5
78.1
90.2
91.8 (+1.6)
AU
95.7
87.5
91.6
95.8
92.0
96.1
95.9
97.7 (+1.8)
Patch Camelyon
78.9
78.5
78.7
85.1
78.2
81.8
87.4
Diabetic Retinopathy
73.9
74.0
69.8
74.2
72.9
68.4
75.5
78.1 (+2.6)
Resisc45
84.2
68.6
73.0
84.7
75.6
83.4
87.4
90.6
Clevr-Count
56.3
34.3
61.5
83.0
50.5
68.5
75.9
82.4
Clevr-Dist
58.6
30.6
55.6
66.9
58.6
60.0
62.3
61.4
DSprites-Loc
57.5
12.5
66.6
80.2
68.7
73.6
77.3
81.7
DSprites-Ori
46.7
20.0
40.0
46.6
36.1
47.9
54.9
55.2 (+0.3)
Smallnorb-Azi
25.7
9.6
15.7
32.2
20.2
32.9
29.5
30.3 (+0.8)
Smallnorb-Ele
29.1
19.2
25.1
41.1
34.1
37.8
37.9
40.4 (+2.5)
DMLab
41.7
33.2
32.4
50.4
40.5
46.5
53.3
54.5 (+1.2)
KITTI/distance
65.5
55.4
55.9
81.4
67.1
72.8
80.6
82.1 (+1.5)
General Image Classification (GIC)
CIFAR 100
93.8
88.7
93.4
93.8
90.4
93.2
94.0
94.2 (+0.2)
ImageNet-IN 1k
83.6
82.0
82.7
82.6
82.1
82.5
83.1
83.8 (+0.7)
Due to the page limitation of the main text, we unable include the comprehensive results for all 25 datasets in vision task. However, the complete results provided in Table 7. These results demonstrate consistent improvement in performance following the implementation of SAN. Furthermore, in certain datasets, we observed a significant increase in accuracy (c.g., +6.5% in CLEVR-Count..
22

---

SAN-PEFT
F. Rank Ablation using LoRA-like Base Method
Table Performance comparison on Commonsense Reasoning using LLaMA-7B Model. Results show accuracy (%) for SAN using different ranks CRS language benchmarks. We colour-coded the colour reflect the baseline type (same in Table 1). The hest results are highlighted in red (Ist) and blue (2nd).
Datasets
Mean Param. % +
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
QBQA
Mean Acc.% %
CharGPT (CoT)
73.1
85.4
68.5
78.5
66.1
89.8
79,9
74.8
77.0
LuRA-16
0.42
69.9
77.8
75.1
72.1
55.8
77.1
62.2
78.0
70.9
DaRA-16
0.43
70.0
82.6
79.7
83.2
80.6
80.6
65.4
77.6
77.5
SAN+LoRA-16
0.42
69.3
82.3
76.8
86.3
80.2
81.0
80.0
77.4 (+6.5)
SAN+DoRA-16
0.43
68.5
81.6
78.9
87.2
81.1
81.4
64.6
79.6
77.9 (+0.4)
LoRA-32
0.83
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
DuRA-32
0.84
69.4
82.4
78.6
85.3
81.0
81.9
66.2
79.2
78.0
SAN+LORA-32
0.83
70.1
82.1
78.6
85.3
81.1
81.5
66.3
78.6
78.0 (+3.2)
SAN+DoRA-32
0.84
71.6
82.6
79.0
84.9
82.4
81.0
66.9
81.8
78.8 (+0.8)
Although DoRA and SAN employ distinct strategies and fully compatible with each other, similar levels of robustness observed in Table 8 when using both methods. We hypothesize that this is due to the decomposition process simplifying the deviation requirements for the limited number of trainable parameters. Furthermore, SAN can be integrated into wider array of tuning methods, including those that do not resemble LoRA-like approaches, which enhances its flexibility.
23

---

SAN-PEFT
G. World Model Video Generation Task
fine-tuned Dynamicrafter (Xing et al., 2023), an open-domain image-to-video diffusion model, into a robotic manipulation-speciife model by following the configurations outlined in Embodied World Model for Future Video Prediction (EVA) (Chi al., 2024). The dataset used for this fine-tuning process was RoboVQA (Sermanet al., 2023).
The primary objective of integrating world knowledge into the video generation model is to predict plausible future actions for specific target objects. This necessitates that the model possess robust capabilities in both maintaining consistency and adhering to instructions.
As shown Figure 10, SAN+LoRA exhibited superior performance compared to LoRA alone. Specifically. LoRA frequently failed to generate any meaningful action, resulting in static videos that remained unchanged from the initial frame. In the limited instances where dynamic results were observed (e.g., the first case), LoRA generated the movement of the robot arm but failed to close the drawer, indicating insufficient acquisition of world knowledge. While FFT mitigated some of these issues, significant problems persisted, preventing strict success criteria from being met. In contrast, SAN demonstrated impressive results. For instance, it successfully closed the middle drawer while leaving the top drawer open as intended. Additionally, SAN not only picked up the green bag but also accurately predicted its cover and displayed it to the camera. Furthermore, SAN exhibited strong world knowledge in determining relative positions, making it the only model to achieve strict success by moving the sponge near the phone.
24

---

SAN-PEFT
Inputs
Outputs (World Knowledge Key Frame)
Prompt
Initial Frame
FFT
LoRA
SAN+LORA
close the middle drawer
pick up the green chip bag
move the sponge to blueberry phone
X
put the glass cup down
ABC action text
ABC target text
target action
success
failure
partial success
Figure 10. Next-frame prediction with world knowledge The outcomes of success, failure, and partial success are evaluated hased on
four criteria: (1) action completeness, (2) action correctness, (3) target accuracy, and (4) frame consistency. the generation process, the inputs consist solely of text prompt and an initial frame. Target actions are provided here for reference to enhance understanding.
25


Reviews:
[{"review_id": "G5DX6t9Xre", "reviewer": "Reviewer_WoLR", "summary": "The authors of this paper introduce a method called Synapse and Neuron (SAN), which decomposes and propagates scaling components from anterior feature adjustment vectors to posterior weight matrices. Extensive experimentation is performed by combining SAN with multiple PEFT strategies demonstrating the performance improvements achievable using SAN.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer WoLR\nThank you for your careful review and feedback. Your positive comments (such as praising our \"sound evaluation criteria\" and \"easy to follow writing\") have greatly support our paper. We have thoroughly reviewed your comments and address your key questions below:\n1.  For SAN+LoRA is the scaling factor propagated to the next layer that uses LoRA or just the immediate next layer of the model?\n    \n2.  The claim regarding LTP/LTD seems a bit stretched.\n    \n## Point-by-Point Response:\n1.  ### Regarding SAN+LoRA propagation mechanism\nIn our experiments, we followed the default LoRA placement settings used in DoRA **[1]** and SSF **[2]**:\n-   For LLaMA/LLaVA models: LoRA was added to `[\"q_proj\", \"k_proj\", \"v_proj\", \"up_proj\", \"down_proj\"]`\n    \n-   For vision models: LoRA was added to all Linear layers (except the head)\n    \nThis naturally led to scaling factor propagation to the immediate next layer. However, your question raises an interesting point: when LoRA modules are sparsely added (e.g., every N block), is it better to propagate the scaling factor to:\n-   The immediate next layer (even without LoRA)\n    \n-   The next layer that uses LoRA (potentially skipping several blocks)\n    \nDue to time constraints, we conducted a simple test on ViT-CIFAR100 using only two LoRA modules (added to the qkv_attn layers in block 0 and block 11). The results are shown below:\n| Method | Linear Probing | LoRA-16 (no propagation) |   LoRA-16 (propagation to next layer) | LoRA-16 (propagation to next LoRA) |\n|--|--|--|--|--|\n| CIFAR100 | 88.74% | 90.82% | 91.03% | 90.38% |\n\nWe found that when LoRA modules are distantly placed, long-distance propagation does not yield superior results. This aligns with our findings in Section 4.5 where long-distance propagation occurs when randomly apply. Conversely, propagating to related adjacent layers shows benefits, even when those layers don't have LoRA modules. We hypothesize that this shifts the LoRA module's learning objective from modeling only the current layer's parameter updates to also accounting for subsequent layers' updates, which enhances LoRA's learning capacity.\n\n2.  ### Connection to LTP/LTD mechanisms\nOur method's connection to Long-Term Potentiation/Depression (LTP/D) is straightforward: we explicitly introduce the propagation mechanism. Existing PEFT methods like LoRA, DoRA, and SSF (as shown in Figure 3) typically only model the current layer's updates. Even though DoRA and SSF decompose scaling factors, they don't leverage propagation.\n\nIn neuroscience, synaptic development is directly modulated by neuronal activation patterns. LTP and LTD exemplify this: high-frequency activation of presynaptic neurons induces LTP, strengthening the synaptic connection, while low-frequency stimulation induces LTD, weakening it **[3] [4]**. This aligns with Hebbian learning—the principle that “what fires together, wires together.”\n\nIn our SAN approach, the explicit propagation of scaling factors mimics this biological process. Rather than treating each layer’s update in isolation, SAN transfers the scaling factor learned from one layer to the next. This is analogous to how the potentiation (or depression) of a synapse in a neural circuit influences the subsequent neurons, ensuring that plastic changes are distributed in a coordinated manner. For example, if a layer’s output is amplified (akin to LTP), the following layer would receives this amplified signal as a prior guidance and adjusts its weights accordingly. Moreover, by propagating scaling factors, our method effectively connects trainable modules throughout the model:\n-   Locally: we shift the learning focus from modeling only single-layer updates to considering groups of related layers.\n    \n-   Globally: All trainable modules throughout the model become \"interlocked,\" simplifying the adjustment range each module needs to cover (Figure 2) and achieving better performance\n    \nThis biologically inspired mechanism, therefore, not only provides a theoretical foundation for our approach but also delivers practical benefits in model transfer learning.\n## Reference\n**[1] Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024, July). Dora: Weight-decomposed low-rank adaptation. In** _**Forty-first International Conference on**_ _**Machine Learning**_**.**\n\n**[2] Lian, D., Zhou, D., Feng, J., & Wang, X. (2022). Scaling & shifting your features: A new baseline for efficient model** **tuning****.** _**Advances in Neural Information Processing Systems**_**,** _**35**_**, 109-123.**\n\n**[3] Malenka, R. C., & Bear, M. F. (2004).** _**LTP**_ _**and**_ _**LTD** **: An embarrassment of riches**_**.** **Neuron****, 44(1), 5–21.** \n\n**[4] Tonegawa, S., Liu, X., Ramirez, S., & Redondo, R. (2015).** _**Memory engram cells have come of age**_**. Neuron, 87(5), 918–931.**", "claims_and_evidence": "The core idea of explicit propagation of scaling components to subsequent layers by decomposing adjusting vectors from preceding layers  is interesting and its advantage is supported well by the experiments. However, the claim regarding LTP/LTD seems a bit stretched.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria used for underscoring the advantage of the proposed PEFT strategy seems sound.", "theoretical_claims": "The paper provides some theoretical insights into how self-regularization is incorporated into the given framework.", "experimental_designs_or_analyses": "The authors present comprehensive experimental results that demonstrate the effectiveness of the proposed method across a wide range of vision, language, and vision-language tasks.", "relation_to_broader_scientific_literature": "This article pertains to PEFT methodologies of LLMs, which is highly relevant for the broader ML and NLP community.", "essential_references_not_discussed": "The paper discusses all major references.", "other_strengths_and_weaknesses": "Strengths: This paper proposes a simple yet novel PEFT approach by involving explicit propagation of the scaling vectors of the current layer to the parameters of the subsequent layer. Experimental evidence suggest improved performance when combined with different PEFT strategies.\n\nWeakness: The connection to LTP/LTD is unclear.", "other_comments_or_suggestions": "The paper is easy to read.", "questions_for_authors": "1) For SAN+LoRA is the scaling factor propagated to the next layer that uses LoRA or just the immediate next layer of the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YUAFJvxH05", "reviewer": "Reviewer_bAk4", "summary": "This paper proposes a fine tuning method based on ideas from the Neural Engram and LTPD literature which gives an alternative method to the popular LORA and DORA updates that have recently been employed for large models. The update consists computing scale vectors $\\gamma_\\ell$ at each hidden layer $\\ell$ of the network from the pooled ratios of activity patterns $y'/y$ of the perturbed $y'$ and unperturbed neural activations $y$.  The authors show that their fine tuning method improves full fine tuning and LORA on a number of vision and language benchmarks. Part of the key idea is to maintain stable subnetworks (engrams) in the model which do not change in topology during fine tuning but rather change in precise weight scales for existing nonzero connections. The authors show that organizing updates by correct layer order is important to reap the full benefits of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer bAk4\nThank you for your thorough review and feedback. Your positive comments (such as acknowledging our solid experiment and idea) and suggestions have greatly helped us improve our paper. To address your concerns, we have carefully read your feedback and provided point-by-point responses below:\n1. Challenging to read / parse / understand partly due to the abundance of acronyms and partly due to poor mathematical notation and some missing explanations:\n\n    a. Minor Typos and overwhelming acronyms\n\n    b. The $W_{down}$ in Equation 1 should be $\\mathbb{R}^{r\\times d}$ and $W_{up}$ should be $\\mathbb{R}^{d\\times r}$. What do $x,y$ represent? \n\n    c. $W$ in Equation 2 should be $\\mathbb{R}^{d\\times d}$ and $γ$ is a vector of size  $\\mathbb{R}^{1\\times d}$. Do the authors mean $(γ1^⊤)⊙W$, where 1 is the vector of all ones? In Equation 9, the $T(\\cdot)$ function is overloaded.\n  \n    d. Why does the update stabilize or become implicitly regularized to be near 1 in Equation 12? Under what conditions do the assumptions in line 184 expect to hold?\n\n2. Is there biological motivation for equation 6? Is it computed per example?? \n3. In the baselines, are all fine-tuning methods employing equal numbers of parameters or similar compute? \n\n## Point-by-Point Response:\n1. ### Regarding mathematical notation and explanations\n    **a:** Thank you for your meticulous review. We will correct these points in the revised version.\n\n    **b:**  In Equation 1, we denote  $r\\ll d$, where $r$ is the rank of the LoRA/Adapter. Accordingly, $W_{down}$ is defined as $\\mathbb{R}^{d\\times r}$ to compress the input $x$ from dimension $d$ to $r$, and $W_{up}$ is defined as $\\mathbb{R}^{r\\times d}$ to recover the original dimension. As noted in [Line 143-144], $x$, $y$, and $\\theta$ represent the inputs, outputs, and the linear/non-linear function of the LoRA/Adapter, respectively. \n\n    **c:** $W$ can also be shaped as $\\mathbb{R}^{d\\times k}$ (e.g., the FFN in transformers), as long as the scaling vector $γ$ of size  $\\mathbb{R}^{1\\times d}$ is available for per-row scaling.  We did not locate the expression  $(γ1^⊤)⊙W$ mentioned in your comment. We assume your concern may relate to whether $γ$ is an all-one vector. Initially, $γ$ is indeed an all-one vector, but after training, it diverges (i.e., $γ$ is trainable or generated by trainable modules). For the function $T(\\cdot)$, defined as $T(y_{l})=\\gamma_{l}\\odot y_{l}$, this represents element-wise scaling of $y_{l}$ prior to feeding it into layer $l+1$ layer. Specifically, we have: $y_{l+1}=W_{l+1}T(y_{l})+b_{l+1}=W_{l+1}\\gamma_{l}\\odot y_{l}+b_{l+1}=\\gamma_{l}\\odot W_{l+1}y_{l}+b_{l+1}=T(W_{l+1})y_{l}+b_{l+1}$, which is automatically managed by PyTorch’s broadcast mechanism.\n\n    **d:** The implicit regularization effect arises from the explicit propagation, which introduces a quadratic effect ($(\\gamma_l)^2$), which discourages extreme values and thus stabilizes the updates. This stabilization prevents overfitting by controlling the magnitude of divergence from the initial value 1. In a linear scenario, scaling the output is equivalent to scaling the weights in the next layer. Although ReLUs are not globally linear, they operate nearly linearly in their active regions (when the output is non-zero).\n\n4. ### Regarding biological motivation for Equation 6\nEquation 6 reduces both the batch and token dimensions (related to the size of data) while preserving the embedding dimension (related to the content of data), resulting in a 1D vector that can be used for scaling weights. It aligns with the principles of synaptic plasticity. Network signals are data-driven, and synaptic development adapts to these signals. For instance, high-frequency stimulation strengthens synaptic connections between neurons, and such strengthened connections maintain over a range of time, forming a specialized engram.\nFurthermore, while methods like SSF and DoRA apply scaling factors at the neuron level (using the same scaling for all parameters in each row of the weight), SAN operates at the synapse level. By introducing a propagation mechanism, we use the previous layer’s scaling as additional per-column scaling for the current layer. Resulting in more complex engram patterns, wherein each synapse within a neuron can have its scaling factor (see Figure 3). This design better aligns with recent neuroscience findings on neuronal engrams, such as the linking, buffer, and feature neuron engrams discussed by Choucry et al. (2024) **[1]**.\n\n3. ### Regarding trainable parameters of baselines\nSAN itself doesn't introduce additional trainable parameters; our experimental parameter settings follow those established in SSF, SPT-LoRA, and DoRA, which are all published papers. We have also provided the parameter ratio in all tables.\n\n## References\n**[1] Choucry, A., Nomoto, M., & Inokuchi, K. (2024). Engram mechanisms of memory linking and identity. Nature Reviews Neuroscience, 25(6), 375-392.**", "claims_and_evidence": "First, on the empirical side, this paper provides a very large set of fine tuning experiments in many modalities including vision, commonsense reasoning, and multimodal visual language tasks. I appreciate the breadth of experiments and the efforts the authors went to to benchmark their method. The fine tuning method shows promise on each of these provided experiments.\n\nHowever, there are a number of claims made about both (1) the motivation and design of the algorithm and (2) theoretical claims about how /why it works that are not clearly tested. I was hoping to see some numerical evidence that their fine tuning method preserved or encouraged formations of new non-overlapping neural engrams which was the motivation of the algorithm. However, I could not find any experimental evidence that this engram formation actually occurs in their finetuning. Second, it is unclear why the pooling and elementwise division to form $\\gamma_\\ell$ should give rise to engrams. Lastly, the claim that $\\gamma_\\ell$ should be close to unity.  The authors write \"This quadratic influence acts as a soft constraint on the magnitude of γl, discouraging extreme values and promoting stability.\" Where is this shown or argued for? Partly, I think these claims are important scientifically since the authors are claiming that this algorithm has something to do with engrams and LTP/D and that these are key insights that enabled the improvement in performance.", "methods_and_evaluation_criteria": "Yes, the benchmark datasets and evaluation criteria make sense.", "theoretical_claims": "There are some issues with mathematical notation that make it difficult to actually understand some of the definitions and mathematical claims. However, I think I understand the intended interpretation of the algorithm and have read through their theoretical results. \n\nThe authors invoke two assumptions which they do not clearly justify or reference to an appropriate article. Specifically, the allude to \"near-linear behaviour of modern activation functions\" and also claim \"Modern optimization methods tend to avoid unstable paths that first reverse and then restore scaling effects.\" What evidence do the authors have for these claims?", "experimental_designs_or_analyses": "I think the empirical design is valid provided that parameter counts are controlled across baselines (see questions below).", "relation_to_broader_scientific_literature": "The authors claim that their algorithm is inspired by neural engrams, which are an idea from computational neuroscience where combinatorial subsets of neurons support representations of new memories that do not interfere with previously acquired knowledge. If they could more clearly make this connection and show how their algorithm generates these engrams, I think this work would be not only interesting to finetuning researchers but also neuroscientists interested in memory formation and continual learning.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "This paper has interesting ideas and good experiments. However, I found it challenging to read / parse / understand partly due to the abundance of acronyms and partly due to poor mathematical notation and some missing explanations (see comments/questions below).", "other_comments_or_suggestions": "1. Abstract final sentence missing final punctuation  “.”  \n2. There are a lot of acronyms like PEFT, SSF, LORA, DORA, LP, SAN, FFT, NE, BNN, LTP. I understand this is a stylistic choice, but sometimes it is hard for the reader to keep straight. Please consider using the full name for some of the less frequently used acronyms. \nLine 92 “Further demonstrated … ” should be “The authors of [cite] further demonstrated …”\n3. The shapes of the matrices reported do not make sense. Notationally, the $W_{down}$ in equation 1 should be an element of $\\mathbb R^{r \\times d}$ and $W_{up}$ should be $\\mathbb R^{d \\times r}$ \n4. More explanation is needed around equation 1, what do x, y represent. Is there one of these for each hidden layer of the network, etc?\n5. Equation 2 shapes don’t make sense. $W$ should be $\\mathbb R^{d \\times d}$ and $\\gamma$ is a vector of size $\\mathbb R^{d}$. Do the authors mean  $(\\gamma 1^\\top ) \\odot W$ where 1 is the vector of all ones?\nIn Equation 9, the T( ) function is overloaded. Suppose that T(y) = A y for matrix A. Then the transformation is  W T(y) = W A y = (W A) y  = T_2( W)  y where T2(W) = W A. \n6. In line 184 the authors introduce two assumptions near linear behavior + optimization stability. Under what conditions do they expect these to hold? Also in what sense is ReLU near linear? For a single input, the function is locally linear but over the space of all inputs, it is a complicated function. \n7. Is equation 6 computed per-example? Does each data point have its own $\\gamma_\\ell$?\nIn Figure 3, the top of the image says “froze” but should say “frozen”\nEquation 10 should (probably, if I understand correctly) read as $W’ =  ( \\gamma_{\\ell+1} \\gamma_\\ell^\\top ) \\odot W$. \n8. The argument below eq 12 is unclear to me. Why does the update stabilize or become implicitly regularized to be near 1? Please provide a derivation or argument in the Appendix. \n9. Line 369 “Topological reasonable” -> “Topologically Reasonable”", "questions_for_authors": "1. Is there a biological motivation for equation 6? Is there any prior work that shows that this kind of plasticity rule encourages formation of engrams?\n2. In the baselines, are all finetuning methods employing equal numbers of parameters or similar compute? This is important to make comparisons across methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7USD1Br9j", "reviewer": "Reviewer_1yzx", "summary": "This work seeks to further advance Parameter-Efficient Fine-Tuning (PEFT) techniques, which reduce memory usage and computational cost compared to full fine-tuning. It draws insights from the Neural Engram (NE) phenomenon, where the brain processes new knowledge by strengthening or weakening existing connections, which helps to preserve energy and reduce the time costs of developing new synapses and enables rapid learning. To do this, the authors propose SAN, with the key innovation lying in explicitly propagating the scaling vectors of the current layer to the parameters of the subsequent layer, mimicking LTP/LTD. In a way, SAN explicitly propagates the layer transformation effect through a scaling approximation. This explicit propagation allows for more efficient parameter adaptation and provides certain implicit regularizations that discourage extreme values and promote stability.\n\n## update after rebuttal\nI maintain my assessment, a weak accept, as I see no major weaknesses left to the best of my knowledge", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 1yzx\nThank you for your thorough review and feedback. Your positive comments (such as acknowledging our clearly explained SAN and its connection to LTP/LTD) and suggestions have greatly helped us improve our paper. To address your concerns, we have carefully read your feedback and provided point-by-point responses below:\n1.  Minor typos\n\n    a.  missing period at the end of the abstract\n   \n    b.  “(see Figure 3)” should be in front of the period on page 4 under “Expressiveness & self-regulation”.\n        \n2.  Provide more discussion on the limitations of the linear transformation approximation and potential failure modes.\n    \n3.  By explicitly propagating the scaling vectors, would it introduce other implicit regularizations besides the self-regularization mentioned, such as effects associated with reducing depth?\n    \n4.  Provide a complexity analysis and more hyperparameter details.\n\n## Point-by-Point Response:\n1.  ### Regarding Typos\nThank you for your meticulous review. We will correct these in the revised version.\n\n2.  ### Regarding potential failure modes\nWe envision SAN as a plug-and-play method adaptable to current and future SOTA PEFT approaches across domains. This implies that SAN's failure mode mainly depends on the selected base method.\n\nTo stress-test pure linear transformations in large models, we evaluated SSF and SSF+SAN on LLaMA fine-tuning. The results (see Table below) indicate that for large models (7B+ parameters), pure linear transformations of features cannot effectively align pre-trained parameters to downstream tasks. This is reasonable, as although such methods achieve excellent results on vision foundation models (which typically have only hundreds of millions of parameters) by providing orthogonal fine-tuning-like **[1]** properties that preserve hypersphere energy and prevent catastrophic forgetting, their expressive power becomes constrained in larger language/multimodality models.\n| Method |  P-Tuning V2 | SSF | SSF+SAN | LoRA | LoRA+SAN |\n| ---- | ---- | ---- | ---- | ---- | ---- |\n| Common Sense Reasoning | 64.60% | 52.60% | 61.10% | 74.70% | 78.00% |\n\n3.  ### Regarding other implicit regularizations\nSelf-regularization emerges directly from the implicit $\\gamma^2$ term during adjustment, which guides scaling factors to reduce extreme values and control variance. Besides, we haven't observed significant regularization effects beyond self-regularization.\n\nTo our knowledge, the depth-reducing phenomenon you mentioned appears in two types of literature: structural pruning (particularly layer pruning) like Layer Folding **[2]**, which aim to improve model efficiency; and layer redundancy studies like stochastic depth **[3]**, which prevent overfitting through random depth dropout during training. Our method doesn't directly relate to these works.\n\n4.  ### Regarding complexity and hyperparameters\nWe have provided basic hyperparameters in the paper, and more detailed parameter settings can be found in our supplementary code. As mentioned in Appendix B, we've recorded thousands of experimental setups and results on WandB, which we will make public.\n\nFor algorithm complexity, SAN, as a plug-and-play method, adds no additional parameters during training. The computational cost only appears in the decompose & propagate operations. Specifically:\n-   The decomposition cost doesn't involve matrix multiplication (using average pooling or direct reuse), making the additional computation negligible.\n-   For propagation, we multiply the decomposed $1×d$ vector with a $d×k$ posterior weight matrix. The complexity for a single propagation is $O(d×k)$, where $d$ is the input feature dimension and $k$ is the output dimension. If we have multiple layer groups $(1,2,...N)$ that need propagation, the total computational complexity would be the sum of each group's complexity: $O(∑(d_i×k_i)$ for $i$ from $1$ to $N$.\n    \nThis minimal computational overhead makes SAN an efficient enhancement to existing PEFT methods. Practically speaking, throughout our experiment, we also did not see a significant slowdown by adding SAN.\n\n## Reference\n**[1] Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., ... & Schölkopf, B. (2023). Controlling text-to-image diffusion by orthogonal finetuning.** _**Advances in Neural Information Processing Systems**_**,** _**36**_**, 79320-79362.**\n\n**[2] Dror, A. B., Zehngut, N., Raviv, A., Artyomov, E., Vitek, R., & Jevnisek, R. (2021). Layer folding: Neural network depth reduction using activation linearization. arXiv preprint arXiv:2106.09309.**\n\n**[3] Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016). Deep networks with stochastic depth. In** _**Computer Vision** **ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part**_ _**IV**_ _**14**_ **(pp. 646-661). Springer International Publishing.**", "claims_and_evidence": "-\t[Strength] The authors performed extensive empirical experiments across well-known vision tasks, language tasks, and visual-language tasks using SoTA architectures for the task. By doing so, the authors demonstrated the effectiveness of incorporating SAN into various PEFT methods (e.g., LoRA and DoRA).\n-\t[Strength] The authors clearly explained how SAN, with the same parameter efficiency as SSF, is more expressive than SSF.\n-\t[Strength] The authors clearly articulated the connection of SAN to LTP/LTD and provided further ablation studies.\n-\t[Weakness] While the authors explained how the principle of SAN, assuming linear transformation, can still remain approximately valid in more complex settings, it would be helpful to provide more discussion on the limitations of such an approximation and potential failure modes.\n-\t[Weakness] Code and hyperparameter details are stated to be released in the future, so I cannot be certain about the reproducibility of this work. That said, the authors did provide basic information on hyperparameter tuning in Appendix B.", "methods_and_evaluation_criteria": "See above", "theoretical_claims": "The results are mainly empirical", "experimental_designs_or_analyses": "I checked all tables and figures in the main text", "relation_to_broader_scientific_literature": "The paper is well-positioned with a clear objective", "essential_references_not_discussed": "None to my knowledge", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "Some typos: missing period at the end of abstract; “(see Figure 3)” should be in front of the period on page 4 under “Expressiveness & self-regulation”.", "questions_for_authors": "-\tBy explicitly propagating the scaling vectors, would it introduce other implicit regularizations besides the one that the author mentioned, such as effects associated with reducing depth? \n-\tCan you provide a complexity analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Concerns were raised that the code and hyperparameter details were not available and will only be released in the future. However, I do not find a strong enough reason for rejection and I would be willing to give the authors the benefit of the doubt, trusting that they will release all code and hyperparameters in the camera-ready version. 
Overall, there was minimal but positive enthusiasm for this contribution.
Decision: Accept (poster)