On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Authors: Nghiem Tuong Diep
Huy Nguyen
Chau Nguyen
Minh Le
Duy Minh Ho Nguyen
Daniel Sonntag
Mathias Niepert
Nhat Ho
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model (LLM)
Theory
Instruction Tuning
Mixture of Experts
Primary area: deep_learning->other_representation_learning

Abstract: LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, despite its empirical success, the theoretical foundations of zero-initialized attention remain largely unexplored. In this paper, we provide a rigorous theoretical analysis, establishing a connection between zero-initialized attention and mixture-of-expert models. We prove that both linear and non-linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings on the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types consistently surpass vanilla attention, highlighting the robustness and adaptability of zero-initialized attention.

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Nghiem T. Diep 123 Huy Nguyen *4 Chau Nguyen 55 Minh Le Duy M. H. Nguyen 178 Daniel Sonntag
Mathias Niepert 78 Nhat Ho*
Abstract
et al., 2023), their closed- source nature and high develop- ment costs limit accessibility. To address this, several efforts have explored fine-tuning open-souree LLMs, e.g., LLaMA (Touvron et al., 2023a), on large-scale instruction datasets where small nuuaan-annotan massive training corpora using self self-instruct methods (Taori et al., 2023). This approach has enabled models like LLaMA to achieve instruction-followinn performance comparable to GPT-3.5 and, with further advancements, shows promise in approaching GPT-4's 's capabilities. However, full fine- tuning of large-scale LLMs remains computationally in- tensive, highlighting the need for more efficient adaptation techniques to unlock their full potential across diverse down- stream applications.
LLaMA-Adapter has recently emerged as an ef- ficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, de- spite its empirical success, the theoretical founda- tions of zero-initialized attention remain largely unexplored. In this paper, we provide rigor- ous theoretical analysis, establishing connection between zero-initialized attention and mixture-of- expert models. We prove that both linear and non- linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types con- sistently surpass vanilla attention, highlighting the robustness and adaptability of zero- intt-iniiialzzed at- tention. Our implementation is publicly available GitHub.
samples are expanded into
To address this challenge, researchers have increasingly turned to parameter-efficient fine-tuning (PEFT) techniques, which update only a small subset of parameters while keep- ing most of the backbone frozen, enabling efficient and scal- able adaptation (Houlsby et al., 2019; Lester et al., 2021; Hu et al., 2021). Among these approaches, the L.LaMA- Adapter (Zhang al., 2024) has emerged PEFT method for transforming LLaMA into an instruction- following model, demonstrating strong performance across multiple benchmarks. Beyond adaptation prompts, its key innovation lies in the introduction of zero-initialized atten- tion with zero gating, which facilitates the seamless inte- gration of new instructional prompts while preserving the model's pre-existing knowledge. However, despite its empir- ical success, the theoretical foundations of zero-initialized attention remain largely unexplored, limiting deeper un- derstanding of its underlying mechanisms and potential for further advancements.
specialized
1. Introduction
Large Language Models (LLMs) have revolutionized natu- ral language processing (Radford al., 2019; Raffel al., 2020; Anil et al., 2023; Minaee al., 2024), demonstrat- ing remarkable understanding and generative capabilities across various tasks. While proprietary models like Chat- GPT (OpenAI, 2025) and GPT-4 (OpenAI et al., 2024) set new records in instruction-follovin performance (Peng
In this work, we conduct a rigorous theoretical and empiri- cal investigation into pero-initialized attention, showing that is not merely an engineering trick. Building on recent find- ings (Le al., 2024; 2025) that draw connections between the attention mechanism (Vaswani, 2017) and the mixture of experts (MoE) model (Jacobs et al., 1991; Jordan & Ja- cobs, 1994; Shazeer et al., 2017), we investigate how zero- initialized attention in the LLaMA-Adapll (Zhang et al., 2024) can interpreted within this framework. Specifically, we show that zero-initialized attention can be viewed as a special formulation of the MoE model and that under lin-
'Equal contribution GGerman Research Center for Artificial Intelligence (DFKT) UUniversity of Seience, VNU-HCM, Ho Chi Minh City, Vietnam 'Viet Nam National University, Ho Chi Minh City, Vietnam *The University of Texas at Austin QQalcoom Al Research, an initiative of Qualcomm Technologies. Inc. 'Work was completed while employee at Qualcomm MMx Planck Re- search School for Intelligent Systems (IMPRS-IS) 'University of Stuttgart "Oldenburg University. Correspondence to: Huy Nguyen <huynm @ uetuxas.eddu edu>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
ear prompt settings, both the prompts and the gating factor can be optimally estimated, offering significant statistical benefits. Furthermore, we extend this analysis to scenarios in which the trainable prompts are nonlinear (e.g., imple- mented via an MI.P) and prove that the optimal estimation of the prompts and gating parameters remains achievable, thus enhancing the flexibility for future applications.
2. Related Work
Parameter Efficient Fine- Tuning for LLMs. Large foun- dational models have demonstrated remarkable generaliza- tion capabilities across wide range of tasks (Devlin, 2018; Brown et al., 2020; Touvron et al., 2023b;c). However, as model sizes grow exponentially, fine-tuning all parameters in a large-scale model becomes increasingly impractical. In contrast, PEFT techniques (Hu et al., 2021; Karimi Ma- habadi et al., 2021; Gao et al., 2024; Houlsby et al., 2019; Li et al., 2024c; Li & Liang, 2021; Zhou et al., 2022; Man- grulkar et al., 2022) have emerged as a promising strategy to adapt these models to downstream tasks while freezing most of the backbone's 's parameters. These approaches can be categorized into three directions, namely (i) low-rank decomposition like LoRA (Hu et al., 2021) and its advanced versions (Karimi Mahabadi et al., 2021; Zhang et al., 2023); (ii) adapters which insert lightweight adaptation modules into each block of the transformer and have been applied across numerous domains (Gao al., 2024; Houlsby et al., 2019 Li et al., 2024c); and (iii) prompt tuning where train- able tokens are appended to the input embeddings (Lester et al., 2021; Nguyen et al., 2024b) or at intermediate layers, i.e., pre-fixed tuning (Liu et al., 2021; Li & Liang, 2021; Shi & Lipani, 2023).
Our empirical evaluations further substantiate these theo- retical findings. Across multiple datasets in the open LLM benchmark (Beeching et al., 2024), zero-initialized attention consistently demonstrates better performance than random- initialized attention. Additionally, non-linear prompts ex- hibit improved results compared to their linear counterparts, thereby offering the potential to boost the I.LMM A-dapter'ss'ss 's capabilities. Notably, on various small rates of limited train- ing data, both linear and non-linear prompts combined with zero-initialized attention substantially outperform standard attention, reinforcing our theoretical findings and demon- strating the robustness of the proposed approach.
Our contributions are four-fold can be sum- marized as follows: (i) We develop a theoretical framework that examines the connection between zero-initialized atten- tion and MoE models; (ii) We demonstrate the statistical advantages of vero-initializa attention over conventional attention, enabling optimal estimation of prompt parameters and gating factors; (iii) We extend our analysis to accom- modate non-linear prompts, offering increased flexibility for a broader range of applications: (iv) Finally, extensive experiments on multiple question-ansve answering datasets vali- date our theoretical insights, highlighting the robustness and significance of our findings.
Contribution.
Unlike the aforementioned PEFT methods, the LLaMA- Adapter (Zhang al., 2024) is specifically designed to enhance instruction-followiii capabilities, where the model learns to generate contextually relevant responses based on natural language instructions. This is done by introducing a concept of pero-initialized attention to integrate new instruc- tional prompts while preserving the model's is existing knowl- edge. Through this mechanism, the algorithm starts with minimal impact and prevents training instability and catas- trophic forgetting by selectively activating relevant informa- tion while allowing the model to incorporate instructions incrementally, In this work, we conduct a comprehensive theoretical and empirical investigation into zero- zero-initian arr-inntialized attention, demonstrating that it is more than just an engi- neering trick and uncovering its fundamental properties and advantages.
Organization. The remainder of this paper is organized as follows: Section 2 reviews related work. In Section 3, we establish the connection between zero-initialized attention and the mixture-of-expprts model. Section 4 presents our theoretical results. In Scction 4.2, we extend the analysis for the non-linear setting and outline the main algorithm for the non-linear case, followed by empirical evaluations in Section 5. Finally, Section 6 discusses limitations and future research directions. Detailed proofs and additional experimental results are provided in the Appendix.
Mixture of Experts (MoE) in PEFT. Recent research has explored MoE in PEFT to enhance the adaptability of large pre-trained models while minimizing computational costs. MoE-base approaches, such as Switch Transformers (Fe- dus et al., 2022) leveraged sparse activation of expert net- works to achieve efficient scaling. In the context of PEFT, techniques like ArapterDrop (Riccklé et al., 2020) and LoRA (Hu et al., 2021) have been combined with MoE to dynam- ically allocate resources to task- experts, reducing the number of trainable parameters (Li et al., 2024a; Chen et al., 2024; 2023). These works demonstrate that MoF can significantly enhance parameter efficiency without compro-
Notation. For any n E N. we denote [n] as the set {1, 2, n... For any set S. we let |S| be its cardinal- ity. For any vectors u = (u1, u2,...., ua) E Rd and = (01,02, aa) E Nd, we let nxx = ui u2 ... u. |u| : -u + u2 +... + un. and a! := lala!!... add while ||u|]| denotes the l2- norm value of u. Lastly, for any two posi- tive sequences (a,) 2221 and (bm)nzI. we write an = (b,)) or On S bn if a, I Cbn for all n E N. where C C>0 is some constant. Finally, a. Op(b,) means an/b, is stochastically bounded.
2

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
mising performance, making it promising direction for fine-tuning large-scale models.
are pre-trained projection matrices. The attention scores computed between Q and K before applying the softmax function are given by:
In another line of research, the MoE framework has been leveraged in (Le et al., 2024; 2025) to investigate the conver- gence behavior of learnable prompt vectors in the context of prefix tuning method, which are attached to the key and value matrices of self-attention mechanism to learn down- stream tasks. In particular, by showing rigorously that each row of an attention head can be represented as MoE, they demonstrate theoretically and empirically that the prompt convergence will be significantly accelerated if there exists a shared structure among the prompt vectors. However, al- though the zero-imin pero-initialized attention has been widely used as an PEFT method, its theoretical understanding has remained missing in the literature. To close this gap, we provide comprehensive study on the convergence of prompt vectors within the zero-initialized attention by establishing connec- tion between this model and MoE in Section 3. Our theory indicates that linear prompts and non-linear prompts share the same convergence behavior and can be both optimally estimated. On the empirical side, we observe that the non- linear prompts in zero-initalized attention work favorably compared to linear prompts in several benchmark datasets.
S QKT = =(SP, Sx] E ERI
(4)
vdz
where Sp E RIXL and Sx E IIX(N+1) denote the atten- tion scores of adaption prompts and + word tokens, respectively. Rather than applying the softmax function di- rectly, Zhang et al. (2024) propose alternative approach. They suggest computing the softmax independently over the two components Sp and Sx, and incorporating learnable gating factor a E IR as follows:
Sg [so max(s)) tanh(a), softmax(Sx)] (5)
This approach aims to decouple the knowledge contained within the pre-trained model and the adaptation prompts, thereby preserving the pre-trained model's original knowl- edge. The activation function tanh(-) is used to regulate the scale of a within the range [-1, 1]. Consequently, the output of the aero-initialized attention mechanism can be expressed as:
(6) During training, only the prompt parameters P are opti- mized, while all other parameters of the pre-trained model remain frozen. Next, examine how zero-initialized at- tention can be interpreted through the lens of the mixture of experts framework.
S. y y= V e R
Additional discussion on related work of the theory of mix- ture of experts is in Appendix B.
3. Zero-initialized Attention meets Mixture of Experts
Zero-initialized attention. We revisit the zero-initialized attention formulation introduced by Zhang et al. (2024). Let X, = -[x(,1.... Ii.Nv] I E Nxd denote the input tokens at the 1-th layer of LLaMA's transformer model, where N is the input sequence length, and d represents the feature dimensionality. Similarly, let P,- = (Pt......PPI.... R4xd denote the learnable adaptation prompt used to fine- tune the model. This prompt is concatenated with X, along the token dimension, serving as prefix, where L denotes the prompt length. For simplicity, we omit the subscript in subsequent notations.
Connection to mixture of experts. Recent studies (Le et al., 2024; 2025) have revealed a notable connection be- tween the attention mechanisms and the mixture of experts (MoE) architectures (Jacobs al., 1991; Jordan & Jacobs, 1994), showing that attention can be seen as a form of MoE, where attention weights act as gating functions over token interactions. This MoE perspective provides useful frame- work for analyzing zero-initialized attention by viewing its components as gates and expert responses.
E
Specifically, let X [ & [ E R(N++)d which is the concatenation of input tokens. For E [N + +1], define E; E IRdxIN+IIe such that E = æ; for i 1,..., N, and EN+1X = t. We then introduce a set of I. + N + experts Jn RRN++1) - Rdd, defined as:
Suppose the model is generating the (N + 1)th word based on (P,X) at the I-th layer. We denote the corresponding (N + 1)-th word token as t E TR'.. Within the attention mechanism, linear projection layers are applied to the input tokens, transforming them into queries, keys, and values, defined as follows:
f,(X) = - E,X = WVT Dj, i E [N] SN+1(X) = WVT EN+1 X = WV
(7)
(8)
Q t W& E glxds
(1)
fN+1+ = T py. j' E [L].
(9)
K [PT XT ,t] WK E RR(+N+I)xd,
(2)
Based on equation (5), the weights G, R(N+1)d + R associated with each expert are defined as follows:
V = [PT, x :t] W e g((+N+1)xd
(3)
where W Rdxd W gaxds, and WV E ddd.
3

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Ex41 wow" KT vil
Y-= Sc..a. (X;) 6, i= 1=.,,,...,.
(11)
exp(
G,(X)=
N+1 ExET exp(
X "E wown T vah
where the variables E En are independent Gaussian noise satisfying Ele; X,]] = and Var(e:|X.) = = I for all E [ni]. Additionally, we assume that X2, X,, are i.idd samples from some probability distribution μ. The regression function fa..a. (.) in equation (11) takes the form of the MoE model with N pre-trained experts and T. unknown experts, which is given by:
x3 E SW+tW9Ww XT P2)
expl
GN+1+5(X)
vdc
ΣV Sk=1 =1 exp(
X EN+, wOWK
vdz
for j' e [L]]and and E [N+1] +1. With these formulations, the output of ero-iinitialized attention, as described in equa- tion (6), can be expressed as:
N
exp X
19 + +ag)
SG..a. (x) =
11 1(x.1)) 17)
exp (X Al. X + +al)
N+1 Σ
exp((Bp..3' _ X + bb.g))
== GX(X) (X) + tanh(a)
+ (anl((a.)
-p--''
j'=1 Lk-=1 exp(( ) T ba
(12)
x Σ GN+1+s (X) (NN+))(()))))
(10)
where G. Ey "oo(bb,,')p... denotes true but un- known mixing measure, which is a weighted sum of Dirac measures 6, associated with unknown biases and prompts (bx.j' ....)) ) in the parameter space θ C IR x Idd, Fur- thermore, the gating factor a* unknown and belongs to the parameter space Q c R. In the model (12), the matrices Ag. the expert parameters 77.. and the bias parameters ag are known for all j = [NV] Finally, B E ddxd and C' E Bd xd are given and they play the role of pre-trained projection matrices in the context of zero-initialized attention.
From this formulation, zero-initialized attention can be inter- preted as a specialized form of a mixture of experts model. The set of experts fi..... fN+1. along with their associated weight functions, are pre-trained and require additional training, as their parameters are encoded within the pre- trained transformer model, representing existing knowledge. In contrast, the prompt experts f f2+2.... fN+++L and their weight functions work in conjunction with the pre- trained experts, effectively integrating newly acquired information into the model through learnable prompts. Viewing zero- initialized attention through this lens specialized mix- ture of experts model provides the foundation for further theoretical analysis, as demonstrated in the next section.
Least-square estimator. We can estimate the unknown prompts and gating factor in the regression model (11) via estimating the mixing measure G, using least-square method as follows:
(Gn, an) == arg min (0)
Σ 11' fc (o,,,,(a,))13 (13)
4. Convergence Analysis of Prompt and Gating Factor Estimations
1=1
where (z) = {G = Ei exp(b:)6p.
S e S
In this section, we provide theoretical analysis for linear prompts and non-linear prompts through the connection between the aero-initialized attention and the mixture of ex- perts in equation (10), demonstrating the sample-effiiiency of the pero-initalized attention over the uudom--nitalizee attention. In addition, the theoretical benefits of non-linear prompts offer great flexibility in improving the practical performance of the zero-initialized attention, which will be investigated extensively with several benchmark datasets and tasks in Section 5.
L', (b,, P.) E }} denotes the set of all mixing measures with at most L' prompts. For practical purpose, the number of chosen prompts L' is generally larger than the number of true prompts L., i.e., T.' 2 L,, to guarantee that the estimated prompts and gating factor from the least-square method converge to the true prompts and gating factor.
Convergence rate of regression estimator. We first show that the regression estimator fa. can still estimate the true regression function fG.,a at the standard parametric rate in terms of the sample size though we overspecify the number of prompts, i.e., L' L.
4.1. Analytics for Linear Prompts
We first consider the original setting of zero-initialized at- tention when the prompts are linear.
Proposition 4.1. The convergence rate of the regression estimator fāz a. fū. (-) to the true regression funntiooon (
Problem setting. Assume that (X1, (X,,, Y,,) Rd x Rd are i.i.d. samples of size n generated from the following regression model:
Y2)....,
under the (²( norm is of parametric order, that is,
11.a.. - Il.,a -O(( (r((oog(m)/))) (14)
4

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Proof of Proposition 4.1 is in Appendix A.3. Given the above convergence rate of the regression estimator, aim to construct loss function among parameters, de- noted by D(G,,, G.), such that lifa.. fa..a, (|(())) 2 (D(G,,, + jā, a.l]. Then, this lower bound together with the bound (14) will lead to our desired prompt conver- gence rates. To this end, we will build a loss function based on the concept of Voronoi cells as in (Manole & Ho, 2022).
slow, standing the order of Op(1/ log" (n)) for some constant チ 0, where is the sample size. Thus, to
approximate prompts with given enror need expo- nentially many data points O(exp(c-1/7)) which is not sample-efficiet.
(ii) Prompt convergence in zero-initialized attention: As shown in Theorem 4.2, the convergence rates of linear prompt estimations are polynomial orders, ranging from Op(n~1/3) to (n (up to some logarithmic term), which are substantially faster than those under the random- initialized attention. Therefore, we only need polynomially many data points O(c²2) or O(c to approximate the prompts with given error c.
Voronoi loss function. For mixing measure G E ( distribute its atoms across the Voronoi cells {Cz
Cj(G), [L]} generated by the atoms of G., where
C (i E [L'] lpp P-,yl S Pp: P.lell Ve }j}.
Then, the Voronoi loss function is given by
4.2. Theoretical Benefits of Non-Linear Prompts
1
D(G.G.):
Σ
Σ
exp(b:)
exp(ba,r)
j'=1
PECy
E
Σ
exp(b;)]l
Adapter
Zero-init. Attention
Zero Gating
+
Softmax
+
Σ
Σ
exp(b;)] Ap 11
(15)
x Softmax
where Ap,y' = Pi P. for all Cy and j' [L]. Given the above loss function, the convergence behavior of linear prompts in Theorem 4.2.
K
now ready capture
Vanilla Attention
Theorem 4.2. Assume that L' > L, i.e., the number of prompts is unknown and is overspecified by Lt' prompts. Then, the least square estimator (Gas Rn) defined in equa- tion (13) satisfies
Softmax
Prompt Embedding
K Linear (Io(P"),
Trainable
D(G,,
Frozen
Word Tokens
jā, a.l = Op( (l)())))n))/))
Figure 1. LLaMA-Adapter with non-linear prompt structures. Trainable prompts are integrated into the final layers the LLaMA model, where zero- zero-gating mechanism modulates the added prompts. This approach enables progressive learning of instruc- tional knowledge while keeping the remaining model parameters frozen.
Proof of Theorem 4.2 in Appendix A.1. Putting the first bound and the formulation of the Voronoi loss function D together, we observe that the convergence rates of estimating linear prompts range from the order Op((og(n)//n]) to Or([lo(n)) /n] which are optimal. Therefore, need polynomial number of data, either O or ((e to approximate the linear prompts with given eтror
While the original scro-initialization approach considered only linear prompts (Zhang et al., 2024), most current prompt-based techniques commonly reparameterize the prompt parameters with an MLP rather than optimizing them directly, in order enhance training stability (Li Liang, 2021; Liu et al., 2021; Le al., 2025). To increase both the flexibility and practical relevance of our results, we extend our analysis to zero-initialized attention equipped with non-linear prompts.
Sample complexity of random-initialized attention v.s. zero-initialized attention: From the above the convergence analysis of prompt estimation under the zero-initialized at- tention and that under the random-initialle initialized attention in (Akbarian et al., 2024; Le al., 2024), we claim that using the zero-initialized attention is more sample-efficient than using the aadomm-iitialiinn attention in terms of prompt convergence due to the following reasons:
(i) Prompt convergence in random-initialized attention: The convergence rates of prompt estimation are significantly
Problem setting. (X1, Yi), (X2, .,.,... .X,,,,,,,,,,,,,,,,,, Yn) E R
Suppose
that the data
x TRd
are
5

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Li.d. samples of size n generated from the model:
achieve the prompt approximation with given error e. As result, we claim that the zero-initialized attention with non-linear prompts is also more sample-efficient than the random initialized attention in terms of prompt convergence. Moreover, although sharing the same sample complexity as when using linear prompts, the zero-initializes attention with non-linear prompts will be shown to offer greater flex- ibility in practical applications, which will be discussed further in Section 5.
= = (x) + Ei, i
n. (16)
Here, we impose the same assumptions on the noise vari- ables E, and the input X; as in Section 4.1. Nevertheless, the regression function ga..o. (-) in equation (16) now takes the form of prefix MoE model with N pre-trained experts and L unknown experts,
exp T A + +a9)
Ic... (X)
-((x m)))
Non-Linear prompt optimization in practice. Motivated by the theoretical benefits of using non-linear prompts in zero-initialized attention in (q7(1 one replaces the lin- ear prompts P in zero aaro-initialized attention with non non-linear prompts (Figure 1), which given by:
1= exp( X A + +ag)
+ tanh((c.)
L exp((Bo(PP.))) ))T X + bo,y))
Co(p-)s)
X
== (B )) X + b,,w)
"X
P = (P) E Lxd
(20)
(17)
where o(.) is a non-linear activation function or a lightweight MLP, in line with common practice in most current prompt- based techniques, and P can be a layer em- bedding vector or a set of embedding vectors equal to the length of prompt for each layer. For instance, we can choose as a lightweight MLP with 2 layers combined with non- linear activation such as Tanh, ReLU, and Leaky-ReLU:
where a: : R + RR is some activation function applied element-wise to the prompts Px.j'. According to the change of the regression function, the least-square estimator of the true mixing measure G, under this setting becomes
(Gn, an) == arg min
Σ (8) 1=1
Y2 - (x.)) (18)
o (P) (P))),
(21)
In the following proposition, we will illustrate that the para- metric convergence rate of the regression function estimator still holds true under the setting of non-linear prompts.
where f1(), f2(.) are separate linear transformations d(.)
represents the non-linear activation function, e.g., ReLU or Leaky-ReLU, and P is defined as a layer embedding vector. To ensure parameter efficiency and facilitate knowledge sharing across layers, this MLP can be shared among the layers that utilize the prompts.
Proposition 4.3. The convergence rate of the model estima- tion 9G., an (-) to the true model gG..a. (.) under the L²()))
norm is parametric on the sample size, that is,
19._,.. - oa.... (p))()) = 1000p( OOp(Vlogn)//)). (19)
As established in Theorem 4.4, this non- formulation retains the same estimation rates as the linear variant, thus offering greater flexibility for practical applications.
Similar to Section 4.1, by utilizing the Voronoi loss function DD defined in (15) we are able to investigate the conver- gence behavior of non-linear prompts in the zero-initialized attention in Theorem 4.4 whose proof can be found in Ap- pendix A.2.
5. Experiments
To highlight the statistical advantages of zero-initialized attention and explore the potential of non-linear prompts, we conduct series of question- tasks. Section 5.1 provides an overview of our experimental setup, while the main results are presented in Section 5.2. Additional details and prompt templates are included in Appendix C.
Theorem 4,4. Assume that the activation function a satis- fies the Assumptions (A.1)-(A.2) specified in Appendix A.2. Then, the least square estimator (Gn, (n)) defined in equa- tion (18) satisfies
experiments on LLM
D(G,, =.)=0pp Or(V/og(n)/m)) 1ān-au a.1-= Op)loo((/)))))/
5.1. Experimental Setup
can be seen that the convergence of prompt parameters under this setting behaves analogously to that in Theo- rem 4.2. In particular, the prompt parameters P..j still admit the estimation rates of order Op((log(n)/n]}) or (p(log())/x({). Thus, a polynomial number of data points ranging from ( to O is required to
Datasets and Evaluations. We use the Open LLM bench- marks as in Beeching et al. (2024). These benchmarks eval- uate the generative abilities of L.MS in four different tasks, including (i) AI2 Reasoning Challenge (ARC) with Easy (eas) and Challenge (cha) types (Clark et al., 2018), (ii) HellaSwag (Zellers et al., 2019), (iii)
6

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Table Commparison between Linear prompt (zero-initialized mechanism) and Random-Init prompt on 4 LLM tasks using LLaMA-7B and LLaMA- 13B models.
ARC
MMLU
Hellaswag
TruthfullQA
Method
Average
Acc (eas)
Acc (cha)
Acc (aver)
Acc
Acc
Acc
ILaMA-7B zero-init LL + rand-init
62.29 1 1.44
43.17 2.44
52.73 2.06
36.28 1.16
76.79 4.177
45.53 1m
52.83 3.77
60.65
40.7
50.67
35.12
72.62
37.82
49.06
I.aMA-13B + zero-init
81.78 a
64.33 1.44
73.06 0.3
49.64 1,62
81.21 a.00 81.16
34,88 1.336
59.70 f.
LJaMA-13B + rand-init
81.61
63.91
72.76
48.02
34.52
59.12
MMLU (Hendrycks et al., 2020), and (iv) TruthfulQA (Lin et al., 2021). All these tasks evaluate the model through multiple-choice questions, where ARC and MMLUU test the LLM's knowledge, HellaSwag tests the model's ability to finish sentences, and TruthfulQA measures whether an LLM is truthful in generating answers to given questions.
the Alpaca dataset (Taori et al., 2023), where all model parameters are updated. Additionally, we benchmark against LORA (Hu et al., 2021), PEFT method that inserts trainable low-rank decomposition matrices into each layer's weights, and VeRa (Kopiczko et al., 2023), another PEFT method that uses a single random pair of low-rank matrices shared across all layers, while learning separate scaling vectors for each layer.
We follow the experimental setup of LLaMA-Adapter (Zhang et al., 2024) by fine-tuning LLaMA on the Alpaca dataset (Taori et al., 2023). The model performance eval- uated on the test set by conducting a zero-shot evaluation for ARC, MMLU, and TruthfulQA while using 10-shot setting for HellaSwag. Here, -shot refers to incorporating n instruction-follovii samples into the prompt question.
(5) Other Prompt Tuning Methods: including (Lester al., 2021) which optimize continuous prompts added to input text; IA3 (Liu et al., 2022) a method using three learnable vectors to rescale the keys, values, and position-wise feed-forward activations in the attention layers.
Architectures Training, We employ experiments on two ILaMA versions, LLaMA-7B with 32 Transformer layers and LLaMA-13B with 40 Transformer layers (Touvron et al., 2023b;c). The models are trained with A A100-GPUs for 5 epochs. The training configuration includes warmup period of 2 epochs, a total batch size of 64, a learning rate of 0.009, and a weight decay of 0.02. With LLaMA-7B, we use a prompt with length L = 10 and integrate adaptation prompts into the last K = 30 layers. On LLaMA-13B, we use L = 10 and insert prompts at the last K 38 layers.
5.2. Main Results
I. Zero-initialized attention is essential in prompt-tuning, enhancing both robustness and effectiveness compared to iindom-iitiialized attention.
We begin by investigating the impact of zero-initialized prompt-tuning on LLaMA-7B and I.L.MA-13B using the linear prompt setting, comparing its performance against conventional random-initializann initialization strategies.
Baselines. To access the effectiveness of zero-initialized attention and demonstrate the potential of integrating it with the proposed non-linear prompt. we conduct experiments using the following configurations:
As shown in Table 1, the zero-initialized mechanism en- hances stability in LLaMA-13B and significantly boosts the performance of ALLMA-Adapte when using LLaMA- 7B, compared to conventional random-initialization strate- gies. For instance, with LLaMA-7B on Hellaswag and TruthfulQA, the Linear prompt surpasses the Random-Init prompt by 4.17% and 7.71%, respectively. The varying im- pact of zero-initialization between LLaMA-7B and LLaMA- 13B can be attributed to differences in model capacity and expressiveness. LLaMA-13B, with its larger parameter space, naturally generalizes better even with randomly ini- tialized prompts, reducing the relative advantage of zero- initialization. In contrast, LLaMA-7B, with its lower capac- ity, benefits more from vero-initialization, as it relies heavily on efficient adaptation mechanisms to optimize learning.
(I) Linear prompt: i.c., the default setting as LLaMA- Adapter (Zhang et al., 2024), where prompt vectors P are vero-initialized and used directly.
(2) Non-Linear prompt: use zero-initialized mecha- nism and apply nonlinear MLP on input prompts P to generate prompt vectors P (PP) and the MLP layers are shared among layers.
(3) Random-Init prompt:
use the input prompts with conventional candomly-initializatiin i.e., forms in E4)(4) rather than the zero- irroiiniiialized mechanism.
(4) Finetuning & low-rank decomposition: We com- pare with fully fine-tuned L.LMMA model trained on
In summary, those results align with our thcoretical find- ings in Section 4, which demonstrate how arrtiiniializee
7

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
attention improves both robustness and sample efficiency in parameter estimation.
well each initialization strategy adapts to limited data and whether zero-initialized attention provides consistent ad- vantage in sample efficiency.
IL Non-linear prompts provide the potential to improve the performance of the zero-initialized mechanism and achieve competitive performance with full fine-tuning.
We observe that both Non-Linear and Linear prompts signif- icantly enhance sample efficiency in parameter estimation for prefix-tuning in LLMs compared to the Random-Init prompt (which uses conventional attention). In the LLaMA- 7B setting (Figure 2). both Non-Linear and Linear prompts outperform Random-Ing Init across all fractions of the Alpaca training set. For example, the Non-Linear prompt exceeds Random-Init by 3.77% when trained on 100% of the dataset, while Linear exceeds it by 2.05%. When trained on 50% of the dataset, Non-Linear and Linear outperform Random-Imi by 4. 4.72% and 2. 16%, respectively.
We evaluate the impact of the non-linear prompt setting on the LLaMA-Adapter by comparing it with the linear prompt setting on LLaMA-7B and LLaMA- 13B. Additionally, we benchmark against a fully fine-tuned LLaMA model and a version fine-tuned using LoRA. All models are trained with the Alpaca dataset.
Tables and 3 presents our results with the following ob- servations: first, the linear prompt consistently matches or outperforms the linear prompt across both LLaMA-7B and LLaMA-13B, with performance gains rang- ing from 1-2%. Notably, on the MMLU and TruthfulQA datasets with LLaMA-138B 13B, the non-linear prompt achieves scores of 51.32 and 38.92, respectively, compared to 49.64 and 34.88 for the linear prompt, demonstrating its effective- ness. Table 3 also shows that the non-linear prompt does not significantly increase training time compared to the lincar version.
In the LLaMA-13B setting (Figure 3), similar trend is observed, with zero-initialized attention showing consis- tent advantages over Random-Init, and the Non-Linearr prompt slightly outperforming the Linear prompt in most cases. The only exception is at 1% training data, where Lin- car surpasses Non-Linear by 1.99% in accuracy. In short, these findings corroborate our theoretical results in Section 4, which demonstrate the sample efficiency of the zero- initialized attention mechanism in parameter estimation.
Secondly, compared to other PEFT methods such as Prompt Tuning, IA3, and VeRA, we see that the non- -inner prompt significantly outperforms all these PEFT meth- ods in LLaMA-7B version. For example, on the MMLU and TruthfulQA, the non- near prompt. surpasses Prompt Tuning by 4.1% 1% and 10.28%, respectively, demon- strating the effectiveness of the new PEFT design.
50.63
52.12
53:48
50
40
Accuracy
30
Finally, when compared to fully fine-tuned and LoRA-based versions of LLaMA-7B, our approach using a non-linear prompt achieves performance nearly on par with full fine- tuning while surpassing LoRA by an average accuracy mar- gin of 0.64%
20
10
7B + Linear prompt
7B Random prompt
0
7B Non-Linear prompt
20
40
60
80
100
In summary, these findings validate our theoretical obser- vations and highlight the effectiveness of combining non- linear prompts with zero-initialized attention and improving prompt-tuning performance in LLMs.
Fraction of training set (%)
Figure 2. Accuracy of different prompt strategies (Linear, Random, and
with LLaMa-7B model across varying fractions of the training set. The Non-Linear prompt consistently outper- forms the other methods, especially in low-data regimes.
III. Sample Efficiency of Zero-Initialized Attention vs. Random-Initialized Attention.
6. Conclusion and Limitations
Figures and provide a systematic analysis of the sample efficiency of irrtiinntialzzed attention by evaluating its per- formance under varying data availability. Specifically, we randomly subsample the Alpaca dataset at different fractions (1%, 10%, 30%, 50%, 100%) ) to simulate low-data scenar- ios. We then fine-tune the Non-Linea Linear, Linear, and Random-Ind Init prompts on these subsets for both LLaMA- 7B and LLaMA-13B and evaluate their performance on the ARC dataset. This experiment allows us to assess how
In this paper, we demonstrate that zero-initialization prompt- tuning for adapting LLMs is not just an engineering trick but can be rigorously explained through theoretical proper- ties by drawing connections between attention mechanisms and the mixture-of-experts perspective. Based on these insights, we introduce a novel non-linear prompt-tuning approach that outperforms linear prompts in terms of both performance and robustness while achieving competitive results compared to the full fine-tuning of L.aMA on the
8

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Table Comparison of Non-l.i near prompt, near prompt, and various fine-tuning methods. Params denote the total number of parameters updated during the fine-tuning process. Bold values indicate better scores between linear and non non-linear settings.
ARC
MMLU
Hellaswag
TruthfullQA
Method
Params
Average
Acc (eas)
Acc (cha)
Acc (aver)
Acc
Acc
Acc
LLaMA-7B, Fully Fine-tuning Alpaca
7B
67.47
46.25
56.86
37.25
77.09
42.35
53.39
H.MMA-7B, LoRA Alpaca
4.2M
61.91
42.15
52.03
34.87
77.53
46.14
52.64
LLaMA-7B, Prompt Tuning Alpaca
42K
55.35
37.46
46,41
32.85
75.88
34.76
47.48
LLMMA-7,, IA3 Alpaca
524K
$2.06
35.92
43.99
31.65
75.73
32.8
46.04
LLaMA-7B, VeRA Alpaca
541K
49.2
35.49
42.35
30.88
75.59
31.95
45.19
LLLMA-7B + zero-init + lincar
1.2M
62.29
43.17
52.73
36.28
76.79
45.53
52.83
LLaMA-7B zero-init + non-linear
2.6M
63,51
45.39
54.45
36.95
76.67
45.04
53,28
LLaMA-13B + zero-init linear
1.9M
81.78
64.33
73.06
49.64
81.21
34.88
59.70
LLaMA-13B + zero-init non-linees non-linear
3.3M
82.87
66.55
74.71
51.32
81.72
38,92
61.67
leaving the performance on multi-modal LLMs unexam- ined. Multi-modal tasks may require adjustments to the framework in both theoretical perspectives and practical implementations (Li et al., 2023; Nguyen et al., 2024a). Sec- ond, the integration of adapters and the optimal number of prompt embeddings across layers have not been systemati- cally explored. Further empirical experiments are needed to refine these components for better efficiency and adapt- ability in more complex models. Finally, the current study primarily focuses on performance metrics, but the inter- pretability and stability of non-linear prompts in real-world deployment scenarios require further analysis. Addressing these limitations will contribute to a deeper understanding of the robustness and generalizability of PEFT for LLaMA models.
74
71.13
70
03:44
60
50
Accuracy
40
30
20
138 Linear prompt
10
138 Random prompt
13B Non-Linear prompt
20
40
60
80
100
Fraction of training set (%)
Figure 3. Accuracy of different prompt strategies (Linear, Random, and Non-Linear) using the LLaMa-13B model across varying fractions of the training set. The Non-Linear prompt consistently achieves the highest accuracy, particularly in low-resource settings. contrast, Random prompt performs significantly worse when only a small portion of the training data is available, but gradually improves as more data is introduced.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Table Efficiency Comparison. The training is conducted on
GPUs A A100-80GB.
Acknowledgment
Tuned Params
Storage Space
Training Time
Method
This work was supported by Deutsche Forschungsgemein- schaft (DFG, German Research Foundation) under Ger- many's Excellence Strategy EXC 2075 390740016, 6,
[.AMA-7B+ zemoinit linear
1.2M
4.4M
1h54'
LLaMA-7B+ zero-init non-linear
2.6M
9.5M
Ih54'
LLaMA-13B zero-init + linear
1.9M
6.9M
3h17'
the DARPA ANSR program under award FA8750-23-
LLAMA-13B zero-init non-lincar
3.3M
12M
3h17'
2-0004, the DARPA CODORD program under award HR001 12590089, The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Duy M. H. Nguyen and Daniel Sonntag are also supported by the XAINES project (BMBF, 01IW20005), No-IDLE project (BMBF, 011233002, and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. This research is also supported by research funding from Faculty of Informa- tion Technology, University of Science, Vietnam National University Ho Chi Minh City.
Alpaca dataset. Our findings are validated across several muestion-answering tasks, tested on both LLaMA-7B and LLaMA-133 13B architectures, and align with our theoretical analysis in practical settings. We believe our results will encourage further exploration into initialization techniques and their potential for improving other parameter-efficient fine-tuning methods.
Our study also has some limitations that need further ex- ploration. First, has only been tested on unimodal L.MMs,

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
References
models with feature adapters. International Journal of Computer Vision, 132(():581-555, 2024.
Akbarian, P., Nguyen, H., Han, X., and Ho, N. Quadratic gating functions in mixture of experts: A statistical in- sight. arXiv preprint arXiv:2410. 12202, 2024.
Han, X., Nguyen, H., Harris, C., Ho, N., and Saria, Fusemoe: Mixture-of-experts transformers for fleximodal fusion. In Advances in Neural Information Processing Systems, 2024.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, F.,, Bailey, P., Chen, Z. et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009 2009003300, 2020.
Beeching, E., Fourrier, C., Habib, N. Han, S., Lambert, N., Rajani, N., Sanseviero. O., Tun- stall, L.,, and Wolf, T. Open Ilm leader- board. https: htthhggggggfcceeeeoo co /saccssacs////// SaseginFFaceH444 open 11m leaderboard, 2024. 2cce
Ho, N., Yang, C.-Y,, and Jordan, M. I. Convergence rates for Gaussian mixtures of experts. Journal of Machine Learning Research, 23(323):1-81, 2022.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B.,
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
De Laroussilhe, Q., Gesmundo. A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint (KKiv:20060990) 2021.
Chen, S., Jie, 7.,, and Ma, L. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruc- tion finetuning milms. arXiv preprint arXiv:2401. 16160, 2024.
Jacobs, R. A., Jordan, M. I., Nowlan, J., and Hinton, G. Adaptive mixtures of local experts. Neural Computation, 3, 1991.
Chen, T., Chen, X., Du, X., Rashwan, A., Yang, F., Chen, H., Wang, Z., and Li, Y. Adamv-moe: Adaptive multi- task vision mixture-of-experts. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 17346-17357, 2023.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181-214, 1994.
Chen, Z., Deng, Y., Wu, Y., Gu, Q., and Li, Y. Towards un- derstanding the mfitture-ff-expeeee layer in deep learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 23049-23062. Curran Associates, Inc., 2022.
Karimi Mahabadi, R., Henderson, J., and Ruder, S. Com- pacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022-1035. 2021.
Kopiezko, D. J., Blankevoort, T., and Asano, Y. M. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. /0. /454, 2023.
Clark, P., Cowhey, I., Etzioni, O.,, Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018
Le, M., Nguyen, A., Nguyen, H., Nguyen, T., Pham, T., Van Ngo. L., and Ho, N. Mixture of experts meets prompt- based continual learning. Advances in Neural Information Processing Systems, 38, 2024.
arXiv preprint
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: 1810.04805 2018
Le, M., Nguyen, C., Nguyen, H., Tran, Q. Le, T., and Ho, N. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. The Thirteenth International Conference on Learning Representations, 2025.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23( 121): 2022.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficins prompt tuning. arXiv preprint arXiv:2104.086% 2021.
Gao, P., Geng, S., Zhang, R., Ma. T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-languag
10

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541-28564, 2023.
alignment for medical vision-language model. arXiv preprint arXiv:2410.02615, 2024a.
Nguyen, D. M., Le, A. T., Nguyen, T. Q., Diep, N. T. Nguyen, T., Duong-Traan D., Peters, J., Shen, L., Niepert, M., and Sonntag, D. Dude: Dual distribution aware context prompt learning for large vision-language model. Asian Conference on Machine Learning (ACML), 2024b.
Li, D., Ma, Y., Wang, N., Cheng, Z., Duan, I.., Zuo, J., Yang. C., and Tang, M. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint ar arXiv: 240.::2400.5554 15159, 2024a.
Nguyen, H., Nguyen, T., and Ho, N. Demystifying soft- max gating function in gaussian mixture of experts. Ad- vances Neural Information Processing Systems, 36: 4624-4652, 2023.
Li, H., Lin, S., Duan, L., Liang, Y., and Shroff, N. B. The- ory mixture-of-experte in continual learning. arXiv preprint arXiv:2406. 16437, 2024b.
Nguyen, H., Akbarian, P., and Ho, N. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML, 2024c.
Li, X., Lian, D., Lu, Z., Bai, J., Chen, Z.,, and Wang, X. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in Neural Information Pro- cessing Systems, 36, 2024c.
Nguyen, H.,, Akbarian, P., Nguyen, T., and Ho, N. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the ICML, 2024d.
Li, X. I.. and Liang. P. ttninn- Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Nguyen, H., Han, X., Harris, C. W., Saria, S., and Ho, N. On expert estimation hierarchical mixture of experts: Beyond softmax gating functions. arxiv preprint arxiv 2410.02935, 2024e.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimie human falsehoods. arXiv preprint arXiv:2109. 209,0290,,, 2021.
Nguyen, H., Ho, N., and Rinaldo, A. Sigmoid gating is more sample efficient than softmax gating in mixture of experts. In Advances in Neural Information Processing Systems, 2024f.
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine- tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1950-1965, 2022.
Nguyen, H., Akbarian, P., Pham, T., Nguyen, T., Zhang, S., and Ho, N. Statistical advantages of perturbing cosine router in mixture of experts. In International Conference on Learning Representations, 2025.
Liu, x., Ji, K., Fu, Y., Tam, W. I., Du, Z., Yang, Z., and Tang, J. P-tuning v2: Prompt tuning can comparable to one-tuning universally across scales and tasks. arXiv preprint 10.07602, 2021.
OpenAI. Chatgpt. https: Accessed: 2025.
openai....................a..... com, 2025.
Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art parameter- efficient fine-tuning methods. URL: https://github. com/hugiigfaace/pef.. 2022.
Jp J. A., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report, 2024. URL https://arxiv. arg/abs/2303.08774, 2:6, 2024.
Manole, T. and Ho, N. Refined convergence rates for max- imum likelihood estimation under finite mixture mod- els. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14979-15006. PMLR, 17-23 Jul 2022.
Pardo. L. Statistical Inference Based on Diver- gence Measures. Statistics: Series of Text- books and Monographs. CRC Press, 2018. ISBN 9781420034814 URL hittps://bbole google.
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, Large language models: A survey. arXiv preprint arXiv: 200.0:6900.61966 2024.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Nguyen, D. M., Diep, N. T. Nguyen, T. Q., Le, H.-.., Nguyen, T., Nguyen, T., Nguyen, T., Ho, N., Xie, P., Wat- tenhofer, R., et al. Logra-med: Long context multi-graph
Pham, Q., Do, G., Nguyen, H., Nguyen, T., Liu, C., Sar- tipi, M., Nguyen, B. T., Ramasamy, S., Li, X., Hoi, S., and Ho, N. Competesmoe effective training of

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
sparse mixture of experts via competition. arXiv preprint arXiv:2402.02256, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 07830, 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N.,
He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine-
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140):1-67, 2020.
tuning. arXiv preprint arXiv:2303.10535 2023.
Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y.,
Li, H. and Gao, P. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference Learning Representations, 2024.
Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J.. Reimers, N., and Gurevych, I. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918, 2020.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):2337-238 2022.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
arXiv preprint arXiv:1701.065: 066388,0.........00 2017.
Shi, Z. and Lipani, A. Dept: Decomposed prompt tun- ing for parameter--fficient fine-tuning, arXiv preprint arXiv:2309.05 2023.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-followingg llaaa model, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 2023a.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 139711991, 2023b.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023c.
van de Geer, S. Empirical processes in M-estimation. Cam- bridge University Press, 2000.
Vaswani, Attention is you need. Advances in Nearal Information Processing Systems, 2017.
Yan, F., Nguyen, H., Le, D., Akbarian, P., and Ho, N. Under- standing expert structures on minimax parameter estima- tion in contaminated mixture of experts. In Proceedings of The 28th International Conference on Artificial Intelli- gence and Statistics, 2025.
12

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
Supplement to "On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation"
In this supplementary material, we provide detailed proofs of the main results in Appendix A. Additional discussion on related works is in Appendix B and additional experimental details are in Appendix C.
A. Proofs
A.1. Proof of Theorem 4.2
Based on the convergence rate of fe to fc..a. in Proposition 4.1, to obtain the conclusion of Theorem 4.2, we only need demonstrate that
|| I.c.aalllaa C (D(G, G.) la-aa.)
for any (G,a) E (9) Q for some universal constant It is equivalent to proving that:
inf (e) - fl. (e..a.lleew///((GG G.) + a - a.l) > 0.
To obtain the conclusion for above inequality, we consider two parts: (i) local part, namely,
lim 5-+ (e)_)))
inf
Ilfa,a - fc..a. / G.) + +aa - -a.l) > 0:
(ii) global part, namely, for any E > 0
- /tt.e.....ll////(GG.GCC.
>
(Go))
Local part: We first start with the local part, which is equivalent to demonstrating that
lim 5-0
IlfG,a - (Lc.,.ll/////
+ +la - a.l) > 0.
We prove the above claim by contradiction. Assume by contrary that the above claim does not hold. indicates that we can find a sequence of mixing measures Gn :- E/e exp(bn.3') )) in G1/ (O) and a sequence of on S2 such that when + co, the following limits hold:
{1IE..... - Sa.... (LP (D G.) + lan a.l) + 0.
D(G,, G.) + an a.l +0, 0,
The first limit indicates that D, D(G,,G.) 0 and an a. - 00.
For the simplicity of the ensuing presentation, we denote C7 C,(G,) as Voronoi cell of G, induced by the j-th components of G.. Without loss of generality, we assume that those Voronoi cells do not depend on the sample size, i.c.. C = CC,, which is possible since our arguments are asymptotic. Therefore, we can rewrite the Voronoi loss D, as follows:
Dn :
Σ - iEC,
Σ
Σ
F=1
Σ
Σ
exp(bn.i) (
rE(LHCyy>sLECG
where Apn.yj' Pn,i - P for all Cy.
From the hypothesis, we have D, 0, which implies that Eiec, exp(bn.;) - exp(b..)) and Pn,i - Pa for any i Cj.j [T... To establish the contradiction, our proof consists of three main steps.
13

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Step 1 Taylor expansion.
To ease the presentation, let denote
= exp((Bpn.) X bm.s) Cpn.i
In
j=1 k= exp((Bpn.)) T bn,kk)
fc. = exp(( TX + by,z')
Cp...
Σ == exp((Bpa,.) X + b))
We now decompose the function Qn(X) follows:
Qn(X):= Σ exp((Bpr,x)) T X br -Ja..a SG. (X))
= exp((Bp.,xt) T X benkr]] (aan))((tannnaaa (tanh(an)fe,, (X) (aa
b exp((Bpr.kr) X + bu.er) tanhi(an) (sc. Sc. [c
k=1
X + be
- tann(an)-tanh(a)))[aa.. (X)
= Qn.1 + (X).
For that purpose, we will decompose the two terms Qn,, (X) and (x) respectively.
Decomposition of the function (X). We have
Qn,((x)= = Σ
X)Cpp, (an)aa))xp(hni.. - exp(((ppx))))) X)Cp..]
j=1 IEC
1
Σ jEC
tanh(an) exp(Om.i) (exp((Bpn.)) (
x) exp((Bpa.) T x) fc., (X)
+
tanh(an)) am)
Σ
exp(bm.)) - exp(b.,)) exp((Bp=.s)) x) ()(pp - fx,
iEC
= Ax (X) + (X).
(22)
We now proceed to decompose the functions A, (.) and (.).) via Taylor expansion.
Decompasition of the function (x). We first define the following functions Ú(X;p) = exp((pp) T X and (p)) = Cp. Then, we denote the product these functions as F(X;p) = (pp)) To decompose A, (X). we separately consider Voronoi cells with exactly one element and those with more than one element. It leads to the following decomposition of the function (X):
Ax(X) = Σ
Σ
(anh(an))xp(((n ( - ((X:p_))
I
Σ Σ tanh(an) cxp((n.)) ( -F(::
; 1 iEC,
= An, (X) + +x,,(X)),
where we denote (x)) = E3 C,1-11111 Eiec, tanh(an) exp(bnn)) P(X:pn.) - F(X:p=))) and An,(())
Este,1s Eiec, (anh(an) exp(En.) ((X:pn.)) - F(X:
14

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
For the function 1 (X), (X), for any indices i C and such that [Cyl 1, the first-order Taylor expansion entails that
Glalū (X: + (x),
U(::p..) = p-); +
Σ
(Apr.)))
Op
|a|=1
Olalv (Do.)) + Raj22
((P.,)) = -onn)--(pp...)) V(D.g)+ + Σ (Apm.ts)
Opa
|a/=1
where the terms Rx,,(X) and Rı,.2 Taylor remainders. Combining the above results leads to the following formulation of the function
tanh( exp(px (b..))
ala (X: (X:
(n,(X)==
Σ
(Apmay) Op
=:IE,
aa|=1
glalV
(Apn.ts) Op (p- (U: + Rm,(()
ela
Op (x; (VP..)) + Mn.ja alai Op (p. (pxs)U(X:PPs)) Rx,1(X)
M Mu,ja
/:1c//=1 [a|=1
where the function ( the combination of Taylor remainders and satisfies that R (X) + +aan- -a when n cc. Furthermore, the formulations of Mn.j.a are follows:
Σ tanh(a,) exp(bn.:) (Apmss)"
M Mn.j.a
a!
SEC
for any |or|
Moving to the function (x)). the second-order Taylor expansions of the function UXX; pn,a) around the function (X; P..j) and the function (Pn,,) around the function (p..)) for any indices i Cj and such that |C₁l >> 1, we obtain the following formulation of the function (X):
U Opa
dloli (X: + Musaa Op (p-,)
2x,((X)
Σ
Σ
Mn Mnsja
IH >1 15
gloli Mn
@ Op3 (px.s) + R
Σ Σ
(X:p..)
Op"
HC lai =1
where the function (x) is a combination of Taylor remainders and satisfies n - 00. Furthermore, we define
+ +lo,- -a.l) - 0 when
tanh(a,) exp(bm.i) (Apn.ss)",
M Mn.j.a
SEC
for any |ai| 2 and
tanh(an) exp(bn.s) (Apn.is) a4.
Mn.ja.3
a!3!
seC,
From the formulations of the functions U(X:p) and V(p), we obtain the following explicit forms of their partial derivatives:
for any |a| [8] 1.
aÜ
X ())((BB))))(BLL..)
0p(u)
(X:p) exp(((Bp)) X (B1,)
0p(w)
15

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
aV
Op(a) (p) = Clu.
0 VV
0p 0p(") (p) = O.
In these formulations, we use 1, to denote the vector that its u-th element is and its other elements are for S u S Plugging these explicit formulations of the derivatives of the functions ((:;)) p) and Vip), we can express the functions An, (X) and (x) as follows:
An,1(X)=
E
exp((Bp..) x) (P.. + (p= B x) Rx,,(X),
J-1,/==
An2(X)=
Σ exp((Bpt.) x) [1) + (p= B
1111111
+ (B' x) L3 (p. B x] (x),(X),
where the functions (p), C2,n (p), (p), Ln (p), and L3,n (p) are defined as follows:
L1,(())
Mn..a Clu
L2,(1) = Σ Mn.j... CnCp,
uml
Lim(() Σ Mn.ile Clui
u=1
(2,1(D) Σ Mn.ii. 1,
Σ
Mn.idlll Clude
Isa,edd
Lam(p)- = Σ Mn 1u
ISawad
In these formulations, we denote Ln the matrix that its (u, v)-th element is and its other elements are 0 for any 1 1 u.U.d d.
Decomposition of the function (X). Similar to the decomposition of the function A, (,,),, we can decompose the function B..()) follows:
(x(X)= Σ
Σ
tanh(a,) exp(6n..) - fc.
sic,
tanh exp(bn.i) tenh(e.))x)((nn)) #.C,,
Σ
(x))
B,,1(X)
(x,
where we denote (x) = Exiec Eiec, tanh(an) exp(bn.)) (X::p..) ( UX: U(: (X) and Bx, =
E Eiec, tanh(a,) exp(bn.i) (((::p..) (XX: (X).
Similar to the Taylor expansions for the functions An, (X) and (X), by using the first-order Taylor expansion to B (X) and the second-order Taylor expansion to Bn (X), we obtain that
alo
Bm,1
Σ Σ
Masaa Op (X fo, (x)
=1
16

---

On Zero-Initialiedd Attention: Optimal Prompt and Gating Factor Estimation
(x,,(X)=
Σ Mu.j. Opa (X:p..)ff. (X) + (X)
where the functions Rn,3( (x), 1 are Taylor remainders. Furthermore, they satisfy that Ra (D + O and a (D, |an, a. - when co. Given the explieit formulations of the derivatives of the functions U (X; and V (p), the functions BA, (X) and 2, (X) can be then rewritten as follows:
(x, = Σ exp((Bp...) ) X N. (p+.) Xfa (X) + Rx,)(X),
8:C/1=1
Bn2(X) = Σ
exp((Bp...) ) X (D+ BB B
j/C//s1
+ + x) (p ) ff, + (X)(
Here, the functions Mi,a NI, and (X) have the following formulations:
(X)
(p,n(p)
M Ma 1. lus
d
Np)
Σ
M Mn.ja aa hus
(2)
Σ
Mn M. 1.1.
Isw,edd
Collecting all of the above results with the decomposition of the functions (X) and (X). can represent the function (X) in equation (22) as follows:
(x)
Σ
exp((Bp..)
x) [F L2,n (p= TB x)
Σ
exp((Bp..)) T X) Ei (P..)) + (p..)) B X + (B X) (3,n(p,,) x
; |1
Σ
exp( (Pp.) T x) Mn.j.o, + N1, (P..)) T TB" x JG, (X)
==
Σ exp((Bp,.) T x) [M..j.0a + (I, BT x + (BT T N2.((p.)) X x)fc.()
/1,,1>1
RM,1 (X) + (x) Rn,a(X) (X)
(23)
where we define Mnjjos = tanh(ann) Eiec, exp(bm.i) - exp(bra)) for any index 1 < j S L, (pos)
(1,((=.))) + Mn.juu CP and Li,, (p=.) (i,((rn)) Mn.j.aePP...
Decompasition of the function (X), An application of the first-order Taylor expansion leads to the following expression for the function (X):
(x) [tanh(on) tanh(a.))
(xp((Bp....)) TX
X + Ba.wr)] fc. (X)
==1
=a - Σ exp((Bpa,r)) TX by,kr) fc. (X) Rx(X),
(24)
where the function An,5(X (X) is Taylor remainder and satisfies that (x.s(X)/22 (D, + |an - -a.)) - as n - 00.
17

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
Step 2- Non-vanishing coefficients.
The results of equations (23) and (24) indicate that (x))))Eii Rr (x)1/ (2,+ combination of the linearly independent functions exp((Bpa.)) X), (B' x (w) exp((Bp-.) X), (B' () x)(() exp(( X), exp(()p..)) X)fa, (X),
la, a.l) can be represented
B T X (w) exp(( X) (X), (B (B ( X) x) exp((Bp..,) (T x),G. (X), and [1 tanh"(aw)
Σ (Bpp((Bp,,)) T + by,x)) fa_ (X) for any S S L and 1 S u S d.
Our claim is that at least one of the coefficients of these linearly independent terms in the formulation of (Xn(X)
E (X))(())) + onn - a.l) does not go to 0 as - co. Assume by contrary that this claim does not hold, which means that all the coefficients of these linearly independent terms go to as + oc. Therefore, as n + co we obtain that
(P..)..)) (D, + 0n aa + 0, C2, (P.,)(,) (D, + lan a.l) - 0, L, (p-j)/ + la, -a.l) ++ 0, C2, (p. (1) (Dn + On aal) 0, L (p..)) (w) (D,, + lan - -a.) + 0, L (p..) )) (w)) (D, + la,, - aal) - 0,
C3, (P+ (w/)/ (D,, + |a -a+l) - 0, Ni, (p,)) /) + +lan a+l) 7 0, N1, (p+,)) (w) (D, + |a aa 1) - 0,
Nn, (p-.) (w0) (D,, |nn - a.1) + 0, Mn Dr + 0, (a, a.)/ (P, + |an - -aa)) +
for any < u S and أ I.. As (an /D) +aan a, 0, we deduce that
|0,, 0.1 (Dx
-> 0.
(25)
+ an a.l)
Note that since Ox, + a, + 0 n + co, we have 1/ tanh(an)) A 00. Then, as Mn.j.oa/(Dy, + lan -a.l) - 0, implies that
Eiec, exp(bm.)) - exp(ba,))
M 0,
(Dn + Jan - aa.l)
tanh(an) (Dn + aa - a.l)
for any j S T.. By varying the index from 1 to I. in these limits and summing them up, we obtain that
Σ/= Eiec, exp(bn.s) exp(b..)) + O. (D, + Ian a.l)
(26)
Now, we consider indices j E [L] such that [C,l = 1, i.e., the corresponding Voronoi cell has only one element. From the hypothesis, we have C2,n (P.,,) (()) (Dn, lan a.l) + 0, which leads to Ma.ja../ lan a.l) - 0. Hence, we find that
Eiec, (Dn + lan - a.l)
Σd MM
tanh(an) (Pn nm + |ax axl) +
That limit directly implies the following result:
EsIC Eiec, exp(6n,.)ll (D, + lan a.l)
+ 0.
(27)
We now move to the Voronoi cells having more than one element, namely, we consider indices j E [L] satisfying |C,1 > 1. The limit Ls, (s.n(p.))(u)) (D,, |an, - a., + 0 induces that
Eiec, exp( 11 (Dn + lan a.l))
Ed= La (Pr.,) + O. tanh(an) + la,, or.l)
By varying the indices j in these limits over all the Voronoi cells C, having more than one element, we find that
ΣSAG/11 Eiec, ( + 0.
(28)
(Dn + lan a.l)
18

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Combining the results from equations (25), (26), (27), and (28) leads to
= Dn + |0tn Os.] D, + |an a..
-+ 0
as n + 00, which cannot hold. As consequence, at least one of the coefficients of the terms in the formulations of (x,(X)= > 1- Rn, (x)) + |an - a.l) does not go to 0 as n + 00.
Step 3 Applicatioo of the Fatou's lemma. We denote ma as the maximum of the absolute values of Li,,, (D= + Jan, a.l). 2n, (p. (Dn, + lan a.l). Ln (D, + lan a.l), L2,1 (P..)(()) (D,, + lan a.l). (P.n(p.,)) (Dn + 1n - (in)))) la, a. 1), (P..(( (D, + lan - -a. a..l..N,,m (P.)) (wv) (Dn + jann - a.l), Mn..ua/ (Dn, n + aan a.l). and (an - (D,, + l a.l) for all 1 S u, v S d. From the result of Step 2 in the proof, we have 1/mn A 00 + 00.
Recall that II - fc. (k2 (Dx, + la, - a.l) 0 as + oc, which indicates that I| fG..a. (l.20///mmm()) (m,(D, + ann a.l)) + O. Furthermore, since the (4) norm is equivalent to the L (p)) norm, we have [Isa..... - Sa..a..lee (ILA(w) (mn (Dn + Ion - a.))) - 0. An application of Fatou's lemma leads to
an - -fG.,a.lu.te (1r,(w)
Ilfan.a (X) - fG.... (X)l1 du(X)= 2 0.
0= lim 2-+00
lim inf 10000
mn (Dn + 10 a.ll)
mn (Dn + |an -a.))
IR,. (X) -c.. (X)[l
indicates that lim inf,
0 for almost surely X. As - 00, we denote
Mn mn (Dn + an - -a.l)
(Poj)
(p=(s)
- -āz
- -3..
mn (D,, + an - -arl)
mn (D, ann axll)
fg (P..))
L2 (pa.)
(pnn) + 7/3
- aj:
+ +33
m, (Dn + an - -0,l)
mn (Dn + |nn - anl)
m, (Dn, + lan - axl)
Mn.10a
(p,)(mn)))
- ويg
+ 333
mn (D, + On - -voa)
mn (Dn + |An - -Ora))
mn (Dn + an - -a,l) + 33, mn (Dn + |an - -a.l)
N (Po.))
N2 (Pa.s)
+ 9).
mn (Dn + lan -a.l) -
On Ok.
for any 1 S j < L. Here, from the definition of ma. least one coefficient among (āz, By, a.j ( bile, =1-
{āj, 8g Tú. jjj B3, 93) 3:101/>1, and T is different from O. Then, the equation
1lQ,(X))
lim inf II ar, (X)-fa. (X)1l1 0 100
lim inf
exp( T X + bexsr) A
10 mn (Dn + Jan - -a+l) =
mn + |0, - (0a-
leads to
Σ exp((Bp..)) (Tp(BB,)))x)(aa + +33 (BTx) (B
|=1
exp((Bp..) x
+B
x)]
H
Σ exp( (Bp..) Tx)
x) (ā, + 8. (B x))
1=
Σ exp((Bp..)) T x aag + 3] (B x) + (B x) Ti, x fx)
>>1
+ + - tanh (a.))
Σ exp((Bp...)) Tx + banr)] Sc. (X) = 0s
#==1
for almost surely X. That equation only holds if and only if all the coefficients (o,, By, (,,,
)ne =1,
19

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
(o B, 75) (,))),)))),, 3:1111>11 and are O, which is contradiction.
It indicates that we indeed have the conclusion of the local part, namely.
inf The result of the local part implies that find a positive constant s' such that
lim 5 (OG -o- I Jo..o.leeta/(P(GG,G. -al)) 0.
Global part:
inf Therefore, to obtain the conclusion of the theorem it sufficient prove that
fac - ft..a.lll2a////(GG.
a.l) >
GE9
inf (e |aa. ll....llew/// la a.l) >
Assume by contrary that the above claim does not hold. Then there exists a sequence of measures G Σj'1 exp(bn.) )0p., in (e) and a', such that have
D(G G.) + lo,, - -all U.lecr,oo - Jo..o. (l2 (D(G) G.) + 10', -cal) - O.
These limits indicate that || fG. ||13())) + 0 as + cc.
Recall that the sets θ and are compact. Therefore, there exists a mixing measure G' in Gr. (r) such that one of (G,, Y's subsequences converges to (G", Since DGG,, G.) + la'n a,l > e', we deduce that D(G", G.) + la' - a.l > 6 An application of the Fatou's 's lemma leads to
0 lim Ilfar,,a, -fG,,a. 2 f lim inf fo of, (X) - fcc... (x)112 du(x).
n-+00
Hence, we have fa', (X) fa.... (X) for plalmost surely X. From the identifiability property (cf. the end of this proof), we deduce that (G', a') = (G), follows that D + lo a.l = O. It contradicts to the hypothesis that D(G", G.) + la' - a.l > E' > 0. As consequence, the proof of the global part completed. We obtain the conclusion of the theorem.
Proof for the identifiability property. We now demonstrate that fa. (X) = -a.... (X) for almost every X, then we obtain that (G, a) = @.).
To ease the presentation we denote the following notations:
exp(u)
softmax Pretmin
(u) N exp(X AQ + a0))
kml
exp(u)
softmax
Prompt (2')
EV exp(( TT X + by))
exp(u.)
Pretrain (u.)) softmax G.
Σn exp(X A + ag)
exp(u.)
softmax G.
Prommt (w,) E exp((((p)))))) TX by,y)
Prompt
Here, u, u', u. and u', in these equations satisfy:
M + a : [N]], v' E ((Bp)) T X by : E [L'}
u. E (X A + a5 :jE [N]]. u., ((Bp..)) T + b, : I [4]].
The equation fa. SG.... (X) for almost every X indicates that
softmax Sretmaxrettiin Pretrain (X 16x + (X)))(X mT) + tanh(a)
softmax Prompt ((Bp,)) TX + by.)) Cpp
20

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
softmax
Pretrain
X A X ( 73 + tanh(
softmax
((Pp.)) X + ) (29)
Prompi
C.
The above equation only holds when = L' Furthermore, wc also have that (softmax Prompt ((Bpy) ) X + by) 5 E [L]]- = {softmax Prompi
C. (BBp..') TX by.g)) : [4]}
for almost every X. By relabelling the indices, we can assume that softmax
Promp Promp ((Bp(((p))))))) X by.) softmax (BB. ((Bp.g") x by.j);
Prompt
for almost every X and any j' E [T]]. From the translation invariant property the softmax function, the above equations only hold when by ba.y' + F for some IR and any j' [L]. Given these results, equation (29) leads
L
tanh( (x)
Σ ==1
exp (by)) exp ((Bp)) X)Cp; = tanh(a.)
exp (bn.) exp ((Bp-s) X)CP...
(30)
for almost surely X.
Now, we partition the set {1, 21.2..... } into m subsets K1, K2...., Km where m S L. such that exp (b,) exp (br.j' ) for any j.j E K, and E [m]. is clear that exp (b,) * exp (b,,r) when j. belong to different subsets K, Collecting these results, equation (30) can be rewritten as follows:
(anh(a) Σ
exp (bx))ep exp ((Bp)) X)Cpj
JEK,
tanh(a.) Σ
exp (br,)) exp ((Bps)) X)Cp..>
jek.
for almost surely X. Hence, we achieve that
((Bp,)).P)): K,) (((Bp..)) .p=s) .p...) ij K.} and tanh(a) = tanh(ā.).
naturally leads to
(py :j E K.} (p+.s ;j K.).
Without loss of generality, P; -Pa. for all j E K,. As consequence, we obtain that ā ā. and
Σ
Σ exp (b,)6p,
Σ
Σ exp (6..))p..-
jEK;
1=1
jER.
It is equivalent to (G, = (G., &.). We achieve the conclusion of the identifiability claim.
A.2. Proof of Theorem 4.4
Similar to the proof of Theorem 4.2 in Appendix A.1, we only need to demonstrate that
l|9 Ic.,a, (1)) C. (D(G, G.) + la la-a.))
for any (G E (+) Ω for universal constant C. It equivalent proving that:
inf (e)x)
119G.a - IG..o. / G.) + la - a.l) 0.
To obtain the conclusion for the above inequality, we consider parts: (i) local part, namely,
lim
inf
ll9G,a - IG..a. (2 (D(G, G.) + la -a,l) 00
c
21

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
(ii) global part, namely, for any E >> 0
inf
-
9G..a.
0:
Since the global part can be argued in similar fashion to Appendix A.I, will focus only on proving the local part in this appendix. Additionally, we will impose the following essential yet mild assumptions in the activation function facilitate our arguments:
Assumptions. We assume that the activation function a meet the following assumptions:
(A.1) (Uniform Lipschitz) Let p( = exp( X)Co(p). Then, for any E {1,2), we have
alal F
Olal Op (X: (X: p)) & cllp - cllp-pp'1(1 p'11
Σ Fa|=r
(X:p)
0pm
for any vector E Rd and for some positive constants & and e which are independent of X and p'p Here, a e N4.
(A.2) (Injective) If there exist parameters p and p' such that opp) = opp'). then we obtain that p = p'.
Local part: We first start with the local part, which is equivalent demonstrating that
lim -
inf
119G, - IG..., -G..,, (1nm),,lll))))GGGG + +-a-).)) > 0.
We prove the above claim by contradiction. Assume by contrary that the above claim does not hold. It indicates that can find sequence of mixing measures Gn Σラ= exp(bn.3) )'p.. in (G) and sequence of an L such that when - co, the following limits hold:
D(G G.) + On a.l +0, { { 9G. (1)) (D G.) + lan axl) + O.
The first limit indicates that D, D(G,, G.) 0 and On -
00.
For the simplicity of the ensuing presentation, we denote сỤ = CG as a Voronoi cell of G, induced by the j-th components G.. Without loss of generality, we assume that those Voronoi cells do not depend on the sample size, i.e., Cj = Cy,, which is possible since our arguments are asymptotic. Therefore, we can rewrite the Voronoi loss D, as follows:
Dn
Σ iECy
exp(bn..) -xp(@n.)-exp(bo..)) exp(t
|'E(M iEg exp(br,)) Alpm..
Σ
Σ
y'=
Σ
Σ
cxp(6 A
J JELLHC,,lDiieE,
where Apn.iy' Pn.i P.,j' for all i Cy-.
From the hypothesis, we have Dn + 0, which implies that Eiec, exp(b...) - exp(b..j) and Pn.: - Paj for any i Cj,j [L]. Similar the proof of Theorem 4.2, to establish the contradiction our proof consists of three main steps.
Step 1 Taylor expansion.
To ease the presentation, let us denote
exp((Bo(Pnn))) X + bng)) Co(pma))
gG,, (X)
exp(( X -bnnk)
exp( + ba.y))
9G. (X) = 5
Co(p..))
Σ 1 (Bo( X + +b,x)
22

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
We now perform the following decomposition:
L
(x) =
exp((Boopp..:))) x + bbawe]] ls (X) - -G.... (X)]
(Bo((Bo()...)))) "X X + br.de]] - (tanhi(an)s,, (X) - tanhh(a))0 (gG.) (x))
exp( exp((Bo(po,:))) X + br Lanh(an) ga. (X) - 9G. (x)
X + bo [tanh(aa)) - tanli(a.)) (ga)). (X)
n (x) + (X),
For that purpose, we will decompose the two terms 2n, (X) and (x)) respectively.
Decomposition of the function (X). We have
(x)
tanh(an) exp(bn.) (pp((Boopn:))"x)(oo)) X)Co() exp((Bo(p_.))) X (O)
5=1 IEC
tanh(ax,) exp(bn.i) exp((Bo(pn..)) x) exp((Bo(p_.))) -xx((()))) x) gG,, (X)
j=1 iEC,
(anh(an)) (ak))
Σ
exp(bm.s) - exp(b-s)) exp((Bo(p..))) x) Co(p.s) 90., ga, (x)
iEC,
An (X)
(X) C.
(31)
Decomposition of the function (X). To ease the presentation, we define the following functions Ú(: == exp((Bo(p)) X) and V(p)- Cp(p). Then, we denote the product of these functions F(X:p) Ü( (p). To decompose A,, (X). we separately consider Voronoi cells with exactly one element and those with more than one element. It leads to the following decomposition of the function (X):
Án(X)= = Σ
Σ
tanh(o..) (x.()...) F(X: - p.oi]
=IC,
Σ
tanh(an) exp(bn.)) (an((a)))xp((m.)))
(m::: -F(::p.))
J:10;1>1 VEC,
Ān, (X) + 2x),
where we denote (x) = E3 E Eiec, tanh(ar) exp(bm.)) F(X: - F(X:P..) and Ānz(X) = Ejie Eiec, tanh(an) exp(bn.i) [F(X:pm.)) F(X:P..))
For the function An, (X), for any indices i Cj and such that |C,l 1, the first-order Taylor expansion entails that
Elal
((x:p.))=ū((x -((:p=.)) + Σ (Apn.)))
)
Op (X:p-s) + Rx),
la|=1
glal
V(p..)) = V(p.,)+ + Σ (Apn.s)) Op (px.) + Riy,21
|a|=1
23

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
where the terms R1 (X) and R2,,2 are Taylor remainders.
Combining the above results leads to the following formulation of the function (X):
tanh((.)) ) (x.p(bn.))
alalū (Apm.y)
(x)
Σ
Σ
(X (X:Po.)V(PP..
Opa
2:CC =1EE
aa/=1
glali
(Apn,ts) Op
(T,))((XE (X: + Rx,((X)
Gla Mu.ja Op (X: (X: + Mn.ja Opm (p.,) } + Rx,1(x)
Glai
J:C,/ =1 |a|=1
where the function ( the combination of Taylor remainders and satisfies that Rn, (X) + lan axl)
when n cc. Furthermore, the formulations of Mn.j.a are follows:
tanh(a,) (a,) exp(bn.t) (Apn.))
Mn.j.a Σ
ad!
IEC
for any |a| 1.
Moving to the function 2n,2(X) (), the second-order Taylor expansions of the function ÚX:; Pn.s) around the function
U(X (X; P..j) and the function V (Pn,s) around the function V(p.,) for any indices i Cj and such that |Cyl > 1, we obtain the following formulation of the function 2 (X):
a Mn.ja Op
alali (X: (ppr)))(pp.) + Mnsia Opa (p.,)((Xiiin))
2n,)(X)=
Σ
Σ
e: > 12/g/
Ula (X:P.s)
alAT Op (pas) + Rx,2(X)
Σ
Σ
Mnjaa
0p"
J1C11> jai =1
where the function Rr, (X) combination of Taylor remainders and satisfies (x)x(X)/(2)) + +aa, 1aan-aa -a-l) + co. Furthermore, we define
when
(anh(a;) exp(bnnt) (Apn.ss)
Mn.j.a
ad!
IEC
for any |a|- = and
tanh(a,)) exp(bnx) )
Mn.j...3
a!3!
iEC
for any |a| |8| 1.
From the formulations of the functions Ú(X:p) and Vp)), we obtain the following explicit forms of their partial derivatives:
aÙ Op(u)
(x:)) = exp((Bo(p)) x) B do 8p(u) (p) X,
(X;p) exp((Bo(()) x){ (B)p
(p) x x (B do
Oplu) (B (p)
(B do
Ep(") (D) T [x]
x
0p (1) Up()) (v)
ov
(p) = c da
ap(u) (p),
0p(u)
0p( (p) ā ga (p).
8²v
7pl))
Plugging these explicit formulations of the derivatives of the functions p(X;p) and (i),, we can express the functions
24

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
(x) and 2n, ()) as follows:
1x, E
exp((Bo(p..)) 3-1/1=1
L2,n (p- BT x) + R
An2(X)= = Σ [ (p)) + (pos) B
B X n, (p... ) B x] + (X),
where the functions Ci, (p), C2,, (p), L i (p), L (p), and L3, (p) are defined follows:
Σ Mn.j..1 C do
(p,)(p) =
Oplu) (p),
u=1
do
(2,((p)= =
Mn Mn 0p(u) Co)).
u=1
Mn C oo
Po
(p, Σ
Oplu)aplo) (p) Σ
Mn Mujila C
(p).
Opl")
Iguoodd
do (p)Co(p)+ +
Mandlend C do
(p) do
(2) Σ
Mudile 0p(u) u=1
Σ
Oplu) 8p(cj (p)
ISuvSd
8 Co())
+ Mu.iilue Dp 0p())p(())
do
do
(p)
E Igwood
Mn.j.wo Spli) (p)
Eplv) (p)) T
In these formulations, we use 1, to denote the vector that its u-th element is I and its other elements are 0 for S u S d. Furthermore, we denote L.v as the matrix that its (u, w)-th element 1 and its other elements are 0 for any S u, v S d.
Decomposition of the function (X). Similar to the decomposition of the function (X). we can decompose the function (X) as follows:
Ba(X) = E
tanh(an) exp(b..)) pm.s) - 9G, (X)
1-1,1-11 1 iEC,
+ Σ
tanh exp(bn.)) - gG. (X)
3:1C, |>1 1 NEC
= BB, (X) + B (X)
where we denote 1 Esic, Eiec, tanh(an) exp(bni)) ((::p..)) ((xp)))))))xxxxx)(p..)))))) Ú(:; (X:p..) E Eiec, tanh(an) exp(bm.i) (U(:: UX:: 9G, (X).
gG. (X) and (x,
Similar to the Taylor expansions for the functions (X) and 2n, by using the first-order Taylor expansion to (x,((X) and the second-order Taylor expansion (X), we obtain that
jlalū Op
Ba,1 (X) Σ
Maja
(X:P-j)gG. (X) (X)+
Rx,,(X);
5:CC,
Ma gla Ū Op
Ba,((X) = Σ
(X:Pos)gc,) (X) (X)
5:CC,
where the functions Rn (X), (X) are Taylor remainders. Furthermore, they satisfy that (x)x(X)((Dn)lll -a.l) +
25

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
O and (x)((X)/(2) la,, -0-, ax,)) when n co. Given the explieit formulations of the derivatives of the functions pp: and (p)). the functions 1,,1 and B, (x) can be then rewritten as follows:
(x) = Σ
exp((Bo(p...)) (i. Xga,
gG., (X) + Rx,a(X),
5:1c/==1
(x,
Σ
exp((Bo X) Ni, (P..)) (P..))BTXX B x
j:Cc/>2
+ (B X) (P. () (B)) (X) + (X).
Here, the functions (X), N1, (X), and n have the following formulations:
do (p),
(p)
Mn...l Op(")
a=1
do Mn.j.l Op(u)
8o
(p)
(p) + Σ Mug apl)) (p).
Isuned
do Magjaus Op(")
(p) Oo
(p) = Σ
0pl") (p)
Isuodd
Collecting all of the above results with the decomposition of the functions (X) and (x),), we can represent the function 2n,1(X) in equation (22) follows:
(x))
Σ
exp(
T [fi, (p..)) T x)
3:Cc/==1
+
Σ
exp(
[ (p..)) + L2, (p+.) B X + (B* (TT (a,((=.))BT x
#
cxp((Bo(p..))) X) M M....aa (N,(( T B X|gG.. gG.. (X) F
Σ
Σ
exp( X) [Mni.aa (pm T B X + (B" X N2. x ga, X|9a,
(X)
(X) 2x,)(X) Rx, (X))(X)
(32)
where we define Mnsjos = tanh(an) Eiec, exp(bm,d) - exp(bra)) for any index 1 s j S L, (pns)
(p.((P..)) + (o and Eg (P-j) = (P. + Mn,.0, Co(p..).
Decomposition of the function (X).. An application of the first-order Taylor expansion leads to the following expression for the function (X):
(x) [tanh(a.)) tanh(a.)] exp (Bp((Bo(pp.)))) X be. gc. (X)
= (an a.)(1 tanh2
exp((Bo(pp.x)) X + bo.k:) (X) + Rx,s(X),
(33)
where the function Rx,s(X) is Taylor remainder and satisfies that R, (D + lan - a++)) - as n 00.
Step 2 - Non-vanishing coefficients. From the results of equations (32) and (33), we can express 1x)
Σim R,, (x) (D,,,,,,, + aa, - -a,l) as combination of the linearly independent terms
exp((B((p.5)) X), (B x)()) T) x),
26

---

On Zero-Initialieed Attention: Optimal Prompt and Gating Factor Estimation
(B' x) ()(0 (B x x (xp((Bo(p..))) X), exp((Bo(p=.)) X)0c. 19GG,.A. (X), (B' () exp(( (o (p=,)) X )g. (X), (B' x) (B x) x)
X X)9G.. (X);
[1 - tanh (a+))
exp((Bo((tr.)) - bo,tr] 90. (X),
for any 1 S S L and S u, u
Our claim is that at least one of the coefficients of these linearly independent terms in the formulation of (X) E.=1 (X)) (X))((P, + aa, a does not go 0 + Assume by contrary that this claim does not hold, which means that all the coefficients of these linearly independent terms go to 0 as - oc. Therefore, as n we obtain that
(p,. // + Jan a-l) + 0, Ln, n (p+.) (w)) (D, + lan a.l) 0, L (p (D, + ann a.l) +0,, 0,
;p) (1) (D, + an a.l) + 0, L (p..) /00//() + |an - -a..) } - 0, L2 (po.)) () (w) (Dn + |a, - a. 1) - 0, La,n (P..j) (111) /(Dn + |a,, -a.)) - 0, Ni, / (D, + |an- -a.)) + 0, Ni,n (p..)) (w) (D,, + |a,, ax, 1) - 0,
N2, Pa.)) (wv) /(n, + [nn o.l) + 0, Mn Mn.. Dn -> 0, (an a.)/(D, + |nn - a.,) ->
for any 1 S u, v S and S L.
Since (an - (D, + Jan a. I), we obtain that
|a ax1 (Dn + an a.l)
O.
(34)
Furthermore, as an A 0 as n2 + oo, we have 1/ taahh(an) A cc. Given that Mn.j.0a/ (Dr + la, - axl) - 0,
demonstrates that
Eiec, exp(bm,i) - cxp(b..)l
Mn.,.0a
+ 0,
(Dn + Jan -a.l)
tanh(a.)
(Dn, + Jan - -a,l)
for any j S L. By varying the index from 1 to L in these limits and summing them up, we achieve that
I exp(Em.)) - exp(b..))l + 0.
(35)
(D, + an -a.l)
Now, we first consider indices j E [Z] such that its corresponding Voronoi cell C₁ satisfying |Cyl = 1. From the hypothesis, (2,n(p.,)) /D,, + lan - a.l) + Therefore, Mn..i./(D, + |an, - a.l)) - 0, which leads to
Eiec, exp(bm..) (l (Dn + lan ax1)
Σd JM tanh(a,) + 1ax aml)
-
The above limit indicates that
EH Eiec, exp(bn,))l (D, + an a
(36)
a.l)
Moving to indices E [Z] such that their corresponding Voronoi cells Cj satisfying |c,l > 1. The limit Ls,n (P. (mn) (D,, + Jon, a.l) + leads to
Eiec, exp(Un.i)llAP.ss))
Ed= La, (Po.) )(us)
tanh(a,) 1ax - anl)) + 0.
(Dn + lan a.l)
The above limit demonstrates that
Ejelss Eiec, exp(bn,)) Ap + 0.
(37)
(Dn + an a.l)
27

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Collecting all the limits in equations (34), (35), (36), and (37), we obtain that
Dn + |0tn Os.l D, + an, a.. as n + oC, which is contradiction. As a consequence, not all of the coefficients of the linearly independent terms in (x)(X) >; i=1 Rr, (D,, + aan - a.l) go to 0 as + co.
++ 0
Step 3- Applicatiion the Fatou's lemma. We denote m. as the maximum the absolute values of L1,,, (P=))/(D, + Ja,, - a.l), Cn, (P.))( (Dn + lan - a.l), f (P..)/(D, + la, a. L.. (P.)))))) ((D,, + la,, - a.l)) C3, (p.,)) (Dn + aan - (in)))),.p-)))((1, la, a. Ni, ((p,)))))))))))) (D, + Jan a.1)..N2,, (p+,s)
(Dn + (p.,)) (wv)
Jan, l,1), Mnj.ua/ (D, aan a.)), and (an (D,, a a.l) for all 1 I u, S d. From the result of Step 2 in the proof, we have 1/m,, A
+
Recall that l19Gn.0r - a [ (D, + la,, - a.l) + 0 as + oc, which indicates that l1g.n..n IG..o. (m)))/mnn(n + ann a. 1) 7 O. Furthermore, since the (²) norm is equivalent to the ('()) norm, we have llgG..an - gG. aGc.a.ll110 (1)) (m, (Dn + lan - aa))) + 0. An application of Fatou's 's lemma leads to
19G...o - aG.,a. ([(((M)
I19a,, (X) - IG..a. (x)1l1 dp(X) 2 0.
0 = lim
lim inf 10000
n-000
mn (Dn + lan -ail)
my (Dn, la,, a.l)
199G..an (X) IG..a (X)Il1 for almost surely X. As n - DO, we denote
indicates that lim inf, n000 mn (Dn + Ian a.l)
(Pon)
(p=(s))))
- -āz;
- 1z..
m, (D,, + an - -ail)
mn (Dn +aan axl))
ann
(1,n(Pa.)
L2 (pa.))
La,((pas) + 7/3
- ag:
+ +33
m, (Dn + an - -],l)
mn (Dn + |nn - anl)
m, (Dn, + lan - axl)
Mn.j0a
- و
+ 33
mn (D, + Onn - -voa)
mn (Dn + |An - -Ora))
mn (Dn + an - -a,l) + 33, mn (Dn + Jan - -a.l)
N (P..))
N2, (Pa.s)
+ 9).
mn (Dn + lan a.l) -
On Ck.
for any 1 S < L. Here, from the definition of ma. least one coefficient among (āz, Bj, ش.j (3 bile, =1-
{āj Bg Tú. jj B5, (3))):)003>11 and T is different from O. Then, the equation
(X)
Σ cxp((Bolp..:)) X + by lim inf
199. (X) - 9G.aa. (X)ll 0
lim inf --000 mn (Dn + ann -a.l)
mn (Dr, + |a,, - axl)
leads to
Σ
X)(a xB)
Σ
x) X) B x) (B x)]
|
Σ
exp( ex
X) 83 (B x)))e.(X)
=1
Σ
exp((Bo(p..)) [y + +3, (B X + (B x) "T x]9G. 1gg..12 (X)
>11,
+ + tanh?(a.))
Σ exp( Tx + ba 9G. (X) = =0a
for almost surely X. That equation only holds if and only if all the coefficients (o,, By,,B 3, )ne, =1-
28

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
{o, Og, B, Ty)::11..,. 3:11,1>11 and are O, which is contradiction.
It indicates that we indeed have the conclusion of the local part, namely.
lim 5-50 (D aa.))s 199,a
inf
gG..o. (ttpl//l////////G/ooooo G.) |a -aa)) 0.
As consequence, we obtain the conclusion of the theorem.
Proof for the identifiability property. We now demonstrate that if (X) n.... (X) for almost every X, then we obtain that (G, ā) = ā.).
To case the presentation we denote the following notations:
exp(u)
softmax& (u)
Pretrain
A== exp(X A +0))
exp(u)
softmax
Prompt (u')
E exp((Bo T TX + by)
exp(u.)
soflmax (u.) Σ exp(X x 0%))
Pretmii
exp(u.)
softmax G.
Prompt (u/)) EJ exp( T + b.gg))
Here, u', u. and u', in these equations satisfy:
E X T 1 x + a e [N]] u' E ((Bo(py)) X + by : E [L'}
Ue E {X A x + a je [N]] u E ((Bo(p-)))) + b i E [b]].
The equation 96.a (X) = 9G. R. X ) for almost every X indicates that
N
L'
softmax
Pretrain
(X
1 X + d))) + tanh(a)
Σ
softmax G
Prompl ((Bo(p;)) (( xx Co,)))))py))))
j'=1
softmax
Protrain
A X + a))))(XX, (3) + tanh(a.) (X)
softmax G.
Prompl
((Bo(pa,)))) x (o)
G.
(38)
The above equation only holds when 7, T/. Furthermore, also have that Prompt
{softmax taattmxxxroopppoo ((Bo(p;)) ((Bor X by): 5 [L']} (softmaxPr.. C. P +bb,)) j E [L]),
Prompt
((Bo(pa)))
for almost every X. By relabelling the indices, we can assume that
softmax Prompt (Bor(p;)) X + by) = softmax Prompt
Prompt
((Bo(p. + b.,y,),
for almost every X and any E [Z]. From the translation invariant property of the softmax function, the above equations only hold when by: = bugy + F for some F E IR and any j' E [L]. Given these results, equation (38) leads to
tanh(a) Σ exp (6,) exp ((Bo(p))) X)Co(p,) tanh(a.) Eexp exp (b_,))ep exp ((Bo(p..)) xx) (39)
for almost surely X.
Now, we partition the set {1, (1. L} into m subsets K1, Kg,..., .,, Km where m S L. such that exp (6,) = exp (b..y) for any j.j E K, and E [m]. It is clear that exp (b,) A exp (b..y) when j. belong to different subsets K, Collecting these
29

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
results, equation (30) can be rewritten follows:
tanh @
exp (6)) ((Bo(p)))"
jeK,
m
tanh(ā,) Σ
exp (b..,) exp ((Bo(p.,))) " ()Co()..))
jEk,
for almost surely X. Hence, we achieve that
{((BolD)) pj) K,} {((Bo()-))) :D-s) j Ki} and tanh(ā) tanh(a.).
naturally leads to
(o(p)) : jE K.) (o(D-)) K.).
Without loss of generality, P, = Pa.j for all j E K5. As consequence, we obtain that ā aa and
m
m
exp (6,)6p,
Σ Σ exp exp (6mr))ppp.
jER;
JER.
equivalent (G, a) = (.). We achieve the conclusion of the identifiability claim.
A.3. Proof of Proposition 4.1
Recall that the i.i.d sample (X,, V2),..., (X,, Vn) E Rdd x TR'" are generated from the model:
Ti=Ja.oo (X,) + =
where E,,...,En,
En are independent Gaussian noise variables such that X,, 0 and Var(s:[X) = = for all 1 S i n. Since X; (Oa,, Ta)), least square estimator (Gn, (.) defined
(Gns an)
arg min GEG, (0)
|| fa.a(x))lf
is exactly a maximum likelihood estimator given by
n
(Gn,an) an) E
arg max
(0)
where π (X.]),,,((x),33203 denotes the probability density function of the multivariate Gaussian distribution with mean vector fc,a (X) and covariance matrix Ig Furthermore, it follows from the result in (van de Geer, 2000) that
h(T(Y)fa., (x), o oTaa),), (Y (Y| (x), (X),
Or Mlg(n)//))
According to Pardo al. (Pardo, 2018), we have
det(21) 1/4 det(222)
(0) 8,) }))) (0,
-1 (0) 0a)
h22 Ei), =
1/2 exp
det 1 Σ1 5522
Therefore, we deduce that
1
(T( xx)
(x)) =
-exp
80 18 (X) Sc. (x)112
As result, it follows that
{ - llli (X) fa. (x)112 =Op(log(n\/)).
1
exp
30

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
This means that with probability one, there exists some constant C and natural number ni such that
1 C'loo(nn)/nn S exp {
1
802 80 fl (x) aa..a_ (X)]2 (x)11)
Assume that the natural number no is large enough so that 1 Clog(n)/n 2 1/2 is true for all no- Then, the above inequality is equivalent to
a. (X) - fa..a. (x)1l2 & 80 log
- (l-Clo(()//m)
C Clog(n)//n
So log
1
(1+
1-olog(n)/n
C Clog(n)//
8
1 Cllog(n)/n
S 16a Clo(mnn,nn)/n,,
which implics that
1 San.a. (X)1l 1 00pllo==p((//////// Ilo(n)//))
Consequently, have
(f 1I (X) (X) = =Or( (r((/lo(n)/n))
1s,., Ja.
Hence, the proof is completed.
B. Additional Related Works
Theory of Mixture of Experts.
While there has been a surge of interest in using Mixture of Experts (MoE) to scale model capacity in large-scale systems, its theoretical foundations remain underdeveloped. From probabilistic perspective, a series of works on Gaussian MoE (Ho et al., 2022; Nguyen et al., 2023; 2024c; Yan et al., 2025) have analyzed the convergence behavior of maximum likelihood estimation, providing insights into optimal expert structures for MoE training. In a related line, (Nguyen et al., 2024f) studied a regression framework where the true regression function follows an MoE form, showing that sigmoid gating yields faster convergence rates for expert estimation compared to softmax gating. Theoretical analyses of MoE for classification tasks were further developed in (Chen al., 2022; Nguyen al., 2024d), where each expert modeled as classifier.
In the context of deep learning, Han et al. (2024) and Nguyen et al. (2024e) incorporated MoE and its hierarchical variants into multi-modal learning, training each expert to specialize in processing specific data modalities such as time series, text, or images. Pham et al. (2024) proposed a competition-basse bassed routing strategy to effectively train sparse MoE models for language modeling, supported by theoretical justification. Additional theoretical insights into MoE have been explored for domain adaptation (Nguyen al., 2025) and continual learning (Li al., 2024b). However, the best of our knowledge, the theoretical analysis of MoE in the context of zero-initialized attention remains unexplored.
C. Additional Experimental Details
C.1. Datasets Description
We first fine-tune LLaMA-Adaphe Adapter with all prefix-tuning settings on Alpaca dataset, which include 52K instruction-following data for training. Then, we evaluate our experiments on LLM benchmarks, including A12 Reasoning Challenge (ARC), HellaSwag, MMLU, and TruthfulQA. The statistics of 4 datasets about testing subset are summarized detail Table 4.
ARC dataset is a multiple-chen question-answering dataset which contain science questions in exams from grade to grade 9. It has two types: Easy and Challenge. In this paper, we report performance on both types and average accuracy.
31

---

On Zero-Initialiedd Attention: Optimal Prompt and Gating Factor Estimation
HellaSwag provides multiple-choice questions evaluate commonsense NLI of LLMs. Given paragraph which is incomplete, the model need find the suitable option complete it.
MMLU is benchmark that covers 57 subjects across STEM, the social sciences, humanities, and more through multiple-choice questions. This dataset test the model on both world knowledge and problem solving ability. Its subjects range from traditional areas such as mathematics to more specialized areas like law and ethics.
TruthfulQA measure whether LLMs is truthful in generating answers given questions. It comprises 817 questions with each question has two types, generative questions and multiple-choice questions. this paper, we evaluate all settings multiple-choice questions.
Table 4. Statistics of 4 LLM benchmarks about the testing subset.
ARC (Easy)
ARC (Challenge)
HellaSwag
MMLU
TruthfulQA
Testing
2376
1172
10042
14042
817
C.2. Prompt Templates
We also provide the prompt templates that use for all settings to evaluate on ARC, MMLU, and TruthfulQA benchmarks in Figure 4, 5, and 6, respectively. These templates are based on the fact that Alpaca Fully Fine-tuning and LLaMt-ddppter both use this prompt template structure in training and we custom little bit for each dataset.
ARC Prompt Template
instruct desoribes Nrite response appropriately completes the milzipla-chcice quastion.
Instruction:
Question:
<question>
optionss
Opt 1
Option 2
Option ln 3
Option
Response:
Figure 4. Prompt templates for ARC dataset
MMLU Promps Template
Re low ta an trstruct ion that descr tes cask. Write a responae that appropriatelly completes the saltippee-hoioe question about (taak),
#+* Instruction:
Question:
<quast'
Options:
Option 1
Optioo 2
Option
Option 4
### Response:
Figure 5. Prompt template for MMLU datasct.
32

---

On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
Table 5. Visualize some question-amswering outputs with different prompt-tuning methods.
Which best explains scientise
Question:
Question: pitcher throws a kg baseball at
Question:
Part of the coast of South Ameri and the west coast of Africa have matching fossils within the same series of rock layers. This provides evidence that these two continenta were once:
are referring
towards catcher. What is the momentum of the baseball while moving at 40 m/s?
they use the term
conservation?
(a) nonliving parts of
the environment.
(a) 0.025 kg x m/s.
(b) living organt sms in
(b) 3.8 kg x m/s.
the environment.
(c) 6.0 kg m/s.
(c) health of the living organisms the
(d) 270 kg m/s.
(a) separated by much
larger
environment.
(b) joined together
(d) protection, management, and renewal of resources.
landmass.
(c) located near the
North Pole.
(d) in a fifrennnt
hemisphere.
Ground Truth: d,
Ground Truth: C-
Ground
Non-Linear prompt:
Non-Linear prompt: 6.0 kg m/s
Non-Linear prompt:
The term conservation refers to the protection, management, and renewal of resources.
joined together one landmass
Linear prompt:
Linear prompt: 3,8 kg m/s
Linear prompt:
protection, management, and renewal
The two continents once joined together one landmass.
of resources.
Random-Inite
Random-Inite
Random-Inie
protection, management, and renewal of resources.
The momentum of the buseball while moving at 40 m/s is 0.025 kg m/s.
located near the North Pole.
TruiluiQA Prompi Template
instruction that describes cask. Nrite response that appropriatell completes the milciple-chotoe question about (taak).
Below is
#+t Instruction:
Intergret each muitiple-choice question literally, and as a question about the
world; carafully to sae the question misses and research each option and only pick option, without falling prey to any corman mythss.
suitable
Question:
squeation>
Options:
Option 1
Option 2
Opt 3
Option 4
* Response:
Figure 6, Prompt templates for TruthfulQA datasets.
C.3. Visualize question-answering
We provide in Table 5 illustrations on output of LLaMA-Adapter trained with different prompt-tuning strategy.
33


Reviews:
[{"review_id": "hCJWqonggP", "reviewer": "Reviewer_oCcc", "summary": "This paper investigates a specific aspect of LLaMA-Adapter, focusing on zero-initialized attention. The zero-initialized attention mechanism is not only initialized with zero values but also involves a structural modification that replaces the traditional softmax function. Instead, softmax is computed independently over two components: the input tokens  $X_l$ and the learnable adaptation prompt $P_l$, i.e., output softmax equals to $S_g=[{softmax}(S_p) \\cdot {tanh}(\\alpha), {softmax}(S_X)]$. The authors demonstrate that this zero-initialized attention can be interpreted as a specialized form of a mixture of experts (MoE). Building on this insight, they also prove that non-linear prompts can offer advantages over linear prompts in the context of zero-initialized attention. Extensive experiments were conducted, and the results were validated.\n\n## update after rebuttal\n\nAlthough the rebuttal responses are satisfactory, I acknowledge that the issues raised necessitate significant revisions to the manuscript’s writing and presentation. As such, I will retain my current evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. We hope that we can address your concerns with the responses below.\n\n### **Q1: Alignment of Theory and Experiments:**\nThanks for your comments. We would like to clarify that the convergence analysis of prompt estimation under the random-initialized attention has been conducted in prior work (see Appendix C in [1] or Appendix A in [2]). Thus, in response to your concern, we will include the following comparison in the revision of our manuscript (below Theorem 4.2), indicating that **using the zero-initialized attention is more sample efficient than using the random-initialized attention**: \n\n*1. Prompt convergence in random-initialized attention [1,2]:* The convergence rates of prompt estimation are significantly slow, standing at the order of $O(1/\\log^{\\tau}(n))$ for some constant $\\tau>0$, where $n$ is the sample size. Thus, to approximate prompts with a given error $\\epsilon$, we need exponentially many data points $O(\\exp(\\epsilon^{-1/\\tau}))$, which is not sample efficient.\n\n*2. Prompt convergence in zero-initialized attention (Ours):* As shown in Theorem 4.2 and Theorem 5.2 in our manuscript, the convergence rates of prompt estimation are of polynomial orders, ranging from $O(n^{-1/2})$ to $O(n^{-1/4})$. Therefore, we only need polynomially many data points to approximate the prompts with a given error $\\epsilon$.\n\n**Hence, in the experiments, we conduct comparisons between *Zero-init* and *random-initialized attention* to validate the above theoretical analytic as presented in Table 1.** Therefore, we believe that our work consistently aligns with our theoretical results.\n\n[1] P. Akbarian et al. Quadratic gating functions in mixture of experts: A statistical insight. arXiv preprint, 2024. \n\n[2] M. Le et al. Mixture of experts meets prompt-based continual learning. Advances in NeurIPS, 2024.\n\n### **Q2: Discrepancy of LLaMA-Adapter on LLM benchmarks and ScienceQA (Multi-Modal):**\nThe discrepancy in performance improvement between our paper and the original study on LLaMA-Adapter can be attributed to **the differences in the datasets and task settings**. The original paper only conduct ablation study to compare Zero-initialized Attention to Random-Init setting (without the zero-initialized mechanism) on the ScienceQA dataset, which is a **multi-modal dataset used for Visual Question Answering**. In contrast, we extend more experiments on **language-only tasks**, specifically fine-tuning on the Alpaca dataset  **based on the original code of  LLaMA-Adapter paper**. The **Reviewer XuJQ** also mention that in these language-only tasks, **the performance gain in Table 1 is notable**. This **fundamental difference in task type** can influence how the zero-initialized mechanism impacts model performance.\n\nMoreover, ScienceQA consists of 21,208 questions, making model convergence slow and unstable when the zero-initialized mechanism is not applied. This issue is **highlighted in Figure 7 of the original paper**, where models **without zero initialization struggle with robustness and efficiency**. The multi-modal nature of ScienceQA, which involves both visual and textual inputs, further complicates the training process, making stabilization techniques like Zero-Init more beneficial.\n\nFor language-only tasks, we fine-tuned the Random-Init setting (without the zero-initialized mechanism) on the Alpaca dataset, which contains 52,000 samples. Although the Random-Init setting led to **unrobust convergence in sample efficiency, as shown in Figure 2 and 3 in our paper**, the final convergence values when fine-tune the Random-Init setting on 100% Alpaca dataset remained **sufficiently low compared to the Zero-Init setting**. This indicates that when fine-tuning a random-initialized model on a sufficiently large language-only dataset, the model can still achieve reasonable stability despite the slower convergence.\n\nOverall, the smaller performance gap observed in our study compared to the original paper can be explained by the **difference in dataset size and modality**. While Zero-Init plays a crucial role in stabilizing multi-modal training on ScienceQA, its impact is **less pronounced in large-scale language-only fine-tuning, where instability can be mitigated to some extent by the sheer volume of training data**.\n\n### **Q3: Fragmentation of Sections:**\nThanks for your suggestion. We agree that both Section 4 and Section 5 focus on the convergence analysis of prompt estimation. Therefore, in the revision of our manuscript, we will merge them into one big section by relabeling Subsection 5.1 and Subsection 5.2 as Subsection 4.2 and Subsection 4.3, respectively.", "claims_and_evidence": "While the paper presents several claims, I believe the main arguments can be summarized as follows:\n\n**(Theoretical Claim)** Zero-initialized attention can be interpreted as a mixture of experts (MoE), and this interpretation suggests that non-linear prompts are more suitable than linear prompts.\n\n**(Empirical Claim)** Zero-initialized attention outperforms random-initialized attention (that is traditional attention), and non-linear prompts outperform linear prompts for zero-initialized attention.\n\nBoth claims are valuable contributions to the field, and I find sufficient supporting evidence for each. As a result, I am inclined to accept the paper; however, I have some minor suggestions.\n\n**Suggestions:**\n1) Alignment of Theory and Experiments: Additionally, the proof in the paper does not directly claim that zero-initialized attention is superior to random-initialized attention. As I understand it, the primary argument is that zero-initialized attention resembles MoE. Given this, I am uncertain if the experiments in Table 1, which compare zero-initialized and random-initialized attention, are the most appropriate. Overall, the theoretical argument revolves around the similarity between zero-initialized attention and MoE, and the benefits of non-linear prompts. However, the experimental results seem to focus on comparing zero-initialized attention with random-initialized attention. I believe it would strengthen the paper if the theory and experiments were more closely aligned.\nMoreover, regarding Table 1, the results presented are indeed similar to those found in the original paper's \"Table 5: Effectiveness of Zero-initialized Attention in our method,\" which specifically covers the ScienceQA dataset. However, the performance improvement in this paper does not seem to be as significant as in LLaMA-Adapter. I am curious if there is a specific reason for this discrepancy.\n\n2) Fragmentation of Sections: The structure of the paper feels somewhat fragmented. In particular, Sections 4 and 5 appear to share similar content and proofs. A possible reorganization could involve grouping these under a common heading, such as \"Linear and Non-linear Prompts.\"", "methods_and_evaluation_criteria": "Refer to Claims And Evidence.", "theoretical_claims": "Although I haven't examined all the proofs, there don't seem to be any major issues.", "experimental_designs_or_analyses": "Refer to Claims And Evidence.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Refer to Claims And Evidence.", "other_comments_or_suggestions": "Refer to Claims And Evidence.", "questions_for_authors": "Refer to Claims And Evidence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9BsDGRJPf7", "reviewer": "Reviewer_XuJQ", "summary": "The paper provides a rigorous theoretical foundation for zero-initialized attention, which has been successfully used in fine-tuning large language models (LLMs), particularly in LLaMA-Adapter. Establishes a connection between zero-initialized attention and mixture-of-experts (MoE) models. It additionally proves that both linear and non-linear prompts, along with gating functions, can be optimally estimated.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and insightful comments. We would like to address your concerns as follows:\n### **Q1: Explanation for results in Table 2.**\nThank you for your comments. **We want to clarify that most PEFT methods (e.g., LLaMA-7B + zero-init + linear, LLaMA-7B + zero-init + non-linear) perform worse than fully fine-tuning of LLaMA-7B, as shown in Table 2 (e.g., ARC dataset)**. While PEFT sometimes matches full fine-tuning, this is likely because the pre-trained model already contains fundamental knowledge for certain downstream tasks. However, in general, PEFT methods lag behind in performance compared to full fine-tuning.\n\nHowever, it's important to emphasize that fully fine-tuning models like LLaMA is computationally expensive, making it impractical for low-resource settings. PEFT significantly reduces the number of learnable parameters while achieving comparable results, offering a more efficient alternative in such environments.\n\n### **Q2: Additional analysis on other Parameter-Efficient Fine-Tuning (PEFT) techniques.**\nThank you for your suggestion. To further validate our method, we conducted additional experiments comparing it with other PEFT methods, including **Prompt Tuning [1]**, **IA3 [2]**, and **VeRA (r=128 and applied to the same modules as LoRA) [3]**, on LLaMA-7B setting. These comparisons provide a broader perspective on how our approach performs relative to established fine-tuning techniques. Additionally, we would like to clarify that the **name Random-Init prompt in our study follows the traditional prefix-tuning approach**. The results is presented in table below.\n|Method|ARC (eas)|ARC (cha)|ARC (aver)|MMLU|Hellaswag|TruthfullQA|Average|\n|-|-|-|-|-|-|-|-|\n|LLaMA-7B, Fully Fine-tuning Alpaca|67.47|46.25|56.86|37.25|77.09|42.35|53.39|\n|LLaMA-7B, LoRA Alpaca|61.91|42.15|52.03|34.87|**77.53**|**46.14**|52.64|\n|LLaMA-7B, Prompt Tuning|55.35|37.46|46.41|32.85|75.88|34.76|47.48|\n|LLaMA-7B, IA3|52.06|35.92|43.99|31.65|75.73|32.8|46.04|\n|LLaMA-7B, VeRA|49.2|35.49|42.35|30.88|75.59|31.95|45.19|\n|LLaMA-7B, Prefix-Tune (Random-Init)|60.65|40.7|50.67|35.12|72.62|37.82|49.06|\n|LLaMA-7B + zero-init + linear|62.29|43.17|52.73|36.28|76.79|45.53|52.83|\n|LLaMA-7B + zero-init + non-linear|**63.51**|**45.39**|**54.45**|**36.95**|76.67|45.04|**53.28**|\n\nFrom our results, we observe that except Fully Fine-tuning setting, **the Zero-Init approach combined with a non-linear prompt consistently outperforms other PEFT methods overall**. This further point out the effectiveness of our method in achieving stability compared to traditional fine-tuning techniques.\n\n[1] The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP, 2021\n\n[2] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS, 2022\n\n[3] VeRA: Vector-based Random Matrix Adaptation. ICLR, 2024\n### **Q3: The paper overlooks the Switch Transformer model.**\nThanks for your comment. There seems to be a misunderstanding about the contributions of the paper. **Our main focus is to study the LLaMA-Adapter for parameter-efficiently fine-tuning the LLaMA models rather than efficiently scaling LLMs with MoE models.**\n\nIn particular, we study the LLaMA-Adapter, a PEFT method for LLaMA models. Since the LLaMA models do not replace feed-forward network layers with sparse MoE layers in their Transformer architecture, we do not consider any MoE-based Transformer variants, including the Switch Transformer. Instead, we establish a connection between the zero-initialized attention and MoE models in Section 3. We show that the zero-initialized attention can be represented as an MoE model. From that perspective, we demonstrate that using the zero-initialized attention with either linear prompts or non-linear prompts is more sample efficient than using the random-initialized attention (traditional attention).\n### **Q4: The novelty of non-linear prompts.**\nThanks for your comment. We would like to emphasize that our main contribution is to provide a theoretical study for understanding the benefits of the zero-initialized attention over the random-initialized attention based on their connection to MoE models. To the best of our knowledge, there had not been any similar studies in the literature prior to our work. \n\nFurthermore, our analysis indicates that in addition to linear prompts in the original LLaMA-Adapter, non-linear prompts can be also optimally estimated with greater flexibility. Therefore, we perform several experiments to justify the efficacy of the LLaMA-Adapter with non-linear prompts. Although the idea of using non-linear prompts in PEFT methods may not be new, **our paper is the very first work to propose employing non-linear prompts in the LLaMA-Adapter to enhance its performance with both theoretical guarantee and empirical evidence.**", "claims_and_evidence": "The claims in the submission are largely supported by clear and convincing evidence, as the paper provides both theoretical justifications and empirical validations for its main arguments. The connection between zero-initialized attention and the mixture-of-experts (MoE) framework is rigorously established through mathematical derivations, and the statistical benefits of optimal prompt and gating factor estimation are backed by well-defined regression-based analyses. The experimental results on open LLM benchmarks align with the theoretical findings, demonstrating improved performance of zero-initialized attention over random initialization and highlighting the advantages of non-linear prompts.", "methods_and_evaluation_criteria": "The paper effectively justifies its focus on zero-initialized attention by connecting it to the mixture-of-experts (MoE) framework and demonstrating its theoretical advantages in prompt estimation. The evaluation is conducted on widely recognized open LLM benchmarks, including AI2 Reasoning Challenge (ARC), HellaSwag, MMLU, and TruthfulQA, which are appropriate for assessing the model’s ability to follow instructions, reason, and generate accurate responses. The choice of LLaMA-7B and LLaMA-13B as base models is also reasonable, as they represent strong open-source LLMs used in real-world applications. Furthermore, the paper compares zero-initialized attention against conventional random-initialized attention and other fine-tuning strategies such as LoRA and full fine-tuning, providing a comprehensive evaluation. However, while the experimental setup is robust, additional analysis on different model scales or alternative PEFT techniques could further strengthen the generalizability of the findings.", "theoretical_claims": "The proofs for Theorems 4.2 and 5.2, which demonstrate the optimality of prompt and gating factor estimation, follow a logical structure, using techniques such as Voronoi loss functions and parametric convergence analysis.", "experimental_designs_or_analyses": "The performance gain is notable, with a clear highlight in Table 1.\n\nIt is strange that LLaMA-7B + zero-init + linear, LLaMA-7B + zero-init + non-linear is worse than LLaMA-7B, Fully Fine-tuning Alpaca. Usually PeFT is much worse than full fine-tuning. Is the full fine-tuned model well-tuned (and overfitted), and what is the trade-off here?", "relation_to_broader_scientific_literature": "By establishing a theoretical connection between zero-initialized attention and MoE models, the paper builds upon the foundational principles of MoE, where a gating mechanism dynamically selects a subset of experts (or parameters) for each input, thereby improving computational efficiency and model capacity. This approach resonates with recent advancements in sparsely-gated MoE layers, which have been employed in scaling transformer models efficiently by activating only pertinent subsets of parameters during processing. Also, the exploration of both linear and non-linear prompts in the context of zero-initialized attention extends the current understanding of prompt-based learning and its integration with attention mechanisms, offering a nuanced perspective on optimizing prompt and gating factor estimation in LLM fine-tuning.", "essential_references_not_discussed": "the paper overlooks the Switch Transformer model, which employs a simplified MoE approach to achieve efficient scaling", "other_strengths_and_weaknesses": "One strength from the theory side is that the paper rigorously establishes a connection between zero-initialized attention and the mixture-of-experts (MoE) framework, providing a solid mathematical foundation for understanding its benefits in prompt tuning. By leveraging regression-based analysis and deriving optimal estimation rates for both linear and non-linear prompts, the paper offers a theoretical justification for why zero-initialized attention improves sample efficiency and stability in parameter-efficient fine-tuning.", "other_comments_or_suggestions": "​The paper demonstrates strengths in its theoretical analysis and empirical validation of zero-initialized attention within large language models (LLMs). By establishing a connection between zero-initialized attention and mixture-of-experts (MoE) models, it offers a novel perspective that could inform future research in parameter-efficient fine-tuning. The empirical results, showing improved performance with zero-initialized attention and the effectiveness of non-linear prompts, add practical value to the theoretical insights. However, the paper's originality is somewhat tempered by its reliance on and builds upon PeFT approaches with on-linear prompt structures, which is not entirely new.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O5z9YqJZun", "reviewer": "Reviewer_jpno", "summary": "This paper provides extensive theoretical analysis for the zero-initialized attention in LLaMA-Adapters and connects it with mixture-of-experts models. Based on this, the author introduces non-linear prompts. Plus, experiments are conducted to demonstrate the performance of non-linear prompts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and insightful comments. We hope that we can address your concerns with the responses below.\n\n### **Q1: Comparison between Non-Linear and Linear prompt on HellaSwag and TruthfullQA with LLaMA-7B setting:**\nThank you for your comments. Our main study in this paper is to **focus on providing detailed theoretical analysis and experiments to understand the benefits of the zero-initialized attention over the random-initialized attention (conventional prefix-tuning approach)** based on their connection to MoE models. Additionally, our analysis also indicates that **non-linear prompts can be also optimally estimated as linear prompt with greater flexibility**, suggesting that non-linear prompts are potential alternative to linear prompts.  \n\nTo justify the potentials of non-linear prompts for LLaMA-Adapter models, we perform several experiments to compare the performance when using the linear prompts versus when using non-linear prompts. From these experimental results, we observe that non-linear prompts achieve **higher performance in most settings (ranging from 0.5% to 4%)**, specifically across **all results on LLaMA-13B and the first two datasets on LLaMA-7B**. However, there are certain cases where non-linear prompts yield only **comparable results**, with performance differences **ranging from 0.1% to 0.5% lower than the linear prompt settings**. \n\nThis variation is expected because no single prompting method universally excels across all tasks and models. The effectiveness of non-linear prompts can depend on several factors, such as the dataset characteristics, model capacity, and the complexity of task-specific adaptations. In some cases, the additional expressivity of non-linear prompts may not be necessary, leading to performance that closely approximates linear prompts. \n\n### **Q2: Comparison among several initialization methods:**\nThank you for your suggestion. Indeed, we would like to clarify that the Random and Zero Initializations in the manuscript do not refer to the initialization of some vector. Let us take this opportunity to explain each of these initializations. \n\nFirst of all, **Random Initialization (equivalently, Random-Init) follows the conventional prefix-tuning approach**, where traditional attention is applied to all tokens, including prompt vectors and previous output tokens, without incorporating the zero-initialization mechanism introduced in the original paper. The term \"Random-Init\" is used because, at the initial stage, all prompt vectors are randomly initialized. When combined with traditional attention mechanisms, this randomness affects the model’s convergence robustness. **The terms \"Random-Init\" and \"Zero-Init\" are also used in the original LLaMA-Adapter paper**.\n\nOn the contrary, **Zero Initialization (in short, Zero-Init) setting introduces a learnable gating factor within the attention layers that use prompt, which is initially set to zero**. Then, tanh activation function is applied to this factor to regulate the scale of this factor into [−1,1]. Additionally, separate softmax operations are applied to the attention scores of prompt tokens and word tokens independently, after which the gating factor is applied to the attention score of prompt tokens. This mechanism, as detailed in Equation (7) of the original LLaMA-Adapter paper, plays a crucial role in controlling the contribution of prompt tokens to the overall attention mechanism. **By initializing this gating factor at zero**, this factor can first eliminate the influence of under-fitted prompts at the early stages of training, allowing the model to gradually adjust its magnitude to incorporate meaningful instruction semantics into LLaMA. \n\nFinally, in our study, we provide a detailed theoretical analysis and experiments to understand the benefits of zero-initialized attention over the random-initialized attention (conventional prefix-tuning approach). We also provide a theoretical analysis and experiments to show the effectiveness and flexibility of the Non-Linear prompt combined with the zero-initialized mechanism. \n\nWe hope our response answers your question about the initialization. Otherwise, please feel free to let us know, we are more than happy to address your further concerns.", "claims_and_evidence": "The claims of the article are consistent with the content he provided. The structure in attention can indeed be regarded as a mixture-of-experts structure (at least similar in form), and the experiments have confirmed the superiority of the method proposed in the paper.", "methods_and_evaluation_criteria": "Yes. This article focuses on treating the zero-initialized attention in LLaMA-Adapters as a mixture-of-experts model and optimizing LLaMA-Adapters using nonlinear prompts. In the article, the author provides the connection between mixture-of-experts and attention and gives proof.", "theoretical_claims": "I have partially checked the relevant derivations and there should be no problem. However, if there are any more questions, I will add them later.", "experimental_designs_or_analyses": "Regarding the experimental part, the experiments on most data sets have verified that zero-init has better results than random-init, and non-linear has greater potential than linear, which to a certain extent can explain the superiority of the method proposed in the paper. However, in the experiments of HellaSwag and LLaMA-7B in TruthfullQA, it seems that non-linear prompts are worse than linear prompts. Can the author explain this part of the problem?", "relation_to_broader_scientific_literature": "The more important finding of the article may be that the introduction of non-linear prompts to LLaMA-Adapters provides better performance, which can inspire new prompt tuning.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "I feel that the author could compare several initialization methods, such as all 1, or other forms of random initialization, and discuss the sparsity of initialization. Intuitively, I feel that sparsity may be a more important point. For example, most of the values are 0 but a small number of values exist.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the theoretical understanding of zero-initialized attention in LLaMA-Adapter and its connection to mixture-of-expert models, and demonstrates the statistical benefits of zero-initialized attention over conventional ones. It also argues that non-linear prompts offer greater flexibility for a broader range of applications and shows that non-linear prompts outperforms the linear ones.

The reviewers agreed upon positive ratings. Overall I recommend accept.
Decision: Accept (poster)