Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts
Authors: Minh Le
Chau Nguyen
Huy Nguyen
Quyen Tran
Trung Le
Nhat Ho
Venue: ICLR 2025 Poster (2025)
Keywords: prefix-tuning
mixture of experts
theory
prompt
reparameterization
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prompt-based techniques, such as prompt-tuning and prefix-tuning, have gained prominence for their efficiency in fine-tuning large pre-trained models. Despite their widespread adoption, the theoretical foundations of these methods remain limited. For instance, in prefix-tuning, we observe that a key factor in achieving performance parity with full fine-tuning lies in the reparameterization strategy. However, the theoretical principles underpinning the effectiveness of this approach have yet to be thoroughly examined. Our study demonstrates that reparameterization is not merely an engineering trick but is grounded in deep theoretical foundations. Specifically, we show that the reparameterization strategy implicitly encodes a shared structure between prefix key and value vectors. Building on recent insights into the connection between prefix-tuning and mixture of experts models, we further illustrate that this shared structure significantly improves sample efficiency in parameter estimation compared to non-shared alternatives. The effectiveness of prefix-tuning across diverse tasks is empirically confirmed to be enhanced by the shared structure, through extensive experiments in both visual and language domains. Additionally, we uncover similar structural benefits in prompt-tuning, offering new perspectives on its success. Our findings provide theoretical and empirical contributions, advancing the understanding of prompt-based methods and their underlying mechanisms.

Published as a conference paper at ICLR 2025
REVISITING PREFI--UNING: STATISTICAL BENEFITS OF REPARAMETERIZATION AMONG PROMPTS
Minh Le Chau Nguyen"), Huy Nguyen"?, Quyen Tran', Trung Le", Nhat Ho
Qualcomm Al Research*
2 The University of Texas at Austin Monash University
ABSTRACT
Prompt-based techniques, such as prompt-tuning and prefix-tuning, have gained prominence for their efficiency in fine-tuning large pre-trained models. Despite their widespread adoption, the theoretical foundations of these methods remain limited. For instance, in prefix-tuning we observe that key factor in achiev- ing performance parity with full fine-tuning lies in the reparameterization strat- egy. However, the theoretical principles underpinning the effectiveness of this ap- proach have yet to be thoroughly examined. Our study demonstrates that reparam- eterization is not merely an engineering trick but is grounded in deep theoretical foundations. Specifically, we show that the reparameterization strategy implic- itly encodes shared structure between prefix key and value vectors. Building on recent insights into the connection between prefix-tuning and mixture of ex- perts models, we further illustrate that this shared structure significantly improves sample efficiency in parameter estimation compared to non-shared alternatives. The effectiveness of prefix-tuning across diverse tasks is empirically confirmed to enhanced by the shared structure, through extensive experiments in both vi- sual and language domains. Additionally, uncover similar structural benefits in prompt-tuning, offering new perspectives on its success. Our findings provide theoretical and empirical contributions, advancing the understanding of prompt- based methods and their underlying mechanisms.
INTRODUCTION
The rapid growth in data availability, along with advances in computational power and training algorithms, has driven the development of numerous foundational models that achieve impressive results across wide range of tasks (Kaplan et al.) 2020 Rae et al. 2021 Dosovitskiy al. 2021) Leveraging these models' strong generalization abilities, fine-tuning them for downstream tasks has become widely adopted and successful approach (Iotinova al. 2022). However, full fine- tuning involves updating all model parameters, demanding storage for separate models per task, which becomes computationally and memory-intensive. especially with models containing billions
of parameters (Dosovitskiy et al. 2021 Dehghani et al. 2023 Lialin et al.) 2023).
To address these limitations, parameter-effiiien fine- tuning (PEFT) has emerged as promising alternative (Hu et al.) 2021 Lian et al.) [022] Xin et al. 2024). By updating only small subset of parameters, PEFT can achieve performance comparable to, or even surpassing, that of full fine- tuning while significantly reducing computational and memory overhead (Houlsby et al. 2019 Jia
ct al. 2022). Among these, prompting (Lester et al. 202T Li & Liang 2021 Jia et al., 2022) is gaining momentum promising solution by updating aesc-specific tokens while keeping the pre- trained transformer model frozen. Specifically, Lester et al. (202T) introduced trainable continuous embeddings, continuous prompts, which appended the original sequence of input word embeddings, with only these prompts being updated during training. Extending this idea, prefix- tuning (Li & Liang. (02T) optimizes not just the input embeddings but also the inputs to every attention layer within the transformer model, appending them to the key and value vectors.
To ensure stability during optimization, vefix-tuning employs a reparameterization strategy Li & Liang 2021
Liu et al. 2021 [an et al. 2024), where prefix vectors are reparameterized rather than
"Equal Contribution
'Qualcomm Vietnam Company Limited

---

Published as a conference paper at ICLR 2025
being optimized directly. After training, only the prefix vectors are retained for inference. However, the theoretical justification for this approach remains remainns largely unexplored. Key questions, such as why reparameterization is necessary and what theoretical principles support its effectiveness, have not been comprehensively addressed. In investigating these questions, we argue that reparameterization is not merely engineering trick but is supported by deep theoretical foundations. Our findings suggest that the reparameterization trick implicitly encodes a shared structure between the prefix key and value vectors. Through extensive experiments, we demonstrate that this shared structure plays a pivotal role in enabling prefix-tuning to achieve competitive performance.
Recent work by Le et al. (2024) has revealed that self-attention (Vaswanil 2017) functions as a specialized mixture of experts (MoE) architecture (Jacobs et al.) [991] Jordan & Jacobs [994).. Within this framework, prefix-tuning serves as a mechanism for introducing new experts into these models. Building on this connection, we provide a detailed analysis of reparameterization from the perspective of expert estimation. We show that the shared structure enhances sample efficiency in prompt estimation compared to cases where the structure is not shared.
Contribution. The contributions of this paper can be summarized as follows: (i) We uncover that the reparameterization trick in prefix-tuning, often regarded as an engineering technique, is grounded in solid theoretical principles. Specifically, we show that reparameterization induces a shared struc- ture between the prefix key and value vectors, which is crucial in enabling prefix-tuning achieve competitive performance. (ii) Through comprehensive experiments in both visual and linguistic do- mains, empirically demonstrate that this shared structure significantly enhances the effectiveness of prefix-tuning, highlighting its importance across diverse tasks. (iii) Via the connection between prefix-tuning and mixtures of experts, we provide theoretical justifications for these empirical obser- vations, showing that the shared structure leads to faster convergence rates compared to non-shared alternatives. (iv) Furthermore, we observe analogous patterns of shared structure in prompt-tune -tninng. Our insights not only explain the role of common practices in -tniin implementation but also offer a partial exploration of the mechanisms underlying the effectiveness of prompt-tuning.
Organization. The rest of the paper is structured as follows. In Section 2 we provide an overview of prompt-based techniques and their connection to the mixture of experts framework. Section introduces the shared structure, which is inspired by the reparameterization strategy. In Section we present theoretical convergence rates for scenarios involving shared structures, demonstrating improved sample efficiency compared non-shared cases. Section 5 details our empirical evalu- ations on visual and language tasks. Finally, in Section 66] we discuss the limitations and suggest future directions. Full proofs and experimental details are provided in the appendices.
Firstly, let we denote [n] = (1,2,....,n) .n] for any n E N. Next, for any vector u E TRdd. we use u = (u(1), (2))) (01, .. au) E Nd, ue uf uf-u. ug² ... ng |u| u1 + + + un and a! := as azlaz!.... aal, while |ul] stands for its 2-norm value. Additionally, let |S| denote its cardinality for any set S. Lastly, for any two positive sequences (an)n21 and (b,)n21 we write On = O(b,) or an S bn if an < Cb, for all n e N, where C' > 0 is some universal constant. The notation a, = Op(bw) indicates that an/ /bn stochastically bounded.
Notation.
u()) and u = (u1, u2..... ua) interchangeably. Given any a :=
2 BACKGROUND
We begin by reviewing the background of prompt-based fine-tuning techniques. Following this, we describe the concept of mixture of experts models and examine how prefix-tuning can be interpreted within the context of MoE models. A detailed discussion of related work is provided in Appendix D
2.1 PROMPT-BASED APPROACHES
The Transformer VVaswani 2017 Dosovitskiy et al. 202T) architecture comprises multiple multi- head self-attention (MSA) layers. To illustrate the function of a single MSA layer, consider an input sequence of embeddings [z]...., INI E Nxd, where N is the sequence length and d is the embedding dimension. The MSA layer processes this sequence as follows:
MSA(X2, Xx. Xv) = Concat(fai,.... hm W E ERNx
h; = Attention( x ? W.K XvW, i [m], where Xo Xx Xv = [01, [..,..., NN T are the query, key, and value matrices, respectively.
(2)
Here m is the number of heads, and wo mmd is the projection matrix. Each attention head h;
2

---

Published as a conference paper at ICLR 2025
is parameterized by W.P Bddd W.K R and W R with dç. d A. Building on this, fine-tume tine-tuning techniques such as prompt-tuning (Lester et al. 2021) and prefix-tuning (Li & Liang/ (021)) have emerged as efficient methods for adapting pre- traineddd transformer- baaseeee-bbaedd based models to downstream tasks. Thess methods introduce prompt parameters P NR, which are used to modify the imput embeddings fed into MSA layers, where Np denotes the prompt length.
Prompt-tuning involves prepending prompt vectors to the input embeddings, which is equivalent concatenating the same prompt parameters P Xo, ., X K. and Xv:
(1
P Xv
Sprompe [x. XK. Xy; ; MSA
[Xo]
[x
(3)
resulting in an output in (R(NNN))) xd with increased dimensions.
Prefix-tuning decomposes P into Pk &R F XK and Xv, respectively:
and Pv
R " which are then appended to
Pre--T (Q. XK. Xy; P) := MSA
XQ=
Pk X
Pv Xv 1)
Chn W (4)
In contrast to prompt-tuning prefix-tuning preserves the output sequence length, keeping it identical to the input sequence length and enabling flexible adaptation across the network.
2.2 MIXTURE OF EXPERTS MEETS PREFIX-TUNING
An MoE model consists N' expert networks, fi IR + Rd for [N'], and a gating func- tion G R2 - R that allocates contributions of each expert based on the input æ. The gating mechanism learned score functions, 8; IRd I., associated with each expert, resulting in:
N' C f((x) = - exp (x(()))) Σド= exp (s)(z))
N'
fi(x),
(5)
where G(
(2) 8N' (x)). Building on this formulation, recent work by Le al. (2044) demonstrates that each attention head within the MSA layer be interpreted specialized architecture composed of multiple MoE models. The study further suggests that prefix- tuning serves as mechanism for introducing new experts into these MoE models, facilitating their adaptation to downstream tasks. Specifically, from equation
(4. consider the output of the 1-th head
hy hh. h R Ndd, Let
Nd represent the concatenated R. Py pp...... ...... .PY. Rxd, where input embeddings, and let Pк [p ..... ...... P E Nz.
[
R
L T. We define N pre-trained experts f3 Nd Rd encoded in the MSA layer, along with L prefix experts fN+J' Nd + Rd introduced via the prompt follows:
(X) W.V E DXX=WV+- Zj fN+y Wy P/r
for j E [N] and j' E [L]. where the matrix Eg E Nd such that E, = Vz. Next, we introduce N x (N + L) score functions, $t,f Rd + R, associated with these experts:
x E W. E,X ( X X W D
Sig (X)
Vdh
VT.
for [N]. E [N] and j' E [L]. Consequently, each output vector hu, can formulated as result of MoE model, utilizing the experts and score functions defined above:
exp(si (X))
h Σ j=l EN (
Et exp(s).4++ (X) (X)
exp(st.++)
7' EN exp(s.. - exp(s..te fN+y (6)
Notably, only Px and Pv learnable, meaning that only the prefix experts fN and their responding score functions 8i,N+J' are trained. These new experts work in conjunction with the pre-trained ones embedded in the original model, enabling efficient adaptation downstream tasks.
3

---

Published as a conference paper at ICLR 2025
3 MOTIVATION: REPARAMETERIZATION STRATEGY
In this section, first introduce the concept of shared structure, derived from the reparameterization technique. We then explain how this structure is integrated into the formulation of prompt-tuning
function
MoE
within
Attention
Output
Pre-cainee expert
Prefis expert
Figure 1: Reparameterization defines both the prefix key p and value p.¹ as functions of shared parameters p/, transformed by go- This introduces parameter sharing between the score functions and expert parameters in the MoE framework in attention. The gating function computes expert weights based on score functions, and the MoE output is a weighted average of expert outputs.
In equation instead of directly updating the prompt parameters Pk and Pv. which can lead to unstable optimization and slight drop performance, Li & Liang 202T proposed reparameteriz-
ing the matrix [Pk; Pv] E IR 4x2d using a smaller matrix p' [p'.....2 Pl. E R2dd which
then composed with a feedforward neural network go RRd + R22d
[p.h .P.] (p())
for 1,..., L, where L, N2. After training, the reparameterization can be discarded, and only the final prompt parameters, Px and Pv. need to be stored. We observe that the reparameterization strategy implicitly encodes a shared structure between the prefix key and prefix value vectors. This relationship can be made explicit by reformulating equation (7) as follows:
p
K = (p)), p. =z((())
(8)
where d Rd and Rd + Rd are two functions derived from go. Both the prefix key p and prefix value p.¹ are functions of the same underlying parameters p but modulated by distinct transformations 0 and 02. We refer this as the shared structure among the prompt parameters.
As discussed in Section 2.2 drawing from the relationship between prefix tuning and MoE models, the prefix key and value can viewed as corresponding to the score functions and expert parameters, respectively. This suggests that the shared structure introduces a form of parameter sharing between score functions and expert parameters within MoE framework in attention, as illustrated in Figure [ perspective of the parameter estimation problem, compared to models without such shared structure.
In Section [ we show that this sharing strategy enhances sample efficiency from the
Shared structure in prompt-tuning. Prompt-tuning, by attaching prompt parameters to the key, query, and value matrices, refines pre-trained MoE models by integrating additional experts, similar to prefix-tuning, and also allows the incorporation of new MoE models. Detailed proof is provided in Appendix A While prompt-tuning can integrate new MoE models, our study focuses on pre- preitrained MoE models within each attention head as preliminary exploration of the underlying mechanism.
As shown equation B)) prompt-tuning employs a single prompt parameter P for both key and value vectors. We find that this strategy also introduces shared structure, similar to the pattern
4

---

Published as a conference paper at ICLR 2025
described in Section Specifically, the prefix key and prefix value vectors expressed as:
Px = ( = P, Pv - 02 (P) P,
where di and 02 are identity functions. Consequently, prompt-tuning encodes a shared structure between key and value vectors, leading to parameter sharing between the score functions and expert parameters in pre-trained MoE. models. As discussed further in Section [ this parameter-sharin mechanism promotes faster convergence in parameter estimation, offering theoretical justifications for using the same prompt parameters for both key and value vectors. We posit that these insights contribute a partial explanation of the efficiency and effectiveness of prompt-tunin which applies the same prompt parameters to the key, query, and value matrices.
4 THEORETICAL ANALYSIS FOR PROMPT LEARNING IN PREFIX-TUNING
The interpretation of prefix-tuning via mixtures of experts in equation (6 provides a natural way to understand prompt learning in prefix-tuning via the convergence analysis of prompt estimation in these MoE models. Moreover, as shown in equation (6). each MoE model in each attention head follows similar structure. Thus, simplify the presentation of our analysis, we focus only on the first head, namely, I = 1 in equation (6). and the first row of the attention this head, namely, i = in equation 66. In particular,
consider regression framework for MoE models as follows.
Rd x IR are i.i.d. samples of size n Setting. We assume that (X1, Y1), (x2, 12). V2)....... (X,; Yn) E generated from the model:
Y fG. (X,) t i= 1.
(10)
where Ei..... En are independent Gaussian noise variables such that E|s; X,] = 0 and Var(e;(X,) 2² for all 1 S samples from some probability distribution μ. The regression function fc. in equation (10) then takes the form of prefix MoE model with N pre-trained experts and T. unknown experts,
Additionally, we assume that X1, .......... Xn, i..d
N
exp(X X ASX + +a))
exp((Bp., )T x + bb.') p)
(x) =
Σ
1(X,
+
.CPY.s (11)
Dx(X)
(x)
1=1
j'=1
T + ba.)), while G. = )0(0 .. ""."') denotes a mixing measure, i..., a weighted sum of Dirac measures 6, associated with unknown parameters (bagr.p.."" "",""". ) in the parameter space C IR Rd Rd. At the same time, the values of the matrix A9. the expert parameter ng. and the bias parameter al are known for all < S N. Additionally, the matrices B d and C RLxd given and they play the role of pre-trained projection matrices in the context of prefi--tuning in equation (6).
where Dy(X) = EN exp(e X 16 + ag) + Σイ
In the sequel, we will investigate the convergence behavior of estimation for the unknown prompt parameters. Our main objective is to show that the convergence rates of the prompts will be ac- celerated when they share the structure, that is, they can be reparametrized
p = a (p) and p = 02 (p),, for some functions σ1 and 02, as motivated in Section 3 To this end, we will con- duct the convergence analysis of prompt estimation when there are non-shared and shared structures among the ground-truft prompts in Section 4. and Section 4.2] respectively. Then, we compare the convergence rates in these scenarios to highlight the sample effeiency of the latter method.
4.1 WITHOUT REPARAMETRIZATION (NONSHARED STRUCTURES) AMONG PROMPTS
In this section, first investigate the scenario when the prompt parameters do not share the inner structure, where we need to learn the prompts p",," and p. in equation (1] separately. To estimate those unknown prompts or, equivalently, the ground-truth mixing measure G., we use the least square method (van de Geer, 2000). In particular, we take into account the estimator
Gn = arg min
(v: (V=-Jo((.))
(12)
G i=1
where we denote (0) (G (G= exp(b.) , 1 $ l < L', (b., .P!) E θ} as the set of all mixing measures with at most T,' atoms. In practice, since the true number of experts I, is
5

---

Published as a conference paper at ICLR 2025
typically unknown, assume that the number of fitted experts L' is sufficiently large, i..., L' > L. In order to characterize the convergence rate of prompt estimation, it necessary to construct a loss function among prompt parameters. To this end, propose using loss function based on the concept of Voronoi cells (Manole & Ho 2022), which we refer to as the Voronoi loss function.
Voronoi loss. For mixing measure G with L L' atoms, distribute its atoms to the following Voronoi cells V = (G), for j E [L]. generated by the atoms of G.:
Vu E [L] [p p.)) (p pX,) = lKpp." .Y) (",,,.P' P., +
(13)
Then, the Voronoi loss function of interest is defined as
DG,
= f=1 LVVY
exp(b:) exp(br.'))
Σ Σ y'=iiV
exp(bi)
|1 I1" + 1|p |r].
for N, where denote App p pt, and = p we are now ready to capture the convergence behavior prompts in the following theorem.
Given this loss function,
Theorem 4.1. The following bound of estimating G. holds for any E N:
E 1i G)] 2 11/2
1/2
(14)
())
where fo indicates the expectation taken w.r. the product measure with fö-
Proof of Theorem 4.1 is in Appendix B.1 The bound in equation (4] together with the formulation of the loss DI, implies that the convergence rates of estimations for both the prompts pr and p',y' are slower than ((n-1/2r) for any N and, therefore, could be as significantly slow as OLL log(n)). This observation indicates that the performance prompt learning will be negatively affected when there are no shared structures among the prompt parameters.
4.2 WITH REPARAMETRIZATION (SHARED STRUCTURES) AMONG PROMPTS
In this section, we consider the scenario when the prompts share their structures with each other. In particular, we reparameterize the prompts as p = a (p) and p o2(p) where p E Rdd the functions T1, Rd' + RR, and the dimension d' 2 is given. That parametrization indicates that the prompts will share the input of the functions σ₁ and d2-
To theoretically demonstrate the benefits of reparametrization among prompts in prompt learning, we specifically take into account the following two settings of the functions 01 and 02.
(i) Simple linear setting: 01 (1(p)= p and 02 (p) = p for any p Rd:
(ii) One-layer neural nelwork setting: a (p(p) ō (W () and 02(2) 52 (Wpp) for any p Rd where W, dxdd and W₂ ddxd are learnable weights.
Here, on and 02 are two given real-valued activation functions, Furthermore, for any vector r = (L ....),.... (dd) ) E Rd, we denote (z) = (z) .x( for any 1 S i S 2, that is, the functions and ō2 are applied to each element of the vector x.
4.2.1 SIMPLE LINEAR SETTING
We begin our analysis with the simple linear setting under which px (p) and pV = 02(p) p for any p Rdd. This setting motivated by prompt-tuning strategy as being discussed in Section 3 Then, the ground-truth regression function in equation (11) turns into
Sc. (X) = En exp(X A9X + (2) TT)) + Σ (xp((Bp..)) + bay') Cp
where G, Σ exp(ba.j)) )0 pp.., is a mixing measure with unknown parameters (ba.j', Pm f= belonging to the parameter space Ω IR x Rdd To ensure the identifiability of estimating prompts in the simple linear setting, assume that Bp.,1..... Bp.,1. pairwise different, Similar to the

---

Published as a conference paper at ICLR 2025
nonshared structure setting of prompts in Section 4.1 we also employ the least square method to estimate the unknown parameters or, equivalently the mixing measure G.. In particular, the least square estimator of interest is given by:
Gn == arg min
Σ - (o()))())) (2) i=1
(15)
where (2) == GG Ex= exp(b.)6p. : S L', (i) E Ω} the set of mixing measures with at most L' atoms, where L' L, and parameters belonging to the space Q. Then, need to build a new Voronoi loss function to capture the convergence rate of prompt estimation.
Voronoi loss. The Voronoi loss tailored to the simple linear setting of prompts is defined
D2(G, =
Σ
Σ
exp(b:)
exp((=-))) +
Σ
Σ
exp(b: xxp(b:)l\Apxr
f'=1 EV
)E y iey
Σ
Σ
exp(b:))l
where denote Apty = - for any i,j' Equipped with this loss function, we wrap up the simple linear setting of prompts by providing the convergence rate of prompt estimation in Theorem 4.2 whose proof deferred to Appendix B.2]
Theorem 4.2. Given the least square estimator G. defined in equation
15 we have that
D2(G,n G.G.n.G.) 00(---00())))//)///
It follows from the bound in Theorem 4.2 and the formulation of the loss D2 that for prompts P--j' whose Voronoi cells have exactly one element, that is [Vyr| = 1, the rate for estimating them of order Op( [log(n)/m)) which is parametric on the sample size n. On the other hand, the estimation rate for those whose Voronoi cells have more than element, that is [ > 1, is slightly slower, standing the order of Op( (/logn)/nn). In both cases, is clear that these prompt estimation rates are substantially faster than those in Theorem 4.1 which could be as slow as O(1 (11/log(n)) (n). Therefore, we can claim that reparameterizing the prompts as px = =p" = sample efficiency of the prompt learning process, thereby leading to superior performance to the scenario when there are no shared structures among prompts in Section 4.1
helps enhance the
4.2.2 ONE-LAYER NEURAL NETWORK SETTING
We now move the setting where the prompts are reparameterized as one-layer neural networks, that is, x (p) σ1 (1(Wp) and p' (p (2(W))) in which W1 R W2 e Rddxd' are learnable weight matrices and Ø1, 32 are two given real-valued element-wise activation functions. Our goal is to demonstrate that the reparametrization among prompts still yields sample efficiency benefits beyond the simple linear setting in Section Different from the simple linear setting, the true regression function under the layer neural network setting takes the form:
fa. (x) = Σ ex
19 X + +a9)
((X,1))
Dx(X)
+ exp(( (W.,)+.))) (X)
X + bbn'') TT
Co
3'=1
where the true mixing measure is of the form G. Σ/ (((. )P that is, weighted sum of Dirac measures associated with unknown parameters (b-.g,, W., W.,IP--.', W. 2f in the parameter space E C R x Rd x Rdd. To guarantee the identifiability of prompt estimation in the one-layer neural network setting, we assume that Bo, (W,,(W,,1 (W..Pt..).... BT, parameters, we utilize the least square estimator, which given by:
are pairwise different. In order to estimate these unknown
G = arg min
Σ (E) tzl
Y, 1x)
(16)

---

Published as a conference paper at ICLR 2025
where (3) -G=EE= set of mixing measures with at most I,' atoms, where > L,, and with parameters in the space E.
: & L', (b,, WiP:, Wap:) E 3] as the
Voronoi loss, In alignment with the regression function change, necessary to construct an appropriate Voronoi loss function for the analysis of this setting, which is given by:
7
Da(G.G.) = 3
exp(b;) -xp((.,)))
iEVY
E
Σ
exp(::) - W..1P.y51 |W?p,
W.,2P.,
8 FELLIV 1 y'
Σ E
W2p: W.
Subsequently, since the prompt reparametrization under this setting involves the activation functions 3 and 02 let introduce two standard assumptions on these two functions prior to presenting the convergence analysis of prompt estimation.
Assumptions. The two activation functions ã and 02 are given such that the followings holds:
(A.1) (Uniform Lipschitz) Let F(X; Wip, Wap) == exp((Bo (Wip)) Co2((WWp)) Then, for any 21, we have
a
glai F
Σ
X; Wap,
(X; Wip', (2p')) 70
8(Wip)a atMMip)aWWWappp O(Wp))
C1I(W W2p) (Wip'. W22p'll
for any vector E Rd2d and for some positive constants ( and C which are independent of X and (Wip, Wep), (Wip', W2p'). Here, a (a1, E N22 where a1 02 E No.
(A.2) (Non-zero derivatives) a(Wap) oWPp) (W 2P ) + 0, for all u E [d] and j' E [L].
(W2h) where the function tanh is applied element-wise, meet both the assumptions (A.1) and (A.2). By contrast, T2 is linear function, e.g. 02 (W2)) W2p, then the assumption (A.2) is violated.
Example. We can validate that 01 (WIp) = tanh (Wi)) and 02 (W2p)
Theorem 4.3. Assume that the given activation functions 51 and ō₂ satisfy both the above assump- tions (A.1) and (A.2), then it follows that
D3 (G,, G.)= (Op(Vlo(nn//))
Proof of Theorem is Appendix B.3 This theorem indicates that the rates for estimating Wr,, Ww,PPr,, are of orders O (og((m)/n) and Op( (P((log(n)/r) if [Vyl and [ > 1, respectively. Furthermore, let 1n,i and Wn,pPn, be estimators of W.,PP=.,' and W..2P+y', respectively. Since the activation functions ōn and ō1 are Lipschitz continuous, that is,
[[o,(W,,PP..) - Te(W.,eP..5)) I [Wn,,Pm.s - W=,eP=.5'll for any f (1,2) |
we deduce that the prompts pkK- = σ estimation rates those of W=.1P.,; and W. W.,PP.,i- Note that these rates are significantly faster than those in Theorem 4.1 where the prompts does not share their inner structures, which could be as slow (11/ log(n)). This observation together with that from Theorem 4.2 demonstrate that the reparametrization among prompts under both the simple linear setting and the one-layer neural network setting helps improve the sample efficiency of prompt learning considerably.
and pV,, (W..PPP-))))) admit the same
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
In our experiments on visual and language tasks, we follow the settings Jia et al. (202)) and Li & Liagg (2021) respectively. Please refer to Appendix for further details.
8

---

Published as a conference paper at ICLR 2025
Table I: Comparison of prefix-tuning with and without reparameterization FGVC and VTAB-1K benchmarks. We report the average accuracy over five independent runs. Best results among all methods except Finetune are bolded.
Method
Mean
CUB-200-2011
NABirds
FGVC Oxford Flowers
Stanford
Stanford Cars
Natural
VTAB-IK Specialized
Stractured
Finetune
87.3
Deep-elareessaaee
84. 36
87.2
81.5
98.6
91.1
63.4
75. 79
79.48
38.53
No-shmereraacco
80. 38
85.1
77.8
97.9
86.4
54.7
69.00
77.20
29.65
Deop-alaane.eu
98.2
77.06
52,00
No-shncrgger
82.32
85.9
79.0
97.9
86.3
62.5
70.29
80.20
37.69
Table 2: Comparison of prefix-tuning with and without reparamcterization language datasets including E2E, WebNLG, and XSUM. Best results among all methods except Finetune are bolded.
E2E
WehNLG MET
XSUM
NIST
CIDEr
TER
$
U
A
8
U
A
U
A
Finetume
46.5
Deep-share
69.9
8.78
46.3
71.5
2.45
63.9
44.3
54.5
0.45
0.36
0.41
0.34
0.52
0.42
42.62
19.66
34.36
No-share
8.61
45.8
71.0
2.41
61.1
42.8
53.5
0.43
0.35
0.40
0.36
0.49
0.42
36.86
15.16
29.89
Datasets and metrics. For visual tasks, use the FGVC and VTAB-IK (Zhai et al. 2019) bench- marks. FGVC includes five Grained Visual Classification datasets: CUB-200 2011 (Wah et al.
2011 NABirds (Van Horn et al. 2015), Oxford Flowers (Nilsback & Zisserman 2008). Stanford
Dogs (Khosla et al. 2011). and Stanford Cars (Gebru et al. al. 2017). VTAB-IK comprises 19 visual tasks in three categories: Natural (standard camera images), Specialized (specialized equipment images), and Structured (tasks requiring structural reasoning like 3D depth prediction). We report accuracy on the test set. For language tasks. we assess performance table-to-text generation and summarization. We evaluate table-to-text generation with E2E (Novikova et al. 2017) and WebNLG (Gardent et al., (017)) datasets, using BLEU (Papineni et al. 2002). NIST (Belz & Reiter 2006), METEOR (Banerjee & Lavie 2004), ROUGE-L (Lin) 2004), CIDEr (Vedantam et al. 2015). and
TER (Snover et 2005). 2005). Summarization assessed with the XSUM dataset (Narayan al. 2018) using BOUGE-1,ROUGER
and ROUGE-L. Table summarizes the metrics for each dataset.
Baselines. To assess the effectiveness the shared structure, we evaluate prefix-tuning under the following configurations: Deep-share: uses prefix-tuning with the reparameterization trick; No- share: applies prefix-tuning without reparamcterization, with prefix key and value vectors as inde- pendent parameters; Simple-share: similar Deep-share, but with and d2 the identity function (see Section 3). Additionally, following Jia et al. (2022), we explore two variants: SHALLOW, where prompts attach only to the first layer, and DEEP, where prompts less otherwise specified, references to prefix-tuning denote the DEEP variant. We also compare prefix-tuning with several fine-tuning techniques: Finetune: updates all backbone model parame- ters; Partial-k: fine-tunes only the last & layers of the backbone while freezing the others; Adapter (Houlsby al., 2019; Lin al. 2020): inserts new MLP modules with residual connections into the Transformer layers; VPT (Jia et al., (022): designed for visual tasks, integrates learnable prompts into the input space of Transformer layers, following prompt-tuning approach.
attached to all layers. Un-
Pre-trained backbones. We use the Vision Transformer (ViT-B/16) (Dosovitskiy et al. 2021 pre-trained on ImageNet-21K (Deng et al., (2009), for visual tasks. For table-to-text, we utilize GPT2MEDICM (Radford et al., 2019), with linearized input tables. For summarization, we employ BARTLARCC [Lewis) 2019), truncating source articles 512 BPE tokens.
5.2
MAIN RESULTS
Tables D and present the performance of prefix-tuning with and without reparameterization. De- tailed per-task results for VTAB-1K provided in Appendix F
Prefix-tuning with reparameterization can achieve competitive performance with full fine- tuning. As shown in Table although prefix-tuning has not been widely explored for visual tasks, our results indicate that Deep-sharenser performs comparably to full fine-tuning, surpassing it of 4 problem classes (13 out of 24 tasks). For instance, prefix-tuning achieved 91.6% accuracy

---

Published as a conference paper at ICLR 2025
on Stanford Dogs, surpassing full fine-tuning by 2.2%, and 52% accuracy VTAB-1K Structured, exceeding fine-tuning by 4.36%. While it underperformed on more challenging tasks like Stanford Cars, Deep-sharepEep still achieved comparable average accuracy (88.28% vs. 88.54%). Similar On E2F, prefix-tuning outperformed
trends are observed for language tasks, as shown in Table 2 fine-tuning across most metrics, though it slightly lagged in the XSUM summarization task.
Reparameterization plays a crucial role in enhancing the effectiveness of prefix-tuning. It can be observed that the performance significantly declines when the reparameterization strategy is omitted. As shown in Table D Deep-share outperforms No-share by substantial margin across both variants, DEEP and SHALLOW. For instance, on Stanford Cars, Deep-share exceeds No- sharepare by 16.8% This trend is consistent across the majority of datasets (22 out of 24 tasks), underscoring the effectiveness of reparameterization in improving prefix-tuning performance. This empirical finding aligns with our theoretical results presented in Section E which demonstrate that reparameterization significantly enhances sample efficiency in parameter estimation. These trends persist across both visual and language tasks. In Table 2 Deep-share surpasses No-share most metries across three datasets. For example, in summarization tasks, Deep-share outperforms No- share on all metrics by considerable margin. This illustrates the critical role of reparameterizztioo in enabling prefix-tuming to achieve competitive performance.
CUB-099
Oeford
Stanlerd
Shallow
Seng
Sinple
Delp
plr
Delp
Deep
Doep
Det 0i
Deep
A0ngle
Dntp
pbo
Deap
Deep
Comparison of prefix-tuning across three configurations: Deep-share, Simple-share, and No-share, referred to as Deep, Simple, and No, respectively, on FGVC benchmarks.
Figure 2:
The shared structure significantly improves prefix-tuning performance. To further assess the impact of the shared structure, we compare prefix-tuning under the Simple-share configuration, where 01 and 02 are identity functions. As discussed in Section 4.2 our thcoretical analysis sug- gests that both Deep-share and Simple-share substantially outperform the No-share baseline. These findings are consistent with our empirical results, as shown in Figure 2 Across all FGVC datasets, both Simple-share and Deep-share consistently yield significantly better performance than No-share. This consistent improvement demonstrates the empirical effectiveness of shared structures in en- hancing prefix-tuning performance. For further experimental results, see Appendix F
DISCUSSION AND CONCLUSION
In this paper, offer theoretical insights into the reparameterizatioon strategy employed in prefix- tuning, which is often regarded as an engineering technique. We demonstrate that reparameterization induces a shared structure between the prefix key and value vectors, which significantly enhances sample efficiency during prompt estimation. Beyond the theoretical analysis, we empirically val- idate the advantages of this shared structure through experiments across both vision and language tasks. However, the current reparameterization implementation, which relies on an MLP to generate prefix vectors during training, introduces a potential memory overhead. Future work could focus on optimizing this implementation to reduce such overhead. Additionally, while our focus is on prefix-tuning, we propose that the benefits of the shared structure may extend to other parameter- efficient fine-tuning techniques, such as LoRA. We also identify similar patterns of shared structure in prompt-tuning, offering a preliminary investigation into the underlying mechanisms contribut- ing to its effectiveness. However, our study is limited to pre- praitedined MoE models in the context of prompt-tuning, serving as an initial exploration. Future research could explore the influence of newly introduced MoE models and the interactions between these models.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In order to facilitate the reproduction of our empirical results, we provide detailed descriptions of the experimental setup in Section 5. and Appendix [ available, enabling full replication of our experiments.
All datasets used in this study are publicly
REFERENCES
Satanjeev Banerjee and Alon Lavie. Meteor: automatic metric for evaluation with high levels of correlation with human judgments. Proceedings of ACL-WMT, pp. 65-72, 2004.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03881. 2016.
Anja Belz and Ehud Reiter. Comparing automatic and human evaluation of nlg systems. In 11th conference of the european chapter of the association for computational linguistics, pp. 313-320, 2006.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 186518)188651188, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of ex-eeerss In International Conference on Machine Learning, pp. 5547- 5569. PMLR. 2022.
D. Eigen, M. Ranzato, and Sutskever. Learning factored representations in deep mixture of experts. In ICLR Workshops, 2014.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning object categories. IEEE trans- actions pattern analysis and machine intelligence, 28(4):594 9-119946661111111 2006.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura challenge: Generating text from rdf data. In 10th International Conference Natural Language Generation, pp. 124-133. 133. ACL. Anthology. 2017.
The webnlg
Timnit Gebru, Jonathan Krause, Yilun Wang. Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- Fine-graine grained car detection for visual census estimation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1211-1237 2013.

---

Published as a conference paper at ICLR 2025
Ben Graham. Kaggle diabetic retinopathy detection competition report. University of Warwick, 22 (9), 2015.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, al. Pframeter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14600 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards
unified view of parameter-efficien transfer learning. arXiv preprint arXiv22110.04366 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7) 2019.
Nhat Ho, Chiao- Yu Yang, and Michael Jordan. Convergence rates for gaussian mixtures of experts. Journal of Machine Learning Research, 23(323):1-81) 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021.
Eugenia lofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse imagenet models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12266-12276, 2022.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3, 1991.
Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214 1994.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR workshop on fine-grained visual categorization (FGVC), volume 2, 2011.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. Mixture of experts meets poompt-based continual learning. In Advances in Neural Information Processing Systems, 2024.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II-104, IEEE, 2004.
12

---

Published as a conference paper at ICLR 2025
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz et al. Standing on the shoulders of giant frozen language models. arXiv preprint wiXv:2204.10019, 2022.
Lewis. Bart: Denoising sqquence-o--squeenee pre-training for natural language generation, trans- lation, and comprehension. arXiv preprint 1910./39900/330 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky, Scaling down to scale up: A guide to parameter-efficient fine-tuning, arXiv preprint arXiv:2303. 15647, 2023.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: new bascline for efficient model tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 71, 2004.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. arXiv preprint (tt:::00000038225 2020.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07002, 2021.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-experts In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 1930-1939, 2018.
Tudor Manole and Nhat Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14979-15006. PMLR, 17-23 Jul 2022.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topie-aware convolutional neural networks for extreme summarization. arXiv preprint 1rXiv:1808.08745 2018
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop deep learning and unsupervised feature
feature learning, volume 2011, pp. 4. Granada, 2011.
Huy Nguyen, TrungTin Nguyen, and Nhat Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems, 2023.
Huy Nguyen, Pedram Akbarian, and Nhat Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? In Proceedings of the ICML, 2024a.
Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. Sigmoid gating is more sample efficient than softmax gating in mixture of experts. In Advances in Neural Information Processing Systems, 2024b.
Huy Nguyen, Nhat Ho, and Alessandro Rinaldo. On least square estimation in softmax gating mixture of experts. In Proceedings of the ICML, 2024c.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEBE, 2008
13

---

Published as a conference paper at ICLR 2025
Jekaterina Novikova, Ondiej Dušek, and Verena Rieser. The e2e dataset: New challenges for end- to-end generation. arXiv preprint 1xXiv:1706.0025::0 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318 2002.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Aleksandar Petrov, Philip HS Torr, and Adel Bibi. When do prompting and prefix-tuning work? theory of capabilities and limitations. arXiv preprint arXiv:2310. 19698, 2023.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse soft mixtures of experts. arXiv preprint arXiv:2308.0095 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 2021.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-6266 2017.
N. Shazeer, A. Mirhoseini, K. Maziarz, Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The pparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.
Mathew Snover, Bonnie Dorr, Richard Schwartz, John Makhoul, Linnea Micciulla, and Ralph Weischedel. A study of translation error rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 06), pp. 223-231, 2005.
S. van de Geer. Empirical processes in M-estimation. Cambridge University Press, 2000.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print fine-grained dataset collection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1955604, 2015.
Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant enns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICC MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II I1, pp. 210-218. Springer, 2018.
14

---

Published as a conference paper at ICLR 2025
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 JEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficiest fine-tuning for pre-trained vision models: survey, 2024.
Bin Yu. Assouad, Fano, and Le Cam. Festschrift for Lucien Le Cam, pp. 423-435, 1997.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformerrbased masked language-models. arXiv preprini arXiv:2106.10199 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1rXiv:I9100 1910.04867 2019.
Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng, Chang Lan, Da Huang, Siamak Shakeri, David So, Andrew M Dai, Yifeng Lu, et al. Brainformers: Trading simplicity for efficiency. In Interna- tional Conference on Machine Learning, pp.
PMLR, 2023.
Wei Zhu and Ming Tan. Improving prompt tuning with learned prompting layers. arXiv preprint arX1v:2310.0112,, 2023.
15

---

Published as a conference paper at ICLR 2025
Supplement to "Revisiting Prefix-tuning: Statistical Benefits of Reparametrization among Prompts"
In this supplementary material, we begin by exploring the relationship between prompt-tuning and mixture of experts in Appendix [A] Following this, we provide detailed proof' for the theoretical
results discussed in Section 4 Additionally, we present in-depth discussion of related work
in Appendix D Appendix offers further implementation details for the experiments outlined in
Section 5 Finally, Appendix includes additional experimental results.
A PROMPT-TUNING AND MIXTURE OF EXPERTS
We demonstrate that applying prompt-tuning not only fine-tunes pre-trained MoE models by incor- porating new experts but also facilitates the introduction of entirely new MoE models within the attention mechanism. Specifically, similar to Section 2.2 we consider the 1-th head within the MSA
layer. Let P pp....,,NN E R. We define new experts SN+j IR - their corresponding new score functions 81,N+j .R. IR for pre-trained MoE models as follows:
Rd along with
(X) = W.T Pj: SL,+)(() X E, WOWK Pj
x. W.TWOW' Pj
(17)
vd
Vd
for E [N] j E [N For Nº new MoE models, we define the score functions SN+i,
R.d
IR associated with pre-trained experts as:
8 (X) P. p.W,WW. E
P. W. Tj
(18)
dd
dd,
for i E [Np] and j E [N]. The score functions 8N+1NN4 RRNd - R for new experts within new MoE models are defined as:
P. WWW Pj
SN+IN+ (X)
(19)
Vd,
for i E [Ny)] and j E [AVp].
Then from equation
3), the output of the (-th head can be expressed as:
h Attention
([
P
P
P
fu..,...,,,,, (....,,, E (N(++)))dd,
(20)
[xQ
X
X
exp(si,(X))
N kk= exp(ss,(())) EE' (x)( 1(()
(X)(
Ek=1 exp(ss,(X)) + - exp(st,)*x fN+
(21)
for E [N + .ypl. grompt-tuning extends pre- treitra MoE models by incorporating N, additional experts JN+j, which are defined by the prompt vectors Pj. Additionally, prompt-tuning introduces new MoE models, hi,441...., NN, that utilize linear and scalar score functions.
B PROOFS
B. PROOF OF THEOREM 4.1
The proof is divided into two step as follows:
Step 1. To begin with, we demonstrate that the following limit holds true for any r 2 1:
Ilfo fG. [[( 0.
lim c GEG (G)...
inf
(22)
DI,
Note that it is sufficient to construct a mixing measure sequence (Gn)az that satisfies both G.) + 0, as n - co.
Pi,,(Gn, G.) + and |lSc. fa. (l.lllL
For that purpose, we take into account the sequence G, ΣLI exp(lm:)))pp wk.)) where
16

---

Published as a conference paper at ICLR 2025
cxp(bn,,1 exp(bn.2) exp(b.,1) 2nr-I and exp(bn.s) = exp(6n,s-1) for any 3
p = = = P and p P for any S 1 - L
Pk1 pk, (1,0,..... Pk,2 pK, - (1, (1,00..... 0) and ph: = P for any 3sisL
Then, compute the loss function D (G, G+))
1
1 O(n
(L, G.) = n2 + exp(b.,1) + n-+1 n
(23)
It can be seen that D1 (Gn; G.) - 0 as -> 20.
Subsequently, we illustrate that [| - fc. (12 (I, G.) - 0. In particular, let consider the quantity
Qx(X) =
Σ exp( 1 a.) + Σ
exp((Bp..)) Tx + b.
(X) fc. (X)).
j'm
which can be decomposed as follows:
(x) =
exp(bm.i) exp((BBp..)) "XCP", 7
j= iEV
exp(bn.))
) exp((Bp".,)) x).a..()) (x) exp(((pp x))G,(X) (X)s,
j=,;ev;
Σ
exp(bm..i) - exp(br.)) exp(((p..)) (epp((.))))ep(((()))) Xx)pp'
(x - exp((Bp(.,) x) X) (x)
j=1
A, (X)= Bn (X) + (X).
follows from the choices of K ph, and b that
A,(X)
cxp(b.,1)
n1+1 exp(((p!))) (x)C(" -PX
1
exp(bb.1) + n1t1
exp((p!;,) X)C((p), - -KK1) + (pk,2 PX,1)
= 2
0.
Moreover, also verify that B,, 0, and C. 1Onn-(++1))) Thus, deduce that Qn(X)/Di,r (G.: G.) + as n + oo for almost every X.
ΣN ex (X A. (xx++a)++ Σ exp((pX, TT X + bagy)) is bounded, we have [Ja,, (X) (X) (Gn, G.) 0 for almost every X. This limit suggests that ||. - fa. (w) Di.r G.) + as 00. Thus, we obtain the claim in equation 22
As the term
Step 2. We will establish the desired result in this step, that is,
inf (.) (11
sup E Ee (D, (G,,, G)
(24)
Since the noise variables 6, follow from the Gaussian distribution, we get that X ~ Nffa. o2) for all i [n]. Additionally, for sufficiently small > 0 and fixed con- stant C > which we will select later. we can find a mixing measure G², E ()) such that
D Di,r (G,, G.) 28 and [so SG. fo.llea()) Cie thanks the result in equation (22). According
17

---

Published as a conference paper at ICLR 2025
to the Le Cam's lemma (Yu 1997). the Voronoi loss function Di, satisfies the weak triangle inequality, follows that
inf G.) (9) (())
E (Di, G)]
PG, G.)
2V V
exp( - (KL(N (W
8
c ex-eepp(nlfff - fa. fa. l1)2(w))
exp(-Cnne*)
(25)
where the second inequality follows from the equality
(L(V (x) ((). (X). (So: (X) - Sc. (x))²
20²
Let E = n n1/2, then we get that E exp(-Cinée Cin = n 11/ exp( -C). Consequently, we achieve the desired minimax lower bound in equation (24
1/2
B.2 PROOF OF THEOREM 4.2
The proof of Theorem consists of two parts. In the first part in Section B.2.1 parametric convergence rate Op (P)/oog(n)// of the estimated regression function fc., to the true regression function fc In the second part in Section B.2.2 establish the lower bound I|fa fe. (I)))))) 2 C'Da(G, G.) for any G ( for some universal constant C². This lower bound directly translates to the convergence rate Or( Or(Vlog(n)/m) of the least-square estimator G, to the true mixing measure G..
we prove
B.2.1 CONVERGENCE RATE OF DENSITY ESTIMATION
Proposition B.1. The convergence rate of the model estimation fö. (.) to the true model fa.() (.) under the (p) norm is parametric the sample size, that is,
[|... Sc.llee Or Or((oog(n)/n)
(26)
Proof of Proposition B.1 is in Appendix C.I
B.2.2 FROM DENSITY ESTIMATION TO EXPERT ESTIMATION
Given the parametric convergence rate of the estimated regression function fc, to the true regression function fa. in Proposition B.J to obtain the conclusion Theorem 4.2] it is sufficient to demon-
strate that |lfe fc. (ا) 2 C"D2(G, G.) for any & E (12) for some universal constant C'. It is equivalent demonstrate the following inequality:
[|fa - fa. / G.) 0. (2),)(9)
inf
We divide the proof of the above inequality into local and global parts.
Local part: We will demonstrate that
Ilfa - Sc. (lw c Ged (C..)s
lim
inf
Assume by contrary that the above claim does not hold. Then, there exists a sequence of mixing measures G, := Ei' exp(bn.y ppm,, in ( such that as n - co, we have
Dzn = D2 (G,, G.) + 0.
[l.o.. - So.li Dzn -
18

---

Published as a conference paper at ICLR 2025
Denote Vy := Vy() Voronoi cell of G, generated by the j-th components of G.. Since arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size. i... Vj = V. Thus, the Voronoi loss D2n can be represented as
D2n = Σ
Σ
exp(bn.)) exp(barr))
E
Σ
(ep(bn.)llApm..t
5'=1 SEVY
VE[LHiyy-=iie
E
Σ exp(bn.))lAAp...)
rELLHU,,IDLEEE
where Apn.y = Pn, Pri for all i Vy: Additionally, since Dzn + O, have Eiev, exp(bn.)) exp(br.)) and Pn.: - P-j for any i E Vj.j E [Z]. Now, we divide the proof of the local part into three sub-steps as follows.
Step 1 Taylor expansion.
this step, we would like to decompose the quantity
N
Qn Qx(X) =
Σ
exp( X AAAxx + ag) +
exp( + bn
===
as follows:
1
Qn(X)= = Σ
exp(bn,i)
"xsp x X)Cp=s]
j=1
L
exp(bnns) exp((Bpn.:)) X) - exp((Bpa.)) xx) Sc., (X)
j=1 VEV,
L
Σ j=1
Σ
exp(bn.;i) -ex((bas)) exp((xs))xxx((Pppp..)) exp((Bp..)) ()ps -6. (x)
(x) Bn + (x)
(27)
Decomposition of (X). To ease the ensuing presentation, we denote E(X:p) = exp((Bp) X) and (p) Cp, and F(X;p) (p)) Since each Voronoi cell V; possibly has more than element, we continue to decompose An follows:
A,(X)= Σ exp(bn.)) [F(X:Pn.) -((ip..)) -FXXP))))))) AlVyl=liEv
Σ
Σ
-F
== A + (X).
By means of the first-order Taylor expansion, we have
glalE
E(X: = E(X:
Σ
(Apnss)
(X:P.s) (x),
Op
aa
glal H
H(pa,)= H(p,,)) +
Σ (Apr, Opa (prs) + Rig
10|=1
19

---

Published as a conference paper at ICLR 2025
for any i E Vj and such that [V51 1. Here, (X) and Rej.2 Taylor remainders. Putting the above results together leads to
exp(bna)
glai E
J-13/== (x)
= E
Σ
(Apm.is)
(:P-.)H(p....
a!
Op
a1/=1
+ (Apn,i) ) alalH
Opm (p..) ) F.) Pos)) + Rx,
(P..))((X
Mn.ja ala E
E
(X; (X: (miD))))(P...
Opa
= al=1
doa H
+ MM.jaa Ope
(p..) (px:) + (X)
where the function 1 satisfies (x), / + 0 when n + cc. Furthermore, the formu- lations of Mnj.a given by:
exp(ban) (Apn.s))
Mnjja
Σ 1EV;
a!
for any |a|
Moving to the term (x, by applying the second-order Taylor expansions to E(X; Pr.,i around E( E(X;p..) and (Pn,s) around H (p=..) for any E V, and j such that [Vy] >> 1, we get that
glai
An,2(X)= = Σ Σ
Mnjaa
H(((s))
Opr
HMII 12
Hlal
Majaa Op (p..) F((:
glalE E (X:P..) Sps (p.g) + (X)
G100 H
Σ
Mn.ja. Opa
where the function R,, ()) satisfies (X),(())/D 0 when - Furthermore, we define
exp(baxx) (Apm.is)
Mnjia
a!
ieV;
for any |a|| and
exp(bnns)
Mnj.o..8
(Apn.i)
a!f!!
TEV,
for any Ja: = |.3] = 1. Direct calculation leads to the following formulations of the partial deriva- tives of E (X:p)) and (p)):
aE pp(u)
(x:) = exp((pp) x)(B1,) x,
X; (.P)p = exxp(Bp)) x)X (BI,).
0p Op(w)p(()
0H
Øpluj (p) -Clu
8
3p(0)p(() ()) (Wpp) 0.
20

---

Published as a conference paper at ICLR 2025
Here, denote lu is the vector that its w-th element is 1 while its other elements are 0 for any $ u S d. Given the above formulations, we can rewrite (X, and (x, as follows:
(x)
E exp((Bp-.))
() (p.)(p B x) + Rn, (X),
An,2(X)= =
Σ exp((Boi((..)) x [1 (Pa.) + L2 (P=.)) B B
+ (BTx) (p- x + (X),
where the formulations the functions Li Lz, Li Ln and Ls are given by:
d Σ
(p, =
Mn.jae Clus
x=1
d
(p) =
Mn. Mn.j. luCp,
==1
(p) Σ Mn.j.1a Clus
(p) Σ Ma.j. Cpp
Σ
Mnslll Ch,d,
15u,edd
(p) = Σ
Iswood
Here, lav is the matrix that its (u, w)-th element is I while its other elements are 0 for any < u S d.
Decomposition of B (X).
We can rewrite (X) as follows:
Ba(X)=
Σ
Σ
exp(bn..) p(X:pn.t) E(X: (X)
exp(bn.:) B((::pm.) [::0)=((Xpppp))) F(X; F(X:P.)) (X)
8: A
Bn, (X) + (X)
By applying the first-order and second-order Taylor expansion, we get
a Op
(x, Σ
Σ
Mn.jo
(x (xpp.))fa (X) + (X)
J=1, |a|=1
gla E
Bx2(X) Σ
Σ
Ma.jaa
(x: p.p. ff.. (X) + (X)
Opa
where Rm, (X)) is Taylor remainder such that Rn 3 (X) D2n - 0, Rn, aX) +
when n + co. Therefore, we can express the functions B,, (X) and (X) as follows:
Bx,((X) Σ exp((Bp...) ) T x)).,.. X fo. (X) + R
J.1V, =1
Bn,(X)= Σ exp((pp..)) T
x
+BBX)² N2. fo, (X) (x)(X),,
21

---

Published as a conference paper at ICLR 2025
where the formulations of the functions Ni, Ni, and 2n given by:
(p) Majjaa lay
N1)
Σ Majjau las
(p)
Σ
M Maj lul
15u,edd
Plugging the above expressions into equation
(77. we can represent (x( as folows:
Qx(X)= = Σ
exp((Bp..)) x) [xp( (I.,.(p..)) + (p..(-.)))) ) BB=
/1
Σ
exp((Bp..) T ( [L, + (p,)) B X + (B x) L3,((=,)B x]
Σ exp((Bp+.) T (p=H) Xf6. (X)
Σ exp( (M.n..p)) T T x (BB+BBB N BTT x]Se. (X)
L
MnJio. exp((Bp..)) T X)fG. (X)+ + 3=1
Σ
MaJios exp((prr) C)
1=1
1
2 R,, (Xx) (XX)
exp((Bp..)) x) ( (p-.) + L2,n(Po.)) B
Σ cxp((Bp.,) T x [ (P.s) + (p,)) T B X + (BT Lzn(p=,)B x
Σ exp((pp,.) T X) M..00 (P. "B" B X (X)
Σ exp((Bp..)) X) Mo. + +N..(p..))) BT x + +B x)T N2, (p.,)) (
fc. (X)
1 + R (X) Ry (xx (x
(28)
where Mnj,0s Esev, exp(bn..) exp(brs) for any j [L], (,n(p L.n(p=)) + Mn.j.o, Cp.,j. and (p,n)p..)) [,, + Mn..o, Cp.
Step 2 Non-vanishing coefficients. From equation
we can represent Qn (X)) as a linear combination of the independent functions exx((Bp..) x), (B X)
35
(()) (Bp.( X),
(B (3) x)()) exp((Bp=.)) X), exp((Bp..)) X).c. (X). (B X (w) exp (Bp..,) (X),
and (B (B x x)(e) exp(((p..) x)fc. (X) for any 1 S < L and 1 S u < d.
Assume that the coefficients of these linear independent functions in the formulation of Qn(X)/D2n go to O as n - Co. It follows that I (P=.)/ Li, ()/s)/D2m Lan(Pes) (Prn)(u)) /D2n. La (p.)))))) (wo)))(u)//Da /D2nn Ni., N1.(Pea)) /Dn. Ni,, ((p-j) (()) /Dzn N2, (10) (Dons and Mn.j.aa/ approach 0 as n + oo for any & u, v S and S $ L.
I2 (p=)))))) /
Then, as Mn..oa/ DDn + 0. indicates that
Ma..oal
Eiev, exp(bn,)) - cxpp(b))) + 00
Dzn
Dzn
22

---

Published as a conference paper at ICLR 2025
for any 1 $ L. By summing these limits up when varying the index j from to L, obtain that
Ej= LiEv, exp(bm.)) exp(b+,)) 0.
(29)
D2n
Now, we consider indices j [L] such that its corresponding Voronoi cell has only one element, i.e. [V = 1. As L (p-,)) DDn + O. it indicates that Mn.j.ae/22, + 0. It indicates that
Ee= exp(ln.)) (I
Eiey, exp( (mp(bn,i)l Aprniil + 0.
D2n
D2n
Putting the above results together, we find that
Em Eiev, exp(Im.i) Ap + 0.
(30)
D2n
Moving to indices j E [L] such that [V,] 1, as (s.n(P.)))un) /ww))//D (D2n - O, we obtain that
d Em exp(bn, LL3, (wv))
Eier, exp(bn.)) Appn,ill -
D2n
D2a
Therefore, we find that
EanDs Eiev, exp(bn.))) -
D2n
Collecting all the above results, we obtain that
D2n
D2n
as n + oo, which is a contradiction. As consequence, not all of the coefficients of the linear independent functions in the formulations of Q (X) to 0 as n + co.
Step Application of Fatou's lemma. In particular, let denote m, as the maximum of the absolute values of (p.,()).))/ DDn. L (p.,)) (()) /D2ns (p, /DDmnn (pn /D2n (a.n(p..)) ((w)))(w)) /'D2n Ni, ()= Ni. ((P.((P.))))))) (w)) D'22n No. (..((P)))(u)) D'2n and M Mn.j.aa/ D2n for all u, S d. From the result of Step 2, it follows that 1/mn oo as n + DC.
Recall that llfc fa. (w) DDn fe. /lle///// D2n) 0. By applying Fatou's lemma, get that
0 as
co, which indicates that llfo.
0 lim llfo. fc. fo.lle((m)
[a. (X) - Sc. (x) du(x)) 2 0.
lim
Mi, Den
m, D2n
S6 (X) fa.(X)
indicates that lim inf,,
O for almost surely X. As n - 00,
mn Dzn
denote
(pnn) - O.j
L2, - 8zj
m,, M2n
mh D2n
Li,n(p=.s) - +āj
L2,((pm5)
Lan (p-s) + 7/g
- Bi.
D2,
m, Din
m, Dan
Mn.5.0s
N1, (P..))
- 6,j
Dzn
Min D2n
NIn(P
N (P=
+ Br,
my, Dan
m. Dzn
23

---

Published as a conference paper at ICLR 2025
for any 1 < 1 L. Here, from the definition of Mny least coefficient among
{aj, 83, ag.
B3bbi,, =11 {āz, Bj. Yj, ag. 3j, Bobsi, is different from O. Then, the equation
[aa, (X) -fc. (x) 0 leads to
lim inf
mn D2n
X) 3 (B (B*
exp((Bp..) x) و + 3/ (B' x) + (B x (B x)]
Σ exp((Bp...) (āā -3 exp((B (B xi) (X)
j: ///1==1
- Σ exp((Bp..) x) [a + 3. ( x) + B (X) =0
for almost surely X. By denoting z B X, this equation also holds for almost surely Z. However, the new equation implies that all the coefficients {aj, Bj. āj B,),,, (ū T5, aj βj 3:13:111>11 are O, which is contradiction.
indicates that we indeed have the conclusion of the local part, namely.
lim 6-0
inf
Ilfc fc. (D G.)
Global part:
From local part, there exists positive constant e' such that
inf (Q)
llfe fc. fte.llean//PPC,... 0.
Therefore, sufficient to prove that
inf GEă (D)
Mc -
0.
Assume by contrary, then can find sequence of mixing measures G E exp(Unns)) (S2) such that as n + co, we have
D,(G.,.G))> Ge) { {1fec - fc. (wm G.) +0
which indicates that l.fa fa. fo. + as n + xo.
Recall that Q2 is a compact set. Therefore, there exists a mixing measure G' in 92 (92) such that one of G, subsequences converges G'. Since D2(G', G.) > e', we deduce that D2 G.) e By invoking the Fatou's lemma, we have that
/ [imiif Ja. - Sc. du(()). n
= lim |lfa; fc. fa.11e 2 no000
Thus, we have for = fa. for μ- u-almost surely X. From the identifiability property (cf. the end of this proof), we deduce that G' G.. follows that D2(G", G.) = contradicting the fact that (G', G.) g' > 0.
Hence, the proof of the global part is completed.
Identifiability property. We now prove the identifiability of shared strutures among prompts. In particular, we will show that fa( fa. (X) for almost every X, then it follows that G G..
For any & E (9)), let us denote
exp(u)
softmax:(u/) (u)
ΣN X A x + a + Σソ'
+ by))
exp(u.)
softmaxa. (u.)
Σn exp A X + og) + Σグ= exp(( T X + b..y))
24

---

Published as a conference paper at ICLR 2025
where
u E {X T AA + + (Bpr) X + by e [N], e [21),
u, E (X A/X + a3 (Bp..)) T X bay : [N].j' [4])
Since fo(X) fc. (x) for almost every X, we have
N
softmaxc X AIX + a.)) m Σ softmaxc((Bp,))
X + TX Cy
softmax (XT Afx + a.))n(X n7)) + softmax,, ((Bp...) X + berj) Cp..
(31)
Thus, must have that L, = T!' As result,
{softmax& ((Bpy) (Bpp)B)) for almost every X. Without loss of generality, we assume that softmaxa((Bp),
" X + by)) j E [L]] = (softmaxg, ((Bp..)) X + bagr) : : j [Z]],
+ by.) = softmaxe, (aftmaax.((BB..)))
+ ba.'):
for any j' E [L]. for almost every X. Since the softmax function is invariant to translation, this result indicates that by: b..z' + for some R and for any j' [Z]. Then, the equation 331 can be reduced to
Σ exp (x,))exp (xp()))))(Bp))) T = Σ exp (b.s) exp ((Bp..) X X)CP-
(32)
j=1
for almost surely X. Next, we will partition the index [L] into m subsets K1. Kg...., Kn where m & L, such that exp (bj) = exp (b..j") for any j.j' E K; and i [m]. It follows that exp (b₂) + exp (bagr) when j do not belong to the same set K,. Thus, we can rewrite equation 32 as
Σ
Σ
exp (b;) ((B,)) X)CP;
JEK,
m
Σ exp (b.,)) exp ((Bp=,) (X) (B)x,,) X)Cp..
JEK,
for almost surely X. Given the above equation, for each E [m], we obtain that
{((Bp,)T P,) : K,} (((Bp..)] .P.d)) :j K.).
for almost surely X, which directly leads to
{p. j E K,} [p-j : j E K,)
Without loss of generality, assume that pj P.. for all j K₁. Consequently, we get that
Σ
exp (b,) =
exp (b.,))pp..)
im JEK,
JER,
or G = G.. The proof is completed.
B.3 PROOF OF THEOREM
The proof strategy of Theorem is also similar that of Theorem 4.2 We first establish the parametric convergence rate Op( log(n) (m)/n) of the estimated regression function fä. the true regression function fG. in Section B.3.1 Then, in Section B.3.2 we establish the lower bound [lfa fa. (اا 2 C'D2(G, G.) for any G (E) for some universal constant C².
25

---

Published as a conference paper at ICLR 2025
B.3.1 CONVERGENCE RATE OF DENSITY ESTIMATION
Proposition B.2. Given the least square estimator Gn in equation (16). the convergence rate of the model estimation fä., (.) to the true model fü. (-) under the L2 (p) norm is parametric the sample size, that is,
Iaa - f/l.llle000 = Orl (r((log(n)/n))
(33)
The proof argument of Proposition B.2 is similar to that Proposition B.I therefore, is omitted.
B.3.2 FROM DENSITY ESTIMATION TO EXPERT ESTIMATION
Given the convergence rate of regression function estimation in Proposition our goal to demonstrate the following inequality:
inf Ge
||aa - (Ds G.) O.
Similar to the proof Theorem 4.2 we divide the proof of the above inequality into local and global parts.
Local part:
We will demonstrate that
lim - GE (...
inf
Assume by contrary that the above claim does not hold. Then, there exists a sequence of mixing measures G. :- Ey exp(bm.3') ((.,,, Wn..Pn,')) in Gr. such that as n co, we have
Dan == Dz G.) +
{
Illea - fa. /Da +
To ease the ensuing presentation, we also denote V == VV,(Gn) as a Voronoi cell of Gn generated by the j-th components of G.. Since our arguments are asymptotic, we may assume that those Voronoi cells do not depend on the sample size, i... V, = Vi- Therefore, we can represent the Voronoi loss Dan as follows:
Dan = Σ
Σ
exp(bm.s) - exp(bas')
==1
TEV
Σ
exp(bn.))llAW,, PPn.i ||AW, Pn..1
1 JE( 1-1 VEV
Σ
Σ
AW w.W.PPn.ii
, ,
where we define AW, AWn.i Wn.1Pm.i IiP.P and for all E Vy.
Wn,ePm,i W..PPe.j'
Additionally, since Dan + O, we have Eier, exp(bn.s) + exp(b..)) ,). Wn,1Pm, W.,IP-.i and Wn,PPa,s - W.2PP.. for any i V E [L]. Now, we divide the proof of the local part into three steps as follows:
Step 1 Taylor expansion. this step, we would like decompose the quantity
Qx( Σ exp(X x A +gg)+ Σ
(Bo x
(X)
(X) (x) (X)),
26

---

Published as a conference paper at ICLR 2025
as follows:
j= (x( exp(bn.)) (xp((
Σ
exp(bn.t) exp((Bo xx) -Csp((B (X) fa. (X)
j=liev,
exp(bn,,) exp(buss) exp((Bo, X) Co2 -fa (x)
3EV;
An (X) (X) + (X).
(34)
Decomposition of (X). To ease the ensuing presentation, we denote B(X;W1P) = (elBBo((W)))))))..... "X) and H(Wpp)
Co2(W2p). and F(X; (1X: W2p)
E(X; W1 (Wpp). Since each Voronoi cell Vj possibly has more than one element, we continue to decompose An as follows:
(x
Σ
Σ
exp(bm.s) F(X; Wn, PPn,i, 2n,)Po.)) -
Σ
exp(bm.i) F(X: (xpp(t,))(((((W..
Wa2Pm.) F(X; W .P.... W. W.22P..)
= A
(X) (X)
By means of the first-order Taylor expansion, we have
alal F (AW,, (Pnn)
E(X; B(X:Wa,)Pns) E(X: W W.,1P=5)+
Σ
O(Wpp)" (X; W.,IP..)) + (
|a|=1
0lai H LaWn.Pmnii 2Pnjj)
H(Wa,2Pn.) = HWW. + Σ (AW,..
O(Wp)) (W.,PP.)) + Roj.2,
|a|=1
for any i E Vi and such that [V,1 1. Here, Rig, (X) and Raj,2 are Taylor remainders. Putting the above results together leads to
exp(bn.a)
glal
(x) = Σ
Σ (AW
(X.
0(Wip)
5:11, =livV,
la|=1
ølai
+ (AW (AWm,Pmn.)))
(W.,pa..) E + Rx,((X)
d(Wpp)a
ala E
M ΣΣ
X W 1P.. H(
n.j.a
a(W (W)p))
JVV, =1
8|a H (
i pW2p)
(W. 2P ((x;
(x)
where 1 satisfies RM, (X) + 0 when oo, which is due to the uniform Lipschitz property of the function F. Furthermore, the formulations of M(1) ajja and M
n.j.a are given by:
(( = Σ
exp(bm.)) a!
(Am.,iPm.i))
VEV,
(2) n.i.a
exp(bn.i)
(AW,, "n.
a!
TEV,
27

---

Published as a conference paper at ICLR 2025
for any |a| 1.
Moving the term (X), by applying the second-order Taylor expansions E(X; in.iP around ( (P-.) and H HWW,,2Pn.) around H H(W.,pP-.) for any i E V3 and such that
[Vy] L, we obtain that
مlai E 1(Wip)
(x,)(X)=
E
E
M (
(X; (...,,,)))
ngg
Hla
+ M O(W2))a
(W, 2 ( W, .P..))
dla E
Σ
M Mn
[ W. Ip..)
(W..P...) (
a(W pp)
1,18/11
where (X) satisfies 2 (x) 0 when - cc. Furthermore, we define
exp(bm.t) (AWn,
Σ
n.j.a
a!
KEV,
M
exp(bn.t)
a! (AW,, 2n..is)
n.ja
iEV,
for any |a|| 2 and
exp(bn.i)
Mnja..8
P ?
a!!!
KEV;
for any la |3| 1. Direct calculation leads to the following formulations of the partial deriva- tives E Wip) and HW2p)):
06 X; WiWi)) exp((Bo X)
(Wip)) T x,
a(W .p) (a)
((Wp))(w) (m)
8
Po WW (Wip)) x
(X; WiW)p) = (W
(W x)
(a (n)
05
05
+ (B
(
(Wip))
((1p)) (v) (Wip) T X
@(W 1p)
0H
0ō2
a(Wep) (w) (Wap) C
(Wp)),
((Wpp) (u)
a w) ( (Wp)) C
(Wap).
(WWp)) (c) a (1) ((W
Given the above formulations, we can rewrite (x) and An. (X) as follows:
(x) Σ exp((Bo [X
BT (1, (1) (2, BT x) + 1X),
Ax
Σ
exp((Bo1((p.))) X [T (P..)) + La,n (p..) T B x
(BT x) La,n (P x + (X),
28

---

Published as a conference paper at ICLR 2025
where the formulations of the functions Li,ss L2, L1, Lz,ns and Ls,n are given by:
(p) =
M (2)
C
802
(W2p).
d(Wpp)(w)
==1
0ō
(p) Σ
M
n.j. ((Wip) (Wip)Cos(Wap),
00
(p)( Σ
C M an ((Wzp) ( 2p) (v) (Wap),
Isu,Dd
80
Σ
nijlun ((Wap)
M3
(W2p),
1 W (1)
L
Σ
M
0o
(Wip)Co>(Wap)
((W)
do2
0o
Σ [Manila. C Iga,edd
(p)
((Wip) (Wip)
O(W (u)
80.
M
Rijilu ((Wi)) (w)
W (ii)) Co2
(p) = Σ
M
n.j.lan 8(Wip) (Wip)
0o,
0o 8(Wi)) (Wip)) Co2(W2p).
Isuped
Here, we denote 1, the vector that its w-th element is while its other elements are 0 for any 1 S S d. Furthermore, Luv is the matrix that its (u, w) element 1 while its other elements are 0 for any S u, u S d. The second equation in the formulation of (1,)(p) is due to the fact that the 8002
function Ø2 is only applied element wise to Wap, which leads to ("Wp)) (W2p) = for all u A
Decomposition of B (X).
We can rewrite B, as follows:
(x(
Σ Σ exp(bn..)
E(X; Pn..) W(X:W.,,P..)) fo. (X)
Σ
exp(bns))
B(X; F. (X)
(x, + (X).
By applying the first-order and second-orde Taylor expansions, we get
Bx,((X) = Σ Σ
(X WP. (X) + Rn,3
O(Wip)
; -1 |=
a @W (X: W, I... (X) + (X)
Bn2((x)=
Σ
naa
where Rn,a(X) Rn, (X)) is Taylor remainder such that R,, X) Dan + 0, (X), +
when n + co. Therefore, we can express the functions Bn, (X) and (,,( as follows:
Bx) = Σ (xp((Bor(p_..)) |) x).1n(pp.))) TB
B X ffa (X) + R (X),
=
(x,((X) Σ exp((Bor(p..)) ()) Xx
(B X x) x|Sa, (X)
R Rn,((X);
29

---

Published as a conference paper at ICLR 2025
where the formulations of the functions Ni, Ni, and 2n given by:
0ō1
N1,
M n.
(WWPp) (Wip).
0ō M M ((Wip) (w)
(p) =
(Wip)
==1
0a
+
Σ
M
nuj.luv a(W (WW (Wip):
Igu,edd
do
0o
(p)
Σ Iguoedd
(1)
(Wip)
O(W p) (v) (Wip)
O(W 1p) (w)
Plugging the above expressions into equation (34. represent (x) as follows:
xn(X)= = Σ (B.. X) (P..)) + (p..)) _ B x)
Σ
exp( (B (W.,, IP..))))) [1, (X (g, BBB T X (B X) (p,n( x
HI,
Σ
XXN Xfa, (X)
Σ
( B + (B x)
(X)
>1
L
Mn.j.o. (B x)fa. (X)
Mn (xp (|)
- (
+
Σ
exp( x ( (on BT
Σ
exp((Bo1 X) ( ( T B X (B X) (T, X]
(p,n)
Σ exp((Bō,(W...P.)) X Mn. + (1, B X fa. (X)
J.
Σ cxp((Bō (W.,IP..))) X) [M [M + (1, B X x) N (p.)) X]Sa, (X)
,1V;> >1
(X) 2 Rn,3( (n.3((]] X) (X),
(35)
where Mn.,0s = Eiev, exp(bn,s) - exp(bmj) for any j E [4]. L1 (p..)) = Lin(p-s)) + Mn,,.0, (W., Co,(W.,PP..) and (p.)) = (p.)) + Mn..o, (FF2(...Po...
Step 2 - Non-vanishing coefficients. From equation (35 we represent Qu (X)DDss
linear combination of the following independent functions:
X), (B" x) exp( X),
(B ()) x (1) exp( (BoI(W..P..)) X), exp((Bo) (W.,P...)) T (X), T (X), (B T (") (B x (v) (B X)fa, (X)
(B x)
30

---

Published as a conference paper at ICLR 2025
for any j & L and u, u d.
Assume that all the coefficients these linear independent functions in the formulation of (n)) go to O as n 7 OO. It follows that (P=.)/ Lzn (P=j) /D [p, /Dn L (p= /Dan. La,, (p+.))((mm)) /Dan. N1, (P,n(P=))/D0 DDns V1,n((=..) /Dan. N2, (p...) ) /Dani and Mn.j.aa/ DDyn approach 0 as n - oo for any u, v S and S S L.
Then, as Mn.j.aa/ Dn + 0, it indicates that
Maj.o.l E,vv, exp(bn.)) exp(on:)-exp(h..))) exp(br,))
+ 0,
D2n
Dan
for any S.j S By summing these limits up when varying the index j from L, we obtain that
Σ Eiev, exp(b..)) -xp(bt_))) + 0.
(36)
Dan
Now, we consider indices j [L] such that its corresponding Voronoi cell has only one element, i.e. 1V,l = 1. As L2, (P+,)) (w)/) /D3n - and the first order derivatives of ō1 are non-zero, it indicates
that ( /Dan + It indicates that
Euzn [M n.j.l
Eiey, exp(bn,))lAW,, I..,.....l O.
D2n
D3n
Σiev, exp (mp(bn,))) AlW...PP..)) + 0. Putting the
Similarly, L1, (P.,(P= + 0 also leads to above results together, we find that
D3n
EApiE Eiev, (Ip(,,,))IAVV n. Pnniill + |AW n2Pniil 0.
(37)
Dan
Moving to indices j E [L] such that |Vyl > 1, as Ls,n (Bn((P))) /D3n - O, we obtain that Eiey, (.Wbnn.)lAAW......s
Ed= Ls.n (p..))
0.
Den
Dan
Likewise, as (p.,,,,, (/wu)/Dan /D + and the second order derivatives of 02 are non-zero, we also obtain that Diev, exp(bn.)) (m, Al n2.nill
O. Therefore, we find that
D3n
ΣE Eier, exp(bn. AW 1P., an 0.
Dan
Collecting all the above results, we obtain that
1 Dan
D3n
as oo, which is a contradiction.
As consequence, not all of the coefficients of the linear independent functions in the formulations of @ (X)) go to 0 as n + 0o.
Step 3 Application of Fatou's lemma. Let us denote m, the maximum of the absolute values of (P.s)) L2, (p=.)) (()) DDDnn L (P..P)) I2,n( (pa.) DDanss (P.,)) (uv) DDD D/nn N1. ((...) (w)
N1. (1.n(p.))/ (Px
(u)) /Dan. Ne. (2.(p).))) (ww)) /Dan. and Mn.j,0a/ Dan for all u, S d. From the result of Step 2, it follows that 1/m, / 30 as n >
Since llfe. fe. (1) /Dan + as +
obtain llfa. - SG. (i (112 (m, Dan) 0. By
applying Fatou's lemma, we get that
fö. (X) fū. (x)
Sz. (122(w)
lim
2
lim n-
du(X)) 2
n-
m, Dan
m. Dan
31

---

Published as a conference paper at ICLR 2025
fa. (X) - faa (X)
Therefore, lim inf m-000
O for almost surely X. As -> oo, we denote
my 22n
(p-j))
12 [p-
- ''j+
+ 8zz
m, D3n
m, D3n
1/ (P-s))
L (p=.) Bjz
Lg,1 (p+,) 7/3
m, Din
mn D3n
mn D3n
Ma.1.0
(P.n))
- +aj.
B3-
Dan
Dnn
(P.)( + B3
N2 (pa.)) + 93
m, Dan
Mnn
for any 1 < j 1 Here, from the definition of Mas at least one coefficient among
{aj, 8g āy. By Bybsi,, =11, {ā,, Bg. Hz, āg. B,, fy is different from 0. Then, the equation
0 leads to fā. (X) fa. (X)
lim inf
MnD
Σ
(W X) + (8, x))
(
X (a, B] (B x) + (B x) x
1
Σ
3J (B x))fe. (x)
#P/=
(W.. X) āū 3 (B x) + (B x (X) =0
for almost surely X. By denoting 2 = B x, this equation also holds for almost
surely Z. However, the new equation implies that all the coefficients (a,, Bj ã. B, Dip,, (ūr āz B, Jo), are which a contradiction.
As consequence, we obtain
lim inf TI (E
lifa - fa. (e 0.
Global part:
From local part, there exists a positive constant e' such that
inf ())
Mefe fā.
Therefore, it is sufficient to prove that
llfe - fa. G D..
inf
G.) 0.
Assume by contrary, then Σ/' exp(bm.j (6 p... W...P... in T.. (3) such that as n + oc, we have
find sequence of mixing measures G
Dz G.) >E'
1110: -f. ( (D G.) + 0,
which indicates that Ifār fa. 11<2(H) + 0 as
Since compact set, there exists a mixing measure G² in ()) such that one of G 's subse- quences converges to GÖ' Since Dz D2GG G.) > E', we deduce that Dz(G", By invoking the Fatou's lemma, have that
GaGGG,GG, G.) >
lim l|fe: fa. (1) 2 f 2-000
inf fay (X) - fxa(() du(X). 2
32

---

Published as a conference paper at ICLR 2025
Thus, have fö fā_ for j-almost surely X. From the identifiability property, we deduce that G.. It follows that D3(G", G.) = 0, contradicting the fact that Dz(G", G.) > e' > 0. Hence, the proof is completed.
Identifiability property. We now prove the identifiability of one layer neural network structures among prompts. particular, we will show that fā(x) = fa. for almost every X, then follows that G = G..
For any G (3) and G., us denote
exp(u)
softmax&(1)
Ex exxpxXXXXxXXX A&X + ag) + (WPP(( by))
exp(u.)
softmax (u+)
Σk= exp A + a.) + Ej'= ( X + b,,)))
where
M E (X 19 x + a): (Bō (WiP;)) X by E E [T]],
U+ {X A + af. (Bō (W. 1P-33'))) X b : [M,,]' e [4]]
Since fx(X) = fc. (X) for almost every X, we have
softmax G ( A a,)) (X, 73) +
(Bftmax;((oo X by Co2
1=1
N
softmax C. (X ATA + ())) n
softmax ((Bō) (W (P),, Tx b., Coo()),, f'=1
(38)
Thus, we must have that L'. As a result, we obtain that (Bo X + by) : j [E]}
= [softmax ((Bō) (W..IP-.)))) X + b E [L]},
for almost every X. We may assume that
softmaxxx(Ba,, (WiP;) Wip;)) X + by) = softmax
(W.....,))) X + ba.j))
for any j' E [L]. for almost every X. Since the softmax function is invariant to translation, this result indicates that by = ba,z' + r for some r E R and for any j' E [Z]. Then, the equation 38 can be reduced to
exp (by) exp (Bo,(Bi)))) C)Coo(W)p,)
exp (b..))
exp exp
())o,)))))) (39)
for almost every X. Next, we will partition the index set [L] into subsets K1, K2... Km where
m < L, such that exp (by) = exp (bx.gr) ) for any j.j' E K; and i [m]. It follows that exp (by) * exp (bx.y) when j.j do not belong the same K₃. Thus, we can rewrite equation 33 m
cxp(b,) (b,) exp (BBo, (Bo,, (WiP,)) X)Co)(Wep))
JEK,
exp (b.,) exp ((B XX)C)
1= jEK;
33

---

Published as a conference paper at ICLR 2025
for almost surely X. Given the above equation, for each E [m], we obtain that (((Bō)(W1p)) Wap;) :j Ki) {((Bo) (W....) W. W..pP.)):
Ki),
which directly leads to
(Wip): : e K,) [W.,IP.. :j Kg} and {W2p) :j E Ki} {W+,2P., : K,).
Without loss of generality, we assume that Wipj = W..PP..j and W2P; = W.,pP., for all j E K; Consequently, we get that
Σ E exp (b,) (ap) =
Σ Σ
exp (b.. (b.pb,))(.........a....
JEK,
i= ic
which implies that G = G.. As consequence, the proof is completed.
C ADDITIONAL PROOFS
In this appendix, we provide proof for the convergence rate of regression function estimation.
C.I PROOF OF PROPOSITION B.
To start with, is necessary to define the notations that will be used throughout this proof. First of all, let denote by FL (S) the set of regression functions w.r.t mixing measures in G2, that is,
Fi) = (Jo(X) : G E (9)).
Next, for each 8 > O, we define the (p)) ball centered around the regression function fa. (X) and intersected with the FL. (12) as
(2, {f (z) Ilf fc. (1)) S 56)
Furthermore, van de Geer (2000) suggest capturing the size of the above set by using the following quantity:
(Q, = 1
6
H (f,, (l)) dt V 6,
(40)
888/218
in which (S2,t)), 1+1l,=()) denotes the bracketing entropy van de Geer 2000 of (9,, under the (no and / V 8 == max(1,
Subsequently, let us introduce a key result of this proof in Lemma C.I which achieved by applying similar arguments those in Theorem 7.4 and Theorem 9.2 in van de Geer (2000).
Lemma C.I. Take (()) 2 J8 Fr (2,,6) 6)) that satisfies &e (8)/82 is non-increasing function of Then, for some universal constant c and for some sequence (6,) such that v vno² 2 cV/ (8x),, achieve that
P Pll5c. fc. > >6) s ceep
no² (-a
for all 8 2 6n-
General picture. We begin with deriving the bracketing entropy inequality
Hr (1xa(w) & log(1/s).
(41)
for any 0 < < 1/2. Then, it follows that
8
/ log(1/t)dt v (42)
1/2 21, (t, (0,t), (1)))) dt dov 88/221
8
Let (1)) = 8 (10) then (8)/682 is non-increasing function of 8. Additionally, equa-
tion 42 indicates that V1))) > Jn Fr 72, 8)). Moreover, by choosing 8, Vlo((n//n,
34

---

Published as a conference paper at ICLR 2025
have that Vno2 2 (6,) for some universal constant Then, according to Lemma the conclusion of Theorem B.1
C.1 reach
As result, it sufficess to establish the inequality in equation (41)
Proof of equation (41). Let fe be an arbitrary regression function in ( As the prompts pj' both bounded, we obtain that (ff(())
M for all X where M 0 is some universal constant.
Next, let < E and (T.....,TT Tx² be the r-cover under the - norm of the set Fg,(() in which N := (I |l (l))())) is the 7-covering number of the metric space (Fi( ll (l))))))-
Then, construct the brackets of the form [L, (X), (X)) for all E [M] as follows:
(xX max [T X)) T,0), O); (x(X) == max T. [X) + M}
can be verified that (1) C (
Furthermore, we also get that
I0; - -U:--illaGGa Li)l
L())
< (f(rduu(x)) 1/2 = 2T,
From the definition of the bracketing entropy, we have that H2 FL. (9), (1r=(w) S log N log (LFFFF -1.-))))))
(43)
Thus, it is sufficient to establish an upper bound for the covering number N. For that purpose, we denote = {(h, p) E x Rd (b. p) E @}. Since fi is compact set, A is also compact. Thus, there exist --covers for A, denoted by A,, respectively. Then, we find that
| S O (d) ((T-((+)))))
For each mixing measure G = E exp(b:)op. E (9)),, we consider a corresponding mixing measure & defined as
b= Σ exp((:))pp.
where (b,, p.) E A, is the closest to (b,, P,) in that set. Let us denote
N
L'
D(X) : Σ exp(x
1 + a +
Σ
exp((Bp,,)' + byy),
j'=1
L'
D(X) : = Σ exp(X 1 + a + Σ
cxp((Bp,))TX + by:).
='=1
Subsequently, we aim show that llfe - fc 00 E T. In particular, we have
xxp((Bp,) TX X + b;). CDp
exp((BD,) X + +b)) Cp.
[l Solloo S sup XEX
((X)
D(X)
Σ exp A9X + (a))n(X nB9) (
1 D(X))
+ sup x EX
D(X)
exp((Bp,) ) T b;)
C(p, - -Bi)
sup X
D(X)
(xxp((B)) / Tx bb)) exp((Bpi,) TX TX + +b,i]
+ sup XEX
Cpp
D(X)
D(X)
Σ 1=1
exp(X xxp(XTAXx ASX + (9)m(xX.)))) ( ((x))
+ sup XEX
T, T2 Ts.
35

---

Published as a conference paper at ICLR 2025
Then, it is sufficient to demonstrate that Ti S T and T2 S r. respectively. First of all, we get that
4' exp((Bp))
T TX bb,)
Ti sup Xex
C(D; - pi))
D(X)
j=1
1/ C(D; - -Di) SC.
1¹'
< sup xex
Σ
Σ /=1
[pj - أاره S CL''S
j==
Here, the first inequality occurs as the softmax weight is bounded by 1. Next, we have
1/'
exp((Bp)) T bx) exp((Bp,) T + bb,) Cpp
T2 = sup XEX
((X)
D(X)
L'
exp(((p)) TX +b,)) exp((Bp,) TX
+ +b,]] CØ:
XEX
sup
((X)
((X)
L'
(T) Tx [X+bi) D(X) D(x))
sup XEX
exp((Bp))
L'
exp( TX x by) ex)))x))((BB)))) x + bز}]
sup XEX
(44)
((x)
Note that since both the input space X' and the parameter space Ω are bounded, the product ((X) is bounded. Thus, we have
1 1 D(X)
S
D(X)
L'
exp((Bp))) TX + by)
exx((BBr) [x + by>
2 Σ
[lpy py | + 1b3 by.1]
y'11
< B + 1)T TT
where B is the bounded constant of ||
|. As result, we deduce that
1
1 D(x)) S D(X) D(X)
1 ST.
exp((Bp,)T Tx + by)
((X)
x + b,) - exp((Bp)) T X + by)] $ [ID, - p,ll ||||| + |b, - [T. ST.
Therefore, it follows that T2 S r. Additionally, we also have
N
T3 S sup XEX
D(X) D(X) (cxp(XTAxx) a))((x
1
j=1
$
sup XEX
exp(X (ooxxAAX-- + ag)h(XX .n3)| ST.
As a result, we achieve that
I18 fallx < Ti + T2 Ts S T.
By definition of the covering number, we deduce that
(T),FL 112=~11~>>))>>>)>1 |
< On (d++))E).
(45)
36

---

Published as a conference paper at ICLR 2025
Combine equations equation 43 and equation 45
achieve that
HB(2T,FL (()))) x log(1/r).
Let E/2, then we obtain that
HB(e,FL (Ire()) & log(1/E).
Hence, the proof is completed.
D RELATED WORK
Carameter-ffficiee Fine- ToninT Full fine-tuning is a common approach for adapting pre-trained foundation models specific downstream tasks. However, this method requires updating all model parameters, which leads to high computational costs and the need to store a separate fine-tuned model for each task. As more efficient alternative, parameter-efficien fine-tuning (PEFT) has emerged to address these limitations (Xin et al. 2024 Li & Liang. 2021; Hu et al. 202T). PEFT updates only a small subset of parameters, offering the potential to achieve performance comparable to, or even exceeding, that of full fine-tuning, For instance, LoRA (Hu et al. 2021) approximates weight updates through low-rank matrices that are added to the original model weights, while Bitfit (Zaken et al. 2021) modifies only the bias terms, freezing all other parameters. Adapters (Houlsby) cct al. 2019) introduce lightweight modules into each Transformer layer, and SSF (Lian et al. 2022) employs scaling and shifting of deep features.
Prompt-based techniques. Unlike the previously discussed methods of fine-tuning backbones, prompt-tuning (Lester et al. (02I) and prefix-tuning (Li & Liang 202I) introduce learnable prompt tokens into the input space. These tokens are optimized while the backbone model remains frozen, offering substantial computational efficiency. Despite its apparent simplicity, prompting has demon- strated notable performance improvements without the need for complex module-specifie designs (Liu et al., 2021). VPT (Jia et al.,, [022) extends this idea to vision tasks by introducing tunable prompt tokens that are prepended to the original tokens in the first or multiple layers. Additionally, (Levine al. 2022) introduces input-dependent prompt tuning, which generates prompt tokens us- ing a generator. SPT (Zhu & Tan (003)) proposes a mechanism that automatically determines which layers should receive new soft prompts and which should propagate prompts from preceding layers.
Analysis of prompt-based techniques. Recent research has increasingly focused on understanding the theoretical foundations that drive the success of prompt-based methods, aiming to uncover the underlying mechanisms responsible for their effectiveness. For instance, He et al. (202T) investigates and adapters, while Le et al. (2024) examines prefix-tuning
the relationship between within the framework of mixture of experts models. Additionally, Petrov et al. (2023) explores the limitations prompting, demonstrating that it cannot change the relative attention patterns and can only bias the outputs of attention layer in fixed direction. Unlike these prior works, our study delves into the theoretical principles behind key implementation techniques, particularly reparame- terization, that enable prefix-tuning to achieve competitive performance.
Mixture of Experts. Building on the foundational concept of mixture models (Jacobs et al. 1991 Jordan & Jacobs 1994), prior works by [Eigen et al. (2014); [Shazeer et al. (207)) established the MoE layer as key component for efficiently scaling model capacity. MoE models have since gained widespread attention for their adaptability across various domains, including large language models (Du et al. 2022; Zhou et | 2023), computer vision (Riquelme et al. 2021 Puigcerver et al.) 2023 and multi- learning (Ma et 2018). Recent studies have investigated the convergence rates for expert estimation in MoE models, focusing on different assumptions and configurations of gating and expert functions. [o et al. (2022), assuming data from an input-free gating Gaussian MoE, demonstrated that expert estimation rates for maximum likelihood estimation depend on the algebraic independence of the expert functions. Similarly, employing softmax gating, NNuuen et al. 2023 2024a) found that expert estimation rates are influenced by the solvability of polynomial sys- tems arising from the interaction between gating and expert parameters. More recently, Nguyen et al. 2024c b) utilized least square estimation to propose an identifiable condition for expert functions, particularly for feedforward networks with nonlinear activations. They showed that under these conditions, estimation rates significantly faster compared to models using polynomial experts.
37

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation metrics for each dataset.
Datasets
Task
Metrics
FGVC
Image classification
Accuracy
VTAB-IK
Image classification
Accuracy
E2E
Table-to-text generation
BLEU, NIST, METEOR, ROUGE-L, CIDEr BLEU, METEOR, TER
WebNLG
Table-to-text generation
XSUM
Summarization
ROUGE-1, ROUGE--2 ROUGE-L.
Table Specifications of datasets evaluated for visual tasks. Following Jia al. (2022), we ran- domly sampled the train and val sets since there are public splits available.
Dataset
Description
# Classes
Train
Test
Five-gruined visual recognition tasks (FGVC)
CUB-200-200 (Wah al. 2011
Fine-grained bird species secognition
200
5,394
600
5.794
NABirds (Van Horn et 2015
Fine-grained bird species recognition
55
21,536
2,393
24,633
Oxford [Nilshack 7iseerman 2008
graiggrannd flower species recognition
Stanford Dogs (Kiosiia
2011
Fine-grained dog species recognition Fine-grained car recognition
120
10,800
1,200
8.580
Stanford Cars Gebru al. 2017
196
7,329
815
8,041
Visual Task Aduptation Benchmark (VTAB-IK)
CIFAR-100 Krizheysky et 2009
Caltech101 Fee--Fer of 2006
102
6,084
DTD (Cimpon et al. 2014
1,881)
FlowerstUZ (Nu1bbcc
2008
Natural
102
800/1000
Pets (Parkh & al 2012
37
3,669
SVHN Netzer et nl. 2011
26,032
Sun397 Xiao
397
21,750
Putch Camelyon Veeling
2018
2
32,768
EuroSAT (Helber et al 2019
10
5,400
Specialized
800Л100
200
Resise45 Cheng el
Retinopathy Grahia
5
42,670
Clevricount (Johnson et al
2017
8
15,000
Clevridistance Uolmson e1
6
15,000
DMLab [Batticc
2016
6
22,735
KITTidistang (Gerggr el ai
2013
4
711
Structured
800/1000
200
dSprites/loc Mattley at
73,728
dSprites/ori Mutthey
73,728
SmallNORBaaai
Lct el 21
2004
18
12,
SmallNORB/ele
LeCun et
2004
9
12,150
E ADDITIONAL. EXPERIMENTAL DETAILS
E.1 DATASETS DESCRIPTION
Table summarizes the details of the evaluated datasets for visual tasks. Each VTAB-1K task contains 1,000 training examples. We follow the protocol from VPT (Jia et 2022 to perform the split of the train, validation, and test sets.
For language tasks, we employ E2E (Novikova et al. 2017) ) and WebNLG (Gardent al. 2017) for table-to-text generation. The E2E dataset comprises approximately 50,000 examples across eight distinct fields, featuring multiple test references for each source table, with an average output length of 22.9 tokens. The WebNLG dataset contains 22,000 examples, where the input consists of sequences of (subject, property, object) triples, with an average output length of 22.5 tokens. For summarization, we utilize the XSUM dataset (Narayan et al. 2018), which is an abstractive summarization dataset for news articles. This dataset contains 225,000 examples, with an average article length of 431 words and average summary length of 23.3 words.
38

---

Published as a conference paper at ICLR 2025
Table 5: Per-task fine-tuning results for VTAB-1k benchmarks. We report the average accuracy
five independent runs. Best results among all methods except Finetune are bolded.
Method
Natural
Specialized
Structured
Pach Cameeyoo
Clevrititanes
SmallNORB
SmallNORBBCC
CIFAR- 100
Caltechiil
BurONAT
Rosseks
Reiiopadyy
Clevreomn
DTD
SVHN
Sun
DML
Finetune
68.9
$7.7
64.3
97.2
$6.9
87.4
38.8
79.7
95.7
84.2
73.9
56.3
58.5
41.7
65.5
57.5
46.7
25.7
29.1
76.8
62.4
50.5
78.6
90.7
75.7
73.7
39.2
55.2
35.4
55.5
35.8
24.4
No-dharequuaioo
78.7
90.5
71.1
Deep-s heop-shareusu
75.5
90.7
65.4
96.6
86.0
78.5
46.7
79.5
95.1
80.6
74.0
69.9
58.2
40.9
69.5
72.4
46.8
23.9
34.4
No-slaveecce
70.0
88.5
62.2
96.7
85.3
43.5
45.8
78.0
93.4
75.7
73.9
11.5
55.0
34.1
60.0
39.6
31.9
15.4
21.0
E.2 IMPLEMENTATION DETAILS
In visual tasks, we preprocess the data by normalizing it with ImageNet's 's mean and standard devia- tion, applying a random resize and crop to 224 x 224 pixels, and implementing random horizontal flip for FGVC datasets. For the VTAB-1k suite, we resize images directly to 224 x 224 pixels. Following Jia et al. (2022), we perform grid search to determine optimal hyperparameters, specif- ically learning rates from the set [50, 25, 10, 5, 2.5, 1, 0.5, 0.25, 0.1, 0.05] and weight decay values from [0.01, 0.001, 0.0001, 0.0], evaluated on the validation set for each task. For prompt length, select Np ensure the number of new prefix experts within each attention head corresponds to the optimal prompt length established by Jia et al. 2022). The SGD optimizer is utilized for 100 epochs, incorporating linear warm-up during the initial 10 epochs, followed by a cosine learning rate schedule. We report the average test set accuracy across five independent runs, maintaining consistent batch size settings of 64 and 128. All experiments were implemented in PyTorch (Paszke et al. 2017) and executed on NVIDIA A A100-40GB GPUs.
In our experiments with language datasets, we adopt the hyperparameter configuration proposed by Li & Liang (022)). which includes the number of epochs, batch size, and prefix length. For the learning rate, we conduct grid search across the following values: [le 5e 2, le 2, 5e 3, 3ele,,, 4, le-44, 5e-5, le-5). 5]. During training, we utilize the AdamW optimizer with a linear learning rate scheduler. For decoding in table-to-text datasets, we implement beam search with a beam size of 5. For summarization, we employ beam size of 6 and apply length normalization with a factor of 0.8.
F ADDITIONAL EXPERIMENTS
F.1 PER-TASK RESULTS ON VTAB-1K
Table summarizes the results for each task on VTAB-IK. Across most datasets, either Deep- sharepsep Deep-sharessanT.n consistently achieves the highest performance, often comparable to full fine-tuning. While prefix-tuning slightly underperforms full fine-tuning on some datasets, its average accuracy remains competitive. These results underscore the effectiveness of reparameteri- zation in enabling prefix-tunins to perform on par with full fine-tuning. Additionally, Decp-share configurations significantly outperform No-share settings on most datasets. For instance, on SVHN, Dhep-share outperforms No-sharogeat.ow by 32%, and on Clevr/count, Deep-sharopear ex- ceeds No-shareuzas by 28.4%. These findings emphasize the critical role of reparameterization, highlighting the benefits of shared structures over non-shared configurations.
F.2 PER-TASK RESULTS ON FGVC
Table 6 presents the detailed results for each task in the FGVC dataset, visualized in Figure 2 Across all FGVC tasks, both the Simple-share and Deep-share methods consistently outperform the No-share baseline. For example, on the Stanford Cars dataset, Deep-sharegeep and Simple- sharepeap exceed the No-share baseline by 16.8% and 20.3%, respectively. Additionally, these methods lead to significantly higher average accuracy, surpassing the No-share baseline by 5.96%
39

---

Published as a conference paper at ICLR 2025
Table 6: Per-task fine-tuning results for FGVC benchmarks. We report the average accuracy five independent runs. Best results among all methods are bolded.
Method
CUB-200-2011
NABirds
Oxford Flowers
Stanford Dogs
Stanford Cars
Mean Acc
Deep-sharegaaaLom
87.2
81.5
98.6
91.1
84,36
Simple-sharessaczee
86.6
79.3
98.4
90.8
82.10
No-sharesuauaon
85.1
77.8
97.9
80.38
Deep-sharegaae
87.8
84.5
98.2
88.28
Simple-shareeree
88.7
84.3
98.8
90.6
82.8
89.04
No-sharcuuar
83.9
79.0
97.9
86.3
62.5
82.32
Table 7: Comparison of fine-tune tieettning results between common techniques on FGVC and VTAB-IK.
VTAB-IK Specialized
Method
FGVC
Natural
Structural
Finetune
88.54
75.88
83.36
47.64
Partial-1
82.63
69.44
78.53
34.17
Adapter
85.66
70.39
77.11
33,43
VPT-Shallow
84.62
76,81
79.66
46.98
VPT-Deep
89.11
78.48
82.43
54.98
No-sharegmatton
80.38
69.00
77.20
29.65
No-sharegzeF
82.32
70.29
80.20
37.69
Deep-sharesuaLLon
84.36
75.79
79.48
38.53
Deep-shareoee 88.28
77.06
82.28
52.00
and 6.72%, respectively. This substantial improvement underscores the empirical effectiveness of leveraging shared structures enhance prefix-tuning performance. Notably, Simple-sharensep achieves the highest average accuracy among all methods, even surpassing full fine-tuning and Deep- share. However, the theoretical comparison between Simple-share and Deep-share remains open question and is left for future investigation.
As shown in Figure and Table 6 the performance of Simple-share and Deep-share varies depend- ing on the task and implementation variant (i.e., SHALLOW and DEEP), with Simple-share some- times surpassing Deep-share and vice versa. Simple-share consistently shows strong performance relative to No-share and remains competitive with Deep-share. Moreover, Simple-share offers more parameter-efficient approach, does not require the additional MI.P used in Deep-share for reparameterization, thereby simplifying the shared structure's implementation. In contrast, Deep- share's use MLP for reparameterizatioo introduces greater flcxibility, which may offer benefits in specific applications. Nonetheless, more thorough thcoretical and empirical comparison between Deep-share and Simple-share remains an open question, which we propose for future research.
F.3 COMPARISON WITH OTHER FINE-TUNING TECHNIQUES
Table and Table g present comparative analysis of prefix-tuning against common fine-tuning techniques. In the vision domain, prefix-tuning demonstrates competitive performance, achieving results comparable to full fine-tuning and surpassing several alternative methods, though it slightly trails behind VPT. No-share, however, shows significantly weaker performance compared VPT, underscoring the importance reparameterization in enhancing deefix-tuning' 's effectiveness. Sim- ilarly, in the language domain, prefix-tuning delivers strong results, with reparameterization once again playing crucial role its success relative to other fine-tuning approaches.
F.4 VISUALIZATION OF PROMPT EFFECTS
To evaluate the effectiveness of the prompt techniques, we conduct a visual analysis to examine the areas of the input data that these techniques guide the model to focus on. Specifically,
40

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of fine-tuning results between common techniques F:E: and WebNI.G.
E2E MET
WebNLG MET
Method
BLEU
NIST
R-L
CIDEr
BLEU
TER
S
A
S
U
A
S
A
Finetune
68.2
8.62
46.2
71.0
2,47
64.2
27.7
46.5
0.45
0.30
0.38
0.33
0.76
0.53
Partial-2
68.1
8.59
46.0
70.8
2,41
53.6
18.9
36.0
0.38
0.23
0.31
0.49
0.99
0.72
Adapter
66.3
8.41
45.0
69.8
2.40
54.5
45.1
50.2
0.39
0.36
0.38
0.40
0.46
0.43
No-share
68.0
8.61
45.8
71.0
2.41
61.1
42.8
53.5
0.43
0.35
0.40
0.36
0.49
0.42
Deep-share
69.9
8.78
46.3
71.5
2,45
63.9
44.3
54.5
0,45
0.36
0.41
0.34
0.52
0.42
Input
No-share
Deep-share
Input
No-share
Decp-share
2011200
NABirds
Oxford Flowers
Stanford Dogs
Stanford Cars
Figure 3: GradCAM visualization of Deep-share and No-share FGVC tasks. Red regions indi- cate high-class activation scores. From left to right: input image after standard data augmentation, GradCAM output for No-share, and GradCAM output for Deep-share.
GradCAM (Selvaraju et al. 2017 to generate attention maps by computing the gradients of target concept with respect to the model's final layer, highlighting the salient input regions most influential in predicting the target concept. Figure illustrates examples from FGVC benchmarks, comparing heatmaps generated for Deep-share and No-share.
The visual differences between the two approaches reveal that Deep-share more effectively guides the model to attend to relevant image regions. For instance, in dog images, while No-share strug- gles to identify the entire structure of the dog, Deep-share successfully localizes and highlights key structural features. This ability to localize salient patterns demonstrates the improved capacity of Deep-share capture meaningful visual information. These findings suggest that the reparame-
41

---

Published as a conference paper at ICLR 2025
terization employed by Deep-share not only enhances the effectiveness of prefix-tuning but also improves the model's 's interpretability by providing more coherent and accurate visual explanations.
42


Reviews:
[{"review_id": "GCoEpHA8dz", "reviewer": "Reviewer_KNxA", "summary": "This paper studies the role of reparameterization in the context of prefix-tuning within foundation models. The authors provide a theoretical study that demonstrates that reparameterization induces a shared structure between prefix key and value vectors in order to attain strong performance. They also include experimentation of language and vision-related models. Furthermore, they also tie the findings to mixture-of-expert approaches and prompt-tuning approaches.", "questions": "[Numbered per Weaknesses above]\n1. The work would be strengthened by more thorough discussion of potential limitations of the problem simplification within the theoretical analysis section.\n2. How does results presented in Section 5.2 translate to larger scale systems?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is clearly written and includes useful foundational knowledge to help set-up the theoretical framework.\n* The work studies both the case of when prompts contain shared structures and when they do not, providing extensibility to the framework\n* The theoretical analysis builds usefully from a linear setting to a one-layer setting. \n* The work contains a broad set of experiments across multiple vision and language oriented datasets. In addition, the experiments study prefix tuning under different kinds of fine-tuning.", "weaknesses": "1. The beginning of Section 4 highlights the simplification of the theoretical analysis (\"we focus only on the first head, namely, l = 1 in equation (6), and the first row of the attention in this head, namely, i = 1 in\nequation (6).\"). The work would be strengthened with greater discussion of how this simplification might provide limitation in the extensibility of the analysis\n2. The work would be strengthened by discussion of how included results on somewhat small-scale set-ups (trained on ImageNet-21k or older versions of GPT-2) might translate to larger scale models and tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkjSPrZdV7", "reviewer": "Reviewer_d1Nj", "summary": "The paper investigates the reparameterization strategy in prompt tuning based on the mixture-of-experts model. By providing theoretical analysis on prompt learning and prefix tuning, the paper proves that the shared structure, e.g., reparameterization among prompts, helps considerably improve the sample efficiency of prompt learning. The experiments on both visual and language tasks demonstrate the theoretical analyses.", "questions": "1. I found the notation $N$ in the paper confusing in the paragraph under equation 5. It is used for both the number of the input embeddings and the number of the experts. Does it means the numbers of the embeddings and experts should be the same? Otherwise, it is better to replace on of them with another notation.\n\n2. What's the basic architecture of the MoE model in the paper? Is it also conducted by prefix tuning with the reparameterization strategy? How are the experts pretrained in the experiments?\n\n3. In Figure 2, sometimes the simple-share method is better than the deep-share method. Any discussions on the selection of the methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides detailed theoretical analyses of the reparameterization strategy for prompt learning.\n\n2. The experiments on several tasks demonstrate the theoretical analyses.", "weaknesses": "1. Although the paper provides detailed theoretical proof of the effectiveness of the reparameterization strategy in prefix tuning, my main concern is about the technical contribution of the paper. Since both the reparameterization strategy and the mixture-of-experts are already investigated in previous works, the technical contribution of the paper seems a bit weak.\n\n2. The theoretical analyses are provided based on the mixture-of-experts models. However, I didn't find the necessary connections between the MoE model and the reparameterization strategy on prefix tuning. Why do the proofs need to be analyzed on the MoE model? Do the proofs still be held without the MoE model? And is there any empirical results to demonstrate the proofs without the MoE architecture? Relying on the pretrained MoE model will also lead to increased costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gDpnOt8bJ8", "reviewer": "Reviewer_jyvW", "summary": "This study primarily explores the theoretical foundations and effectiveness of prompt-based techniques, such as prefix-tuning and prompt-tuning. The authors point out that the key to achieving performance parity with full fine-tuning lies in the reparameterization strategy, demonstrating that this strategy implicitly encodes a shared structure between prefix key and value vectors. Through extensive experiments in both visual and language domains, the research empirically confirms that the effectiveness of prefix-tuning across various tasks benefits from this shared structure. Additionally, the authors find similar structural advantages in prompt-tuning as well.", "questions": "1. It is recommended that the authors include more intuitive visual analyses, such as which areas the prompt techniques guide the model to focus on, to help readers better understand the effectiveness of the prompt techniques.\n2. To improve the reproducibility of the experiments, it is recommended that the authors provide a detailed account of the experimental setup, including hyperparameters and other relevant settings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Many existing studies have only applied prompt techniques in different domains without providing theoretical proof of their effectiveness. In contrast, this research explores the theoretical foundations and effectiveness of prompt techniques from the perspective of reparameterization and other angles.\n2. The authors' theoretical analysis shows that the shared structure significantly improves the efficiency of parameter estimation, resulting in faster convergence compared to non-shared alternatives.\n3. The authors also observed that when there is a lack of shared structure among prompt parameters, the performance of prompt learning is negatively impacted. This finding underscores the importance of shared structure in optimizing the learning process and provides theoretical support for future research on how to effectively design and utilize shared structures to enhance the effectiveness of prompt learning.", "weaknesses": "1. The lack of additional visualization results means that, despite the authors' analysis of the effectiveness of prompt techniques, there is a deficiency in more intuitive visual outcomes. As a reader, I am particularly interested in understanding which specific areas of the images the prompts cause the model to focus on.\n2. The paper primarily discusses the benefits of reparameterization, but the discussion on its potential negative impacts or limitations may be insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vCUx1RftbC", "reviewer": "Reviewer_DYsW", "summary": "This paper explores prefix-tuning—a parameter-efficient method for adapting large pre-trained models to downstream tasks. Specifically, the study investigates the effectiveness of reparameterization in prefix-tuning and offers theoretical foundations for its success. Prefix-tuning, in contrast to traditional fine-tuning, involves modifying only a small subset of model parameters, thereby reducing computational cost and storage requirements. The authors demonstrate that reparameterization is not merely a technical optimization but encodes a shared structure between key and value vectors, enhancing sample efficiency. By empirically validating these theoretical insights across tasks in both visual and language domains, the authors provide a new perspective on prompt-based tuning.", "questions": "1. How does reparameterization improve sample efficiency in prefix-tuning, and what are the core theoretical principles underlying this claim?\n2. What differentiates the shared structure induced by reparameterization in prefix-tuning from other parameter-sharing mechanisms, such as those used in multi-task learning?\n3. How does the proposed shared structure model compare with traditional mixture-of-experts (MoE) frameworks in terms of theoretical benefits and efficiency?\n4. The authors discuss potential memory overhead associated with the reparameterization method. What optimizations could address this issue in practice?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provides a novel theoretical perspective on reparameterization, bridging prefix-tuning and mixture-of-experts frameworks.\n2. Demonstrates empirical improvements across multiple tasks and architectures, supporting the theoretical claims with robust data.\n3. Enhances the understanding of prompt-based tuning mechanisms, potentially broadening their applications.", "weaknesses": "1. The study is limited to specific configurations, leaving questions on broader applicability to different model types and prompt architectures.\n2. While theoretically sound, the real-world implications, particularly computational demands of the reparameterization method, could be explored in more depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)