On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Authors: Fanqi Yan
Huy Nguyen
Le Quang Dung
Pedram Akbarian
Nhat Ho
Alessandro Rinaldo
Venue: NeurIPS 2025 poster ()
Keywords: mixture of experts
contamination
minimax estimation
Primary area: theory
TL;DR: In the paper, we study the convergence rates of the maximum likelihood estimator of gating and prompt parameters of the softmax-contaminated MoE.

Abstract: The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstream tasks by including a new contamination part, or prompt, functioning as a new, trainable expert. Despite its popularity and relevance, the theoretical properties of the softmax-contaminated MoE have remained unexplored in the literature. In the paper, we study the convergence rates of the maximum likelihood estimator of gating and prompt parameters in order to gain insights into the statistical properties and potential challenges of fine-tuning with a new prompt. We find that the estimability of these parameters is compromised when the prompt acquires overlapping knowledge with the pre-trained model, in the sense that we make precise by formulating a novel analytic notion of distinguishability. Under distinguishability of the pre-trained and prompt models, we derive minimax optimal estimation rates for all the gating and prompt parameters. By contrast, when the distinguishability condition is violated, these estimation rates become significantly slower due to their dependence on the prompt convergence rate to the pre-trained model. Finally, we empirically corroborate our theoretical findings through several numerical experiments.

On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Fanqi Yan*.
Huy Nguyen*
Dung Le
Pedram Akbarian
Nhat Ho
Alessandro Rinaldo
Department of Computer Science, Department of Statistics and Data Sciences,
Department of Electrical and Computer Engineering, The University Texas at Austin (fanqi yan, huynm, quangdung0110, akbarian, minhnhat) edu, alessandro. wwn...oodaiiaaonaa.na.inaa. utexas.ed.
Abstract
The contmax--onttmiitteedd mixture of experts (MoE) model is deployed when large-scale pre-trained model, which plays the role fixed expert, is fine-tuned for learning downstream tasks by including new contamination part, prompt, functioning as new, trainable expert. Despite its popularity and relevance, the theoretical properties of the softmax- comtaminnec MoE have remained unexplored in the literature. In the paper, we study the convergence rates of the maximum likelihood estimator of gating and prompt parameters in order gain insights into the statistical properties and potential challenges of fine-tuning with a new prompt. We find that the estimability of these parameters compromised when the prompt acquires overlapping knowledge with the pre-trained model, in the sense that we make precise by formulating a novel analytic notion of distinguishability. Under distinguishability of the pre-trained and prompt models, we derive minimax optimal estimation rates for all the gating and prompt parameters. By contrast, when the distinguishability condition is violated, these estimation rates become significantly slower due to their dependence on the prompt convergence rate the pre-trate pre-trained model. Finally, we empirically corroborate our theoretical findings through several numerical experiments.
Introduction
Mixture of experts (MoE) [14, 16] has emerged as statistical machine learning model that aggregates the power of multiple sub-models. This model consists of two primary components: expert function (or, simply, expert) and gating network. Experts can be, for example, a feed-forward network (FFN) [33, 4], classifier [2, 27], or regression model [7, 17]. The gating network softly divides the input space into multiple regions where the opinions of some experts are deemed be more trustworthy than others. This is done by dynamically allocating higher input-dependent weights instead of constant weights to the various experts, making MoE more flexible and adaptive than traditional mixture models [25]. As consequence, MoE has been leveraged in wide range of fields, including natural language processing [5. 15, 10, 8, 21, 33], computer vision [32, 24], speech recognition [36, 37], multimodal learning [11, 38, 28], continual learning [20, 22], and reinforcement learning [1. 3].
Unlike these applications where all experts are trainable, parameter-efficien fine-tuning methods such as prefix tuning [23, 19, 18] can be interpreted as mixture of frozen or pre-trained expert and
*Co-first authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

trainable prompt expert responsible for learning downstream or more specialized tasks, which we refer as contaminated MoE throughout this paper. Despite the empirical success of this fine-tuning approach, there is a very limited theoretical understanding of their properties and limitations in the literature. To the best of our knowledge, contaminated MoF. has only been previously studied in [35] to characterize expert structures achieving the optimal parameter estimation rates. However, the analysis in that work is conducted under a simplified setting where the gating (mixture weight) is independent of the input value, which is very impractical assumption. To close this gap, undertake a thorough theoretical analysis of the more commonly used seflmax-contaminater MoE model, specified in equation (1) below, a contaminated MoE model whose gating function takes the form of soft-maxed linear network. We analyze the issue of identifiability and the convergence properties of the maximum likelihood estimator of the prompt parameters to shed light the understanding of prompt behavior in prefix tuning methods. A main take-away of our analysis is the potential for the prompt to be exceedingly similar to and thus acquire the same knowledge as the pre-trained model, a situation greatly impacting the estimability of the prompt parameter. To overcome this issue, in Definition 1 we formulate analytical properties of the pre-trained and prompt models, which we refer to as distinguishability, that are guaranteed to rule out excessive overlap between the models and ensure good estimation rates. We make the following contributions.
(i) Distinguishabiiity of the prompt model from the pre-trained model. In Section 2, we propose a novel notion of distinguishability between the pre-trained and prompt models and then illustrate its properties.
(ii) When the distinguishubility condition is satisfied, we show in Section 3.1 that the prompt does not converge to the pre-trained model intuitively, these two models have distinct expertise. In fact, we demonstrate that the convergence rates of the MLE of all the prompt and gating parameters are of Furthermore, we establish minimax lower
parametric order in the sample size n. that is, bounds on the estimation errors with matching rates, thus showing that the convergence rate of MLE is minimax optimal.
(ii)) When the distinguishability condition violated, the prompt will converge to the pre-trained model, that is, both models employ the same expert structure and thus will gain similar expertise. In Section 3.2, we show that, under this setting, the estimation rates for prompt and gating parameters are negatively affected by the prompt convergence to the pre-trained model and, therefore, become substantially slower than the parametric rate O(n-1/2). We confirm that these slower rates are tight by deriving matching minimax lower bounds. See Table for a summary of our results.
Lastly, in Section 4, carry out several numerical experiments to empirically justify our theoretical results, and then conclude the paper in Section 5. Rigorous proofs are provided in the Appendices.
A major technical innovation in our contribution that sets it apart from existing theoretical analyses of MoE models is the fact that we let the parameters of the prompt model to vary with the sample size n, thus potentially allowing for a more challenging estimation task as the sample size increases. This approach necessary to carry out a minimax analysis.
Notation. For any n E N. we let [n] == (1.2.... nn.. For a vector u we denote with ||u]]| its Euclidean norm value. Given any two positive sequences (an)n21 and (b,)n21, we write a, (b,.)
or On S bn if an < Cb, for all n E N and some C' > O. We further write a, = ((b,) to denote an S bn bolyoooog(bb)) (b,),. where polylog(b.) indicate any term that is polylogarithmic in bn Lastly, for any two densities p and 9 (dominated by the Lebesgue measure), their squared Hellinger distance is computed as dz (p. 9) = [[Vp(() - Va(=))]do, while the total variation distance is given by dy (p.q) == 1 (p(x)= - q(x)ldz.
Table 1: Summary of parameter estimation rates in the boftmax-contaminn MoE model. Notice that the rates are in expectation. For the notation, please refer to equations (1) and (2). In addition, we also denote Anj" := 17" ?o and Av" == 1² Vo-
Setting Distinguishable
exp(Fnn) exp(T"))
1|3m
1011
n 17'1 O(n bn)
O(n
Non-distinguishable Ö( -nn=:-1(An) ||(An) Av*))l -22
1-1)
2

---

2
Preliminaries
In this section, we begin with setting up the problem, followed by a discussion on related works in Section 2.1. Then, in Section 2.2, we introduce the distinguishability condition and provide an investigation into the fundamental properties of the softmax-conttaminated MoE, including the model identifiability and the model convergence.
2.1 Problem Setup
X Problem setting. Suppose that (X1, Y1), (X2, (X,, x C Rd are i... samples of covariate-response pairs of size n. We assume that the input covariates X1, Xz...,, X, drown in an i.i.d. manner from some known continuous probability distribution the responses are generated according to a softmax-contaminatee MoE model, which postulates that the conditional density function of the response given the covariates is given by
Rd and that
x + PG_ (ylz) Tx
Solylho(a, vo)
exp(( x 11))
+ 1 ++ cxp((3)))
Above, the pre-trained model corresponds to a fixed and known conditional probability density function fo( (o(-/ha(:: no), vo), parametrized by the pre-trained mean expert function æ + (1r, 70) and variance My- Meanwhile, the prompt model, denoted f(|[((,,) "",,""""" is modeled an unknown Gaussian density function with the prompt mean expert x - la r7") and variance v*. We collect all the unknown parameters of the prompt model into the vector G, (3", " r""",, =²), belonging to some parameter space = Rd R R R+. Note that allow the values of these parameters to vary with the sample size n. However, for notational convenience, we suppress the dependence of G. throughout the paper. In addition, it should also be noted that the "probabilistic" MoE model (1) can be related to "deterministic" MoE models used in deep learning [33] by taking the expectation of the response given the covariate, that is,
x 1 exp((x*)T xx+))) ha(a, no) + 1 exp((3*) + +
E|
h(x,n").
Maximum likelihood estimation (MLE). We utilize the maximum likelihood method [34] to estimate the unknown parameters G_ = (8", , r","", vn) of the softmax-comtaminann MoE model (1) as follows:
Gn = (Bn, Fn, Dn) E arg max GEE
(og(pa(Y) (Y).
For the sake theory, we assume that the input space X is bounded, whereas the parameter space E is compact. In addition, assume that the prompt expert function + h(x, 77) is differentiable with respect to E R for almost all x E X. Note that these assumptions are mild and have been used in previous works [13, 30, 35).
Related work. Mendes et al. [26] considered an MoE model where each expert was formulated as polynomial regression model. Their objective was to address the trade-off between the number of experts and the expert size to obtain the optimal parameter estimation rates. Next, Ho et al. [13] took into account the parameter estimation problem for Gaussian MoE. models with input-free gating. They demonstrated that when expert functions satisfied an algebraic independence condition, the convergence rates of MLE were optimal of parametric order the sample size. Conversely, if the expert functions are not algebraic independent, then the parameter estimation rates became inversely proportional to the number of fitted experts. These results were then extended to more practical settings of input-dependent gatings, including softmax gating [31] and sigmoid gating [29], revealing that the latter
sample-efficiem than former in terms of expert estimation.
was not until 2024 that Nguyen et al. [30] investigated a contaminated MoE where a frozen pre- trained model was fine-tuned by mixture of prompts rather than single prompt model. However, they imposed two unrealistic assumptions on their model of interest: they equipped the contaminated MoE with input-free gating and kept the ground-truth parameters unchanged with the sample size.
3

---

Then, Yan et al. [35] overcame the second limitation by allowing ground-trun truth parameters to hinge on the sample size as in the case of traditional mixture models [6], while the first limitation remained unsolved. Therefore, in this work, our goal is to completely address both limitations by studying the coftmax-contaminate MoF equation (1).
Challenges. There are three fundamental challenges of our analysis compared to previous work.
1. Uniform convergence rates. We allow ground-truth parameters G. to change with sample size n, which is challenging yet closer to practice than the settings in previous works on MoE [31, 29], where G. does not change with n. Thus, the convergence rates of parameter estimations in our work are uniform rather than point-wise as in those works.
2. Minimax lower bounds. We determine minimax lower bounds under both distinguishable and conniistinguishable settings. Based on these lower bounds, we can claim that our derived convergence rates are optimal. However, no minimax lower bounds are provided in [31, 29].
Input-dependent gating. The latest work on understanding the contaminated MoE model is [35], but it considers input-free gating in the analysis. On the other hand, in this paper, we take into account softmax gating, which hinges upon the input value. This input-dependence yields several challenges on the convergence of density estimation and parameter estimation.
2.2 Fundamental Properties of the
MoE
As mentioned above, when the prompt's learned skills overlap with those of the pre-trained model, estimating the prompt parameters becomes challenging due to potential non-identifiability. To capture that issue accurately, we introduce an analytic condition called distinguishability in Definition
Definition (Distinguishabilitt). We say that fo is distinguishable from f if the following hold: for any distinct pairs of parameters (1), (122, 12) ) E O, if there exist measurable real-valued functions X (2x), x E X ++ bi (z), and X + (ca(2))ssla)s) where a (01, E No x N with a |a1| + a2 I 1 such that
(x)) fo (o. vo) + by bz(x) f(u)l( m)) glal
Σ ca(x))
rmpa Oran Svan (1)h(z, 12), O,
0gla|51
for almost every (1r, v) X V, then must be the case that bo(x)) = by (x)
ca(x)= for all S lal for almost every x.
To help understand the notion of distinguishability better, in our next result we characterize the class of pre-trained models distinguishable from the prompt The proof can be found in Appendix B.
Proposition 1. If a pre-trained model fo does not belong to the family of Gaussian densities, then fo is distinguishable from the prompt model f in the sense of Definition 1.
On the other hand, if fo belongs to the family of Gaussian distributions and the pre-trained expert shares the same structure as the prompt expert, that is, ho h. then the above condition is violated. It should be noted that the distinguishability condition ensures that the prompt does not acquire overlap- ping knowledge with the pre-trained model since the equation fo (z, mo)), vo) = f(yl f((x, 1)), v) cannot hold for almost all (xx, y) E X y. Moreover, we illustrate in the following proposition that the distinguishability condition also implies that the dettmax-contaminatee MoE is identifiable.
Proposition (Identifiabilityy. Let G, G' be two components in E. Suppose that f is distinguishable from for then if the identifiaiility equation pa(y)æ) pa (y (x) holds for almost all (æ,y) y) E X V then we obtain G G..
The proof of Proposition provided Appendix B.2. Given the consistency of the softmax- contaminated MoE, we continue to investigate the convergence behavior of density estimation under this model Proposition whose proof can be found Appendix B.3. We conclude this section with a consistency guarantee for the contaminate density itself, which under mild tail conditions on fo. can be estimated at a parametric rate in the Hellinger distance, regardless of the distinguishability between fo and f. Below and throughout the paper, Ep.... denotes the expectation operator with respect

---

To start with, we consider scenario in which the pre-trained model fo is distinguishable from the prompt model f. Recall that given the density estimation rate in Proposition 3, we need to construct loss function between the MLE G. and the ground-trutt parameters G., which should be bounded by the Hellinger distance between the two corresponding densities, in order to capture the parameter to the joint distribution of the data (2,...,..... (X Yu) and assuming the softmax cettmax-contaminatee MoE model (1) parametrized by G E E. i.. Yi|x; ~ Pc. for all i. Instead, E indicates the expectation with respect to the input distribution.
Proposition 3 (Model Convergence). Suppose that the pre-trained model fo is bounded and, for some p 0,
Ex log fo (o(y)ho(( no), vo)] 2 v², for almost every y E V.
(3)
Then, for the MLE G. defined in equation (2), it holds, for almost all x E 2,
du (po. (12)), (. (1x))] Vlo((n//n.
sup
Epo... Ex
(4)
G.EE
The above result shows that the density estimator Pã converges to the true density pc, under the Hellinger distance the near near-parametric rate of order On 1/2). To extract from this result a conver- gence guarantee for the MLE Gn itself, we follow a by-now-standard approach in the latest analysis of MoEs; see, e.g.. [31]. The main idea is that, if one exhibit loss function among param- eters, say D(G,,, G=), such that Ep.... E G.)] E Epc... Ex (H (pe. (-IX), )
then convergence of G. in the expected D(:;)) :) loss, as well potentially information on the rate of convergence, will follow. See Appendix A for further details. Throughout the rest of the paper, we assume that the tail condition (3) on fo and the distribution of X used in Proposition 3 in effect.
3 Convergence Analysis of Parameter Estimation
In this section, we present various convergence rates for the MLE estimator of the model prompt and gating parameters. In Sections 3.1 and 3.2 we provide separate minimax analyses, depending on whether the distinguishability condition of Definition holds not, respectively.
3.1 Distinguishable Setting
estimation rates. Tailored to the distinguishable setting, we measure the discrepancy between two arbitrary parameters G and G_ in E via the loss
DG G.) = (exp(r)= - exp(r") + (exp(T) + exp(r")) 11,, n. - ("","",,"")l (5)
We ready determine the convergence behavior of the MLE under distinguishable settings.
Theorem 1. Suppose that the pre-trained model fo is distinguishable from the prompl model f. For almost every I E X, and for any 17 E 12, we assume that the Jacobian of the prompt expert function functionn does not vanish, i.e., 0h (z,n) # 0. Then, there exists positive constant C1 that depends on E and fo such that the Hellinger lower bound Ex [d (pa (+IX),pG. (-(x))] 2 CGD, holds for all parameters G E E. As a result, we obtain
sup E Epa... -xxp(f))exxp(- )|] < log(n)/n,
(6)
G.E3
sup E Epa... G.E3
(xpp((" (1),,, Am. Dn) (8", n", w² M1]] < log(n)/n.
(7)
The proof of Theorem 1 is deferred to Appendix A. The bound in equation (6) reveals that the gating parameter estimator exp(f..) converges to its ground-truth counterpart exp(r") at a rate of order -1/2). Analogously, looking at the bound in equation (7), since the terms exp(r*) cannot go to zero due to the compactness of the parameter space E. follows that the convergence rates of the parameter estimators 3n, T), and Dn to ,9,, 17" and u² are also of order ((n-1/2) Meanwhile, in the contaminated MoE with input-free gating [35], the estimation rates for prompt parameters
5

---

,"'""" are slower than Ö(n 1/2) as they depend on the convergence rate of the gating parameter to zero. Therefore, replacing the input-free gating with the softmax gating in the contaminated MoE helps reduce the sample complexity of parameter estimation.
Given the pear-parametric convergence rates in Theorem 1, it is natural wonder if they are optimal. To answer this question in the affermative, below we derive minimax lower bounds.
Theorem 2. the pre-trained model fo is distinguishable from the prompt model f. then the following
minimax lower bounds hold for any 0 < <r<, 1:
inf sup Epa.. (lexp(Fn) -xp(()))) 2 1/r, G.EEGGE
inf sup Epa... G. GEB
Thi Pn) (1)1) Zn 1/,,
where the infimum is over all estimators G. := (3 Tu. Tu, .n) taking values in E.
The proof of Theorem 2 can be found in Appendix A.2. The above minimax lower bounds imply that, under distinguishability, the convergence rates of the MLE, of order (n12)/2 is nearly minimax optimal, save for logarithmic factor.
3.2 Non-distinguishable Setting
We now turn to the much subtler case in which the distinguishability condition is violated. Since we assume a Gaussian prompt, it follows from Proposition 2 that the pre-trained model fo necessarily belongs to the family of Gaussian densities. Furthermore, if the pre-trained and prompt model use the same expert function, i.e. ho h, then fo is not distinguishable from the prompt model f. We will thus focus on this challenging scenario.
Under this setting, the prompt model may converge to the pre-trained model. In particular, the pair of prompt parameters (nn","") converge to the pair of pre-trained parameters (No, Po) as n - oc, then it follows that f( ((11k(,,7) (,,,,""),, v** converges to so(- [h(, no); vo), indicating that the prompt learns the same expertise as the pre-trained model. Therefore, it becomes difficult for the gating network to assign higher weight to either the pre-trained model or the prompt than the other as they have similar expertise, As result, one may expect the estimation rates of the gating parameters to be substantially slower. To formalize these setttings precisely, we need to pay more attention to the expert structure.
It should be noted that key step in obtaining the MLE convergence rates in Theorem is to decom- pose the density discrepancy Pa. PGG into combination of linearly independent terms through an appropriate Taylor series expansion of the function g 8, 17, v) == exp( x)) f(y lı(z, 1I), v) with respect to its parameters 3, 17, v. This process involves, particular, higher derivatives of the expert function h with respect to 77. which may not be algebraically independent. To ensure the linear independence of the terms in the Taylor expansion, we formulate a strong identifiability condition that is indeed sufficient for these purposes.
The expert function ++ h(x, 1) strongly identifiable it it twice differentiable with respect 1) I& for almost all X, and if. for any fixed 3 Idd and y E IR?, each of the following sets of real-valued functions (of x) consists of linearly independent functions over TR. For notational simplicity, we write h(()) in place of l((, below.
Definition (Strong Identifiability).
1. The first-order gating independence set:
oh Onlly
exp(3 x) Oh
Onllaj
wE((]
2. The gradient product independence set:
(1, v(0), exp(3 x); Tnp(w) ōn(o] (x x) Oh Oh
Oh. Oh
Onlial Ön((n) u,ve(a), wE(d]
3. The mixed and second-order independence set:
82h { Oh exp(.3 x) Onpuy On(w))* inpl)) exp(3 x) Or ()p(()))((t) uovela,, weld]
oh (w) Oh. 82h
Onlu)

---

Here, the First-order gating independence condition guarantees that changes in h with respect to 1j remain distinguishable, even after modulation by the gating weights exp(8" X). This is a minimal requirement to ensure that the expert and gating mechanisms interact in structurally non-degenerate way. The Gradient product independence condition guarantees that the products of directional derivatives of h are distinguishable from each other (even under modulation by gating terms) and cannot be expressed as a linear combination of basic functions. This prevents higher-order interactions among gradients from collapsing into lower-order structures. Finally, the Mixed and second-order independence condition stronger than the first-order one. It rules out first-order interactions between expert and gating parameters of the form 0h/3r(() = (((), a which would imply 0g/ōn((u (w) d igg/ (a)l"))r
It also requires that second-order derivatives remain linearly independent, even accounting for the effect of the gating function. This guarantees that both first- and second-orde order directional changes in h convey distinet, non-redundant information, and that higher-order structure in h cannot be reduced to absorbed by lower-order terms. This is essential when handling second-order Taylor expansions of the model.
Examples. The expert functions h(2,, 77 = GELU(7 z), h(z, m)) sigmoid(1) x),, and h(r, 1)) = tanh(n T z) satisfy the strong identifiability condition, as their nonlinearities avoid degeneracies. In contrast, hxx,, 77 = (e)U(n) x) fails the second-order independence condition, as the second-order derivatives vanish almost everywhere. Another failure case arises when h n)) aaa x + b). where 17 = (a. b) and a is any scalar activation function. This leads to 0h/ /a = x /h/Ob, directly violating Condition
To determine the convergence rates for the MLE in these settings, we construct the following loss function between parameters G and G*, carefully tailored to the non-distinguishable setting:
D2(G, exp(r)ll(An, Av) 21 exp(r"))l(An) Av")
min(exp(7))
An)] + [I(An), A1²)
+ (exp(r))) / + + (I] Ar*) X (1.3. v, - (8", ,,n"""""))))"""))))""))) >>)]l
(1 Noo V vo) and (An)", Av*) = (11 - =10 1² - vo)) Theorem 3. Suppose that fo belongs to the family of Gaussian densities and ho = h. Then, there exists positive constant C2 that depends on E, no. such that Ex [du (PG (-)X), PG. (-1x))] 2 C₂ C2D2(G,Gr G.) holds for all parameters G. As a result, we obtain
where we denote
sup Epc... m ||A Av")|l x exx((fn) - exp(r*))² S log(n)/n, G.EE(I.)
(8)
sup Epc... [expp(r") 2 x (3 Tr 0x) (0", n" vz )²] < log(n)/n, G.EE(I.)
(9)
for any sequence (In)n21 such that In/ log as - where we denote
In
E(I.) G - (T, B EE:
S Expp())
min ISid, { 12 1/2 36)22
vn
The proof of Theorem 3 is in Appendix A.3. Note that under the setting of Theorem 3, the softmax- contaminated MoE model is not identifiable, that is, the equation Po = Pa. (y/ x)) for almost all (x, y) does not imply G = G.. For that reason, we restrict the parameter space to the set E(I,) to guarantee the consistency of the MLE. Compared to Theorem 1, the above rates exhibit differ in several aspects.
(i) From equation (8), we observe that the convergence rate of exp(fx) to exp(r") becomes slower than the parametric order -1/2) as they depend on the vanishing rate of (Av)", Ar") to zero. For example, if the pair of prompt parameters (11" =²) approach (No, pb) at the rate of (n then the bound (8) implies that exp(fnn goes exp(7*) the rate of (nn-//)) This toy example is indeed confirmed by our numerical experiments in the next section.
(ii) Likewise, the convergence rates of the estimators (Bn, 7m, Pa) are also impacted by the con- vergence rates of the prompt parameters and therefore slower than O112/.). For example, if
7

---

(An", Av") go to zero at the rate of (nnn1//))))) then the bound (9) indicates that 3n, Tr. D/ converges to 9" '7",, v the rate of On= respectively. Again, in numerical experiments below empirically verify this behavior.
In our final result, whose proof can be found in Appendix A.4, we show that the slower converge rates for the MLE under non-disiingushaabiity are fact essentially minimax optimal.
Theorem 4. Suppose that fo belongs the family of Gaussian densities and ho h. Then, the minimax lower bounds
inf sup EPG.. G. GEE(L.)
[1,,,, Av)Il x exp(()) - exp(r)]l Zn
inf sup
E EPG
(xpp Av) x 11(3,,. mn. vn)) (3, w)13 2 n-1/r
G. GEE(U.)
hold for any sequence (hx)21 and any 0 < r < 1,, where the infimum over all estimators GN, taking values in E.
3.3 Practical Implications
There two important practical implications for the design of contaminated MoE model from theoretical results.
1. Softmax gating is more sumple-efficiet than input-free guting. We observe that softmax gating yields faster convergence rates prompt parameter estimation contaminated MoE than input-free free gating in [35]. In particular, when using input-free gating, Tabll 2 reveals that the rates for estimating expert parameters and variance depend on the convergence rate the gating parameter to zero. By contrast, when using softmax gating, estimation rates for expert parameters and variance become significantly faster as the previous rate dependence disappears. Therefore, our theories encourage the of softmax gating over input-free gating when tuning Saraamintte--Mo--aassee models.
Prompt models should have different expertise from pre-trained models. It can be seen from Table 2 that when the prompt model acquires overlapping knowledge with the pre-trained model nonn-iitingguisabbe setting), the convergence rates of parameter estimation are slower than when these models have distinct knowledge (distinguishable setting). Thus, our theories advocate using prompt models with different expertise from the pre-trained model.
Table 2: Comparison of parameter estimation rates in input- free-miatttt MoE [35] and softmax- contaminated MoE (Ours). Below, we consider gating parameters exp())), expert parameters ')", and variance v². In addition, X denotes the constant weight in input-ireeccontaminated MoE.
Distinguishable Setting
Gating parameters
Expert parameters and Variance
Input-free gating [35]
O(n (n/21/2
O(n ((n-1/2(X-)-3
Softmax gating (Ours)
(l2-1/2)
((n1//2)
Non-distinguishable Setting
Gating parameters
Expert parameters and Variance
Input-free gating [35]
O(n [[ Av*)]l-2)
O(n ||(An' Ö(n |(nr" -r")|[
Softmax gating (Ours)
Ö(n ((n=1-1(Ar) |IAn", A
Numerical Experiments
In this section, we present several numerical experiments to verify theoretical findings.
Experimental setup. Recall that, the distinguishable setting, the pre-trained model fo does not belong to the Gaussian density family. Thus, we let fo be the density of a Laplace distribution, with mean function ha. 1))) = tanh(ng x) and variance My- Here, To a d-dimensional vector defined ej == (1, O, .................. 0), and no 0.001. Meanwhile, the prompt ff is formulated Gaussian density,

---

with the same tanh mean function but a different parameter -ie., h(x, n1" = tanh((n")) and variance 2/".
On the other hand, in the non-distinguishable setting, both f and fo belong to the Gaussian density family, and h and hn are expert functions of the same form (albeit parameterized by different values of yo and 17",, As in the previous case, we let the expert function be the tanh function: in the pre-trained model, the expert is hæ, 770) tan x), and in the prompt model, it is h(x, ((r,,"") 77* tanh((n") xx)
Synthetic data generation. We create synthetic datasets following the model outlined in equation (1). Specifically, generate data pairs {(X,, 1, X' x 3 C R IR by first drawing each covariate X, independently from standard Gaussian distribution, for i = 1, ........ n, and consistently set = across all trials. The responses Y, are drawn from the density pG. 12]2)). where G. = (3 .7 v*
(a) In the distinguishable setting, we let 3 1/Vd 1a, 1, 17 0.001.
-1j and V'
(b) In the non aon-stiinguisaable setting, we examine two cases to study the MLE convergence behavior as either rj" or v* varies with n: in the first, 17" is an O(n perturbation of no with v" fixed at un: in the second, 17" = - while v" is perturbed around Py at the same rate. In detail, we set:
(i) In the first case, 8" 1/Vdd la T" = 11" e1 + 1-18)) 77o(1 n n-1/8), and v" 1i = 0.001.
(ii In the second case, 3 3 = 1/vd. la, T 1, -1 -7/0- and 0.001 n 1/8 20 n 11/8).
Training procedure. We conduct 40 experiments and, for each of them, consider 20 different sample sizes n, ranging from 10xh 105, In computing the MLEs, the initialization is set relatively close to the true parameter values to mitigate potential optimization instabilities. We use an EM algorithm [16] compute the MLE, employing an off-the-shelf BFGS optimizer for the M-step due the absence of universal closed-form solution. All the numerical experiments are performed on MacBook Air with an Apple M4 chip.
The experimental results are presented in Figure and Figure 2, where the x-axis displays varying sample sizes n, and the y-axis shows the parameter estimation error. We now present detailed analysis of the results shown in each figure:
Results.
(a) Figure displays the results for Theorem 1. We observe that the convergence rates of O(n (n5-1.)) respectively, aligning with the
(Bn; Fry Ma, (2) 0(n-045),0((00553
theoretical rates of order (m-0.52),
in Theorem 1.
(b) On the other hand, Figure illustrates the parameter estimation errors for the simulations conducted in the non-distinguishable setting Theorem 3.
(i) In the first case, 7/' converges n/o the rate of O(n 1/8), while o² remains fixed, Figure 2a shows that the convergence rate of exp(ff. to exp(T") consistent with the expected rate of -1-44) The convergence rates for 3r, 7m and 222 are O(n -U.37). On -0.39), and O(n -0.35, respectively, all of which are approximately Onn -0.375), as they hinge on the vamishing rate O(n (n/3).)). These empirical rates are consistent with the theoretical rates in Theorem 3.
which is
(ii) In the alternative setting, 17" is held fixed, while 1/* converges to vo at the rate of O(n 1/8). Figure 2b reveals that the convergence rate of exp(f.) to exp(r") is of order O(n On2-2.22). again close to O(n - 1/44). Meanwhile, the MLES Sn, Tm. and D, still empirically converge to 3*, .y",, and v² at rates of O(n -0.39), O (1.37,, and O(n Onn-0.39) respectively, which align well with the theoretical rates O(n (3/)). This observation consistent with the theoretical convergence rates in Theorem 3.
5 Conclusion
In this paper, we characterize the convergence behavior of maximum likelihood estimators for parameters in the
MoE model formulated as mixture of a frozen pre- pre-trained

---

3.278 0.
7,80 630
0.16en 430 Ri-vi
0L06 19
[ooothl - 10p
I4, 11
l '1
Figure 1: (Distinguishable Setting: So is the density of Laplace distribution.) Log-log graphs
depicting the empirical convergence rates of the MLE (Bn, Fn, The Py2) to the ground-truth values (8", ,"" v. The blue lines display the parameter estimation errors, while the orange dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates.
488
2666 4:0 I6-r1 rl
peppft) -aal-1]
18- 11
(a) Case (i): 8" 1/vd 1a,
m e1 (1 + 1/8 0.001.
M4-11
16
(b) Case (ii): 8" = 1/ 1a, T = 1, 1/2 C1, v/ 0.001 + n 1155)
Figure 2: duon-disiinguishable Setting: so is Gaussian density.) Log-log graphs depicting the ,T","" n" v²). The blue lines display the parameter estimation errors, while the orange dashed dotted lines are the fitted lines, highlighting the empirical MLE convergence rates. Figure 2a and Figure 2b illustrates results for Case (i) and Case (ii), respectively.
empirical convergence rates of the MLE (Bn, Fu, ),, Dn) to the ground-truth values (8",'",
model and a trainable prompt model. To capture the challenge in which the prompt model admits the same expertise the pre-trained model, we propose novel analytic distinguishability condition and divide our analysis based on that condition. When the distinguishability condition is satisfied, we obtain minimax optimal parameter estimation rates of parametric order in the sample size, which are faster than those under the contaminated MoE with input-free gating. Conversely, when the distinguishability condition is violated, these rates become substantially slower than the parametric rates as they hinge on the convergence rates of prompt parameters to pre-trained parameters.
Based our theoretical analysis, we make the following observations. First, the softmax gating helps to improve the sample efficiency for estimating the parameters in the contaminated MoE compared to the input-free gating. Second, the convergence rates for parameter estimation will be negatively affected if the prompt model acquires overlapping knowledge with the pre-trained model, thereby increasing the sample complexity of parameter estimation.
In future work, we plan to consider a more challenging setting of the contaminated MoE where the pre-trained model is fine-tuned by multiple prompt models rather than a single prompt as in the current setting. Furthermore, we can also generalize the analysis to the scenario where the prompt models belong to various families of distributions, rather than being restricted to Gaussian distributions.
10

---

References
[1] J. S. O. Ceron, G. Sokar, T. Willi, C. Lyle, J. Farebrother, J. N. Foerster, G. K. Dziugaite, D. Precup, and P. S. Castro. Mixtures of experts unlock parameter scaling for deep RL. In Forty-first International Conference on Machine Learning, 2024. (Cited page 1.)
[2] Z. Chen, Y. Deng, Y. Wu, Q. Gu, and Y. Li. Towards understanding the mixture-of-experts layer in deep learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 23049-23062. Curran Associates, Inc., 2022. (Cited on page 1.)
[3] Y. Chow, A. Tulepbergenov, O. Nachum, D. Gupta, M. Ryu, M. Ghavamzadeh, and C. Boutilier. Mixture-of-Fae Expert Approach to RL-baaed Dialogue Management. The Eleventh Interna- tional Conference on Learning Representations, 2023. (Cited on page 1.)
[4] D. Dai, C. Deng, C. Zhao, R. X. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y. K. Li, Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang. Deepscekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.04088 2024. (Cited on page 1.)
[5] DeepSeek-AI et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 2024, 2024. (Cited on page 1.)
[6] D. Do, H. Nguyen, K. Nguyen, and N. Ho. Minimax optimal rate for parameter estimation in multivariate deviated models. In Advances in Neural Information Processing Systems, volume 36, pages 30096-30133. Curran Associates, Inc., 2023. (Cited on page 4.)
[7] S. Faria and G. Soromenho. Fitting mixtures of linear regressions. Computation and Simulation, 80(2):201-225, 2010. (Cited on page 1.)
Journal of Statistical
[8] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022. (Cited on page 1.)
[9] Gadat, Kahn, C. Marteau, and C. Maugis Rabusssaa Parameter recovery in two-component contamination mixtures: The 1'2 strategy. 2020. (Cited on page 24.)
[10] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, al. The Ilama herd of models. arXiv preprint 20X 2024. (Cited page
[11] X. Han, H. Nguyen, C. Harris, N. Ho, and S. Saria. Fusemoe: Mixture-of transformers for fleximodal fusion. In Advances Neural Information Processing Systems, 2024. (Cited on page 1.)
[12] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, 10:271-307, 2016. (Cited on page 20.)
[13] N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence rates for Gaussian mixtures of experts. Journal of Machine Learning Research, 23(323):1-81, 2022. (Cited on pages and 40.)
[14] R. A. Jacobs, M. Jordan, S. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3, 1991. (Cited page 1.)
[15] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang. T. Lacroix, and W. E. Sayed. Mixtral of experts. arxiv preprint arxiv 2401.04088, 2024. (Cited on page 1.)
[16] M. I. Jordan and R. Jacobs. Hierarchical mixtures experts and the EM algorithm. Neural Computation, 6:181-216 1994. (Cited on pages and 9.)

---

[17] J. Kwon and C. Caramanis. EM Converges for a Mixture of Many Linear Regressions. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1727-1736. PMLR, Aug. 2020. (Cited on page 1.)
[18] M. Le, A. Nguyen, H. Nguyen, C. Nguyen, A. Tran, and N. Ho. On the expressiveness of visual prompt experts. arxiv preprint arxiv 2501. 18936, 2025. (Cited page 1.)
[19] M. Le, C. Nguyen, H. Nguyen, Q. Tran, T. Le, and N. Ho. Revisiting prefix -tnnngg Statistical benefits of reparameterizatioo among prompts. In The Thirteenth International Conference on Learning Representations, 2025. (Cited on page 1.)
[20] M. Le, A. N. The, H. Nguyen, T. T.T.N N. Vu, H. T. Pham, L. N. Van, and N. Ho. Mixture of experts meets prompt-based continual learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. (Cited on page 1.)
[21] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations, 2021. (Cited page 1.)
[22] H. Li, S. Lin, L. Duan, Y. Liang, and N. Shroff. Theory on mixture-of-experts in continual learning. In The Thirteenth International Conference on Learning Representations, 2025. (Cited on page 1.)
[23] X. L. Li and P. Liing. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, Aug. 2021. Association for Computational Linguistics. (Cited page 1.)
[24] H. Liang, Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, and Z. Wang. VIT: Mixture-off of Expeerssts Vision Transformer for Efficient Multi-task Learning with Model- Accelerator Co-design. In NeurlPS, 2022. (Cited on page 1.)
[25] B. Lindsay. Mixture models: Theory, geometry and applications. In NSF-CBMS Regional Conference Series in Probability and Statistics. IMS, Hayward, CA., 1995. (Cited on page I.)
[26]
E. F. Mendes and W. Jiang. Convergence rates for mixture-of-experts arXiv preprint arxiv /I 11.22055, 2011. (Cited on page 3.)
[27] H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho. A general theory for softmax gating multinomial logistic mixture of experts. In Proceedings of the 41st International Conference on Machine Learning, pages 37617-37648, 2024. (Cited on page 1.)
[28] H. Nguyen, X. Han, C. W. Harris, S. Saria, and N. Ho. On expert estimation in hicrarchical mixture of experts: Beyond softmax gating functions. arxiv preprint arxiv 2410.02935, 2024. (Cited on page 1.)
[29] H. Nguyen, N. Ho, and A. Rinaldo. Sigmoid gating is more sample efficient than softmax gating in mixture of experts. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. (Cited on pages 3 and 4.)
[30] H. Nguyen, K. Nguyen, and N. Ho. On parameter estimation in deviated Gaussian mixture of experts. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, 2024. (Cited page 3.)
[31] H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems, 2023. (Cited on pages 3, 4, and 5.)
[32] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pint, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems, volume 34, pages 8583-8595 Curran Associates, Inc., 2021. (Cited on page 1.)
12

---

[33] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra- geously large neural networks: The sparsely-gate gatttdd ofxture-of-experts layer. In International Conference Learning Representations, 2017. (Cited pages and 3.)
[34] S. van de Geer. Empirical Processes in M-estimation. Cambridge University Press, 2000. (Cited on pages 38, and 39.)
[35] F. Yan, H. Nguyen, D. Le, Akbarian, and N. Ho. Understanding expert structures minimax parameter estimation in contaminated mixture of experts. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, 2025. (Cited on pages 2, 3, 5,
and 8.)
[36] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech, 2021. (Cited on page 1.)
[37] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe2: Mixture of of-exerrsss model with improved routing. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7217-7221, 2022. (Cited page
[38] S. Yun, I. Choi, J. Peng. Y. Wu, J. Bao, Q. Zhang, J. Xin, Q. Long, and T. Chen. Flex- moe: Modeling arbitrary modality combination via the flexible mixture-of-expert.. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. (Cited on page 1.)
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The paper's contributions and scope are reflected accurately in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: Section 5, we identify the limitations and present the future development of our analysis.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
13

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Each theoretical result contains the full set of assumptions. All the proof' are deferred to the appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experimental details are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often
14

---

one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We use synthetic data for our experiments. We will consider releasing the code upon the acceptance of our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Coaeeee//ooSSuumiisseee ississssssccciiiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips... wnips.ccpppppp wttpccccpppppccceteoe..eccccccct htelllic/iiieeee deed/Coeeeubbiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All the experimental details are provided in Section 4.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The error bars reported in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The computer resource is reported in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ett::/nneuppppeeeee cttpiippp..pppppeee cc/public/ http/pullc//Ethhttiiiieeeeee celuideeeeel
Answer: [Yes]
Justification:
We have read and followed all the NeurIPS code of ethics.
16

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given the theoretical nature of the paper, we do not think there are any positive negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is theoretical work, and we do not release any data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
17

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We do use any existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
18

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used as any original, important, or non-standard component in the development of the core methods in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: ceeeuripsscccConffeeeeccee for what should or should not be described.
19

---

Supplementary Material for "On Minimax Estimation of Mixture of Experts"
Parameters in
In this supplementary material, we provide the theoretical proofs omitted from the main text. Ap- pendix A presents the proofs of our main results, including the theorems on convergence rates for parameter estimation and the minimax lower bounds stated in Section 3. Proofs of auxiliary results concerning the fundamental properties of the softmax feftmax-contaminatee MoE model, introduced in Section 2, are deferred Appendix B.
A Proof of Main Results
In this section, we present the proofs of the MLE rate theorem and the minimax lower bound theorem from Section 3, covering both distinguishable and non-disinguishable settings.
A.1 Proof of Theorem
We begin by proving Theorem under the distinguishable setting.
Proof of Theorem 1. Let G = -B,7, we need to demonstrate that
Ex (dv (pa(() X), Pa. (+X)) DI(G,G) G) 3) 5) 0.
lim inf 8-0 G.G.
DiGG, G.)
Using the argument with Fatou's lemma as in Theorem 3.1, [12]. is sufficient to show that
|PGG - pc.ll Di(G.G) V 3 < > 0. DI(G,G.)
lim inf e G,G.
Assume by contrary that the above claim is not true. Then, there exist two sequences G, (Bn, Tus In Vn) and G.,,. = (3h, Th Mn vn), such that when n tends to infinity, we get
Di (Gn, G) -0
D. G) - llPG. - PG_,, Dilll//DD(ee (Gn, Ga,1) +
In this proof, will take into account only the most challenging setting of (Bn, Tay Ux) and (8% ny vi)) when they converge to the same limit point (8', n//, v/), where (3', 1f' v/) is not necessar- ily equal to (3, 77. D)
Step 1: Density Decomposition. Subsequently, we consider [PG.. (Y|X) PG... (Y] which can decomposed as
=
(x)( exp(Ta) (()_)))))))))(((((X,,XX,,),))))) -xxp)((( X))(YTb(X, x (2)) (5)] = In
exp(TT) (3x)(((.) X) exp(((;) X)] PG... (YIX) = II,
+ (exp(Tn) - ex)(Tx)) ex²( x) (((/)/(XXX/))) m71),, v/, - PG... (X
Based on the first order Taylor expansion, I, and II, could be denoted as
In = exp(Tn)
Σ
2 (8n 3 (mm -hm )0)00² (2 - yo
|a/=1
I 2 f _X (x==(((()))) X) 0/la2l22ag
8r
(Y)n(X, Mir,,,))))))))))))) vn)
(X, (h, + +Y|())
Joun
= =exp(Tn)
201 (Bn B1) a (mm 17 a2 12
2/11
a
ges
vii 802h (x,1%) + (Y
X (xpp(())TT))
xx)
(V),(X,
(10)
goan
20

---

where L = =01, l2 = |a2l + 203, and
Iey,2 { (a.)i= E No x No x :N:a -1, 62 1
(II)
for all (2) No such that 2 + l2 S 2. Similarly, II, can be expressed as:
1I exp(Ta)
Σ
(Bn 7 X exp(((n) X)po... R2(Y)X). (12)
1-/=1
Here (Y) (G,, G.m)) a - where Ry(X, E [2] are Taylor remainders. Consequently, 2n can expressed as:
Tik X exp((33)) x) 2
j
On
E
(X, m/d
(V)M(X, U3)
gman
oh
2/6 +62=
Σ h|=0
S X exp(((;)
(13)
with coefficients T7,4, and S are defined for any 0 < 2/61 + + 2, and 0 S hi S a
exp(TT) Σ
2as (Bn 07) (mh 1/ ( -0a--v 0, (f1, + (0., 0),
Th
aETrn exp((T) exp(TTn)
(e1, (2) = (0., 0);
and
8 - exp(Tn) (Bn - 1,)7, exp(T,) + cxp(TT))
hr + 0, h 0.
71
where Qn can be viewed as linear combinations of clements of the set Hi defined as
H4 = X exp((3;)] x) g 0 Omps (X, ni)
0h mi)((X,n)) x exp(((;)) T))G, ()).,.()|x))
(14)
Step 2: Non-vanishing coefficients. In this step, we will use contradiction argument to demonstrate that not all the coefficients the set
Tm The Din
5 52 :0:05 0 2, =1
S
(15)
Din
vanish as - Do where Din == Di G.m)). Specifically, suppose that all these coefficients converge to zero, when n oc, then we get,
-xx((=))exp(77)))
[T)
-
(16)
Din
Din
Similarly, by analyzing the limits of Th Din s.t. 1 5 2/6, + 12 - 2, we conclude that:
cxp(Tn)(B, - (3) +0 exp(T,)(n.. - )
00, exp(TT)(U,, - vi) + 0,
Dın
Din
D
as n - oo for all u E [d]. v E [q]. Given that our parameter lies in a compact set, there exists positive constant C such that exp(T;)) exp(Tx)) < C. Thus, we have
(Xp (w)
exp(T;)) (1a (0) 0, exp(Ti) vit) -
0,
D1n
Din
D
21

---

The limits imply that
(exp(Tn) + exp(T,)) Th, Px)) - (5% Thi V,)l// +.
(17)
Combining the results in equations (16) and (17) with the formulation of Din, we deduce that
1= exp(Tn) - exp(r,)) + (cxp(T.) + exp(7))) Ta, Vn) (5% Th v5 Din - O, which is a contradiction. Thus, not all the coefficients in the set S₁ tend to as n - oc.
Step 3 Application of Fatou's lemma. Let us denote by m. the maximum of the absolute values of those coefficients. follows from the previous result that 1/m,, / oc. Then (T) (m, Din) and (/m, Din) remain bounded, we can consider subsequences of these terms, ensuring that: [2E im + 12,, [S 151/m,, - Why as n - 3o for all < 2 + le I 2, 0 < hi & 1. Here, at least one among Me, (f) and w-y (j) is different from zero. By applying the Fatou's lemma, get
Ex|dv(pG, (1x), (+X)))
lim inf |PG. (Y PG. (YXX) d(x,)) n-
lim 100000
(18)
m2
2m, Din
Under the given assumption, the left-hand side of the equation (18) is zero. Consequently, the integrand on the right-hand side of the equation (18) must also be zero almost surely with respect to (X, Y). This results in:
802h 0mpoz
8h (X, 11)
X exp((3*)] x)
(YIN(X,1"))
Th
-
Σ wy X exp(((x)) X)p.. h=
for almost surely (X, Y). Furthermore, by Lemma 1, the collection
Wi = x4 exp((x*) T x) 802 O Th (V)
Orpe (X,
u f (xp(( X)GG (Y
(19)
is linearly independent with respect to (X, Y). Consequently, it follows that ,/, wy 0, for all S 2| + f2 < 2, 0 < by < 1. But this contradicts that from the definition, at least one among 12 Wy is nonzero. Hence,
reach the desired conclusion.
Lemma 1. Suppose that fo is distinguishable with f. then the set Wi defined in equation (19) is linearly independent w.r.t. (X,Y).
Proof of Lemma 1. Recall the set
gash Ómpe (X,
Wi x exp(((-)
Oh
and the density
PG. (YIX): = exp((3t)T T X + 7*)
fu(Y))(XX,))))
(x
F()))((
I exee((())
In words, PG. is a convex combination (depending on X) of and ((Y)b(X.n"),'")
22

---

Noting that the term in set W1 can be divided as the density function or its first and second derivatives
Pa_ Pa_ 10XX)))) of dh (YIN(X,n")), oh (Y)h(X,n))),
8
along with the factor involving only X.
Step 1: Distinguishable property with respect to Y.
Finst, fix X. Suppose for contradiction that there exist real numbers co, c1s C22 d (may depend on X), not all zero, such that
8 Oh
S Öh
Co
ci
02
+ dpc. (VIx) =0, 0, for almost every oh
Note that
8 Dh = f. Hence we have
0f co (Y)())(X)))))) + c1 Ōh
f&
(f)) + ((1((X,")),')) + dpc. (Y =0. 0.
a1e
Since
1
PG. PG.
(1-0()))(() 6(X) = 1+ (xp(())) T X x++7 +7²)²
the above can be rewritten
[o +
d(1-d((1p)) d ((x) + Oh (.) + c2 dh (.) do(X)fo()) = 0.
of
8
Using the hypothesis about the distinguishable property of fo with respect to f as well as the Gaussian property of f. which implies f/hh 1/2 /öv, we have
dx(X)=
for almost all X, and co + d(1 - x(x)) = 0 for almost all X,
and simultaneously c₁ 02 0. But ((X) + 0 on set of X-values of positive measure so O. Plugging d = 0 into co + + - o(X) = yields co = 0. Hence co = c1 = C2 = d =
Since nontrivial linear combination of {f, af 8²
De PG. can vanish almost everywhere, these four functions are linearly independent when is fixed. This completes the proof of step
Step 2: Distinguishable property with respect to X.
Let us consider coefficients appear in each density factor.
Term related to Pa_ (Y|x): The factor appearing along with PG_ (Y| are exp(( X), and (6) exp((3*)T XX), where < S d. Suppose there exists constants c, ai,..., aa such that
cexp((3")T X)
a (1) exp((3*) 0, o.s.
t=1
This equation means that + Σ (1) 0, Given that X has non-vanish almost everywhere density function, this relation implies that c = 0, a; = 0, < i d.
Terms related to f(Y]k(XX v²), The factors appearing along with f(Y))(XX are exp((8*) X), and X exp(((*)T (T X). where 1 < i < d. The identical argument as in the case for PG. (YIX) also gives us the independency.
(X),
Terms related 0f (Y)n(X, m)",, v)): The factors appearing along with pG. (YJX) are
0h
0yj XX, 11" exp((3*) Xx), iisd
23

---

Suppose there exists constants 01, ........ Qd not all equal to zero such that
Oh Or1(3)
(x,1") exp(("²) x) = 0, a.s.
This equation means that
Oh a Ony())
(X, n") = 0, a.s., or vah(X,n")) = 0, a.s.,
where a = (a aa) This is a contradiction.
Terms related f (YIh(X,n") There is only one such term is
ahe
exp((8*) X) S (YIh(X,n")).
01e
Its coefficient obviously vanishes from the independent property with respect to Y.
This completes the proof of Lemma 1.
A.2 Proof of Theorem 2
As a first step in proving the minimax lower bounds for the distinguishable setting (Theorem 2), we define two distances:
dz(G1, = exp(r))]((1, m, 11) - (32, 72, 12)11,
d2(G1, G2) = [ep((T) - exp(72)
for any G1 (1,,T, n, mn) E E and G2 (B2, T2, 72, 22) E. Obviously d2(G1, G2) is proper distance. The structure for di (G1, G2) tells us that is not symmetric. Only when Ti = T2 = T, di (G1, G2) is symmetric. Also di (G1, G2) still satisfies weak triangle inequality:
(G1, G2) + di (G2, Gs) 2 min(dr (G1, G2), d1 (G2, Gz))-
Therefore, will apply the modified Le Cam method for nonsymmetric loss, as outlined in Lemma C.I of [9], to handle this distance. For f satisfies all assumptions in Theorem 2, based on the Taylor expansion, we have the following results:
Lemma 2. Given f in Theorem 2, we denote
n. S = (T, Fr.,1,,1, v1), S2 (r, B2, M2, M2), and St (7,, (1
we achieve for any r < 1 that
Ex Id (ps, (-1X), Ps, (+1X)] d (2) SE =0,
lim inf
c S1.82
d' (51,
Ex(da (ps; (-IX), Ps; (. (X))]
(ii) lim inf
: (S, s =0
+ S4.5%
dī (S), (S),))
We will prove this lemma later.
Proof of Theorem 2. Denote G. (5* T" ,7",, v²) and assume 1. Given Lemma 2 part (i) for any sufficiently small > O, there exists G², = (3T, "" such that (G, G.)) = di dG.",, G.) e, there exists constant Co. s.t.
Ex (d (pa. ([X))PG:(( (+1x) S Coe".
(20)
Now will denote pa. as the density of the n-i.idd. sample (Xi, Yi),... (Xnx Vn). Lemma C.1 in [9] tells us that
2 - (1 (8
inf sup Epa (exp The Dn) (3, w)11² G, Gez
2
24

---

c² 2 V- Cjeer)
Last inequality is from the definition of the Total Variation distance and Hellinger distance and equation (20). Let 2x
Ch then for any < 1 we have
inf sup Ep (exp 7h, Pn) (a, b, x)ll)) 2 ein 1/rr G. GEE
where ej is some positive constant. Following similar reasoning and using Lemma 2 part (ii). we will obtain
inf sup Epc (Iexp(Fx) exp(+))²) 2 c2n 1/r, G. GeB
for some positive constant Consequently, we establish all of the results for Theorem 2.
Proof of Lemma 2 (i). Consider sequences
S1.1 (Fny a1 M1 v1,n);
S2, = (Tn, $2 12,, V2,n):
with the same Tu. By the contaminated MoE model definition, we have
Ps,..(V()) 1 exp(3 X + +
holY)oo(X, no),
exp(8 X + Tn)
f(Y(h(X, Ti.n), Ujsn);
1+ exp(3) X + Tn)
for j 1,2, Since (Tn. Bj.)) lie compact set, and both fo and f are non-negative. Hence, the squared Hellinger distance satisfies
(PS... (Y Ps.. (Y[X))
Ex [d (PSi.n (-1X), (-10)) SC|
d(X,Y)
P8a.. (Y(X)
/ XX((XI,.)) mL,)) 11,n) exp X) 12,,)) v2,)) < C'
d(X,Y),
exp(B ()p(B,,,))(()) XY (XX, v2,n)
for constants C, C' depending the compactness bounds. Consider the Taylor expansion of the map
(3, n. v) + ex X)ʃ ()))(Y)))
at the point (82,.: 17,,, n,,,), expanded up to first order with integral remainder. Let = (a), 022, 0:a) denote a multi-index where an E Nd, 02 E N9, and drg E index components of 3, n, and v, respectively. Then have:
exp(3T,nx X)f (Y))(X, n,,,, v1,n) - exp( X)f (Y h(X, 72.)). v2,0) (3i, - 32.n (M) m2,)) )02 (v1.n - U2
Σ
aazlog!!
|
alaz f a exp(B_,, X)
Opas (//(XXT2.nn)..0
(Bim B2, (m, T2, 02 (vi. 12,n) f x exp ((3g.n t(31.n -),,n x) a laggaa!!
=
flaaltan
(Y)h(X (1lhX,72,, n2,n 1(m,n (2,,)), V2,1 + t(v1,n v22n)) dl.
25

---

So it follows that
Ex (d (psy.,, (-1X), P82.. (18)),882.. (-1x))
(Si(Si,n, )2.n)
since Tn lies in compact set. This establishes part (i) of the lemma.
Proof of Lemma 2 (ii). We consider two sequences
$1.n = (Ti Bno tn vn);
S/ = (T2, Bus Mu vn
with different Ti, * T2,n but the same (Bn, In 1/)) Using the contaminated MoE definition, the difference in conditional densities is:
ex (e ETL.))
Ps,, (Y Ps;... (Y|X)= = (1 + e X x+)
(1+ x0
By the standard bound for squared Hellinger distance.
Ex [x[d2 (S),, (x))) sc c
(P8%... (YIX) ps,,.. (YIX)
d(x,Y).
PSJ XY
Since (Bn, fay Mn) lie in compact set, and both and are bounded away from zero, have pS (Y|X) 2 e > 0. So the denominator is lower bounded.
Then there exists constant C' such that:
Ex [dy (Ps;,,. (IX),ss;,. (+Xx)) SC -
Now recall the definition of the distance:
d2 d2((T1,no BrG Mas Vn), (T vn)) = le 222.1
So we conclude:
Ex [df (ps; (.I),,s) Ps: (+X))] d2((5 S/,,n )
C" le g
C' l' -222m(((--))
letia e22.0 2r
as long as ei. 22. + and r < 1.
Hence,
Ex Ex (Ds, ((xX),ss, (+1x)) 0,
d5 S/,n))
which proves part (ii).
A.3 Proof of Theorem 3
We proceed to prove Theorem for the non-distinguishabl setting.
Proof. Let T (3, F, ", D) and (77, D) can be identical to (7.0%)). Then, we will show that
(i) When (No, 21) + (7. v),
{
[|GG pa. D(G. >0 D,(G,G.)
lim inf s G.G.
26

---

iii When (7o. vo) = (7.
{
IIpG - Pa.la (2(G, D2 G) s 0 >0. 0. D2(G.G.)
lim inf G,G.
(21)
Part (i) can be proved by using the same arguments in the proof A.I. Thus, will consider only part (ii) in this section, specifically the most challenging setting that (No, vb) (1) D). Under this assumption, we know that ho and h are the same expert function, s.t. fu (X, mo), 2)) (X) no); Mb) for almost surely (X,Y) E X x y. Assume that the above claim in equation (21) does not hold, then there exist two sequences G. (Bn, Tny Th: Un) and G., (gn, Ta li: va). such that
D2(Gnn +0
D2(Gr G)
[IGG. -
G.,n) + -
We now analyze the limiting behavior of the sequences (Az, Gx) and (A;,, Gū)) as they approach G). In particular, we distinguish between three asymptotic regimes based on how the expert parameters S. (Mn v.) and s = (hh, vn,) converge.
First, it may occur that both Sn and S converge to the same limit so = (No, vo). Alternatively, both sequences may converge common limit 5' & So, which is distinct from the true expert. Finally, it is also possible that one sequence converges so while the other converges to a different point + So-
In the following, we analyze each of these cases and demonstrate that in all scenarios, the assumption that the normalized difference vanishes leads a contradiction when fo f.
Case 1:
At first we consider that (1n, vn and (vh, (/)) share the same limit of (Nu, vo). Without loss of gener- ality, we can suppose that T 2 Tn- Subsequently, we consider W,, == [PG., (Y PG... (Y [)] [I + exp(();) X + [; [1 + exp((Bm) X + Tn)l, which can decomposed as
Wn exp(Tr) [ Bn, Th kn) - |X 8h 7% =]]] exp(TT) 19 X: Bn, n7, v0) g 3% Th v77]
+ exxp(T)) [x(;|X; 87 70, vo) g( [X; 8% nhe v2)
+ +xx (08) + [Bx)T X + T +T.) [XXY v,), - (XY))(XXn)))))))
II HI, + III, IV,
where we denote [X 3, 7, v) = e(x:
= exp (8Tx) X) (Y|
We expand around the reference parameters 3 ne Vn, where the parameter differences are given by Annn 7/- - '7u, Av, = Vn Vo. and An = the nJ. Au; V,, Vp. Applying a second-order Taylor expansion, then we obtain:
d
I. exp(TT,
Σ |
1
II
(B. - -B,) (w)
II (A - ( (Dv, Auy)os ==1
ad
==1
glalg
az am dpay (Y 3r mh v + (x(,))
<xp((Tn)
Σ
[8,, 3 (w) II I ( (Ain A
!20g
10|=1
exp(();) Tx) ' glaalh h
glaz|+2005
Omlas! (X, mi)
(Y (X, (4, Rı (X.
@hlaglt2ag
(22)
27

---

where B) is the remainder term containing higher-order terms, and the second equality is due 001 Similarly, we will have that
dy 2 26
II. exp(Tn)
Σ
II (Av)) ==1
aa/=1
a g 0 01p Duas
(Y (1 Pn (n (x, r)]
III exp(T;) Σ a!
1
[A
(0)00 (A as alalg
(Y B Ya Un) Rs(X v)
1a/=1
IV,, exp(Tx + Tn) exp ((5) + xx)
Σ |a|=1
(Ann (0)00 (Avn A
al
f
(Y (Y The v,,) (x, y)
dipervas
Then, grouping the terms according to the order of derivative 7 |a2| degree == [01], we can rewrite the expansion in the compact form:
20g and the monomial
SS (Y ((,, m,), exp(( x) x + R( Oh7
L
(x)
C=0 L7=0
where each coefficient la,y,e(X (X) depends on the parameter differences and derivatives of with respect to 1). More specifically we have that
(,,,1(() = exp(Tn)
Σ
(Bn - -3,
1swSd
(Bn (0) (3n - (r)
(x)) = =xxp(Tn) Σ
1 1m=r
Isw
h,,,(()= =ex((T) ((Ann - (n)) Oh
(X, m/))
Orp(c)
(Amm An)) (1)) (A (n7
82h
)n(()))())) (X, m/m)
1 1u=v
15a
Oh
Σ
1(3n - B:)(w]](AAm - Anj)lul] amp(w) mm)))
1 $uu
I2,,0(X== = cxxp(T))
Annn Av
2
(Am (nt)())(An)) (n)) Oh
Oh (X, min anp(o) (X, min)
1 1u=v
Onpu))
(x2,1(x)= = exp(Tn)
Σ
(Bn - (w)(()) (Avr - (w))(())
2
Iwwdd, Sung
exp(Tn)
Oh (nn Au7)) amj(a) (X, m1)
Ix,3,(()==
Anja
[x,4,(()== exp(Tn) (Av, A,7)²²
Similarly, we can rewrite II, in the same fashion as follows:
II, Un. (x) f
0h (Y (Y)h(XX mi), vG) exp((3;)T xx) x² (2
28

---

where
11,,0,1 (X) cxp((n)
(3n (5)((0)
Idw
(5n-
B3)
(xx,, exp(Tn)
1 + -w=r
Isw.rid
4
Oh
= =x)
Σ
[(
(1-Ann
(X, ,1))))
@ny(a)
(-Ann) (-An))
8ph
+
Σ
(X, m7)
1u=e
Org Onp(w)77(0)
15u.59
Oh
(x), = exp(Tn)
Σ
[Bn - (((
(X, m72
3n7(w)
Iswsd.suso
1
(-
Oh
8h
(x),,(()==coa exp(Fn)
(-vV))
+
On((w) (X, ni) Omp(o) (X, km)
2
1 11=v
1x), exp(Tr)
Σ
(8p- - -B))(w) (-)(w)
2
Iundd1SuSy
1x),,0(X)= exp(Tn)
Oh
-nni) (") )ny(a)) (x, (X,
2
u=1
(x), exp(Tn) (-)
In the same way, we rewrite III, in the same fashion follows, here the difference for 8 is zero, so all the coefficients with Ç + 0 is zero, but in order for the alignment of the expression, we will still express III, as follows
f (Y (((,,,, (h(,,n))))) v,) exp(((;)] xx) + (X,
III
Σ III ol (x) 2=1
0h7
where
Oh
Il 1xxx exp(T.))
Σ
1(-A-ni ((- Tyj(ta (X,
1=1
(-Am)) (w) (-Am))))
8h an(()))()) (X, (X.n
Σ
1+ 1u=e
(-Ani)) (-)
Oh
(X, nin) 2h Ōn((n) (X, mi)
[l,,,,(() exp(Tii)
ApNi) +
+ 1u=o
Onpaa)
15n.0S
exp(TTx)
Öh Onp("] (X,
mI,,,,(X)=
(m)
2
um
II1, (1,,,,0(X) exp(TTi)
(-Aun)²
8
Now we consider IV,, exp (13; + BB) TX + T + (.))) ni), v7)1, which is equivalent to
IV.- = Σ IV ( ff
(Y ( mi,, vnt exp((3)) X) exp(((k) + Rx(X,
7=1
29

---

where
2h (Amm - (n)
IV 1V)),(( = exp(Tn + +T))
Σ
Ony(w) (X, mi))
u=
(Amm Am)) (Amm Amg)()) 8hh
)r(())a((()) (X, mm)
1 lu=v
15a,"5
IV,,2,(X) exp(Tn + Tn) 11
( (Au A17)
(Ath - ( - Anj) Oh
Ory(u) (X, (X.1h) ah
)ep(e) (x, (x.1%)
+ lu=v
15u,v5
IV nx,,,,(X)) exp(TT² + Fn)
Oh
(Ann Ant)(u)(Au,, Av Or((u)
(X, m7l)
IV IV,,4,(X)) exp(Tx)
(Avn Avn)?
8
Then could conclude that
1
n Σ
(n)) + 00)) + mm,,,,(()))
7=0
(Inxi(X) (l (x)n X 1V),,((X) exp(3))
<=1
f (Y (x(,,, mn)) vn) exp((3;) x).
Therefore, we can view the quantity Wn D22, G.,n)) as a linear combination of elements of the set L U KC, and - C, = U K where
L., (X 1)), exp X))
C02 (XXT f(Y nh v/)) T X)}
Oh
(X, n X 05 (Y)((X, nhi, vni exp((3;) x) a E [d]
L1,
0h
X OS
(Y|,(X, (1(,)))))) v(,1) exp [x)
22.1
2h2
Oh
(X, mil) 0f (Y (Y Thi, w/ exp(((;) XX) : Tx)
L10
eld
Onpla)
0h
8hh
(X, mhl 0f
(Y| (. ())) exx()) u [4).
U
anp(")a
Oh
S
C2.0
(Y|h(X, ni), exp(((,) x)}
0h
0h
(X, mi) 8
(X, mhn) 0n((v]
0he (x),(()))),
x) u, lal
Onp(a)
Oh
8 (Y)I(X, nh,, v/)) exp((3)) x): : well [d]
L3.0
(X, mh))
Ompla)
0h3
(Y (Y\ Thr v()) exp(((;) x1
L..
2he
and
{ ah
(X, (X, m exp((3.) x) of
X11
0h (3)( : [d]
())(())
30

---

int (1) (X, 1/h exp((3,)) x) Of
8hh
(Y|b(X thd), exp(((,)) X): u, E [d]
0h
SS
12 exp(((3,) T X) 01
(Y ( (() exp((3;) x)
Öh
(X, m 07(10)
Oh
(X, 1m (3.) (3,)
0²
(Y (1(,)((X [,,, v2)) exp(((7) x) x): u, E lul
Oryla)
2he
K3
Oh
(X, mm) exp((3.) X) 801 I
(YII(X, mm) viii exp X) u E [d]].
(ny(n)
Th
exp(((..) X) f (Y ( n, v (x) [x)] x))
K44
The
Assume by contrary that all the coefficients of these elements vanish when - Looking the 0h coefficients of By(")) (x,, 0f an ( m.) v) exp(((;) (X)))))),) get for all w E [d], u [g]
exp(Tn) [(3, - (n) ( [DD(G + +0
(23)
Looking at the coefficients of x (Y (x 1 ( (xp(((3)) X), we get for all w E [d]
- / Gr,n +0
(24)
8hh
0f
Looking the coefficients of 0rl(w)87(()) ah all u, E [g). (exp(Tn)(Ann) Ann)) (n))(r) + (exp(r;) -
(X, m
(YIN(X, (,), exp(((;) T)), we get for
DD2(Gn, Go.n) 0, (25)
Öh 0f
Looking at the coefficients of Tnjla) (X, th) Th (Y (YIM(Xn)h)) (1), exp((()) X). we get for all
E [9].
(exp(Tn)(Ann - (n + (exp(T,)) exp(Tn)((AAn)) DD2(Gn, G.,a) + 0,
(26)
Looking at the coefficients of f Th. (Y|n(X, m7)) exp)))(())) X), we get
(A) Avū)) + [cxp(T,)) exp(T, )]( -v,i)/ D22 G.,nn) +
(27)
Sh
Looking at the coefficients of dml(w) we get for all u, E (g).
(X, mi) )y((")
(X, nin
(Y
(2j) exp((3;)T TX).
012
(exp(Tn)(AT). -
An))(())) + (exp(T,) - exp(T, (-An))
DD2(Gnn Go,n) - 0, (28)
Th (X, n.)
0S (Y (2,,)) (x)( we get for all
Looking at the coefficients of an(())
ah
E [g].
(exp(Tn)(An. - Ant))ul - Avt)) + (exp(Tx)
(xpmt,))(-Am))
DD2(Gn, G.m) - 0, (29)
Looking the coefficients of 8'ʃ T (Y]N(X, ni), exp((())) X), get
(exp(T,) - -2 + (cxp(r.) -l G.,n) + +0, (30)
31

---

0h
05 0h ( (i)) exp((8;) X), we
Looking at the coefficients of get for all u E [4].
Ōny(w) (X, mn) exp(),,)
(exp(Tá + - -nn) Go,n) -0
(31)
8h
Of (Y)h nn) v,), exp(((x)) (B X)
(x, exp((3)) Looking at the coefficients of (n(()))))())
0h
we get for all u, E [g].
[exp(TA + Tn) (Anm Ann) (Am An Gx,1) O,
(32)
Looking at the coefficients of exp((3,)) 8f (Y) (1)((X,,)), ,/), exp((3;) we get
2h?
[exp(Tn + Tn) - Av,)) G..) -
(33)
(X,1)) X) 83 Oh
Looking at the coefficients of Tntor we get for all u, E [g).
8h
(X, 11 लy(()
ah
(Y)h(X, (i),( v2)) exp((())) x)
[exP(T, + (anm -
(m)) 1 G2,n)) - +0
(34)
exp(((n) XX 8
Looking at the coefficients of Tلjt get for all u E [4].
(X, The (h, ah
01.3 [1(X m,), exp(())) X), we
exp(T, +
(Avn Avti/ G..)) +0 811
(35)
Looking at the coefficients of exp((3,) th (Y)h(X, mi),
XX), we get for all
[q].
(exp(Tã + (A) Ar G. + +0
(36)
Now, combining (23) and (24), recall that all the gating parameters are in compact sets, and applying the Mauchy-Schware inequality followed by summation over coordinates, we got that
exp(Tx) Iill Av, DD G..n) 0.
(37)
While it is intuitive that the similar result holds for || Avtil [|. a slightly tricky handle should be employed here. Suppose that
exp( 11 G.,n) A0 0.
By combining this assumption with equation (23), we have there are at least one coordinate u such that (AAn))()/(())) /(Ar,)) )(0l + xo, which implies that AAmt)/(Anr)) - An,) + 1. Thus, by multiplying
equation (31) with (Anr) An,) + 1, we have
exp(T))(An)) (m) DD((G,, G.,)) +
Also noting that ||3,, - B% || is bounded as the parameters belongs to a compact set, we have
exp(T,))1.,, / Gr,n) + 0,
which is a contradiction here. Thus, we have
exp(r,))] 31 // G.,z)
(38)
Similarly, also by combining equation (24) and (33), we have
exp(T,))13,, - S Av ////// Gz,n) +
(39)
As result, have
- -xpp77))1...82111103 IA, Awt))l G..)) +
(40)
32

---

In similar manner, by considering equations (31) through (36), we obtain that
exp(Tn ) (A. Av,) (Amh: Ax,) /D Gam)) +
(41)
Let u = the first equation in equation (25), we achieve that for all u E [d],
(exp(Tx)](A>)) An,) + (exp(r,:) - (etp==e))\))((A)))))))///// Gr.)) + 0, (42)
which implies that
(exp(Tn) (I - An
-x Gr,n) - 0.
(43)
We also have each term inside equation (43) is non-negative, thus (cxp(r,)) - exp((,)) |l D2 G.,n) - 0,
(Tp(Tn)lAAT, An, / D2(Gn, G.,n) + O.
(44)
Applying the AM-GM inequality, we have for all u, [d]. (exp(T,)) (A)) (Am,) 0, (AT)) (mr (ATT Amg)
0,
D2(Gnn Ga,a)
D2(Gnn Go,n)
(45)
0h
0s (Y ())) exp((3;))) and
Next, by considering the coefficients of Öry(w)
(X, mi)
Ōh
0² 0hs
(Y ((X,, mh), v7))
we have
(exp(Tn) (m)) (Amm) 1x G.,n) - 0, u u [ (exp(Tn) Avn) - -exptt)((vv)//22 G.,n) - 0.
(46)
(47)
Noting that for u, [d].
(Xp(F (1) (Ar An (1)
= (A))(TT))An),)))) )()) exp(T, (An) (1)) (exp(T,) (x
exp(Tx)(A1T)) (w) (Am (n))
cxxp(T)) (1)) (Arn, Ant (v) (cxp(T.) |n (w)) (n7
Thus, from equation (45) and equation (46), we achieve that for E [d].
exp(T;)(An)) (() (An /D2() G.,n + 0,
exp(T,)(AAn.)) (1) (An,. (n G.m)
By using the same arguments we will derive
exp(T,,) An,, Al Antll/D2(Ga. G.,n) 0, exp(Tx)) An, ||-|| - Anjil//22(() Ga,n) + 0,
(48)
(49)
By using the same arguments derive equation (42), equation (44) and equation (45), we can point out that
((xx((T)) exp(T.)))lAx.Il 1|2 + exp((T.) ) Avn A G.,n) -
exp(T,) Av || A Dxll//D((Gn. G.m) + 0, exp(7,))
|| Av Av, Au, D/ G.,n) - 0,
exp (w)) Am) ||A A (2../ G.,)) + 0,
exp(7,))
AR, I/ D2( G.,..) + 0.
(50)
Collecting results in equation (37), (40) and (41), and equations (44) to (50), we obtain that
1 = D2(G,, G..n)/ G.,). + 0,
33

---

which is contradiction.
Therefore, not all the coefficients in the representation of W,, Wn/ D2(G G.,r) tend to 0 oo. Let us denote by mn the maximum of the absolute values of those coefficients. Based on the previous result, 1 1/mm A co. Additionally, define
exp(Tx)](3,, B,) (*) [(Amm)
(00) (10))) exp(Tn)[(B, (0)( (Avn)) ///mm/mm. 0:21,
- 01 www0
(
(exp(Ta)(AM), Anj)) + (exp(rx) - exp( - (0)) mm 001,,0a0
(exp(Tx)(An) Ann) (A7T2 - Am7) + (exp(T,) - -xp((x))(-Ar))) (m) (-An,) ) (v) mm.
310,00e
[exp(Tn) - A + (exp(T,)) - (e Amt)() (A (m)) (exр(TT)) -
m + 020,,000
(-An (0)) 1/mm
320,000
(exp(Tn)(Am) (A) - Av + (exp(T;) (x (w) -- )) mmm..
$30,000=
(exp(Tn)(Avn - Avii)² + (oxp(T;) - exp(Tx)) - mn,
//m,, $40,000,
exp(Tī (nm )n, /mnn + P1,00, (1)
exp(TA (An An7 (1) (A An) (1) m/mn 1,80
exp(T (AV,, - Av2))/m, + P2,00,
exp(Ta (Amm )
(w) (A A (0) //m, 7220
exp(T; + (Am Anr) (0) (Avn - Av + T3,u0-
eXp(T, + Tn 7 Au2)² //m - T4,00-
(51)
when + for all w E [d]. u, v E [g]. Note that least one among Cr Src and Pyaav; uy where E [4].5 E (0, 1} must be different from zero. By applying the Fatou's 'ss lemma,
we get
1 2Ex (po. (.1X), PG. (...))))))))
/
/limint n-0
IpG. (Y PG....([X) d(X,Y).
0 lim
2
n 7-000 mn
D2(Gn, G.,n)
D2(Gn, G.,))
On the other hand,
1 PG.. (YIX)- PG.... PG)..(Y())
Mp
D2(Gn, Ga,n))
0S (Y)M(X, no), exp(3 x),
Fac(X)x + K2(X) ex x)
Oh
7
5=0
where
Oh a11,ww0 dn((x)
Ex1(X)
Σ 1Swdd
(X, nm))
1
Ex1(X)
0221,000
Isw5d
Th
B10,0ur
82h
Exo(X)=
010 Om((a) (X, +
Σ
(X. (X.1)). mo).
1 1 u= On (n((0))p(())
u=1
1 1Sn
1
320,000
Oh v=v Onp(
Oh
(x)
0,20,000
(X, mm)
(X, no):
2
Omy(o)
15u.eg
9
Oh
(x)
Fa0,0u0 @nj(n)
(X, no),
2
a=1
34

---

Eso(X)
Bao,or
8
and
Ki(X)= Σ
Pl,
Ory(u)
Oh
(X,
Σ
1 + 1 u=v 01y(0)
Tl
8h
(X, no),
1SuvSq
K2(X)= 2 P2,00 +
1
Σ
2,
Oh
(X, 110) 3ry(1) (X, vo):
Oh
+ 1 u= 0ny(a)
15u,5
Oh
K2(X)=
Σ
7,000
Onyln) (X, n)),
2
==1
Kx(X)= 44,00-
worth noting that for almost surely (X, Y). the set L linearly independent under non- distinguishable setting, which leads to the fact that (x) (X) 0 for almost surely X for any 7 [4]. Ç E 10.
Similar to the proof of Theorem 1, and recalling that the experts are strongly identifiable, we conclude that all the coefficients in Equation (51) must be zero for all w, u, u.
This contradicts the fact that not all coefficients vanish. Thus, we obtain the conclusion for this case.
Case 2:
In this case, we consider that (1h, v/) and (na, vA)) share the same limit, but different from (No, vo). From the formulation of the metric D1 in the proof A. it clear that D2 S D1. Therefore, we - co. Noting that (May Un) and (f V5) share the limit
get (Y,(XYY/DD) (T)DD((G,, G.,n) 0 (1)" "y",, v²) * (No, vo), we have fo = [(Y [h{ (X vjo), uo) and S(Y[h(X, 1)),,, v²" satisfying So and f independent up second order in Lemma Thus, can process similar way in Theorem to draw a contradiction.
Case 3:
Lastly, we consider that one of G, or G converges to Go, while the other converges to G' # Gu- Without loss of generality, suppose that G, + G¹ and G + Go. By passing through the limit for
Ex Ex|hv (.. (1)))))) G;)) - 0.
noting that
D2(G,, GT) + D2(G.G.) (O..)///OO/////...XX)......... + Ex(hv(pa((xx),po. (11x)),,
we have
(xIhv(pa(1xX))pc =0, pG pG.:
This equation implies that
J(Y])(X, no), T exp(B + 7 (Y11((X,10);)))) v0) exp(3 TT X +T*)
exp( + + (Y(1M(X,)))))
which further implies that
exp(BT X +
X x
1 + ex x +T) ())) 1 T
X + / ((Y()(X
and hence
f(Y]h(X.no) = f(Y]/(X,)) v) (as cxp(8 X r") * 0).
This equation means that G' = Go. which is a contradiction.
35

---

A.4 Proof of Theorem 4
In what follows, we present the proof of Theorem 4 for the con-disiinguishable setting.
Proof of Theorem 4. The proof follows similar steps the arguments in the previous sections. Concretely, define for S = (T1, 1/ 11), S2 = (T2, B2, 72, U2)
di (1 52 = | Au 11 -xp(T22)) dn (SI, = (x) ) Am. Avi [|II(3,, m, m1)
(32, (2, 12,
12)l
It is straightforward that d, and du satisfy the weak triangle inequality. Following the same schema in Lemma 2, we can demonstrate two subsequent results for any r
(i) Two sequences can be found
S1,x (TI, 3n Mo: Un) E(In),
12,0 = (T1, 3n, Mh Un)) E 3(In),
such that (S n. S2,n) - and Ex (1 (o1..((11))))))))))) (11), (. (-1x)) S2,n) - 0 as
20.
Two sequences can be found
{ Sn, (Fri B1,n m. V1,n) E(hn),
12, (Tni B2 M2 v2,n) E((n),
such that dy S2,n) + and Ex(lh((ss.. (+IX), PS2.. (1X)))// S2,n. + 0 as
n + D0.
We can omit the justification for the above results as it can follow similar approach as in Lemma 2. This leads the conclusion of the theorem.
B Proof of Auxiliary Results
B.1 Proof of Proposition
Proof. Fix arbitrary E X and abbreviate
91 (v( (ylh(z, v1). ge(y) == (ylh(z, (2))((2n2))..)) go(y) = fo(ylbo(z, mu), vi).
Because f is Gaussian its argument, there exist pl, u2 E and of og 0 such that gy(y)
exp(-(v (2)3)(203)) for j 1,2.
2
Set
H1 (y) =
0gz 3h. (y)
1/ u12
g2()):
Hz(y) = 8292 Oh?
(v)
(y) - 122)
o
9o (v).
of
o²
With these notations the assumed identity becomes
+ (22( =0 for a.e. y E R. (52) (o(x)go(v) (x) co()) + (z)
1. (z) = O. Because go is not Gaussian by assumption, while 91, 92. H1, Hy all belong to the
linear span == span(y +> 9i (v). (y- - n22 / g2(y) == 0, 1,2,, we have go G. Hence the only way (52) can hold on set of positive measure is with bo(x) = 0.
36

---

2. Linear independence inside G. Divide (52) (now with (o(z) = 0) by (2(U); we obtain the polynomial identity
(v 92(U) + co(x) + ei -ー
(xx))" -21 u2 + f02(() (U u2)² 02
bi (x)
oz
for a.c. y/-
02
The ratio 91/92 D2 is the analytic (non-polynomial) function
91(y) K exp [ u2) -y-
g2(u)
with * Since μl # 12 or of + ozz this exponential term cannot be expressed as a quadratic polynomial y. Consequently the set of functions {g1 /2. 1. - 12, (y - )22²²} linearly inde- pendent on any interval. Hence every coefficient in the polynomial identity must vanish:
bi (z) co(x) = ci (x) = c2(2) = 0.
3. Conclusion. We have shown that bo(r) = b1 (z) co (z) = (z) 02() 0 for the fixed I. Because the same argument works for almost every x X, all coefficients vanish almost surely. Thus the unified distinguishability condition of Definition is satisfied, completing the proof. n
B.2 Proof of Proposition
Proof. Write the two (single (single-expert) expert) conditional densities
po(ul=) = [1 - M(x) fo (vlh,( no). vn) + (x))
Par (y/[z) = =[1 - X(x)) fo (ulhola no), + ((=))((ll((,
exp(8 (8TZT)) and ((xx) 1+exp or (Tttr)
exp(3' (+t
where ((x)
14exp (0*
Assume the identifiability equality pa(ulz>) por (ul=:) holds for almost every (x, X V.
Subtracting the two representations gives
[A(x) fo (y ho(z, vo), X(==)(
(x))((lh((,,)) -. (53)
Step 1. If (xx) + (xx). Suppose on a set of positive z-measure, A(x) + M(x). Divide (53) by
((r) X (x)); then for those
fo fulyyoa vo) ((t) + (1)
where
X(x)
-N(x)
b(x) == X(x) A(x)
c(x) = X(x)) - A(x)
Since f is distinguishable from fo. the only possibility is b(x) cxx) 0. hence A(x) X(x) a.e. a.c.-contraiicton. Therefore
X(x) X(xx) for a.e.
Because the soft-max map (B,T) + X(-) is injective, we conclude
β B ==T.
Step Equality of expert parameters. With A(x) = X (x)), equation (53) reduces to
I (y)M( n) v) = ((y)/(a,
for a.e. (x.
Definition 1 forces the situation (n, v) + (T/, v') impossible. Hence the only consistent solution is
(n, v) = (n',v').
Step 3. Conclusion.
== We have shown B B., = T', 1 n/, and 1/: hence G G".
37

---

B.3 Proof of Proposition 3
We begin by introducing several standard notations used throughout this proof. Let (P, be a metric space, where d is metric on P. An e-net of (P, d) is collection of balls of radius whose union covers P. The covering number [ec, P. .) denotes the minimal cardinality of such covering, and the entropy number is defined as (r. P,d) == log (e,,
The bracketing number NB P. d) is the minimal number of pairs [L, such that L < J.s d(f,,,, < e, and P is covered by the union of the brackets. The corresponding bracketing entropy denoted by Hn(e, P,d) == log Na(e, P.d).
When P is a family of densities, we take d to be the (m) distance, where m denotes the Lebesgue measure.
In particular, let P(E) = (px : A E E}, and define the symmetrized density Py == (0 + Px), where p" denotes the true density. We then define the following sets: P(E) == {px X E 3} and
(E) ()) = tok PA P(E))) To study convergence rates, we consider the localized version of the symmetrized class: p²/²(E,e) == (0) E p1/² (3) du (px.p") S e). where d=((.-) denotes the Hellinger distance. Then we assess the complexity of this class via the bracketing entropy
integral defined in [34]: JB(c, (E," m) == S2y22 a V Hs(u, P'/2 (E, m))d V 6, where V == max b}. For brevity, we may omit the dependence on m when it is clear from context.
For the proof at first we consider general lemma that provides the desired convergence rate, provided that bracketing entropy condition is satisfied.
Lemma 3. Assume the following assumption hold: Given a universal constant J > 0, there exists N > O, possibly depending on E, such that for all N and all e > (log(n)/n)) we have
JB(e,P" (E, e)) S Ju/nee
(54)
Then, there exists constant C 0 depending only on E such that for all 2 1,
sup Ep... Ex|d (Pa., (-IX), PG. (+1x)) & C-/log n/n. n.
G.EB
This lemma indicates that suffices to verify the entropy condition Equation (54) in order to obtain the convergence rate. However, this condition is often technically difficult to establish directly. As workaround, we may instead prove the following sufficient condition: Lemma 4. If the distribution satisfies
HB(e,P(EE), dz) S log(1/e),
(55)
it will meet the assumption in Equation (54).
Although we have simplified the condition in Equation (54) to Equation (55), verifying Equation (55) still nontrivial. Fortunately, for the contaminated model defined in Equation (1),
cxp(3TX+7)
PG
fo(YYho(XX ho no), vo) + 1+ exp f(Y[A(X, v)),
1 exp(8 T)
we assume that So is bounded with light tails and that f is a univariate Gaussian density. Under these assumptions, we can verify Equation (55) via the following lemma:
Lemma 5. Let I be compact subsets of Rdd R and @ be a bounded subsets of TR? x R+ f is univariate Gaussian density and fo is bounded with tail Ex (-log fo(Y ( vo)) 2 Y for
almost surely Y E yor some q > O. Then, for any < E < 2, the following results hold:
(i) log (Cc, P(E), II (1x)) & log(1/c).
(ii) HB dn) $ log(1/e).
Combining the above results, we obtain the desired conclusion for Theorem 3.
Now we will prove Lemma 3, Lemma 4 and Lemma 5 in order. At first we need to introduce another Lemma 6 before we prove Lemma 3. Lemma is Theorem 5.11 in [34] and its proof can also be found in [34].
38

---

Lemma 6. Let R > 0, k 2 and G is subset in E where G, E g C E. Given C1 < for all C sufficiently large, and for n and is in the following range
s (8VTR) (CIVnIl /K),
(56)
I"
n
2 (CI
RV 1/(28/m)
H 1/22
p 1// (=,
(E, R), m)du)
(57)
V2
then we will have
sup PG... ( GEg,Ex (MF (+10)) (1))ER (G)] S Ceep
12
CC(
(58)
Proof of Lemma 3. Firstly, by Lemma 4.1 and 4.2 in [34], we have
Exld (l. (+X)) s Ex Id (Pa,
S
Vn Hn (Gn),
16
here Hhn (G.) is an empirical process defined as
Nn (m. = vT
log
Pā.. PG.
(fe.
-PG.)d(X,
PG. >0
Thus, for any 8 > 6n = Vlg fogg//n, we have
Po.,, (Ex dd (pa, (-IX),PG. (+Xx)) 2 5) Mn -na
vTlxx (,a (-1 (+x))) 2 0, Ex (1Ex =
< Pa..
(4x0)))))) [Mn(G) VTllx (1) 20)
< <Pa..
sup
8
(po1://BB{(uoooxxxxxxxccxc((s111 |,, Vii220 ()2)
Σ Pa...
sup
==0
8
|mn(G)] 2 V1222
8 (33)
PG
sup G-Ex (du (eo.
==0
where S is smallest number such that 255/4 1.
Now we will use Lemma 6: choose 2-+18, C1 15 and 1. 1= (n222=2222)))))) We can confirm that condition (i) in Lemma 3 is met since 2--18/4 < for all S S. For the condition (ii), it is still satisfied since
R
H1
DL,2 R),HH du v 2
11/20
(/2''
R\/22
V2
11 1/2 (u, (E),((, du v 2
RR//118
< 2JB (2,p1/2(E, (), R1,
< 2 182
2
Now since the two conditions Lemma 6 are all satisfied, we could conclude that
Pg... [Ex|dn (11)), Pa. (-XX)) > 8) s C
exp
214
n202
S cexp
(-no²)
n182
(59)
0
39

---

here constant is large constant that does not depend on G.. Now we could derive the bound on supremum of expectation:
Epa... Ex (d (x
/ IP P(
> 6) dō
( exp
(- (-no²
s òn +
Jon
d5
S čd
here is independent from G. and on Vlogn/n. So we can conclude that
sup E Epa. n Ex ExIdd(pa.. (1x)) (+x)) S Cllogn/n G.ea
Proof of Lemma 4. Because 1212 (3,0) (3,) C 71/2221) (E) and from the definition of Hellinger distance,
we have
HB(5, (5(6,T" (E,0). (E,),) Hb(6, P 12 [1) HB
( ((E),))
Now, using the fact that for densities / f1, /2, we have 1i2 (Lu ,ar("
((((12)), it easy
2
to verify that H (6/v2, 2, P(3), - (6, P(E),dH) Hence, if equation (55) holds true, then
Hp(6, P 5), (E,))) =
H6 P(B), $ log ())
This implies that
213 Se log( 2 < ne², for all de >
log n
JB
n
Proof of Lemma 5. Proof for (i): Let (S) denote e-net of set under the || ll norm.
Then
18, (S) = log (e,, lloo).
Let P(O) = (pr: Y θ), where pr (Y| f(Y(/((XX)))) v). By Lemma 6 in [13], we have
log P()), (1 & log(1/e).
We now consider the contaminated model pr as composition of smooth components indexed by (8, r, n, v) E E : I x O, where I C Rd++1 and θ C R9 x I are compact.
x Since ( X +T) x +)))++++++eee
X +T)) is infinitely differentiable and Lipschitz (8,7) EL') such that
over compact I, it follows that for any X (B, I, there exists
exp( 3 X 7 1+ exp( (3 T X +)) 1 1 exp
exp( X + +T)
||ox xx sup Xex
Sc.
B x + 7)
Likewise, for any Y = (1, E e, there exists T E (() such that
|pr Pj Pillcc loc Sc.
Now, consider the difference
Px (2(()
(ox(X) xx))) (F(VIN(X,))))) -o(())oo(X, no),
+ox(X) -M
40

---

so that by the triangle inequality and boundedness of fo and f,
l|GG - Polloc & 1 xxlxx (Ilfollec + (1f + 1laxll lppr - py aoo
S..
Hence, the covering number of P(E) satisfies
log P(E), (lx)) & log Ne,,r, |l (l log (c, P(O), (1(b) S log(1/e).
Proof for (ii): First, let r) S E be positive number, which will be chosen later. We consider f is the density function of an univariate Gaussian distribution, so f is light tail: for any Y 2 2a and X X.
I() = 1
Y²
exp
V2
8u2
Also fo is bounded with tail log (o (o), vo) x -Y4 and fo (Y no)), vo) S M for almost surely Y y for some M.q 0. Now let q min 2} and C2 max {M, 1/111//2/// V2rl).
we will have
HX) = exp(( C2,
Y 2 2a Y < 2a
(60)
here Cj is positive constant depending on l and fo- Moreover H(X, Y) is an envelope of P(E). Next, let . IN represent an nenet over (E) Then, we construct the brackets (H,YY),,t (X,Y)) as follows:
(x.) = max (g: (X, n, 0} p (X,Y) = min (X, +7, H(,,))
= .N. As result, Pa(E) (X, min (X, Y)). Consequently,
and (X,Y) -xx,)))
- / (X (x, d(x,Y)
I (p) (x)(X)Y) (x, -() 1 J/11z0 -x d(x,y)
MK22
1 Mizza 2nd((X (X,Y)
H(X, ()(XX) Sn.
Y|22a
This shows that
HB(en, P(E), |1) < N log(1/n)
Setting 77 e/c, we find
Hc P(E),
Since 222 < || 1 holds between the Hellinger distance and the total variation distance, we conclude the bracketing entropy bound.
41


Reviews:
[{"review_id": "k2dLjPCKQG", "reviewer": "Reviewer_mgR9", "summary": "This paper theoretically analyses the softmax-contaminated mixture of experts model in terms of a distinguishability condition, minimax-optimal convergence rates, looking at the fact that estimation rates degrade, and also provide empirical validation with numerics. Importantly it addresses gaps in the MoE literature which historically has been quite empirical/application driven. There are few theoretical works in MoE, so it is a good contribution to close that gap.", "questions": "Can you add a discussion section discussing how this theory can inform practical model tuning decisions for users?\n\nCan the analysis be extended to hard-routing? Similar to: Approximation Rates and VC-Dimension Bounds for (P)ReLU\nMLP Mixture of Experts, TMLR", "limitations": "No real world experiments. Could you add some more realistic experiments for me to raise my score? Could you apply your experiments to a  more realistic prompt-tuning task.\n\nEven a simple simulation involving real embeddings (e.g., from BERT, GPT, or CLIP) with synthetic labels similar in spirit to the DINOv2-based experiments in 'Approximation Rates and VC-Dimension Bounds for (P)ReLU\nMLP Mixture of Experts, TMLR'  would really help. (could be a natural minimal-realism extension of their synthetic setup)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Addresses a novel problem\n- they  derive both upper and matching minimax lower bounds in both distinguishable and non-distinguishable regimes\n- motivated by modern fine-tuning practices like prefix tuning (although the experiments themselves are not really practical enough as later discussed)\n- good paper writing and structure\n\nWeaknesses:\n\n- All experiments are synthetic and low-dimensional\n- prompts are gaussian, doesnt cover other cases\n- there are no practical benchmarks, real NLP/LLM tasks", "was_revised": true, "final_justification": "I share similar concerns to reviewer 2SMz regarding the practical experiments. Due to this I will keep my original score.", "rebuttal": "Dear Reviewer mgR9,\n\nThanks for your constructive and valuable feedback, and giving **good (3) grade** to quality, clarity, significance and originality of our paper. We are encouraged by the endorsement: (i) the problem we address is novel and (ii) the paper has good writing and structure.\n\nBelow are our responses to your questions. We will also add these changes to the revision of our manuscript. We hope that we address your concerns regarding our paper, and eventually convince you to raise your score.\n\n***\n\n**Q1: Can you add a discussion section discussing how this theory can inform practical model tuning decisions for users?** \n\nThank you for your suggestion. There are two important practical implications for tuning models from our theories.\n\n1. From Section 3, we observe that when using the softmax gating, the convergence rates of prompt parameter estimation are faster than those when using the input-free gating in [2]. Therefore, our theories encourage the use of softmax gating over input-free gating when tuning contaminated-MoE-based models.\n\n2. From Section 3.2, we observe that when the prompt model acquires overlapping knowledge with the pre-trained model (the distinguishability condition is violated), the convergence rates of prompt parameter estimation are slow down. Thus, our theories advocate using prompt models with different expertise from the pre-trained model.\n\n***\n**Q2: Can the analysis be extended to hard-routing? Similar to [1]**\n\nThanks for your question. We believe that our analysis can be extended to the setting where only one expert is activated per input as in [1]. For that purpose, we need to employ techniques for analyzing sparsely gated MoE in [3]. In particular, we first partition the input space into two regions corresponding to two experts in the contaminated MoE:\n\n(i) Region 1 contains $x\\in\\mathcal{X}$ such that $\\exp(\\beta^{\\top}x+\\tau)>1$, that is, the weight of the prompt model is larger than that of the pre-trained model;\n\n(ii) Region 2 contains $x\\in\\mathcal{X}$ such that $\\exp(\\beta^{\\top}x+\\tau)\\leq 1$, that is, the weight of prompt model is smaller than that of the pre-trained model.\n\nThen, the conditional density function associated with the contaminated MoE is given by\n\n- $p_{G}(y|x)=f(y|h(x,\\eta),\\nu)$, if the input $x$ belongs to Region 1;\n\n- $p_{G}(y|x)=f_0(y|h_0(x,\\eta_0),\\nu_0)$, if the input $x$ belongs to Region 2.\n\nThen, we can reuse our proof techniques to establish the convergence rates of prompt and gating parameter estimation. However, since this direction lies beyond the scope of our work, we leave it for future development, though we will  mention it in the main text as an interesting extension.\n\n***\n**Q3: All experiments are synthetic and low-dimensional without practical benchmarks and real NLP/LLM tasks. Could you apply your experiments to a more realistic prompt-tuning task?**\n\nThanks for your suggestion. We have added a few-shot fine-tuning experiment on DINOv2 ViT-S embeddings of CIFAR-10 under a domain shift from vehicles to animals. In this setting, the softmax contaminated MoE consistently outperforms both the input-free contaminated MoE and standard fine-tuning. Please see the additional experiments for details.\n\n**Additional Experiments**  \nWe study adaptation of a pretrained classifier to a distributionally shifted dataset.  \nLet $h_{0}$ be a model trained on a “source” subset biased toward *vehicles*.  \nWe then adapt the model on a small “target” subset biased toward *animals* and evaluate on the full, balanced CIFAR-10 test split.\n\n---\n\n### Setup  \nWe extract 384-dimensional embeddings from CIFAR-10 images using a DINOv2 “small” ViT.  \nThe ten classes are partitioned into two groups:  \n* **Animals** — bird, cat, deer, dog, frog, horse (6 classes)  \n* **Vehicles** — airplane, automobile, ship, truck (4 classes)\n\n---\n\n### Data construction (per-class counts)  \n\n* **Distinguishable setting**: the prompt model does not share knowledge with the pre-trained model  \n  * Pre-train: 524 samples exclusively from vehicle classes (131 per vehicle class)  \n  * Fine-tune: 56 samples exclusively from animal classes (≈ 9–10 per animal class)  \n\n* **Non-distinguishable setting**: the prompt model partly shares knowledge with the pre-trained model  \n  * Pre-train: 512 vehicle (128 per class) + 12 animal (2 per class) = 524  \n  * Fine-tune: 48 animal (8 per class) + 8 vehicle (2 per class) = 56  \n\nThe held-out test set is the full, balanced 10 000-image CIFAR-10 split.\n\n---\n\n### Implementation details  \n\n| Setting      | Split       | Animals / class | Vehicles / class | Total |\n|-------------|-------------|-----------------|------------------|-------|\n| Distinguishable  | Pre-train   | 0               | 131              | 524   |\n|             | Fine-tune   | 9–10            | 0                | 56    |\n| Non-distinguishable | Pre-train   | 2               | 128              | 524   |\n|             | Fine-tune   | 8               | 2                | 56    |\n\n---\n\n### Training  \n\n* **Pre-training** — train a linear classifier $h_{0}$ on the pre-training set with cross-entropy, Adam (lr $10^{-3}$, wd $10^{-4}$, batch 32) for 20 epochs, then freeze its parameters.  \n* **Fine-tuning** — attach a contaminated MoE that mixes $h_{0}$ with a new trainable linear expert $h_{1}$ with gating parameters $(\\beta,\\tau)$  \n\n  $$\n    \\hat{y}(x)=\\bigl(1-\\sigma(\\beta^{\\top}x+\\tau)\\bigr)\\,h_{0}(x)\n               +\\sigma(\\beta^{\\top}x+\\tau)\\,h_{1}(x).\n  $$  \n\n  Only the prompt and gating parameters are updated (10 epochs, same optimizer).\n\n*Optimizer:* Adam (lr $1\\times10^{-3}$, wd $1\\times10^{-4}$, batch 32)  \n*Epochs:* 20 (pre-train), 10 (fine-tune)\n\n---\n\n### Evaluation and baselines  \n\nWe report accuracy on the full test set, as well as on the animal-only and vehicle-only subsets.  \nBaselines: Single frozen pre-trained model, Fine-tuning pre-trained model (without prompt), Input-free contaminated MoE [2], Softmax-contaminated MoE (Ours).\n\n---\n\n### Results  \n\n*Columns in each table are accuracies on the indicated test subset.*\n\n### Table 1a – *Distinguishable* setting\n\n| Model | Animals | Vehicles | Overall |\n|-------|:-------:|:--------:|:-------:|\n| Single frozen pre-trained model             | 0.1 | **94.9** | 38.0 |\n|  Fine-tuning pre-trained model  | 69.4 | 91.1     | 78.1 |\n| Input-free contaminated MoE      | 52.6 | 87.9     | 69.1 |\n| **Softmax-contaminated MoE** | **81.7** | 94.1 | **86.7** |\n\n### Table 1b – *Non-distinguishable* setting\n\n| Model | Animals | Vehicles | Overall |\n|-------|:-------:|:--------:|:-------:|\n| Single frozen pre-trained model             | 20.7 | **95.4** | 50.5 |\n|  Fine-tuning pre-trained model   | 71.7 | 91.5     | 79.6 |\n| Input-free contaminated MoE      | 50.3 | 87.4     | 68.3 |\n| **Softmax-contaminated MoE** | **72.3** | 94.7 | **82.4** |\n\n***\n\n### Conclusion\n\nThe above experiments empirically justify our two main theoretical findings:\n\n1. **Softmax-contaminated MoE outperforms input-free contaminated MoE [2]:** Across both distinguishable and non-distinguishable settings, we observe that the Softmax-contaminated MoE achieves the highest and most balanced performance, improving animal accuracy while maintaining high vehicle accuracy. On the other hand, input-free contaminated MoE underperforms, and fine-tuning pre-trained model sits in between.\n\n2. **Performance of softmax-contaminated MoE under distinguishable settings is higher than that under non-distinguishable setting:** It can be seen that the performance of softmax-contaminated MoE under the distinguishable setting presented in Table 1a is generally higher than that under the non-distinguishable setting presented in Table 1b.\n\n***\n**References**\n\n[1]  Kratsios et al. Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts, TMLR.\n\n[2] Yan et al. \"Understanding expert structures on minimax parameter estimation in contaminated mixture of experts.\" In AISTATS, 2025.\n\n[3] Nguyen et al. Statistical perspective of top-k sparse softmax gating mixture of experts. In ICLR, 2024."}, {"review_id": "PDq9XJwIDr", "reviewer": "Reviewer_qXy5", "summary": "The paper analyzes the softmax-contaminated MOE, in which a fixed pre-trained model and a trainable prompt are combined for fine-tuning. The paper investigates the maximum likelihood estimator convergence for gating and prompt parameters, introducing a distinguishability condition to measure model differences.", "questions": "1) I did not understand why in page 6 the identifiability condition requires twice-differentiable expert functions. How does it apply to non-smooth functions like ReLU?\n\n2) What is the network design for the experiments and the proposed model.\n\n3) In Fig. 1, what are the data generation process and sample sizes? It is better to show the performances on a public dataset as well.", "limitations": "In the main paper, limitations are not discussed.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides mathematical proofs and performs evaluations on two case studies.\n\n:Weaknesses:\n\n1- lack of experimental analysis. While mathematical proof was provided, the paper was only evaluated on a synthetic dataset.\n\n2- The presentation of the paper needs improvement. For example, on page 2, the parameters in the table are not defined.\n\n3- in theorem 2 some parameters are not defined, such as $\\langle \\langle \\Delta \\eta, \\Delta \\nu \\rangle \\rangle$", "was_revised": true, "final_justification": "After reading the authors rebuttal and discussions, some of my comments are addressed. While the authors provided some additional experiments on CIFAR10, still my main concern regarding the experiments on the complex dataset is not addressed. By the way, I raised my rate.", "rebuttal": "Dear Reviewer qXy5,\n\nThanks for your constructive and valuable feedback. Below are our responses to your questions. We will also add these changes to the revision of our manuscript. **We hope that we have addressed your concerns regarding our paper and will ultimately convince you to raise your score.**\n\n***\n\n**Q1: Lack of experimental analysis, with sole emphasis on synthetic data.**\n\nThanks for your suggestion. We have added a few-shot fine-tuning experiment on DINOv2 ViT-S embeddings of CIFAR-10 under a domain shift from vehicles to animals. In this setting, the softmax contaminated MoE consistently outperforms both the input-free contaminated MoE and standard fine-tuning. Please see the additional experiments for details.\n\n**Additional Experiments**  \nWe study adaptation of a pretrained classifier to a distributionally shifted dataset.  \nLet $h_{0}$ be a model trained on a “source” subset biased toward *vehicles*.  \nWe then adapt the model on a small “target” subset biased toward *animals* and evaluate on the full, balanced CIFAR-10 test split.\n\n---\n\n### Setup  \nWe extract 384-dimensional embeddings from CIFAR-10 images using a DINOv2 “small” ViT.  \nThe ten classes are partitioned into two groups:  \n* **Animals** — bird, cat, deer, dog, frog, horse (6 classes)  \n* **Vehicles** — airplane, automobile, ship, truck (4 classes)\n\n---\n\n### Data construction (per-class counts)  \n\n* **Distinguishable setting**: the prompt model does not share knowledge with the pre-trained model  \n  * Pre-train: 524 samples exclusively from vehicle classes (131 per vehicle class)  \n  * Fine-tune: 56 samples exclusively from animal classes (≈ 9–10 per animal class)  \n\n* **Non-distinguishable setting**: the prompt model partly shares knowledge with the pre-trained model  \n  * Pre-train: 512 vehicle (128 per class) + 12 animal (2 per class) = 524  \n  * Fine-tune: 48 animal (8 per class) + 8 vehicle (2 per class) = 56  \n\nThe held-out test set is the full, balanced 10 000-image CIFAR-10 split.\n\n---\n\n### Implementation details  \n\n| Setting      | Split       | Animals / class | Vehicles / class | Total |\n|-------------|-------------|-----------------|------------------|-------|\n| Distinguishable  | Pre-train   | 0               | 131              | 524   |\n|             | Fine-tune   | 9–10            | 0                | 56    |\n| Non-distinguishable | Pre-train   | 2               | 128              | 524   |\n|             | Fine-tune   | 8               | 2                | 56    |\n\n---\n\n### Training  \n\n* **Pre-training** — train a linear classifier $h_{0}$ on the pre-training set with cross-entropy, Adam (lr $10^{-3}$, wd $10^{-4}$, batch 32) for 20 epochs, then freeze its parameters.  \n* **Fine-tuning** — attach a contaminated MoE that mixes $h_{0}$ with a new trainable linear expert $h_{1}$ with gating parameters $(\\beta,\\tau)$  \n\n  $$\n    \\hat{y}(x)=\\bigl(1-\\sigma(\\beta^{\\top}x+\\tau)\\bigr)\\,h_{0}(x)\n               +\\sigma(\\beta^{\\top}x+\\tau)\\,h_{1}(x).\n  $$  \n\n  Only the prompt and gating parameters are updated (10 epochs, same optimizer).\n\n*Optimizer:* Adam (lr $1\\times10^{-3}$, wd $1\\times10^{-4}$, batch 32)  \n*Epochs:* 20 (pre-train), 10 (fine-tune)\n\n---\n\n### Evaluation and baselines  \n\nWe report accuracy on the full test set, as well as on the animal-only and vehicle-only subsets.  \nBaselines: Single frozen pre-trained model, Fine-tuning pre-trained model (without prompt), Input-free contaminated MoE [2], Softmax-contaminated MoE (Ours).\n\n---\n\n### Results  \n\n*Columns in each table are accuracies on the indicated test subset.*\n\n### Table 1a – *Distinguishable* setting\n\n| Model | Animals | Vehicles | Overall |\n|-------|:-------:|:--------:|:-------:|\n| Single frozen pre-trained model             | 0.1 | **94.9** | 38.0 |\n|  Fine-tuning pre-trained model  | 69.4 | 91.1     | 78.1 |\n| Input-free contaminated MoE      | 52.6 | 87.9     | 69.1 |\n| **Softmax-contaminated MoE** | **81.7** | 94.1 | **86.7** |\n\n### Table 1b – *Non-distinguishable* setting\n\n| Model | Animals | Vehicles | Overall |\n|-------|:-------:|:--------:|:-------:|\n| Single frozen pre-trained model             | 20.7 | **95.4** | 50.5 |\n|  Fine-tuning pre-trained model   | 71.7 | 91.5     | 79.6 |\n| Input-free contaminated MoE      | 50.3 | 87.4     | 68.3 |\n| **Softmax-contaminated MoE** | **72.3** | 94.7 | **82.4** |\n\n### Conclusion\n\nThe above experiments empirically justify our two main theoretical findings:\n\n1. **Softmax-contaminated MoE outperforms input-free contaminated MoE [2]:** Across both distinguishable and non-distinguishable settings, we observe that the Softmax-contaminated MoE achieves the highest and most balanced performance, improving animal accuracy while maintaining high vehicle accuracy. On the other hand, input-free contaminated MoE underperforms, and fine-tuning pre-trained model sits in between.\n\n2. **Performance of softmax-contaminated MoE under distinguishable settings is higher than that under non-distinguishable setting:** It can be seen that the performance of softmax-contaminated MoE under the distinguishable setting presented in Table 1a is generally higher than that under the non-distinguishable setting presented in Table 1b.\n\n***\n**Q2: The parameters in Table 1 and Theorem 2 are not defined.**\n\nThanks for your feedback. We have defined all the parameters in Table 1 in Eq. (1), Eq. (2), and line 241, respectively, of our initial submission. However, following your suggestion, we will add the definitions of these parameters to the caption of Table 1. \n\nAs for Theorem 2, we assume that the reviewer is referring to the notation $\\bar{\\beta}_n$, $\\bar{\\tau}_n$, $\\bar{\\eta}_n$, $\\bar{\\nu}_n$. These parameters are the components of $\\bar{G}$ given in the infimum operator, that is, $\\bar{G}:=(\\bar{\\beta}_n,\\bar{\\tau}_n,\\bar{\\eta}_n,\\bar{\\nu}_n)$. We will clarify this in the revised manuscript.\n\n***\n**Q3: Why does the strong identifiability condition requires twice-differentiable expert functions? How does it apply to non-smooth functions like ReLU?**\n\nThanks for your questions. It should be noted that a key step in obtaining MLE convergence rates is to decompose the density discrepancy $p_{\\widehat{G_n}}-p_{G_*}$ into a combination of linearly independent terms through an appropriate Taylor expansion. This process involves high-order derivatives of the expert function $h(\\cdot,\\eta)$ w.r.t $\\eta$, which may not be algebraically independent. Therefore, we introduce the strong identifiability condition, which involves the second-order derivatives of the expert function, to ensure the linear independence of terms in the Taylor expansion. \n\nNext, we clarify that the strong identifiability condition requires expert functions which are twice differentiable almost everywhere. Therefore, non-smooth functions like ReLU still can satisfy this condition as long as they meet the conditions of linear independence in lines 216-218.\n\n***\n**Q4: What is the network design for the experiments and the proposed model?**\n\nThanks for your questions. All architectural details are given in Section 4 (Ln 269–278). The frozen expert is $h_{0}(x)=\\tanh(\\eta_{0}^{\\top}x)$, and the prompt expert is $h(x)=\\tanh(\\eta^{\\top}x)$ - both a linear projection followed by $\\tanh$. These two experts are mixed by the softmax–contaminated MoE gate in Eq. (1). We emphasize that we are not proposing a new network architecture; instead, we study the parameter estimation rates of the contaminated softmax MoE. Accordingly, we generate data with exactly this model (with additive label noise) and fit the same class via maximum likelihood, ensuring that the experiments directly validate the convergence rates predicted by our theory.\n\n***\n**Q5: In Fig. 1, what are the data generation process and sample sizes? It is better to show the performances on a public dataset as well.**\n\nThanks for your questions. The synthetic data in Fig. 1 are generated as described in Section 4 (Ln 279–291), we sample inputs $x\\sim\\mathcal N(0,I_d)$, draw labels from the softmax contaminated MoE of Eq. (1) using the ground-truth parameters specified there, and then fit the same model class by maximum likelihood.  \nIn the plot, the x-axis is the training sample size $n$ (log scale), while the y-axis shows the Euclidean distance (log scale) between each learned parameter and its true value; therefore, the slopes illustrate the empirical convergence rates predicted by our theory.  \nTo complement these controlled simulations, we have added a few shot fine-tuning experiments on real data, DINOv2 ViT-S embeddings of CIFAR-10 under a vehicles $\\rightarrow$ animals domain shift, where the softmax contaminated MoE adapter again outperforms static and fine-tuning baselines. Please see the additional experiments for more details.\n\n***\n**Q6: In the main paper, limitations are not discussed.**\n\nThanks for your comment. We have implicitly discussed the limitations of our work in the last paragraph of Section 5. Specifically, we mention that our analysis is restricted to the case of a single prompt and focuses solely on the scenario where prompt models belong to the family of Gaussian distributions. We will further elaborate on these points. In the meanwhile, please let us know if you have further or specific concerns.\n\n***\n**References**\n\n[2] Yan et al. \"Understanding expert structures on minimax parameter estimation in contaminated mixture of experts.\" In AISTATS, 2025."}, {"review_id": "bTK4X64pbv", "reviewer": "Reviewer_2SMz", "summary": "This paper provides the first comprehensive statistical analysis of a mixture-of-experts model used in parameter-efficient fine-tuning, where a fixed pre-trained expert and a trainable Gaussian “prompt” expert are combined via a softmax linear gating network. The authors introduce a novel analytic notion of distinguishability between the frozen and prompt experts that guarantees identifiability, and show that, under this condition, the maximum likelihood estimator for both gating and prompt parameters converges at the near-parametric rate $O(n^{-\\frac{1}{2}})$, matching minimax lower bounds. Conversely, when distinguishability fails—so that the prompt overlaps with the pre-trained expert—they prove that estimation rates slow to $O(n^{-\\frac{1}{4}})$ or $O(n^{-\\frac{3}{8}})$, depending on how the prompt parameters approach those of the pre-trained model, again with minimax-optimal lower bounds. This work clarifies how overlap between experts degrades statistical efficiency and lays the groundwork for extensions to multiple prompts and broader model families.", "questions": "1. What is the most interesting insight compared to [2], [3]?\n2. Can give more empirical results demonstrate the potential of Contaminated MoE that performs better than fully fine-tuning?\n\n\n\n**References**\n\n[1] Zadouri, T., Ust¨un, A., Ahmadian, A., Ermis, B., Lo-catelli, A., and Hooker, S. (2024). Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations.\n\n[2] Nguyen, Huy, Nhat Ho, and Alessandro Rinaldo. \"Sigmoid gating is more sample efficient than softmax gating in mixture of experts.\" arXiv preprint arXiv:2405.13997 (2024).\n\n[3] Nguyen, Huy, TrungTin Nguyen, and Nhat Ho. \"Demystifying softmax gating function in Gaussian mixture of experts.\" Advances in Neural Information Processing Systems 2023.\n\n[4] Yan, Fanqi, et al. \"Understanding expert structures on minimax parameter estimation in contaminated mixture of experts.\" arXiv preprint arXiv:2410.12258 (2024).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n+ This paper proposes \"distinguishability\", which means pre-trained model $f_0$ and prompt model $f$ (and the way $f$ shifts when you tweak its parameters) span a sufficiently rich family so they never lie on the same low-dimensional subspace. When distinguishability holds, the softmax gating can cleanly attribute each sample to one expert or the other, yielding near-parametric $O(n^{-\\frac{1}{2}})$ estimation. If they did lie on the same subspace, you could hide a change in gating or prompt parameters by making the other expert “absorb” the effect—leading to non-identifiable models and slower estimation rates. This result is insightful and and align with intuition.\n\n+ It delivers the first rigorous minimax-optimal convergence rates for both gating (softmax) and prompt parameters under softmax-contaminated MoE settings—covering both distinguishable and non-distinguishable regimes—thus filling the convergence analysis gap of Gaussian MoE.\n\n\n## Weaknesses\n- Assume that \"Contaminated MoE\" only refers to a specific setting where frozen experts and a \"conditioned\" prompt model, excluding efficient tuning such as LoRA, or EP-MoE [1] (Please correct me if I am wrong). Contaminated MoE seems to be a rare setting in application such that few evidences indicate this pipeline has better performance than general supervised fine-tuning. In addition, mean expert function $f_0$ weakens the mutual effect than discussing multiple experts.\n\n- Similar insights seem to be found in [2], [3]. Under the Gaussian distribution assumption, I think the insight is somehow lack of the novelty. Please state the differences (mostly on insights) compared to the similar work. For example, [4] also proved $O(n^{-\\frac{1}{2}})$ convergence under linear and non-linear setting. The novelty of the results seems unclear.", "was_revised": true, "final_justification": "After carefully reviewing the authors' detailed responses to the concerns raised, I am satisfied with the clarifications provided, which address the key points of confusion.\n \nThe authors have clearly distinguished \"Contaminated MoE\" from related parameter-efficient tuning methods like LoRA and EP-MoE [1], emphasizing its unique structure as an input-dependent weighted sum of a frozen pre-trained model and a trainable prompt model. This distinction helps contextualize the work within the broader landscape of efficient fine-tuning, resolving the initial ambiguity about the scope of their framework.\n \nRegarding the comparison to prior work [2], [3], and [4], the authors have effectively highlighted the novelty of their contributions: the focus on a practically motivated contaminated setting (frozen + trainable experts) absent in standard MoE analyses [2], [3]; the consideration of sample-size-dependent ground-truth parameters leading to uniform convergence rates (a more realistic scenario than fixed parameters); and the analysis of input-dependent softmax gating (vs. input-free gating in [4]), with insights into how gating choice impacts estimation efficiency. These points clarify the paper’s theoretical insights and fill a gap in the literature on Gaussian MoE convergence under parameter-efficient fine-tuning regimes.\n\nHowever, as I am not the expert on this field. I am still concern it's contribution impact on the problem setting. Therefore I maintain my original rating.", "rebuttal": "Dear Reviewer 2SMz,\n\nThanks for your insightful reviews, and giving **excellent (4) grade** to the quality and **good (3) grade** to the clarity and significance of our paper. We are encouraged by the endorsement that\n\n*(i) Our result is **insightful and and align with intuition**;*\n\n*(ii) Our paper delivers **the first** rigorous minimax-optimal convergence rates for both gating (softmax) and prompt parameters under softmax-contaminated MoE;*\n\n(iii) *filling the convergence analysis gap of Gaussian MoE.*\n\nThank you for your thoughtful review, which have helped us improved our paper substantially. Below are our responses to your questions. We will also add these changes to the revision of our manuscript. We hope that we have addressed your concerns regarding our paper.\n\n***\n\n**Q1: Assume that \"Contaminated MoE\" only refers to a specific setting where frozen experts and a \"conditioned\" prompt model, excluding efficient tuning such as LoRA, or EP-MoE [1]**\n\nThanks for your question. We confirm that the concept of contaminated MoE is different from that of LoRA and parameter-efficient MoE in [1]. In particular, \n\n- Contaminated MoE is an input-dependent weighted sum of a frozen pre-trained model and a trainable prompt model without rank control;\n\n- LoRA is a normal sum of a frozen pre-trained weight and a product of two learnable low-rank matrices;\n\n- Parameter-efficient MoE is also an input-dependent gating MoE but all the experts are trainable formulated as adapters. \n\nWe will add a remark to ensure that this point is clear.\n\n***\n\n**Q2: Contaminated MoE seems to be a rare setting in application such that few evidences indicate this pipeline has better performance than general supervised fine-tuning. Can you give more empirical results to demonstrate the potential of Contaminated MoE that performs better than fully fine-tuning?**\n\nThanks for your question. Let us clarify that we do not aim to compare the performance of contaminated MoE and full fine-tuning method in this paper. Instead, we focus on analyzing the prompt convergence in the contaminated MoE, a model inspired from a parameter-efficient fine-tuning method known as prefix-tuning, where a frozen pre-trained model is combined with a trainable prompt model to learn downstream tasks. We agree with the reviewer that the comparison between these two fine-tuning methods is worth exploring. However, since this direction lies beyond the scope of our work, we leave it for future development. We will clarify this important point.\n\n***\n\n**Q3: The novelty of our paper compared to previous works [2, 3, 4].**\n\nThank you for your comment. We would like to clarify the key differences in terms of insights and contributions between our work and previous research, particularly [2], [3], and [4]:\n\n**Compared with [2, 3]:** \nThere are four main differences between [2, 3] and our work.\n\n*1. Concepts:* In our work, we consider contaminated MoE, which is a mixture of a frozen pre-trained model and a trainable prompt model motivated from parameter-efficient fine-tuning method, while models considered in [2, 3] are a standard mixture of all trainable sub-models, used for scaling up the model capacity in large-scale AI models.\n\n*2. More practical settings:* We allow ground-truth parameters $G_*$ to change with the sample size $n$, which is closer to practical settings than the assumptions in [2, 3] where $G_*$ does not change with $n$. \n\n*3. Uniform convergence rates:* Since ground-truth parameters vary with the sample size, the convergence rates of parameter estimations in our work are uniform rather than point-wise as in [2, 3]. Furthermore, these rates are able to capture the interaction between the convergences of parameter estimations, while those in [2, 3] cannot.\n\n*4. Minimax lower bounds:* Finally, we determine minimax lower bounds under both distinguishable and non-distinguishable settings. Based on these lower bounds, we can claim that our derived convergence rates are optimal. However, no minimax lower bounds are provided in [2, 3].\n\n**Compared with [4]:** \nThere are two main differences between [4] and our work.\n\n*1. More realistic and challenging setting:* Our work considers contaminated MoE with softmax gating (input dependent), whereas the gating function in [4] is input-free. This input-dependent formulation is not only more realistic and applicable in practice—capturing the natural dependence of the gating mechanism on the input—but also presents additional theoretical challenges. As a result, our analysis is both more rigorous and better aligned with practical modeling scenarios.\n\n*2. Insights:* From Section 3, we observe that when using the softmax gating, the convergence rates of prompt parameter estimation are faster than those when using the input-free gating in [4] (see, e.g., lines 175-183). Therefore, our theories encourage the use of softmax gating over input-free gating when tuning contaminated-MoE-based models.\n\nFrom Section 3.2, we observe that when the prompt model acquires overlapping knowledge with the pre-trained model (the distinguishability condition is violated), the convergence rates of prompt parameter estimation are slow down. Thus, our theories advocate using prompt models with different expertise from the pre-trained model. Meanwhile, Yan et al. [4] focused more on the linearity and non-linearity of expert functions. \n\n***\n\n**References**\n\n[1] Zadouri, T., Ust¨un, A., Ahmadian, A., Ermis, B., Lo-catelli, A., and Hooker, S. (2024). Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations.\n\n[2] Nguyen, Huy, Nhat Ho, and Alessandro Rinaldo. \"Sigmoid gating is more sample efficient than softmax gating in mixture of experts.\" arXiv preprint arXiv:2405.13997 (2024).\n\n[3] Nguyen, Huy, TrungTin Nguyen, and Nhat Ho. \"Demystifying softmax gating function in Gaussian mixture of experts.\" Advances in Neural Information Processing Systems 2023.\n\n[4] Yan, Fanqi, et al. \"Understanding expert structures on minimax parameter estimation in contaminated mixture of experts.\" arXiv preprint arXiv:2410.12258 (2024)."}, {"review_id": "AzLT7q21qO", "reviewer": "Reviewer_at8P", "summary": "This paper provides a theoretical understanding of contaminated mixture-of-expert (MoE) in the presence of input-dependent gating. The analysis implies that input-dependent gating helps finding the optimal solution as compared to the input-free gating. The analysis also implies that having diversity between experts (distinguishability) is beneficial in terms of finding the optimal solution.", "questions": "- Clear motivation of the considered setting -- one frozen, one non-frozen expert; optimal model also assumes the one frozen, one non-frozen expert\n- What is the significance of finding the optimal parameters given that optimal density is already found?\n- Detailed comparison with input-free gating (convergence rate under non-distinguishability, convergence rate in terms of density) and reference [26] (why one cannot restrict the model space of one expert in [26] to get the similar, but more general, result of this work)", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "_Strengths_\n\n- In-depth theoretical analysis of contaminated MoE\n- Near-tightness of the bound (nearly matching the minimax lower bound)\n\n_Weaknesses_\n\n- Single expert is assumed\n- Significance of parameter estimation is not clear\n\n_Detailed Comments_\n\n1) _Motivating contaminated MoE from prefix tuning is not so clear_\n\nIn prefix tuning, only single decision is made unlike MoE where multiple decisions are separately made, and then aggregated. I think in order for prefix tuning to be related to MoE, it should be something like: one decision made by without prefix, the other made with prefix, and then the final decision is made by combining these two. But since existing prefix tuning is not operating like in this way (at least to my understanding), the current motivation seems to be misleading. \n\n2) _Having one frozen expert and one trainable expert, isn't it a too special case of MoE?_\n\nAs far as I am aware of, MoE generally assumes multiple experts with which it achieves higher performance that goes beyond the capacity of individual experts. In this regard, (i) having two experts and (ii) assuming the optimal model (1) is also obtained from these two experts seems to be a too narrow case to call this work as MoE. Also, given that mostly softmax treats multiple inputs and it is more common to call it sigmoid if there is only two inputs, I'm also not sure whether calling this work as softmax-contaminated MoE is the best way one can do. I suggest to provide a clearer, practical, justification of (i) and (ii), followed by theoretical difficulties if one does not assume (i) and (ii). \n\n3) _Proposition 3 already shows that the MoE can find the optimal density in a sublinear manner. Why one needs a further results on the parameter itself?_\n\nProposition 3 shows that the density of MoE converges to the optimal one with sublinear rate. Given that what we need in the end is the density that works fairly well with the optimal one, it is not clear to me why we need to go further to find the optimal parameters. In other words, why knowing the optimal parameters is more informative than knowing the optimal density? Perhaps this can be related to some sort of explainability of the models, but it is not very clear in the current description. \n\n4) _Can we have minimax lower bound for the density result?_\n\n Theorem 1 and 2 show the convergence rate of the parameters while Proposition 3 is about the rate for the density. Theorem 1 can be treated as near-optimal as it nearly matches with minimax lower bound, Theorem 2. Given that Theorem 1 is built upon Proposition 3, I guess Proposition 3 is also somewhat near-optimal in the sense of matching the minimax lower bound for the density. But is it really true? Can this be proved?\n\n5) _Under non-distinguishability, input-dependent gating would be still better than input-free gating?_\n\nThe authors showed that, under distinguishability setting, input-dependent gating helps finding the optimal parameters compared to input-free gating. The authors also showed that, under non-distinguishability, convergence rate of input-dependent gating becomes slower. In such a case, would it be even worse than the input-free gating? Also, how does the convergence rate in terms of density (Proposition 3) compares with that of input-free gating? I guess in terms of density, both input-free and input-dependent gating would admit the same convergence rate (maybe I'm wrong!), and in light of my comment 3), unless there is clear significance of estimating the parameters per se, the advantage of input-dependent gating seems not so clear. \n\n6) _Comparison with [26] is not so clear_\n\nReference [26] also considered softmax, input-dependent, gating with multiple, more than 2, experts. Of course, they did not consider the frozen expert hence it is not directly applicable to the setting of interest, but it would be nice to clarify the connection of this work with [26]. For instance, why one cannot consider the special case of [26] where one expert has its model space with only single element hence essentially serves the role as the frozen expert. \n\n_Minor Comments_\n\n- Using \\$h()$ for both mean expert function and Hellinger distance is confusing. \n- In the proof of Lemma 3 (p 37), for the first inequality of line 1105, why a conservative \\$\\delta/4$ is considered given that considering \\$\\delta$ seems to have no problem in the upcoming steps? \n- In Sec. B. 3, \\$\\mu$ is not clearly defined, e.g., in (57) and line 1116, \\$\\mu$ appears which seems to be not defined before. \n- In proof of Theorem 2, line 873 -- 875 is not clear how \\$\\tau_n$ lies in a compact set makes the desired term goes to 0.", "was_revised": true, "final_justification": "My main concern of this work was (i) why one needs to care parameter convergence rate (which is the main focus of this work) given that we already have density (i.e., model output) convergence rate that achieves the same asymptotic rate; (ii) practical motivating example of parameter efficient fine-tuning is not well aligned with the setting of this work (1), since parameter efficient fine-tuning (e.g., prefix-tuning) does not aggregate the final likelihoods but rather aggregates intermediate, non-probabilistic, decisions, which cannot be covered by (1). And during the rebuttal/discussion periods, the authors clarified the first point for its usefulness in understanding the expert specialization, i.e., how fast an expert specializes in some specific task, by pointing out the existing literature on MoE. However, the second point has not been perfectly addressed since the authors agreed that parameter efficient fine-tuning indeed cannot be expressed in the form of (1) -- aggregation of likelihoods -- and requires additional equation that involves conditional expectation -- aggregation of the non-probabilistic outputs. \n\nAs a result, for its good theoretical contribution in the MoE community where the results of this work can also be well combined with existing literature, e.g., on expert specialization, I recommend this work's score as 5.", "rebuttal": "Dear Reviewer at8P,\n\nThanks for the wealth of your comments, and giving **good (3) grade** to the quality, clarity, significance and originality of our paper. Below are our responses to your questions. We will also add these changes to the revision of our manuscript. We hope that we have addressed your concerns regarding our paper.\n\n***\n\n**Q1: Motivating contaminated MoE from prefix tuning is not so clear.**\n\nThanks for your comment. We clarify that contaminated MoE is a variant of MoE which is inspired by prefix tuning method but does not necessarily share the same concept as prefix tuning. In particular, in prefix tuning [3], the original model parameters are frozen, and then they attend to trainable prefix to learn downstream tasks parameter-efficiently. Inspired from this concept, we consider contaminated MoE, a mixture of a frozen pre-trained model and a trainable prompt expert to learn downstream tasks without having to tune the entire pre-trained model. We will add a clarifying comment.\n\n***\n\n**Q2: Having one frozen expert and one trainable expert, isn't it a too special case of MoE?**\n\n\nThank you for raising this point. We view the “one frozen $+$ one trainable” configuration as a contaminated mixture of experts: an instantiation that preserves the strong pretrained behavior as an anchor while also enabling targeted adaptation via a small trainable expert. This is not a degenerate case but a deliberate design choice that trades capacity for stability, sample‑efficiency, and interpretability. The formulation extends naturally to the case of more than two experts, where we keep some frozen anchor experts alongside trainable ones; the theoretical analysis proceeds in the same direction. Finally, the pattern aligns with the widely used “frozen backbone + lightweight adapter” practice—our contribution is to cast it as an MoE, prove guarantees, and show that it is a strong and extensible baseline rather than a restrictive special case.\n\n***\n\n**Q3:  Proposition 3 already shows that the MoE can find the optimal density in a sublinear manner. Why one needs a further results on the parameter itself?**\n\nThank you for your valuable comment and suggestions. We believe that parameter estimation provides deeper insight into the model by enabling users to identify which components of the parameters contribute to observed differences. This enhances the explainability of the model. Furthermore, in the context of parametric models, knowing the parameters allows for straightforward data generation, whereas relying solely on the density function is often too general and may require additional information. Therefore, parameter estimation not only improves explainability but also offers practical advantages in terms of data generation and model understanding.\n\n***\n\n**Q4: Can we have minimax lower bound for the density result?**\n\nThank you for your suggestions. We believe that a similar result can be established in this setting. One possible approach is to employ Le Cam’s two-point method or its generalization, as presented in Lemma C.1 of [8]. This technique is also utilized in Lemma 2 of our article. The core idea is to construct two sequences of parameters whose distance is sufficiently small to yield a meaningful lower bound. While this construction is relatively straightforward in the context of parameter estimation, it becomes considerably more challenging when extended to density functions, due to the complexity of controlling the Hellinger distance between distributions. For this reason, we defer the derivation of the minimax lower bound for density estimation to future work. \n\n\n***\n\n**Q5: Under non-distinguishability, input-dependent gating would be still better than input-free gating?**\n\nThank you for your comment. We would like to reaffirm the advantage of the input-dependent setting over the input-free setting in terms of the asymptotic behavior of parameter estimation. Specifically, under the assumption that $\\tau$ belongs to a compact set such that $\\exp(\\tau) \\geq c > 0$, Theorem 3 and Table 1 in our paper (under the non-distinguishable setting) indicate that the convergence rate for the prompt parameters estimation $\\eta^\\ast$ and $\\nu^\\ast$ does not depend on the gating parameters. In contrast, as shown in Table 1 and Theorem 8 in [26], the convergence rate in the input-free setting depends on $\\lambda^\\ast$, in particular, as $\\lambda^\\ast \\to 0$, the prompt component vanishes, which significantly slows the convergence rate of parameter estimation.\n\nAlthough the convergence rate for density estimation remains the same in both settings, the input-dependent formulation alleviates the slow convergence issue caused by the vanishing prompt phenomenon. In this sense, the use of a sigmoid gating function helps mitigate the adverse effects of prompt vanishing, which could otherwise hinder the reliability and accuracy of parameter estimation. \n\n***\n\n**Q6: Comparison with [26] is not so clear.**\n\nThanks for your feedback. There are three main differences between [26] and our work.\n\n*1. More general settings:* We allow ground-truth parameters $G_*$ to change with the sample size $n$, which is closer to practical settings than the assumptions in [26] where $G_*$ does not change with $n$. Furthermore, we consider a general expert function $h$, while the expert function considered in [26] is restricted to be of linear form.\n\n*2. Uniform convergence rates:* Since ground-truth parameters vary with the sample size, the convergence rates of parameter estimations in our work are uniform rather than point-wise as in [26]. Additionally, these rates are able to capture the interaction between the convergences of parameter estimations.\n\n*3. Minimax lower bounds:* Finally, we determine minimax lower bounds under both distinguishable and non-distinguishable settings. Based on these lower bounds, we can claim that our derived convergence rates are optimal. However, no minimax lower bounds are provided in [26].\n\n***\n\n**Q7: Detailed comparison with input-free gating (convergence rate under non-distinguishability, convergence rate in terms of density) and reference [26] (why one cannot restrict the model space of one expert in [26] to get the similar, but more general, result of this work).**\n\nThank you for your comment. We would like to clarify the distinctions between this paper and [26], particularly with regard to the role of the softmax gating function.\n\n1. In terms of technique:  The current paper requires a more delicate analysis to handle the differences between two density functions (e.g., as seen in lines 775 and 991), in contrast to the relatively simpler treatment in the proof of Lemma 2 in [26]. This increased complexity arises from the interaction induced by the input-dependent softmax gating.\n\n2. In terms of insight:  As discussed in our response to Q5, the softmax gating function is not only more applicable in practice—reflecting realistic dependence on input—but also plays a crucial role in addressing the issue of prompt vanishing. This property improves the stability and accuracy of parameter estimation, representing an important theoretical and practical advantage over the setting considered in [26].\n\n***\n\n**Minor Suggestions:**\nThank you very much for your thorough revision.\n\n1.  Regarding the notation for the expert mean function and the Hellinger distance, we have already updated the notation for the Hellinger distance as $d_H$ to improve the clarity and consistency of our manuscript.\n2.  As for the use of $\\delta/4$ (line 1105), this choice is made purely for technical convenience and does not affect the generality of the results.\n3. The symbol $\\mu$ refers to the Lebesgue measure, and we have now explicitly clarified this in the manuscript.\n4. Concerning $\\tau_n$ (lines 873–875), since $\\tau_n$ lies within a compact set, $\\exp(\\tau_n)$ is bounded above and below by positive constants. As a result, $d_1(S_{1,n},S_{2,n}) = \\mathcal{O}(\\|(\\beta_1,\\eta_1,\\nu_1) - (\\beta_2,\\eta_2,\\nu_2)\\|)$ , which ensures control over the remainder term and the coefficient of the first-order term in the Taylor expansion above.\n\n***\n\n**References**\n\n[3] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.\n \n[8] S. Gadat, J. Kahn, C. Marteau, and C. Maugis-Rabusseau. Parameter recovery in two-component\ncontamination mixtures: The lˆ2 strategy. 2020. \n\n[26] H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In NeurIPS, 2023."}, {"review_id": "ZfUNKt0mpF", "reviewer": "Reviewer_kSC9", "summary": "This paper provides a theoretical study of a softmax-contaminated mixture of experts model, which corresponds to a parameter-efficient fine-tuning scenario where a large pre-trained expert is kept fixed and a new prompt expert is learned alongside a softmax gating network. The authors analyze the convergence rates of the maximum likelihood estimator (MLE) for the gating parameters and the prompt’s parameters.\nit introduces distinguishability condition that formalizes when the prompt’s learned knowledge does not excessively overlap with the pre-trained model. when the prompt is non-distinguishable (i.e. it effectively learns the same function as the pre-trained model), the authors show the estimation problem becomes much harder.", "questions": "The definition of distinguishability (Definition 1) is quite involved. Could the authors provide more intuition or a simpler characterization for this condition? \n\nhow do these findings manifest in real fine-tuning tasks? For instance, if one fine-tunes a large language model with a small prompt on a new task that is actually already solvable by the base model, does one observe slower convergence or higher data requirements (as the theory suggests)? The current experiments are synthetic for control, but have the authors thought about testing their ideas on a toy real-world scenario (e.g., fine-tuning a small neural network with a fixed part vs. a learned part)?", "limitations": "Yes.  (1) the analysis is restricted to a single prompt expert, and (2) the prompt model family is limited (in this paper,", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThis work provides the first rigorous characterization of parameter estimation rates in a softmax-gated fine-tuned mixture model, highlighting the critical role of the prompt’s distinguishability from the base model in achieving optimal rates.\n\nThe authors derive minimax lower bounds that match the MLE’s rates under both the distinguishable and non-distinguishable regimes\n\nTo the best of my knowledge, this is the first analysis of a softmax-gated mixture of experts with one frozen and one trainable expert (which aptly models modern prompt-based fine tuning). The results of this paper are quite significant for the theory of mixture-of-experts and for understanding fine-tuning with large models. \n\nIt extends prior work by handling a more realistic model (softmax gating)\n\nThe paper is well-structured trying to explain heavy concepts clearly. \n\nWeaknesses:\n\nThe theory’s assumptions, such as strong identifiability and the need to restrict the parameter space in the non-identifiable regime, may limit applicability to certain architectures or require careful interpretation.\n\nThe experimental validation, while appreciated, is limited to synthetic data \n\nMinor:\n38 through-> thorough\n118 challening", "was_revised": true, "final_justification": "The answer is satisfactory. I keep my score.", "rebuttal": "Dear Reviewer kSC9,\n\nThanks for your constructive and valuable feedback, and giving **excellent (4) grade** to the significance and originality and **good (3) grade** to the quality and clarity of our paper. We are encouraged by the endorsement that\n\n*(i) Our work provides **the first** rigorous characterization of parameter estimation rates in a softmax-gated;*\n\n*(ii) The results of this paper are quite **significant** for the theory of mixture-of-experts and for understanding fine-tuning with large models.*\n\n*(iii) Our work considers a **more realistic** model than previous works;*\n\n*(iv) The paper is **well-structured** trying to **explain heavy concepts clearly**.*\n\nBelow are our responses to your questions. We will also add these changes to the revision of our manuscript. We hope that we have addressed your concerns regarding our paper.\n\n***\n\n**Q1: The theory’s assumptions, such as strong identifiability and the need to restrict the parameter space in the non-identifiable regime, may limit applicability to certain architectures or require careful interpretation.**\n\nThank you for your comment. As we noted in our article, any pre-trained model that does not belong to the Gaussian density family is distinguishable from the prompt model (Proposition 1). Moreover, many widely used functions—such as GELU, sigmoid, and tanh—satisfy the strong identifiability assumption (see Examples, line 233-238). Therefore, the assumptions made in our analysis do not significantly restrict its applicability in practice.\n\n***\n**Q2: Could the authors provide more intuition or a simpler characterization for the distinguishability condition?**\n\nThank you for your question. Let us provide an intuitive explanation of the concept of distinguishability. This notion captures the potential interaction between the pre-trained model and the prompt model, particularly in terms of their first-order derivatives. When a pair of pre-trained and prompt models satisfies the distinguishability condition, it ensures that such interactions are avoided. This separation is beneficial for parameter estimation, as it prevents interference between the components and leads to more accurate inference.\n\n***\n**Q3: Experimental validation is limited to synthetic data.**\n\nThanks for your comment. We have added a few-shot fine-tuning experiment on DINOv2 ViT-S embeddings of CIFAR-10 under a domain shift from vehicles to animals. In this setting, the softmax contaminated MoE consistently outperforms both the input-free contaminated MoE and standard fine-tuning. Please see the additional experiments for details.\n\n**Additional Experiments**  \nWe study adaptation of a pretrained classifier to a distributionally shifted dataset.  \nLet $h_{0}$ be a model trained on a “source” subset biased toward *vehicles*.  \nWe then adapt the model on a small “target” subset biased toward *animals* and evaluate on the full, balanced CIFAR-10 test split.\n\n---\n\n### Setup  \nWe extract 384-dimensional embeddings from CIFAR-10 images using a DINOv2 “small” ViT.  \nThe ten classes are partitioned into two groups:  \n* **Animals** — bird, cat, deer, dog, frog, horse (6 classes)  \n* **Vehicles** — airplane, automobile, ship, truck (4 classes)\n\n---\n\n### Data construction (per-class counts)  \n\n* **Distinguishable setting**: the prompt model does not share knowledge with the pre-trained model  \n  * Pre-train: 524 samples exclusively from vehicle classes (131 per vehicle class)  \n  * Fine-tune: 56 samples exclusively from animal classes (≈ 9–10 per animal class)  \n\n* **Non-distinguishable setting**: the prompt model partly shares knowledge with the pre-trained model  \n  * Pre-train: 512 vehicle (128 per class) + 12 animal (2 per class) = 524  \n  * Fine-tune: 48 animal (8 per class) + 8 vehicle (2 per class) = 56  \n\nThe held-out test set is the full, balanced 10 000-image CIFAR-10 split.\n\n---\n\n### Implementation details  \n\n| Setting      | Split       | Animals / class | Vehicles / class | Total |\n|-------------|-------------|-----------------|------------------|-------|\n| Distinguishable  | Pre-train   | 0               | 131              | 524   |\n|             | Fine-tune   | 9–10            | 0                | 56    |\n| Non-distinguishable | Pre-train   | 2               | 128              | 524   |\n|             | Fine-tune   | 8               | 2                | 56    |\n\n---\n\n### Training  \n\n* **Pre-training** — train a linear classifier $h_{0}$ on the pre-training set with cross-entropy, Adam (lr $10^{-3}$, wd $10^{-4}$, batch 32) for 20 epochs, then freeze its parameters.  \n* **Fine-tuning** — attach a contaminated MoE that mixes $h_{0}$ with a new trainable linear expert $h_{1}$ with gating parameters $(\\beta,\\tau)$  \n\n  $$\n    \\hat{y}(x)=\\bigl(1-\\sigma(\\beta^{\\top}x+\\tau)\\bigr)\\,h_{0}(x)\n               +\\sigma(\\beta^{\\top}x+\\tau)\\,h_{1}(x).\n  $$  \n\n  Only the prompt and gating parameters are updated (10 epochs, same optimizer).\n\n*Optimizer:* Adam (lr $1\\times10^{-3}$, wd $1\\times10^{-4}$, batch 32)  \n*Epochs:* 20 (pre-train), 10 (fine-tune)\n\n---\n\n### Evaluation and baselines  \n\nWe report accuracy on the full test set, as well as on the animal-only and vehicle-only subsets.  \nBaselines: Single frozen pre-trained model, Fine-tuning pre-trained model (without prompt), Input-free contaminated MoE [2], Softmax-contaminated MoE (Ours).\n\n---\n\n### Results  \n\n*Columns in each table are accuracies on the indicated test subset.*\n\n### Table 1a – *Distinguishable* setting\n\n| Model | Animals | Vehicles | Overall |\n|-------|:-------:|:--------:|:-------:|\n| Single frozen pre-trained model             | 0.1 | **94.9** | 38.0 |\n|  Fine-tuning pre-trained model  | 69.4 | 91.1     | 78.1 |\n| Input-free contaminated MoE      | 52.6 | 87.9     | 69.1 |\n| **Softmax-contaminated MoE** | **81.7** | 94.1 | **86.7** |\n\n### Table 1b – *Non-distinguishable* setting\n\n| Model | Animals | Vehicles | Overall |\n|-------|:-------:|:--------:|:-------:|\n| Single frozen pre-trained model             | 20.7 | **95.4** | 50.5 |\n|  Fine-tuning pre-trained model   | 71.7 | 91.5     | 79.6 |\n| Input-free contaminated MoE      | 50.3 | 87.4     | 68.3 |\n| **Softmax-contaminated MoE** | **72.3** | 94.7 | **82.4** |\n\n### Conclusion\n\nThe above experiments empirically justify our two main theoretical findings:\n\n1. **Softmax-contaminated MoE outperforms input-free contaminated MoE [2]:** Across both distinguishable and non-distinguishable settings, we observe that the Softmax-contaminated MoE achieves the highest and most balanced performance, improving animal accuracy while maintaining high vehicle accuracy. On the other hand, input-free contaminated MoE underperforms, and fine-tuning pre-trained model sits in between.\n\n2. **Performance of softmax-contaminated MoE under distinguishable settings is higher than that under non-distinguishable setting:** It can be seen that the performance of softmax-contaminated MoE under the distinguishable setting presented in Table 1a is generally higher than that under the non-distinguishable setting presented in Table 1b.\n\n***\n**Q4: How do these findings manifest in real fine-tuning tasks? For instance, if one fine-tunes a large language model with a small prompt on a new task that is actually already solvable by the base model, does one observe slower convergence or higher data requirements (as the theory suggests)? The current experiments are synthetic for control, but have the authors thought about testing their ideas on a toy real-world scenario (e.g., fine-tuning a small neural network with a fixed part vs. a learned part)?**\n\nThanks for your questions. As mentioned in Q3, we include few shot adaptation experiments on DINOv2 ViT-S embeddings of CIFAR-10. We compare two cases: (i) _distinguishable setting_, where the prompt does not share knowledge with the pre-trained model; and (ii) _non-distinguishable setting_, where they share some common knowledge. In both cases, the performance of softmax contaminated MoE under the distinguishable setting (see the above Table 1a) is higher than that under the non-distinguishable setting (see the above Table 1b). This observation aligns with our theoretical findings.\n\n***\n**References**\n\n[2] Yan et al. \"Understanding expert structures on minimax parameter estimation in contaminated mixture of experts.\" In AISTATS, 2025."}]

Author rebuttal:

Meta-review: They provide the first theoretical analysis of a technique for fine-tuning, where a pre-trained expert model and a trainable Gaussian expert ("prompt" expert) are combined via a SoftMax linear gating network.
They consider the maximum likelihood estimator (MLE) for gating and prompt parameters.  They find multiple regimes of convergence rates, depending whether various conditions hold (e.g., a "distinguishability" condition), and establish minimax rates (achieved by MLE) in each regime (the strongest of which is a $1/\sqrt{n}$ rate of convergence).

The reviewers are nearly unanimous in favoring acceptance.  They are encouraged to see a theoretical treatment of this technique motivated by popular approaches to fine-tuning.  They praise the thoroughness of the analysis, which identifies multiple regimes of convergence depending on which conditions hold.

Initial reservations raised in the reviews appear to have been addressed in the rebuttals, which the reviewers seem to be satisfied with.

The one major critique noted by all reviewers is that the experimental evaluation is synthetic and more experiments may be called for.  The authors' rebuttals address this by providing additional experiments on real data.  Most of the reviewers seem content with these additional experiments.
Decision: Accept (poster)