Theory on Mixture-of-Experts in Continual Learning
Authors: Hongbo Li
Sen Lin
Lingjie Duan
Yingbin Liang
Ness Shroff
Venue: ICLR 2025 Spotlight (2025)
Keywords: continual learning
mixture-of-experts
catastrophic forgetting
generalization error
Primary area: learning theory

Abstract: Continual learning (CL) has garnered significant attention because of its ability to adapt to new tasks that arrive over time. Catastrophic forgetting (of old tasks) has been identified as a major issue in CL, as the model adapts to new tasks. The Mixture-of-Experts (MoE) model has recently been shown to effectively mitigate catastrophic forgetting in CL, by employing a gating network to sparsify and distribute diverse tasks among multiple experts. However, there is a lack of theoretical analysis of MoE and its impact on the learning performance in CL. This paper provides the first theoretical results to characterize the impact of MoE in CL via the lens of overparameterized linear regression tasks. We establish the benefit of MoE over a single expert by proving that the MoE model can diversify its experts to specialize in different tasks, while its router learns to select the right expert for each task and balance the loads across all experts. Our study further suggests an intriguing fact that the MoE in CL needs to terminate the update of the gating network after sufficient training rounds to attain system convergence, which is not needed in the existing MoE studies that do not consider the continual task arrival. Furthermore, we provide explicit expressions for the expected forgetting and overall generalization error to characterize the benefit of MoE in the learning performance in CL. Interestingly, adding more experts requires additional rounds before convergence, which may not enhance the learning performance. Finally, we conduct experiments on both synthetic and real datasets to extend these insights from linear models to deep neural networks (DNNs), which also shed light on the practical algorithm design for MoE in CL.

Published as a conference paper at ICLR 2025
THEORY ON MIXTURE-OF-EXPERIS IN CONTINUAL LEARNING
Hongbo Li Sen Lin", Lingjie Duan', Yingbin Liang', Ness B. Shroff
"Engineering Systems and Design Pillar, Singapore University of Technology and Design.
hongbo_ 1 @emymail. sutd. d. edu, sg, lingjie duan8 sutd. edu.sg *Department of Computer Science, University of Houston, slin50ecentraa uh..uu DDepartment of ECE, The Ohio State University, {1i. 15242, 1 liang. 889, shroff. | cosu.edu *Department of CSE, The Ohio State University
ABSTRACT
Continual learning (CL) has garnered significant attention because of its ability to adapt to new tasks that arrive over time. Catastrophic forgetting (of old tasks) has been identified as a major issue in CL., as the model adapts to new tasks. The Mixture-of offturee-offoeee of-Experts (MoE) model has recently been shown to effectively mitigate catastrophic forgetting in CL, by employing a gating network to sparsify and distribute diverse tasks among multiple experts. However, there is a lack of theoretical analysis MoE and its impact on the learning performance CL. This paper provides the first theoretical results to characterize the impact of MoE in CL via the lens of overparameterized linear regression tasks. We establish the benefit of MoE over single expert by proving that the MoE model diversify its experts to specialize in different tasks, while its router learns to select the right expert for each task and balance the loads across all experts. Our study further suggests an intriguing fact that the MoE in CL needs to terminate the update of the gating network after sufficient training rounds attain system convergence, which is not needed in the existing MoE studies that do not consider the continual task arrival. Furthermore, we provide explicit expressions for the expected forgetting and overall generalization error to characterize the benefit of MoE in the learning performance CL. Interestingly, adding more experts requires additional rounds before convergence, which may not enhance the learning performance. Finally, we conduct experiments on both synthetic and real datasets to extend these insights from linear models deep neural networks (DNNs), which also shed light the practical algorithm design for MoE CL.
INTRODUCTION
Continual Learning (CL) has emerged as an important paradigm in machine learning (Parisi et al. (2019); Wang al. 2024 ), in which expert aims to learn sequence of tasks one by over time. The expert is anticipated to leverage the knowledge gained from old tasks to facilitate learning new tasks, while simultaneously enhancing the performance of old tasks via the knowledge obtained from new ones. Given the dynamic nature of CL, one major challenge herein is known as catastrophic forgetting (McCloskey & Cohen (1989;: y Kirkpatrick et al. (2017)) where the expert can perform poorly on (i.e., easily forget) the previous tasks when learning new tasks if data distributions change largely across tasks. This becomes a more serious issue if single expert continues to serve an increasing number of tasks.
Recently, the pattee Mixture-of -xxpeess (MoE) model has achieved astonishing successes in deep learning, especially in the development of large language models (LLMs) (e.g.. Du et al. (2022): ; Li et al. (2024): Lin et al. (2024); Xue al. (2024)). By adaptively routing different input data to one of the multiple experts through gating network (iienn et al. (2013); Shazeer et al. (2016)), different experts the MoE model will be specialized to grasp different knowledge in the data. Thus inspired, there have emerged several attempts leverage MoE in mitigating the forgetting issue in CL (e.g.. al. (2022); Doan et al. (2023); Rypesé et al. (202)): Yu et al. (0224) by training each expert to handle particular set of tasks. However, these studies have primarily
Hihn & Braun| (202T); Wang et

---

Published as a conference paper at ICLR 2025
focused on experimental investigations, whereas the theoretical understanding of MoE and its impact on the learning performance in CL is still lacking. In this paper, we aim to fill this gap by providing the first explicit theoretical results to comprchensively understand MoE in CL.
To this end, we study the sparsely-gated MoE model with M experts in CL through the lens of overparameterized linear regression tasks (Viele & Tong (2002); Anandkumar et al. (2014); Zhong eet al. (2016)). In our setting of CL. one learning task arrives in each round and its dataset is generated with the ground truth randomly drawn from a shared pool encompassing N unknown linear models. Subsequently, the data is fed into a parameterized gating network, guided by which a softmax-bas router will route the task one of the M experts for model training. The trained MoE model then further updates the gating network by gradient descent (GD). After that, a new task arrives and the above process repeats until the end of CL.. It is worth noting that analyzing linear models is an important first step towards understanding the performance of deep neural networks (DNNs), as shown in many recent studies (e.g.. Evron et al. (2022); Lin et al. (2023); Chen et al. (2022)).
Our main contributions are summarized as follows.
We provide the first theoretical analysis to understand the behavior of MoE in CL, through the lens of overparameterized linear regression tasks. By updating the gating network with carefully designed loss function, we show that after sufficient training rounds (on the order of ((M) ) in CL for expert exploration and router learning, the MoE model will diversify and move into a balanced system state: Each expert will specialize either in a specific task (if M > .N) or in a cluster of similar tasks (if M < N), and the router will consistently select the right expert for each task. Another interesting finding is that, unlike existing studies in MoE (e.g., Feduu et al.] (2022); Chen et al.] (2022); [i et al.) (2024). it is necessary to terminate the update of the gating network for MoE due to the dynamics of task arrival in CL. This will ensure that the learning system eventually converges to a stable state with balanced loads among all experts.
We provide explicit expressions of the expected forgetting and generalization error to characterize the benefit of MoE on the performance of CL. 1) Compared the single expert case (M 1), where tasks are learned by a single diverged model, the MoE model with diversified experts significantly enhances the learning performance, especially with large changes in data distributions across tasks. 2) Regardless of whether there are more experts (M > N) or fewer experts (M < NN, both forgetting and generalization error converge to small constant. This occurs because the router consistently selects the right expert for each task after the MoE model converges, efficiently minimizing model errors caused by switching tasks. 3) In MoE, initially adding more experts requires additional exploration rounds before convergence, which does not necessarily improve learning performance.
Finally, we conduct extensive experiments to verify our theoretical results. Specifically, our experi- mental results synthetic data with linear models not only support findings, but also show that load balancing reduces the average generalization error. This effectively improves the capacity of the MoE model compared to the unbalanced case. More importantly, the experiments on real datasets suggest that our theoretical findings can be further carried over beyond linear models to DNNs, which also provides insights on practical algorithm design for MoE in CL.
thove-mentioned" theoretical
2 RELATED WORK
Continual learning, In the past decade, various empirical approaches have been proposed to tackle
catastrophic forgetting in CL, which generally fall into three categories: 1) Regularization approaches (e.g., Kirkpatrick et al. (017)); Ritter et al. (2018): Gou et al.) (2021): Liu & Liu (202))) p).
which introduce explicit regularization terms on key model parameters trained by previous tasks to
balance old and new tasks. 2) Parameter-isolation-aanr
approaches (e.g.. Chaudhry et al. (0188):
Serra et al. (2018): : Jerfel et al. (2019); Yoon et al. (2019); Konishi et al. (2023)). which isolate parameters associated with different tasks to prevent interference between parameters. 3) Memory- based approaches (e.g.. Farajtabar et al. (2020); JJin et al. (202I): Lin et al. (2021); (Saha et al. (2020);
Tang et al. (2021): Gao & Liu (2023). ). which store data gradient information from old tasks and replay them during training of new tasks.
On the other hand, theoretical studies on CL are very limited. Among them, Doan al. (2021) and Bennani et al. (2020) introduce NTK overlap matrix to measure the task similarity and propose variants of the orthogonal gradient descent approach to address catastrophic forgetting. Lee et al.
2

---

Published as a conference paper at ICLR 2025
(2021) consider teacher-studen framework examine the impact of task similarity on learning performance. Peng et al. (2023) propose an ideal CL framework that can achieve no forgetting by assuming i..dd data distributions for all tasks. Evron et al. (2022) provide forgetting bounds in overparameterized linear models on different task orders. Further, Lin et al. (003) provide explicit forms of forgetting and overall generalization error based on the testing error. These works collectively
suggest that learning performance with a single expert tends to deteriorate when subsequent tasks exhibit significant diversification. In contrast, our work the first to conduct theoretical analysis to understand the benefit of multiple experts on CL.
Mixture-of-Experte model. The MoE model has been extensively studied over the years for enhancing model capacity in deep learning (e.g., Eigen et al. (2013); Riquelme et al. (202T): Wang & Van Hoof 2022); Zhou al.] (2022); Chi et al. (2022); Zadouri et al. (2023)). Recently, it has found widespread applications in emerging fields such as LLMs (e.g., Du et al. (2022); Li et al. (2024);
Lin et al. (202)); Xue et al. (2024). To improve training stability and simplity the MoE structure, Shazeer al. (2016) propose to sparsify the output of the gating network. Subsequently, Fedus et al. (2022) suggest routing each data sample to a single expert instead of multiple experts. For theoretical studies, Nguyen & Chamroukhi (2018) propose maximum quasi-likell likelihood method for estimating MoE parameters, while et al. (2022) analyze MoE mechanisms in deep learning for single-task classification. Unlike these works, which do not address sequential task training in CL, our study focuses on MoF in CL., introducing distinct training phases for the gating network. Additionally, we derive explicit expressions for forgetting and generalization errors.
MoE in CL. Recently, the MoE model has been applied to reducing catastrophic forgetting in CL
(Lee et al. (2020); Hihn & Braun (2021); Wang et al. (2022); Doan al. (2023), ; Rypesć al. (2023);
Yu et allaal2.2)) (2024)) For example, Lee et al. (2020) expand the number of experts using the Bayesian nonparametric framework to address task-free CL. Rypešć al. (2023) propose diverse experts by routing data with minimal distribution overlap to each expert and then combine experts knowledge during task predictions to enhance learning stability. Additionally, Yu al. (2024) apply MoE to
expand the capacity of vision-language models, alleviating forgetting in CL., However, these works solely focus on empirical methods, lacking theoretical analysis of how the MoE performs in CL.
3 PROBLEM SETTING AND MOE MODEL DESIGN
Notations. For vector w, let ||w/|2 and ||w|/ denote its 8-2 and t-co norms, respectively, For some positive constant ej and C2, we define x = S2(y) if a > 02 lyl. x = O(y) if cilvl < x < calvl. and Ovv) if ea cilyl- We also denote by x: oly) if xf x/v - 0.
3.1 CL IN LINEAR MODELS
General setting. We consider the CL setting with T training rounds. In each round f [T]. one out of N tasks randomly arrives to be learned by the MoE model with M experts. For each task, we follow most theoretical work on CL fitting linear model f(X) = X w with ground truth w E Rdd (e.g.. Evron et al. (2022); Lin et al. (2023)). ). which serves as foundation for understanding DNN generalization performance (Blkin al. (2018); Du et al. (220)). Then, for the task arrival in the (-th training round, it corresponds to a linear regression problem, where the training dataset is denoted by D.- = y.). Here X, E Rdx* is the feature matrix with & samples of d-dimennional vectors, and y, E I is the output vector. In this study, we focus on the overparameterized regime, where 5t < d. Consequently, there exist numerous linear models that can perfectly fit the data.
Ground truth and dataset. Let W = (w),--. (WN) represent the collection of ground truth vectors of all N tasks. For any two tasks n, n' E [N], we assume ||w, w., bl = O(a)). where do E (0, 1) denotes the variance. Moreover,
assume that task possesses unique feature signal Vn. E R with I/ |/ = O(1) (Chen et al. (0022) Huang et al. )024
In each training round E [T], let n, E [N] denote the index of the current task arrival with ground truth Wn, E W. In the following, we formally define the generation of dataset per training round.
Definition 1. At the beginning of each training round E [T], the dataset D. (X,, Vt) of the new task arrival n, is generated by the following steps:
1) Uniformly draw a ground truth w., from ground-truth pool W and let Wn, Wn.
3

---

Published as a conference paper at ICLR 2025
2) Independently generate random variable Be (0, C), where C is constant satisfying C = O(1)
3) Generate X, as a collection of st samples, where one sample is given by ByVn, and the rest of the 81 1 samples are drawn from normal distribution (O, of Ia), where ot 2 0 is the noise level.
4) Generate the output to be yt -y=xx x Wn.
In any training round E [T], the actual ground truth Wi, of task arrival n4, is unknown. However, according to Definition | task f2y can be classified into one of N clusters based on its feature signal Un.- Although the position of Un. in feature matrix X, is not specified for each task Rdg we can address this binary classification sub-problem over X, using a single gating network in MoE (Shazeer aim investigate whether the MoE. model can enhance the learning performance CL. For ease of exposition, we assume 8, = for all E [T' in this paper. Then we will introduce the MoE model in the following subsections.
al. (201)) |edus al. (2022); Chen al. 2022)) ). In this context,
3.2 STRUCTURE OF THE MOE MODEL
Output
As shown in Figure I. an MoE model comprises a collection of . experts, a router, and a gating network which is typically set to be linear (Shazeer al. (2016); Fedus et al. (2022); Chen et al. (2022)) In the t-th round, upon the arrival of task ny and input of its data D, = yx), the gating network com- putes its linear output hm,,(,,, (X,, (, m( for each expert E [M],
Expert
Expert 2
Expert
(X,, Xd
Seftmax mXX, 0,)
Router
where ((m)) E TRd is gating network parameter for expert m. Define hX,, O₂) == (X,, 0(
hxx,
Update
hM (X,, 0 ((²)
}] and
G.1
Gating Network
O, = [e()) gating network for all experts, respectively. Then we obtain h(X,, O,) = Eieled] O, X,,5, where X,, the i-th sample
e(aD) as the outputs and the parameters of the
Input (K,y))
Figure 1: An illustration of the MoE model.
of the feature matrix X,,
To sparsify the gating network and reduce the computation cost, we employ top-1 "switch routing", which maintains model quality while lowering routing computation, as demonstrated by |edus et al.) (2022); Chen et al. (2022): Yang et al. (202I). Although this top-1 gating model is simple, it is fundamental gaining a theoretical understanding of the behavior of MoE in CL, and its theoretical analysis is already non-trivial. Extending the top-k routing strategy (as introduced by (Shzzeer et al. (2016)) is nontrivial and falls outside the scope of this work. However, we still provide a discussion on the learning performance of the top-k routing strategy later in Section
In each round t, as depicted in Figure [ for task n,, the router selects the expert with the maximum gate output im (Xe, !!())), denoted as my, from the M experts (Chen et al. (2022) Nguyen et al. (2024)). In practice, to encourage exploration across experts and stabilize MoE training, we add perturbations to the router (Shazeer et (2016); (Fedus et al. (2022); Chen et al. (2022) Specifically, task n, will be routed to the expert that satisfies
m arg max, (X,,
(X,, 0 !!!))) + (m))
(1)
where (m)) for any m E [M] is drawn independently from the uniform distribution Unif A]. We analyze in Appendix B that this routing strategy Eq. ( ensures continuous and stable transitions for tasks. Additionally, the router calculates the softmax gate outputs, derived by
exp(h.n (X,,)))))))
Tm X,, O2)
Vm E [M],
ΣM exp(h,,, "x.0""")))
for the MoE to update the gating network parameter O441 for experts.
3.3 TRAINING OF THE MOE MODEL WITH KEY DESIGNS
Expert model. Let w. (m) denote the model of expert m in the (-th training round, where each model is initialized from zero, i.e., w mm)m) for any E [M]. After the router determines the expert my
by Eq. 1 transfers the dataset D, = (X,, ve) to this expert for updating w.(n)) ((me) For any other expert m [M] not selected ( i.e., + m.), its model mn!") remains unchanged from wm))) In
4

---

Published as a conference paper at ICLR 2025
each round t, the training loss is defined by the mean mean-squared error (MSE) relative to dataset Dj: " w. (m)) yrll
L (w) (m)t) D:) J (Xx))
Since focus the overparameterized regime, there exist infinitely many solutions that perfectly satisfy C (w, (m) Dr) in Eq. (3) Among these solutions, gradient descent (GD) starting from
the previous expert model (mm4)) at the convergent point provides unique solution for minimizing
C." (w, (m)) Dr) in Eq. 33. which is determined by the following optimization problem (Evron et al. (2022): Gunasekar al. (2018) Lin et al. (2023))
min
ww w m2: 112.
s.t. X, wi = Yt.
(4)
Solving Eq. ( update the selected expert in the MoE model for the current task arrival n, as follows, while keeping the other experts unchanged:
w (mz)
w + (X, (y: X, un (
(m.))
Gating network parameters. After obtaining wi (m))) in Eq. (5) the MoE updates the gating network parameter from O to 0+1 using GD for the next training round. On one hand, we aim for ofm of each expert m to specialize in a specific task, which helps mitigate learning loss caused by the incorrect routing of distinct tasks. On the other hand, the router needs to balance the load among all experts (Fedus et al. (2022); Shazeer et al. (2016) Li et al. (2024)) to reduce the risk of model overfitting and enhance the learning performance in CL. To achieve this, we introduce our first key design of multi-objective training loss for gating network updates.
Key design I: Multi-objective training loss. First, based on the updated expert model w. we propose the Tollowing locality loss function for updating O
(me) in Eq. 5
Clas D.) = Eme[m] Tm (X,, @r]l|w!T
(6)
where Tm (X,, ④.) is the softmax output defined in Eq. (2). Since our designed locality loss in Eq. (6) is minimized when the tasks with similar ground truths are routed to the same expert m (e.g.. u, = ( it enjoys several benefits as shown later in our theoretical results: each expert will
specialize in a particular set of tasks which leads to fast convergence expert model w (mm) and the performance of CL in terms of forgetting and generalization error will be improved. Note that in Eq. (6). we only need to calculate the locality loss for the single expert m, as || (1m) w|(h ||2 = for any expert m o m, that has not updated its model, leading to low computational complexity.
In addition to the novel locality loss in Eq. (6), we follow the existing MoE literature (e-g.. Fedus ct al. (2022); Shazeer al. (2016); [i et al. (2024)) where an auxiliary loss is typically defined to characterize load balance among the experts:
(edg.Feedus)
Cqua (O,, D.) = a M Eme(n] f(m)) -p(").
where is constant, p(m)
Algorithm Training of the MoE model for CL
D 11 (m, = )) is the fraction of tasks dispatched to expert m since / = 1, and p!(m) 1 D Tm (X,, (x,, θr) 11 1,,, = m] is the average probability that the router chooses expert since / 1. The auxiliary loss in Eq. ( encourages exploration
1: Input: T, [ O( 1.2 x O(o) 7(m) a 0(ag ), ,, = O Ti In M];
2: Initialize mmm)
and (m) Vm E [M];
3: for p 1,... ... do
4: Generate r for any [M];
5: Select m. in Eq. i and update w (me)) in Eq.
across all experts since it is minimized under a uniform routing with f!(m) M
6:
if > then
7:
for Vm E [M] with Im - hmmll I < r do
and p!mm) मे- Although the definition of auxiliary loss in Eq. (7) is not new, it necessary and plays crucial role for balancing the load across experts in the MoE model for CL.
8:
/(m) = 1; Convergence flag
9:
end for
10: end if
11: if 3m, (mm) then
Update o(m) as in Eq. (9) for any mn E [M]:
12:
13: end if
Based on Eq. 3 Eq. 6 and Eq. we fi- nally define the task loss for each task arrival
end for
5

---

Published as a conference paper at ICLR 2025
ny follows:
(m))) Dr) + ( D.) + Li Ciua (o,, Dr). Clask . wi (m))) D:) = L
(w) (mogm))
(8)
Commencing from the initialization n. the gating network is updated based on GD:
(mm)
0++1 0, 0mm)) -n- V ofm)) Ctakk (O2, w. (m)) .D),, V' [M]
where 17 is the learning rate. Note that w (me) in Eq. 5 is also the optimal solution for minimizing
Clask (O,, w (m) D.) in Eq. (8). This is because both L(o0(O); Dz) and 23 (O, Dr) are derived
after updating w. w.!")), making C (z)("))) Dz) the sole objective for w (me) in Eq. (8).
Key design II: Early termination. To ensure the stable convergence of the system with balanced loads among experts (which we will theoretically justify in Section 4. after training sufficient (i.e., T)) rounds for expert exploration, we introduce an early termination strategy in Algorithm by evaluating convergence flag 7(m) for each expert m. This flag assesses the output gap, defined as 1m (X,, 0z) /mm (Xe, 0,)]. between the expert itself and any selected expert my for t > Ti. If this gap exceeds threshold r for expert m, which indicates that gating network parameter 0fm) has not converged. then the MoE model continues updating O for all experts based Eq. (9) Otherwise, the update of O2 is permanently terminated.
4 THEORETICAL RESULTS ON MOE TRAINING FOR CL
In this section, we provide theoretical analysis for the training of expert models and the gating network in Algorithm| D which further justifies our key designs in Section 3 Specifically, (i) we first support our key design I by proving that the expert model converges fast via updating O, under our designed locality loss in Eq. (c) (ii) We then show that our key design II, early termination in updating O., is necessary to ensure a stable convergence system state with experts' balanced loads. For clarity, we study the case with M >> N in this section (labeled as M > N version), and further extend the results to the M- < N version in appendices. To characterize expert specialization, we first show that each expert's 's gate output is determined by the input feature signal Un of Xt.
Lemma 1 (M> N version). For any two feature matrices X and X with the same feature signal On with probability at least o(1), their corresponding gate outputs of the same expert m satisfy
Imm [X, (()) hm 0 =((o)-).
(10)
The full version containing M < N case and the proof of Lemma [ are given in Appendix C According to Lemma] [ the router decides expert m, for task ne based primarily on its feature signal Day- Consequently, given N tasks, all experts can be grouped into N sets according to their specialty,
i.e. their gating parameter o(m))' to identify feature signal Un. where each expert set is defined as:
Mn fm E [M]| I
arg (O) (o(")) ( To;}.
(11)
The following proposition indicates the convergence of the expert model after sufficient training rounds under Algorithm[I [T]
Proposition 1 (M > N version). Under Algorithm 7 with probability at least 1 o(1), for any t > Tr, where Ti MM,, each expert m E [M] stabilizes within an expert set Mn, and its
expert model remains unchanged beyond time Ti, satisfying WT++1 (mm)
(m)
= (m)
The full version and the proof of Proposition] 1 are given in Appendix Proposition 1 demonstrates that after Ti rounds of expert exploration, each expert will specialize in a specific task, enforced by minimizing locality loss in Eq. (6). After that, expert models remain unchanged until the end of T.
Next, the following proposition characterizes the dynamics of gate outputs if there is no termination of updating gating network parameters O in Algorithm [
Proposition 2 (M > N version). If the MoE keeps updating O by Eq. O at any round t E [T'), we
obtain: 1) At round 1 = [[ [1 -0.25 M, the following property holds
(X,,, (m)) Imm (Xe, ofm")) 0( ) ! if.mm m' Mn.
(12)
(( otherwise.
2) At round 1/ 111 0.75 M, the following property holdy Am (X1,, )(m))) hm (X,,; 0/m²) > ( Vm, m' E [M]
:
(13)

---

Published as a conference paper at ICLR 2025
The full version and the proof of Proposition 2 are given in Appendix According to Proposition 2 the MoE updates e at any round t, the gap between gate outputs of experts within the same expert set converges to (( by round t (Eq. ) In contrast, the gap between experts in different sets sufficiently large, i.e., (g (0005)) 70, indicating that the MoR has successfully diversified experts into different sets round 1 in Eq. (TT). However, unlike MoE. in single-task learning that can stop training at any time after the expert models converge (c.g.. Celik et al. (2022); Li et al. (2024)), MoE in CL requires both the gating network and the experts to be suitably updated with the continuous arrival of new tasks. This is necessary to balance the load on each expert and maximize the system capacity utilization. However, continuing updating O according to Eq. (9) will eventually reduce the output gap between any two experts to ( 1.75 in Eq. (13) at training round (2, causing the router to select wrong experts for subsequent task arrivals and incurring additional training errors.
Based on Proposition] 2 it is necessary to terminate the update of θ to preserve sufficiently large output gap between any two experts in different sets, ensuring expert diversity as in Eq. (12) at round ti. This motivates our design of early termination in Algorithm outlined from Line 7 to Line 13
In the next proposition, we prove the benefit terminating updating O in Algorithm
Proposition 3 (M > N version). Under Algorithm [ the MoE terminates updoting O since round T2 On11aa 10000o00.)) M). Then for any task arrival at > T2, the router selects any expert m E Mn, with an identical probability of (Mael
where [Mail is the number of experts in set Mn.
The full version and the proof of Proposition 3 are given in Appendix G
According to Algorithm [
once the MoE terminates updating +, the random noise (m²) in Eq. L will guide the router to select experts the same expert set with identical probability, effectively balancing the loads across experts therein. Our theoretical analysis will be further corroborated by the experiments later in Section 6
5 THEORETICAL RESULTS ON FORGETTING AND GENERALIZZTION
For the MoE model described in Section we define Ee ("( as the model error in the t-th round: (w = ||/| (m,) w,,, 112.
(14)
which characterizes the generalization performance of the selected expert m, with model w, (mi) for task n, at round Following the existing literature on CL. (e.g.. [Lin .ei al. (2018)), we assess the performance of MoE in CL using the metrics of forgetting and overall generalization error, defined as follows:
al. (2023); Chaudhry
(1) Forgetling: Define F; as the forgetting of old tasks after learning task ni, for L E , ,T):
F
Σ (w(( - E, (w(mm))).
(15)
(2) Overall generalization error: We evaluate the generalization performance of the model wm)) from the last training round T by computing the average model error across all tasks:
1 Gr T Σ (wol")))
(16)
In the following, we present explicit forms of the above two metrics for learning with single expert (i.e., M = 1) as a benchmark (cf. Lin et al. (2023)). Here we define r := 1- as the overparameterization ratio.
Proposition 4. M 1, for any training round t E (2, ;T), we have
[ 1 D+=1
ΣN
||w,ll 112 + -- Enfe ||,, w1 112
(17)
E - Σ W 12
--p No
Enfer [|., -on-ww,ll"" w/l
(18)
Note that the setting here (with M 1) differs slightly from Lin et al. (023) we have N tasks in
total. Hence a proof of Proposition 4 is provided in Appendix H Proposition 4 implies that distinct tasks with large model gap Enfrs [w,- w 1 lead to poor performance of both B[F,] in Eq. (17
and B[G.) in Eq. 11). which is missing in the existing CL. literature (e.g.. Lesort et al. (023))
[e.g..Lssort
7

---

Published as a conference paper at ICLR 2025
Next, we investigate the impact of MoE on CL under two cases: (I) when there are more experts than tasks (M > N), and (II) when there are fewer experts than tasks (M < N). The benefit of MoE will be characterized by comparing our results with the single-expert baseline in Proposition 4
5.1 CASE I: MORE EXPERTS THAN TASKS
Based on Proposition | we derive the explicit upper bounds for both forgetting and overall gener- alization error in the following theorem. To simplify notations, we define (m) == L. f/ p(()) as the cumulative number of task arrivals routed to expert up to round t, where f(()) is given in Eq. 7
Theorem 1. If .M Q(N (n) for each round E {2, ,Ti,, the expected forgettinng satisfies [[F]] < + Er=l
(m+)) (
=x=1 || 2 pl (m==) -p"1 (mm)))
Enger |/w wn,l?
N
(19)
For each & (Ti + 1,... ,T) we huve [[F]] t-1 last round T, the overall generalization error satisfies
( Further, after training task wr the
(mmr)
E < <
||
N Enfre 1| - 13
(20)
The proof of Theorem is given in Appendix and we have the following insights.
1) Forgetting. If S Ti, in Eq. (19). the coefficient p.(!"r) - of the term Σ ||,,,|1 is
smaller than O because L. 2 L ( and < 1, indicating that the training of new tasks enhances the performance of old tasks due the repeated task arrivals this phase. Meanwhile, the coefficient of model gap En ||,,'' - w,,ll 2 is greater than O, indicating that the forgetting is due to experts' exploration of distinct tasks. However, as stated in Proposition]] I once the expert models converge at t = Ti, training on newly arriving tasks with correct routing no longer causes forgetting of previous tasks. Consequently, for E (Ti + 1,.... T), E[F] 2-1 [[Frr decreases with & and converges to zero T co. This result highlights that, unlike the oscillatory forgetting observed in Eq. (17) for a single expert, the MoE model effectively minimizes expected forgetting in CL through its correct routing mechanism. Furthermore, decrease in task similarit,, i.e., larger model gaps, further amplifies the learning benefit of the MoE model.
2) Generalization error. Note that the second term En [[w Wn || dominates the generalization
error when tasks are less similar with large model gaps. In Eq. (20). the coefficient
1 of
the model gaps ΣN ||w,, w,, is smaller than - - in Eq. (18). due to the convergence of expert models after round Tj. Therefore, the generalization error under MoE is reduced compared to
that of single expert, especially as T increases (where ,T approaches in Eq. (18))
3) Expert number. According to Theorem 1 for t > Tı, E Fi] increases with Tj as additional rounds of expert exploration accumulate more model errors Eq. 19). Regarding E[GT] in Eq. 200).
longer exploration period Tj for experts increases the coefficient 1 7 (((rr))) of the model gaps, leading to an increase in E[Gy] when the model gaps across tasks are large. Since Ti increases with expert number M, adding more experts does not enhance learning performance but delays convergence. Note that if M = in Theorem L (m+) becomes t for the single expert, causing Eq. (99 and Eq. (0 to specialize Eq. and Eq. respectively.
5.2
CASE II: FEWER EXPERTS THAN TASKS
Next, we consider more general case with fewer experts than tasks, i.e., M < N, where Algorithm I still works efficiently. In particular,
assume that the N ground truths in W can be classified into K clusters, where K < M,, based on the task similarity. Let W3 denote the k-th task cluster. For any two tasks n, n' E [N] in the same cluster with Wn, w. E We. we assume l/w,, ww ())))) Then we let set M include all experts that specialize in tasks within the k-th task cluster.
Recall Proposition indicates that expert models converge after T1 rounds of exploration if M > N. However, in the case of fewer experts than tasks (M < N). each expert has to specialize in learning cluster of similar tasks. Consequently, similar tasks within the same cluster continuously
8

---

Published as a conference paper at ICLR 2025
routed to each expert, the expert models keep updating after round T1, behaving differently from the
M > N case in Proposition I Given the above understanding, we have the following theorem.
Theorem 2. If MM < N and M Q(K In(K)), )) for any t {1. Ti), the expected forgetting
E[Fd] is the same as Eq. (19). While for any E {Tı + +1, T,, the expected forgetring satisfies
B[F] 1-1 E 2
(m)))
N
W2ll 11 + 1-1 -==1
Ti
Enpr
|| wn, II?
N
W4 + *
Lime) (1=))) L.p (m+)-1
(21)
where V -N >n=
is the expected model gap between any two tasks in the same task cluster. After training task Doy in round T, the overall generalization error satisfies
En
Em.'eew eWe
I [Wll
B < T
ybr
m)
2
N
1|
+
T rr
(m=²)
N
(1
(mr)
Enpar ||wl' w.,l 1
V2 +
p
+ Ei=n+
(mm) (mr)
Lig
(1- -1-
(mr)
T
+ 7
4. T
-
pl.y
() L1g (mm))
(22)
where V22 N En Ex=1 [Wal Enews w.,- wnll the expected model gap between
randomly chosen task W and any task in a fixed ground-truth cluster Wk.
The proof of Theorem is given in Appendix | and we provide the following insights.
1) Forgetring- Compared to Theorem EE [[ in Eq. (1) introduces an additional term V1, which measures the forgetting of task arrivals during {Ti + T....... } caused by updating expert models for new task arrival round T E {Ti + 1, T3. However, since tasks routed to the same expert during (Ti +1, 1, T belong to the same cluster, their small model gaps lead to minimal forgetting. If there is only one task in each cluster, V becomes O and F[F becomes the same as Eq. 19
2) Generalization error. As expert models continuously update at any t, E[GT] in Eq. 22 comprises three terms: a) the expected model gap 1. ΣN l|w,, w,, between any two random task arrivals for & Ti b) the expected model gap V between two tasks in the same cluster for / > T1, and c) the expected model gap 2/2 between a random task arrival for t < Ti and any task arrival in a fixed ground-truth cluster W/ for Ti. If each cluster contains only one task, the coefficient of 2 simplifies Ei=n+1 , ((t+)) ). given L( L
without updates after Ti. Moreover,
1 No Enfre |ww- w. 2|? and v = O, resulting in Eq. (2 specializing to Eq. 20
Note that under our assumption ||.,, w., al (a) the router cannot distinguish between similar tasks n and ní within the same cluster. Consequently, adding more experts cannot avoid the errors 1 and 1/2 in Eq. (2T) and Eq. (22). Therefore, similar to our insights of Theorem [ when there are enough experts than clusters (i.c., M Q((K In(K)),, adding more experts does not enhance learning performance but delays convergence. Although the learning performance degrades compared to Theorem it still benefits from MoE compared to the single expert in Proposition
Note that if we extend to the top-k routing strategy in Eq. (I. the router will select k experts to train the same data time. In this case, the forgetting described in Eq. (2I) may decrease, as each expert may handle a smaller cluster of tasks compared to the case with top-1 routing strategy. However, similar tasks that belong to the same cluster in the top-1 case now may be divided into different clusters and handled by different expert, which may reduce the potential positive knowledge transfer among these tasks. Consequently, the generalization error in Eq. (22) may not be smaller for the top-k case.
EXPERIMENTS
In this section, we present extensive experiments on both linear models and DNNs to validate our theoretical analysis. Due to space constraints, we include detailed experimental setups and additional results on datasets such as MNIST[LCuut et al. (1989), CIFAR- Krizhevsky et al. (2009) and Tiny ImageNet Le & Yang (2015) in Appendix|A]
Key design of early termination. In the first experiment, we aim to check the necessity of terminating the update of O in Line 11 of Algorithm I
Here set T 2000, N 6, K 3 and vary
9

---

Published as a conference paper at ICLR 2025
expert number M {1, 5, 10.00020) 20). As depicted in Figure 2(a) and Figure 2 both forgetting and generalization error first increase due to the expert exploration and then converge to almost zero for all MoE models with termination of update, verifying Theorem T and Theorem 2 In stark contrast, learning without termination leads to poor performance with large oscillations in Figure 2(b) and Figure 2(d). the router selects the wrong expert for new task arrival after the continual update of D. In addition, in both Figure 2(a) and Figure 2(c) the MoE model significantly improves the performance of CL compared
single model. The comparison between M 10 and M 20 also indicates that adding extra experts delays the convergence if M > N, which does not improve learnngg performance, verifying our analysis in Theorem and Theorem
02
2002
1050
3250
2002
5.c
2000
(a) With termination.
(b) Without termination.
(c) With termination.
(d) Without termination.
Figure 2: The dynamics of forgetting and overall generalization enrors with and without termination
of updating O Algorithm Here we set N with K -3 clusters and vary M E {1.5, 10, 20).
Real-data validation. Finally, we extend our Algorithm| T and insights from linear models to DNNs by conducting experiments on the CIFAR-10 dataset (Krizhevsky et al. (2009). The details of our experiment setup is given in Appendix A.3 We set K = 4, = 300 and vary M E {1, 4, 12). In each training round, to diversify the model gaps of different tasks, we transform the x d matrix into a d x d dimensional normalized vector to serve as input for the gating network. Then we calculate the variance do each element across all tasks from the input vector, which is then used for parameter setting in Algorithm [ Figure 3 illustrates that theoretical insights from linear models also hold for DNNs, in terms of the impact of MoE and early termination on the performance of CL. in practice.
24
20-
0.0
100
253
201
230
202
100
152
203
10
156
150
200
250
2060
15.5
200
(a) With termination.
(b) Without termination.
(c) With termination.
(d) Without termination.
Figure 3: The dynamics of overall generalization eтror and test accuracy under the CIFAR-10 dataset Krizhevsky et )) Here we set K 4, N 300 and M (1,4,12) 12}.
7 CONCLUSION
In this work, conducted the first theoretical analysis of MoE and its impact learning performance in CL, focusing on an overparameterized linear regression problem. We establish the benefit of MoE over single expert by proving that the MoE model can diversify its experts to specialize in different tasks, while its router learns to select the right expert for each task and balance the loads across all experts. Then we demonstrated that, under CL, terminating the updating of gating network parameters after sufficient training rounds is necessary for system convergence. Furthermore, we provided explicit forms of the expected forgetting and overall generalization error to assess the impact of MoE. Interestingly, adding more experts requires additional rounds before convergence, which may not enhance the learning performance. Finally, we conducted experiments on real datasets using DNNs to show that certain insights extend beyond linear models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work has been supported in part by the U.S. National Science Foundation under the grants: NSF AI Institute (AL-EDGE) 21 12441,, 2NS-2312836, and was sponsored by the Army Research Laboratory under Cooperative Agreement Number W91 INF-23-2-0225. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
implied, of the Army Research Laboratory the U.S.
This work was also supported in part by the Ministry of Education, Singapore, under its Academic Research Fund Tier Grant with Award no. Initiative (SKI) Grant with no. SKI 2021 1122110400000 04_07; and in part by the Joint SMU-SUTD Grant with no. 22-KKCBB-SMU-053
in part by SUTD Kickstarter
The work of Yingbin Liang was supported in part by the U.S. National Science Foundation with the grants RINGS 21 2142253, CNS-2112471, and ECCS-2413528.
We sincerely thank Qinhang Wu for his invaluable assistance in conducting extensive experiments during our rebuttal, which greatly contributed to the acceptance of this paper.
REFERENCES
Animashree Anandkumar, Rong Ge, Daniel J Hsu, Sham M Kakade, Matus Telgarsky, et al. Tensor decompositions for learning latent variable models. J. Mach. Learn. Res., 15()):2773-2832, 2014.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pp. 541-549. PMLR, 2018.
Mehdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006. 1942, 2020.
Onur Celik, Dongzhuoran Zhou, Ge Li, Philipp Becker, and Gerhard Neumann. Specializing versatile skill libraries using local mixture of experts. In Conference on Robot Learning, pp. 1423-1433. PMLR, 2022.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint /rXiv:/8/2..00 2018.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer deep learning. Advances in Neural Information Processing Systems, 35:23049-23000 2022.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 55:34600-34611 2022.
Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pp. 1072-1080. PMLR, 2021.
Thang Doan, Seyed Iman Mirzadeh, and Mehrdad Farajtabar. Continual learning beyond a single model. In Conference on Lifelong Learning Agents, pp. 961-991. PMLR, 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixturt-of-experts. In International Conference on Machine Learning, pp. 5547-5569. PMLR 2022.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint 13/2,413 2013.

---

Published as a conference paper at ICLR 2025
Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp. 4028 4079. PMLR, 2022.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference Artificial Intelligence and Statistics, pp. 3762-3773.
PMIR, R, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Rui Gao and Weiwei Liu. Ddgr: Continual learning with deep diffusion- based generative replay. In International Conference Machine Learning, pp. 10744-10763. PMLR, 2023.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision 129(6):189-1111 2021.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp. 1832-1841. PMLR, 2018.
Heinke Hihn and Daniel Braun. Mixuure-ff-varratoonalleeer for continual learning. arXiv preprint vrXiv:2110.2660 2021.
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence transformers. International Conference Machine Learning. PMLR, 2024.
Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in neural information processing systems, 32, 2019.
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-basse editing of memory examples for online task-free continual learning. Advances in Neural Information Processing Systems, 34: 29193-29205, 2021.
Peizhong Ju, Xiaojun Lin, and Jia Liu. Overfitting can be harmless for basis pursuit, but only to degree, Advances Neural Information Processing Systems, 33:7956-7967, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinste Barwinskw, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 (13):3521-3526, 2017.
Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter- level soft-masking for continual learning. International Conference on Machine Learning, pp.
17492-17505. PMIR, 2023.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back back-propagation network. Advances in neural information processing systems, 2, 1989.
Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. International Conference on Machine Learning, pp. 6109-6119, PMLR, 2021.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free continual learning. In International Conference on Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2025
Timothée Lesort, Oleksiy Ostapenko, Pau Rodriguez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. In Conference Lifelong Learning Agents, pp. 63-65. PMLR, 2023.
Jing Li, Zhijie Sun, Xuan He, Li Zeng, Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, and Xin Chen. Locmoe: A low-overhead moe for large language model training. arXiv preprint arXiv:2401. 13920, 2024.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 15947, 2024.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. In International Conference Learning Representations, 2021.
Sen Lin, Peizhong Ju, Yingbin Liang, and Ness Shroff. Theory on forgetting and generalization of continual learning. In International Conference Machine Learning, pp. 21078 100. PMLR, 2023.
Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. In International Conference on Learning Representations, 2021.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109-165. Elsevier, 1989.
Hien D Nguyen and Faicel Chamroukhi. Practical and theoretical aspects of mixture-of-experts modeling: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 6(4):e1246, 2018.
Huy Nguyen, Pedram Akbarian, Fanqi Yan, and Nhat Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations, 2024.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-7,, 2019.
Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pp. 77585-27610. PMLR, 2023.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture experts. Advances in Neural Information Processing Systems, 34:8583-8595 2021.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
Grzegorz Rypešé, Sebastian Cygert, Valeriya Khan, Tomasz Trzcinski, Bartosz Michal Zielińskii and Bartlomiej Twardowski. Divide and not forget: Ensemble selectively trained experts in continual learning. In The Twelfth International Conference on Learning Representations, 2023.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference Learning Representations, 2020.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference Machine Learning, pp. 4548 4557. PMLR, 2018
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experrs layer. In International Conference Learning Representations, 2016.
13

---

Published as a conference paper at ICLR 2025
Shixiang Tang, Dapeng Chen, Jinguo Zhu, Shijie Yu, and Wanli Ouyang. Layerwise optimization by gradient decomposition for continual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 9634-9643 2021.
Kert Viele and Barbara Tong. Modeling with mixtures of linear regressions. Statistics and Computing, 12:315-330, 2002.
Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Cosel: Cooperation of small continual learners is stronger than a big one. In European Conference on Computer Vision, pp- 254-271. Springer, 2022.
Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Qi Wang and Herke Van Hoof. Learning expressive metaareepresentations with mixture of expert neural processes. Advances in neural information processing systems, 35:26242-26255, 2022.
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: early effort on open mixture-of-exxprts language models. arXiv preprint arXiv:2402.01739, 2024.
An Yang. Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. M6-t: Exploring sparse expert models and beyond. arXiv preprint arXiv:2105. 15002, 2021.
Jaehong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust contin- ual learning with additive parameter decomposition. In International Conference on Learning Representations, 2019.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Dong Wang, Huchuan Lu, and You He. Boosting continual learn- ing of vision-languae models via 2024.
adapters. arXiv preprint arXiv:2403. 11549,
Ted Zadouri, Ahmet Üstüin, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
Kai Zhong, Prateek Jain, and Inderjit S Dhillon. Mixed linear regression with multiple components. Advances in Neural Information Processing Systems, 29, 2016.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 15:7103-7114, 2022.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
Experimental details and additional experiments
16
A.1 Experiments compute resources
16
A.2 Experimental details of Figure
16
A.3 Experimental details of Figure 3
16
A.4 Experiments the MNIST dataset..
16
A.5 Experiments on the CIFAR-100 dataset
17
A.6 Experiments the Tiny ImageNet dataset
17
A.7 Experiments on termination threshold and load balance
18
Smooth router
19
C Full version and proof of Lemma 1
20
D Analysis of loss function
21
Full version and proof of Proposition
23
E.1 Proof of Lemma 7
24
E.2 Proof of Lemma
25
E.3 Proof of Lemma' 9
25
E.4 Proof of Lemma 10
26
E.5 Proof of Lemma 11
26
E.6 Proof of Lemma 12
26
E.7 Proof of Lemma 13
26
E.8 Final proof Proposition
27
F Full version and proof of Proposition
28
F.1 Proof of Lemma 14
29
F2 Proof of Lemma 15
29
F3 Final proof Proposition 2
29
G Full version and proof of Proposition
31
H Proof of Proposition 4
32
I Proof of Theorem
33
1.1 Proof of Lemma 16
33
I.2 Final proof Theorem
34
J Proof of Theorem 2
35
15

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS
A.1 EXPERIMENTS COMPUTE RESOURCES
Operating system: Red Hat Enterprise Linux Server 7.9 (Maipo)
Type of CPU: 2.9 GHz 48-Core Intel Xeon 8268s
Type of GPU: NVIDIA Volta V100 w/32 GB GPU memory
A.2 EXPERIMENTAL DETAILS OF FIGURE
Synthetic data generation. We first generate N ground truths and their corresponding feature signals. For each ground truth w,, E R where n E [N], we randomly generate d elements by normal distribution (0, oo). These ground truths are then scaled by a constant to obtain their feature signals en In each training round i, we generate yz) according to Definition | based on ground-trutt pool W and feature signals. Specifically, after drawing wn, from W, for X, E Rdxa , we randomly select one out of s samples to fill with Vn,, The other 11 samples are generated from from (00, o la). Finally, we compute the output y; X, wn, Here we set og = 0.4, og = 0.1, In Figure 2
and
we set 17 0.5, 0.5 and 0.3.
A.3 EXPERIMENTAL DETAILS OF FIGURE
Datasets. We use the CIFAR-10 (Krizhvssyy et al. 2009) dataset, selecting 512 samples randomly for training and 2000 samples for testing at each training round.
DNN architecture and training details. We employ non-pretrained ResNet- 18 as our base model. Each task is learned using Adam with learning rate governed by a cosine annealing schedule for 5 epoches, with minibatch size of 32, weight decay of 0.005. The initial learning rate is set to 0.0005, and is reduced to a minimum value of 10-6 over a total of 300 rounds.
Task setups. We define the ground truth pool as W (0), (4), (5), (9)). representing K 4 clusters of tasks for recognizing the image classes airplane, deer, dog, truck, respectively. The experiment spans 7' = 300 training rounds with N 300 tasks. We randomly generate the task arrival sequence [nekeeTT where each n, is drawn from (0). (4), (5), (9) with equal probability 1. We then conduct two experiments (with and without termination) using the same task arrival order. For each task I E [T]. we randomly select its type (e.g.. task (0) for recognizing the airplane class image) and 512 corresponding samples, ensuring that tasks have distinct distributions and features.
forgelting
10-1
100
200
300
100
200
300
(a) With termination
(b) Without termination
Figure 4: The dynamics of forgetting under the CIFAR-10 dataset. Here we set N and M E [1,4}.
A.4 EXPERIMENTS ON THE MNIST DATASET
Datasets. We use the MNIST dataset LeCun et al. (1989), selecting 100 samples randomly for training and 1000 samples for testing at each training round.
DNN architecture and training details. We use a five-layer neural network, consisting of two convolutional layers and three fully connected layers. ReLU activation is applied to the first four layers, while Sigmoid is used for the final layer. The first convolutional layer followed by a 2D max-pooling operation with stride of 2. Each task is learned using SGD with learning rate of
16

---

Published as a conference paper at ICLR 2025
0.2 for 600 epochs. The forgetting and overall generalization error evaluated as described in Eq. (15) and Eq. (16), respectively. Here, (ung (m)) is defined as the mean-squared test emror instead of Eq. (14)
Task setups. We define the ground truth pool as W {(1,, (4), (7) representing K clusters of tasks for recognizing the numbers 1, 4, and 7, respectively. The experiment spans T = 60 training rounds with N = 60 tasks. Before the experiments in Figure 5 we randomly generate the task arrival sequence [r)])E[T], where each is drawn from (1), (4), (7) with equal probability 1. We then conduct two experiments (with and without termination) using the same task arrival order. For each task & [T], we randomly select its type (e.g.. task (1) for recognizing the number 1) and 100 corresponding samples, ensuring that tasks have distinct distributions and features.
4199
6.24
3333
395
5
2.00
2
3:50
855
6:34
3800
632
8.00
33
20
10
20
Pounds
(a) With termination.
(b) Without termination.
(c) With termination.
(d) Without termination.
Figure 5: Learning performance under the MNIST dataset (LeCun et al. 1989)) )) Here we set
K 3, N 600 and M (1, 4.
A.5 EXPERIMENTS ON THE CIFAR-100 DATASET
Datasets. We use the CIFAR-100 (Krizhevsky et al. (2009)) dataset, selecting 192 samples randomly for training and 600 samples for testing at each training round.
DNN architecture and training details. They are same as the experiments on the CIFAR-10 dataset in Appendix A.3
Task setups. We define the ground truth pool as W = {(28), (40), (52). (72), (79), (99)), rep- resenting K 6 clusters of tasks for recognizing the image classes telephone, bee, mountain, bear, turtle, tractor respectively. The experiment spans T = 350 training rounds with N = 350 tasks. We randomly generate the task arrival sequence [ne]ee]T] where each n, drawn from (28), (40), (52), (72). (79), (99) with equal probability J. We then conduct two experiments (with and without termination) using the same task arrival order. For each task E [7), we randomly select its type (e.g., task (28) for recognizing the telephone class image) and 192 corresponding samples, ensuring that tasks have distinct distributions and features.
0.8
0.8
0.4
#m1
0.4
R-2
M
10
Rounals
Rounds
(a) With termination
(b) Without termination
Figure 6: Learning performance under the CIFAR-100 dataset. Here we set N = 6 and M {1,8, 10}.
A.6 EXPERIMENTS ON THE TINY IMAGENET DATASET
Datasets. We use the Tiny ImageNet (Le & Yang (201)) dataset, selecting 192 samples randomly for training and 300 samples for testing at each training round.
17

---

Published as a conference paper at ICLR 2025
Table 1: The average incremental accuracy under the CIFAR-100 dataset.
Expert number
Test accuracy (%)
M =
17.1
M 8
25.5
M = 8 w/o ET
10.3
M 10
32.9
M = 10 w/o ET
9.1
DNN architecture and training details. They the same the experiments on the CIFAR-10 dataset in Appendix A.3
Task setups. We define the ground truth pool as W = {(20), (50), (83), (145). (168), (179) ). repre- senting K clusters of tasks for recognizing six disjoint image classes from Tiny Imagenet respec- tively. The experiment spans T 300 training rounds with N 300 tasks. We randomly generate the task arrival sequence [eT), where each n4 is drawn from (20), (50), (83), (145), (168), (179) with equal probability 6. We then conduct two experiments (with and without termination) using the same task arrival order. For each task E [T], we randomly select its type (e.g., task (20) and 192 corresponding samples, ensuring that tasks have distinct distributions and features.
0.6
3 0.6
0.4
0.4
M
34
10
Rounds
(a) With termination
(b) Without termination
Figure 7: Learning performance under the Tiny ImageNet dataset. Here set N 6 and M (1, 1,13.
Table 2: The average incremental accuracy under the Tiny ImageNet dataset.
Expert number
Test accuracy (%)
M =
17.4
M 8
37.6
M = 8 w/o ET
10.5
M 10
28.3
M 10 w/o ET
10.3
A.7 EXPERIMENTS ON TERMINATION THRESHOLD AND LOAD BALANCE
In additional experiments, vary termination threshold I f. ou, 1.25 1. in Line of Algorithm [ to investigate its effect on load balance and learning performance, under the same synthetic data generation Figure in Appendix
Initially, we set on = 0.4. X = 0 1.25 M 5. and N 6 with K 3 task clusters: W, (1,4), 4), W/2 52,5), and W3 (3, 6). Figure 8 illustrates the recorded task arrivals per round and their routed experts. Figure 8(a) and Figure 8(b) depict that the MoE model terminates the update
based r X, the small noise r, maximum gate output for each task cluster (c.g., expert for W. = {2, 5) in Figure 8(a) in Eq. (I).
(m) cannot alter the router's decision from the expert with the
leading imbalanced expert load. While for r < in Figure ((c) and Figure 8(d) the random
18

---

Published as a conference paper at ICLR 2025
(mm) in Eq. can mitigate the gaps of gate outputs among experts within the expert set, ensuring load balance (e.g., experts and 4 for W/2 = 5) in Figure 8(c).
noise 7t
Tark-3
Taak-2
Oycco
166
266
300
400
260
166
286
380
462
30c
100
303
603
550
106
280
200
050
(a) Threshold
0.75
(b) Threshold
do.
(c) Threshold og
(d) Threshold 00 1.5
Figure& 8: The records of task arrivals and their selected experts under different termination thresholds
Line of Algorithm
r {og oo, ob 1.22 o0 (.5). }. Here we set M N and K 3.
To further examine how load balance affects learning performance, we increase the task number to N = 30. We repeat the experiment 100 times and plot the average forgetting and generalization errors in Figure [ Figure ([a) that the forgetting is robust to a wide range of r, due to the convergence of expert models after 7' rounds' exploration. However, Figure [(b) shows that balanced loads under E { 1.25 _.00 lead to smaller generalization errors compared to imbalanced loads
under {oo ). This because diverse expert models help mitigate model errors compared to single overfitted model.
0.4
0.5
froyecigg
0.
0.3
0.0
0.0
100
200
300
400
100
202
200
4.00
500
Figure 9: We run the experiment for 100 times to take the average learning performance under four termination thresholds: I' E {08-75, Jo, of oo.55 Here set M N 30 and K 3.
B SMOOTH ROUTER
We first prove that Eq. ( ensures a smooth transition between different routing behaviors, which makes the router more stable. Suppose that there are two different datasets (X,y) and (X,y) simultaneously acting as input of the MoF. Let h and h denote the corresponding output of the gating network, respectively. Denote the probability vectors by p and p. which tell the probabilities that each expert gets routed for the two datasets. For example, Pm = Parrg ( + p(m²)) = m) and Pm.. P (arg m (dmm + p(m²) m) according to Eq. (1.. Then propose the following lemma to prove the smooth router.
Lemma's 2. The two probability vectors satisfy ||p plloo S AM2 hlloo-
Proof. Let m2 arg max (hm + r(ml) and m2 arg max (hmm p(m)). We first consider the event that mi * m2- In this case, we have
hm, + (111) > hmz + (mma) Tomz + (112) hm, + ((1)))
which implies that
Rmms Rumn (m1) p (172) > lims hm
(23)
hmz Jimm thimg hemn Based Eq. (33 we obtain
Define C(mi, m2)
br - (ma) - C(mim)m2)) him iim: him + fimy
< ll hlloo
(24)
19

---

Published as a conference paper at ICLR 2025
Therefore, calculate that ma my below
P(arg max(hm r(m)) F arg max (m + r())) m
m
<IP mi m2 E [M]] s.t. (r(m)) - -((ma)= ((m1, S (h-hbllo)
Σ
P( P((r(m)) - (m()) Cm,, m2) S (hh-hlloo)
may 5ing
Σ
([p((((ms) + C(mi, m22 - ||h hl1oo S ((m)) S (((2)) + C(m1, m2) + I| hollo))
SAM h|1 oa.
where the first inequality is derived by Eq. 24 the second inequality is because of union bound, and the last inequality is due to the fact that r (mm) is drawn from Unif [0, A].
Then for any E [M]. we have
Ip. - p.l = 1 aar max + (((m) = i}} 1 (arg ftm + r((m) = i3)
m
m
<B I (arg max hm + +((m) = =i}) - 1 max hm + ((m) = 13)
P aar max (hm + p(m) } # arg max + x(m)))
m
<M² h|/
This completes the proof of Lemma
Intuitively, Lemma 2 tells that if the outputs h and h of the gating network are similar for two tasks, their data sets yX and (X, 3) will be routed the same expert with high probability. It means that the router transitions are smooth and continuous.
C FULL VERSION AND PROOF OF LEMMA 1
Lemma 1 (Full version). For any two feature matrices and X with feature signals vn and v',. if wn Wn under M N or wa. W., We under M < N, with probability at least 1 o(1), their corresponding gate outputs of the same expert m satisfy
|m (xm(X,00 m))) (
hm (X, m(m))) 5))..).
(25)
Proof. We first focus the M > N case to prove Lemma Then consider the M < N case to prove Lemmal For dataset (X,, yt) generated in Definition I per round t, we can assume that the first sample of X, is the signal vector. Therefore, we rewrite X, [3, X1,2 Xt,o). Let
X, 13.un, 0 ... 0] represents the matrix that only keeps the feature signal.
Based on the definition of the gating network in Section Ei X,,:- Then we calculate
we have hm hm,X, (X,, m))) =
\ (Xe, mm)) hm (X, m.
mm))l
X
1=
d (o(m)
j=1
d max
m
2=
where ofm) is the j-th element of vector (()) and Ne,(as) is the (i.j)-th element of matrix X,.
20

---

Published as a conference paper at ICLR 2025
Then apply Hoeffding'ss s inequality obtain
2d
X(,ts)) < s.dd. do0 21 21-2expi
X 1000
As (..i)) -N(0, o) we have |x, = =((,), indicating exp
||t,iil d
o(1). Therefore,
with probability least o(1), we have Ei= E Xx,i..) = O(oo). Consequently, we obtain
hmm (X,, m(m) 1m [x,, 0 )l - o due to the fact that Dim 2i Xc,(,s) - O(ao)
and ((m)) = O(08-)) proven Lemma 6 later.
If M < N, calculate
Mm
(X., (X., (Xm) 1im (x., (x.,0 0(m)
(0(m))
X1,0
(0(m)) T
X,,
(0!!m))((()))
(wn, vn))
< max (o(m)) }
((,1,,)) + +0(00)
1=1
where the second inequality is based the union bound, the third inequality is because of 0!m) (a and our assumption ||e,- - Vn | xc = O(ag.)) and the last inequality is based on our proof of the M >> N case above. This completes the proof of the full version of Lemma
Based Lemmal we revisit Lemma to obtain the following conclusion for the smooth router.
Corollary 1. If datasets (X. y) and (X, y) are generated by the same ground truth, then the two probability vectors in Lemma 2 satisfy pp pll O (A
D ANALYSIS OF LOSS FUNCTION
In this section, we analyze the loss function for both gating network parameters and expert models before analyzing MoE.
Lemma Under update rule Eq. (5). if the current task n, routed to expert my have the same ground truth with the last lask nhy, where rouled expert my i.e., Wn, Wn,, then the model w (me)
of expert m, satisfies w (me) 20011
(m.)
is easy to prove Lemma by the updating rule Eq. 5 such that we skip the proof here.
Next, we examine the training of the gating network parameter.
Lemma 4. For any training round 2 1, have that EM
V of C Clask 0.
m)
Proof. As the training loss
V (i) Cit 0, we obtain
V Vom) D Clask V Vitm) cloc
L
caur
(26)
Next, we will prove Σ m
mm Cl and m- Cl
0, respectively. Based on the
two equations, we can prove Lemma
According to the definition of locality loss in Eq. (6. we calculate
0 Cloc Omm. (X,, w
(me)
(m.) (m.)))) 112-
w
(27)
(p(m)
21

---

Published as a conference paper at ICLR 2025
If m me, obtain mmm, (X,, ..)
alm (X,, m)))
(E Tmm (Xe, 0x))
Tmm (x,, θt)
00, m))
m m'
Tma (X,, O2)
Σ
T'm (X,, 0x))
X
(28)
iE|oal
If m # my. obtain
OTm, (x,, ②1)
Tym (x,, Ox) Tm (X,, (x)
Σ X
(29)
00, m)
iE(w,
Based on Eq. 27
Eq.
28
and Eq. (29 we obtain
M
M Tn (X,, O₂)
mn)
ploc
(me)
w.
(mc)
112
00, (m)
= 0.
According to the definition of auxiliary loss in Eq.
7
we calculate
M
ap(mm)
V (m)) Laux aM Σ
(m)))
00,000 (m)
m²==1
aM
e(me)
FT X2, θ2)
(30)
(m) (m)
where the second equality is due to the fact that ar(m))
0%m, Ox)
ap(m') 06 m
and
0 for any
a0
00 m)
m' + me by Eq. 77. Then based on Eq.
(88
and Eq.
29
similarly obtain
o (m)
L
pouz
aM
St
(1m))
@rm, (Xe, O.)
a0, mm)
0.
In summary, we finally prove Σ E==1
V
mm()
C
Ctask
in Eq.
26
In the following lemma, analyze the gradient of loss function with respect to each expert.
Lemma 5. For any training round E {1,.... T.. the following property holds
((oo), ift {1. ,1,,., TT))
mn) Clask 10 (((o)) ift (Ti +1 T]
(31)
for any expert m E [M]. where Ti = [M] is the length of the exploration stage.
Proof. We prove Eq.
by analyzing V gim Lloe O(ao) and Vo(m) Lgux 0 ( in Eq. 26
respectively.
First, we calculate
cCo Cio0 loo. Based on Eq.
(7
we have
00
Clac
Orm (X,, ②:)
(0("m)
w w
(me)
(m²)
2
1 1wwn Wn, 0 f(wn ww,)) } 1|00, (m²) (m))
(m²) 0T,, (X,, O,) )
w [|2
00 (m)
wi (mp))) 1 Otm, (x)
< <1lws (mi1)
00 (m)
where T the index the last task that routed to expert my the second equality is derived by Lemma 3 As 05m, (X, (1) and w, ~N, d.). we finally obtain
m)
| (m) cloc xo O(ao).
Next, we further calculate
V (m) Cana loos which contains the following two cases.
22

---

Published as a conference paper at ICLR 2025
Ift {1,... ,Ti,, by Eq. (30), we have
a.M f(m)). @nm, (X,,0))
| mm) Ltue Ixc 2 11
1
Ti
00(m)
!looft ((m))
0nm, (x,, (2)
1100 = S2(oo),
a0, mm
where the second inequality is derived by setting 17 (((0))5) to make Ti If (Ti + 1, ,T,. we calculate
1000" 0.5 M
pl
Caws
80
a.M
Ti
((m))
@h (X,, ++)
m)) 100
=
Based on the derived || ml Cloc
Doo and g(
Lt oo above, we can finally calculate
|| o! Llask)l 11oo based on Eq. 26
Fort (I, Ti}, if 7 m4, have
V m/m)) Lt
Clask
0c =
V afm) fr
V ofm) Le
Lewe 000
a.M (mm)) dtm, (X,, O2)
< (10()))
lt
Tj
00 mm)
(0(n))
= =((o))-
Similarly, for any E {T1 + 1,....
T, we can can obtain
V mmm)
Clask| 1l = V ml!) Cloc + V mm Lr 100
2 110(oa) + aM
St p(mm). Ozm. (X,, ②2)
Ti
0 mm)
00
O
This completes the proof of Eq. 31
Given (mm) 0 for any expert m E [M], in the next lemma, we obtain the upper bound of ((m) at any round (1,... ,.).
Lemma 6. For any training round E (1,..,,7 ,),, the gating network purameter of any expert m satisfies ||0] [m]
b| = o).
Proof. Based on Lemma 5 for any & E {1,-.. ; the accumulated update of 0."" (m) throughout the exploration stage satisfies
10 mm) 1lo nn Ti aM = !....)
For any t E (Ti + 1, TT. the accumulated update of 0 (m) throughout the router learning phase satisfies
a.M Ti
10 (m) a S $100 (m)mm)) 1o
00. + Olob 0v)
(
summary, ((o(()) 1oo = (o for any round E (1....,)
E FULL VERSION AND PROOF OF PROPOSITION i
Proposition 1 (Full version). Under Algorithm 7 with probability at least 1 - o(1), for any t > Ti where MM] each expert m E [M] satisfies the following properties:
1f M > N, expert m stabilizes within an expert set Mn, and its expert model remains unchanged beyond time Tı, satisfying u (m))
w (m)
w
23

---

Published as a conference paper at ICLR 2025
If N, expert m stabilizes within an expert Mg. and its expert model satisfies [!mo!")) (mm) w Ti +1 ol = ( for any t E (Ti + ,,,,.,
We first propose the following lemmas before formally proving Proposition Then we prove Proposition I in Appendix E.8
Lemma 7. At any training round E {1,-.. ,Ti).,,T for any feature signal On the gating network
parameter of expert m E [M] satisfies
(-O(o0), ifm m, - if + m.-
Lemma tells that for any expert my being selected by the router, its softmax value under the updated 024 is reduced since the next task t. While for any expert m without being selected, its softmax 0!!h) Un)
0 (m))
value is increased. This is to ensure fair exploration of each expert m under the auxiliary loss function in Eq. 77) In addition, for any expert without being selected, its gating network parameter m!m) is updated at the same speed with others for any signal vector on.
Lemma 8. At the end of the exploration stage, with probability at least 1 - ô, the fraction of tasks dispatched any expert m [M] satisfies
(m)
1 =1 0
(32)
M
Lemma tells that during the exploration stage, all the M experts expected to be evenly explored by all tasks. Therefore, the gating network parameter !!")) of expert m updated similarly to all the others.
Lemma 9. At the end of the exploration stage, i.e., t Tı, the following property holds
0 mm)
3(m') O1 0.5 ou).
T
for any m, m,m' m' [M] and m + m'.
Define d6 = (X,, of.,, hime (X,, !!m))) Then we obtain the following lemma.
Lemma 10. At any round if εω is close to 0, satisfies [m (X,, O:) Tmm (X,, 0.) = 0(6e). Otherwise, [m (X,, ④2) Tmmm (Xt, +.) ((8e).
Lemma 11. If M = SIN (nN))) we have [Ma] > for all n [N] with probability at least
1 1- -1). M < N, given M = (2( In(K)), we have [M&l 2 1 for all E [K) with probubility at least - o(1).
Lemma 12. At any round t, we have the following properties:
for task arrival nu with ground truth wn, = wn, under M N, if routed to a correct expert m4 Mu. then V oomm L. 0 for any expert m [M].
2) for task arrival nt with ground truth Wn. E Wy under M < N, it is routed to a correct expert m M2, then |/ mm C O(o).)) for any expert E [M].
Let X, and X, denote two feature matrices containing feature signals vn, and Un', respectively.
Lemma 13. In and n' sutisfy: 1) + n' under M > N or 2) wn E We and w. E WWe with & K' under M < N, then if expert m satisfies 1) m E M, under M N or 2) m MM under M < N. at the beginning of the router learning stage Ti + 1, then the following property holds at any round t E (T1 + ........ T):
7m
Tu Xw', Ot), Vm E [M].
(33)
Based on these lemmas, the proof of Proposition | is given in Appendix E.S
E. PROOF OF LEMMA
Proof. According to Lemma 5 for any round t E (1, Ti the auxiliary loss is the primary loss
update A of the gating network. Then based on the update rule of o(()) in Eq. (9) and the gradient
24

---

Published as a conference paper at ICLR 2025
of V p(me) Ly f(aux)
in Eq. (30 obtain
10
r+1
0 (mme)
cc ||n- V ofm L
(aaw))
1100
=0(00 n) ||amm (X,, Oi)
(X,,
E
Tmm (X), X1i
m m/ymm
sE/sel
_((o))
based on the fact that Rm, X,, 1 amd (x,, O2) Tm 0.) k,, (X,, 0.)) s
and || ll (().
While for any m * my, we calculate
(mm)
0!!² >| =11m V Volmo Li
clanx) 1000
++1
=0(00 0.0 m) T (X,, (T)))Tmm
Tm (X,, 0,)
iE(si]
-0(M-1oo) o0);
due to the fact that Tm, (X,, 4r) Tm (X,, O
Note that by Eq. (3)), we have V (m C. (aux)) 0 and V mm m) c(aux)
for any # me. Conse- quently, for expert my, its corresponding output h. at the gating network will be reduced by O for the same feature signal v, since task + While for any expert m m,, its corresponding output hm is increased by O(M-lap)
o
E.2 PROOF OF LEMMA g
Proof. By the symmetric property, we have that for any m [M]. E
at.
By Hoeffding's inequality, we obtain
P(I
((m) 1
(-2e2Ti).
M
Then we further obtain
P(IS M
(m))
S, Vm [M]) 2(1 2exp (-2e?
21 - 2M exp (-2e
Let 8 = 1 - 2M exp (-2e2Ti)). Then we obtain e 0..p.5511 Subsequently, there is
05 probability of at least 8 that f(m)) M
E.3 PROOF OF LEMMA
Proof. Based on Lemma and Lemma g and their corresponding proofs above, we can prove Cemma@helowe
For experts m and m', they are selected by the router for f1, and Ti f1, ((m)) times during the exploration stage, respectively. Therefore, we obtain
[|0((T) |1 = <((m) Tı O(au) - (1-f1,) ((m)) Ti OMM oo).
|0
(m))
a1.0
5/4 Ti ((oo) - (1 - --f(m') (((m)) ) Ti O(M
Then by Eq.
mlm)) 9 and Lemma
7 we calculate
0
I1
(5/ (m))
)) Ti O(oo) - ((1 - ((ا)) (1 (( )) Ti O(M
, - _ (mm)
O(on)
On 0.5 Go))
where the first equality is derived based on the update steps in Lemma 7 and the last equality is because of Ti
(f(m) fr, ((m')
O0n 0.5) by Eq. (32 in Lemma
25

---

Published as a conference paper at ICLR 2025
E.4 PROOF OF LEMMA 10
Proof. At any round caleulate
Im [1, O:)
Tmm (X,, 0:) Tmm (m/(X, (X,, ①2) exp (Am (0,,XX,, 0) hm (X,, 00.m)/)))) Tim (X,, 21)
Tim X,, 2.) exp (hm (X,, 0 0l))) hm (x,, . ((m²m)) (1) -
where the first equality is by solving Eq. (2 Then if de, close to O, by applying Taylor series with sufficiently small Se, we obtain
mm x h,) Tm (X,, +,)
23 Tm (X,, be (X,, 0!T)) hmm (X,, 0, ml)) = =((e)
where the last equality is because of Tm (X,, ,) 1.
While if de, not sufficiently small, we obtain
Tmm (X,, O:)l > Tmd (X,, 2)) (X,, 0!!)) Nim (X,, mlm))))
mm X,, ..)
(((d)).
This completes the proof.
E.5 PROOF OF LEMMA 11
Proof. If M by the symmetric property, we have that for all n [N], m E [M],
IP(m E M,,) N
Therefore, the probability that [Mm] least includes one expert is
1
P(MM,] 2 1) 2 1-
N
By applying union bound, we obtain
P(IM,) 2 Vn) 2 (1- (1- 1 M N
M 21 N Nexp
M
2 1 N
21 5,
N
N
where the second inequality is because (1 N N-NNN-- W small enough, and the last inequality is because of M (2V In ))
While if M N.
the same method to prove that P[[M&] 2 1, Ak) 2 1 5, given M = ((K In )) and Pmm MM)) K by the symmetric property.
E.6 PROOF OF LEMMA 12
Proof. In the case of M > N, as [Ma]l = 1, if task with ne is routed to the correct expert my Mn. we have wn!t. w by Eq. (5) Consequently, the caused locality loss (O,,Di) (O, D.) = 0, based on its definition Eq. 6
In the case of M N, as [M
for each cluster k. if task ne with Wn, E W/ is routed to the
correct expert m, E MR, we have
1/ (mmm.)))) w (m..)
(m)))
|XX x,) (y. X (o(m)))]le lloo
|X,(X] X,) -xx (wn. (0!mm))
(mm)) )1ooo
-O[llu wi11 (m=ا)) (m))lloo)
-0
where the second equality because of y: = X T wi, and the third equality is because of X
(L)). Therefore, we obtain V v Clos = 0 1.5) by solving Eq. (27), for any [M].
E.7 PROOF OF LEMMA 13
Proof. We first focus on the M > N case to prove Lemma 13 Based on Eq. (35
we obtain
Tm (X,, Or,) Fmm (Xa,, Or,) (.5 for m Mns given [0(")) oo O derived in
26

---

Published as a conference paper at ICLR 2025
Lemma 6 To prove Eq. 33 we will prove that 7m (X,; O1) Tm (X,,, Ot) (2 holds for
any 11 + 1.
V o Ctask 3o -((o)), leading to (0(#1) - - vn) - -((00-)
Based on Lemma
we have
for expert my and (0[m) 100 1) 1.S]--1] M,, we calculate
0 m!)) Un) = O for any other expert m & my- As T2 =
mm) 10
occ < O V
0 T 11 17
(T2 - -
12
I ((0..) ( 0 0.25) 00005
Therefore, |10, m1)mm) 8o (o is always true for t E {Ti + 1, .T].. and thus Tm (Xn; O.) Tm Tm.(Xn (X,', ).) (((00))) holds, meaning T. (X., θ,) > Tm (X,', θ.), Vm E [M].
For the case of M < N, we can use the same method to prove Eq. (33). This completes the proof.
E.8 FINAL PROOF OF PROPOSITION
Proof. In the of M > N, to prove Proposition 1 we equivalently prove that the end of
the exploration stage with / = 71, for any two experts m E M,, and mn' E Mn with n * n', the following properties hold:
mm (X,, θ.) Tm, (X,, Ot), Tud (X,, w O2) Tm (X,', Ot),
(34)
where X, and X. contain feature signals and Uw', respectively. Based on Eq. 34 we have each expert m E [M] stabilizes within an expert set Mn.
According to Lemma I the gating network only focuses on feature signals. Then for each expert m, we calculate
hmm (X,, ( m(m)) hm (X,,, ((m))) ((0(m) Vn vn:)l
|10| (m) |1 1|00, t 11x0
((.5.)),
where the last equality is because of ( 1l,0 ((0..5) in Lemma
6 and |/ U 1oo O(1).
Then based on Lemma 10 we obtain
IT (X,, O.) Tm Xx', O = 2(00
(35)
Next, we prove Eq. by contradiction. Assume there exist two experts m Ma and ml E Mn such that
Tm (X,, O₂) > Tm (X,; Ot), Tm (Xnr, O.) > Km ( Ot),
which is equivalent to
Tm (X,, O,) > Tm X θ)) > Tmd (Xn', O2) > nim Xn,s Ot),
(36)
because Tm (Xn, ..) > Tm (Xn', +,) and Tm
X Kn, O.) Tm (X,, θ,) based on the definition
of expert set Mn, in Eq. (II. Then prove Eq.
36
does not exist at / Ti.
For task = Ti, we calculate
hn,(X,, 0 (/m)) Naw (Xn
m))
10 (m)
0%, "Im'l |/
O ( 0.5),
(37)
where the first inequality is derived by union bound, and the second equality is because of ||v,,|oo
(1) and [|0[-) 0l olco = O(oon derived in Lemma 9 the end of the exploration phase.
Then according to Lemma 10 and Eq. Jm (X,, Or: Tmm (Xn, Or, ) = (0a0n-0.0)
(37 we obtain
(38)
Based on Eq. 36).
we further calculate
(Xns O7,) Tm (Xn, Or,) 2 mm (Xn, OT,) - nm (Xn', θT,)
27

---

Published as a conference paper at ICLR 2025
where the first inequality is derived by Eq. (36 and the last equality is derived in Eq. (35). This contradicts with Eq. 38) as do7 0.5 < 0 given 7 = (o Therefore, Eq. (36) does not exist for T, and Eq. (34) is true for t Ti.
Based on Lemma 13 we obtain that each expert set M, is stable during the router learning stage. Therefore, any round t E {Ti + 1, T) task n, with ground truth Wn, will be routed one of its best experts in M,.- Then based on Lemma 12 we have that V vi!n) Li 0 holds in the router
learning stage. Subsequently, w!!") of any expert m remains unchanged, based on Lemma
In the case of M < N, we similarly obtain that at the end of the exploration phase with t = Tí, for any two experts m MM and m' E .M with k o k', the following property holds
Tm (Xk; O.) > Tmm (Xk; Ot), Tim (Xx, O₂) > Tm (x),
where X and X contain feature signals 24 and uy, respectively.
Then during the router learning stage with t > Tı, any task n, with Wn, E We will be routed to the correct expert m E MA. Let (n) denote the minimum e--ormm offline solution for expert m.
Based on the update rule of w. (m)) in Eq.
(5). we calculate
Wi (m²)
w (me) wi-1
(m))
XX, X.) -yt- X, T,T (w (m))) w (m))
= (I-
x,) X
Jen 1-1 XX -X, () w (mo)
(m))
= =II-X,(X, x,) 'x, (w (ma)
t-1 - w (m)))
where the second equality because of y: = X, wmmm), Define P. = Xx! -Xp for task ne, which is the projection operator the solution space Wne Then obtain
w (m)
w (m) (I P,) (I Pri+)( (m) (m))
for each expert m
[M.
Since orthogonal projections P. 's are non-expansive operators, it also follows that Vt E (Ti + 1...., T) ||er (m) w
(m)) S
| (m)) || S ... S ||an (mm)
w fm) 11.
As the solution spaces Wi fixed for each expert m MR,, we further obtain
||w! /1+ll
(m)
[!wm (m) (mn) + (m) ww -w +1 +o
(m)
|! (m)
w wm())) 11 + WT w (m) 1loc
max w.
||l, - w,, 1l
where the first inequality is derived by the union bound, the second inequality is because of the
orthogonal projections for the update of w!!") per task, and the last equality is because of l| wy Iloo = ((ad.5) for any two ground truths in the same set Wi.
F FULL VERSION AND PROOF OF PROPOSITION
2
Proposition 2 (Full version). If the MoE keeps updating θ by Eq. (9) at any round t [T]. we obtain: 1) At round tı 1 0..5 M7, the following property holds
OO)))
if m' E MM, under M > N or m, m' E MM under M < N otherwise.
Am (X,, (m)) hm (Xe, 0 (m)))
(
2) At round to = In-lop 00 0.75 M) the following property holds X
Am
12 hm (X,,, 0 (mm))) Oa 1.75, Vm, m' E [M].
(m))
We first propose the following lemmas as preliminaries to prove Proposition Then we prove Proposition 2 Appendix
F3
For any two experts m, m', define Ao = | (X,, ①.) T.m (X,, 0.)l-
Lemma 14. At any round E (Ti ,tth Vm # M,, the following property holds
| mm) Ctask V oommes C task
(
(39)
28

---

Published as a conference paper at ICLR 2025
Let X,, and X, denote two feature matrices containing feature signals Un and vn', respectively.
Lemma 15. In the router learning stage with E {Ti +,....
1......,,,, ,.i b. for any two experts satisfying 1) m E Mn and m' E MM with n + n' under M > N, or m E MM and m' E MM with Wn E Wy, Ww E Wy and le & k under M < N, the following properties hold:
Tm (X,s Oe) >> Tmd (X Ot), Tim (X,', O.) > Tm (X,', Ot).
(40)
F.1 PROOF OF LEMMA 14
Proof. Based on the proof of Lemma 5 for any m o me, we calculate
C
Clask
V
@ Ck
Clask]
xc = ||([12o (m))
wi (mm)) l12 +
aM
St stmi)(
@T.. a0mm)
0mm
lloo
a0mmm)
aM, ÖT
dam:
= -
00 mm)) a0(m)) 11
5g(m) (m)
= =(Oo);
where the last equality is because of Tm. Ox)
S oo for any 1, X 0(1) and aM
e (Ti
F.2 PROOF OF LEMMA 15
Proof. We will use the same method in Appendix E.8 to prove Lemmal 15 by contradiction. Here, we only prove the case of M > N. The proof for the case of M < N is similar.
Recall Proposition that Eq. (40) is
at L Ti 1. Based
Lemma
13 we have
IT Xx,, ,t)
Xn, θ
S2 0.5 Then in the following, we aim to prove Imm X
θr,
Tm' (X,z On = ( 0.3 in Eq. (38 is always true for any m .,, and m' MM during
the router learming stage. Then according to the proof of Proposition in Appendix E.8 Eq. 40) is
also true.
Under Eq. 440 at t = Ti + 1, the router will route task t to its best expert my E MM,,, leading
to V @lm) Cloe 0 by Lemma 12 Therefore, @oim) C Clask ((oo) makes (0 (0(m:) 0 0(mm) vn)
( for expert my and (0 p(m) vn) O())-0) any other expert m m. at task
t= +1
Subsequently, for any two experts m E Mn and m' E Mn', we calculate
(0(m) o!m²²
211 1l s [10[ml) o llc (t1 Ti) 0c0
olm²)
s O 0.3) O1 (,o 0225 -
O(aon 0.5)
,,) = 5 and the second incquality is where the first inequality is because of (m) m))
because of t Ti S f1.
As 10 m) 0. ((m²)) ll = O0aon 0.5) and | (X,, ,,) Tm (X ..) = l(ag Eq. 400 is true
f for any t (Ti .....
... 1. based our proof of Proposition
F.3 FINAL PROOF OF PROPOSITION 2
Proof. We first focus on the M > N case to prove Imm (X,, (())) - hm (X, (mn |>| (m)))))
for any two different experts m, m' Mn in the same expert set in Eq. 12p. Then we prove
I/m , ((m) Am (X, 0fTh different expert sets in Eq. 12). After that, we prove Eq. (13) at round = [m>_20 o 0.75 M]. Finally,
(m') )) = g for any two experts m E MM and m E M, in
we prove that the above analysis can be generalized to the case of M < N.
In the case of M > N, let M' and m denote the two experts within set Mn with the maximum and minimum softmax values, respectively. In other words, we have
M' = arg max {Tm (X,, O,)], m' arg min { (Xt, 9.) mEM,
(41)
mEM,,
29

---

Published as a conference paper at ICLR 2025
where Un X, If the two experts satisfies Ihw (X,, 0(141) Ay (X,, (X,,,//T))) -000--7) then this equation holds for any two experts in Mr.
At the beginning of the router learning stage, we have M (XT, θr,) - Tmm (XT,: θT,) O(oon -((o
based on Proposition
According to the routing strategy in Eq. T. if the new task has ground truth Wn. it is always routed to expert M until its softmax value is reduced smaller than others. Therefore, we calculate the reduced output of gating network for expert M' in the router learning stage:
(o(Ar) +1 b vn) S ((oo) 17 (t1 - -i)
((M))
While for expert m', it will not be routed until its softmax value increased to the maximum. Therefore, we similarly calculate the increased gating output (0(,') o - v,,) Oo for expert m'.
Based Lemma 4 and Lemma 14 the gating network parameters of experts m' and M' will converge to the same value, with an error smaller than the update step of O, Therefore, we obtain
[((( a(m²) 11.00 = V Dlask
7/120
V
Caur 7/100
= Oaa 1.75
based on the fact that V
0 (oree ( -1) Ca O and r7 ( Then according to Lemma 10
we obtain Ihw (X,, : 0! in the same expert set Mn.
(M
hm (X,,, 0 ((mm))) (a) which also holds for any two experts
Next, we prove I (Xe, (mm)) hm (Xe, o0T O 7.7 in Eq. 12 for expert mn' in Eq. (41 and another expert m $ Mn.
Let Ti M,, denote the index of the expert in other expert sets with the maximum softmax value of dataset X,, where v, E Xt. other words, m = arg maxmg. { FTm (X,, O.)). According to the proof of Proposition [ we obtain Tm, (Xr,, θr,) Tm (XT, Or indicates that during the router learning stage with E {Ti + 1, f1). any task arrival n, with ground truth w,, w,, will not be routed to expert m. Therefore, nm (X,, Q,) keeps increasing with t. Then we calculate the difference between the parameter gradient of expert m. and expert m per round V 0. Clask V (lmnn)) C Clask ald frh Tm, (X,, Oc) (X,, O,) TT (X,, O.)) Eieles) X1,i
(... This equation
2 0, based on the fact that Tm (X,, O.) Tm (X,, O.) > under Lemma As V gln) Crask 0 and
0 (mm) C Clask < O, we obtain that hm (X,,0! mmm)) hm (Xe, ml) increases with t during the router (mm)) )) we obtain
learning stage. According to our former analysis of hm (X,, 0(m²)
Im (X,, 0(m²)) hm (x,, ((m))) ) (m))
= =l/m (XT, T. m0m! )- nm (T, T
(m) 0
V ml (
Closk V
of (mm)s Ctaskh 1100 -1 (t1 Ti)
Ti
=O(o8 + (a 0.75 8
where the first equality because of |/ mmm) proof of Eq. (2).
Ctask
11 = O(oo). This completes the ormn') Ct Ctask
Subsequently, we prove |/m (X1,, (mm) hm (X,,, 0 mmm)) = (s) Vm, m' E [M] in Eq. (13
by proving [X,,, 0 (M')) hm (X,,, Q0 (a 1.75) and any other expert m z Mn.
between expert M' E M, in Eq. 41
30

---

Published as a conference paper at ICLR 2025
Based on Lemmal 7 we obtain
- if me: m) m2) Un) 10(M- if my-
According to our analysis above, expert M E MM, is periodically selected by the router for training task arrivals nt n during & E (t1....,. .2)). After each training of task expert M', its gate output has hX,, (x,,0 ((a))) is reduced by (ggg22)). While at other rounds without being selected, its Under such training behavior, we obtain
gate output increased by OMM-la0
| o(A")) hM (X
((4))) (ag)
by assuming vn E X1, and Un E X12-
While for expert m & Mn. its gate output Am Xe, 0g (m)) keeps increasing for any data X, of task For any E (...... ,12), we have || (()) Lt 100 Oab 25) for expert m. Assuming that expert
m is never selected by the router for training task ny = n in the period, we obtain
Imm (x (m)) hm (x 0 ml = V g L2 Laux
1100 (12 (1)
(m)) 1oc (122 - (1) 7 Oa0 and p (X,,, M()))
Ihas (X : 0 ((5') hm X 0n
where the inequality is because of |/
a(m)) Lt Leue
hm (X1 {(())) O(08-75). This inequality indicates that there exists training round t' (f,,... )2) such that htm (X.,, o(())) > hmr(X,, o!A²)) for task arrival ny expert m' is selected to train task n again, meaning that m' E Mn, at round t'. Then the gating network parameters of experts m and M' will converge to the same value, with error of (
Consequently,
based on Lemma and Lemma 14 This completes the proof of Eq. (13) in the case of M > N.
In the case of M N,, let M' and m' denote the experts of set ME with the maximum and minimum softmax values, respectively. In other words, we have
M' arg max {T O.)], m'' arg min (Tm (X,, θ1)),
mE
mEM,
where the ground truth of task satisfies w,, Wk-
According to the proof of Proposition 2 in Appendix F3 the gating network parameters of experts m' and M' will converge the same value the end of the router learning stage, with error smaller than the update step of O, Therefore, we obtain
10 (M)
0!Th 1100
V
cask 77 00
|I(V
doc
77/000
0 011-1
0 (nn
011-
Ola0 1.75 }
based on the fact that V
2 = -O(o0 1.55) and
0 (m -
=O(ab-)) derived in
0
Lemma 12 Then we obtain I (X,, 0 ((M)) for any two experts in the same expert set Mr-
hm (X g(mm)) Oa. which also holds
Similarly, for any two experts in different expert sets, we can derive IX,, (())
hm (X,, e (m))) = 7)).. For training round t2, the proof for the case M < N is the same as the case of M N above, and thus we skip it here.
G FULL VERSION AND PROOF OF PROPOSITION
Proposition (Full version). Under Algorithm the MoE terminates updating O since round T2 = -op 0.25
-025 0.) Then for any task arrival na at >>> T2, the following property holds:
If M > N, the router selects any expert m E Mr, with an identical probability of [...l where [mng is the number of experts in set Mn.
31

---

Published as a conference paper at ICLR 2025
2) If < N and Wn, E Wk. the router selects any expert m Mg. with an identical probability of MMll where [Mx] the number of experts in set My.
Proof. In the case of M N, according to Algorithm I and Proposition
2 after the termination
of gating network update, the following properties hold: 1) (X,,, 06"²! hmm (X,: 0 0.m²!) =
( for any m E Mn and m' _M and 2) Imm (X,, 0(!)) hm (X,, 0l any m, m' E Ma, where I' = O(ag.2))
(TT) for
If the ground truth of task arrival ny satisfies Wn. = Wn. for any experts m Mn and m' $ Ma. we have
+ rt (m) hm (X,, 0, m))
(Iem' (X,, (X,, m(m²)) + rt f(m²) hm (X,, (()) (()) hm (X,, mmm²)) (m')
=
given rt (mm) @ 25) and hm (X, m(()) - Nym (X, o!!²) ((ff.15). Therefore, any expert
m' & Mn will not be selected to learn task t, and only experts in set Mn will be selected.
For any experts m E Mn. we calculate
P m, m/m E Mn. - P
P(m m = arg max { (Iim (X,,, Opm)) + +r ((mn)) m M'EMa
P Nm (X,, mm)) f(()) (hhme (X,, mm ) + +ree" r.(mi))
0, Vm' Mn)
P (m)) r( > ri (mn') vm' E M,))
[M
where the third equality is because of (m)) 2ag)-25) and [/mm (X,, m!!)) hm (Xe, o!m²))
r,m))
Oat derived under Algorithm I and the last equality is due to the fact that r!!)) satisfies uniform distribution Unif(0, AJ.
In the case of M < N, we similarly derive the following properties: 1) Mm (Xe, o(m)) hm (X,, ((m') I) 0(0g-75) for any m MM and m' MM and 2) ( (X,, m)) hm (X,, o fmm') 1)1 O(T) for any m, m' E .M.. Based on the two properties, fm (X,, (m)) rt ((m) (nm (X,, "...(mm)) + ((n')) @(0
always true for any two experts m and m' not in the same expert set Mp. Furthermore, we can similarly calculate
P P(m== m+ = m|m E MM2 = P ((m) ri (mm))
(m)
Vm' E EM) = "Mal'
This completes the proof of Proposition 3
H PROOF OF PROPOSITION 4
Proof. In the single-expert system, based on the definition of forgetting in Eq. (15) and Eq. (42 in Lemma 16 for any training rounds & E [T'] and E {I, .t. we calculate:
E[F(]
1
E
w
(ms)
wn,, 112 (m):) wn.ll 1?]
(r*
)B] Wn: 11 + Σ (1 ))n EEllwwm - wn,
Σ (1 r)r 1111 [lll wn. 111]
1=1
N
|w 1|2 + pd pt N
Σ nz
www wn
N2
32

---

Published as a conference paper at ICLR 2025
where we let Wn; denote the ground truth of the task arrival round i. Here the last equality is derived by I=ll w,, = के Σ l|,,ll and
BiIl,,, wn, | - Σx Wn,, wn.l?] NoW Emgent n Wn w.,l 11
when there only single expert.
Similarly, we can calculate the generalization error
E|Gr] = + E W
(m.)
Wn. [12]
+ >im ['Ell |1 + (1 r)r "Bll- -nn, 112]
N Σn
"
||,nll² +
Lrr Enyon' 1w0,
w/,,"
No
where the second equality is because of Eq. Proposition 4
42 in Lemma T6
This completes the proof of
PROOF OF THEOREM 1
Before proving Theorem we first propose the following lemma. Then we formally prove Theorem | in Appendix 1.2
For expert m, let 7 (mm) E {1...., , } represent the training round of the I-th time that the router selects expert m during the exploration stage. For instance, (2) = 5 indicates that round t = is the second time the router selects expert 1.
Lemma 16. At any round E (Ti + 1.... Tor {Ti + 1, 1.... t}. have |mm (m.) Wn,ll 12 1/ (m²) Wn,! 11
While at any round {1. Ti for any {1. ,t), we have
L (mm))
w (m).) Wn.1
12] = (mm) B Wny
12
Σ
(1
r)r
(mme) '[ll w (m)
wn,11 (42)
1=1
where I.( (m=) t. s(mm)) and r 1
1.1 PROOF OF LEMMA 16
Proof. At any round t E (Ti 1.... T), we have w. WT, based on Proposition 1 and
(mi) (m) (m)
Proposition Therefore, ||/! (m²) Wn,, 112 ( W., 1| is true for any round E {Ti-
1,... T and i E [Ti + ,,...,,t. t}.
Next, we prove Eq. (42) for round E {1,.... }. Define P. XX, - -X, for task t. At
current task I, there are totally Tm(m) = 1. S (m)
tasks routed to expert m. where ((m) is in Eq. (7)
Based on the update rule wx (m) in Eq. 5 we calculate
w (m.) Wn, 2 w/ (im) (mmm)))
(m.)
w.:ll 1|
(m)))
[[ P.)a (m²)
(L(m)))))))))))))))))))))))))( + P.w (m(ml)) wn,l!
= [ - P.) (m²)
(z(")) Wn,) + Pe(w (m)(((mm) - Wn. Wn.)l1
(m.) (m.) for t (r(mi) (T.m))(.( (I (mi)) (mmi)....... ... f). and
where the first equality is because there is update of w.
the second equality is by Eq. (5).
As P, is the orthogonal projection matrix for the row space of X., based the rotational symmetry of the standard normal distribution, it follows that E P. (26m4)) wn,) )ll = =|l mm (z.m)))
33

---

Published as a conference paper at ICLR 2025
Wp, 112. Then we further calculate
E w
(m)) Wn,
12 (1
E 2 (ma))
-wm,11 d
2Bll/00 ( wn. 1|2]
d'
(1
d
5 L (m))
)
10ll (m.) wnn []
kma)
(1 L (m)))
Wn: Elll (oms)(
1=1
D Dmme
11 + E1- (mm) "EE (00 (mms)) Wnx, 11
where the second equality is derived by iterative calculation, and the last equality is because of Wi [m)
= 0 for any expert m. Here we denote by r = 1 to simplify notations.
1.2 FINAL PROOF OF THEOREM
Proof. Based on Eq.
in Lemma 16 we obtain
1 kmm)
w (m.)
Wn: [13] 4 mm)
Win
[3++
Σ
(1
r)r
-Elllw, -w,,111
1=1
where 7(m)))
(L, (mi)) =
Then at any round t E 12, , Ti}, we calculate the expected forgetting as:
E B[F:]
E (mmmm)) wn, 11.111 -1100
(mm))
(m,)
m. 11
L (mm))
(n (m))
- - 1.
(m))
W 1
+
(1 r)r (m))
"El 2 (m) - -w,,1"
1=1
1=1
1. (mm)
- (1 r)r (m
- - [B(l(2)mmo0)) - Wn. [31}
j=1
where C22 (1 (1 (m)) mm)))) p -j ب m.mm)))
As task i's ground truth wn, is randomly drawn from ground truth pool W with identical probability N. we have
Bl]lwn, 12]
N
According to Lemma X and Proposition each expert m will converge to an expert set M,, before t Ti. Therefore, we obtain
N
E un (m)())
Wn, 122
-w,,l3
N
N
|m 20 1|
N
n=
R==1
Wn 11?
(43)
N2
nyf''
where the inequality is because the expected error E[[ w.,l 1 per round for < Ti is smaller than the uniformly random routing strategy with expected error E Blllw0, w, ||
N D. |!w. w., 11 and the last equality is because of l,,,, w.rl 2 0 for n' n.
34

---

Published as a conference paper at ICLR 2025
Therefore, finally obtain
E|A < उ
(m)) (m)))
N
[2en 112 + Lrr
Et
(mm))
- ||00n w.n 11
No
I- (m)))
p -s Enniin
|00r - -w wnm12
No 2===1
E i==
2 xp (m (mm))
(mm))
|enn 1| + lmpky 2
(mg)
2 Znnghn Ww - -w,1 20211
1-1
Nº
1-p (mn))
1/pp''
wnll3]
N2
I. (m))
(m,)
N
mm
p
(mm))
N
||w
+
1 wmll?
N
N2
(mn)
For any E (Ti+ +1, m E [M]. Therefore, we calculate the caused forgetting
T) based on Proposition the expert model w
wtz for any expert
t-1 (m))
E|
E
w
wn. 112 - |100 (m)) Wm, 12]
(m.)
Wny, 112 |120 (m.) mmmmm)) win 12)
E
Ti -
B[F.].
i
Based on Eq.
42
also calculate the close form of the generalization error:
E
(m:)
Wn,, ||
T
1
Σ
E[[
(m:)
wnn |1
T
(mmg)
Toy
1
Σ
(mm)
E
Wnn 12] + Σ
r)r
(mm) -1 -B[1|
(mm)())) -w.11
T
1=1
T
L. (m²)
71
N
--r
T.y
(mm)
(mn))
iim
|w,||1
| wonl"
NT
N2T
i=1
1=1
T (m))
N
(mm)
r
T1
||
Ei=1
wn,l?
NT
2
where the inequality is because of Eq.
43
PROOF OF THEOREM 2
Proof. For any round t E (1.... T. the forgetting is the same as the case of M > N Theorem tasks randomly arrive and routed to different experts. Therefore, skip the proof for t Ti.
For E {Ti + 1.... T}, the router will route tasks in the same cluster to each expert per round. Therefore, divide the i rounds into two subintervals: E (1..... ,Ti) and (Ti + 1, to
35

---

Published as a conference paper at ICLR 2025
calculate the forgetting 1
B
Σ
E (m))) Wn. 112 - 11/0 (mmmm)) wme1 112
{ (1_ mm
(mm) 11lll [1 + Ei=i mmmm
(1 r)r
(m))
10 20 (0m))) -wn, [12]
i-1 i=
دكر (1 r)ndi
(m)))
smm) - - Wnn 112]
L (mg)
T-T Ci=T++1 (1r4) p 1 113] + Σ
(1 r) Li -1 (mm))0) -Wn, 13]
+1
lerm
(m))
(1-r)pf) (ime) -5 211112, (m))()) - wn, wn,1191
j=bTy +1
tern
where the first term in the second equality is similarly derived as forgetting in Eq. 19 For the term term b in the second equality, we calculate
term term b
3
+
(1- (-r)r" (mm)
E W (mm))())) Wn. 11
t= L= (m +1
(m)))
(m²)
(1
I.
(m))
(mp)
E (m))((()))
Wn, 11
(mn)) - -Lyg
(one)
-1 (1
liz (mp) (mm)
N
w (m))(1) -w,,l'
N
n=1
1/( (mm) (img)
(m)) (m))
(1
[ 20,l1
N
W
n=1 n.W
where the third equality is derived by [|| (m Wn,, |1 EN and the last equality is because the router always routes task (,(m,) (1) = w within the same cluster We to expert m. Taking the result of term a term b into E F.F.). we obtain
== B[[lw (1,l,,,,,a) (m)) w., 12].
E[F]
1
pli
(mm)
p
(mm))
(msz)
Σ
N
Σ
Ti
(m))
(mm)
N Σ
w,,l
N
1
N2
(mn)
(ms)
(1
I (mm)) (mm))) b. (mm)
N
[w,' w.
+
N
[e
1=1++1
==1 n.WEWa
36

---

Published as a conference paper at ICLR 2025
Similarly, calculate the overall generalization
E
E
1
E WT
(m)))
wn,
[1
I km))
1
(mm)
Elllwn, 11 + Σ
(1
r)r
"[[llw- (m)))())
-wn, 11?1)
T
1=1
kmm)
1
1
Wn: [|]]
r)r
(mi))) wp:
T
T
LT (m)))
(1- r)-
(mm)
Bill ((m)())))))))))
((m)(())
Ww.l 113
6m0 t= T
+1
tem
kmg)
r)r
2 (mm - Wn,1 |!
i=1+
1m1
term
(1- r) (ome)
Blllw, (m)())
Wn. 11
1= +1
temm
where the second equality is derived by Eq. 42 and the third equality is derived by dividing T rounds into two subintervals (1, Ti and [T] 1 I...... T}.
In the above equation, term means both wn, and w, (m=))) are randomly drawn from the N ground truths, due to the fact that i < Ti and -(m=)(() Ti. Term 2 means one of wn, and (m is randomly drawn from the N ground truths before the Ti-th round, while the other one has fixed to cluster Ma after the Ti-th round (i.e., < Ti and 7(m=)() (t) > Ti or i > Ti and ((mi) < Ti). Term 3 means Wn; and (1))(()) are in the same cluster Mr. as > Ti and 7 ((m)) (1) > T1.
For i E (1.... Ti based on the proof of Theorem D in Appendix L we obtain term 1:
E W (1))) Wn, 11 < N²
Σ || w,,ll
ngnnn
While for (Ti 1,... t). we calculate term as:
N
1
B Bllle,tmmm wn,ll =
E
[1 (m))()) w,,ll
N
n=1
1
N
| w0, 1|
N
Finally, we caleulate term
N
1
E w (1))))))
wn.1 1
WC (0)))
wn, 121
N
mm
N
K
Σ
|Rw
wo 112
N
K
WAl
k=
N'EW;
37

---

Published as a conference paper at ICLR 2025
Based on the expressions of the three terms above, obtain (m))
r
(
N
Ti
(mm)
(mi))
G
<
1t 11
+
Wm wmll
T
N
Nº
1=1
(mmg)
N
K
1/ wrll
T
N
K
Wkl
1=1
1=1
n'EW
T
(m4)
Lop (1
(m))
(mm))
N
K
Ww Wn, 1|
T
N
K
W
i=1++1
72
k=
N'EW
T
(m))
-bi
(mm)
WWy
7
N
[WA
1=T++1
n=1 B.W'
which completes the proof.
38


Reviews:
[{"review_id": "uIQd1ahlJu", "reviewer": "Reviewer_DWT2", "summary": "This paper provides a theoretical study of Mixture-of-Experts (MoE) models for Continual Learning (CL). Specifically, it examines the CL of linear regression tasks in an overparameterized regime. The benefit of this regime is that each task has multiple possible solutions, increasing the likelihood of expert specialization and transfer.\n\nThis paper implements a one-layer MoE architecture with the following design choices:\n\n- **Experts**: Each expert is implemented as a linear model.\n- **Router**: A top-1 router is implemented as a linear model.\n- **Router Training**: The router is trained using gradient descent.\n- **Expert Parameters**: Expert parameters are found through a closed-form solution\n- **Training regime**: routing is performed at a per-task level, at each new trainign iteration a new task is sampled from a limited pool of tasks. \n\nAuthors propose two key design choices to facilitate learning in their MoE:\n\t\n- Training loss: in addition to the standard load balancing loss (eq. 7), they propose to use a **locality loss** (is a novel contribution afaik), that facilitates routing of the similar tasks to the same experts\n- Early termination: intuitively, once an expert has specialized sufficiently (expected to be the case after T_1 updates), further updates of the routing parameters can result in instabilities. This is achieved by terminating training of the router if the current expert's routing \"dominates sufficiently\" the other experts' routing.\n\nAuthors address the following setting:\n- at each update step t, a feature matrix X is randomly sampled following the data generating process described in Definition 1\n- among the s_t examples in the feature matrix, there is one example that contains the feature signal\n- in the addressed setting, identical tasks can reoccur, as tasks are sampled independently in each update step\n\nAuthors derive the following properties of the MoE with previously mentioned design choices:\n- the router routes experts primarily based on feature signal, and all experts can be clustered into experts sets, with specialized experts in each expert set\n- therefore, under the proposed algorithm (due to the locality loss and termination criteria), experts will likely specialize on certain task clusters after sufficient training round T_1 minimizing the effect of forgetting\n- if no termination criteria is applied, proposition 2 states that the specialization will break at round t_2 (the gap between any two experts is predicted to be the same)\n- if early termination is applied, then experts within the same expert set are chosen uniformly after T2 updates\n\nThe authors derive upper bounds on forgetting and generalization error for the MoE, showcasing that **both are reduced compared to a monolithic single-expert model**. When there are more tasks than experts, forgetting is bounded due to the router’s tendency to route similar tasks to the same experts.\n\nOverall, the intuition that MoE models, with correct routing and module specialization, can address the challenges of CL is natural and compelling. This paper effectively demonstrates how this intuition can be implemented and validated in a controlled, toy setting. The findings can offer a useful starting point for scaling these ideas to settings more relevant for practical AI applications, a very relevant work in this context is [2].\n\n[2] Rypeść, Grzegorz, et al. \"Divide and not forget: Ensemble of selectively trained experts in Continual Learning.\" arXiv preprint arXiv:2401.10191 (2024).", "questions": "-  is expert specialization necessary for CL with MoEs? \n- are modern-day large-scale AI systems operating in an overparameterized regime?\n- for load balancing loss, why not use entropy maximization? What are the benefits of the proposed load balancing loss compared to entropy maximization?\n- is cross-expert and cross-task transfer possible in the proposed system design?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality**: \n- the idea that forgetting in MoE can be mitigated solely through specialized experts and correct routing is not entirely new, e.g. see [1]\n- this paper is original in its theoretical contribution (to the best of my knowledge), providing proofs and bounds on Cl metrics with MoEs\n- the proposed locality loss and load balancing loss provide clear mechanisms for task clustering and specialization (even though load balancing loss is not a contribution of this work)\n\n**Quality**: \nI think the quality of this work is decent, due to the rigorous theoretical work and relevant experiments. Even though the focus of the work is mainly on the theoretical side, further validation on more complex datasets would be interesting and benefit the credibility of the paper's contributions. \n\n**Clarity**: while the theoretical analysis appears to be rigorous and well presented, I think the presentation would benefit a lot from incorporating more **intuitive explanations**. E.g. authors could explicitly state the idea that forgetting may be prevented solely through correct routing. Also the MNIST experiment's design is somewhat hard to follow.\n\n**Significance**:\nDespite the small scale of the experiments, I think this paper opers some interesting directions for future research, mainly among the lines of scaling these ideas to larger systems.\n\nOverall, I appreciate how this paper demonstrates how under proper specialization and routing CL can be addressed with modular solutions like MoEs.\n\n[1] Ostapenko, Oleksiy, et al. \"From IID to the Independent Mechanisms assumption in continual learning.\" AAAI Bridge Program on Continual Causality. PMLR, 2023.", "weaknesses": "- the scale of the experiments is small, while one has to uknowledge that the contributions are mainly theoretical\n- I would appreciate more intuitive lingo and explanations\n- the current implementation is essentially on the one extreme of the parameter sharing trade-off where no transfer happens between tasks? \n- it is not exactly clear how these ideas can be extended to large scale MoE with multiple expert layers and per-token routing, where correct routing as well as expert specialization is not guaranteed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HT7LgF2eLU", "reviewer": "Reviewer_WfDL", "summary": "The paper \"Theory on Mixture-of-Experts in Continual Learning\" analyzes the Mixture-of-Experts (MoE) model's effectiveness in addressing catastrophic forgetting in continual learning (CL). It establishes that MoE can enhance learning by diversifying expert specialization through a gating network, which routes tasks efficiently. The study provides theoretical insights into expected forgetting and generalization error, demonstrating that MoE outperforms single-expert models, especially with diverse data distributions. However, it notes that adding more experts requires additional training rounds before achieving convergence. Empirical results support the theoretical claims, extending the findings to deep neural networks (DNNs) for practical algorithms", "questions": "1) Line 56-57 “One learning task arrives in each round and its dataset is generated with ground truth randomly drawn from a shared pool encompassing N unknown linear model”. What is the intuition behind generating the ground truth from the linear model?\n\n2) It might make sense if the number of experts is less than the number of tasks But in Line Line 69 if M > N implies the number of experts more than the number of tasks. This is not continual learning if you are training task-specific experts. Furthermore, in Section 4, you have done the main study with more experts than tasks (Line 296).\n\n3) In Figure 2, on which datasets the experiment is performed?\n\n4) For MNIST which DNN model is used? As from the setup number of task N = 3, how did the 10 classes were split?\n\n\n## After Reading Comments\n\nI raised my score to 6 as an experiment on MNIST and CIFAR10 is not enough.\n\nAgain, thank you for your contributions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Theoretical Foundations: The paper provides a comprehensive theoretical analysis of MoE in the context of continual learning, establishing clear benefits over single-expert models through explicit expressions for expected forgetting and generalization error.\n\n2) Load Balancing: The model ensures balanced utilization of experts, which can lead to improved generalization performance as it reduces the risk of overloading any single expert with too many tasks.\n\n3) Empirical Validation: Experiments conducted on both synthetic and real datasets support the theoretical findings, demonstrating that MoE can effectively improve learning performance across diverse scenarios. It is interesting to note that even with a higher number of experts than tasks, the model might not perform well.", "weaknesses": "1) Validity of Proposition (4): The model gap term ∑n≠n′∥wn−wn′∥2\\sum_{n \\neq n'} \\|w_n - w_{n'}\\|^2∑n=n′​∥wn​−wn′​∥2 only considers the Euclidean distance between weights. This may not fully capture the complex relationships between tasks. In practice, tasks could overlap in non-trivial ways (e.g., in feature space or output space), and simple weight differences do not reflect true \"task divergence\" accurately.\n\n2) Limited Experiments: Though the main contribution is to present the theoretical analysis of forgetting and generalization error for MoE in CL, the main objective of the model is to reduce catastrophic forgetting. Without presenting enough empirical evaluation in terms of accuracy in continual learning settings for benchmark datasets like CIFAR10/100, ImageNet is not enough. I would encourage the authors to extend the simulation to the more complex dataset. For example, SEED [1] uses a mixture of expert networks and selects a single expert to finetune downstream tasks. If the authors could provide details of this work by forgetting a generalization error it would provide better judgment of where the current state-of-the-art MoE methods for CL stands in terms of forgetting and generalization error and strengthen the contribution of the work.\n\n[1] Rypeść, Grzegorz, et al. \"Divide and not forget: Ensemble of selectively trained experts in Continual Learning.\" arXiv preprint arXiv:2401.10191 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jcRiZ9UuEK", "reviewer": "Reviewer_nnJV", "summary": "This work studies the theoretical understanding of mixture-of-experts (MoEs) module in continual learning. To examine the role of MoEs, the authors conduct experiments on overparameterized linear regression tasks. Theoretically, this work identifies the importance of the gating network update’s termination and analyzes the catastrophic forgetting and generalization.", "questions": "(1) In Line 76-78, it sates the MoE enhances performance over the single expert case, which seems trivial. Is the condition, such as with the same model complexity, missing in the statement?\n\n(2) In Line221-224, it seems the update of model parameterizes use the inversion matrix, which brings more computational overhead over gradient-based methods.\n\n(3) In Line 293-294, it seems the early termination achieves stable convergence, but the motivation is also related to overfitting for alleviating imbalanced loads. So does the early termination improve balanced loads?\n\n(4) In Line 532, “the first theoretical analysis of MoE” is overstated considering the existing work [1]. And I am also wondering how the developed strategy works in more complicated experiments.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is written clearly and well structured. Overall, the theoretical analysis over MoEs is necessary in developing MoE-based large models, and this work discusses some insights into the catastrophic forgetting and generalization. The experiments include the overparameterized linear regression and MNIST cases.", "weaknesses": "(1) The scope of this work seems a bit wide according to the title. I suggest to use terms like “theoretical understanding” to modify.\n\n(2) The theoretical analysis is mainly on overparameterized linear regression cases, which might be a limitation in this work as nonlinear deep neural network cases can be more practical.\n\n(3) There are some related work on MoE theories, continual learning with MoEs or MoEs for adaptation in the field that require discussions [1-4]. \n\nReference:\n\n[1] Nguyen H D, Chamroukhi F. Practical and theoretical aspects of mixture‐of‐experts modeling: An overview[J]. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2018, 8(4): e1246.\n\n[2] Jerfel, Ghassen, et al. \"Reconciling meta-learning and continual learning with online mixtures of tasks.\" Advances in neural information processing systems 32 (2019).\n\n[3] Wang Q, Van Hoof H. Learning expressive meta-representations with mixture of expert neural processes[J]. Advances in neural information processing systems, 2022, 35: 26242-26255.\n\n[4] Lee S, Ha J, Zhang D, et al. A neural dirichlet process mixture model for task-free continual learning[J]. arXiv preprint arXiv:2001.00689, 2020.\n\n---\n\n***Post Rebuttal**\n\nAfter reading the rebuttal, all my questions are well answered. I updated the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)