Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
Authors: Yehya Farhat
Hamza ElMokhtar Shili
Fangshuo Liao
Chen Dun
Mirian Del Carmen Hipolito Garcia
Guoqing Zheng
Ahmed Hassan Awadallah
Robert Sim
Dimitrios Dimitriadis
Anastasios Kyrillidis
Venue: NeurIPS 2025 poster ()
Keywords: Mixture-of-Expert
Decentralized Training
Primary area: general_machine_learning

Abstract: Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components.
Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions.
As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing *Dynamically Decentralized Orchestration of MoEs* or *DDOME*. *DDOME* leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, *DDOME* achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. 
We empirically validate *DDOME* within a Federated Learning (FL) context: *DDOME* attains from 4\% up to an 24\% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings
Yehya Farhat* f
Hamza ElMokhtar Shili*
Fangshuo Liao*t
Chen Dun
Mirian Del Carmen Hipolito Garcia*
Guoqing Zheng
Ahmed Hassan Awadallah
Robert Sim
Dimitrios Dimitriadis
Anastasios Kyrillidis
Department of Computer Science, Rice University Microsoft
Abstract
Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allo- cate domain-specifer expertise across multiple underlying data distributions. As an outcome our framework, we develop an instance tailored specifically decen- tralized training scenarios, introducing Dynamically Decentralized Orchestration of MoEs or DDOME. DDOME leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically, By inte- grating a pretrained common expert to inform a gating function, DDOME achieves personalized expert subset selection on-the-fly, facilitating just-in-time personal- ization. We empirically validate DDOME within a Federated Learning (FL) context: DDOME attains from 4% up 24% accuracy improvement
state-of the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.
Introduction
5 18 it is now widely accepted design philosophy that "the larger (model/dataset), the better". Yet, the computational and economic costs associated with training and deploying large monolithic models raise concerns: Are we wisely allocating resources by training single, monolithic model ruther than employing specialized submodels that operate efficiently and adaptively?
Due to the success of large-scale deep learning 28 3 15
Mixture of experts (or MoEs) [10 12
represent a well-known model architecture that embodies this principle. Unlike traditional ensemble methods [7]. MoEs train expert submodels jointly, using
Equal contribution.
farhat, fyehya.farhat, hamza. shili, fangshuo _laa, cd46, anastasios)
[mirianh, aaadal.. rsim) @ microsofi.com
Work done while at Microsoft. [gzheng] @ perceepta.a, (dbdim) @amazon.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

gating function to selectively activate subsets of experts based on input data. Recent successful instances of MoEs demonstrate impressive scalability benefits by leveraging sparse activation of network layers 30 5
18
27
25
32
Despite these successes, achieving semantic specialization -where experts learn distinct functions- remains challenging in single-domain tasks, where data does not naturally partition into subtasks or distinct distributions. Recent findings even suggest that sophisticated gating strategies might offer limited benefit compared to random gating functions in such single-domain contexts [36., indicating that explicit specialization does not emerge without additional mechanisms or constraints. Existing methods [19 [ addressing meaningful specialization primarily focus on distinct task boundaries and separate datasets per expert to facilitate explicit expert specialization. This observation leads to a fundamental question: How can we effectively foster meaningful specialization within MoEs in single- single-domain scenarios, where clear task boundaries are not naturally available?
Understanding this question has broad implications for real-world settings where data exhibits subtle variations rather than clear domain boundaries. For instance, consider pretraining a model predict the type and stage of cancer from CT scans. In this case, cancer prediction is the shared task, while the clients-differees
hospitals or clinics-each possess private datasets reflecting their own data distributions (e.g., one may specialize in prostate cancer, another in breast cancer). Similar situations arise personalized content recommendation language modeling across dialects, and client-specific adaptation in decentralized environments. Studying this phenomenon can lead to strategies for dynamically allocating computational resources, thereby improving both model efficiency and overall performance.
Main hypothesis and our contributions. We explore how joint training of gating mechanisms and experts within MoE frameworks can enable adaptive specialization under single-domain settings. We hypothesize that MoEs can leverage implicit data heterogeneity to achieve expert specialization, guided by a dynamic gating function that learns concurrently with experts.
As an instance, we introduce decentralized MoE framework designed to leverage data characteristies across different nodes for personalized expert specialization. We dub this system Dynamically Decentralized Orchestration of MoEs DDOME, distributed MoE system tailored for decentralized learning scenarios (Figure D. DDOME maintains a collection of independent expert models (we consider image and text classification tasks in this work), adaptively selected by a gating function influenced by shared representations from a pretrained common expert. This design allows DDOME to dynamically specialize subsets of experts across heterogeneous data distributions without explicit task annotations. Some of our findings include:
We theoretically show decoupling the training of the gating and expert modules leads to suboptimal specialization, highlighting the necessity of joint training for effective expert allocation.
DDOME effectively leverages implicit client-specific data characteristics to dynamically specialize experts during joint training, without the need for explicit task definitions.
DDOME can dynamically select experts and achieve just-in-time personalization unseen clients during testing. DDOME accurately classify unseen data with small adaptations.
We achieve these reducing the overall communication cost by not sending the whole MoE module
to all clients, compared to sate-of-the-an methods [29].
Some highlights of DDOME in practice: Within Federated Learning (FL) setup, DDOME achieves
95% accuracy on FL CIFARIO, 78% accuracy on FL CIFAR100, and 75% on FL Yahoo! Answers text classification as just-in-time personalization method on unseen clients, where the second best SOTA method achieves ~ 71% ~ 74%, and 69% respectively.
2 Background
Notation.
Vectors and matrices are represented with bold font (e.g., x), while scalars by plain font (e.g., or S). Capital letters distinguish matrices from vectors (e.g., W vs w). Calligraphic uppercase letters denote sets (e.g.. D); the cardinality of 'D is represented as |D]. [N] is [N] = N1...N).
Problem formulation. Let S be the total number of training clients. Each client has its own local data, denoted as D.. We will assume that D.- {xx, yi IP-l, where x; is the i-th input sample and yi its corresponding label in a supervised setting. Abstractly, let W denote the collection of trainable model parameters. The goal is to find values for W that achieve good accuracy on all data D = U.D.. by minimizing the following optimization objective:
2

---

W E arg min ((W) = Σ Ee(WDa))
where (W, D.) P. E EE..NED. L (W, 9:1). Here, with slight abuse of notation, (W,D, denotes the local loss function for user s, associated with a local model W, (not in- dicated above), that gets aggregated with the models of other users. W, a could be a full copy of the global model at the current training round or a selected submodel out the global one, randomly chosen based on the client's characteristics.
desired that the trained global model W W applied to unseen test clients that come with different non-i.i.d local data. Previous approaches handling a similar scenario 33 31 assume have access part of the new client's labeled local data and fine-tune W. We consider this limitation since new users are likely unwillinggunable to provide accurate labeled data and might not have sufficient resources to contribute a fine-tuning phase of the whole model.
3 The Necessity of Joint Router-Expert Training
To motivate our system, we first examine whether joint training of a gating function and experts is necessary for such specialization to emerge. This question is nontrivial, as it remains unclear whether decoupled training
support effective expert allocation in the absence of explicit task boundaries. We provide a theoretical justification for this necessity by studying a simplified scenario with top-1 routing and linear experts on Gaussian input data.
Formally, consider a ground-truth orthonormal list vi...., V E Rd.. This partitions Rd into m subsets, denoted by:
C₁ = X c==(XERR
X 2 max x v Vj E [m].
EE|ma
We consider learning a function f that maps the input space Rd to labels in IR that depends on which region Cy input data x is sampled from. In particular, consider there exists another orthonormal list wi....; w that connects the input data x with the output data E RR by:
y - I x wj: x N (0. Va JaIn)
Our goal is to learn an MоE model parameterized by 0 = {(v,, w, w where {y are the parameters of the gating function, and w, is the parameter of the jth expert:
m
f x) = Σ
x
max vg x w x. FE(m)
In particular, given an input vector x, we define the output of the jthh expert as w/Xx, x, and the vector Vx [vIxx...,, vTxx] the gating output. Therefore, the indicator function I [VV]x 2 max Ve x) can be seen as the top-1 routing based on the gating function's 's output. We formulate the training of the MoE model as minimizing the following MSE loss:
L (0) = 2 ((xx)) [f (0,x) - w)]
2 EE (x..)
Σ
I
vJ
x ) (w] x - y)² max v. x EE [m]
Note there exists 0* such that L (00) = O. by defining 0- = {v]] wii,:w) }- To this end, the theorem below characterizes the limitation of the disjoint training of experts and gating parameters. Theorem 1. Assume that m S Vd. Assume the gating parameters are initialized according to Y,00 N' (I,) independently for all j E [m]. Let i be the parameter obtained by training the experts first, and then the gating function. Formally, let Ô = {(W,, v,)1 be defined as:
m
(w,)m arg min 1 (X..) [w))
1,0 > max Ve,x (w] x w)
(I)
RE(m)
arg min
1 (x.w) 2 bm
v.
max vi x (w: x y)² [E(m)
(2)
3

---

Then with probability at least 1 exp (-0 (m²)). we have that E (vv,,)
[1 w3 [1|] 2 (1)
for all j.J E [[ml, and that E Byyy,0)5= [< (0) 2 2().
Remarks: Here, Tp describes the process of training the experts' parameters {W,)}=1> )=1= while fixing
the gating's ss weights initialization {V3,0}ī- 1- After the experts parameters learned, (2) trains the gating's parameters, while fixing the experts' weights at { { learned in dT.. What Theorem 1
demonstrates is that training experts first followed by the training of gating function with the trained
expert weights frozen incurs test loss that is lower bounded by constant factor. Note that w; has unit norm; then, by the standard concentration property of Gaussian random vectors, almost all labels y should have constant magnitude. Therefore, even a trivial choice of the parameters by choosing all vj and w to be O will incur a constant loss. This means that the joint training of the expert and the gating's 's parameters will not improve upon the trivial choice of the parameters by more than constant factor, motivating the need to perform joint training of the two parts. The proof of Theorem -with all the details of the assumptions made- provided in Appendix [
Guided by this result, we now turn to the design of our proposed system, DDOME. DDOME operational- izes this principle in decentralized learning setup and leverages data heterogeneity across different nodes to achieve expert specialization, even within a single domain.
4 Overview of DDOME
System components. Our system is depicted in Figure II with components grouped across the server (using purple boxes) and client (using cyan boxes).
Server-side. Parts (a), (b), (f), (h) in Figure 1 The server maintains two key modules: i) pool of M experts (MoE module), each initialized with the same architecture (e.g., TinyBert); and di) a gating function that ranks experts based on data characteristics. The experts can be randomly initialized or be pretrained; their weights are denoted as W., for i E [M]. The gating function is a small MLP with parameters W, that outputs relevance score for each expert based input representations.
Client-side. Parts (c), (d), (e), (g) in Figure Each client has access to the same frozen, pretrained common expert that serves local feature extractor. This common expert transforms raw inputs into embeddings, which are further fed into server's s gating function. Note that the common expert is not retrained during gating function is a sparse-enforced selection of experts. The selected experts, say experts and j. dispatched the client to be locally trained. Each client jointly updates the assigned experts' parameters denoted as Wise., with slight abuse of notation- along with the gating function W,. Finally, the updated parameters Wige, ig,, and W, are sent back to the server to be aggregated with other updates coming from other participating clients during the training round.
procedure but used as an embedding mechanism. The result of the
The gating function. The gating function consists of two components: i) a pretrained common model that acts as feature extractor, converting each client's local data samples into embeddings. By design, our gating function should be model agnostic to the pretrained common expert. This model is treated as a fixed, black-box encoder and does not require further training. éi) an expert-ranking network, which takes the extracted embeddings as input and outputs relevance score for each expert. This network is updated locally by each participating client in that training round using its own data. The expert-ranker ranking network aggregates scores across local data samples and selects the most relevant experts to be sent to each active client. This sparse-enforced, per-round selection ensures that each client only interacts with a targeted subset of experts, promoting both efficiency and specialization.
Expert-Cliie Client relationship. To encourage early and stable specialization, we introduce client activation strategy, called anchor clients. This strategy serves two key purposes: i) it guides experts toward meaningful initial specialization, and ii) it helps the gating function better characterize each expert's behavior. Specifically, given M experts in the system, larger S > M) to act as anchor clients. Each anchor client is persistently aligned with a specific expert in one-to-one fashion and is activated more frequently than regular clients. During training, these clients are optimized with an independent loss. This design encourages consistent, distinctive expert behaviors; this strategy improves convergence stability and expert diversity (Section 5)
pre-select M clients (out of a much

---

Server
MoE Ensemble
W.
W:
Exprrt
Espert
Expert
Gating Function W
Select appropiate experts based on C₁
Aggregate
[Wr r. W Wee.
W.) = te e.
from active clients
(b)
Expert/
Axper/
(h)
W,
Client
W icc.
(W,, W Wien!
Train s/W. Wan,
g W,
Tork Σ stt,, w.)
Send updates of (W., Won.)
Common Expert
(c)
Figure I: For each client: i) The server uses the gating function to select a subset of experts based on the local data distribution (parts (a), (b), (d), (e)); it) The client updates expert and gating function's weights (part (g)) and sends these back to the server. iit) the server aggregates and update the new weights (part (h)). The above are repeated for all FL rounds.
Module details
Pretraining, Each client utilizes a pretrained common expert with parameters W. E.g., such an expert could be a pretrained model on ImageNet for image classification purposes. We restrict our methodology such that: i) we ensure our algorithm is agnostic to both the common expert's architec- ture and performance; ii) we assume only access to the common expert's 's embedding capabilities; and tii) we do not modify/retrain the common expert. The common expert is sent to/downloaded by all clients only once, before training. For client s, we perform one-time inference on all local data using the common expert and store the corresponding output features, noted as X for each X; E D..
The set of expert models. Our methodology involves M experts, each being independent model of the same architecture For the i-th expert, i E [M] we denote its parameters as W; and the corresponding model function f(. W.)). See also Figure [a). We consider two cases in experiments for completeness: The M experts are randomly initialized in our image classification experiments to provide full plasticity during training, while for text classification the M experts are initialized using weights transferred from pretrained TinyBERT model. In each round, different subsets of experts are selected to be communicated and updated by active clients based on their local data (see Figure ID). Per round, the updated experts are sent back to the server to be aggregated before the next round starts; see Figure [[)).
The gating function mechanics. We randomly initialize an expert-ranking network with parameters W,. This is a small-scale, two-layer MLP that, for each active client, takes embeddings from the common expert and predicts a relevance score over all M experts. Specifically, for client 8, we denote the score as ( W,) R-M, for the i-th data sample, based on:
g ( W+) Softmax(MLP(Xf W,)) R.
The final decision on the top-K experts is made via the rule:
e, ToPK Σ Eg(x W.)).
over all embedded local data samples x i E [[Da]] where the TopK(.) function selects the dominating experts, based the current state of W, and {x;),
The "anchor clients" mechanism. Given M experts, we designate a subset of M clients, with roughly distinct local data distributions, as anchor clients. [ All remaining clients are referred to as normal clients. Each anchor client is pre-assigned
unique expert in a one-to-one relationship.
"This choice made for simplicity. Diverse architectures per expert are left for future work.
'The anchor client selection process is detailed in Appendix A.
5

---

Algorithm DDOME
Parameters: T' rounds, S training clients, U test- ing clients, M experts, local iterations, experts' function and parameters f(;, W.), gating function's function and parameters g(: Wr), common ex- pert's parameters W..
for I,..., No do
Select subset of experts e, for client s;
e, TopK (Pxl o W.));
Send experts W. 6, E e, client 8;
for = 1,..., do
az awww. W.
Wiee, BOWW.,iee, P.).
Pretraining
W: W
Send W. to all clients; // Data embedding
BC W...W.
W W.en, W ign,
Wice,
for .,..., S do
end for
x = fxt, W.) end for
end for
Send to server for aggregation
W, Aggregate (W;, W Vq,
Training
W WereegateWWigz, Wiee,), Vg,8;
for O, ...,,........................... T 1 do
end for
Activate N. anchor and No normal clients;
Send g(- W,) to all activated clients; for q 1,..,, .N do
Testing
for = 1...., do
Send expert Wi, to client q;
Send g(: W,) and common expert, Wa:
eu (T lPul g(f(x", W.), W,));
for 1 1, ....,,, E1 do
W W ac D.),
Send experts W J.j e to client |/ Perform inference
j' = maxjee, [x("(x", W.), W.).;
Wi, = Wi,
Wig
9. = f(xx", ,
end for end for
end for
and we denote the index of the expert assigned to the g-th anchor client as ll.. At the beginning of each communication round, subset of N clients are activated and selected participate, where N < S and S is the total number of clients in the system. We divide the N active clients into two groups: No anchor clients and Ne normal clients, such that N Na + NN. Anchor clients Nu are sampled from the set of M anchor clients, and normal clients No from the remaining S - M clients |
The idea is that, since M < (S M), we more frequently sample the anchor clients. This frequent activation, coupled with fixed expert assignments, encourages each expert to specialize to the data distribution of its associated anchor client. In other words, experts consistently trained on similar data distributions, fostering more stable and distinct specialization over time.
The training process. For both normal and anchor clients, the server sends the current copy of parameters of the gating function, W,. The gating function selects a subset of experts; the output e, abstractly contains the set of chosen experts W, for client where E eg. The server receives e, and sends the parameters W., for E e., to the corresponding client s; this routine reduces the communication cost as compared to existing methods 2 and encourages expert specialization. We provide a theoretical analysis of the communication cost in Appendix K
Per training round, each normal client, using the standard cross-entropy loss, will locally update both W, and W,'s. 's. Formally, this amounts to (see also Figure] part (g)):
L (WW,, W Wiee. D.) : 1 E <E W,), f(x,, W.), y/
(XD,
On the other hand, for an anchor client q, we only send the II, expert to encourage expert specialization. Such an expert is trained regularly on the anchor's local distribution. Accordingly, we encourage the expert ranker network to recognize such rough specialization of the selected expert by using simple independent loss. The two loss functions for anchor clients are as follows:
C (Wi, Da)
1. Σ (f(x,, Wi,).,), L(W (W), P
1
Σ Eg(Rs, w.).
D.
where 11, is the one-hot encoding indicating Ml.
The anchor-to tormoaal sampling ratio is discussed in Section

---

After all clients finish the local training round, the server applies a simple aggregation step to average the updated copies of W, and W; ss. Adapting the MoE loss to this setting is non-trivial. After expert selection, each client only observes and trains small subset of experts (K & M), posing major challenge for the gating function: if it selects "incorrect" experts, may not only harm performance but also interfere with the specialization process by misaligning the experts. Despite this, we find that the gating function is able to learn effective expert assignments.
Test-time generalization to unseen clients. We are given new clients with unseen local data distributions during testing. We only send K experts to each test client, and we cannot get access to local test data labels to perform fine-tuning. We first send W, to the test client and select the top- experts, according to aggregated expert ranking score. for each test sample, instead of using the weighted average of the output of all selected experts, we use the output of the expert with the highest expert ranking score to fully utilize the specialization of the expert. Le.. both experts might be utilized for different data samples instead of averaging their performance. See Algorithm [
5 Experiments
The learning scenario we consider. We focus our experiments on supervised image and text classification learning tasks within a Federated Learning (FI.) setup [23 21 13,, as FL. offers a practical and representative environment for evaluating our system. We use the CIFAR data suite and EMNIST [4] for image tasks, and the Yahoo! Answers dataset [35] for text. Following common FL practice, we partition data by class to transform the full datasets into non-i.id. subsets. We assume the FL server-client protocol, where clients participate in training using local data; we assume there are 100 clients while we can only activate 10% clients per round. However, our system deviates from traditional FL implementations [23] : in those, one assumes a sole global model that is being shared with active clients, and updates to this model are being aggregated by the server per synchronization round. E.g., in image classification scenarios a large, ResNet-type network -like ResNer34, ResNet101, ResNet200- could be used Ω.. For system, the "global" model consists of multiple independent models (experts), all sharing the same architecture. 'he pretrained common expert, architecturally identical to the task-specific experts, supports on-the-fly expert selection. In our experiments, we assume between 5 and 10 experts per deployment.
17 8
Task and model description. For the image classification task, we use ResNet-34 as the expert model architecture [9] for the federated CIFAR experiments, and a two-layer MLP for the federated EMNIST experiments. For text classification,
use TinyBERT [II]. In all cases, the gating function is implemented as a two-layer MLP followed by a softmax layer, which outputs relevance scores over all experts. For the image classification, clients are trained using the SGD optimizer with momentum (SGDM), with a learning rate of 0.01, momentum of 0.9, batch size of 256, and one local epoch per round. For text classification, all client models are trained using SGDM with identical hyperparameters. All clients use a batch size of 16, and one local epoch per round. The gating function is trained using an SGD optimizer; for image classification, this uses fixed learning rate of 0,001, while for text classification the same initial learning rate is used but it incorporates a step decay learning rate schedule. The model aggregation on the server side is performed with FedAvg 24
System. Experiments were conducted on different hardware setups. For image classification tasks, we used an NVIDIA RTX A6000 GPU with 46GB of VRAM. Training the default configuration with 10 experts required approximatell 6 hours. For text classification tasks, we used an NVIDIA A40 GPU with 48GB of VRAM, and training the default configuration with 5 experts required over 100 hours (using 2 workers) due to the increased computational cost. Training was performed in a distributed fashion using between 2 and 10 workers.
Dataset, We conduct experiments on CIFARIO, CIFARIO0 17], EMNIST 4.. and Yahoo! An- swers (35. For EMNIST, we use the "ByClass" split, which contains 814,255 character images across 62 unbalanced classes. For Yahoo! Answers, we randomly sample 50,000 training and 10,000 test examples to create dataset of the same size CIFAR, with an equal number of samples per 81 To increase task difficulty, we exclude the answer text and train models solely on the question title and content. For all cases, the training dataset is randomly partitioned across 100 clients. We followed the same procedure for the anchor clients but avoided replacement, aiming to preserve the label diversity in each subset. We establish one-to-one mapping between these clients and the experts, corresponding to each group of labels. This path allows to i) have one expert available for
class
Fuull dataset training computationally expensive, given the hardware configuration had.
7

---

each group; and ii) retain the flexibility to activate the anchor clients during the training rounds. The complete client distribution for all datasets is detailed in Appendix A
Zero-Shot Personalization. Let us first descrihe the baselines to compare against
FedMix [29] trains an ensemble of models adapted to the data space's sub-regions. By definition,
FedMix sends all the experts to each client to specialize them, heavily increasing communication costs. For this implementation, we initialized the common expert from the initial pretrained model checkpoint, and we used it embed local data in the gate function and help the routing.
FedAvg 24 is the de facto approach for FL and allows fair comparison regarding fixed commu- nication cost. Here, we initialize the global model with the initial common expert checkpoint and aggregate the updates from all sampled clients per iteration.
FedProx [2 tackles heterogeneity by introducing regularization term that limits the distance between local/global models at the cost of additional computation overhead per round. We follow the same strategy for initialization with FedAvg.
Scaffold [14] handles non non-iidness by applying control variates for the server and clients at the expense of doubling communication cost per round compared FedAvg. This method tends to become unstable during training, as previous studies have shown 20 We follow the same strategy for initialization with FedAvg.
The Average Ensembles [16] train two models (initialized from the common expert) as in FedAvg, but with different random seeds. It then combines them by averaging output probabilities. While provides flexibility with respect to resources, it has higher inference costs.
CIFAR 10
CIFAR 100
Method
# Clients
Rounds
M
K
Acc.
Acc.
Rounds
M
K
Acc.
Acc.
Common Expert
73%
93%
67%
73%
FedMix 29
100
1250
2
2
31.3%
42.9%
2000
2
2
49.7%
48.3%
FedAvg
24
100
1250
31.2%
58.4%
2000
72.9%
74.0%
FedProx
22
100
1250
72.7%
71.4%
2000
72.8%
74.0%
Avg Ensemb embles 16
100
23.9%
53.7%
72.8%
74.1%
DDOME
100
1250
5
2
91.8%
95.7%
2000
10
2
75.7%
78.6%
Table Average zero-shot personalization score for unseen test clients on C1FAR10/100 See subfigures (d) and (b) of Figure Siin Appendix[C] (C statistical significance of the respective results. We use two different pretrained common experts as feature extractor for each dataset: a) The lower bound model at which the gating function can outperform the initial common expert accuracy, illustrated in Figure & in Appendix; b) The average model that represents good accuracy that relatively casy to achieve using ResNet-34 architecture. Sampling is performed under the scheme of N. = 5 anchor and N = 5 normal clients per training round; here, N = No + No = 10. See Appendix
E for DDOME gating function's 's effectiveness on individual samples.
Tables 3 summaaize our findings on this setup. Whereas FedMix requires all experts to be transmit-
ted to each client, i.e., M = K, DDOME allows the selection of K experts, here K = 2, without the need to send them all. This reduces communication costs and ensures the client receives the most pertinent information from the relevant experts.
In terms of baselines, we observe that for the CIFAR datasets both behave differently. We attribute this gap to the number of classes each client holds. In the CIFAR10 10 scenario, each client has fewer classes, which can amplify the model drift problem in all baselines. Furthermore, FedAvg's 's performance deteriorates sharply when we test it on the new CIFAR10 clients that were not used for training due to the heterogeneous data distribution during training and then in the testing phase. Similarly, Average Ensembles faces performance ceiling, the ensembles inherit the limitations of the FedAvg aggregation method. On the other hand, FedProx can surpass the initial performance of the common expert for the CIFARI00 100 scenario but degrades quickly when using few labels per client, as in the CIFAR 10 setup. To the best of our ability, we attempted multiple hyperparameter settings for Scaffold, yet we were unable to produce useful model under this distribution; it became unstable during training (achieving only 10% for CIFAR1O / <5% for CIFAR100), Further comparison against domain adaptation methods, as in FedADG 34 and FedSR 26). is shown in Appendix [] for the cases we consider, we observe that current implementations are bound to having a small number of clients to perform competitively. These trends are not limited to vision tasks.
NNot that we are aware that there are dozens more generic FL algorithms to compare against; yet, our aim is provide proof-of-concee for methodology training and selecting the right experts in such setting.
8

---

Yahoo! Answers
On the Yahoo! Answers dataset (Table 2). we observe similar pattern. All baselines struggle to consistently outperform the pretrained common expert, with FedAvg again showing limited generalization. However, unlike CIFARIO, the severity of model drift is reduced-likes likely due to clients having access to more di- verse text samples per class. De- spite this, the limitations FedAvg and Average Ensembles persist. As in CIFARI00, FedProx slightly outperforms the common expert in one configuration, but fails to do consistently.
Method
# Clients
Rounds
M
K
Acc.
Acc.
Common Expert
61% %
69%
FedMix
100
2000
58 08.08%
59.46%
FedAvg
100
2000
60.75%
60.18%
FedProx
100
2000
60.15%
61.18%
Avg Ensembles
100
62.42%
69.36%
DDOME
100
2000
5 2
64.55%
74.90% %
Table 2: Average zero-shot personalization score for unseen test clients on Yahoo! Answers. The structure of the table follows that Table
For data diversity, we report results on the EMNIST dataset; please refer to Table for more information. We note that we have also considered lower than 73% accuracy for the common expert (e.g., 67%). Yet, such an initial performance was too low to improve further using any of the methods in comparison. This led to the inclusion of the 73% and 80% cases. This highlights the importance of the common expert in our framework, underlying that our methodology does not "magically" work for all cases. Still, proper preparation is needed to obtain favorable performance.
The global accuracy reported at the end of training demonstrates the effectiveness and consistency of DDOME in all datasets, with significantly better performance than other algorithms. Please re- fer to Appendix [ for a detailed end-to-end performance of the methods Table under differ- ent clients' distribution.
EMNIST
Method
Clients
Rounds
M
K
Acc.
Acc.
Common Expert
73%
80%
FedMix FedAvg FedProx
100
2000
9.4%
15.9%
100
2000
72. 1%
72.1%
100
2000
72.0 %
72.0%
Avg Ensembles
100
74. 54%
74.4%
DDOME
100
2000
10
2
79.9%
80.5%
Table 3: Average zero-shot personalization score for unseen clients on EMNIST: The structure the table follows that Table See Figure 6 in Appendix D for statistical significance of the respective results.
Additional ablation studies and experiments. Appendix con- tains thorough ablation studies
on the initial conditions of the common expert and how it boosts the performance, and the value of anchor clients and their ratio with normal clients. Appendix [E contains assessment of the DDOME gating function's effectiveness on individual samples. Appendix considers the incremental learning scenario, where either the pool of clients dynamically increases over time or changes over time. Appendix H considers the where M K and compares FedMix versus DDOME. Appendix I compares DDOME against domain generalization methods. Finally, Appendix discusses how DDOME differ from other clustering- tussed based methods applied similar scenarios.
6 Broader impacts and limitations
Broader impacts. Our work advances efficient and adaptive specialization of coxxuur-oof-Expeeee models, enabling personalized ML in decentralized environments. Positive impacts include privacy- preserving FL and sesource-efficiens personalized applications, reducing computational and com- munication costs. However, we acknowledge potential risks, such exacerbating fairness issues expert specialization amplifies biases inherent in heterogeneous data. Future deployment should carefully monitor and mitigate such risks.
Limitations. Despite the practical and theoretical promise, our study exhibits several limitations that could inform future research directions:
Dependence pretrained common experts: The success of approach relies heavily the availability of an effective pretrained common expert (see Appendix
Communication overhead: While DDOME reduces communication costs relative to sending all experts to clients, it still incurs higher communication overhead compared traditional single- model methods.

---

Expert inirialization and diversity: Our experiments indicate that expert diversity at initialization significantly impacts specialization effectiveness. We primarily evaluated homogeneous expert architectures; further study is necessary to understand the impacts of architectural heterogeneity.
Limited scalability testing: Current experimental setup tests to a moderate number of clients. Complexity of gating mechanism luning: Finding optimal gating configurations might become computationally expensive scale and diversity increase.
7 Conclusions
In this work, we investigated how joint training of gating mechanisms and experts can enable adaptive specialization in Mixture-of Experts (MoE) frameworks under single-domain heterogeneous data settings. We introduced DDOME (Dynamically Decentralized Orchestration of MoEs), a distributed MoF: architecture specifically tailored for decentralized training scenarios.
Our empirical evaluations across various datasets demonstrate that DDOME achieves performance, surpassing existing FL methods by leveraging implicit data heterogeneity. Comple- menting empirical findings, we provided a rigorous theoretical justification for the necessity of joint training between gating and experts. Our analysis highlights that disjoint sequential training of these components significantly limits achievable specialization, reinforcing the importance of coordinated parameter updates. Future work will explore architectural diversity among experts, scalability enhancements, and extensions to multi-domaa settings.
10

---

References
[1] Ruqi Bai, Saurabh Bagchi, and David I. Inouye. Benchmarking algorithms for federated domain generalization, 2023.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arrXiv:2244.02311 2022.
[4] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921 IEEE, 2017.
[5] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-52700 2022.
[6] Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5557-5576, 2022.
[7] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, (100):993-1001.1 1990.
[8] Chaoyang He, Alay Diliphhai Shah, Zhenheng Tang, Di PanIAdarshar Naiynar Sivashun- mugam, Keerti Bhogaraju, Mita Shimpi, Li Shen, Xiaowen Chu, Mahdi Soltanolkotabi, and Salman Avestimehr. FedCV: a federated learning framework for diverse computer vision tasks. arXiv preprint arXiv:2111 1066,, 2021.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991.
[11) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 19X.::1909.10311 2019.
[12] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181-21)) 1994.
[13] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning, 2019. http:::ttt//arattt wwtp://..rrrrabaa//eeeee abw/1910........... 06378.
[14] Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. 2021.
[15] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Moham- mad Shocybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. arXiv preprint arX5v:2205.05778, 2022.
[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neurul Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https: httpttps://prroceednnee www.ris.....eeeeeeeeeeeeee...eee http:////.pppp..ee////////////// papps////a///////e//e// c3996226b666663333444444666
pdf

---

[17] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[18] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[19] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arKiv:2208. 03306, 2022.
[20] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study, 2021.
[21] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2018. URL https://arxxi org/ abs/1812.0611
[22] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020.
[23] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effccii learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[24] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera
y Arcas. Communication-efficccnn learning of deep networks from decentralized data, 2023.
[25] Basil Mustafa, Carlos Riquelme. Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with LIMoE: the Lanuuage-mage mixture of experts. arXiv preprint 220.::2000002770, 2022.
[26] A. Tuan Nguyen, Philip Torr, and Ser Nam Lim. Fedsr: A simple and effec- tive domain generalization method for federated learning. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 38831-38843, Curran Associates, Inc., 2022. URL http http:///ppppccoccoooooccoocceepp /ttp://///ppppiieeeeeeeee/////// pttpp/pppp///////epeee htw.laacce..fcccccoooaaaaaaaaaea
pdf]
[27] Joan Puigcerver. Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXhv:2009.13239 2020.
[28] Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:21 2021.
[29] Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling, Federated mixture of experts, 2021.
[30] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 000.:::1701.06...0. 2017.
[31] Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint 1910v 10252, 2019.
[32] Zhao You, Shulin Feng, Dan Su, and Dong Yu. SpeechMoF: Scaling to large acoustic models with dynamic routing mixture of experts. arXiv preprint arXiv:2105.03036, 2021.
[33] Tao Yu, Fugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation. arXiv preprint 20X2:22002.04758. 2020.
12

---

[34] Liling Zhang, Xinyu Lei. Yichun Shi, Hongyu Huang, and Chao Chen. Federated learning with domain generalization, 2023.
[35] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[36] Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:2110.042660 2021.
13

---

A Clients distribution
We created federated version of the datasets by introducing two partitioning strategies to split the samples across 100 clients:
Quantity-based label imbalance: Each client holds data samples of K labels. We first randomly assign K different labels to each client. Then, per label, we randomly assign samples to clients along with labels (with replacement). This way, the number of different labels for each client is fixed. For the CIFARIO0 dataset, use K 10. For the CIFAR1O and Yahoo! datasets, use K
Anchor clients: We followed the same method to create the anchor clients, except we prevented replacement when randomly selecting the labels. This way, we created a) 5 anchor clients with K = 2 CIFARIO and Yahoo! and b) 10 anchor clients with K 10 the CIFARI00
dataset.
cetrribution-based on label imbalance: We simulated the label imbalance of each client by allocating a portion of the samples (with replacement) of each label according to the Dirichlet distribution (a 0.1). As illustrated Figure 2 the test clients are randomly unseen combinations of K labels that never appear during training.
Anchor clients: We use the same Dirichlet distribution (a = 0.1) randomly create a) 5 anchor clients on CIFARIO and b) 10 anchor clients on the CIFAR100 dataset.
Training Clients
Testing Clients (a 0.11
Loada
Unseen dients during trariing
Figure 2: Example of distribution-based label imbalance partition on CIFARIO dataset (a 0.1)
We conducted simulation of federated version of 100 clients on the EMNIST dataset, using the "ByClass" split. This split presents a greater challenge than the CIFAR 10/100 datasets, as some classes have much larger number of samples, resulting in 62 unbalanced classes. The clients were created in accordance with the sitantity-basel label imbalance approach, with K 5.
Note that for test users, we do not repeat any distribution from the training clients; this way, we create an example where the distribution of the images over all users is different.
B DDOME end-to-end performance
B.1
Quantity based strategy
We begin to evaluate the performance of our method and baselines by measuring the zero-shot personalized model accuracy on several unseen test clients with Quantity-based label imbalance distribution strategy, as explained in Appendix A The results are illustrated in Figure
In Figure 3. we can observe that FedAvg cannot keep improving once it's initialized from the pretrained checkpoint. This surprising result stems from three major issues: the clients' learning rate parameters are inconsistent with previous training, ii) the heterogeneous data distribution on the training clients introduces a high degree of model variability, and iti) the pretrained expert struggles to improve or adapt to the federated distribution. Moreover, implementing FedProx required careful fine-tuning of the j parameter to achieve good accuracy and fast convergence. On the other hand, despite trying multiple hyperparameter settings, we could not produce a useful model using the Scaffold method; became unstable during training and often collapsed or got stuck in a poor model. This suggests that our method is more robust than these basclines in the current setup.
14

---

Test Accuracy
Test Agcuracy
06-
Dosemble
84
Breemble
011
600
1000
1200
1000
1500
2000
Communication Rounds
Communication Rounds
Boselime
Fod
FedMit
DDOME
-
FedParx
Scafiold
Figure 3: DDOME CIFARIO (left) and CIFAR 100 (right) datasets, against FedMix, FedAvg and Average Ensembles based Table using initial common expert of 73% accuracy.
B.2 Distribution based strategy
Using the distribution-basee strategy detailed in Appendix A implement additional chal- lenging scenarios, where further heterogeneity and complexity are inserted via labels distribution: i) we use the Dirichlet probability rule generate skewed and imbalanced label distributions, mimick- ing real-world applications; úi) we relax the assumption of disjoint labels for the anchor clients and allow label overlap, creating a more complex scenario, given that experts are initialized from scratch.
CIFAR 10
CIFAR 100
Common Expert
73.39%
73.73%
FedAvg
51.3%
73.6%
FedProx
52.8%
73.6%
Scaffold
10.0%
01.0%
FedKix
29.8%
65.3%
DDOME
80.8%
77.8% %
Table 4: Best global test accuracies from the last ten evaluation rounds reported on different non-iid algorithms under Dirichlet distribution (a 0.1).
Table indicates DDOME leverages the common expert's original 73% accuracy to reach up to 80% accuracy,
highly skewed scenarios. While heterogeneity should decrease the overall performance, DDOME outperforms the methods under comparison, where experts learn to better generalize to unseen data.
Further, Figure f we show the test accuracy envelope curves for all the algorithms under consider- ation. It is clear that DDOME show superior performance throughout execution, often surpassing the text accuracy of the common expert, which is already sufficiently trained, and there limited space for improvement. This figure also shows the behavior of the models during training: even DDOME shows the variability of test accuracy over iterations, indicating that keeping the model at the very end of the execution might not always be the best practice.
0.8
0.8
Test Accurace
0.6
Test Acuuaayy
11.6
0.4
0.4-
0.0
500
1003
15000
500
1000
1500
2000
Communication Rounds
Communication
Rounds
BedAog
DDOME
Figure 4: Evaluation of different non-iid algorithms under Dirichlet distribution (a = 0.1) on CIFAR1O dataset.
15

---

C Performance under different sampling ratios
There is an initial degree randomness in the gating function. During the first couple of iterations, it sends random top-K experts to each client while the experts learn to specialize in the different regions of the label space. However, we found a way to keep consistency during these initial rounds: through the anchor clients. Figure each round, we can ensure a balance against the wrong selection of the gating function by letting them as regularizers. Additionally, Vigure5sshow shows the impact the performance when remove the anchor clients rule from sampling and allow only random selection from the pool of available clients. It is clear from Figure that "warming-up" phase is necessary for DDOME: using sufficient number of anchor clients, one achieves stability and better final accuracy, by warm-starting the system using more specialized clients.
that by introducing least 30% anchor clients during
b)
0.8
Test Accuracy
06-
Ensemble
Test Test Accuracy
0.6
Ensemble
Test Accuracy
Test Test Accuracy
0.8
0.6
0.
0.8-
3.4
02
0.3
02
Ereemble
0.0
0.0
2000
1000 Communication Rounds
Communication Rounds
Communication Rounds
Communication Rounds
Baseline
FedAvg
FedMix
DDOME
Figure 5: Global testing accuracy for CIFARIO0 (a-b) and CIFARIO (c-d) datasets on two different sampling strategics: a) + c): 10 random clients without replacement per itcration. b) d): 5 random anchor clients + normal clients without replacement per iteration along different methods.
D EMNIST Byclass statistical significance results
We evaluate a two-layer MLP on the federated EMNIST dataset, running each algorithm for 1000 communication rounds sistently surpasses the performance of the initial common expert, demonstrating its robustness and effectiveness across varying accuracy levels.
three different initialization seeds. Results show that DDOMe con-
EMNIST ByClass (3E)73%)
EMNIST ByClass (CE=80%)
Test Accuracy
0.8
Baseline
Test Accuracy
0.8
Baseline
FedAug
Fedfig
0.8
FedMix DDOML
DDOME
Fedirrox
Avg Ensembles
Avg Ersembles
0.2
0.2
0
200
400
600
B00
1000
200
400
600
800
1000
Communication Rounds
Communication Rounds
Figure 6: Statistical significance results for EMNIST dataset using a common expert with 73%
accuracy (left subfigure) and 80% accuracy (right subfigure).
E Gating function Per-Sample Performance
After training, we thoroughly evaluate our gating function, using the checkpoints trained with the 73% common expert on CIFAR 10 and CIFAR100 datasets on the DDOME algorithm. Our fine-grained evaluation demonstrates that our gating function can analyze the characteristies of each unseen test client's 's local sample and adaptively select subset of experts that match those characteristics. This crucial ensuring that our gating function can generalize well to new data. After selecting the top- K experts, the gating function chooses the highest score/confidence expert to predict each test
16

---

data sample. Our results, reported in Table show that our gating function can achieve high accuracy on the selection.
CIFARIO0
CIFARI0
Client
Incorrect
Correct
Error Rate
Client
Incorrect
Correct
Error Rate
0
278
722
27.8%
227
3773
5.7%
1
281
719
28.1%
122
3878
3.1%
2
263
737
26.3%
3437
14.1%
3
251
749
25.1%
103
3897
2.6%
4
261
739
26.1%
78
3922
2.0%
5
309
691
30.9%
6
260
740
26.0%
7
285
715
28.5%
8 255
25.5%
9 267
733
26.7%
Average Error Rate
27.1%
Average Error Rate
5.5%
Table 5: Evaluation per-sample level on CIFARIO and CIFARI00 datasets.
Ablation studies
Initial common expert impact. We study performance tradeoffs when utilizing different common experts for the gating function decisions. Our findings indicate that the amount of training allocated in the initial common expert has critical effect the overall performance of DDOME. For example, suppose the gating function uses a poor common expert for training. In that case, it can lead to poor performance (collapses to selecting single expert) and, therefore, unable to improve beyond the baseline.
Ablation study:
Figures 7 8 show that the breakpoint of the gating function for the CIFARI00 dataset is approximately 66% accuracy by the common expert. In Figure it becomes clear that a significant cause this breakpoint is that the experts cannot surpass the common expert's 's initial accuracy. This is attributed to the lack an adequate selection of experts, which essential for the gradient updates of each expert to be aligned with the same part of the task. Figure 7 also reveals the following: the 67% case, given a few more iterations, can match the performance of the 73% case. This suggests "phase-transition" might exist, where more effort (i.e., communication) is needed to improve beyond the common expert's performance. This implies that the performance of DDOME depends on the quality of the experts.
0.8
Test Accuracy
0.6
64
0.4
65
66
0.2
67
73
0.0
1000
2000
Communication Rounds
Figure 7: DDOME's performance CIFAR 100 dataset, using different initial accuracy for common expert (legends of the plot); the setup Table is used.
17

---

Initial Acc: 64%
Initial Acc: 65%
Initial Acc: 66%
Initial Acc: 67%
Test Accuracy
1.5.
Test Accuracy
Test Accuracy
Test Accuracy
64-
82
2000
2000
2000
200 Communication Rounds
Communication Rounds
Communication Rounds
Communication Rounds
Common
expu
expl
exp3
expl
exp5
expó
exps
expo
Figure 8: Zero-shot personalization accuracy per expert during training on CIFAR100.
Ablation study: Common expert boosts experts' performance. To test this, we initialized each expert from the common expert and continued training for 2000 rounds. In Table we observe the final score of each method. Surprisingly. for DDOME, takes a few more rounds overeome the baseline than when the experts are initialized from scratch. This is because the pretrained model is optimal for the entire dataset. successfully specialize experts, retraining the model on the specific subset of labels is necessary.
Common Expert
73.73%
FedMix
73.78%
FedAvg
73.99%
Average Ensembles
70%10%
DDOME
83.27% %
Table 6: Average zero-shot accuracy for CIFAR100 after 2000 rounds.
We also plot in Figure g the performance of each expert (denoted expX) over the communication rounds for different initial accuracies the common expert. evident that, for our setting, using common expert with accuracy below 67% We can outperform the other methods once the gating function can utilize a slightly better common expert.
Ablation study: The "anchor/normal" client ratio. The sampling scheme must be carefully studied ensure the best performance of DDOME. Each expert has distinct distribution; i.c. their local objectives only align with a particular subset of labels. Ensuring consistency in the experts updates essential prevent them from drifting away from their own "task". As mentioned, we assume we have some control over the activation of the clients during training.
0.8
Test Accuracy
0.6
Baseline
0.4
Na 3 Nc 7
Na 5 Nc 5
0.2
Random
1000 Communication Rounds
250
500
750
1250
Figure 9: DDOME, _ X_No_ means that N and N. + N.. 30% anchor/normal client ratio enough to match baseline accuracy. However, the model becomes more inconsistent by converging more slowly.
Our solution is the of anchor clients, whose primary purpose is to act as regularizers, ensuring consistency the expert updates during training. find the optimal ratio of anchor/normal clients Na, N. we conduct experiments varying this ratio; see Figure Sampling half of the clients per round as anchor quickly surpasses the baseline of the common expert and maintains high consistency in subsequent iterations. Using a lower ratio of 30% anchor clients per round also achieved similar performance, allowing some flexibility in the sampling. Contrarily, when we sampled clients randomly from the available pool (i.c., no "anchor clients"), DDOME shows difficulty improving performance,
18

---

as experts' updates become inconsistent. Appendix [ shows the end-to-end performance difference across different methods using these sampling ratios for both datasets.
G Incremental Learning
Incremental learning is paradigm that aims to update and refine existing knowledge from new data rather than discarding retraining from scratch. This benefit scenarios where data is dynamic, searce, or costly to acquire and where learning models must adapt to changing environments or tasks. We performed comprehensive comparison using the same benchmarking methods in Table [ to contrast each algorithm's 's learning process.
G.1 Dynamically increase the client's pool
For this setup, we split the CIFARI00 dataset into five groups with cenrolllagpinn labels. Each group held 20 different clients with random samples within the label range. Then, we allowed only one group of labels be trained for 200 iterations. Afterward, we increased the pool of clients with new group each 200 iterations, monitoring the global accuracy of the models over time. In Figure TT we can observe that DDOME not affected if the entire of clients not present from the outset; its gating function develops adaptively, without compromising its ability to capture the old distributions. In contrast, Fed-Mix drops its performance by approximately 4% compared to the original results in Table [
0.8
Test Accuracy
0.6
Ensemble
0.4
Baseline
0.2
FodAvg
FedMix
DDOME
0.0
500
1000
1500
2000
Communication Rounds
Figure 10: Incremental Learning scenario on CIFAR100, dynamically increasing the total pool of clients.
G.2 Dynamically switch the client's pool
We employ cyclical learning approach based on the first setup for the second scenario. Instead of simply increasing the total pool of clients, we only allow one of the five groups clients to contribute to the training process at a time. This means that every 600 iterations, we switch the pool of available clients, allowing us to see new labels and ensuring that the labels seen during the initial iterations will never be seen again during the training process. This cyclical approach allows to benefit from the data's 's diversity while ensuring that the model constantly being exposed to new information.
Figure IT illustrates that even when DDOME is approximately 2% below FedAvg at the end of the training. the former continues to improve. At the same time, the other methods begin to decline over the iterations. This is likely due to the anchor clients acting as regularizers to adjust the gradient directions during optimization, as the client pool presents more complex setup. The anchor clients provide a more stable optimization process.
19

---

0.6
Ensemble
TestAccuracy
Baseline
0.4
FodAvg
FedMix
DDOMF
0.2
0.0
0
500
1000
1500
2000
Communication Rounds
Figure 1I: Incremental learning scenario on CIFARIO0, dynamically switching the total pool of clients
H Performance under matching number of experts M K
We present additional experiments to compare FedMix vs. DDOME, using the same number of total experts. Le., MM- K to disentangle our method's ' behavior under different experts. The results are shown in Table 7 where it is evident that even if we send the complete set of experts per worker, our approach performs better. Yet, the resulting model might be less accurate than the common expert, implying that malicious "interference" exists.
M K =2
M 5
Common Expert
73.39%
73.39%
FedHix DDOME
42.76%
43.86%
60.16%
75.77%
Table 7: Best glohal test accuracy reported during training on the CIFAR10 dataset under Dirichlet distribution
(a 0.1) with fixed number of models communicated to each client. Both methods were initialized from the common expert with an initial accuracy of 73.39%.
I Comparison against Domain Generalization Methods
Our scenario can also framed as Domain Generalization problem. Thus, we evaluate DDOME against state-of-art methods, such FedSR 26 and FedADG [34), that handle robustness to distri- bution shifts on test-time. Results in Table demoonsrate that the ability of FedADG and FedSR to evaluate unseen domains tightly bound to small number of clients. Once increase the under- lying distribution (e.g., 100 different clients), these methods cannot exploit the cross-relationship among domains (1]
Common Expert
93.05%
FedSR 26
28. 28.24%
FedADG 34
41.83%
DDOME
87.86 %
Table 8: Best global test accuracy reported during training on the CIFARIO dataset using quantity-based label imbalance. We sample 10 (of 100 available) random clients during 900 iterations with replacement. All methods were initialized from the same common expert reported the Table.
J Clustering analysis
To provide more extensive comparison of our expert models, it is essential to highlight that the core idea summarize clients into several models, as many clustering relatedd works do. Clustering methods are limited to scenarios where clients are inherently grouped; all clients in the same group will have similar local data distributions, while clients across groups will share few data. Instead, we target more realistic scenario, where each client has non-iid and mixed-data distribution, making client clustering based on local distributions less meaningful. To illustrate this, we have performed an example of client clustering using K-means local class distributions as shown in Figure 12
20

---

where each dot represents one client and the annotated numbers are this client's two main data classes. The color represents the K-means clustering result. Clearly, clustering does not create meaningful groups of clients, and training individual experts in each group does not provide any specialization of experts.
1.00
6.6
0.5
0.30
6,8
8.8
0.6 5.6
0.2 9.2
6.6
4
-0.25
0.9
-0.50
0.7
077
47
7.9
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Figure 12: Clients clustering with label frequency.
K
Theoretical communication cost analysis of DDOME and FedMix
Variable definitions
M: number of expert models.
R: number of communication rounds.
S: number of clients.
N: active clients.
k: topk experts (k < M).
Pr: parameters of router.
P.: parameters of single expert.
Pa: parameters of common expert.
6: size to communicate an expert index.
K.1 Analysis of DDOME
Communication Cost Derivation
Total initial setup + cumulative round cost.
Initial Cost (Cont): This one-time, server-to-de to-client cost involves sending the common expert model to all clients S.
Cini = S.S. Pa
Per-Round Cost (Cround): For each of the R rounds, the cost is the sum of downlink (Server-to- Client) and uplink communication Slient-to-Serverry
Downlink Cost (Coown)): Router (P,) and & experts to N active clients.
Cdown N(P, k Pa)
Uplink Cost (Cup): top-k indices + updates for router + experts to N active clients.
Cup N(k- eilk-eeP...
21

---

Note: The communication here does not represent computational order.
The total cost for single round is therefore:
Cround Cdown + Cup N2P,, 2 + ke)
Total Communication Cost (Crotat): The total cost over R rounds is the sum of the initial cost and the cumulative round costs.
Crotel = Cintt + R Cround = Pc) + N2P., + 2kP. + ke)
Since the parameter sizes are much larger than index sizes (Pr, Pe > c), we can approximate the total cost as:
Ceocal x (S- + 2R N(P, kP.)
K.2 Analysis of FedMix
The FedMix algorithm communicates all experts in every round.
Communication Cost Derivation
Per-Round Cost (C (Crundd, FedMis) In each round, every active client downloads and uploads all the models.
Downlink Cost (Cdown, FedMixx: The server sends all M experts (M P.) to each of the N active clients.
Coown, FedMix = N(M Pa)
Uplink Cost (Cup, FedMix): Each of the Nactive clients sends back the updated parameters of the M experts.
Cup. FedMix
((M- Pe)
Total Communication Cost (Crotal, FeaMix) The total cost is the per-round cost multiplied by the number of rounds, R.
Ciotal, FedMix
2R N(M P.)
K.3 Limit Analysis
This analysis examines the asymptotic behavior of the communication costs under the following assumptions:
The number of active clients is fixed fraction of the total: N S/c for some constant
c 0.
The parameter size of the initial common expert is equal to that of a single expert: P. Pe.
Note: This is true in our experiments.
K.3.1 Analysis as the number of experts M + 00
DDOME:
lim Cioaa lim P() +2 N(P,+ PP) 2R N(P, kP.) M-000
M-+000
The cost is independent of M, implying that the communication cost is bounded and does not increase the number of experts grow.
FixMix:
lim Ciotal, FedMix M M
lim [2R N(M P.)] 0 A5+00
The cost is a linear function of M. As the number of experts grows, the cost grows without bound. This means the communication cost unbounded, making the algorithm very unscalable compared to DDOME.
22

---

K.3.2 Analysis as Number of Rounds R + oo
We analyze the cost as the training process becomes infinitely long.
DDOME The cost is of course unbounded but grows linearly with R. The rate of growth is given by the partial derivative with respect to R:
@Cuaal 0R
25 (P, kP.)
FedMix The rate of growth with respect to R is:
oC FedMix
25 (MP₂)
ÔR
Comparison Both costs are unbounded as Rt + oc. However, comparing their growth rates reveals that for 1 < M, DDOME grows significantly slower, making more efficient for longer training durations.
(P.+ <
(MP..
K.3.3 Analysis as Number of Clients - 00
In this case we will add the e term back in, directly depends on
DDOME The rate of growth with respect S is:
oC
R
P. + (2P, + 2kP. + ke)
as
FedMix Algorithm The rate of growth with respect to S is:
O FedMix
2R
(MPc)
0S
Comparison. The goal is to prove that FedMix rate of change is greater. We will show that this is indeed the case under certain (realistic) conditions.
We seek to demonstrate that: 2 MP. > Pe + n (2P, + 2kP. + kc)
We proceed by rearranging this inequality to find the condition under which it holds true.
2(M k))
2P. P.
P.
2(M - k)
be P.
a positive number slightly less than 2(M - k) and almost certainly greater
P.
than 1.
Thus, the inequality FedMix > DDOME holds true if:
R 2(M-k)
This condition is generally satisfied in practical scenario. For example we can plug in our own experimental values from e0 and Yahoo!
M 5 2, c= 10, R = 2000 P. > Pr.P.
In this case it is easy to see that this inequality is indeed satisfied, showing us that the rate of change of the communication cost FedMix, as a function the number of clients will remain bounded below by DDOME.
Conclusion. Across all scaling dimensions of interest- experts (M), rounds (R), and clients (S) DDOME communication is fundamentally more efficient and scalable.
L Missing Proof from Section 3
Here, we first state the assumption of the theorem in Section
23

---

Assumption 1. Let E E Raxd be a positive definite matrix. There exists a constant > 0 such that
1)
Amin (x
I x[1] z max x [
xx
Amin (2))
EE(d]
d
[ x(|| 2 maxx({]
Csc
Ex-N(0.E)
2
EE|d]
d
For the simplicity of the analysis, we define
1 (x) (x)== [xxx=
L.1 Analysis of the Initialized Router Weights
Let Vi...., Vm ~ Af (0, La). Write V = Mi...., Vml E Rd By standard concentration of covariance matrix, when m < d, we have that with probability at least 1 - cxp (d)
Vv 1m S 2v7
m
This gives that om (V)= 2 '2" For the simplicity of the analysis, we assume that this event happens. Let the singular value decomposition of V be given by V VU Lemma 1. Let .i...., u be fixed orthonormal list, and let U ju u with some m' S d. Then with probability at least exp (-0 (m²m?)) we have that
mm d
| s 2
Proof. Since v, N (0, (Ta for j [m], we have that U V, =: a, N (
Therefore
m
m
m
V
Σ
|
1
1=1
==
Therefore
E
[||
U [F]
mem'
d U'
1 E subE (e (
=
By the standard concentration inequality of Gaussian random vector, we have that
V U
mam'
Pr [|VTU'II mm' 2 mml) S 1 1-exp (-0 (m?m!?)
Pr
Thus, with probability at least
exp (-O (m2m!?) we have that
22mmn'
| S
Vd
Recall that V = Uv Ev R Therefore
V Ry
| UVU
Amin (V) U v2
1
|uT
Thus, we have that
mm!
ITU 2 d
Recall that the ground-truth router parameters vi.... v is an orthonormal list, and the concatena- tion is denoted by V*, and that Uv denotes the left singular vectors of V. By Lemmal have that
|
22
m vd
<
24

---

when dd Therefore, dim (span (Uv) + span (V*))
2m.
Otherwise, if
dim (span (Uv) + span (V*) < 2m, there exists x E span (Uv) n span (V*), which implics that there exists s' R such that Uvs V's'. This gives
|s = | = ||s|| | V < |
which contradiction. Therefore, we can let Üy _dx2 be the matrix representing an orthonormal basis of span (Uv) + span (V*) where the top m columns are Uv. Let Uv
[dd((-2m) the orthogonal complement Üv. Then we can re-write x ~ N~N (0, Ja)) as
x Üvsi + Üv82; 81 ~ N (0.2 82 ~N (0, 1 T0-
We make the following assumption:
Assumption Let RRaxd be a positive definite matrix. There exists a constant Osc > such that
Amin
Ex-
I x(1] [XXX11)
2 max x({]
Xx
Amin (Σ)
kE(d]
d
Ex
IT x(1]
max maxx EE [d]
d
Let C2m denote the lower bound in Assumption 2 with Gaussian measure 2m
Lemma 2. Let Üv Üv $1. $2 be defined above. If Assumption hold, then we have that
Ex 1 (x) 2 Ω m ():
1 md
Amin (Bs, 14 (x) 2 Ω
Proof. Write 81 = |q 4b where qi, 92 E R Then we have that qı, 92 ~ N (0, 1 Jm)
d
Moreover, by the construction of Üv we have that q2 is independent of Vi.-.ra Vm and qı U Therefore
Es, [4, (x) s1ss
Pi Eq. [L (x) bbh P. + Ea. [4 (x)] Pallag [929]] T1 P
where P1 I and P2
Recall that qi
U Then we have that x Tv, vv - v
Im
Since V has the SVD
V = U
we have that UTv, = Evr, where r, is the jth row of Rv. Thus
x V, q Evr, = Eye
Let 9ı = Evyn- Then we have that = Ev RTQ., and
E [[ = E [Rv Evqiqi EvR = Rv R
1
Moreover, indicator function can thus be written as
(x) = q e, 2 max को es }
=T [ārbi) 2 |
Thus, the original objective is re- re-written
Eg, [1 s1s = Pi Ey R RTE [[ 2 |1 qıqف] Ry E P.T
ā Ean (á 2 1
By Assumption 2 have that
( m
2 (21) md
Eq, [] [qrl)) 2 |qq H 2
Amin (Eq, [1( 2
25

---

Therefore, we have that
m (m):
1
Ex (1) 2 $2
Amin (Es, [1g (x) (x)s11]]] 2 s
md
Proof of Theorem 4 Notice that the objective in (??) is a quadratic form with Hessian Recall that x = Uvs1 + Uvs2. Thus, we can write
Ex [1, (x) xx
Ex [4) (x) xx
UvL [4 siss UF + Es, 14, Üvū
since $2 is independent of 1 (x). Based on Lemma we have that the objective in (??) is strongly convex quadratic since the two expectation terms are strietly positive. Therefore, w, has the form
W EÜv (x) Ü. + Es (x)] Üvūt) Exx M15 (x) xy]
Notice that for y have
V = Σ 1=1
I (x)x
Therefore, w, can be written as
m
w, (ÜvEn, [T, (x)s1s) s18s UT + Es [T, (x)] -1 Tv
Bx (x)) w;
1=1
= -ÜUVEs [4, (x) $15 s1s
T + E, 14, (x) Üvūt).
m
Σ ÜvlEs [4 (x) 12 (x) s1 oJ + Es, U1 (x) Li] Üvū w
Σ ÜUVEs I[ (x) Es, [1 (x) 1 (x) ÜJ + E,, 11 (xx] E,, 14, (x) 12 Üvū w/
Let w be given for any r e [m]. Then we have that
w Üvū
Therefore, the difference between w, and w is given by
Wj w: = -Üv
Σ Es, M4 (x)
Es, 14 (x) (x) UT - -ūvÜtw;
m
+ Úv Σ Es, [L (x)s Es, [1 (x) 1 STS - Ürūīw;
Due to the orthogonality between Úy and Üv, the magnitude of wj w must be lower bounded by the magnitude of its projection onto Uv. Thus
ww: )
ww, w:'l2 > Ú
Es, (L, (x)] Es, I1 (x) 12 (x) [x)]w; w/
m
Σ
Es, [L, (x) -1 Es, [1, (x) I (x) w w;
m
(Ia (a--vüt))
Σ
Ez, [L, (x)) Eg, [T, (x) 17 (x)) w -w;
26

---

Similarly, we can obtain that
T (wr, -.i)l 2
BsI [Tg (x) 33 17, (x) 17 (x) w w;
1=1
m
Bsg (x)) 1 Esz (13 (x) 13 (x) w w::
Σ
- (1.-
m
Bag [1 (x)] Esn 11, (x) 11 (x)) w3 -w:)
Σ
We further notice that
[T (x)) E,, 1 (x) 17 (x) w = Wy 1919 = Exi [x)) Bay 11/ (x) 17 (x) =}
t=1
Therefore, we have that
| will2 2 | 2 (1-1 115;
=
we have that with probability at least 1 - exp (-0(m²)) (mi)),, we have that | s
By Lemma
3m vd
Moreover, diving deeper into ||<|I,, we have
Ex [4 (x) Ex, 1 (x) 17 - 2
11 =
8=1
Σ Es [4, (x)] Egg [1, (x) 13 (x)) + EEs, [T, (x)] En, (x) 12 (x)) -1
2 Σ
Exy [1 (x) 17 (x)] + Ez, [4 (x)] Exy (x) (x)] (x)] -1)
Since Ezz [1 (x)) Exz [T (x) 1 (x)) S 1, and by Lemma 2
we have
1
Es, M4 (x)] 2 Ω
m
Therefore, we can obtain that
Fmn [T (x) E, (x) (x)) -1) -1)² 2
m
Ex [I, (x) 13 (x)) - -1)²
c
for some constant c. This gives that
||5 - Σ Rss (T, (x) 12 (x))² +
(TTF.. Fx, [T, (x) 1/ -1) s $1,, l
m
c
Exr
where
Es, [4, (xx) 13 (x)]
if
5 =
m Es, [Li (x) 13 (x) -1 if(=j
This gives
Ev (5 [0]
{ m² if(#i
1 if(=j
me
Therefore,
(1 1 (1-1) )² = =1(
Ey [|w, - w;lla] [ Bv 1 N Ev
2 111 =
+
33
mc|
27

---

Using a similar approach, we can conclude that
Ev Uv (w, wi) 20(1)
Therefore, for the test loss, we can write
C 0
Exx,y
(x( x y)
2
j=1
27
m
m
Ex,y
Σ j=1
1 (xx
w > 5'=1
1 (x) w.. x
(wy wz.) Ex [1, (x) 15. (x) 1 (w; - wz.)
2 J/
1
Σ (w, wy''' ' (xx [1 (xx 13. (x) 11) + Ess [Tg (x) Ij. (x)) ÜvūT) (w,- -w))
J,''=l
m
Σ
Es. [4, 13. (x)] (w, - w3.) T Ü.ūT (w, - -w3)
3-'==1
Üv (w, - 1
M
2/
This gives that
Ev
[ [0)
2 E (w, (w)] v [ - 2
(1)
28

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly and accurately reflect the contributions and scope of the paper, explicitly highlighting empirical improvements and theoretical insights. The stated claims match the provided experimental and theoretical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A dedicated section clearly outlines limitations, including dependency on pretrained models, communication overhead, expert initialization concerns, scalability tests, and gating complexity.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: The paper clearly states assumptions and provides the complete and detailed proof of the main theoretical result (Theorem in the Appendix, including references to relevant lemmas and explicit assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper clearly discloses architectures, hyperparameters, optimizers, dataset splits, and methodological details necessary for reproducing the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Although detailed instructions for reproducibility are provided, the paper does not explicitly mention releasing data or code publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcc/pcccccpcccccooooooop httpa///giidees/deeuubbbbiiiii semiisssonni swiniiossiiiyccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details including datasets, data splits, hyperparameters, optimizers, and procedures are clearly specified in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not explicitly include error bars, confidence intervals, or other statistical significance measures, due to computational constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper clearly describes the hardware, GPU types, memory, execution times, and the distributed training setup for both image and text classification tasks.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////aarrpp..cccoooolhhcchs
Answer: [Yes]
Justification: There is no indication of ethical concerns, and the research aligns fully with the NeurlPS Code of Ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes a dedicated "Broader Impact" section that explicitly dis- cusses positive impacts, such improved privacy-preserviee federated learning, enhanced resource efficiency, and personalized learning in decentralized environments. It also ac- knowledges potential negative impacts, highlighting the risk of exacerbating fairness issues
32

---

if biases present in heterogeneous data are amplified through expert specialization. The authors clearly recommend monitoring and mitigation strategies during deployment. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve releasing datasets pretrained models that pose high risks of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification: Although datasets and model architectures are cited, explicit mentions of licenses or terms of use for these assets are missing.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
33

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce any new datasets, models, or code assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The research does not involve crowdsourcing or human subject experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not involve human subjects.
Guidelines:
34

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The research does use LLMs as any original or core component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
35


Reviews:
[{"review_id": "KSjcsEkvQA", "reviewer": "Reviewer_7ym6", "summary": "This paper investigates how the joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. Based on this insight, the authors propose the DDOME (Dynamically Decentralized Orchestration of MoEs) framework. Experimental results demonstrate noticeable performance improvements.", "questions": "- Include more recent methods in the experimental comparisons and explicitly evaluate communication cost as part of the comparison.", "limitations": "Please refer to the weaknesses section above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Applying Mixture-of-Experts (MoE) to federated learning is a relatively novel perspective that opens up a range of interesting research questions.\n- The paper is clearly written, and both the methodology and experimental results are presented in an intuitive manner.\n\nWeaknesses\n- Although communication cost is listed as a contribution, it is not thoroughly evaluated or compared in the experiments.\n- The experimental section lacks comparisons with more recent federated learning methods that incorporate MoE, and additional baselines should be included.\n- DDOME appears to benefit from using a larger number of experts (M), which may give it an unfair advantage in comparisons.", "was_revised": true, "final_justification": "My concerns have been well addressed and I will raise my score.", "rebuttal": "We thank the reviewer for their feedback. We are encouraged that they found our application of MoE to federated learning to be a novel and promising direction, and that they appreciated the clarity of our writing and the intuitive presentation our methodology and experimental results. We are also pleased that the reviewer acknowledged DDOME as a technically sound framework that offers meaningful performance improvements across federated domains. We address the reviewer’s concerns below: \n\n1. **\"Although communication cost is listed as a contribution, it is not thoroughly evaluated or compared in the experiments.\"**\n\n      - We acknowledge that our current draft does not include an experimental evaluation of communication cost. **_However, we would like to clarify that our communication-related claim was specifically made in reference to FedMix,_** which requires all expert transmissions per client per round. In contrast, DDOME limits communication by decentralizing expert selection via the router. **_To back up our claim, we derive the communication cost of DDOME and FedMix in terms of the of parameters being transmitted_**. The cost is a function of the number of clients, communication rounds, and number of experts. We show via a simple analysis that FedMix's communication cost is bounded below by DDOMES communication cost. Proving that DDOME is indeed more efficient across all metrics of interest. Please find the analysis at the end\n\n\n  2. **\"DDOME appears to benefit from using a larger number of experts (M), which may give it an unfair advantage in comparisons\"**\n\n        - We believe that the reviewer might have some confusion about the conclusions from our experimental results. **_In all of our experiments, the number of experts communicated to and processed by each client is identical between DDOME and FedMix (Please refer to table 1, 2, and 3)_**. In fact, one of DDOME's contributions is its efficiency in managing a larger server-side expert pool without increasing this client-side cost or communication (Please refer to our communication cost analysis provided).  One of the research thrusts behind this work, and MoEs more broadly, is efficient and intelligent scaling. \n\n   3. **\"The experimental section lacks comparisons with more recent federated learning methods that incorporate MoE, and additional baselines should be included.\"**\n\n      - We thank the reviewer for this suggestion. We agree that comparing against more related methods is essential for situating our contribution, and we would like to clarify the scope, intent, and novelty of our work to address this concern thoroughly. **_A central goal of our work is to provide a proof-of-concept for how distinct expert models — selected dynamically through a lightweight router — can lead to client-specialized training in a single domain_**.  This setting has proven itself to very relevant in decentralized training, specifically FL. As an outcome of this study we developed a state of the art decentralized MoE framework that is competitive with our FL baselines. We agree that additional baselines might provide more support to the study, but we believe that the experimental results provided in the paper already lead to a meaningful conclusion.\n\n\n\n**Communication Cost Analysis**\n\n**_Variable Definitions_**\n\n$M$: number of expert models.\n\n$R$: number of communication rounds.\n\n$S$: number of clients.\n\n$N$: active clients.\n\n$k$: topk experts ($k \\le M$).\n\n$P_r$: parameters of router.\n\n$P_e$: parameters of a single expert.\n\n$P_c$: parameters of common expert.\n\n$\\epsilon$: size to communicate an expert index.\n\n**_Analysis of DDOME_**\n\n**Communication Cost Derivation**  \nTotal cost = initial setup + cumulative round cost.\n\n- **Initial Cost ($C_{init}$):**  \nThis one-time, server-to-client cost involves sending the common expert model to all clients $S$. \n\\begin{equation}\nC_{\\text{init}} = S \\cdot P_c\n\\end{equation}\n\n**Per-Round Cost ($C_{round}$):** \nFor each of the $R$ rounds, the cost is the sum of downlink (Server-to-Client) and uplink communication (Client-to-Server).\n\n- **Downlink Cost ($C_{down}$):** Router ($P_r$) and $k$ experts to  $N$ active clients.  \n\\begin{equation}\nC_{down} = N(P_r + k \\cdot P_e)\n\\end{equation}\n\n- **Uplink Cost ($C_{up}$):**  top-$k$ indices + updates for router + $k$ experts to $N$ active clients.\n\\begin{equation}\nC_{up} = N (k \\cdot \\epsilon + P_r + k \\cdot P_e)\n\\end{equation}\n\n**Note:** The communication here does not represent computational order.\n\nThe total cost for a single round is therefore:  \n\\begin{align*}\nC_{\\text{round}} = C_{\\text{down}} + C_{\\text{up}} \\\\\n= N (2 P_r + 2 k P_e + k \\epsilon)\n\\end{align*}\n\n**Total Communication Cost ($C_{total}$)**  \nThe total cost over $R$ rounds is the sum of the initial cost and the cumulative round costs.  \n\\begin{align*}\nC_{total} &= C_{init} + R \\cdot C_{round} \\\\\n= (S \\cdot P_c) + R \\cdot N (2 P_r + 2 k P_e + k \\epsilon)\n\\end{align*}\n\nSince the parameter sizes are much larger than index sizes ($P_r, P_e \\gg \\epsilon$), we can approximate the total cost as:  \n\\begin{equation}\nC_{\\text{total}} \\approx (S \\cdot P_c) + 2 R \\cdot N (P_r + k P_e)\n\\end{equation}\n\n**_Analysis of FedMix_**  \n\nThe FedMix algorithm communicates all experts and the router in every round.  \n**Communication Cost Derivation**\n\n**Per-Round Cost ($C_{\\text{round, FedMix}}$)**  \nIn each round, every active client downloads and uploads all the models.\n\n- **Downlink Cost ($C_{\\text{down, FedMix}}$):** The server sends all $M$ experts ($M \\cdot P_e$) to each of the $N$ active clients.  \n\\begin{equation}\nC_{\\text{down, FedMix}} = N (M \\cdot P_e)\n\\end{equation}\n\n- **Uplink Cost ($C_{\\text{up, FedMix}}$):** Each of the $N_{\\text{active}}$ clients sends back the updated parameters of the $M$ experts.  \n\\begin{equation}\nC_{\\text{up, FedMix}} = N (M \\cdot P_e)\n\\end{equation}\n\n**Total Communication Cost ($C_{\\text{total, FedMix}}$)**  \nThe total cost is the per-round cost multiplied by the number of rounds, $R$.  \n\\begin{equation}\nC_{\\text{total, FedMix}} = 2 R \\cdot N ( M \\cdot P_e)\n\\end{equation}\n\n**_Limit Analysis_**  \nThis analysis examines the asymptotic behavior of the communication costs under the following assumptions:  \n. The number of active clients is a fixed fraction of the total: $N = S/c$ for some constant $c > 0$.  \n. The parameter size of the initial common expert is equal to that of a single expert: $P_c = P_e$.  **Note:** This is true in our experiments.\n\n- **Analysis as the number of experts $M \\to \\infty$**\n\n  - **DDOME**  \n  \\begin{align*}\n  \\lim_{M \\to \\infty} C_{\\text{total}} &= \\lim_{M \\to \\infty} \\left[ (S \\cdot P_c) + 2 R \\cdot N (P_r + k P_e) \\right] \\\\\n  = (S \\cdot P_c) + 2 R \\cdot N (P_r + k P_e)\n  \\end{align*}  \n  The cost is independent of $M$, implying that the communication cost is bounded and does not increase as the number of experts grow.\n\n  - **FedMix**  \n  \\begin{align*}\n  \\lim_{M \\to \\infty} C_{\\text{total, FedMix}} &= \\lim_{M \\to \\infty} \\left[ 2 R \\cdot N ( M \\cdot P_e) \\right] \\\\\n  = \\infty\n  \\end{align*}  \n\n The cost is a linear function of $M$. As the number of experts grows, the cost grows without bound. This means the communication cost is unbounded, making the algorithm very unscalable compared to DDOME.\n\n\n- **Analysis as Number of Rounds $R \\to \\infty$**  \nWe analyze the cost as the training process becomes infinitely long.\n\n  - **DDOME**  \n  The cost is of course unbounded but grows linearly with $R$. The rate of growth is given by the partial derivative with respect to $R$:  \n  \\begin{equation}\n  \\frac{\\partial C_{\\text{total}}}{\\partial R} = \\frac{2S}{c} (P_r + k P_e)\n  \\end{equation}\n\n  - **FedMix**  \n  The rate of growth with respect to $R$ is:  \n  \\begin{equation}\n  \\frac{\\partial C_{\\text{total, FedMix}}}{\\partial R} = \\frac{2S}{c} (M P_e)\n  \\end{equation}\n\n**Comparison**  \nBoth costs are unbounded as $R \\to \\infty$. However, comparing their growth rates reveals that for $k \\ll M$, DDOME grows significantly slower, making it more efficient for longer training durations.  \n\n$\\frac{\\frac{2S}{c} (P_r + k P_e)}{\\frac{2S}{c} (M P_e)} < 1$\n\n- **Analysis as Number of Clients $S \\to \\infty$**  \nIn this case we will add the $\\epsilon$ term back in, as it directly depends on $S$.\n\n  - **DDOME**  \n  The rate of growth with respect to $S$ is:  \n  \\begin{equation}\n  \\frac{\\partial C_{\\text{total}}}{\\partial S} = P_e + \\frac{R}{c} \\left(2P_r + 2kP_e + k \\epsilon \\right)\n  \\end{equation}\n\n  - **FedMix Algorithm**  \n  The rate of growth with respect to $S$ is:  \n  \\begin{equation}\n  \\frac{\\partial C_{\\text{total, FedMix}}}{\\partial S} = \\frac{2R}{c} ( M P_e)\n  \\end{equation}\n\n**Comparison**  \nThe goal is to prove that FedMix  rate of change is greater. We will show that this is indeed the case under certain (realistic) conditions.\n\nWe seek to demonstrate that:  \n$2\\frac{R}{c} M P_e > P_e + \\frac{R}{c} (2P_r + 2 k P_e + k \\epsilon)$\n\nWe proceed by rearranging this inequality to find the condition under which it holds true.\n\n$\\frac{R}{c} \\left[ 2(M - k) - \\frac{2P_r}{P_e} - \\frac{k \\epsilon}{P_e} \\right] > 1$\n\n$2(M - k) - \\frac{2P_r}{P_e} - \\frac{k \\epsilon}{P_e}$, is a positive number slightly less than $2(M-k)$ and almost certainly greater than 1.\n\nThus, the inequality FedMix $>$ DDOME holds true if:\n\n$\\mathbf{R > \\frac{c}{2(M - k) - \\frac{2P_r}{P_e} - \\frac{k \\epsilon}{P_e}}}$\n\nThis condition is generally satisfied in practical scenario. For example we can plug in our own experimental values from cifar10 and Yahoo!.\n\n$M = 5$, $k=2$, $c = 10$, $R = 2000$  $P_e \\gg P_r$, $P_e \\gg \\epsilon$. \n\nIn this case it is easy to see that this inequality is indeed satisfied, showing us that the rate of change of the communication cost of FedMix, as a function of the number of clients will remain bounded below by DDOME.\n\n**_Conclusion_**\n\nAcross all scaling dimensions of interest—experts (M), rounds (R), and clients (S)—DDOME communication is fundamentally more efficient and scalable."}, {"review_id": "Yf4SONcGi6", "reviewer": "Reviewer_pNK2", "summary": "Paper investigates how joint training of gating functions and experts in MoE frameworks can enable adaptive specialization in decentralized settings, especially in those without explicit task partitions. From what I understand the method uses data heterogenity across clients to achieve expert specialization.", "questions": "1. How does your method improve privacy preservation in the current federated learning scinarios. Detailed analysis would strengthen the paper\n2. Is K in Top-K experts a hyperparameter? Is ther anyway K could be better estimated using the number of experts, nature of the task and other additional metrics?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. paper is well written with formal proofs using theorem 1, which proves that sequential/disjoing training of gating functions and experts leads to suboptimal specialization\n2. Elaborate ablation studies are performed, in situations involving increasing the clients pool, dynamically switching the client's pool\n3. The section on Broader impacts and limitations outlines various limitations of the proposed method, that help in caping the expectations of the proposed method\n\n\nweakness\n1. Although the experiments show that the proposed method performs better than the baselines, I think the authors need to include more recent baselines. the most recent baseline that I see on the comparision table is from 2023.\n2. As someone who worked in a related field but not the same research area, it'd be helpful if the authors can have a figure on an example of what are tasks and domains. This would help appreciate the contributions of the paper better\n3. It'd be helpful to have one medicore expert architectures along with ResNet-34 and TinyBERT for text to understand how the perfrance of the method various with the change of expert architecture\n4. It is slightly surprising to find the paper lacking error bars or stastical sgnificance testing across multiple runs, which make it difficult to asses the uncertainity and realiability of the perforamnce of the method", "was_revised": true, "final_justification": "Although this paper has some flaws, I think the contributions have significant scope and would help future work build better MoE systems. Besides, this paper is based on strong foundations as seen in the proofs from the Appendices. Due to this reason, I'm increasing my score to \"accept\"", "rebuttal": "We are encouraged that the reviewer found our paper to be well written and clear. We also appreciated the acknowledgment of the significance of the formal proof (Theorem 1) demonstrating the suboptimality of disjoint training of gating functions and experts, as this is something we wanted to highlight in our paper. We are pleased that the reviewer valued our elaborate ablation studies. We address some of the concerns and weaknesses raised by the reviewer below.\n\n  1. **\"Although the experiments show that the proposed method performs better than the baselines, I think the authors need to include more recent baselines. the most recent baseline that I see on the comparision table is from 2023.\"**\n     - We would like to note that a similar concern was raised by one other reviewer, and we appreciate that multiple reviewers highlighted the importance of broader comparisons. We fully agree that including more baselines is valuable for situating our contribution within the current landscape. **That said, we would like to clarify the scope, intent, and novelty of our work. A central goal of our study is to provide a proof-of-concept for how distinct expert models—selected dynamically through a lightweight router—can lead to client-specialized training within a single-domain.** This setup is highly relevant for decentralized training, and through this investigation, we developed a state-of-the-art decentralized MoE framework that is competitive with strong FL baselines. While we agree that additional baselines could further reinforce the empirical analysis, we believe the current experimental results are sufficient to demonstrate the promise of our approach and lead to meaningful conclusions. \n\n   2. **\"As someone who worked in a related field but not the same research area, it'd be helpful if the authors can have a figure on an example of what are tasks and domains. This would help appreciate the contributions of the paper better\"**\n\n      - Thank you for the opportunity to clarify this. We did our best to provide some real-life examples of the setting in the introduction, but we understand that the terms \"task\" and \"domain\" can carry different meanings depending on the research context, and we appreciate the different perspective. In our work, we primarily focus on single-domain federated learning, where the data across clients may differ in distributional characteristics but generally share the same task. For example, consider pretraining a model that predicts the type and stage of cancer using CT scans. Here, cancer prediction is the task, and the clients participating in this training process are different hospitals or clinics with private medical data. Each client has labeled CT scans, but likely follows its own underlying distribution of the same task (e.g., one client might specialize in prostate cancer, while another specializes in breast cancer). In the final version, we will include better-contextualized explanation in the introduction (Lines 39 - 44)\n   \n   3. **\"It'd be helpful to have one medicore expert architectures along with ResNet-34 and TinyBERT for text to understand how the perfrance of the method various with the change of expert architecture\"**\n\n      - We agree, this is something the group has been very interested in studying. Diverse architectures across the server has been something we want to study to better understand what can be achieved in such settings. The impact of varying expert architectures is indeed an important and interesting direction, but also a very big one. We have left a broader investigation of this question to future work.\n\n   4. **\"It is slightly surprising to find the paper lacking error bars or stastical sgnificance testing across multiple runs, which make it difficult to asses the uncertainity and realiability of the perforamnce of the method\"**\n\n       - We thank the reviewer for raising this important point. We fully agree that statistical robustness is essential for assessing the reliability of experimental results. **_As shown in Figure 5 and Figure 8, several of our experiments do include statistical error metrics and are averaged over multiple runs. In particular, subfigures (b) and (d) in Figure 8 correspond to results reported in Table 1_**. We omitted error bars in the main tables for consistency but included statistical significance results in the appendix. That said, we acknowledge that not all experiments include this level of statistical detail, especially those involving larger-scale setups. For instance, many of the configurations in Table 2 require 3–7 days per run, depending on available hardware. Running these experiments multiple times under our current resource constraints would require several additional months of compute. We appreciate the reviewer’s emphasis on this aspect, and we will aim to address it for the EMNIST dataset in the final version.\n\nQuestions:\n\n   Q1. **\"How does your method improve privacy preservation in the current federated learning scinarios. Detailed analysis would strengthen the paper\"**\n\n   - To clarify any confusion, our paper does not claim to introduce new privacy-preserving mechanisms in the formal sense. Rather, we emphasize that our method operates fully within the standard privacy-preserving framework of federated learning. Within this framework, our method goes a step further by enabling zero-shot personalization without requiring any additional data or fine-tuning from new clients. While some FL approaches also support this property, many require further adaptation, data, or finetuning, which can introduce additional privacy risks. Our design avoids this, reducing client burden while maintaining personalization.\n\n Q2. **\"Is K in Top-K experts a hyperparameter? Is ther anyway K could be better estimated using the number of experts, nature of the task and other additional metrics?\"**\n   - Yes, to the best of our knowledge, \nK is treated as a hyperparameter in nearly all existing MoE-based work, including ours. However, it is not arbitrary, choosing \nK involves balancing computational efficiency, communication cost, and model performance. While some empirical studies explore the effect of different \nK values, we are not aware of a principled or theoretical framework for setting \nK. Developing such a framework by tying \nK to the number of experts, task complexity, or client heterogeneity is a rich and largely unexplored direction that could support an entire line of future research. Thank you for this very interesting question!"}, {"review_id": "Oy4kGL9rNH", "reviewer": "Reviewer_Uwk3", "summary": "This paper introduces Dynamically Decentralized Orchestration of MoEs (DDOME), a novel framework tailored for decentralized learning environments, particularly Federated Learning (FL). DDOME addresses the challenge of achieving meaningful expert specialization in single-domain tasks where explicit data partitions are absent, by leveraging implicit data heterogeneity across decentralized sources. The authors given a theoretic justification for why joint training of gating mechanisms and experts is necessary, to avoid suboptimal specialization. DDOME utilizes a pretrained common expert as a feature extractor to inform a dynamic gating function, which adaptively selects subsets of experts for just-in-time personalization on unseen clients. The framework incorporates \"anchor clients\" to guide early and stable expert specialization. Empirical validation within FL demonstrates DDOME's superior accuracy (4% to 24% improvement) over state-of-the-art FL baselines in image and text classification, while also reducing communication costs compared to other MoE based FL methods.", "questions": "- Robustness to Common Expert Quality: Given the heavy reliance on a high-quality pretrained common expert, what are the proposed strategies for DDOME's performance in scenarios where such an expert is unavailable, or its performance is sub-optimal? Have the authors considered methods for adaptive or periodic updates to the common expert itself within a federated setup to alleviate this dependency?\n- Scalability to Million-Client Scale: The current experiments use 100 clients. Could the authors discuss the major practical hurdles (e.g., communication bandwidth, computational cost of gating function for very large M, synchronization overhead) and potential solutions for scaling DDOME to federated deployments involving millions of clients and a much larger pool of experts?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work. They provide a dedicated \"Limitations\" section discussing key constraints such as the dependence on pretrained common experts, remaining communication overhead, expert initialization challenges, limited scalability testing, and complexity of gating mechanism tuning. Furthermore, a \"Broader Impacts\" section thoughtfully covers both positive societal impacts (e.g., privacy-preserving FL, resource efficiency) and potential negative impacts (e.g., exacerbating fairness issues due to bias amplification) while suggesting mitigation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Novelty in Decentralized MoEs: DDOME presents a novel solution for dynamic expert specialization in challenging decentralized and single-domain settings. The integration of a pretrained common expert to guide the gating function and the \"anchor clients\" mechanism are novel contributions.\n- Theoretical Foundation: The paper provides a rigorous theoretical justification for the necessity of joint training of gating and expert modules, demonstrating that decoupled training leads to suboptimal specialization. This theoretical insight is a significant contribution that underpins the proposed methodology.\n- Harder setup: Previous approaches assume access to part of the new client’s labeled data is available. This limitation is not imposed in the paper, as new users are likely unwilling/unable to provide accurate labeled data and might not have sufficient resources to contribute to a fine-tuning phase of the whole model.\n- Impressive Empirical Performance: DDOME consistently achieves state-of-the-art accuracy improvements (ranging from 4% to 24%) over leading FL baselines and text classification tasks. Its performance in zero-shot personalization on unseen clients is particularly noteworthy.\n- Robustness to Data Heterogeneity: The framework effectively leverages implicit client-specific data characteristics to dynamically specialize experts and performs strongly in scenarios with highly skewed data distributions (e.g., Dirichlet distribution).\n\nWeaknesses:\n- Heavy Dependence on Pretrained Common Expert: The success of DDOME is critically dependent on the availability and quality of an effective pretrained common expert. The ablation study explicitly shows that a poor common expert can lead to performance collapse. This is a significant practical limitation, as acquiring such a well-performing expert might be costly or not feasible in all decentralized scenarios.\n- Small and Uninteresting datasets: Datasets considered from FL literature are too old and toyish from before 2021, and appear to be non-interesting. The authors should consider more interesting and larger datasets.\n- Lack of Statistical Significance: The paper does not report error bars for its empirical results. This makes it challenging to ascertain the robustness and generalizability of the reported accuracy improvements across multiple runs and different random seeds. \n- Choice of hyperparameters: The authors consider a different optimizer for DDOME than the one considered for baseline models in lines 241-243. This appears problematic without a solid justification. Perhaps the change in optimizer is the cause for some of the gains.\n- Ambiguity in M=K Scenario: In experiments where the number of experts equals the number of selected experts (M=K), particularly for M=K=2 on CIFAR10, DDOME's performance (60.16%) is lower than the initial common expert's accuracy (73.39%) [Table 7]. The authors attribute this to \"malicious 'interference'\", which is a vague explanation. This suggests a potential fragility or a specific failure mode in DDOME under certain configurations that needs deeper investigation and mitigation.\n- Relative Communication Cost: While DDOME reduces communication compared to some MoE methods like FedMix, the authors acknowledge it \"still incurs higher communication overhead compared to traditional single-model methods\". This is an important trade-off that should be more explicitly highlighted in the main claims regarding communication efficiency.\n- Reproducibility: The paper does not provide code.", "was_revised": true, "final_justification": "I have updated from score from 3 to 4 after the clarifications provided by the authors.", "rebuttal": "We thank the reviewer for their constructive evaluation of our work. We are encouraged by the recognition of DDOME’s novelty in decentralized setting. We are also pleased the acknowledgment of the theoretical contributions and anchor-client mechanism. We also appreciate the attention to detail in identifying typos, which we will address in the final version. We respond to the main concerns raised.\n\n  1. **\"Ambiguity in M=K Scenario: In experiments where the number of experts equals the number of selected experts (M=K), particularly for M=K=2 on CIFAR10, DDOME's performance (60.16\\%) is lower than the initial common expert's accuracy (73.39\\%) [Table 7]. The authors attribute this to \"malicious 'interference'\", which is a vague explanation. This suggests a potential fragility or a specific failure mode in DDOME under certain configurations that needs deeper investigation and mitigation.\"**\n     - _We believe the concern raised here is mostly due to a misunderstanding of the purpose of the experiment._ This is indeed a corner case. While the exact mechanism behind the drop in performance remains an open question, we believe this behavior is not entirely surprising. In the $M=K=2$ setting, all experts are sent to all active clients, which removes the intended benefit of selective routing. With a low number of experts, specialization becomes difficult as interference between updates is more likely.\n    **_But we would also like to clarify the purpose of this ablation experiment: it was designed to test DDOME under the same expert-sharing protocol as FedMix, where all experts are sent to each client. Even in this setup, our lightweight server aggregation and inference protocol still outperform FedMix._** So while performance does degrade relative to the initial expert, the experiment shows that DDOME still offers advantages even under these edge cases.\n\n   2. **\"Small and Uninteresting datasets: Datasets considered from FL literature are too old and toyish from before 2021, and appear to be non-interesting. The authors should consider more interesting and larger datasets.\"** \n\n      -  We respectfully disagree with the premise of this statement. The datasets are far from toyish or small. While CIFAR10/100 and EMNIST may be considered outdated by some in the broader community, **_we emphasis that we are using their federated learning versions_**. These federated versions introduce significant non-i.i.d. data skew across clients, making them much more challenging. Federated CIFAR10/100 in particular continue to serve as standard and difficult benchmarks across numerous FL studies. Additionally,  Additionally, the Yahoo! dataset is one of the largest publicly available text classification datasets. Even our subsampled version demands several days of training, with some baselines taking up to a week to complete. DDOME itself requires several days of training. We also note that recent FL papers from NeurIPS 2024 continue to rely on the same datasets we use: for example, **A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings** (Makhija et al., 2024) uses CIFAR10/100 and MNIST; **FedAvP** (Hong et al., 2024) uses CIFAR10/100 and FEMNIST; and **FedGMKD** (Zhang et al., 2024) uses CIFAR10/100 and SVHN.\n\n  3. **\"Heavy Dependence on Pretrained Common Expert: The success of DDOME is critically dependent on the availability and quality of an effective pretrained common expert. The ablation study explicitly shows that a poor common expert can lead to performance collapse. This is a significant practical limitation, as acquiring such a well-performing expert might be costly or not feasible in all decentralized scenarios.\"** \n\n     - We agree the common expert is critical to DDOME, as it informs routing decisions. As acknowledged in section 6, the method can indeed degrade when a high-quality expert is not available. That said, this reliance is not unique to DDOME—many FL and personalized learning methods depend on embedding mechanisms; **Federated Learning of Models Pre-Trained on Different Features with Consensus Graphs** (Ma et al., 2023), **FedPETuning** (Zhang et al., 2023). In practice, we believe the growing availability of high-quality pretrained models can help mitigate this limitation. Nonetheless, we acknowledge this as a key constraint and an important area for future work\n\n  4.  **\"Choice of hyperparameters: The authors consider a different optimizer for DDOME than the one considered for baseline models in lines 241-243. This appears problematic without a solid justification. Perhaps the change in optimizer is the cause for some of the gains.\"** \n\n\n       - We highly thank the reviewer for bringing this up, as it gives us the opportunity to correct the record. The statement that DDOME’s main model used a different optimizer is inaccurate. All the main models for all methods were trained using the same SGDM optimizer to ensure a fair comparison. **A step decay learning rate (not exponential) was applied to the gating network**. This was done to stabilize the training of the routing policy. We would also like to clarify that router decay was limited to a single experiment—specifically, DDOME on the Yahoo! dataset as it presented additional challenges due to the complexity of its features. The correct text in the manuscript (Lines 241-243) should be:  \n       \"_For text classification, all client models are trained using SGDM with identical hyperparameters. All clients use a batch size of 16, and one local epoch per round. The gating function is trained using an SGD optimizer; for image classification, this uses a fixed learning rate of 0.001, while for text classification the same initial learning rate is used but it incorporates a step decay learning rate schedule_\"\n\n   5. **\"Relative Communication Cost: While DDOME reduces communication compared to some MoE methods like FedMix, the authors acknowledge it \"still incurs higher communication overhead compared to traditional single-model methods\". This is an important trade-off that should be more explicitly highlighted in the main claims regarding communication efficiency.\"**\n\n      - We agree and explicitly acknowledge this trade-off in our Limitations. All communication claims are explicitly made only in reference to FedMix, not traditional FL baselines. We’ve done our best to frame them accordingly.\n\n  6. **\"Lack of Statistical Significance: The paper does not report error bars for its empirical results. This makes it challenging to ascertain the robustness and generalizability of the reported accuracy improvements across multiple runs and different random seeds.\"**\n\n      - We appreciate this point; another reviewer raised it as well. Several experiments include error metrics and were averaged over multiple runs (e.g., Fig. 5 and 8). Subfigures 8b and 8d correspond to Table 1. We omitted error bars in the main tables for consistency but included significance results in the appendix. That said, not all experiments include this level of statistical detail, particularly those involving larger-scale configurations. Many baselines from Table 2 require 3–7 days per run, depending on hardware availability. Repeating these experiments under current resource constraints would require several additional months of compute. Nonetheless We aim to include variance estimates in the appendix for EMNIST in the final version.\n\n**Questions**\n\n _Q1._  **\"Robustness to Common Expert Quality\"**\n \n  - Thank you for this question, as it touches on a key aspect of DDOME’s deployability in varied environments. We address each part below \n\n     - **\"(1) Adaptive or periodic updates to the common expert?:\"**\n    \n        This introduces challenges. The common expert is meant to be static and shared once. Updating it periodically would require re-sharing, effectively becoming a central router. While viable, this introduces synchronization and consistency issues—an interesting future direction.\n\n      - **\"(2) Unavailable or sub-optimal common-expert?\"**\n\n        There are several possible strategies: \n\n          - Leverage the rich transfer learning literature using pretrained models as plug-and-play components or with minimal fine-tuning.\n\n         - Explore lightweight data embedding or preprocessing techniques. An interesting direction that drops the common-expert entirely.\n     \n        - Draw inspiration from the learning-to-optimize literature, by training a models solve the optimization based topk routing problem. Further fine-tuning on the actual federated task is possible.\n\n        -  A simple strategy is to increase the capacity of the router itself. A more expressive router might help mitigate the negative effects of sub-optimal embeddings.\n\n\n _Q2._ **\"Scalability to Million-Client Scale\"**\n\n - This is a very thoughtful question and sparked cool discussion in our group. At such scales rethinking the server–client protocol for any FL system is necessary. Communication and synchronization become key bottlenecks. In the context of DDOME, the gating function is the component most directly implicated. While it would certainly need to be adapted, we do not believe it needs to scale into a massive monolith to remain effective. One promising direction we see is client clustering. Clients are grouped based on various criteria, and routing is organized hierarchically: \n\n   -  At the first level, run the gating network once per cluster to select a small subset of experts. \n\n   -   At the second level, clients within each cluster: \n        -   Use local lightweight routers to select from the expert subset, or\n        -   Delegate routing and coordination to a cluster-specific intermediary server\n\n     To reduce global communication further we can leverage peer-to-peer (P2P) communication within clusters, and cache frequently used (“hot”) experts locally. Furthermore, we can apply asynchronous communication between clusters to avoid global synchronization bottlenecks."}, {"review_id": "nLNiFGpmNU", "reviewer": "Reviewer_tKQk", "summary": "The authors study the issue of expert specialization in an MoE framework (specifically in the context of federated learning). They argue through theoretical insights in a Top-1 setting with linear experts that joint training of experts’ weights and their gating function is necessary, else the loss is lower-bounded by a constant. The authors then propose a dynamic expert allocation framework to use implicit client information to achieve specialization without the need for explicit labels (unlike prior work). The authors perform experiments on a range of both image and text classification tasks (with thorough ablation studies), with SOTA performance on zero-shot personalization scores.", "questions": "- **[Q1]**: On [L79], should the local loss function be $\\ell(\\mathbf{W}_s,\\mathcal{D}_s)$ (i.e., with the subscript)? Might the authors clarify whether is a typo, or instead further elaborate on the distinction they intend to draw here between this and the definitions of the partial loss terms above it?\n\n- **[Q2]** In the “remarks” on Theorem 1, the authors state the implication of the loss lower bound when updating the *experts first*, followed by the weights, and then conclude with the importance of joint training. In my impression, it’s important that the result also holds when the update is performed in the reverse order. I think this might follow trivially from the authors’ theorem, but i believe it’s important to spell that out explicitly. Might the authors have comments on whether this holds, or instead perhaps: to what extent this is not important? \n\n- **[Q3]** Is there any important reason why the authors name the shared branch an “expert” (e.g., [L161])? I think this could be confused with the recent design trend of using a shared expert [1] (that runs in parallel with the conditional ones, rather than before as a standalone feature extractor). In my current understanding, I would avoid calling this step an “expert”. \n\n- **[Q4]** The authors state that load balance is a problem with certain choice of pre-trained backbones. This acknowledged limitation is insightful, but I wonder whether the use of a standard load balancing loss [2] might mitigate the issues experienced by the authors. Did the authors consider this at all?\n\n---\n\n[1]: Dai, Damai, et al. \"Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.\" *arXiv preprint arXiv:2401.06066* (2024).\n\n[2]: Fedus, William, Barret Zoph, and Noam Shazeer. \"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.\" *Journal of Machine Learning Research* 23.120 (2022): 1-39.", "limitations": "Yes; the authors' limitations are listed. An extra comment here about the experiments’ limited scope to classification tasks would be useful however.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n## [S1] Timely and important topic\n\nThe paper tackles some very interesting and important problems. With the popularity of MoEs, it’s of great value to study how, why, and when expert *expertise* emerges. The topic here studied by the authors is of great interest to the community.\n\n## [S2] The paper writing is clear\n\nThe paper is very well written--the authors do a fantastic job of introducing the issues with expert specialization and its importance. Ultimately, the high quality of the writing not only makes the paper crystal clear and compelling, but also an enjoyable read.\n\n## [S3] Theoretical framework is compelling and insightful\n\nWhilst I do have a question regarding the scope of the result shown ([Q2] below), the theoretical framework presented with top-1 routing and linear experts is a remarkably clean and useful setting in which to motivate the authors’ argument theoretically. Whilst it is admittedly contrived (specific to top-1 routing and makes the assumption of fixed ground-truth expert assignments), I believe it constitutes an insightful framework with which future researchers could further study MoEs theoretically. \n\n# Weaknesses\n\n## [W1] clarifying federated learning scope\n\nThe authors do a good job of being fairly explicit in the paper’s title and abstract that the work explores issues with MoEs in the context of federated learning. My impression however is that adding additional clarification when discussing the contributions [L30-L48] about the federated learning scope would help make the work even clearer. More than this, however, the authors should elaborate on the extent to which the insights in the paper do (or do not) relate to standard MoE settings outside of the federated learning paradigm. I imagine a non-trivial percentage of readers may not be experts in the federated learning MoE setting specifically, in which case some discussion in the introduction would help the insights bridge this gap.\n\n## [W2] experiments are limited to classification tasks\n\nFirstly, I must state that I am not an expert on federated learning, so I understand that it might be non-trivial to adapt FL pipelines to all experimental setups. Whilst, I appreciate the authors’ efforts to perform experiments on both the vision and text domains, it would strengthen the paper even more to have even small, proof-of-concept experiments on a modern generative task. For example, the authors could (partially) train a NanoGPT (https://github.com/karpathy/nanoGPT) with the RTX A6000 used in their other experiments on a toy data dataset such as TinyStories. I appreciate there might be difficulties that i do not foresee here however, and if that is the case, a discussion of why this is hard to extend to the generative paradigm will be well-placed in the authors’ limitations section.\n\n## [W3] Confusing notation and formatting\n\n1. I find the number of sentences typeset in *“quotations and italics”* in the introduction to be mildly confusing (e.g. [L37]). It’s not clear to me if these are direct quotes from prior work or not--my personal suggestion here as a reader would be to remove the quotations if these are not verbatim quotes from prior work to avoid confusion.\n2. I think the choice to use $\\mathbf{W}_r$ for the gating function’s parameters (when $\\mathbf{W}_i$ is already used to denote the *weights* of a particular expert) is confusing. Furthermore, I do not understand the need for the $_r$ subscript. Might the authors have a justification for this that I overlooked? If not, I might suggest using a totally different letter for the expert gate (e.g. $\\mathbf{G}$) could prevent confusion here, without any subscript.", "was_revised": true, "final_justification": "The paper and theoretical framework would be of interest to the research community; but I am unconvinced by the authors' rebuttal to strongly support the paper", "rebuttal": "We sincerely thank the reviewer for their thoughtful and encouraging feedback. We are especially grateful for the recognition of our theoretical framework, writing clarity, and the significance of the problem tackled. The reviewer’s comments on the relevance and presentation are deeply appreciated, as we invested significant effort in making the paper both accessible and compelling to the broader machine learning community. The reviewer raises several concerns regarding scope clarification, notation, and the potential for broader applicability. We address these points below.\n\n 1. **\"clarifying federated learning scope:\nThe authors do a good job of being fairly explicit in the paper’s title and abstract that the work explores issues with MoEs in the context of federated learning. My impression however is that adding additional clarification when discussing the contributions [L30-L48] about the federated learning scope would help make the work even clearer. More than this, however, the authors should elaborate on the extent to which the insights in the paper do (or do not) relate to standard MoE settings outside of the federated learning paradigm. I imagine a non-trivial percentage of readers may not be experts in the federated learning MoE setting specifically, in which case some discussion in the introduction would help the insights bridge this gap.\"**\n\n    - Thank you for the opportunity to clarify the scope of the work here. As we tried our best to frame the results and insights obtained from the study for general MoE setups, not just for FL. **_To clarify the potential misunderstanding: all the results and insights in the paper indeed apply to standard MoE settings._** The theoretical result is concerned with the dynamics of learning underlying distributions and is not specific to the federated learning setting. Furthermore, the DDOME system is not tied exclusively to FL but applies to any decentralized framework. The primary motivation of this paper is to investigate whether expert specialization can occur in a single-domain setting where explicit task boundaries are not available. From this perspective, **we ask: _what learning setup severely suffers from diverse data distributions in a single-domain scenario?_** The answer is Federated Learning. FL has long been studied not only for its practical relevance but also because of the unique challenges it presents, particularly due to the inherently non-i.i.d. nature of its data, a scenario that conventional centralized methods often struggle to handle effectively. **_In this work, we not only show that expert specialization is possible under such conditions, but also propose a general decentralized framework that, when applied to FL, yields highly competitive results._** \n\n\n2. **\"Confusing notation and formatting:**\n\n    2.1. **I find the number of sentences typeset in “quotations and italics” in the introduction to be mildly confusing (e.g. [L37]). It’s not clear to me if these are direct quotes from prior work or not--my personal suggestion here as a reader would be to remove the quotations if these are not verbatim quotes from prior work to avoid confusion.**\n      \n      2.2. **I think the choice to use $W_r$ for the gating function’s parameters (when $W_i$ is already used to denote the weights of a particular expert) is confusing. Furthermore, I do not understand the need for the subscript. Might the authors have a justification for this that I overlooked? If not, I might suggest using a totally different letter for the expert gate (e.g. $G$) could prevent confusion here, without any subscript\"**\n\n      - To clarify the first point (2.1.), the use of quotation and italics in the introduction was purely stylistic and intended to emphasize key statements or ideas. However, we understand that this formatting may have caused confusion but those lines are not direct quotes from prior work. As for the second point (2.2.) the notation $W_r$ is standard in some seminal MoE literature, where $r$ denotes the router. We followed this convention for consistency with prior work, **Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity** (Fedus et al., 2022).\n\n 3. **\"experiments are limited to classification tasks\nFirstly, I must state that I am not an expert on federated learning, so I understand that it might be non-trivial to adapt FL pipelines to all experimental setups. Whilst, I appreciate the authors’ efforts to perform experiments on both the vision and text domains, it would strengthen the paper even more to have even small, proof-of-concept experiments on a modern generative task. For example, the authors could (partially) train a NanoGPT with the RTX A6000 used in their other experiments on a toy data dataset such as TinyStories. I appreciate there might be difficulties that i do not foresee here however, and if that is the case, a discussion of why this is hard to extend to the generative paradigm will be well-placed in the authors’ limitations section.\"**\n\n    - In principle, training specialized NanoGPT models within our framework is possible.  However, such a setup introduces significant practical and infrastructure-related challenges. The server would need to manage and store multiple NanoGPT models concurrently, while also handling the large-scale parameter updates returned from clients. Although possible in principle, given hardware and time constraints, such experiments were infeasible. **That said, a key future direction addresses this. We are exploring coordinated systems of small, diverse models to emulate LLM-like performance.** Our goal is to demonstrate that a coordinated system of diverse lightweight expert models can achieve competitive performance with larger models such as NanoGPT or Multimodel LLMs. We believe this direction holds great promise and aligns closely with the motivations behind this point.\n\n**Questions**\n\n  Q1. **\"On [L79], should the local loss function be $\\mathcal{L}(\\mathcal{W}_s, \\mathcal{D}_s)$\n (i.e., with the subscript)? Might the authors clarify whether is a typo, or instead further elaborate on the distinction they intend to draw here between this and the definitions of the partial loss terms above it? \"**\n\n   - Yes, the local loss function is $\\mathcal{L}(\\mathcal{W}_s, \\mathcal{D}_s)$. The notation without a subscript was an intentional abstraction to represent the shared objective across both local (client) and global loss functions.\n\n  Q2. **\"In the “remarks” on Theorem 1, the authors state the implication of the loss lower bound when updating the experts first, followed by the weights, and then conclude with the importance of joint training. In my impression, it’s important that the result also holds when the update is performed in the reverse order. I think this might follow trivially from the authors’ theorem, but i believe it’s important to spell that out explicitly. Might the authors have comments on whether this holds, or instead perhaps: to what extent this is not important? \"**\n\n   - Thank you for asking the question. Indeed, the proof strategy for the \"router-first\" case would be similar to our current result. In particular, in proving Theorem 1, the core idea lies in showing that the data points assigned to train each expert is a mix of roughly equal number of data points from each cluster. To show the \"router-first\" case, we simply notice that a data point from cluster $j$ will be assigned to a randomly initialized expert with parameter $w_r$ if $w_r - w_j^\\star$ projected onto $x$ has the smallest magnitude. Therefore, the learned router will try to minimize the projected magnitude of directions $w_r - w_j^\\star$. Due to the randomness of $w_r$, the learned router will also be roughly random.\n\n Q3. **\"Is there any important reason why the authors name the shared branch an “expert” (e.g., [L161])? I think this could be confused with the recent design trend of using a shared expert [1] (that runs in parallel with the conditional ones, rather than before as a standalone feature extractor). In my current understanding, I would avoid calling this step an “expert\"**\n  \n  - We are assuming the reviewer is referring to the “common-expert.” On second thought, a more appropriate name could have been “common-embedder.” The original term was chosen because this component embeds the data to guide expert selection. We will consider renaming it.\n\n Q4. **\"The authors state that load balance is a problem with certain choice of pre-trained backbones. This acknowledged limitation is insightful, but I wonder whether the use of a standard load balancing loss [2] might mitigate the issues experienced by the authors. Did the authors consider this at all?\"**\n\n   - Although this is an interesting direction, its implementation is not as straightforward. In standard MoEs load balancing is used to encourage uniform expert utilization across input tokens. DDOME operates under a slightly different paradigm: the router selects a topk subset of experts to be sent to each client, based on the entirety of each client's data. This expert-to-client assignment is explicitly designed to reflect the client's unique data characteristics. In this setup, enforcing uniform expert usage—i.e., ensuring all experts are equally used across clients—could undermine DDOME’s core objective of learning specialized experts aligned with client-level heterogeneity. When imbalance does arise, we do not view it as a flaw, but rather as an emergent property of specialization. To address router collapse, we introduced a fundamentally different mechanism: the anchor-client strategy. Unlike traditional load balancing, which injects uniformity through the process, our approach establishes persistent relationships between certain clients and experts to encourage stable and diverse expert specialization. We believe this novel contribution could have value even beyond FL and may inspire future work in general MoE designs."}]

Author rebuttal:

Meta-review: This paper studies expert specialization in MOE setting in context of Federated Learning. Using theoretical insights from their framework, authors argue that joint training of experts and gating weights is necessary to make sure training loss continues to decrease. Authors additionally propose dynamic expert allocation framework to achieve speciliazation without the explicit need for labels. Experiments on both image and text classification tasks and thorough ablations are convincing and achieve SOTA performance. 

All the reviewers find the setting of MOE in FL challenging, impactful and interesting. Additionally the paper is well written, easy to follow, experiments are comprehensive and theoretical insights are used to propose new methodology. While reviewers outlined several problems with an experimental setup, authors did a great job during rebuttal phase, with several reviewers increasing the score due to author's clarifications. We expect that all the additional experiments and clarifications are incorporated in the final draft.
Decision: Accept (poster)