Retraining-free Merging of Sparse MoE via Hierarchical Clustering
Authors: I-Chun Chen
Hsu-Shen Liu
Wei-Fang Sun
Chen-Hao Chao
Yen-Chang Hsu
Chun-Yi Lee
Venue: ICML 2025 poster (2025)
Keywords: Sparse Mixture-of-Experts
Merging
Compression
Primary area: deep_learning->large_language_models
TL;DR: Our method (HC-SMoE) offers an efficient method for merging experts of large Sparse Activated Mixture of Experts (SMoE) models without retraining under task-agnostic settings.

Abstract: Sparse Mixture-of-Experts (SMoE) models represent a significant advancement in large language model (LLM) development through their efficient parameter utilization. These models achieve substantial performance improvements at reduced
inference costs. However, the deployment of SMoE models faces constraints from extensive memory requirements of expert components in resource-limited environments. To address these limitations, this paper introduces Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE), a task-agnostic expert merging framework for parameter reduction without retraining. HC-SMoE introduces a novel hierarchical clustering approach based on expert outputs to ensure merging robustness independent of routing decisions. The proposed output-based clustering method enables effective capture of functional relationships between experts for large-
scale architectures. We provide theoretical analysis and comprehensive evaluations across multiple zero-shot language tasks to demonstrate HC-SMoE’s effectiveness in state-of-the-art models including Qwen and Mixtral. The experimental results validate HC-SMoE’s superior performance and practical applicability for real-world deployments. Our implementation is available at https://github.com/wazenmai/HC-SMoE.

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
I-Chun Chen
Hsu-Shen Liu
Wei-Fang Sun?
Chen-Hao Chao Yen-Chang Hsu Chun- Yi Lee
Abstract
Team et al., 2024). This increase in size creates substantial challenges for real-world deployment due heightened in- ference latency and computational demands (Bommasani et al., 2022). Sparsely activated Mixture of Experts (SMoE) models offer a promising solution through their sparse acti- vation mechanism, where only selected model parameters ('experts") activate per input token. This architecture en- ables extensive parametric capacity without proportional computational costs during inference (Shazeer et al., 2017; Fedus et al., 2022). However, the total size of SMoE ar- chitectures presents significant memory constraints, which positions parameter reduction as a critical research prior- ity. Recent studies have revealed important insights about expert redundancy. Liu et al. (2023) identified high represen- tational similarities among experts and proposed methods to enhance expert diversity. Lu et al. (2024) provided addi- tional empirical evidence to support these findings. These studies demonstrate substantial parameter redundancy in current SMoEs and highlight opportunities for optimization.
Sparse Mixture-of-Experts (SMoE) models repre- sent a significant advancement large language model (LLM) development through their efficient parameter utilization. These models achieve sub- stantial performance improvements at reduced inference costs. However, the deployment of SMoE models faces constraints from extensive memory requirements of expert components in resource-limited environments. To address these limitations, this paper introduces Hierarchical Clustering for Sparsely activated Mixture of Ex- perts (HC-SMoE),, task-agnostic expert merg- ing framework for parameter reduction without retraining. HC-SMoF introduces a novel hierar- chical clustering approach based on expert out- puts to ensure merging robustness independent of routing decisions. The proposed output- based clustering method enables effective capture of functional relationships between experts for large- scale architectures. We provide theoretical analy- sis and comprehensive evaluations across multi- ple zero-shot language tasks to demonstrate HC- SMoE's effectiveness in state-of-the-art models including Qwen and Mixtral. The experimen- results validate HC-SMoE's superior perfor- mance and practical applicability for real-world deployments. Our implementation is available http://ggitbb.comm//acc//////CC
Several approaches have emerged to address redundant parameters in SMoE models. Early research introduced task-specific expert pruning (Chen et al., 2022), which pro- gressively eliminates non-essential experts and produces single-expert dense model for specific downstream tasks. However, such approaches often necessitate extensive fine- tuning to maintain performance levels. Recent studies have advanced toward retraining-fre free expert pruning methods (Lu et al., 2024; He et al., 2024). Lu et al. (2024) proposed an approach to trim experts based output loss comparison with the original model. He al. (2024) developed more scalable solution through routing score-based pruning. How- ever, the complete removal of experts and their parameters can cause irreversible loss of learned representations. An alternative research direction explores expert merging rather than pruning. Li et al. (2024) introduced a method to con- solidate information from significant experts. Nevertheless, our experimental results presented in Section reveal limi- tations in this methodology's 's task-agnostic generalizability.
1. Introduction
Transformer-based architectures in natural language pro- cessing (NLP) have demonstrated significant performance improvements across various tasks with exponential param- eter growth (Chowdhery al., 2022: OpenAl et al., 2024:
'Department Computer Science, National Tsing Hua Uni- versity, Taiwan 'NVIDIA AI Technology Center (NVAITC) "Department of Computer Science, University of Toronto, Canada "Samsung Research America 'Department of Computer Science and Information Engineering, National Taiwan University, Taiwan. Correspondence to: Chun- Yi Lee
In response to these challenges, this paper introduces the Hierarchical Clustering for Sparsely Activated Mixture of Experts (HC-SMoE), a retraining scalable, and task- agnostic framework. HC-SMoE reduces SMoE model pa- rameters through hierarchical clustering based on expert outputs and freguency-weighted merging. The hierarchi-
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
cal clustering methodology provides two significant advan- tages. First, the approach advances beyond the single-pass grouping method from Li et al. (2024) through iterative com- parisons, which maintains superior inter-cluster diversity and intra-cluster similarity. Second, HC-SMoF utilizes ex- pert outputs rather than router logits as its similarity metric, which enhances generalizability beyond dataset-specific pat- terns, For fair comparison with previous work, we follow standard evaluation protocols with clustering and merging on the C4 dataset (Raffel et al., 2020) and evaluate accu- racy across eight zero-shot language tasks (Lu et al., 2024). Our extensive evaluations in the supplementary material fur- ther demonstrate HC-SMoE's effectiveness across diverse dataset domains and tasks. Our results in Fig.1 illustrates HC-SMoE's s achievement of comparable performance to the original Qwen model, with improvements of 6.95% and 2. 2.14% over the strongest baseline in 8B and 11B parameter configurations. Section 4 further validates HC-SMoE's s su- perior performance across all baselines on Mixtral 8 x 7B. The primary contributions of this study are summarized as:
28
52
48
14
Ours (avg)
O-prune (10")
10
F-prune
S-prune
M.SMot
Owen tunpruned!
Model Parameters
Figure 1. Effectiveness of expert parameter reduction approaches Qwen 1.5-MoE-A2.78 Ch-Cha (Team, 2024). Average accuracy across 8 LM-Harness benchmarks demonstrates HC-SMMEEss supe- rior performance over existing retraining pruning and merging baselines at 25%, 37.5%, and 50% expert parameter reduction rates. * indicates the original unpruned Qwen model performance.
2024) and Mixtral (Jiang et al., 2024), which adopt the LLaMA (Touvron et al., 2023) structure. The feed-forward network (FFN) in each expert implements three linear lay- ers as shown in Eq. (2), where element- wise multiplication C operates with weight matrices Wup, W gate E Rd xdm W down E TR. Th xdy, and Sigmoid Linear Unit (SiLU) acti- vation function a (Elfwing et al., 2018). The routing im- plementation employs an efficient top-k strategy (Shazeer et al., 2017: Fedus al., 2022) to select experts with the highest logits from linear input transformation. A subse- quent softmax operation on these k largest logits enables sparse expert activation, which reduces computational over- head. This selective mechanism is formulated as follows:
This work proposes HC-SMoE, a promising retraining- free and lask-agnostic merging strategy with efficient scaling characteristics for different numbers of experts.
We propose expert outputs as an effective similarity metric for clustering, which offers advantages over tra- ditional router logits or weights from prior approaches.
Our analysis substantiates the importance of cluster- ing quality for merging effectiveness. The proposed hicrarchical clustering method produces thcoretically guaranteed and empirically validated expert groupings.
Our extensive experiments demonstrate HC-SMoE's consistent superior performance across diverse bench- marks and its effectiveness on multiple SMoE models.
P(x) softmax(topK((()))) = softmax(top (rRR)):
(3)
where R(xx) represents routing-logits and WR denotes the learnable parameter matrix. This sparsely activated archi- tecture enables efficient scaling with preserved performance through selective computation. In turn, this mechanism al- lows the SMoE model to optimize computational efficiency and task performance through focused expert utilization.
2. Background and Related Works
2.1. Sparsely Activated Mixture-of-Expert. (SMoE)
The SMoE model comprises multiple SMoE layers, each of which contains set of expert neural networks and a router network. Consider an input token x, a set of expert neural networks {E1, ..., }, and a router network R. The output y of an SMoE layer is computed as weighted sum of the expert network outputs, which can be expressed as:
2.2. Expert Pruning and Merging
This section reviews existing methods for expert reduction in SMoE architectures, summarized in Table 1. Recent research has focused on various pruning strategies. Chen et al. (2022) proposed Task-Specific Expert Pruning (TSEP). (L) which reduces active experts through iterative fine-tuning for specific downstream tasks. Although effective, this ap- proach requires extensive computational resources and time for fine-tuning, which limits its applicability to large-scale models. Lu et al. (2024) introduced a method, which we refer to as O-prune in this study, for retraining free and task- agnostic expert reduction in zero-shot settings. The method
y= Σ P.(z)-E
Fi(x) = (o( gate) ④
Wdow: (2)
where (x(x) represents the 7th expert routing score from
R. and (x) denotes the 7th expert network output. This architecture extends to recent models like Qwen (Team,
2

---

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
based on router information can be problematic, it de- pends on dataset-dependent statistics. Together, the limita- tions can potentially hinder the model's ability to maintain performance over diverse tasks without access to task data.
Table 1. A Comparison of different approaches for reducing the number of experts in SMoE. We evaluate approaches based on their retraining free nature, task-agnostic applicability, scalability, and strategies. Our method, HC-SMoE, is compared with TSEP (Chen al., 2022), O-prune (Lu et al., 2024), S-prune (He et al., 2024), F-prune, and M-SMoE (Li ct al., 2024). Please note that F-prune is detailed in Section 4,
3. Methodology
3.1. Problem Definition
Method
Retraining-fe free
Task-agnostic
Scalable
Strategy
In this study, we address the challenge of reducing the space complexity of an SMoE model through a process termed expert merging. This process consolidates existing experts in an SMoE layer into smaller set while preserving the model's performance. Each SMoE layer initially contains n experts, as defined in Section 2.1. We aim to merge these experts into r clusters, where r represents the tar- get number of experts after merging. For the i-th cluster, denoted as C. = (E Ei..... ................................ Eic,), ]. |Ca] represents the number of original experts assigned to this cluster. Unlike conventional model merging (Yadav et al., 2023; Xu et al., 2024) with predefined element combinations, expert merg- ing in an SMoE necessitates a two-phase procedure due to its flexible solution space: first grouping experts into clus- ters, then merging within each cluster. During the merging phase, experts within each cluster combine into a single new expert, which reduces the total number of experts to r. The distribution of original experts across clusters satis- fies Σi= |C,| = n, which ensures that all original experts are accounted for in the merging process. The grouping phase encompasses two distinct strategies. Static grouping maintains exactly r experts per layer after merging, while dynamic grouping permits variable expert numbers per layer with an average of r. HC-SMoE implements static group- ing, aligning with O-prune, whereas F-prune and S-prune in Table utilize dynamic grouping. The router network It after expert merging remains unchanged throughout this process, with inputs previously routed to any expert within a merged group now directed to the corresponding merged expert, as illustrated Fig. This preserves router di- mensionality while achieving expert reduction. The expert merging problem presents several unique challenges absent in conventional model merging. First, experts exhibit high- dimensional, task-specific functional overlap that cannot be trivially disentangled through parameter-space similarity alone. Second, the lack of predefined grouping criteria ne- cessitates clustering methods for determining which subsets of experts can be grouped in manner that minimizes per- formance loss in the model. Since training MoE model demands substantial GPU memory, we address this prob- lem without retraining and utilize a non- non-bencchmar dataset to collect information for the expert merging process as calibration dataset.
TSEP
X
Pruning
O-prune
x
Pruning
S-prune
/
Pruning
F-prune
5
/
Pruning
M-SMoE
X
x >
Merging
HC-SMoE
Merging
determines expert retention counts per layer and evaluates all possible expert combinations to select configurations that minimize output deviation from the original model. However, this approach discards potential knowledge from pruned experts. Moreover, its computational requirements become prohibitive for large expert counts. For example, a 50% reduction in Qwen's 's 60 experts requires evaluation of approximately C(60, 30) = 10 combinations per layer. He et al. (2024) proposed an efficient expert trimming tech- nique, denoted as S-prune, based on router scores. This method accumulates global router-scores P(x)) and retains top-scoring experts, which offer enhanced flexibility over O-prune by allowing variable expert retention across layers.
Model merging techniques have emerged as a promising approach to combine the strengths of multiple models. Ziplt (Stoica al., 2024) introduces a model merging technique that allows models with the same architecture but trained on different tasks to be merged without retraining. It utilizes pairwise feature correlation to merge features both within single model and across different models, and provides flex- ibility in choosing correlated features. Since expert merging can be considered a multi-model merging problem, we ex- tend ZipIt to this context. However, its extensive feature correlation computation makes it coneucinu and less effective for expert merging scenarios. M-SMoF (Li et al., 2024) proposes a three-step pipeline for expert merging in SMoE models. It first selects dominant experts based on activation frequency to decide which experts to retain in each layer, then uses router logits R(x) to group experts, followed by frequency- merging, Nevertheless, task- agnostic settings without retraining, relying on frequency information for clustering proves ineffective in Table 2 and Table 3. This approach faces three main issues. First, fre- quency varies across tasks, as shown in Appendix F, making it an unreliable indicator for deciding how many experts to retain in each layer. Second, high-frequency experts within the same layer are rarely merged, overlooking their func- tional similarities in the feature space. Moreover, grouping
3

---

Retraining-Free Merging of Sparse MoE via Hicrarchical Clustering
3.2. Hierarchical Clustering for Sparsely Activated MoE
Exports
Grouped cluster
In light of the challenges from expert clustering and merg- ing, we introduce new framework for SMoE model com- pression through expert parameter merging. Our approach achieves the balance between model size reduction, per- formance retention, and computational efficiency without retraining requirements. It advances beyond previous ap- proaches by addressing three crucial aspects of SMoE com- pression: (1) expert similarity metries, (2) grouping method- ology, and (3) merging strategies. We present that utilizing averaged expert outputs similarity metrics (Section 3.2.1) preserves functional equivalence among grouped experts more effectively than router-space comparisons. We then demonstrate that our hierarchical clustering approach (Sec- tion 3.2.2) enables progressive expert grouping with reduced sensitivity initialization variations. Finally, we elaborate on the expert merging process and demonstrate that effective clustering enables our method to preserve the capabilities of the original model across diverse merging strategies (Sec- tion 3.2.3). The integration of these components enables HC-SMoE to produce reliable and stable cluster quality, and trade-off.
B
RS
Stez 1
Stop
Suap
E4
ER
ES
Figure 2. Illustration of the proposed hicrarchical clustering strat- egy based on expert outputs. Each blue circle denotes the outputs of an expert in the embedding space. Hierarchical clustering would iteratively group the expert clusters with minimum cluster distance.
EB
Router
Router
superior in addressing
Group A
Group B
3.2.1. EXPERT OUTPUTS AS SIMILARITY METRIC
E1
I:33
ES4
E:S
E1
I:2
E:
E:4
ES
The primary objective of expert merging process is min- imize functional divergence between the compressed and original models. M-SMoE falls short in this objective as it adopts router-logits as the similarity metric, which leads to the deviation of the function outputs. Motivated by evi- dence that output similarity correlates with functional equiv- alence (Li al., 2016; Stoica et al., 2024), we propose utilizing average expert outputs over calibration dataset Deal with T tokens address the issue. Specifically, for expert Ej, the representative vector computation follows:
Router
Router
Experts Pruning
Experts Merging
Figure 3. Comparison of expert pruning and merging strategies.
tasks, as validated in Section 4.3, where HC-SMoE outper- forms router-logits- and weight-based baselines. Note that additional quantitative results are available in Appendix D.
3.2.2. HIERARCHICAL CLUSTERING OF EXPERTS
With reliable expert similarity metric established, the sub- sequent step involves clustering SMoE experts into groups for the merging process. To achieve this objective, we em- ploy Hierarchical Clustering (HC) as the core mechanism for grouping experts based on its capability to dynamically adapt cluster assignments while maintaining initialization robustness. Unlike static partitioning methods, HC com- bines experts through bottom-up agglomerative process: starting with each expert as singleton cluster, it recursively combines the most functionally similar pairs while contin- uously recalculating inter-cluster distances. This iterative recalibration reflects current functional affinities of evolving clusters and enables adaptation to emergent behaviors, a capability absent in static partitioning based approaches.
T
0j = Ra (x))
Σ Fx))
(4)
DED cat
This formulation captures both contextual input information and expert learned transformations, validated by L2 errors of last layer outputs presented in Table 23 in the Appendix.
Prior arts face fundamental limitations in this context. For assign- ment preferences rather than intrinsic expert functionality, which creates task-specific biases that hinder generaliza- tion. Parameter-space comparisons, such as concatenat- ing flattened weights (e.g.. Wgane: Wdown, Wap). face com- putational inefficiency and require O(3?) operations for d-dimensional experts and parameter redundancy in SMoE structures, as shown by Liu et al. (2023). In contrast, our average-output-basse metric operates in (dd) space with reduced memory overhead, aligns directly with the merging objective, and maintains consistency across input distribu- tion shifts. This property ensures robustness across diverse
instance, router logits R(z) capture
Prior approaches suffer from fundamental limitations in this context. The one-shot grouping strategy proposed by Li et al. (2024), for instance, freezes cluster assignments after an initial partitioning, disregarding how merged experts alter the functional landscape of remaining clusters. Similarly,

---

Retraining-Free Merging of Sparse MoE via Hicrarchical Clustering
K-means (Ikotun et al., 2023) imposes restrictive assump- tions about cluster geometry (e.g., spherical, equally sized) and exhibits sensitivity to centroid initialization, a flaw am- plified in high-dinne -ithesninnii expert output spaces where ran- dom initialization often traps clusters in suboptimal local minima (Table 5 and Appendix D). HC circumvents these issues through its dendrogram-
other hand, fixed-dominant merging, a methodology intro- duced in this study, represents efficient adaptation of ZipIt specifically developed for merging experts in SMoE models. As developing novel merging methods extends be- yond the focus of this article, we present the comprehensive analysis of fixed-dominant merging and comparisons with Ziplt in Appendix B.2. HC-SMoE implements frequency- weighted merging to maintain merging strategy flexibility, while preserving the task-agnostic nature of hierarchical clustering. The experimental results presented in Table 7 further reveal that merging strategy selection has marginal impact when utilizing a general-purpose calibration dataset.
structure, which ac- commodates heterogeneous cluster shapes and sizes without requiring explicit assumptions. HC recursively applies deter- ministic linkage criteria (e.g., average linkage) to preserve hierarchical expert relationships while mitigating initial- ization dependencies. The clustering process requires two essential components: (1) a distance metric for measuring differences between expert output vectors, and (2) a link- age strategy for determining inter-cluster distances. Our implementation uses the Euclidean distance, expressed as:
4. Experimental Results
4.1. Experimental Settings
d(e;; e,) = ||e; - e,l12
(5)
We conduct experiments on two SMoE models: Qwen MoE-A2.7B (henceforth Qwen) (Team, 2024) and Mix- tral 8x7B (Jiang et al., 2024). For Qwen, we explore two levels of reduction: merging the number of experts from 60 to 45 and further to 30 per layer. This corresponds to a reduction in parameters from 14.3B to 1 11.2B (denoted as Qwen 45x2.7B), and subsequently to 8. 8.1B (denoted as Qwen 30x2.7B). Similarly, Mixtral 8x7B undergoes reduc- tion from eight to six experts and then four experts per layer, decreasing the total parameters from 46.7B 7B to 35.6B (denoted as Mixtral 6x7B) and further to 24.3B (denoted as Mixtral 4x7B). This graduated approach enables the evalua- tion of expert merging impact at different levels of model reduction. Experiments on Mixtral 8x7B and Qwen are conducted on eight NVIDIA A100 GPUs and four NVIDIA V100 GPUs, respectively.
where e; and ej represent the metric values for computing distances between experts and j. For the linkage strategy, we investigate three methods: single, complete, and average:
single:
min b(a,b),
(6)
complete:
max d(a,b). DE
(7)
ΣΣ Σ d(aab). DEA
average:
A B
where A and B represent clusters, and a and b denote ex- perts that belong to these clusters. Single linkage defines cluster distances through the closest pair of elements, while complete linkage uses the maximum distance and often pro- duces overly compact clusters that miss subtle similarities. Average linkage considers the mean pairwise distance be- tween cluster elements and achieves an optimal balance. As a result, the proposed HC-SMoE framework employs average-linkage HC to optimize the trade-off between intra- cluster homogeneity and inter-cluster distinctiveness. The theoretical justification is provided in Appendix A.
To evaluate our method in a task-agnostic setting, we utilize eight tasks using the EleutherAl Language Model Evalua- tion Harness (Gao et al., 2024). These are designed to cover various aspects of language understanding and reasoning, including both Challenge and Easy sets in A12 Reasoning Challenge (ARC) (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), Massive Multi- task Language Understanding (MMLU) (Hendrycks et al., 2021a), OpenBookQA (Mihaylov et al., 2018), Recognizing Textual Entailment (RTE) (Bentivogli et al., 2009) and Wino- grad Schema Challenge (Sakaguchi et al., 2021). We report zero-shot accuracy on those benchmarks. To demonstrate HC-SMoE's 's domain adaptability, additional evaluations on medical reasoning tasks (are offered in Appendix B.4.2).
3.2.3. EXPERT MERGING
Following cluster formation, HC-SMoB merges experts within each cluster via parametrized weight-space ag- gregation. Our empirical evidence indicates that while the choice of merging method does influence the overall performance, its impact is relatively modest compared to the significance of clustering results. Specifically, HC- SMoE merges the clustered experts on the weight space as: É. = Eja E, where E) aj = 1, and a denotes the weight for merging expert j. This study considers three different merging strategies: average merging, frequency- weighted merging, as well as fixed-dominant merging. In average merging, aj- = -े. In frequency-weighted merg- ing, a, denotes the usage frequency of expert j. On the
For our comparisons, three pruning baselines are employed: O-prune (Lu et al., 2024), S-prume (He et al., 2024), and F-prune. F-prune, where 'F' denotes frequency, adheres to the same methodology as S-prune. However, it employs frequency as the criterion for pruning experts, in contrast to S-prune which utilizes router logits. Due to the high
5

---

Retraining-Free Merging of Sparse MoE via Hicrarchical Clustering
Table 2. Zero-shot comparison of Owenl.5-MoPF-A2.7 layer. HC-SMoE (avg) stands for average linkage when performing hierarchical clustering. HC-SMoE (single) stands for single linkage.
original architecture v.s. reduced versions with 45 and 30 experts per
Model
Method
ARC-c
ARC-
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Qwen 60x2.7B
None
0,3951
0.7012
0,8135
0,5932
0.6047
0.310
0,7329
0.6559
0.6008
O-prune (10%)
0,3268
0.6111
0.7566
0,5388
0.5150
0.268
0.6498
0.6330
0,5374
F-prune
0,3490
0,5989
0.7618
0,5441
0.4560
0.282
0.7690
0.6409
0,5502
S-prune
0.3464
0.6061
0,7128
0,5228
0.4930
0.264
0.6534
0.5935
0.5240
Qwen 45x2.7B
M-SMoE
0,3473
0.6157
0.7544
0,5157
0,4182
0,262
0.7292
0.6377
0,5350
HC-SMoE (avg)
0.3660
0.6578
0.7948
0.5520
0.5332
0.272
0,7509
0.6464
0.5716
HC-SMoE (single)
0,3592
0.6578
0.7942
0.5578
0,5360
0.270
0.7292
0.6472
0.5689
O-prune (10%)
0,2568
0.4449
0.6496
0,4351
0,2907
0,202
0.6065
0.5375
0.4279
F-prune
0,2765
0.4718
0.6587
0,4330
0.3023
0,230
0.6570
0.5927
0.4528
S-prune
0,2500
0.4756
0.6388
0.4041
0.3471
0.196
0.6209
0.5146
0.4309
Qwen 30x2.7B
M-SMoF
0.1945
0.2786
0,4462
0.2837
0,2475
0,160
0,4477
0,5185
0.3221
HC-SMoE (avg)
0,3532
0.6149
0.7535
0.4695
0.4534
0.228
0.6606
0,6456
0,5223
HC-SMoE (single)
0,3524
0.6153
0.7661
0,4661
0.4537
0.228
0.6534
0.6306
0.5207
computational complexity of O-prune on Qwen, a random sampling of 10, 000 possible expert sets in each layer is per- formed instead. The set with the smallest output difference from the original model is selected, denoted as O-prune (109) in the Qwen experiments. In addition, M-SMoE is in- cluded as the merging baseline and applied in a lask-agnostic setting without retraining to ensure a fair comparison. All baselines and HC-SMoE require a calibration dataset to estimate input statistics. This dataset is constructed by sam- pling from the C4 corpus (Raffel et al., 2020), concatenating extracted text into 32 sequences of 2, 048 tokens each. To further validate the independence of HC-SMoE from the cal- ibration dataset, we construct two additional datasets from MATH (Hendrycks et al., 202 and CodeQA (Liu & Wan, 2021). Please refer to our Appendix B.3 for more details.
vation substantiates the scalability of HC-SMoE to SMoE models with a higher number of experts. With a 25% re- duction in experts, our method even surpasses the original model on certain tasks, such as Mixtral 6x7B on BoolQ and Qwen 45x2.7B on RTE. This improvement can be attributed to the reduction of expert redundancy after merging. In this configuration, both Qwen and Mixtral exhibit an average performance gap of less than 3% compared to their original models. Even with a 50% reduction, HC-SMoE applied to Qwen maintains a gap of merely 7.43% and outperforms the best baseline, F-prune, which lags behind HC-SMoE by 7.46%. The experimental results validate the robustness and efficacy of HC-SMoE over diverse model sizes and tasks.
4.3. Ablation Study
Ablation on Different Linkage Methods among Different Metries. Table 4 presents a comparison of different link- age methods in hierarchical clustering according to various metrics: router-logits, weight, and expert-ouiput. Hierar- chical clustering exhibits stability due to its deterministic nature. This stability is evidenced by consistent performance across benchmarks and the highest average scores. Unlike K-means, it is not susceptible to initialization randomness, which establishes it as more reliable clustering method. Among the different linkage methods, single linkage gen- erally performs satisfactorily. However, average linkage emerges as the superior option and achieves the highest scores in most of the evaluated settings. The experimental results further reveal an interesting pattern in the perfor- mance of complete linkage across different metries. When applied with the expert-output metric, complete linkage yields suboptimal results, achieving only 0.3909 on aver- age. The performance further deteriorates with the weight metric, which reaches a mere 0.3682. On the contrary, the
4.2. Performance Comparisons
This section presents a comprehensive comparison of the performance of the models reduced by HC-SMoE against the original SMoE models and the baselines. The analysis encompasses various model sizes and tasks, and provides insights into the efficacy and scalability of the proposed HC- SmoE method. As presented in Tables 2 and 3, the M-SMoE baseline exhibits the lowest performance across all bench- marks, indicating the ineffectiveness of router-logit-base based grouping in a task-agnostic setting. O-prune demonstrates suboptimal performance, particularly on Qwen, due to its limitations in evaluating all possible expert combinations. This results in a substantial performance decline compared to Mixtral. In contrast, HC-SMoE demonstrates consistent superiority over these baselines, irrespective of model size, and proves applicable to different numbers of SmoE experts.
Qwen 45B2.BB and Mixtral 4x7B achieve comparable scores despite a twofold difference in parameter count. This obser-
6

---

Retraining-Free Free Merging of Sparse MoE via Hicrarchical Clustering
Table 3. Zero-shot comparison of Mixtral 8x7B: original architecture v.s. reduced versions with six and four experts per layer.
Model
Method
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Averago
Mixtral 8x7B
None
0,5648
0.8422
0.8505
0.6490
0.6712
0.350
0.7112
0.7593
0.6748
O-prune
0.5205
0.8009
0.8352
0.6115
0.5741
0.316
0.6606
0.7719
0.6363
F-prune
0,5009
0.7904
0.7725
0.5990
0,5099
0.326
0.5596
0.7672
0.6032
S-prune
0.4991
0.7891
0.7801
0,5984
0,5103
0.340
0,5704
0.7735
0.6076
Mixtral 6x7B
M-SMoF
0,2619
0.5564
0.5208
0.4320
0.2503
0.194
0.5271
0.5848
0.4159
HC-SMoE (uvg)
0.5145
0.8043
0.8554
0.6142
0,6043
0.324
0.6715
0.7514
0.6425
HC-SMoE (single)
0,5154
0.8123
0.8554
0.6163
0.6053
0.310
0.6715
0.7403
0.6408
O-prune
0,4394
0.7327
0,8046
0.5660
0.4584
0,286
0.5668
0.7285
0.5728
F-prune
0.4352
0.7290
0.7520
0.5293
0.3739
0.290
0,5560
0.7245
0,5487
S-prune
0,2235
0.4339
0.6300
0,4250
0,2554
0,188
0,5235
0,5699
0.4062
Mixtral 4x7B
M-SMoE
0,2116
0.2765
0.4954
0,2767
0,2452
0.108
0,4910
0,4964
0,3251
HC-SMoE (avg)
0.4573
0.7454
0,8018
0,5709
0.4571
0.270
0.5523
0.7285
0,5729
HC-SMoE (single)
0,4642
0.7483
0,8321
0.5781
0,4895
0.280
0.5884
0.7206
0.5877
Table 4. Different linkage method comparisons of hierarchical clus- tering on Qwen 45x2.7B, where 'rl' denotes router-logits and 'eo' denotes expert-output.
Table S. Performance comparison between the proposed HC- SMoE and K-means clustering approaches on Qwen 30x2.7B. K-means-fix designates the first r experts as initial centers, while K-means-rnn randomly selects r' experts as initial centers. Note that the best performance across all methods marked in bold, with the best performance in each clustering method marked with
Linkage
Metric
ARC-c
BoolQ
OBQA
RTE
Average
None
None
0,3951
0.8135
0.310
0.7329
0.5629
s! 0,2398
0.3792
0.180
0.5054
0.3261
underline.
Single
weightt
0,3695
0.7676
0.254
0,7004
0,5229
eo
0,3592
0.7942
0,270
0,7292
0,5382
Cluter
Metric
ARC-c
BoolQ
OBQA
RTE
Average
r! 0,3677
0.7694
0,248
0,7329
0,5295
Complete
weightt
0,2363
0,4446
0.178
0.6137
0,3682
n! 0,2031
0.4015
0.162
0,4838
0.3126
0,2338
0.6037
0,210
0,5162
0.3909
K-fix weight
0,2073
0.4960
0.166
0.509
0.3446
s! 0,2073
0.3801
0.172
0,5018
0,3153
0,2184
0.3786
0,148
0,5343
0.3198
Average
weigght
0,3788
0.7645
0.250
0.7004
0.5234
n!
0,2014
0.4168
0.142
0,5018
0.3155
eo 0,3660
0,7948
0.272
0.7509
0.5459
K-nn
weight 0.2108
0.533
0.174
0.5379
0.3639
ew 0,3370
0.6398
0.224
0.6065
0.4518
nuter-logiit-baseed approach excels exclusively with com- plete linkage, and attains an average score of 0.5295. This disparity substantiates the distinctive properties of router- logits compared to weights and expert outputs. This obser- vation can be attributed to the inherent characteristics of the similarity metrics. Router-logits align well with com- plete linkage since they capture the maximal boundaries between clusters. This alignment effectively reflects distinct activation patterns. In contrast, expert outputs and weights benefit from single or average linkage methods. These met- rics reveal more subtle, internal similarities that may not manifest through extreme distances. Therefore, they favor linkage methods that consider average or minimal distances between cluster elements.
HC
eo
0.3532
0.7535
0.228
0.6606
0.4988
age of language abilities, encompassing common sense rea- soning, basic knowledge questions, and semantic similar- ity between sentence pairs. The evaluation results reveal that most post-merged models utilizing K-means experi- ence substantial decline in their original capabilities. For instance, even the best-performing model employing the expert-output similarity metric achieves a score 4. 4.75% lower than our HC-SMoE results. This performance gap validates the effectiveness of our proposed HC-based method.
K-means also exhibits significant instability, particularly when juxtaposed with HC. The final performance of K- means demonstrates high sensitivity the choice of ini- tial cluster centers. In experiments conducted on the Qwen45x2.7B model using the weight similarity metric, we observe a substantial average accuracy reduction of 12.96% when transitioning from a fixed to random initialization strategy. This sensitivity illuminates K-means' inherent ran- domness and lack of robustness. The observed instability and performance degradation in K-means clustering fur-
K-means Clustering v.s. Hierarchical Clustering. We next present a comparative analysis between our hierarchi- cal clustering (HC) method and various K-means cluster- ing strategies, underscoring the superiority of HC. Table 5 reports the performance of different initialization strate- gies and similarity metrics in K- K-means, evaluated across four benchmarks: ARC-c, BoolQ, OBQA, and RTE. These benchmarks were selected for their comprehensive cover-
7

---

Retraining-Free Free Merging of Sparse MoE via Hierarchical Clustering
Table Comparisons for different similarity metric to single-shot grouping method and HC-SMoF. on Mixtral 8x7B when reducing SMoE experts to average six and four per layer.
Model
Metric
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogramde
Averuge
Mixtral 8x7B
None
0.5648
0.8422
0.8505
0.6490
0.6712
0.350
0,7112
0.7593
0.6748
router-Eogits
0.2619
0.5564
0.5208
0.432
0.2503
0.194
0.5271
0.5848
0.4159
weight
0.4974
0.7955
0.7810
0.6131
0.5244
0.340
0.6715
0.7585
0.6227
Mixtral 6x78
expert-ouput
0.5060
0.8056
0.8373
0.6130
0.5595
0.306
0,6318
0.7474
0.6258
MMOE
0,5145
0,8043
0.8554
0.6142
0.6043
0.324
0.6715
0.7514
0,6425
mater-logits
0.2116
0.2765
0.4954
0.2767
0.2452
0.108
0.4910
0.4964
0.3251
weight 0,4172
0.7382
0.7862
0,5457
0,4223
0.256
0,5523
0.7143
0,5540
Mixtral 4x7B
expert-ourpur
0.4326
0,7386
0.8021
0,5467
0.4290
0.278
0,5704
0.7245
0,5652
HC-SMoF
0.4573
0.7454
0.8018
0.5709
0,4571
0.270
0,5523
0.7285
0,5729
Table 7. Various merging methods with HC average linkage based on expert outputs. Fix-Dom represents fixed-dominant merging
described in Section 3.2.3. Avg in the Merge column denotes the average score among all the merging strategy under same model settings.
Model
Merge
ARC-c
ARC-e
BoolQ
HellaSSag
MMLU
OBQA
RTE
Winogrande
Average
Qwen 60x2.7B
None
0.3951
0.7012
0.8135
0.5932
0,6047
0.310
0.7329
0.6559
0.6008
Frequency
0.3660
0.6578
0.7948
0.5520
0.5332
0.272
0.7509
0.6464
0.5716
Average
0.3584
0.6553
0.7936
0,5516
0.5348
0.270
0.7473
0.6559
0.5709
Qwen 45x2.7B
Fix-Dom
0.3695
0.6692
0.7896
0.5555
0.5338
0.262
0.7365
0.6535
0,5712
Avg
0.3646
0.6608
0.7927
0.5530
0.5339
0.268
0.7449
0.6519
0.5712
Frequency
0.3532
0.6149
0.7535
0.4695
0.4534
0.228
0.6606
0.6456
0.5223
Qwen 30x2.7B
Average
0.3575
0.6145
0.7554
0.4706
0.4531
0.228
0.6643
0.6488
0,5240
Fix-Dom
0,3439
0.6132
0.7544
0.4679
0.4445
0.228
0.6643
0.6504
0.5208
Avg
0.3515
0.6142
0.7544
0.4693
0,4503
0.228
0.6631
0.6483
0.5224
ther accentuate the stability and efficacy our HC-based method. These findings reinforce the superiority of HC in maintaining model performance post-merging and its resilience to initialization variability.
grouping. The results reveal that HC-SMoE demonstrates a clear advantage over the one-shot grouping approaches. achieves average improvements of 1.98% and 1.67% in the 25% and 50% parameter reduction settings,
respeetiseee respectively.
Single-shot Grouping v.s. Hierarchical Clustering. In this analysis, we follow the single-shot grouping methods outlined in (Li et al., 2024) to compare results on Mixtral 8x7B, and report the results in Table 6. Among the similarity metrics evaluated, router-logits exhibits the poorest perfor- mance, indicating its unsuitability for task-agnostic settings due to its reliance dataset-specific statistics. In both the 25% and 50% parameter reduction scenarios, all one- shot grouping methods underperform compared to O-prune presented in Table 3. This observation suggests that these grouping methods fail to form effective clusters, and can potentially result in lower performance even when attempt- ing to absorb all expert knowledge. The method based on the expert output metric demonstrates superior performance over other similarity metrics. It outperforms router-logits by 24.01% % and weights by 1. 1.12% when reducing 50% of the expert parameters. This finding highlights the importance of selecting appropriate similarity metrics for effective expert
Ablation on Different Merging Methods. Table 7 presents the results of hierarchical clustering with three merging strategies: frequency, average, and fixed-dominant merging. For Qwen30x2.7B. the average merging method demon- strates superior performance. It exceeds frequency merging by 0.17% and marginally enhances overall performance, This outcome substantiates assertion that once high- quality cluster is identified, the specific merging method becomes modestly influential on the final performance. The rationale behind this phenomenon lies in the functional sim- ilarity exhibited by experts within the same group, as evi- denced by their similar outputs. Thus, the model maintains robust performance irrespective of the merging strategy em- ployed. It is noteworthy that all three merging methods outperform the four baselines in Table 2. This observa- tion further substantiates the effectiveness of HC-SMoE in preserving model performance during the merging process.
8

---

Retraining-Fre Merging of Sparse MoE via Hierarchical Clustering
5. Conclusion
Arora, S., al. On the opportunities and risks of founda- tion models, 2022.
In this paper, we presented HC-SMoE, a retraining-free, task-agnostic, and scalable expert merging framework that employed hierarchical clustering to reduce the parameters of SMoE models. By employing expert outputs as the similarity metric and leveraging hierarchical clustering, HC- SMoE effectively captured functional similarities between experts, surpassing previous merging and pruning meth- ods. Our comprehensive evaluation on two representative large-scale models, Qwen and Mixtral, demonstrated that HC-SMoE retained the models' general language abilities even when significantly reducing the number of experts. The experimental results also validated the robustness and scalability of our approach. HC-SMoE achieved notable improvements over existing baselines. This work not only provided a practical solution for optimizing SMoE models but also opened up broader domain for further research on aask-agnostic model compression strategies for SMoE.
Chen, T., Huang, S., Xie, Y., Jiao, B., Jiang, D., Zhou, H.,, Li, J. and Wei, F. Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiv:2206.0027 2022.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., et al. Palm: Scaling language modeling with path- ways, 2022.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Volume I (Long and Short Papers), pp. 2924-2936. Association for Computational Linguistics, June 2019.
Impact Statement
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
In compliance with ICML's guidelines, we affirm that our HC-SMoE algorithm does not involve human subjects, per- sonal data, or any components that could raise ethical con- cerns. The algorithm focuses solely on clustering machine learning model experts based on output similarity to opti- mize parameter efficiency, ensuring ethical issues are present.
arXiv preprint /803.00:/000055
Elfwing. S., Uchibe, E., and Doya, K. Sigmoid-weightee
linear units for neural network function approximation in reinforcement learning. Neural nefworks, 107:3-11, 2018.
Fedus, W., Zoph. B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Acknowledgements
The authors gratefully acknowledge support from the Na- tional Science and Technology Council (NSTC) in Taiwan under grant numbers MOST 111-222---002-001--M
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., et al. A framework for few-shot language model evaluation, 2024.
NSTC
and NSTC 113-2640- E-002-003. We also express our sincere appreciation to NVIDIA Corporation and the NVIDIA AI Technology Cen-
ter (NVAITC) for the donation of GPUs and access to the Taipei-1 supercomputer. Furthermore, we thank the Na- tional Center for Pigf-Peeroormncc Computing for provid- ing computational and storage resources.
Hc, S., Dong, D., Ding, L., and Li, A. Demystifying the com- pression of mixture-of experts through a unified frame- work. arXiv preprint (xXi:224660025000 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Mcasuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
References
Bentivogli, L., Clark, P., Dagan, I,, and Giampiccolo, D. The fifth pascal recognizing textual entailment challenge. TAC, (18):12009
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurlPS, 2021b.
Bezdek, J. C., Ehrlich, R., and Full, W. Fem: The fuzzy c-means clustering algorithm. Computers & Geosciences, 10(2):191-20) 1984. ISSN 0098-3004. doi: https://doi.
Ikotun, A. M., Ezugwu, A. E., Abualigah, L., Abuhaija, B., and Heming, J. K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data. Information Sciences, 622:178-2 220,
000/00100000008-0000444000000000
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R.,

---

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
2023. ISSN 0020-0255. doi: http:///bb.orr//0..001..... 2022.11.139.
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medm- cqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceed- ings of the Conference on Health, Inference. and Learning, volume 174 of Proceedings of Machine Learning Research, pp. 248-260.. PMLR, 07-08 Apr 2022. URL https: http//eeeooogg..////////// mlr. www.innssm.r....r.s...m press/ v174 1774/pa1220 html.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., et al. Mixtral of experts. arXiv preprint AXXi:24401040080 2024.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21
Li, P., Zhang, Z., Yadav, P., Sung, Y.-L., Cheng, Y., Bansal, M.. and Chen, T. Merge, then compress: Demystify ef- ficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Represen- tations, 2024.
(1
Sakaguchi, K., Bras, R. L., Bhagavatula, C.,, and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, August 2021.
Li, Y., Yosinski, J., Clune, J., Lipson, H.. and Hoperoft, J. E. Convergent learning: Do different neural networks learn the same representations? In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
ISSN 0001-0782.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q.. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-expeee -f-eeee layer. In International Conference on Learning Representations, 2017.
Liu, B., Ding, L.,, Shen, L., Peng, K., Cao, Y., Cheng, D., and Tao, D. Diversifying the ceixture-of-xxper repre- sentation for language models with orthogonal optimizer. arXiv preprint arXiv:2310.09762. 2023.
Stoica, G., Bolya, D., Bjorner, J. B., Ramesh, P., Hearn, T., and Hoffman, J. Zipit! merging models from differ- ent tasks without training. In The Twelfth International Conference on Learning Representations, 2024.
Team, G., Georgiev, P., Lei, V. L., Burnell, R., Bai, L., and others. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Liu, C. and Wan, X. Codeqa: A question answering dataset for source code comprehension. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pp- 2021.
Team, Q. Qwen1.5-moe: Matching 7b model per- formance with 1/3 activated parameters", February 2024. URL httpt:/t///// /ttp::::://///n/aa ///w///wnnnl woweelm.ape 1m..mmm.g...... wwt.mll...........m..... wttccuub.o/bbog... qwen-most
Lu, X., Liu, Q. Xu, Y., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1159-6172, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B.. Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391 2018.
Xu, Z., Yuan, K., Wang, H., Wang. Y., Song, M., and Song, J. Training free pretrained model merging. In Conference on Computer Vision and Pattern Recognition 2024, 2024.
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. TIES-merging Resolving interference when merg- ing models. In Thirty-seventhh Conference Neural Information Processing Systems, 2023. URL https: // //opneeeview... net/ fww.mmiiddoooooo 2orummid=xxtxx3Wy111
Moseley, B. and Wang, J. R. Approximation bounds for hierarchical clustering: Average linkage, bisecting k- means, and local search. Journal of Machine Learn- ing Research, 24(1):1-36, 2023. URL http:/ /mnr.. org/ heg/papers/ /vwp///22-//-/// 080. html.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Canatiioonlinnnniiiiii Linguistics, 2019.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., et al. GPT-4 technical report, 2024.
10

---

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
Appendix: Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
A. Theoretical Justification
In this section, we present a theoretical rationale for the HC-SMoE algorithm, which employs hierarchical clustering with unweighted average linkage to group experts into clusters and reduce the number of parameters a Mixture-of-Expern (MoE) model. Our objective output while leveraging the performance guarantees for hierarchical clustering.
minimize the approximation error between the original MoE output and the HC-SMoE
Let {G denote partition sets of the expert indices into r clusters. We define the average-merged expert for group j as:
(x(x) = Σ
Bi(x),
iEG,
where represents the expert index and g(i) is the function that maps expert Eg to its cluster index j = =g()). The HC-SMoE output can be expressed as:
12 P(x)) (g(n)
ync(
Σ Pa(x)) By(x) j=1 iEG,
(10)
where (z( denotes the routing probability assigned expert E, for input x, and represents the total number of experts. The approximation error between the original MoE output and the HC-SMoE output is formulated as:
Boriig (x) |)c(z)]1== Σ (E())(E(()) - (2)) &
Σ P((x) [[E(()) Eoce) (x)11,
(11)
by Jensen's 's inequality applied the convex function || 1 This inequality demonstrates that the HC-SMoE approximation error is bounded by the weighted intra-cluster variance. To reduce the approximation error Vorig (x) (HC(()) ?| we opt to to minimize the intra-cluster variance [IE())) - Eyto (2)
A.1. Clustering Optimality and Approximation Bound
Finding the optimal solution, defined as OPT := min E 1 is an NP-hard problem. A naive approach would involve exhaustively evaluating all possible partitions of the experts into r clusters, where the total number of such partitions is given by:
Σ E-ny ( 1)) 2)a
(12)
Hierarchical clustering with average linkage is a polynomial-time algorithm that provides an approximation to the optimal clustering. In the worst case, guarantees solution satisfying | E1i < 3 OPT (Moseley & Wang, 2023). Therefore, we adopt this computationally efficient method in place of the naive exhaustive approach.
Furthermore, unlike K-means and K-center which depend on random initialization, HC operates as a deterministic algorithm. This deterministic nature ensures stable results across executions. Our empirical evidence indicates that HC with average linkage performs effectively in practice, and is able to deliver robust clustering results without randomness requirements, which benefits parameter reduction in MoE models.
A.2. HC-SMoE Algorithm
The HC-SMoE algorithm performs hierarchical clustering on expert outputs and merges experts within each cluster based on freiuency-weighted averaging.

---

Retraining- Free Merging of Sparse MoE via Hicrarchical Clustering
Algorithm HC-SMoE: Hierarchical Clustering for Sparse Mixture-of-Experts
Require: Calibration dataset Deat with I' tokens, number of experts within an MoE layer n, set of experts E {Ei,..., }n target number of clusters r.
Ensure: Merged expert set { (F.,..., }.
// Collect averaged expert outputs
2: for each expert E, E do
3: 0; = EeaDaa [x)] // Compute mean expert output
4: end for
5: // Hierarchical clustering
6: Initialize clusters C = [1 (F2). (F,}) and set m +- n
7: while m > r do
8: Compute pairwise Euclidean distance between expert outputs:
d(Ca, Cb) = CallCol
Σ
|10,
01112
Merge two clusters (Ca; Cb) with smallest d(Ca, Cb) using average linkage. 10: Update C + C (Ca, C₁} U (Ca U 3 and m + m I.
9:
end while
12: //
Frequency-weightee merging 13: for each cluster Cj C do
14:
Let fi be the frequency of tokens assigned to expert E,. ft
15:
Normalize: ft = En,ec, Fe'
16:
Compute merged expert:
E, Σ J.E F.EC,
17:
end for
12

---

Retraining-Fre Merging of Sparse MoE via Hierarchical Clustering
B. Exploratory Experiments
B.1. Non-Uniform Hierarchical Clustering
In our main experiments, the number of clusters in each layer is fixed and uniform due to model design choices. Here, we explore more flexible approach that allows different numbers of clusters in each layer while maintaining an overall 25% or 50% reduction of experts. To determine the cluster count per layer, we first select the top r% most frequently activated experts based their activation frequencies across layers. We then count the number of these experts that remain in each layer to guide the selection of clusters in that layer, followed by hierarchical clustering.
For example, in the uniform clustering setting for Qwen with a 25% reduction, the distribution will be [45, 45, 45, 45] across all layers. In contrast, the non-uniform setting might result in a distribution like [48, 45, 40, 42, 02,422500... 50,.... as long as the overall number of clusters aligns with the target reduction. Table 8 presents the results of this non-uniform clustering strategy.
B.2. Fixed- Dominant Merging
The fixed-dominant (Fix-Dom) merging approach modifics the traditional Ziplt (Stoica et al., 2024) feature similarity calculation. Rather than concatenating features from all experts and computing pairwise correlations, we fix the feature order of designated dominant expert as reference point. Correlations then computed between this fixed order and the features of other experts, shown in Figure 4. Features from secondary experts are grouped with their most correlated counterparts in the dominant expert. The merging process then applies an appropriate weighting scheme, such average merging, preserving the dominant expert's 'ss weight feature order while simplifying the merging process.
Feature similarity is defined as the pairwise correlation between these output features, using formulas adapted from (Li et al., 2016). the original Ziplt model merging, output features are taken after each linear layer. However, since we aim to merge entire experts, each containing three linear layers. we use the intermediate activation features, which is the activations after the non-linear function and before feeding into Woow:: act (x:W gate) O zW up to compute similarity. This approach considers expert similarity from an activation perspective, but we can also the experts' weights the "feature" for correlation or even combine both activation and weight features.
The Fix-Dom merging technique has two main advantages: it preserves the structural integrity of the dominant expert's feature arrangement and accelerates the merging process compared to the original Ziplt method. Instead of iteratively selecting and merging highly correlated features until the target dimension is reached, fix-dom merge performs a more efficient grouping. For example, in Mixtral8x4B, ZipIt takes approximately 725 minutes, while Fix-Dom merge completes in just 7 minutes, making it over 100 times faster. For performance comparisons between Ziplt and fix-dom merge using various feature selections (activation, weight, and activation + weight), refer to Table 9.
B.3. Calibration Dataset
We evaluated our approach using three different calibration datasets: C4 (Raffel et al., 2020), MATH (Hendrycks et al., 2021b), and CodeQA (Liu & Wan, 2021). These datasets vary in their domain focus, where C4 contains general-purpose
Table 8. Zero-shot performance evaluation of non-uniform hierarchical clustering for reducing 25% experts of Qwen. We present the clustering results under single and average linkage with weight and expert-output as similarity metric.
Linkage
Metric
Merge
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Freq
0.2108
0.3493
0.5086
0.4536
0.2296
0.170
0.5596
0.5801
0,3827
weight
Fix-Dom
0.2133
0.3531
0.4847
0.4588
0.2303
0.168
0.6101
0.5714
0.3862
Single
Freq
0.3686
0.6604
0.7960
0.5587
0.5290
0.254
0.7401
0.6543
0.5701
expert-output
Fix-Dom
0.3660
0.6612
0.7917
0.5564
0,5302
0.262
0.7292
0.6527
0.5687
Freq 0.2125
0.3535
0.5024
0.4543
0.2287
0.174
0.5560
0.5785
0.3825
weight
Fix-Dom
0.2116
0.3497
0.4951
0.4565
0.2327
0.164
0,5921
0.5738
0,3844
Average
Freq 0.3575
0.6561
0.7933
0.5538
0,5319
0.272
0.7365
0.6551
0.5695
expert-output
Fix-Dom
0.3558
0.6582
0.7917
0.5558
0.5306
0.270
0.7256
0.6559
0.5680
13

---

Retraining- Free Merging of Sparse MoE via Hierarchical Clustering
Table The comparison between Ziplt and Fix-Dom merging for reducing Mixtral 8x7B
Mixtral 4x7B under the same expert clustering
groups.
Feature
Merge
ARC-c
ARC-e
BooiQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
zipit
0.3959
0.6978
0.7352
0.5350
0.4256
0.252
0.5776
0.7080
0.5409
act
Fix-Dom
0.4036
0.6873
0.7951
0,5351
0.4471
0.278
0.6462
0.7174
0.5637
zipit 0.3959
0.7062
0.7976
0,5376
0,4318
0.266
0.5848
0.6993
0,5524
weight
Fix-Dom
0.4334
0.7290
0.8009
0.5608
0.4913
0.280
0.5596
0.7253
0.5725
zipit 0,4078
0.7146
0.8125
0,5389
0,4364
0.270
0.5921
0.7009
0,5592
act+weight
Fix-Dom
0.4283
0.7184
0.7774
0.5501
0.4737
0.264
0.5921
0.7388
0.5679
Table 10. Ablation study on choice calibration dataset cight zero-shot language tasks Qwen model.
Model
Calib-Dataset
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Qwen 60x2.7B
0.3933
0.6982
0.8119
0.5938
0.6034
0.314
0.7401
0.6606
0.6029
C4
0.3660
0.6578
0.7948
0.5520
0.5332
0.272
0.7509
0.6464
0.5716
Qwen 45x2.7B
MATH
0.3746
0.6662
0.7917
0.5478
0.5293
0.284
0.7437
0.6425
0.5725
CodeQA
0,3490
0.6578
0.7976
0.5275
0,5194
0,260
0.7581
0.6448
0,5643
C4
0,3532
0.6149
0.7535
0.4695
0.4534
0.228
0.6606
0.6456
0.5223
Qwen 30x2.7B
MATH
0.2978
0.5513
0.7593
0.4151
0.4415
0.244
0,6823
0.6164
0.5010
CodeQA
0.3089
0.5720
0.7633
0.4179
0.4412
0.214
0.6606
0.6069
0.4981
tasks and closely aligned with language tasks, MATH focused on math question answering, and CodeQA addresses Python code question answering.
The results presented in Table 10 and Table 11 demonstrate that the performance across the eight evaluated language tasks remains consistent, regardless of the calibration dataset used. Although C4 aligns most closely with general language tasks, the clustering results remain stable even when using domain-specific datasets such as MATH or CodeQA. This finding indicates that for general language tasks, the choice of calibration dataset has only negligible impact on the effectiveness of our method.
B.4. HC-SMoE on Various Models and Domains
B.4.1. DEEPSEEK-
We evaluate HC-SMoE on the DeepSeek-MoE--16BBDB model with pruning ratios of 12.5%, 25%, 37.5%, and 50%. In addition to the first layer, the model originally contains 64 experts and one shared expert per layer. When calculating similarity and merging experts, we only consider those 64 experts in the layer. Reducing the number of experts in MoE models introduces complexity due to interactions among neighboring experts within the same layer and across adjacent layers. Our results demonstrate that even after removing 50% of the experts, the model maintains its performance on language benchmarks and retains much of its knowledge, including well-preserved RTE score. With method, the expert merging process is completed within ten minutes single NVIDIA A100 GPU.
B.4.2. MEDMCQA
To evaluate HC-SMoE on more complex domain specific tasks, we conducted additional experiments on MedMCQA (Pal et al., 2022), a challenging medical question strate HC-SMoE's robust performance in domain semefff applications. The performance of HC-SMoE, exceeds all three baseline methods in this specialized domain.
question-answering answering dataset. Table 15 presents the experimental results, which demon-
hhttp:: htap:::://hgggnggaaccec
http://giiinnnc..c.eeeeeeeeeeeee wwwp:eseeeeeeeeeeeeeeeeeeeeee... http://deppeee-aaadeeeeeeeeeeeee /eeeppeeeemoeeeeeeeeeeeeee moe- 11---aae base
14

---

Retraining- Free Merging of Sparse MoE via Hierarchical Clustering
Table 11, Ablation study on choice of calibration dataset of eight zero-shot language tasks Mixtral model.
Model
Calib-Dataset
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Mixtral 8x7B
0.5648
0.8422
0.8505
0.6490
0.6712
0.350
0.7112
0.7593
0.6748
C4
0.5145
0.8043
0.8554
0,6142
0.6043
0.324
0.6715
0.7514
0.6425
Mixtral 6x7B
MATH
0.5102
0.7992
0.8547
0.6178
0.6026
0.322
0.6643
0.7561
0.6409
CodeQA
0.5196
0.7896
0.8456
0.6104
0.6152
0.316
0.7256
0.7561
0.6473
C4
0.4573
0.7454
0.8018
0.5709
0,4571
0.270
0,5523
0.7285
0.5729
Mixtral 4x7B
MATH
0.4522
0.7546
0.8333
0,5674
0.4985
0.292
0.5884
0.7024
0.5861
CodeQA
0.4411
0.7348
0.8416
0,5666
0.4983
0.288
0.6390
0.7372
0.5933
Table 12. Zero-shot performance evaluation of HC-SMoE: (avg) CeepSeck-Moo-16B-BBB with reducing experts to 56, 48, 40, and 32 experts per layer.
Expert Prune Ratio
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
0%
0.4445
0.7605
0.7251
0.5805
0,3830
0.316
0.6390
0.7040
0.5691
12.5%
0.4403
0.7588
0.7358
0.5661
0.3531
0.206
0.6390
0.7103
0.5512
25%
0.4036
0.7197
0.7339
0.5379
0.3090
0,270
0.6390
0.6977
0.5389
37.5%
0,3831
0.7033
0.7232
0.5008
0,2882
0,248
0.6534
0.6890
0.5236
50%
0.3251
0.6275
0.6810
0.4439
0,2470
0.212
0.6426
0.6448
0.4780
MedMCQA comprises a large-scale Multiple-Choice Question Answering (MCQA) dataset designed for real-world medical entrance examinations. The dataset contains over 194,000 high-quality multiple-choice questions from AIIMS and NEET entrance exams, which span 2,400 healthcare topics across 21 medical subjects. Our experiments utilize two-shot prompt format Table 14. The evaluation protocol assesses whether the model outputs {A. B, C. D) in the subsequent three tokens.
The experimental methodology employs the MedMCQA validation set for evaluation and its training set for calibration. Due to the imbalanced answer distribution in the validation dataset (answer A, B, C, and D, has 950, 735, 617, and 514 samples, respectively), we present a comprehensive analysis through precision, recall, and F1-score metrics.
B.5. Soft Clustering
To explore soft clustering against hard clustering, we implemented the Fuzzy C-Means (FCM) algorithm (Bezdek et al., 1984), which allows each expert to belong to multiple clusters with varying degrees of membership.
Let E .1....,f. 6n denote the a experts to be clustered. The FCM algorithm outputs e cluster centers C Cı.... C. and partition matrix. The degree of membership of expert e; in cluster ej is denoted by uij E [0, 1]. FCM minimizes the following objective function:
J
U ee c212
(13)
where the membership degrees and cluster centers updated iteratively follows:
les - Cjl le cx
-1
Σ N
Σ Uij C
Uij
Oj
(14)
Σ "
For our experiments, we set the hyperparameter m = 2.
Nevertheless, applying soft clustering introduces ambiguity in our frequency-weighted merging method. To address this, we modified the merging process. For each cluster, the final merged expert is computed as a weighted sum the experts, where
15

---

Retraining- Freen Merging of Sparse MoE via Hierarchical Clustering
Table 13. Zero-shot performance evaluation of HC-SMoR (avg) on Mixtral&x7B-Insttuue with reducing experts six and four experts per
layer.
Expert Prune Ratio
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
0%
0.6237
0,8708
0.8850
0.6751
0.6822
0.364
0.7112
0.7656
0.6972
25%
0.5913
0.8443
0.8749
0.6457
0.6284
0.360
0.7329
0.7672
0.6806
50%
0.5162
0.7925
0.8606
0.5995
0.5358
0.308
0.6606
0.7395
0.6266
Table 14. Two-shot prompt lemplate of MedMCQA dataset. <data www.tuueestioo... will be replaced with the specific question well <data. opp>, <data. opb>, <data. opc>, <data. apd> will be replaced with options of corresponding question during evluation.
Please choose option among A,B,C,D to answer the question. Question: Chronic urethral obstruction due to benign prismatic hyperplasia can lead to the following change in kidney parenchyma
Options: A. Hyperplasia B. Hyperophy C. Atrophy D. Dyplasia Ans:C
Question: All of the following are surgical options for morbid obcsity except Options:
A. Adjustable gastric banding B. Biliopancrcatic diversion C. Duodenal Switch D. Roux Y Duodenal By pass Ans:D
Question: <data questi inn> Options: A. <data. opa> B. <data. opb> C. <data. opc> D. <data. opd> Ans:
the membership degree serves the weight:
uiee
(15)
Since HC-SMoE does not modify the router weights (as presented in Fig. 3), input tokens are routed to the corresponding merged expert based on their original assignment. In the FCM setting, this direct routing becomes infeasible, every expert belongs all clusters some degree. To adapt, we merged the router weights using the same weighted formula as above.
Table 16 and Table 17 compare HC-SMoE with FCM. The results indicate significant accuracy degradation when using FCM, which is likely due to interference in the router weights. This performance decline can be attributed to the fundamental differences between hard and soft clustering. hard clustering, each expert belongs to exactly one cluster, which allows for clear and unambiguous assignment of input tokens merged experts. In contrast, soft clustering assigns each expert to multiple clusters with varying degrees of membership, which could lead more complex and potentially less effective routing process. Moreover, the weighted merging of router weights in the FCM setting may introduce noise and dilute the specialized knowledge captured by individual experts. This dilution can hinder the model's ability to effectively route input tokens to the most relevant experts, resulting in suboptimal performance.
These findings highlight that applying soft clustering methods in the MoE merging framework requires a more sophisticated design to handle router weights, particularly retraining-fres free settings. We consider it an interesting direction for future exploration develop novel techniques that can leverage the benefits of soft clustering while mitigating the challenges associated with routing and merging in the context of MoE models.
B.6. Extreme Reduction
To evaluate extreme pruning scenarios, we conducted additional experiments at substantial compression rates of 62.5% and 75%. Our analysis compares HC-SMoE against four established baselines: F-prune, S-prune, O-prune and M-SMoE. The results for the -wwen..5-Mo--A2 Ahhhh-hhtt and Mixtral 8x7B models presented in Table 18 and Table 19, respectively.
Several benchmark tasks employ multiple-choice formats. ARC-c, ARC-e, HellaSwag, MMLU, and OBQA require selection from four options, with 0.25 as the random-gues baseline. BoolQ, RTE, and Winogrande utilize binary choices, with 0.5 as the random- guess baseline. Scores in proximity to these baselines indicate a substantial deterioration of model capabilities.
Under extreme reduction settings, our experiments reveal that baselines experience significant accuracy degradation, with
16

---

Retraining- Ferainiig--Free Merging of Sparse MoE via Hierarchical Clustering
Table 15. Experimental results of HC-SMoF. and three baselines McdMCQA with Mixtral 8x7B and the compressed version of reducing experts to six and four per layer.
Model
Method
Accuracy
Precision
Recall
F1
Mixtral 8x7B
None
0.5930
0.5876
0.5918
0.5888
F-prune
0.3615
0.4109
0.3786
0.3498
Mixtral 6x7B
S-prune
0.4794
0.4755
0.4721
0,4703
M-SMoE
0,1818
0.0909
0.1990
0.0634
HC-SMoB (ours)
0.5018
0.4950
0.4785
0.4828
F-prune
0.3249
0.4363
0.3197
0.2404
Mixtral 4x7B
S-prune
0.0000
0.0000
0.0000
0.0000
M-SMoE
0.0160
0.0720
0.0165
0.0263
HC-SMoE (ours)
0.3817
0.4015
0.3883
0.3705
Table 16. Comparison of HC-SMoE and Fuzzy-Cmeans on Qwennl.5-MoB-A2277B-hhhh
Model
Method
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Qwen 60x2.7B
None
0.3951
0.7012
0.8135
0.5932
0.6047
0.310
0.7329
0.6559
0.6008
HC-SMoE
0,3660
0.6578
0.7948
0.5520
0.5332
0.272
0.7509
0.6464
0.5716
Qwen 45x2.7B
Tuzzy-Cmeans
0.1954
0.282
0,4471
0.2707
0.2658
0.138
0.4910
0.5020
0.3240
HC-SMoE
0.3532
0.6149
0.7535
0.4695
0.4534
0.228
0.6606
0.6456
0.5223
Qwen 30x2.7B
Fuzzy-Cmeans
0.1954
0.2816
0.4428
0.2708
0.2655
0.136
0.4946
0.5043
0.3239
performance often falling below random-guess baselines. In contrast, HC-SMoE maintains competitive accuracy even at a 75% reduction through its output-basee clustering and merging strategy. This finding can be attributed to the fact that experts with similar outputs are likely to capture related features or patterns the data. Merging these experts allows the model preserve essential information while reducing redundancy, which enables a more compact representation without compromising performance.
C. Efficiency Discussion
We evaluate computational and memory costs on the Mixtral 8x7B and Qwen .5--MME-2..7B-Che models in both their original and merged versions. All experiments use the same calibration dataset as the main experiments and consist of 32 sequences of 2048 tokens sampled from the C4 corpus (Raffel al., 2020). The results in Table 20 show that reduction in the number of experts leads sigmificant decreases memory usage and GLOPs without impact on throughput and latency, The ideal benefits of reduced router latency from fewer output channels not realized since retain the original router weights prevent accuracy degradation. As result, the router functions if the original number of experts exists, with experts within the same group producing identical outputs through their corresponding merged experts.
In addition to the inference costs, we also compared the runtime and memory usage of HC-SMoE algorithm against various baselines Table and Table 22. The results demonstrate that HC-SMoE achieves competitive runtime and memory efficiency across different models while maintaining superior performance on benchmarks.
D. Cluster Quality Analysis
In this section, we analyze the characteristics and measure the cluster quality of K-Means and hierarchical clustering to shed a light on the motivation of using hierarchical clustering instead of K-Means.
We selected hierarchical clustering over K-means based on two primary reasons: its stability and determinism. The initialization of cluster centroids in K-means is often random, which can result different clustering outcomes across
17

---

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
Table 17. Comparison of HC-SMoF and Fuzzy-Cmeans Mixtral 8x7B-v0.1.
Model
Method
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Mixtral 8x7B
0.5648
0.8422
0,8505
0.6490
0.6712
0.350
0.7112
0.7593
0.6748
HC-SMoE
0.5145
0.8043
0.8554
0.6142
0.6043
0.324
0.6715
0.7514
0.6425
Mixtral 6x7B
Tuzzy-Cmeans
0.4804
0.7694
0.8297
0.5953
0.5282
0.308
0.639
0.7427
0.6116
HC-SMoE
0.4573
0.7454
0.8018
0.5709
0,4571
0.270
0,5523
0.7285
0.5729
Mixtral 4x7B
0.3609
0.6456
0.7339
0.4704
0.3725
0.240
0,5343
0.6654
0,5029
Table 18. Zero-shot performance evaluation of HC-SMoE and three baseline methods on Aieal.5-MoE-A2.7B--C0 with expert reduction to 23 and 1.5 per layer. We exclude O-prune for this experiment due to the large search space.
Model
Method
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Qwen 60x2.7B
None
0.3951
0.7012
0,8135
0.5932
0.6047
0,310
0.7329
0.6559
0.6008
F-prune
0.2287
0.3763
0.5957
0.3627
0,2413
0.186
0.5668
0.5280
0.3857
S-prune
0.2150
0.4200
0.5945
0.3307
0.2725
0,166
0.5343
0,5406
0.3842
Qwen 23x2.7B
MC-SMoE
0.2014
0.2803
0.4410
0.2743
0,2292
0.158
0,4982
0,51
0.3242
HC-SMoE (ours)
0,3319
0.5720
0.7554
0.4111
0.3957
0.216
0.6606
0.6117
0.4943
E-prune
0.2176
0.3026
0.5269
0.2871
0.2358
0.154
0.5018
0.5185
0.3430
S-prune
0,1954
0,3114
0,5275
0,2803
0.2537
0.136
0,5199
0,5122
0.3421
Qwen 15x2.7B
MC-SMoE
0,1903
0.3035
0.3966
0,2741
0.2295
0.160
0,5199
0,5028
0.3221
HC-SMoE (ours)
0.2662
0.5034
0.7046
0.3664
0.3629
0.196
0.6173
0.5777
0.4493
multiple runs on the same dataset (Ikotun et al., 2023). This non-deteriinistic behavior of K-means makes it less suitable for tasks where the reproducibility and consistency of clustering results are crucial, such in the evaluation of downstream tasks. In contrast, hierarchical clustering generates deterministic results for given dataset and linkage method. This deterministic property ensures that the clustering results are reproducible and consistent across different runs. Furthermore, hicrarchical clustering employs a systematic approach to merge clusters based on specified linkage criterion. This linkage criterion determines the distance between clusters and governs the merging process. By optimizing the linkage criterion, hierarchical clustering guarantees the formation stable clusters that minimize the intra-cluster distance and maximize the inter-cluster distance. This optimization ensures that the resulting clusters are compact and well-separated, which desirable for effective pruning and merging of experts in the context of model compression.
In addition to the theoretical justification, further validated the effectiveness of hierarchical clustering compared to K- K-means through experiments. We conducted experiments using both methods across three similarity metrics (i.e., router logits, expert outputs, and expert weights) and evaluated six clustering criteria:
1. L2 distance: [|T(x) - (2)) where T(x) and S(x) represent the outputs of the original and pruned models,
respectively. Lower values are better.
2. Cosine similarity: cosine-similarity(((e).
Higher values are better.
3. Silhouette score (Euclidean): Measures how similar an object is to its cluster compared to other clusters, using Euclidean distance. Higher values are better.
4. Dunn index (Euclidean): Evaluates cluster compactness and separation, using Euclidean distance. Higher values are better.
5. Silhouette score (Cosine): Similar (3) but based cosine similarity. Higher values better.
6. Dunn index (Cosine): Similar to (4) but based on cosine similarity. Higher values are better.
18

---

Retraining-Free Free Merging of Sparse MoE via Hierarchical Clustering
Table 19. Zero-shot performance evaluation of HC-SMoF. and four haseline methods Mixtral 8x7B-v0.1 with expert reduction to three and two per layer. The runtime for each algorithm is provided in seconds.
Model
Method
ARC-c
ARC-e
BoolQ
HellaSwag
MMLU
OBQA
RTE
Winogrande
Average
Time (s)
Mixtral 8x7B
None
0.5648
0.8422
0.8505
0.6490
0.6712
0.350
0.7112
0.7593
0.6748
F-prune
0.2253
0.399
0.6024
0,3663
0.2414
0.168
0.5379
0.5249
0,3832
61.070
S-prune
0,2082
0,3826
0.5951
0.3648
0.2315
0.154
0,509
0,5383
0.3729
$4,000
Mixtral 3x7B
O-prune
0,4471
0.7210
0,7761
0.5377
0.3847
0,264
0,5921
0.7024
0.5531
2530.584
MC-SMoE
0,2125
0.2963
0.6131
0.2699
0.2513
0,126
0.5162
0.5185
0.3505
43.544
HC-SMoE (ours)
0.4078
0.7138
0.7755
0.5402
0,4156
0.268
0.5451
0.7001
0.5458
253.299
F-prune
0.2329
0.2689
0.6214
0,2681
0.2574
0.150
0.491
0,5162
0.3507
61 61.868
S-prune
0,2022
0.2929
0.6193
0,2942
0.2356
0.142
0,5199
0,5083
0,3518
55.718
Mixtral 2x7B
O-prune
0.3481
0.6540
0,7043
0.4846
0,3163
0.214
0,5451
0.6685
0.4919
1181.15
MC-SMoE
0,2116
0,2908
0.6196
0.2697
0.2370
0.132
0,4729
0,5107
0.3430
43.778
HC-SMoB (ours)
0.3746
0.6721
0.7541
0.4786
0.3606
0.236
0.5307
0.6582
0.5081
267.134
Table 20. Evaluation of computational and memory efficiency across multiple models. For Mixtral: Mixtral 8x7B (original). Mixtral
6x7B (25% pruned), and Mixtral 4x7B (50% pruned). For Qwen1.5-ke 5-MMoEAA.77----- Qwen 60x2.7B (original), Qwen 45x2.7B (25% pruned), and Qwen 30x2.7B (50% pruned). All measurements use identical input sequences and include throughput (tokens per ms), latency (8), GFLOPs, model memory, and model size (number parameters).
Models
Throughput
Latency
GFLOPs
Memory
Model Size
Mixtral 8x7B
13.45 + 1.30
2.854 + 0.333
2989
87.49GB
46.7B
Mixtral 6x7B
13.87 + 0.47
2.666 + 0.093
2267
66.49GB
35.4B
Mixtral 4x7B
13.96 + 0.65
2.599 = 0.166
1546
45.49GB
24.2B
Qwen 60x2.7B
24.08 + 0.17
1.593 + 0.168
916
27.04GB
14.3B
Qwen 45x2.7B
23.95 + 0.24
1.541 = 0.011
717
21.23GB
11.2B
Qwen 30x2.7B
23.16 + 0.42
1.583 + 0.034
518
15.44GB
8.1B
Silhouette score evaluates clustering quality at the data point level (i.e., expert level), while the Dunn index evaluates it at the cluster level. The Dunn index considers maximum intra-cluster and minimum inter-cluster distances, while the Silhouette score uses mean distances. Both metrics highlight clustering compactness and separability. We excluded evaluations involving the cosine similarity of expert weights due to the high computational cost of processing concatenated weight tensors, which would require excessive GPU memory. The detailed formulation is provided at the bottom of this response.
Table 23 summarizes the results. Hierarchical clustering with expert outputs achieves the lowest L2 error and the highest co- sine similarity with the original model outputs 25% and 50% pruning ratios. Moreover, hierarchical clustering consistently outperforms K- K-means across most clustering metrics, demonstrating better clustering quality. These results substantiate the stability and effectiveness of hierarchical clustering in producing compact, well-separated clusters. Furthermore, the zero-shot performance on eight language tasks, as demonstrated in Table 5 in our manuscript, further supports the superiority of hierarchical clustering. Across all tasks, hierarchical clustering consistently outperforms K-means, and achieves better and more stable accuracy.
Based on the above reasons, hierarchical clustering is preferable for its deterministic nature, superior clustering quality, and consistent performance across similarity metries and benchmarks.
E. Frequency Analysis
F. Mixtral 8x7B
We present the activation frequency analysis of all experts Mixtral 8x7B (Jiang et al., 2024) using our sampling dataset from C4 (Raffel et al., 2020) and eight language benchmarks. The results provide evidence against using frequency as the sole criterion for determining the number of experts in each layer. The analysis reveals variability in activation frequency
19

---

Retraining- Frree Merging of Sparse MoE via Hierarchical Clustering
Table 21. Runtime and memory consumption of HC-SMoE and baseline of Mixtral87BB NVIDIA V100 GPUs.
Model
Method
Runtime (sec)
Memory (GB)
F-prune
65
106.600
S S-prune
63
106.592
Mixtral 6x7B
O-prune
1891
122.640
M-SMoR
47
106.601
HC-SMoE (ours)
111
138.039
F-prune
63
106.6005
S-prune
64
106.5926
Mixtral 4x7B
O-prune
3605
122.640
M-SMoE
50
106.601
HC-SMoB (ours)
112
138.039
Table 22. Runtime and memory consumption of HC-SMoE and baseline of Awvenl.5MMoE-2.7755 that since O-prune has non-feasible computation time on Qwen model, we only run 100 iterations for each layer.
4 NVIDIA V100 GPUs. Note
Model
Method
Runtime (sec)
Memory (GB)
F-prune
95
61,605
S-prune
63
106.592
Qwen 45x2.7B
O-prune (100)
824
70.849
M-SMoE
107
48.829
HC-SMoE (ours)
290
48.701
F-prune
95
61.605
S-prune
95
61.605
Qwen 30x2.7B O-prune
840
70.849
M-SMoE
107
48.829
HC-SMoE (ours)
323
48.701
across different tasks, highlighting the fact that this metric is not consistent reliable indicator for expert selection in task-agnostic settings
E.2. TinyLLana-41.IB-NG
The activation frequency analysis of all experts in TinyLLaMa-4k. 2 on sampling dataset of C4 (Raffel et al., 2020) and eight language benchmarks. It can be the evidence of poor expert utilization in SMoE, since one of the experts is seldom chosen among all tasks.
4x1. BB- MoR
20

---

Retraining- Free Merging of Sparse MoE via Hicrarchical Clustering
Table 23. The ablation study of measuring the error of the last layer with the original model, and the cluster quality with different clustering methods and similarity metrics on the Qwen model. We conduct the measurement on Qwen 45x2.7B (pruning ratio 25%) and Qwen and Dunn Index every two because these criteria
30x2.7B (pruning ratio 50%). We bold the of highest Silhouette cannot directly compared to methods using different metrics, ie.. methods using expert outputs cannot compare with methods using weights.
Model
Cluster
Metric
error
Cosine Similarity
Silhouette-Ear
Dunn-Euc
Silhouette-Cos
Dunn-Cos
HC
3,806.8332
0.9972
0.7909
0.8252
0.7090 0.6489
0.5136
Kmeans
eo
6,769.3674
0.9910
0.6093
0.2145
0.5300
HC
weight
9,307, 8304
0.9874
0.7358
0.9801
Qwen 45x2.7B
Kmeans
weight
7,962.1227
0,9919
0.6125
0,9225
HC
rl
7,572.2246
0.9897
0,6688
0.8453
0.7169
0.5204
Kmeans
7.463.5265
0,9900
0.6196
0.4810
0.6389
0.3469
HC
8,142.6961
0,9878
0.6104
0.8126
0.4233
0.4939
Kmeans
eo
9,796.4269
0.9842
0.1851
0.2210
0.2375
0.5136
HC
weight 11,400.2119
0.9791
0.4876
0.9734
Qwen 30x2.7B
Kmeans
weight
10,894.4059
0.9808
0,2380
0,9245
HC
rl
10,022.4158
0.9826
0.4495
0.5536
0.4673
0.2115
Kmeans
10,348.098]
0.9825
0,2566
0.3731
0.2653
0.2030
21

---

Retraining- Free Merging of Sparse MoE via Hierarchical Clustering
C3
Cluster
Dominant Expert
Input token x
Expert network parameters
Expert
Intermediate feature
Step
gate
up
gate
gate
Expert 4
Expert
down
down
down
Expert
Expert
Expert 3
Step 2
Step 3
Expert 2
Expert 3
Assign grouping based on dominant expert
Expert 1
Expert 2
Expert
Expert
Expert
similarity matrix
similarity matrix
Step
Expert
Expert
Expert
gate
gate
up:
SELU
SLU
X
down
down
Merged Expert
original expert
merged expert
Figure 4. Fix-dominant merging. Given experts within cluster and dominant expert index. Step J. first collect intermediate features from each experts. Step Then, we use pairwise correlation to compare similarity between dominant expert and non-dominant experts. Step 3. Each non-dominant expert's dimension choose the dimension of highest similarity with its in the dominant expert feature group. Step 4. Based on this grouping, we average merge each expert weights each dimension.
22

---

Retraining-Free Free Merging of Sparse MoE via Hierarchical Clustering
Transformer block
SMoE Layer
Expert module
Add Norm
X
down
SMoE
Expert I
Expert
Expert
Expert
SiLU
Add Norm
gale
up
Top-k routing
Self-Attention
Router
Assign
Figure 5. General architecture of SMoE. The router uses top-2 routing to assign each token to the two experts with highest scores.
Mixtral8x7B C4
Mixtral8x7B ARC-challenge
0.13
0.12
0.12
0.12
0:13
0.13
011
0.14
0
0:12
0.15
011
0.16
0.10
0.11
0.13
1
0.12
013
0.11
0.12
013
0.14
0.12
0:14
2
0.13
0.13
0.12
0.13
0.12
032
0.13
0.11
2
0.10
0.14
0.12
0.10
0.15
0.14
0.14
0.11
0.12
3
0.16
0.16
0.08
0.11
011
0.14
0.13
0.11
0.200
0.12
0:12
0.12
0.13
0.12
0.13
0.12
0.13
0.14
4
0.14
0.11
0.10
011
017
0.11
0.09
0.35
0.11
0.12
5
0.11
0.14
0.12
0.10
012
0.13
0.17
0.12
0.14
0.13
0.13
0.13
0.11
0.12
011
0.13
6
0.14
0.12
0.15
0.09
0.12
0.13
0.11
0.15
0.14
011
0.13
0.13
7 0.11
011
0.11
0.11
016
0.15
0.11
0.12
0.175
0.12
0.13
0.12
0.12
0112
0.13
0.13
0:12
8
0.11
0.37
0.08
0.15
012
0.33
0.15
0.09
0.13
0.15
0.12
0.12
9
0.13
0.10
0.3
0.09
013
0.35
0.19
0.10
10
0:12
0.12
0.12
0.14
0.14
0.13
0.09
0.14
10
0.11
0.13
0.08
0.18
0.1
0.17
0.10
0.17
0.15
0.15
0.18
0.150
12
0.13
0.14
0.14
0.16
0.13
0.14
0.13
0.04
12
0.13
0,34
0312
0.13
0.1
0.14
0.14
0.05
0.12
0.11
013
0.14
0.14
0.14
0.08
011
0.16
0.12
14
0
0.14
0.07
0.13
0.13
0.13
0.15
0.13
0.10
14
0.10
0.13
0.16
0.12
0.08
0.10
0.14
0.17
0.12
033
0.14
0.08
0.14
0.14
value
Laye
0.13
021
0.125
Value
0.12
0.13
0.14
0.11
0.13
012
0.12
0.14
16
0.15
0.18
0.09
0.11
012
0.08
0.15
0.13
17 011
0.14
0.14
0.12
012
012
012
0.11
0.14
0.16
0.10
18
0.13
0.12
0.13
0.13
0.14
032
0.13
0.12
18
0.14
0.10
0.15
0.12
0.1
0:13
0.15
0.32
19
0.12
0.11
0.13
0.12
013
0.1
0.13
0.13
0.08
0.100
20
0.12
0.14
0.12
0.1
0.12
0.14
01R
0.13
20 0.13
0.16
0.14
0.09
0.0
0.14
0.1
0.14
21
0.13
0.12
0.13
0:13
013
011
0.13
0.12
0.14
0.08
22
0.13
0.13
0.12
0.12
0.13
0.13
0.12
0.13
22
0.13
0.16
0.08
0.13
0.15
013
0.11
0.30
23
031
0.13
0.12
0.11
013
0.13
0.12
0.13
0.10
0.075
24
25
0.13
0.12
0.13
032
0.13
0.13
012
0.12
012
011
013
0.12
0.12
0.13
0.13
0.13
0.06
25
24 0.11
0.14
0.13
0.13
0.0
0.15
0.44
O12
031
0.11
0.13
0.1
0.33
0.14
26
0.12
0.12
0.12
0.13
0:12
0:13
0.12
0.13
26 0.13
0.14
0.14
031
0.1
0.11
011
0.32
27
0.12
0.11
0.13
0.12
0.12
0.15
0.13
0.12
0.11
0.14
0.14
0.10
0.050
28
0.13
0.14
0.14
0.12
0.12
0.10
0.12
0:13
28
0.11
0.15
0.18
0.08
013
0.14
0.09
0.12
29
0.12
0.13
0.13
0.15
0.13
0.11
0.13
0.11
29
0.11
0.16
0.13
0.15
0.08
0.14
0.08
0.12
0.14
0.13
0.09
0.11
30
0.11
0.16
0.17
0.14
0.15
0.09
0.10
0.08
0.11
0.13
0.10
0.16
0.10
0.13
0.14
0.12
31
0.13
0.13
0.09
0.19
0.10
0.12
0.15
0.09
0.025
2
6
Expert
Expert
Figure 6. The frequency anslysis of Mixtral 8x7B on ARC-c and our sampling dataset of C4.
23

---

Retraining-Free FFee Merging of Sparse MoE via Hicrarchical Clustering
Mixtral8x7B ARC-challenge
Mixtral8x7B eRcy
0
0:12
0:13
0.15
0.11
0.11
0.16
0.10
0.11
0
0.12
0.13
0.14
0.11
0.11
1
0.12
6.13
0.11
0.12
0.13
0.14
0.12
0.14
1
0.12
0.12
0.11
0.11
013
0.14
0.11
0.15
0.10
0.14
0.10
0.14
0.14
0.11
0.10
0.14
0.10
0.14
3
0.16
0.16
0.08
011
0.11
0.11
0.14
0.13
0.15
011
0.200
3
0:14
0.15
0.17
0.08
0.11
0.11
0.14
0.13
0.09
0.11
0.200
5
0.11
0.14
0.12
0.10
0.12
0:13
0.17
0.12
5
0.11
0.14
0.11
0.10
0.12
0.13
0.17
0.13
6
0.14
0.09
0.13
0.15
6
0.15
0.14
0.09
0.10
8
7 0:11
0.11
0.12
0.11
0:15
011
0.16
0.15
0.13
0.11
0.09
0.12
0.175
7 0.11
0.11
0.32
0.10
0.14
0.12
0.15
0.16
0.16
0.11
0.12
0.175
9
0:13
0.10
0.13
0.09
0.13
0:15
0.19
0.10
9-
0.14
0.10
0.12
0.08
0.13
0.15
0.19
0.09
0.11
0:18
0.12
0,12
10 0.10
0:13
0.08
0.18
0.16
011
0.11
012
11
0.11
0.16
0.14
0.02
0.08
0.18
0.22
0.08
0.150
11 0.11
0.16
0.14
0.02
0.08
0.18
0.22
0.08
0.13
0.13
0.14
0.05
12- 013
0.14
0.11
013
0.16
0.4
0.14
0.05
0.150
13
0.14
0.10
0.08
0.17
0.11
0.16
0.12
0.12
13
0.14
0.10
0.08
0.17
0.11
0.16
0.13
0.12
0.10
0.12
0.10
0.17
14
0.10
0.13
0.17
0.12
0.08
0.10
0.14
0.17
15
0.15
0.08
0,09
013
0.11
011
0.21
0.08
0.08
0.15
0.13
0.14
0.125
Valuee
3
15
16
0.15
0.07
0.19
0.09
0.12
0.09
0.10
0.12
0.22
0.11
0.08
0.09
0.14
0.15
0.14
0.13
0.125
Value
17
0.14
0.12
0.16
0.10
0.10
0.11
0.12
0.15
0.10
0.13
0.12
0.13
18
0.14
0.10
0.15
0.12
0.10
0.14
0.15
039
19
0.15
011
0,09
0.09
012
0.09
0.14
0.14
0:14
0.14
0,14
0.13
0.100
20
0.13
0.09
0.16
0.15
0.09
0.09
0.13
0.15
0.12
0.13
0.100
21
0.14
013
0.13
010
0.16
0.09
0.17
0.08
0.14
0:13
0.09
0.17
0.09
0.13
0.13
22
0.13
0.16
0.08
0.14
0.15
0.13
0.11
0.10
23
0.10
0.14
0.14
010
0.13
0N1
0.15
0.13
0.10
0.14
0.10
0.16
0.15
24
0.11
0.13
0.13
0.15
0.12
0.11
0.11
0.14
0.075
25
0.14
0.14
0.10
012
0.14
0.11
0.13
0.13
0.12
0.13
012
0.11
26
0.13
0.14
0.13
DBB
0.14
0.11
0,10
0.13
27
0.11
0,10
0.14
013
0.14
0.14
0.14
0.10
0.050
0.11
0.10
0.13
0.14
0.14
0.08
28
0.11
0.15
0.19
0.08
0.13
0.14
0.09
0.11
0.050
29
0.11
0.16
0.13
0.15
0.15
0.08
0.14
0.08
0.12
0.15
0.14
0.14
0.09
0.08
30
0.10
0.15
0.17
0.15
0.15
0.10
0.10
0.07
31
0.13
0.13
0.09
0.19
0.10
0.12
0.15
0.09
0.025
0.19
0:15
0.09
2
5
7
0
1
3
4
5
6
7
0.025
Expert
Expert
Figure 7. The frequency anslysis of Mixtral 8x7B on ARC-c and ARC-e.
Mixtral8x78 BoolQ
Mixtral8x7B Winogrande
0
0:13
0.14
0.14
0.11
0:12
0.15
0.09
0.13
0
0.13
0.15
0.15
0.09
0.12
0.17
0.10
0.09
1-
0:13
0:12
0:13
0114
0.13
0.11
010
0:13
0.15
0.12
2
0:13
0.14
0.13
0.13
0.13
0.12
0.13
0110
2
0.09
0.13
0.12
013
0.16
0.16
011
0.08
3
0.15
0.11
0.09
0.12
011
0.12
0.14
012
0.15
0.09
0.12
0.25
4
0.11
0:13
0.13
0.11
0.14
0.11
0.12
0.15
4
0.36
0.11
0.09
011
0.13
012
0.11
5.16
0.15
0.11
0.13
0.13
0.09
0.12
0.14
0.13
6
0.10
0:13
0.12
0.11
0.10
0.15
0.13
0.16
6
0.16
0.13
0.09
0.19
031
030
0.12
0.10
7
0.10
0.12
0.10
0.15
0.13
0.09
0.12
8
0.11
0.14
0.10
0.09
0.22
0.16
0.16
0.12
0.14
8
0.32
0.10
0.08
0
0.16
012
0.12
0.14
0.16
0.11
0.11
0.12
0.19
0.20
0.08
0.77
0.07
0.20
10
0.14
0.16
0.12
0:13
0.14
0.11
0.07
0.14
10 0.07
0.16
0.10
0.21
0.10
0.15
0.09
0.12
0:13
0.05
11 0.10
0.09
0.14
0.01
0.12
0,20
0.28
0.05
12
0.17
0.16
0.13
0.14
0.11
0.12
0.11
0.06
12 0:17
Q14
0.15
015
0.14
0.4
0.09
0.02
14
0.13
0.12
0.08
0:13
0.14
0.14
0.11
0.14
0.13
0.12
14 0.08
13
0.08
0.15
0.17
0.03
0.22
0.14
0.14
0.04
0.12
0.09
0.20
0.19
0.14
0.12
0.18
15
0.14
0.13
0.10
0.12
011
Value
15 0.07
0.08
0.17
0.05
0.19
0.09
0.18
0.17
0.15
Value
16
0.15
0.14
0.11
0.11
010
0.12
0.12
0.15
16 0
Q.4
0.15
0.09
0.13
0.07
0.11
0.19
0.13
0.12
0.11
0.10
0.14
0.13
0.14
17 0.13
0.18
0.15
0.15
0.10
011
0.08
0.10
18
012
0.14
6.13
0.10
011
0.14
012
0.14
18 0.08
0.08
0.16
0.15
0,14
0.12
19+
0.13
0.12
0.14
031
0.11
012
0.13
0111
0.10
19
0.14
0.08
0.11
031
0.13
031
0.18
0.13
20
0.11
0.14
0.13
0.13
0.10
0.12
0.14
0.12
20 0.12
Q17
0.09
0.08
0.14
0.15
010
0.16
21
0.12
0:13
0.11
0.32
0.14
0.10
0.13
0.14
21
0.11
0.11
0.13
0.16
0.14
0.10
0.14
0.11
22-
0.12
0.14
0.12
0.15
0:13
0.11
012
22
0.12
0.11
0.13
23
0.12
0.14
0.13
0.12
01
0.11
0.13
0.15
23
0.12
0.12
0.13
0.10
0.11
0.12
0.13
0.17
0.1
031
010
0.08
24
0.10
0.15
0.11
25
0.12
011
0.13
0.15
0.12
0.12
0.13
0.10
25
0.14
0.13
0.14
0.10
0.13
0.10
0.12
0.14
0.12
0.12
0.13
0.12
26
0.12
0.11
0.11
27
0.12
031
0.12
0.10
0.13
0.13
0.14
0.14
27
0.11
0.08
0.15
0.12
0.17
0.14
0.16
0.07
0.05
0.12
010
28
0.10
0.16
0.08
29
0.10
0.15
0.13
0.12
0.12
0.11
0.12
0.15
0.06
29
0.14
0.14
0.15
0.13
0.15
0.09
0.13
0.07
0.11
30
0.17
0.08
31
0
0.13
0
0.11
0.13
0.14
0.11
0.14
31
0:33
0.14
0.08
19
0.10
0.13
0.18
0.06
6
0
2
5
7
Expert
Expert
Figure& 8. The frequency anslysis of Mixtral 8x7B on BoolQ and Winogrande.
24

---

Retraining- Free Merging of Sparse MoE via Hicrarchical Clustering
Mixtral8x7B MMLU
Mixtral8x7B HellaSwag
0:14
0.12
0.12
011
0.35
0.12
0.12
0-
0.13
0.13
0.12
0.12
0.12
0.16
0.08
0.13
0.225
1-
0.10
0.12
0.16
0.11
0.13
0.13
0.12
0.13
1-
0.11
0.15
0.10
0.17
0.12
0.11
0.12
0.11
0115
6,13
0.13
0.12
0.11
0.12
0.12
0.11
0.14
0.15
012
0.12
3
0.14
0.12
0.09
0.13
0.14
0.12
0.12
0.14
0.16
0.15
0.11
0.12
0.15
0.12
0.11
0.12
0.32
0:12
0.12
0.12
0.14
0.11
0.14
0.11
0.12
0.11
0.11
013
0.16
0.200
5-
031
0.11
0.11
0.15
0.11
0.31
0.15
0.14
0.11
0.14
0112
0.10
0.12
0315
0:3
0.15
0:12
0.18
0.11
0.10
0.14
0.12
0.30
0.12
0.11
0.12
0.13
7
D17
0.12
0.14
0.12
0.16
0:13
0.11
0.10
0.14
0.14
0.10
013
0.09
0.17
011
0.15
0.13
0.13
0.15
0.14
0.09
0.30
0.09
0.13
0.12
0.13
0.175
0.13
010
011
0.10
0.3
0.16
0.16
0.13
0.14
0.16
0.14
0.08
0.17
0.09
30
0.14
0144
011
0.15
0.15
0.10
0.12
0.10
10
0.11
0.14
0.11
0.18
0.14
0.15
0.07
0.1
11
1:15
0.14
0.17
0.12
11
0.14
0.12
0.13
0.03
011
0.16
0.18
0.32
12
1.15
0.17
0:13
0.14
0.14
0.32
0.12
0.04
12
0.13
0.08
0.18
0.15
0.14
0.13
0.14
0,06
0.150
13
013
0.11
0.13
011
0.13
0.11
0.33
0.06
0.12
0.15
0.14
14
0.13
0.13
0.10
0.13
0.12
0.13
0.13
0.13
14
0.10
0.13
0.12
0.13
0.12
0110
0.18
0.14
Laver
15
0.12
0.15
0.12
0.10
0.15
0.13
0.14
Veluve
16
011
0.15
010
0.11
0.12
0.12
0.12
0.16
Velue
16
0.13
0.12
0.16
0.10
0.13
0.09
0.12
0.14
0.125
17
0.16
17
0.14
0.12
0.11
0.14
18
0.14
0.12
0.13
0.13
0.14
031
0.12
0.12
18
0.12
0.10
0.14
0.15
0.13
0.11
0.14
0.10
19
0.13
0:13
0.10
0.13
0.13
0.15
0.08
19
0.10
0.14
0.14
0.10
20
013
0.22
0.1
0.15
0,1
0:13
0.12
0.13
20
0.12
0.18
0.09
0.07
0.17
0.12
0.1
0.15
0.100
21 011
0.16
0
0.12
0.12
0.13
0.12
0.11
21
0.15
0.11
0.10
0.12
22
012
015
010
0.14
0.15
0.12
0.11
0.12
22
0.12
0.18
0.12
0.10
0.11
0.11
0.12
0.14
23
0.12
0
0:13
0.13
0.14
0.13
0.12
0.11
0.06
23
0.14
0.14
0.13
0.14
0.15
24
25
0.12
0:13
0.15
0.14
011
0.12
0.10
0.09
0
0.16
0.12
0.11
25
24
0.10
0.12
0.14
0.14
0.16
011
0.09
0.10
0.11
0.12
0.13
0.16
0.075
26
012
0.17
0.12
0.33
0.12
26
0.11
0.14
0.12
0.14
0.13
0.12
0.11
0.13
27
0.13
0.13
0.12
0.11
0.12
0.14
0.12
0.14
0.04
27 0.12
0.07
0.14
0.14
0.14
0.13
0.16
0.11
29
28
0.15
012
0.13
0.13
0.15
0.13
0.12
0.12
0.11
0.10
28
29
0.13
0.13
0.15
0.13
0.11
0:12
0.09
0.16
0.14
0.13
0.10
0.11
0.12
0.14
0:13
0.11
0.050
0.14
0.12
30
0.12
0.15
0.16
0.16
0.13
0.07
0.13
0.09
31
0:13
0.12
0.12
0.16
0.11
0.32
0.13
0.13
0.02
0.07
0.11
0.12
0.08
2
5
7
0
2
4
5
7
Expert
Expert
Figure 9. The frequency anslysis of Mixtral 8x7B on MMLU and HcllaSwag.
Mixtral8x7B RTE
Mixtral6x7B OpenBookQA
0
0:13
0.18
0.16
0.10
0.11
0.14
0.09
0.11
0.10
0.14
0.15
011
0.13
0.17
0.10
0.10
1
0.13
0111
0.14
0.11
0.15
0.14
0.10
0.12
0.11
0.16
0.1
0.13
0.13
0.30
2
0.11
014
011
0.11
0.12
0.13
0.16
0.09
2
0.09
0.15
0.09
0.13
0.18
0.13
0.13
0.09
3
0.16
0.12
0.08
0.13
0.12
0.2
0.34
0.13
0.18
0.14
0.18
0.08
0.08
0.1
013
0.17
0.11
4
0.13
0.14
0.12
0.10
0.14
0.11
0.10
0.16
4
0.15
0.10
0.11
0.13
0.14
0.12
0.10
0.16
5
0.12
0.14
0.12
0.11
0.12
0.10
0.13
0.15
0.10
0.15
0.11
0.11
0.11
0.15
0.12
0.15
011
0.12
0.10
0.14
0.10
0.14
0.17
019
0.12
0.15
0.17
0.10
0.08
0.10
0.08
7
0.13
0.12
0.12
031
0.12
0.17
0.10
0.13
7
0.11
0.10
0.12
0.13
0.14
0.16
0.12
0.12
0.25
0.09
0.15
0.10
0.14
0.12
0.14
0.13
0.12
0.13
0.18
0.13
0.13
9
0.15
0.08
0.11
0.10
0.15
0.13
0.15
0.13
9-
0.13
0.10
0.17
0.09
0.13
0.17
0.13
0.08
0.13
0.16
0.09
0.13
0.06
0.12
11
0.12
0.13
0.20
0.04
0.13
0.14
0.15
0.11
0.14
11
0.10
0.10
0.11
0.01
0.07
0.22
0.32
0.07
12
0.16
0.18
0.11
0.15
0.08
0.03
0.16
0.15
0.16
0.20
0.20
13
0.14
0.15
0.14
031
0.10
017
0.12
0.12
13
0.09
0.17
0.02
0.16
0.10
0.20
0.15
0,10
14
0.14
0.09
0.13
0.15
0.12
0.10
14
0.07
0.05
Layer
15
0.15
0.12
0.11
0.12
0.14
0.3
0.10
0.14
Volue
15
0.07
0.08
0.16
0.06
0.18
0.11
0.17
0.15
Velue
16
0.09
16
0.12
0.19
0.10
0.12
0.06
17
0.12
0.10
0.13
0.11
0.09
0.15
0.15
0.15
17
0
0.16
0.14
0.16
0.09
0.08
0.10
0.14
0.15
18
0.17
0.10
18
0.09
0.08
0.15
0.15
0.14
0.15
0.15
19
0.13
0.13
0.13
0.12
0.09
0.12
0.14
0.12
19
0.11
0.09
0.07
0.13
0.14
0.15
0.17
0.13
0.12
0.15
0.08
20 0.14
0.18
0.11
0.07
0.10
0.14
0.10
0.16
21
0.11
0.16
0.13
0.12
0.11
0.13
0.11
0.14
21
0.15
0.11
0.15
0.13
0.14
0.09
0.15
0.08
23
0.13
0.13
012
0.13
0.11
0.14
0.10
0.16
0.16
0.10
23
22
0.10
0.15
0.13
0.17
0.13
0.10
0.10
0.10
011
0.13
0.15
0.13
0.12
0.12
0.12
0.14
0.100
24
0.11
0.12
0.14
0.15
0.12
0.10
0.14
0.14
25
0.13
0.1
0.13
0.14
011
0110
0.13
0.12
25
0.14
0.13
0.13
0.12
0.09
0.06
26
0.13
0.12
0.13
0.13
0.14
0.13
0.09
0.12
27
0.14
0.12
0.12
0.10
0.16
0.11
0.13
0.13
27
0.11
0.17
0.10
0.11
28
0.11
0.14
0.18
0.06
0.14
0.12
0.12
0.12
0.05
29
0.11
0.16
0.12
0.11
0.12
0.13
0.12
0.13
0.04
29
0.13
0.15
0.16
0.07
0.11
0.10
0.13
0.17
30
0.08
0.15
0.16
0.18
0.14
0.10
0.10
0.08
31
0.12
013
0.14
0.11
0.11
0.13
0.12
0.15
0.11
0.10
0
2
3
4
5
7
Expert
Expert
Figure 10. The frequency anslysis of Mixtral 8x7B RTE and OpenBookQA.
25

---

Retraining-Free Free Merging of Sparse MoE via Hicrarchical Clustering
TinyLLama4x1. 1B BoolQ
TinyLLama4x1.1B MMLU
TinyLLama4x1.1B (32)
0.11
0.17
0:34
0.38
-040
0:27
0.11
0.29
0.30
0.25
0.30
0.29
6.10
025
0.35
0.29
0.11
024
0:30
6.11
026
0:31
0.31
0.12
0.25
032
0.6
0.33
0.99
024
0.34
0.33
0.10
0.23
0.34
0.78
0.11
0.27
0.34
0.28
0.13
0.27
0.30
0.31
0.3
0.39
0.34
0.33
0.2
0.28
0.14
0.26
0.2
0:33
0.09
0.23
0.35
0.30
0.12
0.28
0.32
Layer
0:31
0.15
0.27
0.26
0:35
0.09
0.21
0.35
Layer
8
6.32
025
0.26
0.30
12
0.30
0:24
0.33
0.25
0.34
0.31
0.08
027
0.35
0.29
6.11
0.32
0.08
0.25
0.15
0.33
0.26
0:31
0.09
025
0.35
0.27
0.13
9.32
0.28
0:20
0.12
0.31
0.32
0.08
0.23
0.36
0.15
0.31
0.33
0.30
0.29
0.28
0.24
0.32
0.10
0.30
0.29
0.11
0.10
0.73
0.31
0.32
0.09
0.21
0.34
0.09
0.28
0.30
000
0.32
0.25
0:31
0.08
0.23
0.31
Pxneet
Evnort
Panert
Figure 11. The frequency anslysis of TinyLLaMa-44..BBMMM on BoolQ, MMLU and sampling dataset of C4.
TinyLLama4x1.18 ARC-challenge
TinyLLama4x1. ARC-easy
TinyLLama4x1.11 1BB OpenBookQA
0:30
0:18
0:33
0.10
5:38
0.39
3.33
0.25
0.10
6.30
0.35
0.24
9.30
0.31
0.35
0.31
0.89
623
037
0.30
0.08
637
0.34
0.35
0.29
0.30
611
0.29
530
0.3
0.38
6.7
0.09
0.24
0.12
0.26
0.38
0.24
8:33
0.25
0.35
622
0.13
027
0.57
0.32
0.11
6.23
0.35
032
0.34
0.31
0.11
0.23
0.3
6.32
0.09
0.88
0.31
0.25
Layer
33
0.32
6.1
0.29
0.79
0.29
0:2
0.27
0.32
0.79
0.11
0.75
0.14
0.29
0.12
0.25
0.34
0.31
0.10
0.20
0.28
0.11
0.25
0.36
831
0.35
0.28
0.00
6.10
0.33
0.30
0.31
5:00
0.37
0.34
0.77
0.1
0.32
0.09
0.22
0.37
0.33
3.09
0.22
0.37
0.35
0.06
0.26
0.33
0.35
0.10
8.19
0.36
0.55
8.10
0.19
0.36
0.32
0.10
0.21
0.37
8.30
0.37
0.36
0.06
0.24
0.31
9.10
0.35
Export
Expert
Expert
Figure 12. The frequency anslysis of Txy on ARC-e, ARC-e and OpenBook
26

---

Retraining- Free Merging of Sparse MoE via Hicrarchical Clustering
TinyLLama4x1.18
TinyLLama4x1. Winogrande
TinyLLama4x1..11 HellaSwag
0.13
0.2
9.36
0:13
0:00
D:33
037
0.31
5:31
0.09
035
035
0.9
012
0.33
0.33
069
D37
0.40
633
0.29
6:34
669
631
0:11
0.08
0.30
0.30
0:30
0.10
0:34
0.2
067
632
0:44
033
0.1
0.89
0.8
0.34
0.30
0.39
0.33
0.25
0.31
0.26
0.11
039
832
0.12
0.52
0.33
0.9
025
Layer
22 92
6.31
0.32
0.77
0.28
Layer
0.36
0.9
631
524
0.25
0.12
6.2
0.36
8:41
0.18
0.77
8:90
0.33
0.09
631
5.55
0.20
018
0.11
0.32
0.28
0.99
0.33
0:34
067
6.32
626
0.33
0.30
6.38
0.88
8:35
0.7
0.31
0.28
0:38
0:6
6.31
6:25
0.15
0:30
0:32
0.33
0.08
6.30
0.30
0:35
6:40
064
0.28
0:28
0:38
0.09
0.23
0.32
0:42
0.09
6:43
0:4
0:25
0.09
0.27
0.36
0.06
0.33
0.34
0.27
805
0.44
062
0.27
0.37
0.06
0.29
0.28
Expert
Expert
Figure 13. The frequency anslysis of TanyL.LaM--411. 1B-Mo... on RTE, Winogrand'e and HellaSwag.
27


Reviews:
[{"review_id": "rfu38E2H85", "reviewer": "Reviewer_A14Y", "summary": "This paper proposes an untrained sparse expert merging strategy, HC-SMoE, which reduces the parameters of  Sparsely activated Mixture of Experts (SMoE) models through expert merging. The clustering strategy adopts hierarchical clustering based on expert output similarity to progressively group experts, while the merging strategy selects frequency-weighted merging to maintain flexibility. Experiments show that HC-SMoE, when applied to Qwen1.5-MoE-A2.7B-Chat and Mixtral 8x7B, achieves an average accuracy drop of 8% and 8.7%, respectively, when reducing the number of experts by half, outperforming existing pruning and merging methods.\n\n## update after rebuttal\n\nThanks for the authors' feedback, which addresses most of my concerns. I have updated my score accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.\n\n**Q1.** Theoretical analysis is still lacking: the theoretical advantages of hierarchical clustering are described vaguely.\n\n**Response:**\nPlease refer to theoretical justification part in link.\n\n- [Theoretical justification](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Theoretical_Justification.md)\n\n---\n\n\n**Q2.** However, the generalization capability to multimodal domains is not covered, and details on calibration dataset sampling are missing, which may introduce bias. \n\n**Response:**\nWe appreciate the question from the reviewer. This paper concentrates on the MoE-based language domain. Extension to multimodal domains represents an interesting direction for future research but exceeds the current scope of this work. Future investigations could profitably explore the application of HC-SMoE to additional modalities.\n\n---\n\n**Q3.** Comparison of Memory Consumption, Computational Cost, and Overheads\nWe have incorporated a table that compares the runtime and memory usage of HC-SMoE against various baselines. The results demonstrate that HC-SMoE achieves competitive runtime and memory efficiency across different models while maintaining superior performance on benchmarks.\n\n- [Mixtral model experiments](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_runetime_on_mixtral8x7B.md)\n- [Qwen model experiments](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_runtime_on_qwen.md)\n\n---\n\n**Q4.** Choice of Euclidean Distance as Distance Metric for Expert Outputs\n\n**Response:**\nIn response to the reviewer’s question about the choice of Euclidean distance as the distance metric for expert outputs in hierarchical clustering, we selected the Euclidean distance due to its effectiveness in measuring the similarity between averaged expert outputs, which are high-dimensional vectors. This choice aligns with prior work such as [3][4], where the Frobenius norm was used to measure pruning error between the original model and the pruned model. In our case, the Euclidean distance is a natural fit, given that the expert outputs are vectors in Euclidean space.\n\nIn addition, we show in Table 20 of the paper that even with Euclidean distance, our approach yields high cosine similarity in the final layer output, demonstrating that the choice of Euclidean distance does not hinder performance. We also present empirical results indicating that HC-SMoE produces the best cluster quality, as measured by the silhouette score and Dunn index, when compared to K-means. Please note that in Table 20 the similarity scores between different metrics cannot be directly compared, as the silhouette score and Dunn index are computed on different bases.\n\n---\n\n**Q5.** What is the specific sampling scheme for the calibration dataset?\n\n**Response:**\nRegarding the calibration dataset, we exactly follow the protocol in [4], using a sampling scheme where we randomly select 32 sentences from the C4 dataset, with each sentence containing 2,048 tokens. The same sampling scheme is used across all experiments to ensure fairness and reproducibility.\n\n---\n\n**Q6.** Related Work on Model Soup\n\n**Response:**\nWe acknowledge the related work on Model Soup and its connection to our approach. Our average merging technique shares similarities with the concept of uniform model soup, where multiple models are averaged to create a unified model. However, while uniform model soup typically involves the combination of multiple models into a single entity, HC-SMoE focuses on merging a set of experts into $r$ clusters, where $r<n$. Our approach exhibits greater complexity, as it incorporates expert clustering based on similarity metrics and a frequency-weighted merging process. We will include citations to the relevant work on uniform model soup in the paper to highlight this connection and differentiate our methodology.\n\n---\n\nDue to space limitations, we can only provide concise responses here. We have more detailed and comprehensive answers regarding the concerns on limitations discussion of HC-SMoE and other questions, which we look forward to discussing thoroughly with the reviewer in the next phase of the review process.", "claims_and_evidence": "The main claims of the paper are generally supported by experiments, but some evidence needs further clarification:\n1.\tUsing expert outputs for hierarchical clustering similarity is claimed to be superior to existing router logit and weight metrics, and experiments on Qwen 45x2.7B validate the effectiveness of output features.\n2.\tRegarding task independence, experiments are conducted on the C4 dataset and eight zero-shot tasks, along with two domain-specific datasets in the appendix, which partially verify task independence. However, the generalization capability to multimodal domains is not covered, and details on calibration dataset sampling are missing, which may introduce bias.\n3.\tRegarding the efficiency of MoE models, comparisons with other baselines in terms of memory consumption and computational cost are absent.", "methods_and_evaluation_criteria": "Methodological soundness: Hierarchical clustering based on expert output similarity is intuitively reasonable, but the rationale for directly adopting Euclidean distance is not discussed.\nEvaluation criteria: Using zero-shot tasks and the C4 calibration dataset aligns with the task-independent setting, but the specific sampling scheme for the calibration dataset is not explained.", "theoretical_claims": "Compared to other single-step grouping methods, the paper provides some qualitative analysis of hierarchical clustering, but no detailed theoretical proof is presented in the main text.", "experimental_designs_or_analyses": "In the experimental design, the paper does not compare the memory consumption, computational cost, and other overheads of the SMoE model introduced by other baselines.", "relation_to_broader_scientific_literature": "Regarding the field of model merging, the paper only discusses ZipIt but does not address the theoretical connections between HC-SMoE and general model merging techniques (e.g., model soups), making it seem isolated.", "essential_references_not_discussed": "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time", "other_strengths_and_weaknesses": "Strength:\nInnovatively introduces hierarchical clustering based on expert output into SMoE expert merging, effectively reducing model memory requirements and computational costs.\nLarge-scale experiments on Qwen and Mixtral demonstrate practical deployment potential, indicating certain applicability.\nWeakness:\nTheoretical analysis is still lacking: the theoretical advantages of hierarchical clustering are described vaguely.", "other_comments_or_suggestions": "It would good to include some discussion of limitations of this work.", "questions_for_authors": "Q1: Why was Euclidean distance chosen as the distance metric for expert outputs in hierarchical clustering?\nQ2: What is the specific sampling scheme for the calibration dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55EouTGkRK", "reviewer": "Reviewer_YKXx", "summary": "This paper proposes a simple and effective task-agnostic method named HC-SMoE to merge the experts in pre-trained mixture-of-expert models. HC-SMoE first obtains expert outputs on a calibration dataset, and then conducts a hierarchical clustering of the experts based on these outputs. The experts inside a cluster are then merged together. Experiments show that HC-SMoE applied on Qwen and Mixtral achieves better performance than the baseline methods in most cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.\n\n---\n\n**Q1.** Please consider adding baseline results to see if HC-SMoE still outperforms the baselines when the calibration dataset and the evaluation dataset have larger distribution differences.\n\n**Response:**\nWe acknowledge the reviewer's valuable inquiry regarding HC-SMoE's performance under distribution shifts. To evaluate the robustness of HC-SMoE against calibration dataset distribution shifts, we conducted additional experiments with MATH and CodeQA as calibration datasets. The results demonstrate the following:\n- HC-SMoE consistently **maintains superior or equivalent performance compared to all baselines** across all configurations on Qwen1.5-MoE-A2.7B-Chat, which demonstrates its capacity to generalize effectively across diverse calibration distributions.\n- When implementing 25% pruning with MATH calibration on Mixtral8x7B, S-prune exhibits marginally superior performance compared to HC-SMoE. However, S-prune demonstrates significantly inferior performance in all other experimental scenarios.\n- F-prune exhibits substantial performance degradation when utilizing MATH and CodeQA, which indicates that **pruning methodologies based solely on frequency or routing scores lack stability**, whereas HC-SMoE maintains robust performance regardless of the calibration dataset employed.\n\nThe experimental results with calibration datasets MATH and CodeQA are presented as follows. The red color indicates performance decreases relative to calibration dataset C4, while the green color signifies performance improvements. We excluded O-prune from these experiments as its original publication [4] provides comprehensive analysis regarding the impact of calibration datasets.\n\n- [Qwen on MATH](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_calib_math_on_qwen)\n- [Qwen on CodeQA](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_calib_codeqa_on_qwen)\n- [Mixtral on MATH](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_calib_math_on_mixtral8x7B)\n- [Mixtral on CodeQA](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_calib_codeqa_on_mixtral8x7B)\n\n---\n\n**Q2.** The proposed method requires a calibration dataset. How large is this dataset in the experiments? How sensitive is the performance of the proposed algorithm to the size of this dataset?\n\n**Response:**\nSection 4.1 of our original manuscript provides comprehensive details regarding the calibration dataset size. To further evaluate the sensitivity of our method to dataset size, we conducted an additional experiment using Qwen with 50% expert parameter pruning while varying the calibration dataset size (16, 32 (original), and 64 examples). The results indicate that the average performance across eight zero-shot benchmarks remains remarkably consistent, which highlights the robustness of HC-SMoE with respect to calibration dataset size.\n\n- [Different size of calibration dataset](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/ablation_different_size_of_calibration_dataset.md)\n\n---\n\n**Q3.** Related Works on EEP\n\n**Response:**\nTo provide a clear comparison, we include a table that summarizes the performance and runtime of EEP and HC-SMoE. It merits mention that the reported accuracy for BoolQ and RTE in the EEP paper differs significantly from our results, likely due to differences in evaluation protocols—we use the EleutherAI Language Model Evaluation Harness.\n- [EEP comparison table](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/eep_comparison.md)\n- [Mixtral8x7B-Instruct result](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Tables/experiments_on_mixtral8x7B-Instruct.md)\n\n---\n\n**Q4.** Fix-Dominant Merging\n\n**Q4.1** What is the definition of \"dominant\" in Figure 4?\n\n**Response:**\nThe definition of “dominant” in fixed-dominant merging refers to a “concept” of dominant expert within a cluster, since it would be the only one expert within the cluster to preserve the original weight vector order. In HC-SMoE, the dominant expert refers to the expert that exhibits the closest proximity to the cluster center. \n\n**Q4.2** Could you explain in more detail how the merging is done?\n\n**Response:**\nOur default fix-dominant merging methodology implements simple average merging, wherein all experts within a cluster contribute equally to the merged representation. \n\n---\n\nDue to space limitations, we can only provide concise responses here. We have more detailed and comprehensive answers regarding the concerns on section 3.2, section 4.3 and table 4, fix-dominant merging, explanation on table 4 and 5, as well as EEP comparison, which we look forward to discussing thoroughly with the reviewer in the next phase of the review process.", "claims_and_evidence": "* Section 3.2 claims that \"effective clustering enables our method to preserve the capabilities of the original model across diverse merging strategies (Section 3.2.3).\" However, the experiment results show that the method does have performance loss than the original model in most cases (see Table 2 and Table 3).\n\n* Section 4.3 claims that \"Hierarchical clustering exhibits stability due to its deterministic nature. This stability is evidenced by consistent performance across benchmarks and the highest average scores.\" However, Table 4 shows that the results have a large variance across different settings. For example, on BoolQ, the performance ranges from 0.3792 to 0.7948.", "methods_and_evaluation_criteria": "* I read Section B.2, but still find it hard to understand the technical details of the Fixed Dominant Merging approach. For example: \n  - What is the definition of \"dominant\" in Figure 4? \n  - Line 577 states that \"The merging process then applies an appropriate weighting scheme, such as average merging, preserving the dominant expert’s weight feature order while simplifying the merging process.\" Could you explain in more detail how the merging is done?\n  - What do \"feature\" and the stars mean in Figure 4?", "theoretical_claims": "The paper does not have theoretical claims.", "experimental_designs_or_analyses": "* Section 4.1 mentioned the experiments of checking how the choice of the calibration dataset affects the results: \"To further validate the independence of HC-SMoE from the calibration dataset, we construct two additional datasets from MATH (Hendrycks et al., 2021b) and CodeQA (Liu & Wan, 2021). Please refer to our Appendix B.3 for more details.\" This is an important experiment. I checked Appendix B, but did not find the results of the baseline methods. Please consider adding baseline results to see if HC-SMoE still outperforms the baselines when the calibration dataset and the evaluation dataset have larger distribution differences.", "relation_to_broader_scientific_literature": "The key difference to prior work is (1) the use of hierarchical clustering to find the expert merging sets, and (2) the expert similarity is measured based on expert outputs. These are simple changes, and the results show clear improvements.", "essential_references_not_discussed": "EEP [1] also proposes a training-free algorithm to do expert pruning and merging for MoE models. I understand that it is hard to compare every method in the experiments. But at least, the paper should discuss it, given the high relevance and the fact that EEP has appeared online more than 6 months before the ICML submission deadline.\n\nIn addition, given the existence of this paper, the paper might need to tone down some of the claims, such as \"task-specific expert pruning ... often necessitate extensive finetuning to maintain performance levels\" in Section 1.\n\n[1] https://arxiv.org/abs/2407.00945", "other_strengths_and_weaknesses": "Strengths:\n\n* Overall, the paper is well-written. I really appreciated that the paper not only discusses the proposed approach, but also discusses the rationale behind the design choices and why the choices could be better than other alternatives. These discussions provide useful insights to the readers.", "other_comments_or_suggestions": "Typos:\n\n* Line 51: \"from (Li et al., 2024)\" --> \"from Li et al. (2024)\"\n* Line 115: \"F-prune and M-SMoE\" --> \"F-prune, and M-SMoE\"\n* Line 142: \"Fig.3.\" --> \"Fig. 3.\"\n* Table 2: The best number is not made bold in the last column of the Qwen 30x2.7B rows.\n* Line 584: ”feature” --> \"feature\"", "questions_for_authors": "In addition to the questions mentioned before, I also have the following questions:\n\n* The proposed method requires a calibration dataset. How large is this dataset in the experiments? How sensitive is the performance of the proposed algorithm to the size of this dataset?\n\n* I found that some of the results across different tables are not consistent. For example, the last row of Table 5 does not match the numbers in Table 4.\n\n\n# Review summary\n\nOverall, the paper is of good quality. However, given that there are too many unclear points as discussed above, I have to give a negative score. That said, I believe it is feasible to address all these questions during the rebuttal. I would appreciate it if the authors can help clarify these questions, and I would be happy to adjust the score accordingly.\n\n# Reply to \"Reply Rebuttal Comment by Authors\"\n\nThank the authors for the further clarification. The numbers in the updated Table 5 still do not match the ones in Table 4. Is it because some settings (e.g., the model) are different? Please make it clearer in the revision. \n\nSince most of my concerns are addressed, I increase the score from 2 to 3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qVC03qnzYU", "reviewer": "Reviewer_39Jv", "summary": "The paper presents HC-SMoE, a new framework for reducing SMoE model parameters that doesn't require retraining and works across different tasks. HC-SMoE uses hierarchical clustering on expert outputs and frequency-weighted merging, which offers two main benefits over previous approaches. First, it uses iterative comparisons to create expert groups, leading to better diversity between groups and similarity within groups. Second, it measures similarity based on expert outputs rather than router logits, making it more generalizable across different datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.\n\n**Q1.** It was not entirely clear to me why Li et al. (2024) proposal is one-shot and why HC is iterative. \n\n**Response:**\nThe fundamental distinction between these approaches lies in their underlying methodologies. Li et al.'s expert pruning operates through a one-shot mechanism, wherein only the top $r$ experts with the highest routing scores are retained based on a single evaluation of the data. This process involves computing routing scores by averaging results from a forward pass across the entire calibration dataset, followed by sorting experts according to these averaged scores to identify the top rr r candidates. This methodology proceeds in a direct, non-iterative manner.\n\nIn contrast, our HC-SMoE method employs an iterative approach. Hierarchical clustering necessitates multiple sequential steps to systematically merge experts into clusters. Each step selects the optimal pair of clusters (or experts) to combine based on minimizing the average intra-cluster distance. This iterative procedure continues until precisely $r$ clusters are established, ensuring that each clustering decision optimizes the process by minimizing clustering error at every iteration.\n\n**Q2.** It would have been nice to include an algorithm in the paper.\n\n**Response:**\nWe acknowledge the reviewer's recommendation regarding a more detailed description of the algorithm, and we have provided [the link to algorithm](https://anonymous.4open.science/r/ICML_Rebuttal-0632/algorithm1.png). Specifically, in Algorithm 1, we outline the steps of hierarchical clustering and expert merging in our methodology.\n\n**Q3.** When using HC, are the experts merged at every step when creating the dendrogram? Or the experts are merged only at the desired step of say 25% sparsity?\n\n**Response:**\nTo clarify the merging process: The procedure initially involves clustering experts through the construction of a dendrogram, which provides a hierarchical representation of expert relationships. Upon completion of the dendrogram, we proceed to merge the experts at the final stage to achieve the specified sparsity level. This means that experts are not merged at every step of the dendrogram creation. Instead, they are merged only once the final clustering decision is made, based on the desired sparsity or number of clusters.\n\nWe hope this explanation adequately addresses the reviewer’s concerns.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, the methods, models and datasets make sense.", "theoretical_claims": "No theoretical analysis", "experimental_designs_or_analyses": "The experimental design and analysis looks sound.", "relation_to_broader_scientific_literature": "The paper is contextualized properly in the context of broader scientific literature.", "essential_references_not_discussed": "Related works that are essential to understanding the (context for) key contributions of the paper are discussed.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well written\n- The proposal of averaged expert output seems novel and using HC to find cluster seems appropriate.\n- The experiments are thorough\n\nWeaknesses:\n- It was not entirely clear to me why Li et al. (2024) proposal is one-shot and why HC is iterative. It would have been nice to include an algorithm in the paper. When using HC, are the experts merged at every step when creating the dendogram? Or the experts are merged only at the desired step of say 25% sparsity?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VShmackBZO", "reviewer": "Reviewer_cVaE", "summary": "This paper introduces HC-SMoE, a retraining-free framework for merging experts in Sparsely Activated Mixture-of-Experts (SMoE) models via hierarchical clustering. The key idea is to group experts based on their output similarities over a calibration dataset, followed by frequency-weighted merging to reduce model parameters while preserving performance. The authors validate HC-SMoE on Qwen and Mixtral models, demonstrating superior performance over pruning baselines (e.g., O-prune, S-prune) and merging methods (e.g., M-SMoE) across multiple zero-shot tasks. The main contributions include: (1) output-based similarity metrics for clustering, (2) hierarchical clustering for improved robustness, and (3) empirical validation across diverse benchmarks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable feedback and effort spent on the review, and would like to respond to the reviewer’s questions as follows.\n\n**Q1.**  Theoretical Gaps: No formal analysis of clustering quality or merging stability.\n\n**Response:**\nWe appreciate the reviewer's suggestion regarding theoretical analysis. We would like to direct the reviewer to refer to the general response section. Please refer to the theoretical justification link below.\n- [theoretical justification](https://anonymous.4open.science/r/ICML_Rebuttal-0632/Theoretical_Justification.md)\n\n---\n\n**Q2.** Experiments are comprehensive but lack latency/throughput comparisons (Table 19 only reports FLOPs and memory). Inference efficiency gains from merging are unclear.\n\n**Response:**\nWe thank the reviewer for this observation. Our primary objective focuses on reducing parameter count in MoE models without retraining while maintaining computational efficiency. Table 19 demonstrates significant reductions in FLOPs and memory usage, which highlights the efficiency gains achieved through our approach.\n\nIt is important to note that our method preserves the top-K routing mechanism, wherein each token continues to be assigned to $K$ experts per MoE layer, the same as the original model. As a result, when we reduce the number of experts in each layer to $r$ while ensuring $r \\ge K$, the inference cost remains equivalent to that of the original model. While merging reduces storage and computational requirements per expert, the routing mechanism dictates that inference cost per token primarily depends on K rather than the total number of experts.\n\n\n---\n\n**Q3.** Does HC-MoE cause a huge impact on the semantic space of the original model? It is a pity that neither of these issues is explicitly explained in the text.\n\n**Response:**\nTo address concerns regarding potential semantic shifts, we have incorporated t-SNE visualizations that compare expert representations before and after merging. These results demonstrate that HC-SMoE effectively preserves the model's semantic coherence, as the expert distributions maintain substantial consistency after the merging process.\n\nSpecifically, we visualize the first MoE layer of Mixtral-8x7B, utilizing the same calibration dataset described in Section 4.1. We collect 65,536 output vectors per expert, each with a hidden dimension of 4,096, by directing identical MoE input tokens to all experts. Each point in the t-SNE plot represents the average of 128 token outputs, which then undergoes projection into two dimensions using sklearn.manifold.TSNE with n_components = 2. The analysis yields several key observations:\n- With perplexity = 8, the t-SNE visualization of the original model reveals a clear eight-cluster structure, which corresponds to the eight experts.\n- After applying HC-SMoE to reduce the expert set to six experts, the resulting t-SNE plot maintains a well-defined six-cluster structure, despite some overlap. This indicates that HC-SMoE maintains expert specialization to a significant extent, and preserves the output distribution of the original model.\n\nIt is important to note that t-SNE visualization at the single-token level would appear as random noise without discernible cluster structure. This occurs because individual token embeddings exist in high-dimensional space and do not inherently form clusters without appropriate aggregation.\n\n- [t-SNE of each expert’s output of the original Mixtral8x7B's first layer](https://anonymous.4open.science/r/ICML_Rebuttal-0632/t-SNE/t-SNE-8e_layer_0_output.png).\n- [t-SNE of each expert’s output of first layer after HC-SMoE which reduced each layer in Mixtral8x7B to 6 experts](https://anonymous.4open.science/r/ICML_Rebuttal-0632/t-SNE/t-SNE_6e_layer_0_output.png).\n- [t-SNE of each expert’s output of the original model layer where each point indicates a single output token](https://anonymous.4open.science/r/ICML_Rebuttal-0632/t-SNE/t-SNE_150tokens_for_each_expert_output.png).\n\n---\n\nDue to space limitations, we can only provide concise responses here. We have more detailed and comprehensive answers regarding the concerns on ethical statement, recommend related works on cluster-based routing MoE as well as [figure 2's enhancement](https://anonymous.4open.science/r/ICML_Rebuttal-0632/figure2_modified.png) and other questions, which we look forward to discussing thoroughly with the reviewer in the next phase of the review process.", "claims_and_evidence": "The central claim—that HC-SMoE outperforms existing pruning/merging methods—is are supported by extensive experiments (Tables 2–3, 6–7). However, theoretical justification for why hierarchical clustering is optimal is lacking.", "methods_and_evaluation_criteria": "The article lacks rigorous analysis and mathematical modeling of the rationality of the method, and the article mainly provides empirical explanations.", "theoretical_claims": "No theoretical proofs are provided. For example, the claim that hierarchical clustering \"produces theoretically guaranteed groupings\" (Section 1) is unsupported. A theoretical analysis of clustering robustness or error bounds is missing.", "experimental_designs_or_analyses": "Experiments are comprehensive but lack latency/throughput comparisons (Table 19 only reports FLOPs and memory). Inference efficiency gains from merging are unclear.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Key Contributions vs. Literature:\n1.\tThe work builds on SMoE compression methods like TSEP (Chen et al., 2022), O-prune (Lu et al., 2024), and M-SMoE (Li et al., 2024). The novelty lies in hierarchical clustering for merging, contrasting with prior pruning or single-pass grouping. \nMissing Citations:\n1.\tCluster-based routing MoE has been proposed in many papers before(e.g., \"On the Benefits of Learning to Route in Mixture-of-Experts Models\" and \"Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge\"), but it is not proposed and referenced in this paper. Such papers have a more detailed analysis of the clustering phenomenon in the representation space, but in this paper, it is missing.", "other_strengths_and_weaknesses": "Strengths:\n1.\tPractical Impact: HC-SMoE offers a deployable solution for resource-constrained settings.\n2.\tScalability: Validated on large models (Mixtral 8x7B) with significant parameter reduction.\nWeaknesses:\n1.\tTheoretical Gaps: No formal analysis of clustering quality or merging stability.\n2.\tSemantic Preservation: Claims about preserving semantic spaces are unsubstantiated (see Questions).\n3.\tEthical Statement: Missing required impact statement on societal/environmental implications.\nClarity:\n1.\tThe method is well-described, but Figure 2 (clustering illustration) is too simplistic.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The proposed clustering method is multi-level, but it takes into account two premises: a. Many papers pointed out that experts of token-level MoE did not have a preference for actual professional semantics(e.g., \"DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models\" and \"OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models\"). b. The professional semantic preferences of different experts are also not clearly indicated in this article. Then the questions that authors should explain are: \na. How does HC-MOE guarantee that experts merging based on Hierarchical Clustering will not produce semantic conflicts? \nb. Does HC-MoE cause a huge impact on the semantic space of the original model? It is a pity that neither of these issues is explicitly explained in the text.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "As the impact statement was not provided in the article as required, I cannot comment on this."}}]

Author rebuttal:

Meta-review: [1] The main claims of the paper are generally supported by experiments [Reviewer A14Y, cVaE]
[2] This paper proposes a simple and effective task-agnostic method named HC-SMoE to merge the experts in pre-trained mixture-of-expert models. [Reviewer YKXx]
[3] The paper presents a new framework for reducing SMoE model parameters that doesn't require retraining, reduces the parameters of Sparsely activated Mixture of Experts (SMoE) models through expert merging and works across different tasks, making it more generalizable across different datasets. [Reviewer 39Jv]
Decision: Accept (poster)