LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
Authors: Xiaonan Nie
Qibin Liu
Fangcheng Fu
Shenhan Zhu
Xupeng Miao
Xiaoyang Li
Yang Zhang
Shouda Liu
Bin CUI
Venue: NeurIPS 2024 poster (2024)
Keywords: mixture of experts
locality-sensitive hashing
communication optimization
Primary area: infrastructure

Abstract: Larger transformer models perform better on various downstream tasks but require more cost to scale up the model size. To efficiently enlarge models, the Mixture-of-Expert (MoE) architecture is widely adopted, which consists of a gate network and a series of experts and keep the training cost constant by routing the input data to a fixed number of experts instead of all.
In existing large-scale MoE training systems, experts would be distributed among different GPUs for parallelization, and thus input data requires additional all-to-all communication to access the target expert and conduct corresponding computation. 
However, upon evaluating the training process of three mainstream MoE models on commonly used GPU clusters, we found that the all-to-all communication ratio averaged around 45\%, which significantly hinders the training efficiency and scalability of MoE models.

In this paper, we propose LSH-MoE, a communication-efficient MoE training framework using locality-sensitive hashing (LSH). 
We first present the problems of scaling MoE training in existing systems and highlight the potential of exploiting token similarity to facilitate data compression.
Then, we introduce an efficient LSH-based compression technique, which utilizes the cross-polytope hashing for rapid clustering and implements a residual-based error compensation scheme to alleviate the adverse impact of compression. 
To verify the effectiveness of our methods, we conduct experiments on both language models (e.g., RoBERTa, GPT, and T5) and vision models (e.g., Swin) for both pre-training and fine-tuning tasks. The results demonstrate that our method substantially outperforms its counterparts across different tasks by 1.28-2.2$\times$ of speedup.

LSH-MoE: Communication-effiienn MoE Training via Locality-Sensine Hashing
Xiaonan Nie Qibin Liu'
Fangcheng Fu'
Shenhan Zhu'
Xupeng Miao²
Xiaoyang Li
Yang Zhang Shouda Liu Bin Cui
Peking University Purdue University SByteDance
shenhan. zhu, bin. cui cuil@pku. edu. cn xxupengppurdue. edu ${lixiaoyang. zhangyang. elfin, Iiuhhoudaa@byteddneeee
[xiaonann nie, 2101212782 ccchengff 2,
Abstract
Larger transformer models always perform better various tasks but require more costs to scale up the model size. To efficiently enlarge models, the mixture-of- experts (MoE) architecture is widely adopted, which consists of gate network and a series of experts and keep the training cost constant by routing the input data fixed number of experts instead of all. In existing large-scale MoE training systems, experts would be distributed among different GPUs for parallelization, and thus input data requires additional all-to-all communications to access the target experts and conduct corresponding computations. However, upon evaluating the training process of three mainstream MoE models commonly used GPU clusters, we found that the all-to-all communication ratio averaged around 45% %, which significantly hinders the efficiency and scalability of training MoE models. In this paper, we propose LSH-MoE, a communication-efien work using locality-sensitiv hashing (LSH). We first present the problems of scaling MoE training in existing systems and highlight the potential of exploiting token similarity to facilitate data compression. Then, we introduce efficient LSH-based compression technique, which utilizes the cross-polytope hashing for rapid clustering and implements a residual-bare -ased error compensation scheme to alleviate the adverse impact compression. To verify the effectiveness of our methods, conduct experiments on both language models (c.g.. RoBERTa, GPT, and T5) and vision models (c.g., Swin) for pre-training and fine-tuning tasks. The results demonstrate that our method substantially outperforms its counterparts across different tasks by 1.28x 2.2x of speedup.
MoE training frame-
Introduction
In recent years, large-scale pre-trained models have significantly advanced the performance of deep learning across various complex tasks, including computer vision [ 20 natural language
processing [3] 7 28), and multi-modal learning [19]. Commonly referred to as foundation models,
these pre-trained models are primarily built on Transformer architectures 34] and undergo extensive pre-training on large datasets, utilizing substantial GPU resources. OpenAl has validated the scaling law for large language models [15] and suggests that increasing the model's parameter size, the volume of training data, and the duration of training can significantly enhance the model's performance. However, this approach results in considerable rise training costs, making the development of foundation models extremely expensive.
Xiaonan Nie, Qibin Liu, Fangcheng Fu, Shenhan Zhu, and Bin Cui are with the School of Computer Science and Key Lab of High Confidence Software Technologies (MOE), Peking University. Bin Cui is also with the Institute of Computational Social Science, Peking University (Qingdao).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To reduce the high computational costs, the sparse mixture-of-expeee (MoE) architecture is often adopted, which comprises a sparse gate network and series of expert networks. This architecture routes input data to only a subset of experts, resulting in sparse activation of the experts and thereby reducing the model's computational FL.OPs (float point operations) as well as training costs. Prominent models such as Google's s
Transformer [9], ST-MoE [4T, Meta's 's Hash Layer [31] and Mistral-Al's mixtral models [[4] have successfully implemented this design, demonstrating improvements in both performance and efficiency with MoE models.
Meanwhile, effectively scaling the training of MoE models across hundreds or even thousands of GPUs remains a significant challenge. Researchers from Google have proposed the expert parallelism approach [17], which replicates the gating network on each GPUs and distributes different experts across multiple GPUs for parallel processing. Specifically, each input token is initially processed by the gating network to select the appropriate expert, after which it is routed to the designated experts via peer-to-peer (P2P) network communication. Once the designated experts complete their computation, the token is returned to the original GPU for further processing through an additional P2P communication. Since each GPU typically needs to exchange data with many other GPUs, these P2P transmissions results in an all-to-all communication pattern. Moreover, because the computation of the expert network relies on the outcomes of these communications, the communications cannot be effectively overlapped with ongoing computations. This dependency creates a significant performance bottleneck in model training across most commonly used GPU clusters. We conducted experiments on three widely-used -used MoE models, including RoBERTa-Mode GPT-MoE and Swin-MoE. on four A100 servers, each with cross- moos-me bandwidth of 200Gb/s. The results, shown in Figure 3 reveal that the time cost of all-to-all communication constitutes an average of 45% and can reach up to 67% of the total model training time.
Existing methods to improve distributed MoE training on bandwidth- limited clusters tackle communi- cation challenges in various ways. TA-MoE 4 reduces cross-machine communication by adjusting the gating network to favor experts on the same server, while Pre-gated MoE [33 reduces dependency between communication and computation through pre-gating mechanism that plans token routing in advance. However, both approaches require modifications to the gating mechanism and model structure, limiting their universal applicability. DeepSpeed-Mod [29] introduces PR-MoE, which selects one expert plus a shared expert, halving the all-to-all communication load. SCoMoE [40] organizes all-to-all communication by structuring data transfers along different dimensions and controlling data volumes across network levels, and also clusters tokens to improve routing. However, none of these works consider reducing the All-to-All communication volume in MoE training by compressing the forward activations. Therefore, they can be intergrated with our method for further improvement.
In this paper, we present LSH-MoE, a communication-effiiceen MoE training framework that leverages Socality-sensitie hashing to group similar tokens. Our key contributions
follows:
We begin by identifying key challenges scaling MoE training existing systems, noting that all-to-all communication constitutes an average of 45% of the total training time. Addi- tionally, we investigate the potential of using token similarity to facilitate data compression to reduce communication costs.
We propose an efficient LSH-based compression technique that employs cross-polytope hashing for rapid clustering. This approach transmits only the clustering centroids, sig- nificantly reducing communication costs. To further enhance accuracy, we implement residual-based error compensation scheme to mitigate the negative effects of compression.
Through extensive experiments with language models (RoBERTa-MoE: GPT-MoF, and T5-MoE) and vision models (Swin-MoE), across both pre-training and fine-tuning tasks, we demonstrate that our method maintains model quality while achieving speedup of 1.28x 2.2x in end-to-end training time.
2 Background
2.1 Mixtures-of-Exper Architecture
To enhance the training efficiency of Transformer models, William et al. (2022) 9 introduced an innovative paradigm, the sparse mixture--f-eexperrs (MoE) architecture, illustrated in Figure]

---

f(x E()x)
AlltoAl
Alltoi
16500)
X
Cating Network
1 Expert
GPU
(x)
X1
Gating Network
2 Expert
GPU
1a
2 Export
3rs Expert
1ph Expert
Node
Expert
X2
Gating Network
3 Expert
GPU
X3
Gating Network
4th Expert
GPU
Gating Network
G:R [1,N]*
Node
Intra-ndee Comm.
Inter-aode Comm.
Figure 2: Training Mixture-of offture-of-Experts on multiple GPUs expert parallelism.
Figure I: cexture-of-Experti on single GPU.
This architecture effectively balances parameter capacity and training costs, and comprises two key components: an expert network (E) and a sparse gate network (g). It is evident that MoE models, with equal number of active parameters per input, significantly surpass the performance of dense models. This breakthrough has also catalyzed further research and their application across various industries, as highlighted by numerous subsequent studies |
14
22
23
25
30 39
The expert network to experts, denoted as {E,}, fi where N represents the number of experts. Additionally, E, (x) denotes the output produced when the input z is processed by the i-th expert. Each expert trained to excel in a specific sub-task, such in multi-task learning, seen in language modeling and multi-modal learning, thereby increasing the overall model capacity. In foundational models, the MoE layer often serves substitute for the traditional feed-forwart network (FFN) layer. Within each MoE layer, each FFN function works as an individual expert, significantly enhancing the model's capability to process diverse and complex data inputs.
composed of multiple specialized and separate networks, commonly referred
handle specific segments of data, as
The gating network plays a crucial role in the sparse MoE architecture. For example, in K-way gated MoE. system, the gating network outputs set of integers Equation to determine which experts should be activated. This decision is based on the characteristies of the input itself, allowing for dynamic and efficient allocation of computational resources. By only processing each input token with a selected subset of the expert network, the MoE model achieves computation sparsity, effectively decoupling parameter capacity from training costs.
G:RM
N1K
Through the integration of multiple specialized experts, described by Equation 2 the sparse MoE model is capable delivering more accurate and efficient predictions as (x)). This is achieved by leveraging the specialized knowledge embedded within each expert, combined with the strategic input allocation managed by the gating network.
f f(x)
Σ (,) 1E9(x)
While MoE's primary advantage is decoupling parameter capacity from network cost, key challenge lies in learning the gating parameters effectively, as the output's sparsity makes non-differentiable. Consequently, much of the research in the MoE field has centered on developing methods for learning gating functions. These methods fall into three main categories, as outlined in [6]: routing via learnable weighting (9) 24, 33), deterministic hash routing [31), and reinforcement learning- based These approaches primarily differ in the design of the gating network & rather than the expert network ε, and therefore all encounter similar scaling challenges.
routing (2 32 33
2.2 Challenges of Scaling MoE Model Training
While MoF models were initially developed to facilitate efficient scaling during training, deploying these large-scale models in practical GPU-intensive environments poses significant challenges in

---

180%
80%
50%
68%%
40%
85%
49%
2815
35%
RoREAT
GFT-SNE
BoBERT=
GFT MoR
Soste MARL
3 MA O0lrr
GAA----11 200birs
(a) 16GPUs
(b) 32GPUs (double #GPUs)
(c) 16GPUs (double #experts)
Figure 3: Proportion of all-to-all communication time relative to total training duration across different configurations: scaling the number of training servers (Figure ((b) and scaling the parameter size of models (Figure 3(c)..
distributed computing. Specifically, the MoE layer harbors a considerably higher number of param- eters and requires additional memory, yet it maintains almost the same computational demands as the dense layer. This leads to unique compute density defined as the ratio of the layer's 's FLOPs (Floating Point Operations) to its number of parameters. Therefore, traditional parallelism methods such as lensor parallelism and pipeline parallelism are insufficient for achieving effective parallelism in the scenarios of MoE training.
To improve the efficiency and scalability of training large-scale MoE models, expert parallelism [77 has been introduced as specialized model parallelism strategy. This approach distributes experts within an MoE layer across multiple GPUs, while leveraging data parallelism for replicating non-MoE layers, thus efficiently managing the training workload of MoE models. The workflow of distributed training for an MoE layer depicted in Figure 2 Once the target expert for each token is determined, an all-to-all communication process is triggered to distribute tokens to their corresponding target experts for computations, denoted as (z). Subsequently, another round of all-to-all communication executed gather the outputs from all experts, which produces the MoE layer's output (represented as f (xr), Equation? 2 Subsequent operations involve executing the hata-parallel non-MoE layers.
We first profiled the training process of three popular MoF models employing expert parallelism (detailed in Table [ on a cluster comprised of four A100 machines, each equipped with an inter- connect RDMA bandwidth of 200Gb/s. The proportion of all-to-all communication time relative to the total training duration is illustrated in Figure ((a) We then double the number of machines, and the number of experts increase the model scale. The results are shown in Figure 3(b) and 3(c) respectively. Our findings reveal that all-to-all communication accounted for substantial portion of the total time: approximately 30% in GPT-MoE (15B), 40% in RoBERTa-Men in Swin-MoE-L. And this overhead remains nearly constant in larger models and at larger machine scales. These results highlight a significant bottleneck that hampers the scalability of the training process. Consequently, the duration of all-to-all communication substantially constrains training with expert parallelism, leading reduced overall throughput and limiting the potential to scale up the number of experts effectively.
and 70%
2.3 Locality-Sensitive Hashing Algorithms
Locality-Sensitive Hashing (LSH) is a probabilistic method primarily used to approximate nearest neighbor search in high-dimensional spaces, which reduces the dimensionality of data by mapping similar data to the same "buckets" with high probability using hash functions. This approach offers substantial reduction in computational complexity, particularly beneficial for large-scale data applications. The key operations in LSH including:
The core of LSH is family of hash functions that maximize the probability of nearby points the original space staying close in the hashed space, while distant points are likely to end up in different buckets. Each hash function h is characterized by the property: P(h(x) h(yy)
Mapping Data into Buckets:
d(z; v)/D D. where d(x, y) is the distance between points æ and y, and D denotes the diameter of the space. To map similar data into the same bucket, multiple hash functions from this family are selected based on the specific attributes of the data (e.g., Fuclidean distance, cosine similarity) and the desired granularity of the buckets. Data points then hashed by these

---

residuals
devicég
experto
tokens
centroids
E(centroids)
E(tokens)
device
LSH-Based Based Clustering
experts
Residual- Error Compensation
All-To-All
AlI-To-AII
device; expert,
tokens
hash buckets
tokens
centroids
residuals
centroids
calculate centroids
E(centroids)
residuals
E(tokens)
5
average
hash function
Figure 5: Schematic of MoE training with Locality-Sensitive Hashing (LSH-MoB).
functions, and each point is assigned to buckets according to its hash values, effectively categorizing similar items together for clustering.
Calculating Cluster Centroids: By grouping data points into buckets as determined by their hash values, data points are effectively clustered. Each bucket represents cluster of data points and the centroid of each cluster is then calculated the mean of all points within that cluster, formulated as Cj = 1 Ei 2s, where Cj is the centroid of the j-th bucket, n, is the number of points in the j-th bucket, and are the data points in the bucket.
3 Methodology
3.1 The Motivation of Token Similarity
To explore the potential optimization for all-to-all communications in MoE training, we conducted an in-depth analysis the data involved in these all-to-all communicationss identifying high degree of similarity, termed token similarity. Specifically, we applied Principal Component Analysis (PCA) to reduce the dimensionality of the input tokens of all-to-all communications and observed distinct clustering phenomenon, illustrated in the Figure Our analysis suggests that the observed similarity among tokens may stem from two primary factors:
Data Related Influences: The similarity is partially due the nature of real-world data, which often adheres to Zipf's Law [8]. This results in skewed distribution, with certain data elements appear more frequently than others.
Figure Principal Com- ponent Analysis (PCA) Visu- alization of input tokens in- volved in all-to-all communi- cation.
Model Structure Related Influences: The design of Trans- former architecture [34], especially its attention mecha- nisms, significantly impacts token similarity. In models like BERT (7). attention layers are designed to capture and integrate context information across tokens, thus homoge- nizing token representations and emphasizing their shared semantic relationships at the sentence level.
5

---

3.2 LSH-MoE
Motivated by the Token Similarity observed Section 3.1 we introduce LSH-MoE, a novel MoE training framework that integrates locality-senne sensitive hashing (LSH) for rapid clustering of input tokens. Our method transmits only the clustering centroids, significantly reducing communication volumes. To counteract the negative effects of compression, we also implement residual-based error compensation scheme.
As depicted in Figure 5 LSH-MoE initially employs (1) an LSH-based clustering method to compress tokens into centriods for subsequent processing, effectively reducing communication overhead. It then sequentially executes (2) all-to-all communication, expert computation, and another (3) all- to-all communication to produce the processed outputs E(centriods). Finally, introduces (4) residual- based eтror compensation method to approximate the expert-procee processed results E E(tokens), by integrating E(centriods) with residuals. Meanwhile, we also outline the workflow of our LSH-MoE framework in the Algorithm Tof Appendix A. The key components of our LSH-MoF framework includes an efficient LSS-based clustering algorithm for rapid processing and an residual-based error compensation scheme to minimize quality degradation.
Efficient LSH-based Clustering Algorithm. Since the data to be compressed (the input data for all-to-all communication) is generated dynamically and in real time, pre- pre-compressing.. it overlapping compression time with other processing tasks is not feasible. Consequently, selecting an efficient online compression algorithm is crucial. Traditional clustering algorithms, such as K-Means, often encounter computational challenges and efficiency limitations. Locality-sensitive hashing (LSH) address these issues by hashing similar data points into the same buckets, enabling faster similarity detection in high-dimensional spaces.
Numerous LSH algorithms have been developed, each employing a unique hashing approach for mapping data onto buckets. We conducted experiments to evaluate several popular hashing algorithms, including pross-polyton
hashing and spherical hashing. Based on evaluations in Section 4.5
selected cross-por pross-polytope hashing as the optimal algorithm for our application. Cross-polytopy hashing stands out for its method of mapping input vectors to the nearest vertex cross poossopp. This process is facilitated by applying randomly rotated cross-pol gross-polytopes, which effectively segment the surface of the unit sphere. The algorithm can be mathematically represented follows:
(SH(x) (
[Rx|;
(3)
where R is a random rotation matrix, the dimensionality of the space, and |Rx; denotes the absolute value of the i-th component of the rotated vector Rx.
This formula encapsulates how the input vector transformed by the rotation matrix R and then mapped to the nearest vertex of the cross-polytope by selecting the dimension that maximizes the absolute value of the components of Rr. This method effectively segments the high-dimensional space and enhances the clustering efficiency by rapidly identifying similar data points.
Residual-basee Error Compensation Scheme. In our LSH-MoE framework, we compress the intermediate activation values within the model network. Unlike gradient compression, this process does not tolerate errors well. Therefore, it essential to minimize compression-induces errors to ensure minimal impact model performance. To address this, we implement novel residual-based gradient compensation strategy, outlined as follows:
1. We first capture the residual for each data point relative to its cluster centroid, defined by the equation:
Acluster, + {x cluster, x E clustery
2. After the expert network computes outputs for the cluster centers, the final step is to restore the processed result for each token by adding back the previously recorded residual:
Y13 +- (E(cluster,) + AClusterzi A: 1,2...,N,J
5
This error compensation scheme effectively mitigates potential accuracy loss caused by data compres- sion in all-to-all communication, ensuring the fidelity and robustness the LSH-MMB framework. The experimental results in Section show that implementing this compensation mechanism enables

---

Table 1:
Models for evaluation, where indicates that the values are different across layers.
Model
#Layer
dmodel
dmn
#Experts
#Params. (MoE)
#Paarass. (Total
RoBERTa-MoE
12
768
3072
16
302M
394M
T5-MoB
16
1024
16384
16
8594M
9288M
GPT-MoB (LSB)
12
768
3072
512
14507M
14629M
GPT-MoE (S2B)
1024
4096
512
51539M
51740M
Swin Swin-MoE-L
24
32
946M
the model trained with LSH-MoE to achieve an accuracy comparable to that of a model trained without compression. This outcome highlights the effectiveness of our proposed error compensation strategy in preserving model performance despite the challenges posed by data compression in all-to-all communication.
3.3 Scalability Analysis of LSH-MoE
To effectively demonstrate the scalability of our approach, particularly in terms of its applicability to both larger models and larger computational clusters, we conducted a theoretical analysis. This analysis primarily focuses on the computation overhead and the communication costs associated with Mixture of Experts (MoE), specifically considering all-to-all communication overhead. We derived the ratio of communication time to computation time, highlighting how this ratio evolves as both the scale of the servers and the model size increase. This relationship is crucial for understanding scalability and can be formally expressed follows:
Tall
FLOPs
K:
w
X
(6)
T Toompute
6B 6Bmtee
1 2k
wh
where & represents the number of experts activated per token, FLOPs and Binter denote the GPU's computation ability and the network performance, w is the number of GPU servers, and h is the hidden size of model. Notably, the first term, FL.OPs remains constant under fixed hardware conditions. GB.ncer'
Additionally, scaling MoE models typically emphasizes increasing the number of layers and experts, while the growth in hidden size (h) tends to be gradual, as seen in models like Switch-Transforme [9]. Consequently, when both the model scale and the number of training servers grow, the proportion of all-to-all communication time remains nearly unchanged. This insight underpins the scalability of the LSH-MoE method, demonstrating its robustness in larger-scale settings and supporting its potential in future large-scale applications. For a detailed derivation, please refer to Appendix A.2
4 Experiment
4.1 Implementation
Our LSH-MoË comprises data compressionn/estoraaiio component and communication compo- nent. We utilize PyTorch 1.11 for developing the LSH clustering and NCCL for implementing the communication. Additionally, our method is Hameewrk-iindepender and can be easily applied to
other MoE training frameworks such as Hetu-MoE [21 26), DeepSpeed-Moo [29] and Tutel [12].
4.2 Benchmarks and Datasets
Our evaluations are conducted by scaling pre-trained models equipped with MoE architecture across various application domains. This includes models like RoBERTa-MoE, T5-MoE and GPT-MoE in natural language processing (NLP), as well as Swin-MoE in computer vision (CV). Among these models, RoBERTa-Moo and T5-MoE are evaluated on pre-training task, while GPT-MoE and Swin- MoE undergo fine-tuning evaluation based on their official open-sourced model checkpoints We also evaluated the zero-shot accuracy of the pre-trained T5-MoE. Model configurations detailed in [
http://ghhhheeeecccorrerrr/e//ea 1m

---

The RoBERTa-Mon model is pre-trained with masked language modeling tasks on combined dataset, which includes BooksCorpus (~ 800M words) and English Wikipedia (~ 2,500M words). This dataset is tokenized using tokenizer with vocabulary size of 50,257. To assess the impact of our MoE method in compressing all-to-all communication on large model training, the T5-MoR model, which is with about 10B parameters, is pre-trained on an industry dataset (~ 500M words) using a span-masked language modeling task. In addition to pre-training tasks, we further evaluate our work on fine-tuning tasks. To be specific, we fine-tune two open-sourced models, including the language model GPT-MoE on the General Language Understanding Evaluation (GLUE) benchmark and the vision model Swin-MoE on the ImageNet classification benchmark.
4.3 Software and Hardware Environments
To thoroughly evaluate the effectiveness of our method, we conducted experiments on two clusters, V100 cluster and A A100 cluster. Additionally, to ensure consistency in software versions, we performed experiments on both machines using the same docker image.
Software Environment. Our experiments were conducted using a docker image built upon the official NVIDIA GPU containers, which includes Ubuntu 20.04, CUDA 11.3, cuDNN 8.2.0, and NCCL 2.12.7, accessible at NVIDIA GPU Containers
V100 Cluster: The first hardware environment includes two servers, each outfitted with eight NVIDIA V100 (32GB) GPUs Within each server, GPUs are interconnected using NVLink 2.0 technology. The servers are interconnected via RDMA NIC, providing network bandwidth of 100 Gips..
A100 Cluster. The second hardware environment consists of four servers, each equipped with eight NVIDIA A100 (40GB) GPUs. Within these servers, GPUs utilize NVLink 3.0 technology for interconnection. The servers are linked through two RDMA NICs, enhancing the network bandwidth to 200 Gbps.
We allocated the experiments involving RoBERTa- MoE and GPT-MoE to the V100 cluster, while T5-MoE and Swin-MoE were tested on the A100 cluster. This setup allowed us to effectively compare the performance impacts across different hardware configurations.
4.4 Overall Performance
In general, to evaluate our LSH-MoB training approach, which compresses communication data, there are two crucial questions:
1. Does the LSH-MoE method enable normal model convergence, and is there risk of increased loss variability during this process, potentially leading to instability in training?
2. Might the implementation of the LSH-MoE method adversely affect the model's performance on downstream benchmarks?
Therefore, conducted experiments focusing on both Convergence Performance and Benchmark Performance to validate the effectiveness of our method. In this section, due to the necessity of selecting several hyperparameters for LSH, such as the type of hash function and the quantity of hash functions, we have opted for the cross-polytope hash function based on empirical evaluation, setting the number of hash functions A detailed examination of the effects stemming from variations in these parameters will be methodically addressed in the upcoming ablation study (Section 4.5)
Convergence Performance. We pre-trained the RoBERTa-MoE and T5-MoE using open-source datasets and industrial datasets, respectively. In our approach, we substitute the FFN (Feed-Forward Network) layer with an MoE (Mixture of Experts) layer in alternating layers, as detailed in Section 4.2 We meticulously tracked the time required to achieve equivalent model performance levels (perplexity) during training, as depicted in Figure 6 The results indicate significant acceleration in training convergence when employing the LSH-MoE method: 1.6x faster for RoBERTa-MoE and 2.2x faster for T5-MoE, compared the original models' convergence rates. Furthermore, we investigated the role of error compensation in this process. Our findings reveal that omitting error compensation in the LSH-MoE model led to a 0.3 point increase in perplexity, given the same training duration. This observation underscores the efficacy of the error compensation algorithm.
https://catalog. wgw.eaattttoo........ooooooooeee www.:nnnn..iiiiiieeeee

---

RoBERTa-Moo
T5-MoE
4.5
Validation Perplexity
22-
LSH-MoE LSH-MoE w/o EC
Vall Talidatoon Perplexity
LSH-MoE
4.0
Origin
Origin
20
18
3.0
38.99
1.6 speedup
2:3
16-
2.2 speedup
2.0
50
100
150
200
250
300
25
50
75
100
125
150
Wall Time (Hours)
Wall Time (Hours)
Figure 6: Comparative analysis convergence performance. This includes a comparison between the original models, LSH-MoE without Error Compensation, and LSH-MoE implementations. The perplexity curves are applied 1D Gaussian smoothing with o = 0.5.
Table Evaluation of L.SH-MoR on the GLUE benchmark.
Table 3: Results of fine- tuning Swin-MoE on the ImageNet- dataset.
GPT-MoE (15B)
GPT-MoE (52B)
T5-MoE (10B)
Dataset
Origin
Ours
Speed
Origin
Ours
Speed
Origin
Ours
SST-2
93.8%
93.8%
1.3x
94.5%
94.3%
1.4x
51.6%
50.9%
Origin
Ours
MNLI
82.8%
82.7%
1.4x
84.1%
84.3%
1.4x
52.6%
52.1%
Top-1 Acc. f
84.7%
84.5%
QNLI
86.6%
86.7%
1.3x
90.2%
90.0%
1.5x
49.5%
50.0%
Top-5 Acc. f
97.0%
97.1%
QQP
88.8%
88.7%
1.3x
88.9%
88.9%
1.2x
Compression
MRPC
71.3%
71.1%
1.3x
76.3%
76.1%
1.3x
Rate Sample/s
11.7%
COLA
72.3%
72.4%
1.4x
73.5%
73.8%
1.5x
184.3
236.6
Benchmark Performance. To better validate the performance of LSH-MoB on downstream tasks, we fine-tuned the GPT-MoE and Swin-MoE on different datasets using open-source model checkpoints, and evaluated zero-shot performance of our internal pre-trained T5-MoE model, adhering to their original architectural designs that incorporate Top-2 gating, as detailed in [1 17]
We first utilized the LSH-MoE method for fine-tuning the GPT-MoE of two model scales (i.e. 15B and 52B) on the GLUE benchmark, yielding impressive outcomes. As detailed in Table 2 the implementation of the LSH-MoE method substantially reduced communication overhead while maintaining nearly the same level of accuracy. This strategy resulted in significant performance boost, achieving an acceleration rate ranging from 1.2x to 1.5x. The results also demonstrate that as the parameter size of MoE models increases, LSH-MoE continues to achieve significant improvements without compromising model accuracy. Additionally, we report the zero-shot accuracy of the pre-trained T5-MoB, showing that the T5-MoE models trained with LSH-MoE achieved accuracy comparable to standard T5 models, confirming LSH-MoEE' efficacy in pretraining. Because the limited number of tokens in the pre- pre-trained dataset and its out-of-doman nature compared to the GLUE evaluation data, the zero-shot performance metrics are relatively low.
Furthermore, our evaluation of the L.SHMMB method in fine-tuning the Swin-MoE. on the ImageNet- 1K dataset demonstrated noteworthy efficiency. We achieved a communication compression rate of 11.7%, which led to a 1.28 x increase in acceleration, reported in Table 3 Notably, this was accomplished while preserving almost the same level of accuracy.
4.5 Ablation Study
To study the impact of the quantity and types hash functions, we conducted ablation experiments by fine-tuning the GPT-MoE (15B) model on the MNLI and SST-2 datasets in the GLUE benchmark.
Impact of the Quantity of Hash Functions. We controlled the number of hash functions indirectly adjust the LSH compression rate, exploring its effect on model performance. Specifically, we utilized 2, 4, 6, 8, and 10 hash functions. As shown in the middle sub-figure in Figure we observe that an increase in the number of hash functions enlarges the number of buckets, enhances data distinction and, consequently, the compression rate. Besides, we indicate from the left sub-figure in Figure) 7 that more hash functions leads to improved model convergence quality and worse compression rate.

---

MNU Accuracy
MNLI Compression Rate
MNU Accuracy
40
hashes
hashes
Compression Raceeks
255
B hastes
hashis
Accuracy %5)
80
Pashes
Accurano
80
25-
Origin
Pastes hoshes
78
EP 19%
15%
hashes
CP 10%
SP 10%
24
8 hashes
2 hashes
Origin
10
Epoch
Epoch
SST-2 Accuracy
S5T-2 Compression Rate
SST-2 Accuracy
35
92-
Compression Ratee
30
10 hashes
B hastws
hashes hashes
92.5
Acouraonss
90
hashes
90.0
20-
36-
Crigin
Pastoss
15%
15%
hashes
hoshes
-0-
CP 10%
SP 10%
B hashes
2 hashes
Origin
Epoch
Epoch
Figure 7: An in de-tepth analysis of the compression rate and the model performance by adjusting the quantity and types of hash functions. The left and middle sub-figures are results for diverse quantities of hash functions. The right sub-figure the result for diverse types of hash functions (CP for cross-polytops and SP for spherical) with different compression rates (20%, 15%, 10%).
Importantly, our results indicate that compression rate of approximately 20% (achieved with about hash functions) optimal for maintaining nearly identical convergence as uncompressed models. Therefore, we choose 6 as the default number of hash functions in Section 4.4
Impact of the Types of Hash Functions. We further explored the impact of the types of hash functions with Cross-Polytope Hashing (CP) and Spherical-Plane Hashing (SP). The outcomes are illustrated in the right sub-figure in Figure 7 CP generally achieves better convergence than SP at the same compression rate. This is attributable to CP's ability to more effectively handle a variety of complex data patterns. CP encodes data based on an n-dimensional pross-polytope, while SP relies on the geometric relationships between spheres and planes. Thus, CP is more generalizable across variety of complex data patterns while SP performs better with data that has spherical distribution characteristics. Other works (e.g. Reformer (16)) also use CP to leverage the sparsity of attention mechanisms. Therefore, we finally choose eross-poly hashing the default type of hash functions in Section 4.4
5
Conclusion
Our study tackled the latency challenges inherent in training sparse-gated -ffpeeerts----- (MoE) models with our innovative LSH-MoE framework. Utilizing locality-sensitiig hashing to harness token similarities, our approach significantly reduces communication overhead. The integration of residual-basee error compensation scheme further preserves model integrity under compression. Empirical tests across various models, including RoBERTa, GPT, T5, and Swin, showcase LSH- MoE's capability to accelerate both pre-training and fine- the-tuning phases by up to 2.2x, paving the way for efficient and scalable MoE applications in real-world settings.
Limitations
At the current stage, our work only considers MoE models. Nevertheless, we want to clarify that MoE models are also mainstream class of deep learning models that are increasingly adopted due to rising model computational demands and training costs, such as Mixtral- and GPT-4. Hence, accelerating MoE training is indeed a critical direction. Additionally, the core of our work leverages data redundancy, which is also presented in non-MoF, model training. We hope our observations and utilization of data redundancy can inspire more refined work in optimizing training for non-MoE models well.
DeepSeek-MofE
10

---

Acknowledgments and Disclosure of Funding
This work is supported by Beijing Natural Science Foundation (4244080), National Natural Sci- ence Foundation of China (U22B2037, U23B2048), Beijing Municipal Science and Technol- ogy Project (Z231 100010323002), China National Postdoctoral Program for Innovative Talents (BX20230012), China Postdoctoral Science Foundation (2024M750103), research grant No. SH- 2024JK29, ByteDance-PKU joint program (PI20230002900063) and Sigh-performance Computing Platform of Peking University. Fangcheng Fu and Bin Cui are the corresponding authors.
References
[1] Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Vic- toria Lin, Jingfei Du, Srinivasan lyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684 2021.
[2] Emmanuel Bengio. On reinforcement learning for deep neural architectures: conditional
computation with stochastic computation policies. McGill University (Canada), 2017.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Chang Chen, Min Li, Zhihua Wu, Dianhai Yu, and Chao Yang. Ta-moe: Topology-ane -wware aware large scale mixture-of-experr training. Advances in Neural Information Processing Systems, 35:22173-221 22186. 86, 2022.
[5] Tianlong Chen, Xuxi Chen, Xianzhi Du Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamv-moc: Adaptive multi-task vision mixture-of-experts. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 17346- 17357, 2023.
[6] Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning. pages 4057-4086 PMLR, 2022.
[7] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171 4186, 2019.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[9] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research. 23(120):1-39, 2022.
[10] Fangcheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang. Yingxia Shao, Ce Zhang, and Bin Cui. Don't tt waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In International Conference on Machine Learning, pages 3314,3314, 2020.
[11] Keshi Ge, Yiming Zhang, Yongquan Fu, Zhiquan Lai, Xiaoge Deng, and Dongsheng Li. Accelerate distributed deep learning with cluster-aware sketch quantization. Science China Information Sciences, 66(6):162102, 2023.
[12] Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts scale, Proceedings of Machine Learning and Systems, 5, 2023.

---

[13] Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, Mao Yang. and Minsoo Rhu. Pre-gated moe: An algorithm- system co-design for fast and scalable mixture-of-expert inference. arXiv preprint arXiv:2308. 12066, 2023.
[14] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
[15] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alce Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
[16] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, 2020.
[17] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference Learning Represen- tations, 2021.
[18] Wentian Li. Zipf's law everywhere. Glottometrics, 5(2002):14 2002.
[19] Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823, 2021.
[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[21] Xupeng Miao, Xiaonan Nie, Hailin Zhang, Tong Zhao, and Bin Cui. Hetu: A highly efficient automatic parallel distributed deep learning system. Science China Information Sciences, 66(1):1-2, 2023.
[22] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-ing image mixture of experts. Advances in Neural Information Processing Systems, 35:9564 9576, 2022.
[23] Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, and Bin Cui. Angel-ptmi -tme: A scalable and economical large-scale pre-training system in tencent. arXiv preprint arXiv: 020888 2023.
[24] Xiaonan Nie, Xupeng Miao, Shijie Cao, Lingxiao Ma, Qibin Liu, Jilong Xue, Youshan Miao, Yi Liu, Zhi Yang, and Bin Cui. Evomoe: An evolutional of training framework via dense-to- to--sparseeeeeeee gate. arXiv preprint arXiv:2 12. 14397, 2021.
[25] Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement. Proceedings of the ACM on Management of Data, 1(1):1-19, 2023.
[26] Xiaonan Nie, Pinxue Zhao, Xupeng Miao, Tong Zhao, and Bin Cui. Hetumoe: An efficient. trillion-scale offtuere distributed training system. arXiv preprint arXiv:2203. 14685, 2022.
[27] Dmytro Nikolaiev. How estimate the number of parameters in transformer mod- wtw.e///oooooo-ssiittttoe--aeeee
els. models-ca0157ddddn0 2023. Accessed: Oct 22, 2024.
[28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
12

---

[29] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-Me MoE: Advancing mixture-of- experts inference and training to power next-generation Al scale. In International Conference on Machine Learning, pages 18332-18346, 2022.
[30] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, $4:8583-8599 2021.
[31] Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 14:17555-17566 2021.
[32] Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint 1904. 2019.
[33] Richard Sutton and Andrew G Barto, Reinforcement learning: An introduction. MIT press, 2018.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30:6000-6000 6010, 2017.
[35] Guozheng Wang, Yongmei Lei, Zeyu Zhang, and Cunlu Peng. A communication efficient admm-based distributed algorithm using two-dimensional torus grouping allreduce. Data Science and Engineering. 8(1):61-72, 2023.
[36] Adam Weingram, Yuke Li, Hao Qi, Darren Ng. Liuyao Dai, and Xiaoyi Lu. xeel: survey of industry-led collective communication libraries for deep learning. Journal of Computer Science and Technology, 1661::166669999 2023.
[37] Shuo Xiao, Dongqing Zhu, Chaogang Tang, and Zhenzhen Huang. Combining graph contrastive embedding and multi-head cross -ttsttnttention transfer for cross- recommendation. Data Science and Engineering, 8(3):247-266 2023.
[38] Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. Improving BERT fine-tuning via self- ensemble and self-distillatton. Journal of Computer Science and Technology, 38(4):853-866, 2023.
[39] An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang. Yong Li, et al. M6-t: Exploring sparse expert models and beyond. arXiv preprint arXiv:2 15082, 2021.
[40] Zhiyuan Zeng and Deyi Xiong, Scomoe: Efficient mixtures of experts with structured commu- nication. In The Eleventh International Conference on Learning Representations, 2023.
[41] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint
2022.
13

---

Algorithm Training MoB models using our LSH-MoE framework
Input: X: sequence of tokens
Output: {Yi S i I n, Σ S m), where Vig is the output for tokens in the j-th cluster
assigned the i-th expert
function MoE LAYER WITH VITH_LSH(X)
2: Calculate the token-to-exper mapping ( using the gating network: 1,2,....ni, based on Ç; X; are tokens assigned the i-th
3: Dispatch X into (X;
expert
4:
for t- 1,,,,,,,........ ..,,....77 do
5:
IDX; + LS:((:); // Get the LSH bucket for each token Divide X; into {cluster, j .12,....,m. based on IDX;
6:
7:
for j + 1, ........1 do cluster, f- Mean(cluster,)) Get the centroids for each cluster
8:
9:
Acluster, f- {x cluster, x E cluster, }; // Get the difference between each token and its cluster centroids
10:
C fcluster, J = AX; AX; - Acluster;
m)}
11:
12:
C+ 1.2...., .n);
.....
13:
Input + all-to-all(C);; Transmit the cluster centroids through all-to-all
14:
Oulpul Expert( [nppud:; // Perform computations centroids
15:
E(C)) &- all-to-all( (Oulpud): Transmit the results back through all-to-all
16:
for (i, -1,2,...., n)
m) ...; do
17:
Yo (E(clustery) + AClusterya .......: :;):; // Apply the residual-basee error
compensation scheme
18:
return (Y).
A
Appendix
A.1
Framework of LSH-MoE
As illustrated in Algorithm 1 our LSH-MoE training process begins by dispatching each input token X to its designated expert based on the gating network (Line 2-3). then utilizes locality- sensitive hashing cluster tokens into groups, calculating the centroid for each cluster represent the mean its tokens, and recording the differences between each token and its centroid for later error compensation (Lines 4-11). These centroids subsequently transmitted to the experts via all-to-all communication for processing and their results are sent back in a similarly manner (Line 13-15). Finally, residual- based error compensation is applied to determine the output of the MoE layer (Lines 16- (6-18). This method effectively minimizes the communication load, thus improving the scalability and efficiency of MoE model training.
A.2 Scalability Analysis
First of all, we want to highlight that as the scale of both models and machines increases, the proportion of all-to-all communication time relative to the total time remains nearly constant. This consistency suggests that the LSH-MoB method remains effective even at larger scales. will now present our derivation step by step, using the notations listed in Table 4
Formulate all-to-all communication. For any given training server. the amount of tokens (i.e. m) communicated with any other GPU node can be expressed as m = n x k/w. Similarly, the volume of communication within the same GPU node is also equal to m n x k/w. Consequently, the time required for all-to-all communication during model training can be modeled as follows, with each layer involving two instances for the forward pass and two the backward pass:
m h
m
(w 1)
h(w 1w-11)
TallJo = 4
4
X
(7)
B
Bintra
Binter
Binter
Formulate model computation. Based on the derivation
27]. for standard decoder model, given the number of layers I, and the hidden size h of the model, the activated parameter count per token
14

---

Table 4: Notations used in scalability analysis.
Notation
Description
The number of tokens processed per GPU
m
The number of tokens communicated between two training servers The number of experts activated per token
Hidden size for each token
The number of layers for the model
The number of training servers network bandwidth network bandwidth
Bintra
The
Binter
The
can be formalized as #ActivatedParams. 4(1 2k)lh?. According the theory in the appendix of the GPT-3 paper [3) the computation time per GPU can be formalized as T computee where FLOPS represents the computation ability of GPU.
#tokens #ActivatedParams.
24(1 + 2k)mlh?
T coopuute
FLOPSs
FLOPPs
Formulate all-to-all communication computation. Therefore, as the machine scale (w) and model scale (1 and hi) increase, the ratio of computation time to communication time can be formalized as:
h(w-1)
Tail tolll T Tcompute
Binter
FLOPs
w
(24(1 + mlh? /FLPPs
6Binter
1
2k wh
where the first term 6B:nter is constant. As MoE models scale up, the emphasis is generally placed on increasing the number of layers and experts, with a more gradual increase in hidden size (e-g. Tritch-Transformer (9). Consequently, the proportion of communication time remains significant as both the model size and the number servers increase. These observations and theoretical proof's underscore the sustained effectiveness of the LSH-MoE method in larger environments, thus reinforeing its scalability and applicability for future advancements.
FLOPS
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper demonstrates the token similarity in Section 3.J introduces the criti-
cal residual-based error compensation scheme in Section 3.2 and verfies the effectiveness of the method in Section4.44
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[ Yes]
Justification:
The paper discusses the limitations of this work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed illustration of the LSH-MoE method in Algo- rithm T Additionally, we thoroughly explain the model setup, dataset configurations, and other relevant details in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Most of the datasets and model checkpoints used in the paper are open- and publicly available. In addition, we provide our code in the supplementary material submitted along with the paper. However, the T5-MoR model is built on a proprietary company platform and training framework. Due company regulations, we are unable to release the training codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
paips.cc/publiccuuudess wttp:e/////oooSSeeiisiiiiiommeee
www.oiiiiisssooooooo.oocoo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides detailed experimental settings in Section 4.2 for datasets, Section 4.3 for environments, and Table [ for model hyper-parameters. In addition, we submit our code in the supplementary material along with the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because would be too computationally expensive for pre-training tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides detailed information the computer resources and environments needed to reproduce the experiments in Section 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the NeurlPS Code of Ethics and preserves anonymity in the submitted code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
19

---

There is no societal impact of the work performed. This paper addresses foundational research problem and is not tied particular applications.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper introduces a method to accelerate the training of MoE models, and does not release any new datasets or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators and original owners of assets used in the paper are properly credited. The paper includes URLs as footnotes for the code base used in the experiments in Section 4.2
Guidelines:
The answer NA means that the paper does not existing assets.
20

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We submit our code in the supplementary material and the documentation is provided alongside the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "olvZtSCHA2", "reviewer": "Reviewer_7fVV", "summary": "This paper presents a method to speed up MoE large model training with locality-sensitive hashing. It conducts experiments on both language models and vision models for both pre-training and fine-tuning tasks, and achieves 1.28-2.2× of speedup.", "questions": "1. What's the difference between compression rates and compression ratio in Fig 6, and why we choose 20%,15%,10% compression rates for accuracy comparation.\n2. We only try  cross-polytope and spherical hashing for comparation, are there any other hashing types could be adopted?\n3. Do we consider  to adopt some learning to hash method for MoE model training speedup? Because as far as I see, learning to hash method usually keep semantic similarity better than LSH method.", "limitations": "The authors have not  addressed the limitations and negative societal impact of their work. This work gives a good training speedup solution for MoE large models. But the proposed method could not work well on none-MoE models. Maybe the author can consider some speedup method for general large models.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces an efficient LSH-based compression technique to apply tokens on similar expert model, which is both effective and fast.\n2. This paper is well written for understanding.\n3. This paper conducts experiment on different types of large model and training tasks, which shows the generality of proposed method.", "weaknesses": "1. There is a lack of the whole framework figure for the proposed LSH-MoE method for better understanding of reader.\n2. More deeper analisis should be added for  ablation study,  the experiment result shows the effect of different quantity and type of hash functions. But we still don't have even a guess of reason.\n3. Some expressions need to be improved for example, what is the difference between compression rates and compression ratio in Fig 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# W1\nTo address the reviewer's comment, in the attached PDF file of our global response, we have included a figure (Fig A) to depict the schematic of MoE Training with Locality-Sensitive Hashing (LSH-MoE). \nThis figure highlights the key components of LSH-MoE, including the *LSH-Based Clustering* and the *Residual-based Error Compensation*. These components play a significant role in leveraging data redundancy to accelerate the training process.\n\nAs illustrated in Fig A, LSH-MoE initially employs (1) an LSH-Based Clustering method to compress tokens into centriods for subsequent processing, effectively reducing communication cost. It then sequentially executes (2) all-to-all communication, expert computation, and another (3) all-to-all communication to produce the processed outputs E(centriods). Finally, it introduces (4) a Residual-based Error Compensation method to approximate the expert-processed results E(tokens), by integrating E(centriods) with residuals.\n# W3 & Q1\nWe use the terms \"compression rate\" and \"compression ratio\" interchangeably to refer to the ratio of the data volume after compression to that before compression, where a lower value indicates more effective compression.\nWe apologize for any confusion due to typographical errors. We will carefully revise our manuscript.\n# W2 & Q1\nThere are two key hyperparameters in LSH to reduce hash collisions and optimize model performance, which are the type and quantity of hash functions. Thus, we carried out two kinds of ablation studies in Section 4.5 (Fig 6) of our submitted manuscript, aiming to provide guidance on hyperparameter selection:\n\n**Impact of the Quantity of Hash Functions**: We controlled the number of hash functions to indirectly adjust the LSH compression rate, exploring its effect on model performance across different settings. Specifically, we utilized 2, 4, 6, 8, and 10 hash functions, observing that an increased number of buckets enhances data distinction and, consequently, the compression rate. Results are shown in the left and middle subfigure of Fig 6, where we see improved model convergence quality and worse compression ratio with more hash functions. Importantly, our results indicate that a compression rate of approximately 20% (achieved with about 6 hash functions) is optimal for maintaining nearly identical convergence as uncompressed models, without significantly slowing down training. As described by Zipf's Law [1], there is an imbalance in data distribution (in the corpus of natural language, the frequency of a word's occurrence is inversely proportional to its rank in the frequency table). Therefore, most of the data will be repetitive, leading to data redundancy.\n\n**Impact of the Types of Hash Functions**: We further explored the impact of the types of hash functions with Cross-Polytope Hashing (CP) and Spherical-Plane Hashing (SP).\nAs shown in the right sub-graph in Fig 6, CP generally achieves better convergence than SP at the same compression rate. This is attributable to CP's ability to more effectively handle a variety of complex data patterns.\nCP encodes data based on an n-dimensional cross polytope, while SP relies on the geometric relationships between spheres and planes. Thus, CP is more generalizable across a variety of complex data patterns while SP performs better with data that has spherical distribution characteristics. Other works (e.g. Reformer [2]) also use CP to leverage the sparsity of attention mechanisms.\n\n# Q2 & Q3\nIndeed, other hash types can be considered, such as Random Projection Hashing, MinHash, and SimHash. We chose Cross-Polytope Hashing and Spherical-Plane Hashing in this work because they are commonly used in deep learning [2,3,4].\n\nIt’s an interesting topic to explore how to integrate our work with learning-to-hash techniques, which often preserve semantic similarity more effectively than tranditional LSH methods.\nHowever, as model parameters are continuously updated during training, the hidden states also change accordingly, so it necessitates dynamically adjusting learning-to-hash strategies to adapt to the dynamic data distribution, which introduces significant time overhead and affects training efficiency.\n\nAlthough it is important and challenging to determine the optimal hash functions (such as exploring other hash types or learning to hash), we would like to highlight that this does not hamper the significance of our work. In particular, the core contribution of our work is the identification of data redundancy during the training process and designing a framework that utilizes the LSH method to compress the AllToAll communication volume in MoE training. This allows us to accelerate model training while ensuring model convergence accuracy. We believe that our work will spark further interest in data redundancy and inspire more research in this area. Consequently, we would like to leave the investigation of the optimal hash functions as a potential future direction.\n\n# Limitation\nThank you for raising this concern, and we acknowledge it as valid. \nMeanwhile, we want to clarify that MoE models are also a mainstream class of deep learning models that are increasingly adopted due to rising model computational demands and training costs. Examples include Switch-Transformer, Mixtral-7Bx8MoE, DeepSeek-MoE, and possibly models like GPT-4. \nTherefore, we believe accelerating MoE training is indeed a critical direction and our work is of significant value. \n\n[1] Zipf's law. https://en.wikipedia.org/wiki/Zipf%27s_law\n\n[2] Kitaev N, Kaiser Ł, Levskaya A. Reformer: The efficient transformer. ICLR 2020.\n\n[3] Bojarski M, Choromanska A, Choromanski K, et al. Structured adaptive and random spinners for fast machine learning computations. PMLR 2017.\n\n[4] Dahlgaard S, Knudsen M, Thorup M. Practical hash functions for similarity estimation and dimensionality reduction. NeurIPS 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpjS8HEpzW", "reviewer": "Reviewer_ooKw", "summary": "The paper introduces LSH-MoE, a communication-efficient training framework for Mixture-of-Experts (MoE) models using Locality-Sensitive Hashing (LSH). The authors identify the inefficiencies in existing MoE training methods, particularly the high communication costs due to all-to-all communications among GPUs. The proposed method leverages token similarity for data compression, significantly reducing communication overhead and achieving substantial speedups in training time while maintaining model quality.", "questions": "How does the authors view the potential speedup of LSH-MOE over larger MoE models like Mixtral 8x7B?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of Locality-Sensitive Hashing for compressing communication data in MoE training is novel and addresses a significant bottleneck in distributed training systems.\n2. The authors conduct extensive experiments on various language and vision models, demonstrating the effectiveness of their method across different tasks and datasets.\n3. The experimental results show impressive speedups (1.28-2.2×) in training time, making the approach highly beneficial for large-scale model training.", "weaknesses": "Maybe it would be better to include more larger MoE results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to address the reviewer's concerns about the speedup over larger MoE models with both experiments and analysis.\n\n**Experiments**\n\nWe conducted experiments on the GPT-MoE 52B model, and the results are summarized in Fig. E of the one-page PDF file attached with our global response. \nThe results demonstrate that as the parameter size of the MoE models increases, LSH-MoE continues to achieve significant improvements without comprising model accuracy.\n\nOther larger MoE models, such as Mixtral 8x7B, operate on the same mechanisms as GPT-MoE. Unfortunately, due to the time constraint of rebuttal, we were unable to finish the coding development and coordination of experimental resources for experiments on the Mixtral 8x7B model. However, we believe that the results on the GPT-MoE 52B model sufficiently demonstrate the applicability and effectiveness of our method on larger models\n\n**Analysis**\n\nAs analyzed in our global response, the ratio of communication time to computation time can be formalized as:\n\n$\\frac{T_{all-to-all}}{T_{compute}} = \\frac{k \\times FLOPs}{24 B_{inter}} \\frac{w-1}{wh}$\n\nwhere $k$ is number of experts activated per token, $FLOPs$ and $B_{inter}$ represent GPU and network performance, $w$ is the number of devices, and $h$ is the hidden size of model. \n\nWe have the two facts that\n- the first term $(k \\times FLOPs)/(24B_{inter})$ is constant;\n- scaling MoE models often focus on increasing the number of layers and the number of experts, while the growth in hidden size (i.e., $h$) tends to be gradual, e.g., Switch-Transformer. \n\nTherefore, when both the scale of models and the scale of training servers expand, the proportion of all-to-all communication time remains nearly constant. This observation supports the continued efficacy of the LSH-MoE method in larger settings, thereby affirming its scalability and potential in future applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NvcKjmoZx3", "reviewer": "Reviewer_4XaA", "summary": "The authors focus on the communication overhead in large-scale MoE training, specifically under the expert parallel plus the data parallel regime. They reduce the communication workload by transmitting only the clustering centroids, which are calculated on the fly using LSH functions. To reduce the compression-induced error, the authors also propose a residual-based error compensation scheme. The authors perform evaluations using 4 different architectures  in both language and vision, and for both pertaining and finetuning tasks.", "questions": "1. Why did T5-MoE achieve a much faster convergence than RoBERTa-MoE? \n\n2. Figure 5 shows a difference in validation perplexity( 2.37 vs. 2.5), while LSH-MoE seems not quite flattened out. Would continuing training LSH-MoE reach the same perplexity? \n\n3. For the pertaining setting, what is the accuracy difference on zero-shot or few-shot downstream tasks between origin and LSH-MoE? \nThe", "limitations": "The authors claim there is no limitations and no societal impact of the work performed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is primarily well-written, and the algorithm is presented clearly. \n2. As far as I know, the idea of compress activation using cluster to reduce communication for MoE training is rather original. \n3. The authors focus on a timely and important problem, especially because there has been a trend of adopting MoE in large-scale models recently.", "weaknesses": "1. To convince the audience of this problem's importance and increase the work's significance, the authors may provide a more detailed analysis of the communication overhead. One example is given in Section 2.2. However, it would be better if the authors could study what factors influence the percentage,  perhaps the relationship between the communication overhead and the scale of the training servers, the scale of models, etc. \n\n2. Lack of background and related work. I would suggest the authors provide background on LSH algorithms, such as how to calculate centroid using LSH. Further,  as far as I know, there are more works that improve the efficiency of MoE training, such as DeepSpeed MoE, DeepSpeed-TED, and SCoMoE, which should be discussed and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# W1\nThank you for your suggestion to deepen our analysis of communication overhead.\n\nTo address the review's comment, we have conducted a detailed deduction in the global response to analyze the communication overhead. In particular, the ratio of communication time to computation time can be formalized as:\n\n$\\frac{T_{all-to-all}}{T_{compute}} = \\frac{4 l \\frac{nk}{w} \\times \\frac{h (w-1)}{B_{inter}}}{(96 n l h^2) / \\text{FLOPs}} = \\frac{k \\times \\text{FLOPs}}{24B_{inter}} \\times \\frac{w-1}{wh}$\n\nwhere the first term $\\frac{k \\times \\text{FLOPs}}{24 \\times B_{inter}}$ remains constant, $k$ is the number of experts activated per token, and $\\text{FLOPs}$ and $B_{inter}$ represent GPU and network performance respectively. Here, $w$ denotes the number of devices, and $h,l$ denote the hidden size and number of layers of the model. \n\nAs MoE models scale up, the focus is typically on adding more layers and experts, while the growth in hidden size (i.e., $h$) tends to be gradual. Therefore, the proportion of communication time remains substantial as both model size and server scale increase. This observation underscores the ongoing effectiveness of the LSH-MoE method in larger environments, thereby reinforcing its scalability and future applicability.\n\n\n# W2\n\nThank you for pointing out the need for more background on LSH algorithms and the missing comparisons with other MoE work about efficiency improvements. In our revised manuscript, we will include a more comprehensive background as well as a related work section.\n\n**Background on LSH algorithms**. Locality-Sensitive Hashing (LSH) is a probabilistic method primarily used to approximate nearest neighbor search in high-dimensional spaces. The key operations in LSH including **Mapping Data into Buckets** and **Calculating Cluster Centroids**. Due to the response word limit, please see the official comment for further details.\n\n**Related Work**. Several advancements have been made to improve the distributed training of MoE models on bandwidth-constrained clusters. DeepSpeed-TED integrates Tensor Parallel, Expert Parallel, and ZeRO-powered Data Parallel to enable training of larger models, and introducing the Duplicate Token Dropping technique to eliminate unnecessary communications. DeepSpeed-MoE introduces a novel MoE architecture, PR-MoE, which selects one expert combined with a shared expert instead of the top-2, thus halving the all-to-all communication volume. \nSCoMoE addresses all-to-all communication in a structured manner, rather than uniformly across different devices. It partitions the data along the sequence or feature dimension and controls the data volume communicated  at different network levels. Furthermore, SCoMoE proposes a token clustering approach that aggregates related tokens before the SCoMoE layers to alleviate routing locality in the structured communication. \n\nHowever, none of these works consider reducing the All-to-All communication volume in MoE training by compressing the forward activations. Therefore, they can be intergrated with the idea in LSH-MoE for further improvement.\n\n# Q1\nThe faster convergence of the T5-MoE model compared to RoBERTa-MoE is mainly due to differences in hardware setups and training tasks. We trained the T5-MoE on an A100 cluster, which offers a computational power of 312 TFLOPs, significantly higher than the 125 TFLOPs of the V100 cluster used for RoBERTa-MoE. Additionally, T5-MoE was trained on a language modeling task, while RoBERTa-MoE used a masked language modeling task. These factors contributed to the observed faster convergence of the T5-MoE model.\n\n# Q2\nWe apologize for any confusion caused by the unclear graphical representation in our results. In fact, both models converged at a perplexity of 2.37. However, because the LSH-MoE was plotted with a dashed line, only a single point at 2.37 was visible, which did not clearly convey this outcome. (Please zoom in the right sub-figure of Figure 5 of our manuscript and the single point at 2.37 can be found.)  To fix this issue, we have redrawn the figure in the PDF file attached to our global response (Figure C).\n\n# Q3\nWe appreciate your feedback. To demonstrate the generalization capabilities of our MoE models trained with LSH-MoE, we compared their zero-shot performance on the GLUE benchmark. The results, summarized in Figure E of the attached PDF, show that the T5-MoE models trained with LSH-MoE achieved accuracy comparable to standard T5 models, confirming LSH-MoE’s efficacy in pretraining. Because the limited number of tokens in the pre-trained dataset and its out-of-domain nature compared to the GLUE evaluation data, the zero-shot performance metrics are relatively low.\n\n\n# Limitations\nWe acknowledge the oversight that our initial submission did not explicitly address potential limitations and societal impacts. At the current stage, our work only considers MoE models. Nevertheless, we want to clarify that MoE models are also a mainstream class of deep learning models that are increasingly adopted due to rising model computational demands and training costs, such as  Mixtral-7Bx8MoE, DeepSeek-MoE, and GPT-4. Hence, accelerating MoE training is indeed a critical direction. Additionally, the core of our work leverages data redundancy, which is also present in non-MoE model training. We hope our observations and utilization of data redundancy can inspire more refined work in optimizing training for non-MoE models as well. We will include a section in our final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers for their thorough evaluations and valuable feedback.

In the attached PDF, we have provided a detailed response to each comment from all reviewers. To aid in understanding our responses, we have included the figures recommended by the reviewers.We hope that this detailed reply adequately addresses the issues raised and demonstrates our commitment to improving the quality of our work.

To demonstrate the scalability of our work (its applicability to larger models and larger clusters), we theoretically derived the ratio of communication time to computation time as the machine scale and model scale increase.

First of all, we want to highlight that as the scale of both models and machines increases, the proportion of all-to-all communication time relative to the total time remains nearly constant. This consistency suggests that the LSH-MoE method remains effective even at larger scales. We will now present our derivation step by step, using the following notations:

+ $n$: number of tokens processed per GPU
+ $m$: number of tokens communicated between two training servers
+ $k$: number of experts activated per token
+ $h$: hidden size for each token
+ $l$: number of layers for the model
+ $w$: number of training servers
+ $B_{intra}$: the intra-machine network bandwidth
+ $B_{inter}$: the inter-machine network bandwidth


# 1. Formulate all-to-all communication
For any given training server, the amount of tokens (i.e., $m$) communicated with any other GPU node can be expressed as $m=n \times k/w$. Similarly, the volume of communication within the same GPU node is also equal to $m=n \times k/w$. Consequently, the time required for all-to-all communication during model training can be modeled as follows, with each layer involving two instances for the forward pass and two for the backward pass:
   
$T_{all-to-all} = 4  \times  l  \times  (\frac{m \times h}{B_{intra}} + \frac{m \times h \times (w-1)}{B_{inter}}) \approx 4  \times  l  \times  \frac{n \times k}{w} \times \frac{h \times  (w-1)}{B_{inter}}$


# 2. Formulate model computation
Based on the derivation in [1], for a standard decoder model, given the number of layers $l$, and the hidden size $h$ of the model, the parameter count can be formalized as $|Params| = l  \times  16  \times  h^2$. According to the theory in the appendix of the GPT-3 paper [2], the computation time per GPU can be formalized as $T_{compute}$, where FLOPs reprensents the computation ability of GPU.

$T_{compute} = 6 \times |tokens| \times \frac{|Params|}{FLOPs} = \frac{96  \times  n  \times  l  \times  h^2}{FLOPs}$


# 3. Formulate all-to-all communication/computation
Therefore, as the machine scale ($w$) and model scale ($l$ and $h$) increase, the ratio of computation time to communication time can be formalized as:

$\frac{T_{all-to-all}}{T_{compute}} = \frac{4  \times  l  \times  \frac{n \times k}{w} \times \frac{h \times  (w-1)}{B_{inter}}}{(96  \times  n  \times  l  \times  h^2) / FLOPs} = \frac{k \times FLOPs}{24 \times B_{inter}}  \times  \frac{w-1}{w \times h}$

where the first term $\frac{k \times FLOPs}{24 \times B_{inter}}$ is constant. 


# Conclusion
As MoE models scale up, the emphasis is generally placed on increasing the number of layers and experts, with a more gradual increase in hidden size. Consequently, the proportion of communication time remains significant as both the model size and the number of servers increase. These observations and theoretical proofs underscore the sustained effectiveness of the LSH-MoE method in larger environments, thus reinforcing its scalability and applicability for future advancements.

[1]. How to Estimate the Number of Parameters in Transformer models. https://towardsdatascience.com/how-to-estimate-the-number-of-parameters-in-transformer-models-ca0f57d8dff0

[2]. Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. NeurIPS 2020.

[3] Fedus W, Zoph B, Shazeer N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR 2022.

Meta-review: This paper proposes a communication-efficient method to train MoE models. The method leverages token similarity and LSH to significantly reducing communication overhead while maintaining model quality.

The AC agrees with the reviewers that the problem being studied is important and timely. The proposed method is technically sound. The paper is in general well written. The method is well evaluated showing impressive improvements.

The authors have addressed most of the concerns from the reviewers, and the final scores of the work are unanimously acceptance (7, 6, 6, 7). Please add the additional discussions especially the scalability results in the final version.
Decision: Accept (poster)