Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
Authors: Young Jin Kim
Raffy Fahim
Hany Hassan
Venue: Submitted to ICLR 2023 (2023)
Keywords: MoE
Quantization
Mixture of Experts
Sparse Model
Machine Translation

Abstract: Large Mixture of Experts (MoE) models could achieve state-of-the-art quality on various language tasks, including machine translation task, thanks to the efficient model scaling capability with expert parallelism. However, it has brought a fundamental issue of larger memory consumption at deployment time. Furthermore, this results in significant inference speed degradation at auto-regressive decoding steps due to the increased memory transfers. In this paper, we propose a simple weight-only quantization method using ultra low-bit such as 2-bit, 3-bit and 4-bits to effectively mitigate the increased memory and latency issues of MoE models. We show that low-bit quantization together with the MoE architecture delivers a reliable model performance while reducing the memory size significantly even without any additional training. Especially, expert layers in MoE models are much more robust to the quantization than conventional feedforward networks (FFN) layers. In our comprehensive analysis, we show that MoE models with 2-bit and 80\% sparse expert weights can deliver better model performance than the dense model trained on the same dataset. We present how quantization of different parts of models affects the performance with various experiments using a large MoE model (5.3 B). As a result of low-bit quantization, we show the model size can be reduced by 4.9X smaller than the original half precision floating point (fp16) MoE model. This cuts down the model size of 5.3B parameters from 8.4x of the dense model to only 1.7x of the dense model after 2-bit quantization. It still preserves 1.88\% higher accuracy than the dense model. Combined with an optimized GPU runtime implementation, it also achieves 2.7X speed-up which is even slightly faster than the FLOPs equivalent dense model.

Under review as a conference paper at ICLR 2023
MIXTURE OF QUANTIZED EXPERTS (MoQE): COMPLEMENTARY EFFECT OF LOW-BIT QUANTIZATION AND ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
Large Mixture of Experts (MoE) models could achieve state-of-tee of--he--rttttttt quality on various language tasks, including machine translation task, thanks to the efficient model scaling capability with expert parallelism (Fedus et al.) 202T). However, it has brought fundamental issue of larger memory consumption deployment time. Furthermore, this results significant inference speed degradation at auto- regressive decoding steps due the increased memory transfers. In this paper, we propose Mixture of Quantized Experts (MoQE) which is simple weight-only quantization method applying ultra low-bit down 2-bit quantizations only to expert weights for mitigating the increased memory and latency issues of MoE models. We show that low-bit quantization together with the MoE architecture delivers a reliable model performance while reducing the memory size signifi- cantly even without any additional training. Especially, expert layers in MoE models are much more robust to the quantization than conventional feedforward networks (FFN) layers. In our comprehensive analysis, we show that MoE models with 2-bit and 80% sparse expert weights can deliver better model performance than the dense model trained on the same dataset. We present how quantization of different parts of models affects the performance with various experiments us- ing a large MoE model (5.3 B). As a result of low-bit quantization, we show the model size can be reduced by 79.6% of the original half precision floating point (fp16) MoE model. This cuts down the model size of 5.3B parameters from 8.4x of the dense model only 1.7x of the dense model after 2-bit quantization. It still preserves 1.88% higher accuracy than the dense model. Combined with an optimized GPU runtime implementation, it also achieves 2.7X speed-up which is even slightly faster than the FLOPs equivalent dense model.
INTRODUCTION
Large Language Models (LLMs) have shown their effectiveness on various language tasks by in- creasing the number of trainable parameters together with the framework of pre-training model on a large scale data and using it to different downstream tasks (Devlin et al.] 2018]] Radford et al.
2018 Liu et 2019 Raffel et al., 2220).. With the advancement of distributed large scale training methods (Shazeer et al. 2018] Rasley et al. [2020] Ren et al. [2021] Baines et al. 2021) and large scale data collection (Raffel al.) 2020; Hoffmann et al. (022), the models get even larger and break state-of-the theooff-t--rr-- art performance with the increased model capacity (Brown et al.
2020 Rae et al.,
2021 Zoph al. [0022] Zhang et al., 2022] Smith al.) [2022] Chowdhery al.]
2022 However,
the cost of training these models increases whenever more parameters are added, and this may not be sustainable.
As solution to address this issue, sparsely activated models (Shazeer et al., 2017) are more widely adopted and show significant efficiency improvements in terms of model size scaling while enabling up to trillions of parameters be trained more efficiently and achieving better model accuracy (Lepikhin et al., 2020; Fedus et al. 2021 Kim et al. 2021 Artetxe et al. 2021). Mixture-of ofituure-o--Experts type of sparsely activated models replacing single layer in model with a group of parallel layers which are called experts combined with a gate layer. For a given input, the gate layer selects subset of the experts from the group, and use them for processing
(MoE) models

---

Under review as a conference paper at ICLR 2023
the input. By limiting the number of subset layers for given input to one or two, the theoretical FLOPs stays almost constant even if we add hundreds of parallel layers into the MoE group. Thus far, most studies have shown that it is effective increase the capacity of the models by replacing feedforward networks (FFN) of Transformer (Vaswani et al. (017) blocks with MoE layer consists of multiple FFN layers together with a gating network (Lepikhin al. 2020 Fedus et al.) 202T
Kim et al. 2021; Artetxe et al. 2021). One of the most unique and critical components of MoE models is the gating network which decides how to conditionally select experts for each input, and there have been various studies to improve it to achieve better training convergence ([Lewis et al.,
2021 Roller et al., [021]] Zuo al. [02T] Clark et al. 2022 Liu al. 2022
Zhou et al., 2022) and
they are well surveyed in Fedus et al. (2022).
In spite of the progress on the training of MoE models, there have been only a few handfuls of studies related to MoE model inference. Rajbhandari et al.] (2022) designs a more efficient MoE architecture and distributed runtime to achieve 7.3X inference speed-up. [Kudugunta et al. (202T) uses task specific information to reduce the size of the model at deployment time by only loading task specific experts. Kim et al. (202T) prunes some experts at deployment time to reduce the model size by trading-off model performance. Zoph et al. (2022) uses knowledge distillation technique to distill a large MoE model into a smaller dense model to reduce the memory consumption and improve the throughput. Even with all the proposed techniques, there has not been solution to accelerate the inference of MoE models while maintaining the accuracy.
Quantization is a type of model acceleration and compression techniques by estimating a floating point number into smaller precision number. There are various studies that show quantization is effective to accelerate neural network model inference (Rodriguez et al., 2018; Stock et al. 2019
Choukroun et al. 2019; Gholami et al. (022). Especially, it has been known to be very effective in natural language generation such as machine translation ((Kim al. 2019; Aji & Heafield 2020 Fan et al. 2021)) and natural language understanding (Kim & Awadalla (020) tasks. However, there has not been an in-depth study about how quantization works with large MoE models.
Recently, Dettmers et al. (2022): Yao et al. (0022) have studied how quantization works on large scale language models. Dettmers et al. (2022) looks at outlier features in the activations of large language models, and proposes to decompose them while performing matrix multiplications. In our quantization method, this is not needed because is a weight-only quantization and outliers in acti- vations cannot affect the performance. And, the weights are dequantized back to fp16 while matrix multiplication is done. This also makes our approach not require a special low-bit instructions. And, show that this can be applied to lower bits than 8-hit for large MoE models. ZeroQuant (Yao) et al, 2022) presents a series of techniques including knowledge distillation (Kim & Rush 2016) for achieving higher quality quantization. Our foeus is to exploit the intrinsic characteristics of MoE layers based on our investigation, and we show that a simple quantization algorithm can achieve significantly higher efficiency and maintain the quality at the same time.
Our contributions in this paper are as below.
We present extensive studies about how applying low-bit (down 2-bits) quantization to different layers of MoE transformer models affects the model accuracy together with comparisons the corresponding dense model with the same embedding size.
We show that expert weights are highly robust to the quantization, therefore they can be quantized to 3-bit without additional training calibration data and to 2-bit with Quan- tization Aware Training (QAT) which results in 79.6% reduction in memory size. Com- bined with a runtime optimization, we show that the method boosts the inference speed significantly more than 2.7X faster. We leverage the memory bounded characteristic of auto-regressive decoders, reduced memory bottleneck improves the overall efficiency even with additional dequantization steps in our procedure. Based on the observations, we propose a new framework named Mixture of Quantized Experts (MoQE) which is simple weight-only quantization method only applied to MoE expert weights.
Finally, we show emerging sparsity of more than 80% in the expert weights to be zero from 2-bit quantization. The expert weight matrices are sparse and very low-precision at the same time, while still outperforming the dense counterpart trained on the same dataset.
2

---

Under review as a conference paper at ICLR 2023
2
BACKGROUND CHALLENGES OF DEPLOYING MOE MODELS
In the widely used MoE architecture, even with a constant or only sub-linearly higher theoretical FLOPs by using top-1 or top-2 gating. the increased model size with additional experts has serious negative impact on the inference performance in various aspects.
2.1 INCREASED MEMORY FOOTPRINT
First of all, due the increased model size, the model requires much more accelerator memory. With modern accelerators like GPUs, the accelerator memory size is limited. So, more accelerators are required to handle model which causes communication problem described next. Also, the model takes up more memory, so the batch size is limited to be small which prevents the optimal utilization of processing cores.
2.2 SLOWER INFERENCE SPEED
Increased communication overhead. In the distributed training and inference set-up for large scale models, is natural to use many GPUs or accelerators for a single model. The model weights can be distributed across different accelerators with various techniques (Ren al. 2021) and expert parallelism (Fedus et al. 2021). However, in Liu et al. (002)), it is shown that the communica- tion overhead with expert parallelism at training time could take up more than half of the entire end-to-end time depending on the number of GPUs and clusters. This could affect inference effi- ciency even more severely because inference usually needs fewer FLOPs numbers than training, and communication bottleneck will stand out more. Therefore, it is desirable to use as few numbers of accelerators as possible to avoid this overhead.
Memory bandwidth bottleneck with MoE layers. The increase in the model size not only causes communication overhead, but also brings a significant inference speed impact on the modern pro- cessor architectures. While performing beam search decoding, the size of activation (an individual token) is relatively small and the decoding operation memory bandwidth bounded. This means transferring model weight matrices in memory hierarchy is huge bottleneck. With the increased number of experts, the burden of memory transfer increases even more, and directly impacts the inference speed.
Inference speed measurement. Table I shows an actual speed difference measured with dense NVIDIA's V V100 GPU. Two models are encoder and decoder based the
and MoE models transformer architecture (Vaswani et al. 2017). and have exactly the same model settings except for the number of experts. The speed measurements English using auto-regressive beam search with beam size of five. Both models are evaluated on the same PyTorch, with half-precision floating point (fp16). The MoE model uses top-1 gating which assigns only one expert for a given input token which provides the same theoretical FLOPs as the corresponding dense model (with the same embedding size). Due the excessive memory transfer caused by the increased number of experts, the actual inference speed decreases by 60% of original dense model's speed as shown in the table.
done on the translation task from German to
Table 1: Inference speed measurements and model sizes of dense and MoE models. Both models run with batch size of 24 and the throughput is measured with the number of sentences processed for one second.
Throughput
Model size
% of MoF weights
Model
(sentences/second)
(fp16) in GB
Dense
14.02
1.18
MoE (32 experts)
5.37
9.91
92.8 %
Difference
0.38X
8.38X
To overcome these challenges, we focus on reducing the model size utilizing quantization. Espe- cially, increased model size and latency are mostly from the expert FFN weights which contribute

---

Under review as a conference paper at ICLR 2023
92.8 % of all weights in this specific model setting, the FFN weights main target for the optimization. With an emerged sparsity in expert weights from the low-bit quantization, we also explore further sparsification opportunity with simple magnitude pruning technique.
QUANTIZATION METHODS FOR MoE LAYERS
There are multiple design choices to quantize model weights. In this section, we analyze the nu- merical characteristics of different layers in large MoE model, and describe the decisions have made to most effectively quantize the MoE layers.
3.1 NUMERICAL DISTRIBUTION OF MODEL WEIGHTS
While quantizing matrices, it is desired not to have outliers, but to have smoothly distributed numer- ical values. Outliers usually skew the range be quantized and scaling factors get too large. Figure
1 shows weight distribution box plots of linear layers in the MoE model's ' FFN blocks. Following
the widely used practice, an MoF layer is in every other layer (Lepikhin et al. 2020 Fedus et al.)
2021 Kim ct al. 2021). Even number layers {0, ... are expert FFN layers and odd number layers 11.3.... ...j are normal dense FFN layers. First, all of them are centered around zero. However, dense FFN layers have much larger range than MoE FFN layers. This indicates that dense FFN layers have more outliers than MoË FFN layers. This phenomenon is more prevalent in the second linear layers sometimes reaching down -8.0 which is shown in Figure Tb Figure shows example his- tograms of an expert FFN weight and dense FFN weight. As can be seen in Figure 2b the example dense FFN layer suffers from outliers seriously. However, expert FFN weights in Figure 2a show smooth distribution without any major outliers. We observe the similar pattern all across different we additionally include the statisties of overall layers.
layers and different experts. In Appendix C This statistical observation indicates that MoE FFN layers would be well suited for the quantization.
Based on the observation, the FFN weights are following a normal distribution with a mean value near zero. Therefore, we use symmetric quantization without needing to shift the zero point. Even for the dense FFN layers, the means and the standard deviations are around zero except for the outliers which can be seen in the box plot of Figure [ This symmetric quantization also gives an advantage to quantize many weight values near center to zero which could result in a sparse model weight.
0-
Box plut
-4
30
Layers
Layers
(a) PFN linear weight distribution across layers (b) FFN linear 2 weight distribution across layers
Figure/ FFN weight distribution across layers. Even number layers (0, 2,.} are expert FFN layers and odd number layers (1.3...) are normal dense FFN layers. (a) shows the first linear layer FFN and (b) shows the second linear layer in FFN.
3.2 QUANTIZATION ALGORITHMS
3.2.
QUANTIZATION TECHNIQUES
We try two quantization techniques, they are (i) linear quantization which is mapping quantized integer values and the original float value uniformly and (ii) log- quantization from Aji & (202)) which maps integer and float ranges a log scale. In both cases, we choose channel- wise quantization matrix-wise quantization based on the experiment in Appendix A
4

---

Under review as a conference paper at ICLR 2023
200900
30030
2000
330307
22281
455 (b) Example FFN weight distribution (layer 7. FFN 2)
(a) Example expert weight distribution (layer 6, FFN 2, expert 15)
Figure 2: A comparison of example weight distributions from MoE and dense FFN layers.
Linear quantization with absolute maximum. The first technique is linear quantization which, given a matrix A and bits, encodes as follows:
2 x max Al
$)
2
Q = int
8j
where s the scaling factor which can be chosen per channel as shown or per the whole tensor. At inference time, quantized Q dequantized back to A with the scaling factor a as follows:
A Q.. X X
Log-scale quantization. The second technique is log-scale quantization where bit kept for the sign and (b 1) bits are used to encode the log-scaled values. Given a matrix A, the quantization formula as follows:
P sign(A)
T = dipp
A
1.2 1.22-2"-1)
Q = [log2( 2 T)
where 8 can be chosen in two ways, either (i) the absolute maximum or (ii) the optimal value to min- imize the mean squared error (MSE) between the quantized and original values which is described in Aji & Heafield (2020). We use the second algorithm which we observe a better accuracy with the quantization. At inference time, the quantized weight values are dequantized based on the formula as follows:
A' PXPxxx2
Comparison quantization techniques. Figure shows the comparison between two quantization techniques with low bits applied on expert FFN layers and dense FFN layers. For dense FFN layers, log-scale quantization performs slightly better, but both not work well on 2-bit resulting in almost zero evaluation scores. For expert FFN layers, both techniques work similarly for 3 and 4 bits, but log-scale quantization loses the accuracy seriously with 2-b This is because there are only 4 bins for the integer values to quantize with 2-bit quantization and one of them is zero. Log-scale tries to split values near zero in a more fine- mee-griieed way, but this actually hurts the performance compared to having enough zeros with linear quantization. Based this experiment, for compressing MoE FFN layers.
linear quantization
3.2.2 ROBUSTNESS OF EXPERT LAYERS TO QUANTIZATION
To better understand how applying quantization on different parts of an MoE model affects the accuracy, conduct set of experiments with various quantization bits. We divide MoE model
5

---

Under review as a conference paper at ICLR 2023
40-
Linear
BLEU (DF-EN)
35- 30- 20-
33
Log-scale
(DB
23
16-
Linear
Log-scale
8
3
Quantisation bits
Quantization hits
(a) Expert FFNs
(b) Dense FFNs
Figure Linear quantization vs log-scale with optimal 8 quantization.
into four parts: (i) expert FFNs, (ii) dense FFN layers, (iii) self-attention layers and (iv) cross- attention layers. Based on the observation that linear quantization works better with lower bits, we use it for this set of experiments.
Figure 4 shows evaluation BLEU scores when quantizing different parts of the MoE model. We ob- serve that quantizing expert FFN layers to 2-bit does not seriously impact the overall model quality. However, quantizing other parts of the model into 2-bit hurts the output quality significantly. Quan- tized cross-attention and self-attention blocks still can maintain the quality with 3-bit quantization, but their performance gets impacted with 2-bit quantization. On the other hand, dense FFN layers get significant impact with lower bit quantization of 2-bit and 3-bit. With 3-bit quantization, the model score drops 23 of original score, and 2-bit quantization on dense FFN layers gives almost zero score. We also include the same study a dense model in Appendix the similar pattern with 2 and 3 bit quantization is observed.
40
BLEU (DE- (DE-EN) BNN)
35
30
Expert
Dense FFN
Self attention
5
Crossinttenton
4
6
8
Quantization
Figure 4:
Quantization impact on different MoE model parts (channel-wise linear quantiztation
without any additional training).
3.3 MIXTURE OF QUANTIZED EXPERTS (MOQE)
Based on the experiments from the previous parts of this section, we propose a very simple, highly effective and accurate quantization recipe for MoE models.
Apply weight-only quantization while keeping activations in fp16.
Quantize expert FFN layers only.
Use channel-wise and symmetric guantization.
Choose one of either two quantization methods depending the quantization precision
1. (3-bit or higher bit): Directly quatize trained MoE models without additional calibra- tion.
(2-bit): Fine-tune the model with Quantization Aware Training (QAT) which the de- scribtion follows.
Sparse experts with quantization aware training QAT is a well-known method used to recover the accuracy loss from the quantization (Gholami al. 2022). In our to quantize to 2-bit precision, we can continue training the model with the original training data while applying quantization only the forward pass computation as presented in (Wu et al.] 2020: Bengio

---

Under review as a conference paper at ICLR 2023
al. 2013) for recovering the accuracy loss. As symmetric quantization with 2-bit, zero numerical value is always included. Due to the normal distribution of expert weights centered around zero, many weight values naturally turn into zeros. This procedure results in sparse expert matrices.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Task. We use multilingual machine translation task for our experiments with two different dataset which are 20 language directions and 10 language directions respectively. We also evaluate the proposed method on different task presented in Appendix (D We use sacrebleu the detokenized output to measure the accuracy of the models. A single NVIDIA PCIE V100 running inside a docker container running Ubuntu 20.04 and CUDA 11.6 is used for all experiments, and all code compiled with nvec and goc/g++ 9.3. We measure end-to-end runtime of the inference for the evaluation dataset.
Datasets. We use two different datasets described below. For the larger dataset setting, we use inter- nally collected dataset consists of different languages which German (de), French (fr), Italian (it), Spanish (es), Dutch (nl) and English (en). They are crawled from web, and each language pair has at least several hundred million sentences. We use 128,000 sub-words vocabulary built with sentencepiece library. librrr. The number of training sentences is included in Appendix G
For the smaller dataset setting, we use WMT-10 benchmark dataset widely used for public bench- marks (Wang et al. (0020) Kim et al. 2021). There are 32.5 million sentence pairs for English-centrie 20 language pairs including French (fr), Czech(es), German (de), Finnish (fi), Latvian (lt), Estonian (et), Romanian (ro), Hindi (hi), Turkish(tr) and Gujarati (gu).
Model architecture. For all the experiments with large dataset, we use 24 transformer (Vaswani et al. encoder layers and 12 transformer decoder layers following the deeper encoder and shal- lower decoder practice (Kim et al. 2019 Kasai et al. 202T) to more efficient at auto-regressive decoding. The embedding dimension is 1, 022 and FFN hidden dimension is 4, 096. For the po- sitional information encoding to the hidden state, we use Transformer with Untied Positional En- coding (TUPE) proposed in Ke al. (2021) instead of the conventional sinusoidal positional em- bedding. Another design choice is the location of layer normalization. For the training stability, pre-layer normalization proposed in Xiong et al. (2020) instead of the original post-layer normalization from (Vaswani et al. (017).. We train MoE and dense models for the comparison. The model architecture choices mentioned here are common for both models. The only difference between dense and MoE models is the number of experts. We use 32 experts for the MoE model trained with the larger web data. We use beam search decoding with beam size of 5. For the experi- ments with smaller dataset, we use 12 transformer encoder layers and 6 transformer decoder layers. The embedding dimension is 768 and FFN hidden dimension 3, 072. this setting, we use MoF layers with 128 experts at every other layer.
MoE architecture. For the MoE model specific settings, we use top-1 learned gating from Fedus et al. (202T) and use MoE layer at every other layer which are even numbered layers (Lepikhin eet al. 2020; Fedus et al. 2021 Kim et al. 202T). During the training of MoB models, we use jittering noise and balancing loss (ratio of 0.01) suggested in Lepikhin et al. (2020); |Fedus et al. (2021) to more uniformly distribute expert utilization. To prevent overfitting and better regularize
the model, we gating dropout (0.2) (Liu et al. 2022) as well.
4.2 MOQE PERFORMANCE RESULTS
We apply MoQE quantization recipe to an MoE model and compare the performance with several dense models in Table This experiment is done on the larger web dataset. The baseline is dense model trained on the same dataset as the MoE model. Throughput, memory size and sparsity are all measured with the fp16 precision model. As additional comparison points, the dense model is also
7

---

Under review as a conference paper at ICLR 2023
Table 2: The model performance comparison. All the models are trained on same data up to the convergence with 200,000 update steps. The baseline is the FLOPs equivalent dense model's BLEU score and speed.
Average BLEU
Throughput
Size (X times)
Model type
Precision
Sparsity
(difference %6
(X times)
Dense (baseline)
fpl6
45.06 (0)
1X
IX
3.8e-5
int8
45.04 (-0.02)
Not optimized
0.88X
1.28
Dense
int4
44.84 (-0.47)
Not optimized
0,82X
21.31
tpl6
46.35 (+2.87)
0,38X
8,38X
3.8c-5
int8
46.34 (+2.85)
L,0XX
4.57X
1.24
int4
46.18 (42.49)
L,0XX
2.67X
20.68
inL3
46.01 (42.11)
Not optimized
2.19X
42.15
MoE 5.3B (32 experts)
int2
45.90 (+1.88)
Not optimized
1.71X
79.10
QAT
quantized to 8-bit and 4-bit only on the even numbered FFN layers which is the best configuration for quantizing the dense model described in Appendix [ For the MoE model, various quantization settings ranging from 8-bit to 2-bit are measured together with the original fp16 performance. For 2- bit quantization, additional QAT is applied. Finally, applied magnitude based pruning approach to the 2-bit quantized model to acquire a sparser model.
First of all, the MoE model achieves 2.87% improvement on the BLEU score while increasing the model size to 8.38X of the original dense model. When 4-bit post-training quantization is applied, still maintains 2. % higher BLEU score than the original dense model. And, could achieve even faster speed than the dense model which is 2.7X speed-up from the fp16 MoE model. This also reduces the memory consumption significantly from 8.38X to 2.67X compared model. With 2-bit QAT, the MoE model can still maintain 1.88% higher quality than the original dense model, but the model size is now only 1.71X of the original dense model. Also, the matrices sparse up to 79.1 19.1% 1% of the values are zeros.
the dense
Figure shows the sparsity distribution of different layers. The second linear layers after the non- linear activation layers show higher sparsity compared to the first linear layers. Some layers could reach up to 85% sparsity. We include a further investigation of sparsity with magnitude based pruning approach in Appendix
0.86
0.84
0.8
0.78
0.72
0.7
Encoderilnes
Encoder 2
Decoder Incor
Decader Incor
Figure Sparsity distribution of 2-bit quantized MoE layers.
4.3 ROBUSTNESS COMPARISON BETWEEN MoE AND DENSE MODELS
We compare robustness against low-bit quantization between MoE and dense models using the post- training quantization without any QAT. For the dense model, quantization with different bits is applied to the numbered FFN layers. Appendix B shows this is the best layer selection for the dense model. We use two different datasets to verify the proposed quantization method works in different model settings.
8

---

than Under review as a conference paper at ICLR 2023
(10 langrage paLas)
DLEU aveeage
85
25
15-
MoE model - expert weights
Dense model mumber layer FFNs
Quantization bits
Figure Quantization performance comparison between MoE and dense models. 10 different language pair scores are averaged.
Figure 6 presents the experiment with the model trained with the larger dataset. shows the average BLEU scores with different quantization precision for both MoE and dense models. The MoE model can maintain accuracy within -0.3 down to 3-bit and -1.82 for 2-bit. On the other hand, the dense model preserve the accuracy only down to 4-bit, but starts lose significant accuracy 2 BLEU scores when it goes down to 3-bits. In case of 2-bits, dense model loses most of capability by -42.96 BLEU scores. Table shows the score differences by quantization for both MoE and dense models on 10 different language pairs translations.
Figure presents the experiment with the model trained with the smaller dataset. In this setting, each individual expert is smaller, but there are 4 times more experts in one MoE layer. And, they are trained with smaller dataset, they do not have equivalent knowledge as the previous model trained on the larger dataset. As can be seen in the Figure, the quantization performance shows similar pattern. The MoE model preserves accuracy even when it quantized model quickly loses the performance when it is quantized down to lower than 4-bit. Again, the MoE model much more robust to quantization than the dense model.
or bits. However, dense
21
(10 language paiss
21
BLEU sverag
12
MoE model - expert weights
6
Dens: model even number layer FFNs
Quantixation bits
Figure 7: Quantization performance comparison between MoE and dense models. different WMT language pairs averaged.
CONCLUSION AND FUTURE WORKS
This paper shows how much MoE models are robust to the low-bit quantization with various ex- periments. By analyzing component-wise sensitivity and various quantization design choices, we present an efficient and effective way to reduce the model size which results in 4.9X model size reduction. With an optimized runtime, 4-bit quantized model can run 2.71X faster than the fp16 model. also show 2-bit quantization could achieve more than 79% sparsity in the expert weights. The results naturally bring interesting future research directions. The discovered robustness of pert layers can guide a better way to train MoE models. If we can better control the splitting of latent space, better MoE models can be acquired. Analyzing the interactions between expert FFN layers and the other common layers in the model could guide a way to build composable model. Especially, presented in Appendix we observe that quantization sometimes improves the accu- racy on tasks in a specific situation. Another important direction will be studying how to accelerate sparse expert computation on modern hardware with software/hardware co-design. This will even- tually make MoE models much more efficient in both training and inference.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alham Fikri Aji and Kenneth Heafield. Compressing neural machine translation models with 4-bit precision. In NGT, 2020.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Out, Sam Shleifer, Xi Victo- ria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112.10684 2021.
Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, et al. Fairscale: A general purpose modular pytorch library for high performance and large scale training. 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 300 343..333.2222 2013
TrXXiv:300.33322
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 202..
Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3009-3018, 2019.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03111, 2022.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pp. 4057- 4086. PMLR, 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int((; 8-bit matrix multi- plication for transformers scale. ArXiv, 0axx2200.073333 2022.
Jacob Devlin, Ming- Heig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. ArXiv, abs/2004.073200 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.0166) 2022.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint arXiv:1902.099
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. ArXiv, abs/2103.13630, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint 2rX1v:2203.15556 2022.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In ICLR, 2021.
Guolin Ke, Di He, and Tie- Yan Liu. Rethinking positional encoding in language pre-training. ArXiv, abs/2006.1555 2021.
10

---

Under review as a conference paper at ICLR 2023
Yoon Kim and Alexander M Rush. 0rXiv:1606.07947, 2016.
Sequence-level knowledge distillation.
arXiv preprint
Young Jin Kim and Hany Hassan Awadalla. Fastformers: Highly efficient transformer models for natural language understanding. arXiv preprint 200::::2000.33882 2020.
Young Jin Kim, Marcin man Grundkiewicz, and Nikolay Bogoychev. From research to production and back: Ludicrously fast neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 280-288, 2019.
Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Ro-
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and effi cient moe training for multitask multilingual models. arXiv preprint arXiv:2109. 10465, 2021.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In EMNLP, 2021.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 2002:::000..6668, 2020.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In ICML, 2021.
Rui Liu, Young Jin Kim, Alexandre Muzio, and Hany Hassan. Gating dropout: Communication- efficient regularization for sparsely activated transformers. In International Conference on Ma- chine Learning, pp. 13782-13792. PMLR, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745 2018
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 22::211/2111 1446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 21(110) (140): 2020.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-mon Advancing mixture-of-experts inference and training to power next-generation ai scale. In ICML, 2022.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyang Yang. Min- jia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training. In USENIX Annual Technical Conference, 2021.
Andres Rodriguez, Eden Segal, Etay Meiri, Evarist Fomenko, Young Jin Kim, Haihao Shen, and Barukh Ziv. Lower numerical precision deep learning inference and training. 2018.
11

---

Under review as a conference paper at ICLR 2023
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:17555-17566 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprini urXiv:/701.000 017770006388, 2017.
Noam M. Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanan- takool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Mesh-tensorflow: Deep learning for supercomputers. ArXiv, abs/181 1.02084, 2018.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprini arXiv:2201.11999 2022.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. arXiv preprint N.ii::9900006880 2019.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Yiren Wang, ChengXiang Zhai, and Hany Hassan Awadalla. Multi-task learning for multilingual neural machine translation. arXiv preprint arXiv:2010.02523, 2020.
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quan-
tization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602, 2020.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architec- ture. In ICMI., 2020.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-traiming quantization for large-scale transformers. ArXiv, abs/2206.01861, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106. 2022.
Yan- Gan-Quan Zhou, Tao Lei, Han-Chu Liu, Nan Du Yanping Huang, Vincent Zhao, Andrew M. Dai, Zhifeng Chen, Quoc V. Le, and James Laudon. Mixture-of-experts with expert choice routing. ArXiv, abs/2202.09368,, 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906, 2022.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:21 10.:22100.04200 2021.
12

---

Under review as a conference paper at ICLR 2023
A CHANNEL-WISE VS NATRIX-WISE QUANTIZATION
Scaling factors are calculated by the quantization algorithm and stored in half precision floating- point (fp16) numbers to dequantize the matrices with. These factors can be chosen on the channel scale or the whole matrix scale. As shown in figure g channel-wise quantization gives quite higher scores than tensor-wise especially for low precision. Additional parameters to store channel- -iie scaling factors small, because only one value is needed for a channel and less than 1% of to- tal parameters in matrix. Therefore, experiments.
channel-wise quantization for all the quantization
BLEU DDE-E EN)
10
39
38
37
36
35
Channel wise quantization
34
Matrix-wisc quantization
33
2
3
8
Quantization bits
Figure 8: Linear quantization of expert FFNs with channel-wise and matrix-wise scaling factors.
B QUANTIZATION OF DIFFERENT LAYERS IN A DENSE MODEL
In the paper, we compare dense model and an MoE model terms of quantization robustness. To make a fair comparison, we consider quantizing only half of the dense transformer blocks' FFNs, because we quantize expert weights only which exist only in every other block (even numbered). We compare three different configurations (1) quantizing even numbered blocks' FFNs only, (2) quantizing odd numbered blocks' FFNs only and (3) quantizing all FFN layers. As can be seen in Figure B quantizing
numbered blocks' FFNs affects the accuracy the least, and quantizing all FFN layers give the worst result. Based on this experiment, we quantize only even numbered transformer blocks' FFNs for the dense model in all the experiments and comparisons.
40
(10 language pairs)
BLEU average
30
25
20
Even mumber layers
15
10
Odd number layers
All layers
0
2
3
5
7
Quantization bits
Figure 9; Quantization impact of different layers in a dense model.
C SKEWNESS OF WEIGHT MATRICES IN MOE AND DENSE MODELS
In the analysis of model weight distribution in Section observe that dense models' FFN layers tend have more outliers than MoEs' expert FFN layers. We measure the skewness of weight distribution of those in Table 3
13

---

Under review as a conference paper at ICLR 2023
Table 3: Expert vs non-expert FFN layers parameters distribution skewness
Parameter
skew
encoder expert 1.5 FFN fol layer O
-0.002
encoder expert FFN fc2 layer
-0.190
encoder expert 1.5 FFN fol layer 6
-0.002
encoder expert 1.5 FPN Tc2 layer 6
-0,002
encoder non-expert FFN fel layer
-0.199
encoder non-expert FFN fc2 layer
-10.729
encoder non-expert FFN fel layer? 7
0.003
encoder non-expert FFN fc2 layer
-1.574
encoder expert FFN encoder expert FFN fc2 mean
mean
0.00
-0.63
decoder expert FFN fcl mean
0.00
decoder expert FFN mean
0.48
encoder non non-expert FPN fol mean
0.00
encoder non-expert FFN [c2
-.84
decoder non non-expert FFN fel mean
0.00
decoder non non- expert FFN fc2 mean
-0.09
D ABSTRACTIVE SUMMARIZATION TASK PERFORMANCE
To validate the quantization performs well on different task and model, we evaluate B MoE (64 experts) model's quantization performance on an abstractive summarization task called XSUM (Narayan et al. 2018). Table shows that the MoE model performs well with low-bit quantizations such as 2-bits and 3-bits.
Table 4: The ROUGE score differences in percentage (%) after applying post-training quantization on XSUM evaluation.
Quantization Bits
R1
R2
RL
8-bit
+0.17
+0.13
+0.26
4-hit
-0.10
-0.26
-0.34
3-bit
-0.21
-0.82
-0.28
2-hit
-1.61
-5.94
-2.37
E BETTER GENERALIZATION WITH EXPERT QUANTIZATION
We observe an interesting phenomenon that quantization actually improves the score of evaluation on different domain dataset. We trained MoE model with 64 experts (I0.IB) 50 different language translations (98 English-centric language pairs). When we evaluate this model on a differ- ent domain subset 6 languages (German, Spanish, French, Italian, Dutch, English), the evaluation BLEU score increases until we quantize the experts down to 3-bits without any additional QAT or calibrations. With 3-bit quantization, the score increases more than 6.42% on non-English En- glish and 6.96% on English to the others. Especially, from English to Italian and from Italian to English scores increase more than 10% which quite significant. The results are summarized in Table 5 We are analyzing what could be the reason for this phenomenon, but we think this is related to how MoE models learn representations. MoE layers might learn very specific knowledge with its increased capacity, but the shared layers learn more generic knowledge. By blurring the representa- tion from the MoE layers, the model becomes more general task capable. This one of our future research areas.
14

---

Under review as a conference paper at ICLR 2023
Table 5: pairs.
The BLEU score differences in percentage (%) after quantization on different language
Quantization Bits
de-en
fr-en
it-en
nl-en
Average (xx-English)
fp16
40.43
47.83
46.46
39.67
43.13
43,50
8-bit
-0.01
-0.08
+0.17
+0.25
+0.06
+0.08
4-bit
-0.04
+5.60
+2.83
47.72
+5.39
+4.30
3-bit
-0.47
+9.22
+4.02
+11.97
+7.38
+6.42
2-bit
-7.59
-0.30
-7.52
-4.45
-4.34
-4.84
en-de
en-es
en-fr
en-it
en-nl
Average (English-xx)
fp16
36.74
40.98
45.99
29.13
38.72
38.31
8-bit
-0.03
+0.13
-0.19
+0.06
+0.29
+0.05
4-bit
+0.88
+6.86
+3.55
+6.52
+2.59
+4.08
3-bit
+0.25
+9.65
+5.24
+16.09
+3.57
+6.96
2-bit
-19.58
-9.11
-6.99
-2,22
-14.68
-10.52
F ADDITIONAL SPARSITY DISTRIBUTION
We include sparsity distribution across different layers in a dense model quantized with 4-bit. As can be seen in Figure I0 the sparsity is overall low. Second linear layers in FFN show slightly higher sparsity, but all of them are smaller than 30%.
0.7
0.6
0.5
0.4
0.3
0.1
23
10
Encoder linear
Encoder linear
Decoder linear
Decoder linear
Figure 10: Sparsity distribution of 4-bit quantized dense layers.
G MACHINE TRANSLATION DATASET SUMMARY
Table shows the number of parallel sentences used to train dense and MoE models. All languages have at least 300 million sentences and the differences in the number among languages are less than two times.
Table 6: The number of parallel sentences including backtranslation data.
Number of parallel sentences (million)
Language
English
English
DE (German)
505
ES (Spanish)
448
407
FR (French)
448
376
IT (Italian)
447
303
NL (Dutch)
302
15

---

Under review as a conference paper at ICLR 2023
H QUANTIZATION AWARE TRAINING (QAT)
For the QAT with straight through estimator, we use the hyper-parameters in Table 7 Figure 11 shows the validation loss curve of one training run with 2-bit expert quantization.
Table 7: Expert vs non-expert FFN layers parameters distribution skewness
Parameter Name
Value
Start learning rate
0,001
Warm-up up steps
1000
Effective batch size
1. 1.5M tokens
Optimizer
Adam
Betas (Adam)
(0.9, 0.999)
Weight decay (Adam)
0
Gradient clipping
0.1
Training steps
60,000
Validation loss (cross entropy)
3.40
3.39
3.38
3.37
10000
20000
30000
40000
50000
60000
Steps
Figure 11: Validation loss curve for QAT.
I MAGNITUDE PRUNING EXPERIMENTS
Inspired by the emerged sparsity in expert layers, we apply a simple magnitude based pruning to the MoE model we experiment with. We apply different threshold values from 0.00001 to 0.5. We make all the weight values less than the threshold to be zero. We apply 2 to 8 bit quantization together. Figure shows how model performance varies with the achieved sparsity. Even with sparsity level of 90%, the model preserves a certain level of task capability. Compared to Gale et al. (2019). this shows much higher performance with the same sparsity. This could be another example showing the robustness of expert weights.
J DETAILED BLEU SCORE DIFFERENCES WITH QUANTIZATION APPLIED TO THE MODEL TRAINED ON PUBLIC WMT DATASET
Table shows individual BLEU score changes with various quantization bits for MoE and dense models trained on public WMT dataset.
16

---

Under review as a conference paper at ICLR 2023
45
BLEU (DE-EM BE-EN)
30
20
15
10
5
0
0.0
0.1
0.2
0.3 0.4 0.5 0.6
0.7
0.8 0.9
1.0
Sparsity
Figure 12: Sparsity of MoE layers vs. BLEU scores with magnitude pruning.
Table 8: The BLEU score differences in percentage (%) after quantization on different language pairs in WMT dataset. The rows with fp16 show the baseline BLEU scores.
Bits
Model
en-es
en-le
en-el
en-fi
en-fr
en-gu
en-hi
en-ly
em-rr
en-ir
Arg.(en-xx)
Ip16
31.46
17,80
10.34
27.22
(BLEU)
MoB
26.09
34.36
18.27
22.17
31.34
13.04
12.16
23.26
27.95
16.89
22.55
8-bit
Dense
.0.99
.0.99
40.32
0.60
0.01
40.00
0.61
.0.26
0.17
00.09
40.05
MoE
-0.15
0.64
-0.33
0.19
0.86
0.02
-0.15
-0.03
0.05
4-bit
Dense
-1.11
-1.91
-3.15
-1.50
1.03
-7.08
4.44
-2.38
-1.65
-1.89
-1,90
MoB
00.0
-0.62
0.30
-0.62
-0.13
-0.97
1.53
-0.81
-0.82
40.22
40.36
-1.88
-32003
-7.05
11.74
11.24
3-bit
MoB
-0.84
-1.06
-1.79
-1.97
0.35
-2.80
-0.70
-1.98
-1.05
-1.64
-1.21
2-bit
Dense
99.44
-6629
-91.79
-91.02
-85.75
-98.26
-96488
-94.14
-87.30
-95.02
.91.21
MoE
-9.15
-17.06
-25.24
-16.11
-11.04
-12.34
Bits
Model
even
desen
el-en
fiven
fr-en
gu-en
hi-en
Iv.en
moeen
tr-en
Arg.(xx-en)
Ip16
35.62
(BLEU)
MoB
31.25
38.21
23.67
25.64
32.59
19.55
15.89
25.22
34.80
20.27
26.71
8-bit
Dense
0.02
-0.02
0.10
-0.33
-0.15
40.37
40.40
0.33
-0.34
0.14
40.09
MoB
0.12
0.08
0.06
-0.10
0.14
-0.49
-0.03
0.05
-0.17
0.00
4-bit
Dense
.0.44
.0.78
-3.74
-1.72
-1.69
4.58
40.56
-1.97
-0.15
-1.84
-1.53
0.01
-1.00
0.67
-0.56
3-bit
Dense
7.25
-7.11
10.44
10.36
6.44
18.67
16.68
11..2
-7.39
-00.9
9.68
MoF
-0.66
-0.14
-2.04
-1.10
1.02
-2.55
1.11
-2.11
-1.45
-2.91
-1.01
81.78
-74.17
-3.008
-2.44
-9425
-81.50
-80.4
-55.00
-8.33
2-bit
MoE
-6.12
-7.69
-16.78
-11.29
-2.16
-20.14
-16.42
-15.82
-12.34
-17.61
-11.54
K DETAILED BLEU SCORE DIFFERENCES WITH QUANTIZATION APPLIED TO 5.3B MODEL.
Table D shows individual BLEU score changes with various quantization bits for MoF and dense models measured with the internal validation dataset. Table 10 shows the same model's evaluation performance on two WMT public dataset.
17

---

Under review as a conference paper at ICLR 2023
Table 9:
The BLEU score differences in percentage (%) after quantization on different language
pairs. The rows with fp16 show the baseline BLEU scores.
Quantization Bits
Model
de-en
es-en
fr-en
it-en
nl-en
Avg. (xx-English)
Ipl6 (Baseline BLEU)
Dense
40.31
53.09
49.13
44.03
46.23
46.56
MoE
41,49
53.79
50.26
46.97
47.53
48.01
8-bit
Dense
-0.03
-0.08
-0.02
0.01
-0.05
-0.04
(% difference)
MoE
-0.10
-0.06
0.00
-0.02
0.03
-0.03
4-bit
Dense
0.78
0.29
0..3
-0.93
-0.20
-0.37
(% difference)
MoE
0.50
-0.11
-0.10
-0,39
-0.02
-0.22
3-bit
Dense
-6.36
-2.51
-4.24
-5.93
-2.67
-4.34
(% difference)
MoE
-0.2
0.26
-0.26
-1.26
0.29
-0.38
2-bit
Dense
-95.44
-94.42
-95.51
-95.10
-93.31
-94.76
(% difference)
MoE
-4.35
-1.00
-2.64
-7.01
-0.70
-3.14
en-de
en-es
en-fr
en-it
en-nl
Avg. (English-xx)
fp16 (Baseline BLEU)
Dense
38.74
46.44
50.82
40.09
41.69
43.55
MoE
39.90
47.47
52.45
41.25
42.36
44.69
8-bit
Dense
00.04
-0.07
0.02
-0.05
0.09
-0.01
(% difference)
MoE
0.05
-0.01
-0.03
0.00
0.00
4-bit
Dense
0076
-1.11
-0.29
-0.70
-0.26
-0.62
(% difference)
MoE
0.31
-0.90
-0.74
-0.45
-0.68
-0.49
3-bit
Dense
-5.2
-4.79
-3.96
-5.41
-4.54
-4.91
(% difference)
MoE
-0.21
-2.12
-1.41
-0.87
-0.89
-1.10
2-bit
Dense
-97.28
-96.16
-95.52
-96.68
-94.83
-96.09
(% difference)
MoE
-5.24
-6.19
-5.19
-5.30
-4.48
-5.28
Table 10: The BLEU score differences in percentage (%) of 5.3B MoE model after quantization on different language pairs on WMT datasets. The rows with fp16 show the baseline BLEU scores.
Quantization Bits
Model
de-en
fr-en
Avg. Ex-English)
fp16
Dense
50.11
42.98
46.54
(Baseline BLEU)
MoE
52.73
44.04
48.39
8-bit
Dense
0.04
0.11
0.07
(% difference)
MoE
0.09
-0.04
0.03
4-bit
Dense
-0,59
-1.27
-0.91
(% difference)
MoE
-0.47
-0.36
-0.42
3-bit
Dense
-5.75
-6.17
-5.94
(% difference)
MoE
-1.15
-0.90
-1.03
2-bit
Dense
99.888
-95.59
-96.28
(% difference)
MoE
-5.37
-3.68
-4.60
en-de
en-fr
Avg. (English-xx
fp16 (Baseline BLEU)
Dense
50.90
44.47
47.68
MoB
52.90
45.51
49.21
8-bit
Dense
0.00
0.02
0.01
(% difference)
MoE
-0.05
0.23
0.08
4-bit
Dense
0.24
-1.31
-0.48
(% difference)
MoE
-0.93
0.25
-0.39
3-bit
Dense
-5.86
-7.53
-6.64
(% difference)
MoE
-1.41
-0.69
-1.08
2-bit
Dense
997.7
-96.22
-97.05
(% difference)
MoE
-6.34
-6.15
-6.25
18


Reviews:
[{"review_id": "sxFNbvmOwcA", "reviewer": "Reviewer_P5VH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an investigation of Mixture of Experts (MoE) transformer models with aggressive quantization (2-4 bits) of the Feed Forward Networks weights. 2-bit quantization of these layers limits model size increase due to MoE usage to 1.7x of the original model (instead of 8.4x), while retaining some of the higher-precision MoE accuracy gains (+1.88% instead of +2.87%). 2-bit quantization also results in significant sparsity of the FNN layers (~80%).", "summary_of_the_review": "This paper presents an empirical investigation that applies known quantization techniques to transformers models with MoE. Although from this perspective novelty is limited, results are interesting as the authors identify a path towards aggressive quantization of these models and their observations may be leveraged in future works.", "strength_and_weaknesses": "Strengths:\n- paper convincingly demonstrates that aggressive quantization of the expert FNN retains significant accuracy improvement while limiting the increase in model size\n- ablation studies give valuable information on where quantization is best applied (expert FNN)\n- observation of high sparsity resulting from aggressive quantization may also be leveraged by future studies\n\nWeaknesses:\n- primarily an observational study which applies known quantization techniques to known architectures\n- a trade-off remains between model size and accuracy improvement (i.e., the MoE model is still larger than corresponding dense models, especially when compared to the quantized versions)\n- only weights are quantized, not activations. Consequently, inference speedup is necessarily limited\n- several throughput values on Table 2 are missing for both the dense and MoE model. What's the reason?\n- as a result of the authors attempting to emphasize their best results, some statements can be misleading. For example, the conclusions read \"We also show 2-bit quantization could achieve _more_ than 80% sparsity in the expert weights\" (emphasis mine) which refers to the 2-bit pruned results which comes at the expense of losing most MoE gains in accuracy. I would recommend to tune down this and similar statements\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iTAnBLGLgs", "reviewer": "Reviewer_MzdQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple weight-only quantization method to quantize the MoE model. This method combines several common quantization techniques to quantize and reduce the model size. The experiment results show that the method can be used for accelerating the MoE model.", "summary_of_the_review": "This paper proposes a simple weight-only quantization method to quantize the MoE model. The experiment results show that the method can be used for accelerating the MoE model. However, this paper lacks a comparison with other quantization methods for MoE, and there is not any new technique for solving the MoE. Moreover, this paper needs more polishes. To sum up, I do not recommend this paper. ", "strength_and_weaknesses": "Strength: The paper applies the quantization to reduce the model size and accelerate the model inference. The paper conduct several experiments to show the effectiveness of the method.\n\nWeakness: This paper lacks the comparison with other quantization methods for MoE. This paper needs more valuable analysis for applying the quantization into MoE since using quantization to reduce the model size to speedup is a common technique and result.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "77eNlgOFRor", "reviewer": "Reviewer_ZfnQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigated the characteristics of Mixture of Experts (MoE) models when their expert layers are quantized. The authors revealed that the expert layers have more evenly distributed data than dense layers, and thus they are more robust to quantization. Motivated by this observation, the authors applied low-bit quantization to MoE's expert layers and achieved memory savings with slight accuracy degradation.", "summary_of_the_review": "The authors presented the discovery that the distributions of the expert layers of MoE models are less skewed than typical FFN layers, and thus the authors claimed that these expert layers are more friendly to quantization. Although these observations are interesting, it would be nicer to see further innovations in the quantization technique to squeeze more bits out of the MoE models.", "strength_and_weaknesses": "(Strength)\n- One of the first efforts to investigate the quantization impact on MoE models\n\n\n(Weakness)\n- Although the observations are interesting, there is little technical innovation based on them. The proposed quantization schemes follow conventional quantization techniques.\n\n- The authors claimed the gain in throughput, but the experimental settings for measuring hardware speedup are not clear.\n\n- The evaluation of accuracy is quite limited (presented only one translation task).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2UvmYryFeB", "reviewer": "Reviewer_5W77", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the effects and technique of quantizing the mixture of experts (MoE) models originating in the field of NLP. These MoE models are constructed to have gating layers that route inputs through certain matrices, and typically, the chosen gating settings are such that overall FLOPs are the same as of a single (non-MoE) network. As such, the only issue is saving and loading the larger MoE network: according to Table 1, MoE network of 32 experts is requires 9x more storage and has only 0.4x thorouput of non MoE network (despite having same theoretical FLOPs). In this paper, authors empirically study whether quantization is a viable mechanism to reduce the size of MoE networks and present their findings: using 2-bit quantization on expert layers and performing additional finetuning (QAT), it is possible to achieve an MoE network of 32 experts that is only 1.7x times larger than dense baseline (non-compresses MoE is 8.4x of dense network) Their full recipe is available in Section 3.3.", "summary_of_the_review": "Overall, the main contribution of the paper is empirical evaluation of various quantizaiton options/techniques when applied to MoE networks. Although I believe such evaluations would be very helpful to the community, the current state of this paper requires having additional empirical studies to make its contribution/claims stronger.", "strength_and_weaknesses": "After reading the paper I can identify following strengths and weaknesses. Strengths first:\n- The paper is one of the first to give an empirical study of quantization of MoE networks. It would be a good manual/starting point for practitioners in the field. \n\nWeaknessess:\n- Thoroughness: Despite having good results and having investigated several quantization options, one would still have questions of \"what if?\" style. There are many additional experiments and empirical evaluations that are needed to make it a stronger contribution, and to be certain of presented recommendations. For instance here are additional questions: 1) if inference happens in fp16, why to stick with uniform or log-uniform quantization schemes? how about non-inform quantization akin k-means? 2) why not to consider finer grouping for quantization instead of per-tensor and per-channel? 3) why PTQ calibration techniques are not discussed? are all calibrations work the same? 4) what is the tradeoff between # experts vs bit-width of compression? are there certain recommendation? and many other questions of this format\n- The paper would benefit from another proof-reading pass: there are many places where it is hard to understand what was exactly meant.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1cAiM1OcxJ", "reviewer": "Reviewer_X5nA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a quantization technique for MoE models. The authors first study the weight distributions of different layers (MoE layer, Dense layer) and found that the MoE layer has fewer weight outliers, which is why the authors claim that the MoE layers are more suited for quantization. The authors then studied two different quantization methods, i.e. linear quantization and log-scale quantization. The authors then show performance after quantization on different modules, and then propose a quantization recipe. ", "summary_of_the_review": "Given the limited novelty and unclear presentation, I currently give a score of 3. However, I am also open to change the score during the rebuttal time. ", "strength_and_weaknesses": "+ [+] The idea is straightforward enough\n+ [+] The topic is timely\n- [-] The presentation is not clear\n- [-] The novelty is limited\n\nPlease refer to the next section for more details. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject