Mixture of Lookup Experts
Authors: Shibo Jie
Yehui Tang
Kai Han
Yitong Li
Duyu Tang
Zhi-Hong Deng
Yunhe Wang
Venue: ICML 2025 oral (2025)
Keywords: Mixture-of-Experts
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Mixture-of-Experts (MoE) activates only a subset of experts during inference, allowing the model to maintain low inference FLOPs and latency even as the parameter count scales up. However, since MoE dynamically selects the experts, all the experts need to be loaded into VRAM. Their large parameter size still limits deployment, and offloading, which load experts into VRAM only when needed, significantly increase inference latency. To address this, we propose Mixture of Lookup Experts (MoLE), a new MoE architecture that is efficient in both communication and VRAM usage. In MoLE, the experts are Feed-Forward Networks (FFNs) during training, taking the output of the embedding layer as input. Before inference, these experts can be re-parameterized as lookup tables (LUTs) that retrieves expert outputs based on input ids, and offloaded to storage devices. Therefore, we do not need to perform expert computations during inference. Instead, we directly retrieve the expert's computation results based on input ids and load them into VRAM, and thus the resulting communication overhead is negligible. Experiments show that, with the same FLOPs and VRAM usage, MoLE achieves inference speeds comparable to dense models and significantly faster than MoE with experts offloading, while maintaining performance on par with MoE. Code: https://github.com/JieShibo/MoLE.

Mixture of Lookup Experts
Shibo Jie'
Yehui Tang?
Kai Han
Yitong Li
Duyu Tang
Zhi-Hong Deng'
Yunhe Wang" 2
Abstract
Mixture-of- Eepeeetssss (MoE) activates only subset of experts during inference, allowing the model maintain low inference FLOPs and latency even the parameter count scales up. How- ever, since MoE dynamically selects the experts, all the experts need to be loaded into VRAM. Their large parameter size still limits deployment, and offloading, which load experts into VRAM only when needed, significantly increase infer- ence latency. To address this, propose Mixture of Lookup Experts (MoLE), new MoE archi- tecture that is efficient in both communication and VRAM usage. In MoLE, the experts are Feed-Forward Networks (FFNs) during training, taking the output of the embedding layer as in- put. Before inference, these experts can be re- parameterized as lookup tables (LUTs) that re- trieves expert outputs based input ids, and offloaded to storage devices. Therefore, we do not need to perform expert computations dur- ing inference. Instead, we directly retrieve the expert's computation results based on input ids and load them into VRAM, and thus the result- ing communication overhead negligible. Ex- periments show that, with the same FLOPs and VRAM usage, MoLE achieves inference speeds comparable to dense models and significantly faster than MoE with experts offloading, while maintaining performance on par with MoE. Code: https:/ / thub. com/3 Ji eSh bo MMLLE.
42
44
0.0
0.5
1.0
100
200
Average Accuracy
# Parameter in VRAM (B)
Latency (ms/step)
Dense
MoF
MoF (Expert Offlodding)
MoLF (Qurs)
Figure 1. With the same 410M activated parameters, MoF outper- forms the dense model in of performance, but it comes with significant VRAM usage. If experts are offloaded, inference la- tency will increase. Our MoLE maintains competitive performance without increasing the model's VRAM usage or decoding latency.
1. Introduction
Scaling laws indicate that, with sufficient data for training, the performance large language models (LLMs) improves as the model size increases (Kaplan et al., 2020). However, larger LLMs also result slower inference speeds, which can degrade the user experience. For this reason, the archi- tecture of LLMs has increasingly focused Mixture of Experts (MoE) (Jiang et al., 2024; Dai et al., 2024). MoE models use several Feed-Forwart Networks (FFNs) as ex- perts and employ a router to determine which subset of ex- perts needs to be activated, rather than activating the entire model. This allows the model maintain a large number of parameters while keeping the computational cost low.
Although MoE reduces the computational cost, the num- ber of parameters does not decrease. This means that the VRAM requirements during inference remain unaffordable. For example, although the Mixtral-8 x 7B (Jiang et al., 2024) model only has 13B parameters activated a time, its to- parameter count reaches 46B, making it impossible load into single 80GB GPU with FP16. Existing meth- ods (Eliseev & Mazur, 2023; Xue et al., 2024; Shen et al., 2022) reduce VRAM usage by offloading experts to larger storage devices (e.g., CPU RAM, disk, or cloud storage), and loading the selected experts into VRAM at each in- ference step. However, there are two drawbacks to this approach: i) Since the selection of experts dynamically
'State Key Laboratory General Artificial Intelli- gence, School of Intelligence Science and Technology, Peking University "Huawei Noah's Ark Lab *Consumer Business Group, Huawei. Correspondence to: Yunhe Wang <yunhe, waneewanngge huawei.com>>, Zhi- Hong Deng <zh- deng @ww.eppw.uwuu.....eeeeeee Ychui Tang <ychui.tang huawei.com>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mixture of Lookup Experts
determined by the router, we must load different experts into VRAM each inference step. Frequent transfer of large numbers of parameters can significantly increase inference latency, as in Figure| 1. ii Since different samples select different experts within a single step, loading only a subset of experts may not meet the needs of batched generation.
2. Related Work
2.1. Mixture-of-Experts
The concept of MoE was initially introduced by Jacobs et al. (1991); Jordan & Jacobs (1994) and has been widely explored and developed through subsequent research (Col- lohert al., 2002; Rasmussen & Ghahramani, 2001; Shah- baba & Neal, 2009; Eigen et al., 2014; Theis & Bethge, 2015; Deisenroth & Ng, 2015; Aljundi et al., 2017; Shazeer et al., 2017). MoE posits that different parts of the model, Le., the experts, focus on distinct tasks or encapsulate dif- ferent kinds of knowledge. In this paradigm, only the experts relevant to given input are activated, which al- lows the model to scale its capacity while keeping computa- tional costs manageable and making full use of specialized knowledge across many experts. As the scale of LLMs increases, reducing computational overhead has become a key focus. This leads to its application in transformer- LLMs (Lepikhin et al., 2021), making MoE a widely used architecture.
To address the issues mentioned above, we propose Mix- ture of Lookup Experts (MoLE), a new LLM architecture. MoLE has different structures in training and inference. During training, MoLE is similar to MoE, with router and several experts. However, unlike MoE, where experts take intermediate features as input, MoLE's experts are fed with embedding tokens (i.e., the output of the embed- ding layer) instead. Additionally, MoLE allows all experts to be activated simultaneously. After training, MoLE is not directly used for inference but undergoes a series of re-parameeerizatioon. Since the output of the embedding layer is fixed for specific input ids, the inputs to the experts have only a limited number of choices, equal to the model's vocabulary size. Therefore, for each token in the embed- ding layer, we pre-compute the outputs corresponding to all experts, creating lookup tables (LUTs) that replaces the original experts.
Recently, series of industrial-scate large language models have been released, including Mixtral (Jiang et al., 2024) and DeepSeek-No aim to improve the efficiency of MoE (Jin et al., 2025) or increase expert capacity (Yan et al., 2025) by modifying the model architecture.
(Dai et al., 2024). Some works also
During inference, MoLE demonstrates several advantages:
Computation-free experts. The experts are re- parameterized from FFNs into LUTs, eliminating the need for any computation. Each expert only requires a single lookup operation.
2.2. Expert Offloading
Offloading techniques typically transfer a portion of the model parameters to CPU RAM or disk when GPU memory is insufficient. However, current mainstream offloading frameworks, such as Zero-Infinity (Rajbhandari et al., 2021), are designed for dense LLMs and load model parameters layer by layer on-demand. This approach overlooks the sparse activation property of MoE models, leading to the unnecessary loading of inactive experts.
Low VRAM overhead and communication latency. Although the size of the LUT is much larger than the model itself, it can be entirely offloaded to storage devices. During inference, since the output of each expert is the same number of tokens the input, the communication required to load the lookup results into VRAM is negligible, thereby avoiding increased infer- ence latency.
Building on this, some studies have proposed expert offload- ing, a form of parameter offloading specifically designed for the sparse activation characteristic of MoE models (Eliseev & Mazur, 2023; Xue et al., 2024). These methods stores non-expert weights and portion of the expert cache in VRAM, while the remaining experts are offloaded to CPU RAM or disk and loaded on-demand.
Batch-generation friendly. Traditional expert offload- ing methods introduce additional VRAM usage and communication latency during batched generation be- cause different samples in a batch may select different experts. MoLE only transfers the pre-computed ex- pert outputs, making its communication overhead still negligible even during batched generation.
Despite being effective, existing expert offloading tech- niques still suffer from high latency. Subsequent research includes optimizing prefetching techniques and cache re- placement strategies to accelerate inference speed (Shen et al., 2022), designing MoE architectures that are more friendly to prefetching (Hwang et al., 2024), or employing other model compression techniques to reduce prefetching latency (Yi et al., 2023).
Through extensive experiments, we validated the effective- ness of MoLE scales of 160M, 410M, and IB parame- ters. As in Figure 1, with equivalent computational cost and VRAM usage, MoLE significantly outperforms dense models while maintaining the same inference speed. Com- pared to MoE with expert offloading, MoLE achieves better performance and with significantly faster inference speed.
2

---

Mixture of Lookup Experts
Shared FFN
FFNI
FFN2
FFN3
FFN4
Shared FFN
FFNI
FFN2
FFN3
FEN4
Shared FFN
Off-Device
RRuueer
Router
Router
[Cptional)
Attention
Attention
Attention
Embedding
Embedding
Embedding
Training
Inference
MoE
MoLE (Ours)
Figure 2. Illustration of MoLE. During training, MoLE differs from MoE in two key structural aspects: i) The routed experts in MoLE take embodding tokens input. ii) All experts in MoLE are activated. During inferenee, the routed experts in MoLE re-parameterized as zero-computation, offloaded LUTs. For simplicity, normalization layers and residual connections of attention layers are omitted.
3. Mixture of Lookup Experts
MoE layers, with experts per layer, but only 2 experts are activated per token. Although only 13B parameters are acti- vated per token, the total parameter count reaches up to 46B, requiring least 92GB of VRAM for FP16 deployment.
3.1. Preliminary
First, briefly introduce the structure of MoE and the challenges it faces during inference.
If temporary loading is chosen, each expert is 176M in size, and loading the necessary experts for single decoding step would require up to 1..3 of parameter transfer. If offloading to CPU VRAM is selected, using GPU with PCIe 4.0x 16 would still incur a transfer latency of 0.7s per step. Offloading to disk, on the other hand, results in an unacceptable transfer latency of over 10s per step. More importantly, since the selection of experts is dynamically determined by the router, the experts chosen for different samples are highly likely to differ when batch size > 1. This requires loading all the selected experts (may be all (2) experts when batch size is large) into VRAM, which not only increases VRAM usage but also further exacerbates communication latency.
For MoE, each expert is typically FFN module. As illus- trated in Figure 2 (left), MoE layer contains N routed experts, represented as [ and a linear router, denoted as { J. Some models may also introduce a shared expert FFN shared that is activated in all cases. Given input token h E Rd, the output token h' Rdd of the MoE layer is computed as
= ArgTopk((h (1h,r)))-)
{gibee SoftMax((h Felh--yyhe rybee)
h' = dd==(O, ((GFFN((()) + FFN FFNshared(h) (h)
(3)
JEG
The reason experts need be loaded into VRAM that they participate in the computation, which relies GPU, In other words, if the experts do not require computation, we do not need to load them, thereby avoiding significant communication overhead. To address this, we introduce MoLE, a new MoE architecture whose experts can be re- parameterized as computation free LUTs in inference.
where G' denotes the indexes of the activated experts, and denote the gate value for the i-th expert.
The computational efficiency MoE lies in the fact that, in Eq. (3), only à routed experts are involved the computa- tion. However, we cannot determine which experts need to participate until Eq. (1) is completed. This means that either need store all the experts in VRAM or temporarily load the required & experts into VRAM after Eq. (1) is computed.
3.2. Training Phase
As illustrated in Figure 2, during training, MoLE and MoE have similar structures, including N routed experts
However, both of these solutions present deployment chal- lenges. Taking Mixural-8x7B as an example, it has
3

---

Mixture of Lookup Experts
Table 1. Complexities of different architectures. We report the statistics a single FFN or MoE layer.
Models
FLOPs
# Param in VRAM
# Param Offloaded
# Param Loaded per Token
Dense
4dD; 4d(kD, + D.) 4d(D:Dr 1 D.)
2dD. 2d(ND, + D.) 2d(kD, | D.) 2dDD
0 0
0
MoE
0
MoE Expert Offloading
2dN D, dN
2dkD,, (worst case)
MoLE LUT Offloading
4dD,
dN
[FFN; و and a linear router {r; {r j=1 Specifically, MoLE also includes a shared expert FFN shared. which al- ways activated for any input and does not receive weighting from the router.
which allows us to obtain ej for all i. The items of the LUT at the I-th layer can be represented as
LUT, = ((0))
Since the experts will be transformed into LUTss after train- ing, MoLE differs from MoE in the following ways. First, LUT is computation-fiet free, eliminating the need for sparse activation to reduce computational cost. Therefore, MoLE activates all experts, rather than just the top-k experts. The computation for the router is as
where | denotes the size of the vocabulary.
After re-parameterization, the LUT offloaded to the stor- age device, and the computation of the MoLE layer can be represented as
h' n'= = E-E0 (0,0,) + FFN (h)r +
(8)
{ SoftMax((h ri)
(4)
Second, since the LUT is essentially mapping between finite set of input-output pairs, the key to re-parameterizing the experts into LUTss is ensuring that they only have a lim- ited number of possible inputs. To this end, MoLE uses the output of the embedding layer, ie., the embedding tokens, as the input to the experts. After training, the embedding layer is only related to the input ids, which means that the inputs to the experts are limited to finite set. The computation for the layer is as
The inputs of the LUT in MoLE are the input ids, meaning that no context information is included. This a trade-off to ensure that the experts can be re-parameterized, but it does not imply that the expert layers do not contribute to context- related knowledge. Firstly, the router and shared experts still take intermediate features input, which means they can access contextual information. Secondly, the output of the expert layer is part of the input to subsequent attention layers, allowing the experts to influence the behavior of later attentions. This enables the experts to adjust how the model processes the same words different contexts, thereby still enhancing the model's capacity.
N h'= = Σ (c)) + FFN sharea(h) + h.
(5)
3.4. Complexity Analysis
in which Embedding(i) E Rd is the embedding token, and deonotes the input id.
Consider an MoE layer with MI.P-based FFNs as experts. Let the hidden layer dimension of the routed experts be D,,, and the hidden layer dimension of the shared experts be D,- When single token is used as input, the FLOPs for this MoF layer can be computed as
All experts are activated and receive gradients during train- ing. Therefore, we do not need to add any auxiliary losses to prevent collapse or maintain training stability. MoLE trained solely using the cross-entropy loss of language modeling just like dense model.
FLOP 4d(kD, D.)
(9)
in which the router and normalization is neglected.
3.3. Inference Phase
To save VRAM, we assume that all routed experts of- floaded, and then the offloaded parameter count is
After training, MoLE can be directly used for inference like other LLMs. However, to further reduce VRAM overhead, can re-parameterize the experts. For each possible input id i, we pre-compute the outputs of expert FFN, as
OffParamMor 2dN
(10)
In the worst case, during inference, we need to load the experts that the current token is routed into VRAM. The number of per-step loaded parameter is
0ÿ = FFN, (Emmbedding(i) E Idd
(6)
In practice, we only need to perform a single forward pass with the embedding layer's weights as the input to FFN,,
I LoadParammor 2dk Dr
(11)

---

Mixture of Lookup Experts
Table Model architectures. We ensure fair comparison by keeping number of activated parameters same inference the dense, MoE, and MoLE models.
Activated Param in Inference
Attention Heads
Parameters in Training
Models L
d
D>
D,
N
k
Dense
12
768
3072
12
0.16B
MoE-10E
12
768
1536
10
2
12
0.39B
160M
MoLE-4E
12
768
3072
3072
4
0.39B
MoE-34E
12
768
1536
34
2
12
1.07B
MoLF--16
12
768
3072
3072
16
16
12
1.07B
Dense
24
1024
4096
16
0,41B
MoF- IOE
24
1024
2048
10
2
1.21B
410M
MoLE-4E
24
1024
4096
4096
4
4
16
1.21B
MoE-34E
24
1024
2048
34
2
16
3.63B
MoLE-I6E
24
1024
4096
4096
16
16
3.63B
Dense
16
2048
8192
1.01B
IB
MoF-IOE
16
2048
4096
10
2
3.16B
MoLE-4E
16
2048
8192
8192
4
4
8
3.16B
For MoLE, since the experts are transformed into computation-frs free LUTs, its FLOPs can be computed as
no shared experts and activating the top-2 routed experts, since Muennighoff et al. (2024) suggest that shared experts lead performance degradation. To ensure the number of activation parameter is the same as that of the dense model, the hidden dimension of the MoE FFNs is set to half the hidden dimension of the dense model's FFNs.
FLOPSMBLE = 4dDs
(12)
The number of parameters contained in the offload L.T
OfiParamMoLE = d
(13)
For the MoLE model, since routed experts have no computa- tion during inference, we use FFNs identical to those the dense model's FFNs as the shared experts keep the same FLOPs the dense model. For the routed experts, since their hidden dimension does not affect the model architec- ture during inference, we set it to be the same as the shared experts for simplicity. We implemente MoE with both 10 and 34 experts and MoLE with both 4 and 16 experts for comparison.
In each inference step, since we only need to load all the vj from Eq. (8) into VRAM, the amount of parameters loaded only
LoadParammaLL dN
(14)
We summarize all these comparisons in Table 1. Since | is typically on the order of tens of thousands, for example,
|V] = 32k for Mixtral (Jiang et al., 2024) and |V] = 50k Pythia (Biderman et al., 2023), and D, varies from thousands to tens of thousands depending on the model size, the number of offloaded parameters in MoE and MoLE will not differ by order of magnitude. However, the number of parameters loaded per token in MoLE will be only fraction - often hundreds or even thousands of times smaller - compared to the number of parameters loaded per token MoE.
Data & Tokenizer. We train all models on 100B-token subset of the deduped Pile dataset (Gao et al., 2021), us- ing the GPT-NeoX tokenizer employed by Pythia, with vocabulary size of 50k.
Sypee-Parammttrss We follow the learning rate settings used by Pythia, specifically 6.0 x 10 for the models with 160M activated parameter, and 3.0 x 10-4 for the models with 410M and IB activated parameter. For the MoE model, the coefficients for the 2-loss and load balance loss are set to 0.001 and 0.01, respectively, suggested by Muennighoff et al. (2024).
4. Experiments
4.1. Experimental Setup
Benchmarks. We use the Im-evallatioon-hamess package for evaluation. The benchmarks used include ARC-C (Clark et al., 2018), ARC-E (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), RACE (Lai al., 2017), SIQA (Sap al.,
Model Architectures. As shown in Table 2, we imple- ment models with activation parameter counts of 160M, 410M, and IB. For the dense model, we basically follow the Pythia (Biderman et al., 2023) setup. For the MoE model, we adopt configuration similar to Mixtral, with
5

---

Mixture of Lookup Experts
Table Comparison of complexities different architectures. MoE offloads experts and MoLE offloods the LUT, ensuring that both models maintain the same VRAM usage as the dense model. "# Param Loaded per Token" considers the worst-case scenario. when the experts of MoE activated the current step have overlap with those the last step.
# Param
# Param Offloaded
Models
Loaded
ARC-C ARC-E BoolQ
HellaSwag PIQA RACE SIQA LAMBADA
AVG
per Token
160M Activated
VRAM Parumeters
Dense
OB
0M
20.3
45.9
57.1
29.7
64.0
29.4
37.8
26.2
38.8
MoE-10E
0.3B
57M
21.7
49.5
51.6
32.0
66.8
30.6
39.1
31.0
40.3
MoLE-4E
1.8B
0.037M
21.9
48.5
60.7
31.2
65.1
29.4
38.4
31.1
40.8
MoE-34E
1.0B
57M
20.5
50.0
57.5
34.5
67.3
28.6
39.9
36.4
41.8
MoLE-16E
7.4B
0.15M
22.4
48.6
60.3
32.7
68.3
30.9
38.6
33.3
41.9
410M Activated & In-VRAM Parameters
Dense
OB
0M
21.8
50.8
56.8
33.8
66.5
29.6
39.2
36,2
41.8
MoE-10E
1.0B
201M
24.1
53.5
54.5
37.1
69.0
30.8
40.8
41.5
43.9
MoLE-4E
4.9B
0.098M
22.0
54.8
61.1
35.9
69.6
30.9
40.1
42.2
44.6
MoF-34E
3.4B
25.0
57.0
59.7
39.9
71.5
32.3
40.4
47.1
46.6
MoLE-I6E 19.7B
0.39M
23.6
57.0
60.9
37.6
70.8
32.0
40.2
43.5
45.7
IB Activated & In- VRAM Parameters
Dense
OB
0M
24.1
56.9
52.8
37.6
69.5
31.6
39.1
43.1
44.3
MoE-10E
2.7B
537M
25.9
53.8
40.7
72.0
33.6
41.3
46.6
MoLE-4E
6.6B
0.26M
25.5
61.7
39.8
71.7
32.1
40.9
48.3
47.4
Table Ablation study on training loss. The base model MoI.E-16E with 160M activated parameters, After adding the auxiliary loss used by MoE, the performance decreased.
Training Loss
ARC-C
ARC-E
BoolQ
HellaSwag
PIQA
RACE
SIQA
LAMBADA
AVG
LM loss only (ours)
22.4
48.6
60.3
32.7
68.3
30.9
38.6
33.3
41.9
LM loss load balance loss
21.2
50.8
60.2
32.4
66.5
31.5
37.7
33.1
41.7
LM loss load balance loss 2-loss
20.7
50.5
51.7
32.5
67.7
30.8
38.2
32.5
40.6
2019), and LAMBADA (Paperno et al., 2016). For all these benchmarks, we report the zero-shot accuracy.
parison of five pairs of MoLE and MoE models with the same number of training parameters, MoL.E outperforms the five comparisons terms of average
MoE in four accuracy. Notably, for MoLE- 16E with 160M, 410M, and IB activated parameters, the number of per- per-token loaded parameter is only about 1/1500, 1/2000, and 1/2000 of that of MoE, respectively. This demonstrates that MoLE can maintain outstanding performance while significantly reduc- ing communication overhead, making feasible offload to lower-tier storage devices.
Offloading Setting. To measure the deployment efficiency of different models in apply offloading strategies to both MoE and MoL.E EE to ensure their VRAM usage is consistent with that of dense model with the same number of activated parameters. For the MoE model, we adopt an expert offloading strategy, where only the parameters of the activated experts and all non-expert parameters are stored in VRAM. During each inference step, if the required activated expert is not in VRAM, it is loaded from the storage device into VRAM, replacing the previously loaded expert. For MoLE, we offload the LUT, while keeping the other parameters stored in VRAM.
environments,
We note that the size of the LUTs in MoLE is 2.4 to 7.4 times larger than the size of the offloaded experts in MoE, However, since these parameters are offloaded to large, scal- able storage devices, we believe the storage overhead of the LUTs remains within an acceptable range. Specifically, as the model size increases, the proportion of the LUTs also decreases accordingly. On models with IB activated parameters, the size of the LUT's in MoLE-4E becomes
4.2. Main Results
As shown in Table 3, both MoE and MoLE significantly improve performance over the dense baseline. In the com-
6

---

Mixture of Lookup Experts
Table Ablation study the hidden dimension routed experts. The model MoLE- with 160M activated parameters.
# Param Loaded Offloaded
# Param in Training
# Param
D,
ARC-C ARC-E BoolQ HellaSwag PIQA
RACE SIQA LAMBADA AVG
per Token
768
0.4B
7.4B
0.15M
20.6
48.1
58.7
31.4
66.9
30.3
38.7
31.9
40.8
3072
1.1B
7.4B
0.15M
22.4
48.6
60.3
32.7
68.3
30.9
38.6
33.3
41.9
12288
3.8B
7.4B
0.15M
21.8
52.4
58.1
33.1
67.6
29.2
38.7
32.3
41.7
Table Ablation study on the number routed experts. The model is MoLE 160M activated parameters.
Param Loaded ARC-C ARC-E
# Param in Training
Param Offloaded
N
BoolQ
HellaSwag
PIQA
RACE
SIQA LAMBADA
AVG
Token
2
0.3B
0.9B
0.02M
19.5
48.2
57.7
30.2
64.7
29.5
38.1
29.4
39.7
4
0.4B
1.8B
0.04M
21.9
48.5
60.7
31.2
65.1
29.4
38.4
31.1
40.8
8
0.6B
3.7B
0.07M
19.4
50.5
60.2
32.0
66.4
29.9
37.4
32.6
41.1
16
1.1B
7,4B
0.15M
22.4
48.6
60.3
32.7
68.3
30.9
38.6
33.3
41,9
32
2.0B
14.7B
0.29M
21.8
53.1
59.0
33.5
68.4
30.7
38.8
33.0
42.3
comparable the size of the experts MoE.
Architecture designs. To ensure that routed experts can be re-parameterized, we change the input the routed experts from intermediate features to embedding tokens. Intuitively, this modification means that the experts only receive the raw word features and no context-related information, which is likely to lead to decrease model performance. But on the other hand, since the re- re-parameterized experts not require computation during the inference phase, we can activate all the experts while keeping the model's inference FLOPs unchanged. This helps compensate for the performance loss mentioned earlier. To conduct ablation study on this, we trained the following model variants, evolving from MoE-106 10E to MoLE-4E of 160M activated parameters:
4.3. Ablation Experiments
Training loss. Unlike MoE, MoLE is fully differentiable model, so during training, we do not encounter issues like router collapse instability. Therefore,
do not any additional auxiliary losses. To illustrate this, attempt to add the load balance loss and 2-loss from MoE. As shown in Table 4, after adding these losses, the model's ' performance decline. This because the additional losses caused the model's optimization objectives to become misaligned with the inference requirements, leading to negative effects.
Number and size of experts. We experiment with varying the size and number of experts. As shown in Table 5, when the hidden dimension of the experts increases from to 4d, the model's performance improves. However, further increasing the dimension to 16d leads to performance sat- uration. This because increasing the size of the experts does affect the re-parameterized models or the size of the LUTSs during inference. It indicates that the knowledge embedded in LUT's with constant size reaches saturation as the expert size increases, meaning that there is no "free lunch" - further increases in expert size do not lead to additional performance gains.
Full activation. All experts of MoE-10E are activated, which causes the number of activated parameters increase from 0.16B to 0.39B. Meanwhile, all auxiliary losses are discarded.
Reconfiguration. Based on the above model, modify the experts by changing from 10 routed experts to I shared expert and 4 routed experts. Additionally, the hidden dimension of the experts is increased to twice its original size. The total number of parameters remains unchanged.
Unlike the increase in size, the increase in the number of experts results in continuous performance improvement, demonstrating certain level of scalability. At the same time, the size of the LUTs and the amount of parameters transferred will also increase proportionally.
Embedding as inputs. Based on the above model, we change the input to the routed experts to embedding tokens.
Re Re-parameterization Based on the above model, we reparameterize the routed experts as LUTSs. Then the

---

Mixture of Lookup Experts
Table Ablation study on the designs structure.
# Param Activated in Inference
# Param in Training
Model
ARC-C ARC-E BoolQ HellaSwag PIQA RACE SIQA LAMBADA AVG
MoE-106 10E
0.39B
0.16B
49.5
51.6
32.0
66.8
30.6
39.1
31.0
40.3
Full activation
0.39B
0.39B
21.6
50.2
58.6
33.3
66.8
30.5
39.6
33.5
41.8
+ Reconfiguration
0.39B
0.398
21.5
48.7
58.0
32.7
67.5
31.1
38.5
33.7
41.5
+ Embedding as inputs
0.39B
0,39B
21.9
48.5
60.7
31.2
65.1
29.4
38.4
31.1
40.8
+ Re-param. MoLE-4E
0,39B
0.16B
21.9
48.5
60.7
31.2
65.1
29.4
38.4
31.1
40.8
Table 8. Post-training quantization for LUTs. The base model is MoLE-4E with 160M activated parameters.
Size Param
LUT Precision
Size of Param
Loaded per Second
ARC-C ARC-E BoolQ HellaSwag PIQA
RACE SIQA LAMBADA
AVG
Offloaded
FP16
3,SGB
72KB
21.9
48.5
60.7
31.2
65.1
29.4
38.4
31.1
40.8
NF4
0,9GB
18KB
21.5
48.5
61.7
31.3
64.7
29.6
38.6
30.9
40.9
NF3
0.7GB
14KB
22.3
48.1
59.8
31.0
65.3
28.9
38.5
30.1
40.5
number of activated parameters during inference re- turns O. 16.... This model is referred to MoLE-4E.
Latency (ms/step)
Dense
MoE-10E
MoLE-4E (Ours)
As shown in the table, using embedding tokens as the input to the routed experts only results in a 0.7 performance drop,
MoE-34E
MoLE-16E (Ours)
but brings significant benefits of enabling the experts be p--arammterized, allowing fully activated MoE yields 1.5 performance gain com- pared top-2 MoE, which leads to an overall performance improvement for MoLE over MoE.
200
activate all experts. A
100
32
Batch Size
4.4. Efficiency
Figure 3. Decoding latency. use experts offloading for MoE, The light-colored portion of the bars represents the delay caused by loading.
We measure the per-step decoding latency of models with 410M activated parameters on NVIDIA V100 GPU using Huggingface's
s package. Since the spe- cific speed of parameter loading is largely influenced by the underlying implementation, we estimate the latency of loading parameters based on the maximum PCIe bandwidth of the V100, which is 16GB/s. For the MoE model, when the batch size is 1, the experts activated in the previous de- coding step are retained in VRAM. When the batch size is greater than 1, random two of the activated experts from the previous decoding step are retained VRAM for each layer. the experts activated in the current step overlap with those VRAM, they will be reloaded. Under this setup, the average number of experts loaded per step for batch sizes of 1, 8, and 32 are 1.6, 6.7, and 8.0 for M0E-10E, or 1.9, 12.3, and 27.4 for MoE-34E, respectively. The input length is fixed to 512.
higher latency than the dense model. As the batch size increases, the number of experts being loaded also increases, further adding to the latency of MoE, but the latency of MoLE has almost no increase.
4.5. Reducing the Size of LUTSs
Although MoLE significantly reduces data transfer in of- floading scenarios compared MoE, it has larger storage footprint. While storage space may be as constrained VRAM, reducing the size of the LUTs ate deployment burdens. To address this, we conduct simple experiment to compress the LUTs. We apply post- training quantization to the FP16 LUTs,, quantizing them to NF4 and NF3 (Dettmers et al., 2023) data types. The token-wise block sizes for quantization are 768 and 128.
still allevi-
As shown in Figure 3, the latency of MoLE is comparable that of the dense model, while MoE exhibits significantly
8

---

Mixture of Lookup Experts
respectively. As shown in Table 8, the model's performance suffers minimal loss, while the storage burden and size of transferred data are reduced to 25.3% and 19.5% of the original size, respectively. This indicates that the LUTs still contains significant redundancy and has the potential for further compression. We leave this as future work.
Collobert, R., Bengio, S., and Bengio, Y. A parallel mixture of svms for very large scale problems. Neural Compud., 14(5):1105-1114, 2002.
Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-expprts language models. In Proceedings of ACL, 2024.
5. Conclusion
In this paper, we address the issues of high memory con- sumption and loading latency in MoE by proposing MoLE, a novel language model architecture. MoLE restricts the input to experts to a limited set (embedding tokens), allow- ing the experts to be re-parameterized into LUTs before inference, thus avoiding the need to load expert param- eters. MoLE demonstrates competitive results down- stream tasks, while significantly outperforming MoE with expert offloading in terms of inference speed. This work provides new direction for designing ddge-friendly lan- guage models. Future research could explore more diverse discrete spaces and expert architectures.
Deisenroth, M. P. and Ng. J. W. Distributed gaussian pro- cesses. In Proceedings of ICML, 2015.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer,
Qlora: Efficient finetuning of quantized Ilms. In Proceedings of NeurlPS, 2023.
Eigen, D., Ranzato, M., and Sutskever, I. Learning factored representations in deep mixture of experts. In Workshop of ICLR, 2014.
Eliseev, A. and Mazur, D. Fast inference of mixture-of- experts language models with offloading. arXiv preprint, arXiv:2312.17238, 2023.
Impact Statement
This paper advances LLMs, which have potential societal impacts, including concerns around bias, misinformation, and accessibility. Continued ethical oversight and interdisci- plinary collaboration are essential as the field evolves.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint, 100:02, 2021.
References
Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T. and Yang. M. Pre-gated moe: An algorithm-system co- design for fast and scalable mixture-of-exppe inference. In Proceedings of ISCA, 2024.
Aljundi, R., Chakravarty, P., and Tuytelaars, T. Expert gate: Lifelong learning with a network of experts. In Proceedings of CVPR, 2017.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brieen, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of ICMI., 2023.
Jacobs, R. A., Jordan, M. I., Nowlan, J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Comput., 3 (1):79-87, 1991.
Jiang, A. Q.. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplor, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.. Stock, P., Sub- ramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral experts. arXiv preprint, arXiv:2401.04088. 2024.
Bisk, Y., Zellers, R., Bras, R. L.,, Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In Proceedings of AAAI, 2020.
Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of NAACI. HLT, 2019.
Jin, P., Zhu, B., Yuan, L., and Yan, S. Moe++: Accelerat- ing mixture-of-experts methods with zero-computation experts. In Proceedings of ICLR, 2025.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the A12 reasoning challenge. arXiv preprint, arXiv:180s 0188803..555 2018.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the EM algorithm. Neural Comput., 6(2): 181-214, 1994.
9

---

Mixture of Lookup Experts
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint, AXiv:2001.0836// 2020.
Xue, L., Fu, Y., Lu, Z., Mai, L., and Marina, M. K. Moe- infinity: Activation-awan expert offloading for efficient moe serving, arXiv preprint, arXiv:2401.14361 2024.
Yan, S., Bin, X., Zhang, S., Wang, Y., and Lin, Te- moe: Augmenting mixture of experts with ternary expert choice. In Proceedings of ICLR, 2025.
Lai, G., Xie, Q. Liu, H., Yang, Y., and Hovy, E. H. RACE: large-scale reading comprehension dataset from exami- nations. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of EMNLP, 2017.
Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., and Xu, M. Edgemoe: Fast on-device inference of moe-based large language models. arXiv preprint, arXiv:2308. 143522, 2023.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. In Proceedings of ICLR, 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Hellaswag: Can machine really finish your sentence? In Proceedings of ACL. 2019.
Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., Gu, Y., Arora, S., Bhagia, A., Schwenk, D., Wadden, D., Wettig, A., Hui, B., Dettmers, T., Kiela, D., Farhadi, A., Smith, N. A., Koh, P. W., Singh, A., and Hajishirzi, H. Olmoe: Open mfxpure-of-experts language models. arXiv preprint, arXiv:2409.02066 2024.
Paperno, D., Kruszewski, G., Lazaridou, Pham, Q. N., Bemardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of ACL, 2016.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. Zero-infinity; breaking the GPU memory wall for extreme scale deep learning. In de Supinski, B. R., Hall, M. W., and Gamblin, T. (eds.), Proceedings of SC, 2021.
Rasmussen, C. E. and Ghahramani, Z. Infinite mixtures of gaussian process experts. In Proceedings of NIPS, 2001.
Sap, M., Rashkin, H., Chen, D., Bras, R. L., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. In Proceedings of FMNN.P-IJCNIP,, 2019.
Shahbaba, B. and Neal, R. M. Nonlinear models using dirichlet process mixtures. J. Mach. Learn. Res., 10: 1829-1850, 2009.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. F.,, and Dean, J. Outrageously large neural networks: The sparsely-gatee mixture-of-experts layer. In Proceedings of ICLR. 2017.
Shen, L., Wu, Z., Gong, W., Hao, H., Bai, Y., Wu, H., Wu, X., Xiong, H., Yu, D., and Ma. Y. Sc-moe: A scalable and efficient mixture-of cocture-of--xpertt. distributed training and infer- ence system. arXiv preprint, arXiv:2205.1 10034, 2022.
Theis, L. and Bethge, M. Generative image modeling using spatial Istms. In Proceedings of NIPS, 2015.
10

---

Shared Expert' Mixture of Lookup Experts
Appendix
A. Pseudocode
A.I. Training Phase
class MoleCecoderlayer (nn Module)
def _init_ config):
super(). ini.__nntt111
sclf.sclf_attn = Attention
self self.shharddeexprtt = MLP (config)
self.router in (c.inearrcoofiighiiiii (config.hidden size, config.num nun__perss_ bias=False)
self.routed_exxertt nn momouelelist [MLP (config) for
config.nu_eexerraae
self.innut_layerrom
(esseermcoonigg.hidde sizze)
selt.post_attention__ayaee layerroom = RKSNorm ( aidden__dddd__iiii
selt.expert__ayernaaa (MSNerm.connfi.... hww../eooong..iiddnnss.....
def (self, hidden_states, embsdding_states) "'Attention''
residual hidden_states
hidden_staten self.input, Lay.eoaaryyeoahheeeea
layeemm (hidden states)
hidden_ staatsssttese_ self.self_atee set.selfftttteeeeeeeeeeeeeeaeeea (hidden (hideee ) states)
hidden_states = residual hidden_states
residual bidden_states midden_stater
(h (hidden_states, shared_cutput. - sel self.shared_expert (hideer_ttidddddttttat
"Rouee Expert'
router router_value value nn. wwn.fuutiinal..ooooooo.... softmax (self. roaeer (hidden (hidden_sttttee))) dim=-1)
embeddinggsteeeeeeeedd states = self.expert e (em states)
routed_output = torch. stack ( ([epeer (embedding_statee) for expert in self. rottferootedeepprtt] dim-2)
routed_cutput (routed_output
router_value unsqueeze (-1)) sun (dim=2)
hidden_states residual shared_outpit + routed_outpe
return hidden_states
A.2. Inference Phase
class MoleDecoderlayer (nn. Module) i
def _nnit_ config):
(norrt)
self.self_attn (eeenttionccnnfi
self.shared_expert MiP (config)
self.router nn. ln
config.num_expertee bias=Falae)
self.lut lookup'rable
configgnuu_experts config.hiddee sieei_
self.input layernorm Lnttiinpt___ayyrmommm self.post_attentine latrttioon_ayeeerrrr layernorm RMSNorm (ww.fccoooioii... hidden size)
(config size)
def (self, hidden_states, input_ids)
Lookkpp
lookup_results self. (input_ids) to (hidden non_blocking-Trus)
"'Attention""
residual hidden_se saaden_states
hidden_states = self. Loptt_ hidden self.self_ate wtttfeatttttteeeeeeeeeeeeeaaaaat (hidden stattessssstts))))
hiaden_states residual iidden_states
rrrShared Ex
residual hidden_states

---

Mixture of Lookup Experts
hidden_states eel. (hidden_states) shared_cutput exre (hideensstttdd
""Routed Expert'
router_value nn. functional. softmax
(hiddenn___ sttadddessssts dim=-1)
lookup_results lookup_results
config. _eariiggnueeeeetts..re_e
experts, config. hww.ffig..iidnddiieee
routed_output rLookup_reeults router_value unsquezze (-1)) ) sum (dim=2)
hidden_states = residual + output + routed_output
return hidden_states
B. Hyper-parameters
Configuration Key
Value
attention-dropout
drype
bf16
global-batch-size gradient-clipping
1024
1.0
hidden-dropout
Ir-decay-style
cosine
nux-position-embeddinn min-Ir
2048
0.1 optimizer.paamm.ll
no-weight-tying norm
True
RMSNorm
ceiimizer.pramsshette hetas hetas
[0.9, 0.95]
gpaimizer.prams..pss
1e-08
optimizer.type
Adam
pos-emb
rotary
rotary-pet
0.25
seq-length
2048
train-iters
50000
warmup
0.01
weight-decay
0.01
12


Reviews:
[{"review_id": "pMg88tAnfg", "reviewer": "Reviewer_iLJn", "summary": "The paper proposes Mixture of Lookup Experts (MoLoE), a new variation of Mixture-of-Experts (MoE) architectures that significantly reduces GPU inference latency for batched generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address specific concerns and questions below.\n\n\n> Q1. The comparison between MoE and MoLoE is inconsistent.\n\nThe different settings for MoE and MoLoE are chosen in order to prioritize aligning both **total parameter count in training** and **activated parameter count in inference** for a fair comparison.\n\n+ MoLoE adopts shared experts because, unlike MoE, MoLoE reparameterizes all routed experts into LUTs during inference. If shared experts were not used, MoLoE would not include any FFN layers during inference, which would make it difficult to align its activated parameter count (VRAM usage) with that of dense and MoE models.\n+ The reason MoE does not adopt shared experts is that, in our experiments, MoE without shared experts performs better, which has also been thoroughly validated in [1]. We provide our experimental results below (No.1 vs No.2).\n+ In inference stage, MoLoE only has 1 shared expert, while MoE activates 2 experts. Therefore, to make a fair comparison, we set the size of each expert in MoE to half of that in MoLoE (to align activated parameter count in inference), but double the number of experts (to align total parameter count in training). As a result, the total number of experts cannot be aligned (No.2 vs No.3). But as a reference, we have still provided the results below for MoLoE and MoE both using 1 shared expert and 33 routed experts (No.2 vs No.4).\n\n|No.|Model|#total param in training|#activated param in inference|ARC-C|ARC-E|BoolQ|HellaSwag|PIQA|RACE|SIQA|LAMBADA|average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|1|MoE (0 shared, top-2 from 34 routed)|1.1B|160M|20.5|50.0|57.5|34.5|67.3|28.6|39.9|36.4|41.8|\n|2|MoE (1 shared, top-1 from 33 routed)|1.1B|160M|20.6|48.4|57.3|31.5|65.2|30.5|37.9|34.7|40.8|\n|3|MoLoE (1 shared, 16 routed)|1.1B|160M|22.4|48.6|60.3|32.7|68.3|30.9|38.6|33.3|41.9|\n|4|MoLoE (1 shared, 33 routed)|2.1B|160M|22.0|52.4|59.3|33.5|68.3|30.7|38.8|33.1|42.2|\n\n[1] OLMoE: Open Mixture-of-Experts Language Models. ICLR 2025\n\n> Q2. Results under complex language modeling tasks like WikiText or Winogrande.\n\nWe reported the results for a 1B activated parameter model on WikiText and Winogrande. MoLoE outperforms MoE on Winogrande, but performs slightly worse on WikiText.\n\n|Model|#Param Loaded per Token|WikiText (ppl↓)|Winogrande (acc↑)|\n|-|-|-|-|\n|Dense|0|17.7|54.0|\n|MoE-10E|537M|15.6|55.3|\n|MoLoE-4E|0.26M|16.2|56.5|\n\nAlthough the routed experts in MoLoE do not directly receive context information, they can still indirectly influence the model's handling of context by altering the input to subsequent attention layers. Additionally, the retained shared expert operates similarly to MoE and dense models, taking the output from the previous layer as its input, and thus can also handle context-related information.\n\n> Q3. Different tokenizer settings.\n\nIn addition to Pythia's tokenizer with a vocabulary size of 50k that we used, we also tried the 32k vocabulary tokenizer from Mixtral. The results are shown below.\n\n|Model|#offloaded param|#activated param|#Param Loaded per Token|8-task average|\n|-|-|-|-|-|\n|MoE-10E (Pythia tokenizer)|0.3B|160M|57M|40.3|\n|MoLoE-4E (Pythia tokenizer)|1.8B|160M|0.037M|40.8|\n|MoE-10E (Mixtral tokenizer)|0.3B|134M|57M|39.8|\n|MoLoE-4E (Mixtral tokenizer)|1.2B|134M|0.037M|40.2|\n\nThe primary impact of the tokenizer on MoLoE is that its vocabulary size is proportional to the size of the LUTs. However, since different tokenizers also affect the performance of all kinds of models, it is difficult to isolate the tokenizer's specific effect on MoLoE's performance. Nonetheless, at least from the experimental results, we can observe that MoLoE is effective under different tokenizers.\n\n> Q4. Compared with Pre-gated MoE.\n\nPre-gated MoE allows for the parallelization of expert prefetching and computation. Compared to this approach, our method has the following advantages:\n+ Pre-gated MoE does not increase inference latency under the assumption that the time to load parameters is less than or equal to the computation time. However, during batch decoding, since different tokens require different experts, the number of experts to prefetch increases (in the worst case, all experts need to be loaded), which in turn increases the latency.\n+ Additionally, when deployed on edge devices, it may be necessary to offload to lower-tier storage devices such as SSDs, where the transfer bandwidth is much smaller, causing the time to load parameters to be significantly higher than computation time, leading to a substantial increase in overall latency.\n+ In contrast, our MoLoE fundamentally reduces the amount of data that needs to be transferred by more than three orders of magnitude, thus avoiding additional latency during decoding. The two methods actually offer two different solutions in distinct directions.", "claims_and_evidence": "I think the claims and evidence are good.", "methods_and_evaluation_criteria": "Something is not aligned with the baselines.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "I checked the experimental designs and analyses. Overall, it is good, but it needs more alignment and baselines.", "relation_to_broader_scientific_literature": "This work mainly focuses on the loading latency for MoE models in the offloading scenario.", "essential_references_not_discussed": "I think it's better to compared with Hwang, Ranggi, et al. \"Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference.\" 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024.", "other_strengths_and_weaknesses": "**Strengths:**\n1. Introduce the LUT to combine the advantages of the Dense Model (Balance Training) and MoE architecture (Parameter Scaling).\n2. Training all experts simultaneously removes the need for auxiliary losses, simplifying training and improving efficiency.\n\n**Weaknesses:**\n1. In Table 3, the comparison between MoE (0 shared, top-2 routing) and MoLoE (1 shared, 4 routed) is inconsistent. I think a direct comparison under identical conditions would strengthen the evaluation.\n2. I have concerns about the context information in your MoLoE model. After being re-parameterized, during the inference, the input ID and output are fixed for each token. As Table 3 shows, performance under LAMBADA is weaker than MoE. Could you provide results under complex language modeling tasks like WikiText or Winogrande?\n3. As I understand, the tokenizer is important for the LUT's construction. Have different tokenizer settings been explored? An ablation study or discussion regarding tokenizer will be better.", "other_comments_or_suggestions": "I think some system-level optimization work should be discussed or as a baseline.\ne.g. Hwang, Ranggi, et al. \"Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference.\" 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024.", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nYOS3QRdL5", "reviewer": "Reviewer_xkaR", "summary": "This paper proposes MoLoE architecture to address the high memory overhead of MoE architectures. The key difference is that MoLoE converts experts into external LUTs before inference, eliminating expert computation and allowing experts to be stored outside of VRAM. Additionally, since only the lookup results need to be transferred to the GPU, MoLoE nearly eliminates the parameter transmission overhead associated with offloading. Experimental results demonstrate that MoLoE achieves competitive performance with MoE while maintaining latency and VRAM usage comparable to dense models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address specific concerns and questions below.\n\n\n\n> Q1. Will MoLoE require more training time than MoE?\n\nSince MoLoE activates all parameters during training, its training cost is approximately equal to that of a dense model with the same number of parameters, and higher than that of a sparse MoE model with the same parameter count. However, it should be noted that our method is designed for edge deployment of LLMs, where the parameter size is limited by the computational resources available on the edge, typically resulting in smaller models. Therefore, the cost of training these smaller-scale LLMs on a cluster remains manageable.\n\n> Q2. Why is MoLoE able to perform better than MoE?\n\n+ Although the experts' inputs do not contain context information, they can still indirectly influence the model's handling of context by altering the input to subsequent attention layers. Additionally, the retained shared expert operates similarly to MoE and dense models, where it takes the output from the previous layer as its input, and thus can also handle context-related information.\n+ By changing the expert inputs to embedding tokens, the routed experts can be reparameterized as LUTs, resulting in almost zero computation during inference. This allows us to activate all experts without incurring additional costs, which helps recover some of the performance loss caused by altering the expert inputs (see Table 7). Additionally, since activating all experts does not require auxiliary losses, the training and inference objectives are more aligned, leading to improved model performance (see Table 4).", "claims_and_evidence": "The claims made in the paper are supported by experiments.", "methods_and_evaluation_criteria": "MoLoE meets the requirements of the intended application scenarios. The benchmarks used in the paper are commonly adopted for LLM evaluation.", "theoretical_claims": "No theoretical proof is presented.", "experimental_designs_or_analyses": "The experimental section includes comparisons with MoE and dense models, as well as ablation studies, which are sound and valid.", "relation_to_broader_scientific_literature": "MoLoE discretizes inputs into a finite set by using embeddings as expert inputs, which is conceptually related to quantization methods. Input quantization could be a potential alternative solution for MoLoE.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Pros:\n1. This paper focuses on the challenges MoE faces during deployment, and MoLoE offers a novel and practical solution.\n2. The writing is clear.\n3. The experiments are thorough and convincing.\n\nCons:\n1. MoLoE has more offloaded parameters than MoE. Although off-device storage is larger, this still requires additional deployment overhead.\n2. MoLoE activates all expert parameters during each training step, which means that during training, the FLOPs of MoLoE will be higher than that of MoE with the same number of parameters.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. When the number of parameters is the same during the training phase, will MoLoE require more training time than MoE?\n2. Intuitively, using embeddings as expert inputs would limit the experts to only accepting context-independent, shallow features. So, why is MoLoE able to perform better than MoE?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMNAC6r6Aq", "reviewer": "Reviewer_vBLo", "summary": "This paper presents MoLoE, a new MoE architecture designed to address the high VRAM usage of traditional MoE models. MoLoE uses the output of the Embedding layer as input for the experts during training. Before inference, it pre-computes the output of each expert for each token ID, reparameterizing the experts as LUTs. By offloading the LUTs to other storage devices, MoLoE reduces the memory overhead of the experts while introducing negligible data transfer latency, and achieves performance superior to MoE with the same number of activated parameters.\n\n## update after rebuttal\n\nThanks for addressing my questions, I keep my rating as accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address specific concerns and questions below.\n\n> Q1. Pruning LUT entries corresponding to rare token IDs.\n\nWe prune half entries of the LUTs of MoLoE-16E with 160M activated parameters based on frequency. When the input token ID is pruned, the LUTs return a zero vector. The results are as follows:\n\n|Model|#offloaded param|#activated param|ARC-C|ARC-E|BoolQ|HellaSwag|PIQA|RACE|SIQA|LAMBADA|average|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|MoLoE-16E|7.4B|160M|22.4|48.6|60.3|32.7|68.3|30.9|38.6|33.3|41.9|\n|MoLoE-16E-pruned|3.7B|160M|21.7|44.8|61.8|31.2|64.5|28.8|39.3|26.6|39.8|\n\nFrom the experimental results, it is evident that post-training LUT pruning significantly degrades model performance. If smaller LUTs are required, using a more compact vocabulary during the training phase is a better choice.\n\n> Q2. Why did the MoE baselines not use shared experts?\n\nThe different settings for MoE and MoLoE are chosen in order to prioritize aligning both **total parameter count in training** and **activated parameter count in inference** for a fair comparison.\n\n+ MoLoE adopts shared experts because, unlike MoE, MoLoE reparameterizes all routed experts into LUTs during inference. If shared experts were not used, MoLoE would not include any FFN layers during inference, which would make it difficult to align its activated parameter count (VRAM usage) with that of dense and MoE models.\n+ The reason MoE does not adopt shared experts is that, in our experiments, MoE without shared experts performs better, which has also been thoroughly validated in [1]. We provide our experimental results below (No.1 vs No.2).\n+ In inference stage, MoLoE only has 1 shared expert, while MoE activates 2 experts. Therefore, to make a fair comparison, we set the size of each expert in MoE to half of that in MoLoE (to align activated parameter count in inference), but double the number of experts (to align total parameter count in training). As a result, the total number of experts cannot be aligned (No.2 vs No.3). But as a reference, we have still provided the results below for MoLoE and MoE both using 1 shared expert and 33 routed experts (No.2 vs No.4).\n\n|No.|Model|#total param in training|#activated param|ARC-C|ARC-E|BoolQ|HellaSwag|PIQA|RACE|SIQA|LAMBADA|average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|1|MoE (0 shared, top-2 from 34 routed)|1.1B|160M|20.5|50.0|57.5|34.5|67.3|28.6|39.9|36.4|41.8|\n|2|MoE (1 shared, top-1 from 33 routed)|1.1B|160M|20.6|48.4|57.3|31.5|65.2|30.5|37.9|34.7|40.8|\n|3|MoLoE (1 shared, 16 routed)|1.1B|160M|22.4|48.6|60.3|32.7|68.3|30.9|38.6|33.3|41.9|\n|4|MoLoE (1 shared, 33 routed)|2.1B|160M|22.0|52.4|59.3|33.5|68.3|30.7|38.8|33.1|42.2|\n\n[1] OLMoE: Open Mixture-of-Experts Language Models. ICLR 2025\n\n> Q3. Add citations to the papers mentioned above.\n\nThank you for your suggestion, we will update it in the revised version.", "claims_and_evidence": "The paper primarily makes the following claims:  \n1. MoE faces deployment challenges, including high VRAM usage or significant offload latency.  \n2. MoLoE can reparameterize the experts during the inference stage, resulting in lower VRAM usage and inference latency.  \n3. MoLoE achieves performance comparable to or better than MoE.  \n\nThese claims are either theoretically valid or can be empirically demonstrated through experiments.", "methods_and_evaluation_criteria": "The proposed method reduces data transfer by several orders of magnitude compared to MoE, without sacrificing performance, while maintaining the same number of activated parameters and VRAM usage. This aligns with the objectives of the paper. However, a side effect of this approach is the need to maintain large LUTs in external storage devices, which may limit the applicability of the model in certain scenarios.", "theoretical_claims": "The paper does not introduce particularly complex techniques, and its effectiveness is primarily demonstrated through experimental results. The ablation study in Table 7 effectively highlights the validity of the model design.", "experimental_designs_or_analyses": "The experiments are conducted on 8 tasks, with dense and MoE models used as baselines for comparison. The metrics include accuracy, number of activated parameters, number of parameters during training, offloaded parameters, and number of data transferred. The experiments demonstrate the validity and effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "The method seems to be related to reparameterizable network architectures in computer vision, as both approaches simplify the model by reparameterizing after training. For example:\n\n+ RepVGG: Making VGG-style ConvNets Great Again", "essential_references_not_discussed": "Although it can be considered as concurrent work, the authors are encouraged to include the latest papers from ICLR 2025 in the related work section.\n\n+ MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts\n+ TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice", "other_strengths_and_weaknesses": "Strengths:\n+ MoLoE addresses the challenges of MoE deployment -- high VRAM requirements and the large delays associated with offloading experts, offering practical value.\n+ The experiments demonstrate that the model achieves better performance than MoE while reducing the number of transferred parameters by over a thousand times.\n\nWeaknesses:\n+ As a trade-off for the strengths mentioned above, the model needs to store larger LUTs on storage devices, potentially several times larger than those of MoE, which may limit the applicability of the model in certain scenarios.", "other_comments_or_suggestions": "+ Add citations to the papers mentioned above.", "questions_for_authors": "+ Are there other ways to reduce the size of the LUT, such as pruning LUT entries corresponding to rare token IDs?\n\n+ Why did the MoE baselines not use shared experts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3DhGFwKGEO", "reviewer": "Reviewer_VEVw", "summary": "This paper introduces a new LLM architecture, MoLoE. MoLoE enables the conversion of experts into lookup tables before inference, which does not require computation and is placed outside of VRAM, thereby reducing its VRAM usage to levels comparable to dense models. At the same time, MoLoE only needs to transfer the lookup results into VRAM during inference, without the need to transfer any model parameters, making its extra latency for data transfer negligible. This addresses the issues of high VRAM usage in MoE models and the significant latency caused by expert offloading. Experiments on a range of language understanding datasets demonstrate that MoLoE can achieve performance comparable to, or even better than, MoE models.\n\n\n## Update After Rebuttal\n\nThe authors’ response has adequately addressed my key concerns, particularly regarding the long-context optimization strategies and MoE’s inefficiency in single-GPU deployment. I am keeping my rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We address specific concerns and questions below.\n\n> Q1. As the model scales up, the usability of MoLoE may come into question.\n\nFirstly, the number of LUT parameters that need to be offloaded by our method is $dN|V|$, whereas MoE needs to offload the expert size of $2dND_r$. Since $|V|$ is typically on the order of tens of thousands, the difference between the two is not greater than an order of magnitude. Additionally, the method is designed for edge deployment of LLMs, where the parameter size is constrained by the available computational resources, typically resulting in smaller models. Therefore, the size of the LUTs stored on lower-level storage devices is generally acceptable.\n\n> Q2. When the prompt length exceeds 10k, MoLoE-4E would need to load more parameters than MoE-10E\n\nIn the extreme case of very long texts, the amount of parameters our method loads during the pre-filling phase may indeed be greater than that of MoE. However, during LLM inference, the time required for multi-step decoding remains the dominant factor, meaning the cost during the prefilling phase can be somewhat offset. Additionally, directly loading the expert parameters before reparameterization during the pre-filling phase and using computation to replace loading is another potential solution.\n\n> Q3. Why is MoE significantly slower than the dense model?\n\nOur implementation is based on the Mixtral code from the Huggingface transformers package. Unlike cluster environments that can utilize expert parallelism, edge-side single-card deployment requires each activated expert to process tokens individually, resulting in lower parallelism. When the batch size is greater than 1, the parallelism is further reduced due to different tokens requiring different experts, leading to slower speeds. It is worth noting that the inefficiency of MoE computation could potentially be addressed in the future through more friendly designs of hardwares or better operator implementations. However, even if the single-card execution efficiency of MoE becomes comparable to that of dense models, our method still maintains a significant advantage in terms of the latency associated with loading parameters.\n\n> Q4. Is it possible to use modules other than FFN as experts?\n\nIn fact, as long as the module does not involve information exchange between tokens, i.e., the module’s input is context-independent, it can serve as an expert in MoLoE. This characteristic is similar to the FFN layer in transformers. Therefore, we adopt FFN as the expert, ensuring that the type of expert is consistent with the MoE model.", "claims_and_evidence": "The paper claims that MoLoE achieves lower latency and better performance compared to MoE, with the same VRAM usage and number of activated parameters. These claims are supported by experimental results.", "methods_and_evaluation_criteria": "This paper essentially reduces VRAM usage and inference latency at the cost of increased storage overhead outside of VRAM. For scenarios with limited VRAM and sensitivity to latency (such as edge-side large language models), MoLoE holds practical value.", "theoretical_claims": "This paper is application-oriented and does not involve theoretical proofs. The explanatory formulas and parameters/FLOP calculations presented in the paper are correct.", "experimental_designs_or_analyses": "The paper evaluates the model on language understanding tasks and compares it with dense and MoE models. The comparison is conducted under the control of the same number of activated parameters and VRAM usage, making it convincing.", "relation_to_broader_scientific_literature": "Since this paper aims to address the issue of large parameter sizes in MoE, it seems to align with the goal of expert pruning.", "essential_references_not_discussed": "As mentioned above, due to the similar objectives, expert pruning methods should be discussed, such as [1].\n\n[1] Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models, ACL 2024.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The motivation is clear, and the method makes sense. This paper introduces a novel approach for reducing the VRAM usage of MoE.\n2. The paper demonstrates the effectiveness of MoLoE through comprehensive experiments.\n\n**Weaknesses:**\n\n1. In certain settings, MoLoE’s offloaded parameters are significantly more than those of MoE. For example, in the setting with 160M activated parameters, the offloaded parameters of MoLoE-16E are more than seven times those of MoE. As the model scales up, the usability of MoLoE may come into question.\n2. Theoretically, when the prompt length is sufficiently long, the lookup results that MoLoE needs to load during pre-filling could surpass the size of MoE's experts. For example, in a model with 1B activated parameters, the MoE-10E expert parameter count is 2.7B (assuming all experts are loaded for each layer and then deleted after computation). On the other hand, MoLoE-4E loads 0.26M parameters per token. This means that when the prompt length exceeds 10k, MoLoE-4E would need to load more parameters than MoE-10E.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1. In Figure 3, why is MoE significantly slower than the dense model, even when ignoring the loading time, despite both having the same number of activated parameters?\n2. Since the experts in MoLoE can be reparameterized as LUTs, the specific architecture of the experts during training should not affect the architecture during inference. Is it possible to use modules other than FFN as experts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission worked on mixture of experts (MoE) and proposed mixture of lookup experts (MoLoE), a new MoE architecture that is efficient in both communication and VRAM usage, where the experts can be re-parameterized as lookup tables that retrieves expert outputs based on input IDs and offloaded to storage devices. The authors did a particularly good job in their rebuttal to address most of the concerns raised by the four reviewers. More specifically, three reviewers gave the accept; the only negative reviewer whose overall recommendation is still the weak reject, Reviewer iLJn, did not argue against novelty or practicality, and his or her requirements for more experiments were satisfied by the authors (especially the first weakness). I think the paper is potentially very impactful in the near future given its important topic and nice idea with good performance so that it may significantly push forward the research and/or applicability of MoE architectures. Therefore, we should definitely accept this submission to be published at ICML 2025.
Decision: Accept (oral)