CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing
Authors: Yifan Zhou
Tianshi Xu
Jue Hong
Ye Wu
Meng Li
Venue: NeurIPS 2025 poster ()
Keywords: Privacy-Preserving Deep Learning
Homomorphic Encryption
Multi-party Computation
Mixture of Experts
Large Language Model
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose CryptoMoE, the first framework to enable private, accurate, and efficient inference for MoE-based LLMs.

Abstract: Private large language model (LLM) inference based on cryptographic primitives offers a promising path towards privacy-preserving deep learning. However, existing frameworks only support dense LLMs like LLaMA-1 and struggle to scale to mixture-of-experts (MoE) architectures. The key challenge comes from securely evaluating the dynamic routing mechanism in MoE layers, which may reveal sensitive input information if not fully protected. In this paper, we propose CryptoMoE, the first framework that enables private, efficient, and accurate inference for MoE-based models. CryptoMoE balances expert loads to protect expert routing information and proposes novel protocols for secure expert dispatch and combine. CryptoMoE also develops a confidence-aware token selection strategy and a batch matrix multiplication protocol to improve accuracy and efficiency further. Extensive experiments on DeepSeekMoE-16.4B, OLMoE-6.9B, and QWenMoE-14.3B show that CryptoMoE achieves $2.8\sim3.5\times$ end-to-end latency reduction and $3\sim6\times$ communication reduction over a dense baseline with minimum accuracy loss. We also adapt CipherPrune (ICLR'25) for MoE inference and demonstrate CryptoMoE can reduce the communication by up to $4.3 \times$.

CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing
Yifan Zhou Peking University
Tianshi Xu* Peking University
Jue Hong
Independent Researcher
Ye Wu
Meng Li" Peking University
Independent Researcher
Abstract
Private large language model (LLM) inference based on cryptographic primitives offers promising path towards priivacy-preeeree deep learning. However, isting frameworks only support dense LLMs like LLaMA-1 and struggle to scale mixttre--o-eepeere (MoE) architectures. The key challenge comes from curely evaluating the dynamic routing mechanism in MoE layers, which may reveal sensitive input information if not fully protected. In this paper, we propose Crypto- MoF, the first framework that enables private, efficient, and accurate inference for MoE-based models. CryptoMoE balances expert loads to protect expert routing information and proposes novel protocols for secure expert dispatch and com- bine CryptoMoE also develops a confidence-aware token selection strategy and a batch matrix multiplication protocol to improve accuracy and efficiency further. Extensive experiments on DeepSeekMoE- 16.4B, OLMoE-6.9B. and QWenMoE- 14.3B show that CryptoMoE achieves 2.8 ~ 3.5x end-to-end latency reduction and 2.9 4.3x communication reduction over a dense baseline with minimum accuracy loss. We also adapt CipherPrune (ICLR'25) for MoE inference and demonstrate CryptoMoF can reduce the communication by up to 4.3x. Code is available at: https: thub. com SOom/PKU-SEC---O Lae--Laa//CCCyyyre
Introduction
Sparsely-gated mixture-of-expert (MoE) models have emerged as a powerful architecture for scaling up large language model (LLM) capacity without proportionally increasing the computation cost. As a result, many state-of-the- the-art LLM families, including LLaMA-4 [1], DeepSeek- V3 [2], and QWen-3 [3], have adopted MoE as their core architecture.
Driven by the high model capacity, MoE-based based LLMs are increasingly adopted in real-world ap- plications, some of which involve sensitive user data, e.g., person re-identification [4] and medical diagnostics [5]. Therefore, data privacy has become major concern and has propelled the devel- opment of privacy-preserviv
inference frameworks. Hybrid cryptographic approaches combining Homomorphic Encryption (HE) and Secure Multi-Party Computation (MPC) are considered a promis- ing solution. They enable the user and model provider (server) to jointly compute LLM outputs without exposing either the user inputs or the model weights.
However, existing private inference frameworks primarily support dense architectures such as GPT- 2 [6] and LLaMA-1 [7], and lack support for MoEebased models. core challenge lies in how to
These authors contributed equally to this work. "ooresppoddigg author: meng. @ @ku.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Aod--Nomolie
Insecure
secure
Expert
Expert
Expert 63
"Mith Expers
MDE Layer
Ado-Moom
Router
wwwwm
www.mw
Self-Allntion
Toben
Token
Triten
Toom
Expert
Expert ids Math Task
Text Unddrstannngg
w/ CryptoMoE
(a)
(b)
(c)
Figure 1: (a) Structure of MoE- LLM; (b) Expert activation in 10-th layer of DeepSeekMoE differs notably between text understanding and math reasoning tasks. The underlying data is provided in Appendix A; (c) CryptoMoE features
balanced expert routing.
securely evaluate the dynamic routing mechanism inherent to MoE layers. As illustrated in Figure I(a), MoE operates by activating a subset of experts for each input token, where each expert is distinet sub-network. Figure (bb further demonstrates that expert activation patterns are highly input-dependent and need to be protected: in 10-th layer of DeepSeekMoE [8], experts #3 and #60 frequently activated for mathematical tasks across two math datasets [9, 10], but exhibit uniform distribution across eight textual reasoning datasets [11, 12, 13, 14. 15. 16, 17J. Similar findings have been reported in previous work [18, 19]. This indicates that individual experts often specialize in specific semantic domains. Consequently, revealing expert routing information may leak sensitive details about both the input type and the internal specialization of different experts.
natural solution to protect the routing information to eliminate the sparsity of MoE and route all tokens through all experts, denoted as the dense baseline. However, this approach protects privacy the cost of significant computation. For instance, in QWenMoE (4 out of 60 experts), it increases computation by about 15x, eliminating the efficiency benefits of the MoE structure.
To this end, we propose CryptoMoE, the first framework enabling private, efficient, and accurate inference for MoE-based LLMs. CryptoMoE features a key idea we term Inference-Time Balanced Expert Routing. Each expert processes a fixed number of tokens, denoted by t. regardless of the actual routing results. Tokens beyond this limit are discarded, making expert contributions input-independent and preserving privacy. By carefully selecting t, we can achieve strong privacy guarantees with little no increase in overall computation.
However, naively discarding tokens exceeding the threshold leads to significant accuracy degradation. To alleviate this, we introduce a confidence-aware selection strategy. Among the tokens assigned to given expert, we re-rank them by their routing confidence and retain only the top-/ tokens. To further support private inference under the balanced expert routing, we design confidence-awanl secure dispatch protocol that privately assigns tokens to their target experts, ensuring each expert receives the top-t tokens with the highest routing probabilities. A corresponding secure combine protocol aggregates the expert outputs and reconstructs the final result for each token. Together, these protocols introduce only around 18% additional communication and computation overhead, while reducing expert computation by 8 ~ 15x, without leaking any routing information.
We further identify the expert linear layers the main computational bottleneck in MoE inference. To address this, we propose an efficient Batch Ciphertext- Eiphertext-Plaintxtt Matrix Multiplication (Batch MatMul) protocol, which packs tokens assigned to different experts into a single ciphertext. This reduces the number of costly HE rotation operations by factor of n, where n is the number of experts, significantly improving inference efficiency.
We evaluate CryptoMoE on three representative MoE-based LLMs: DeepScekMoE-2.88B// [88), 1.MMMoE-..7B14.31 [20], and OLMoE- 1.3//6 99 [21], across eight zero-shot reasoning tasks. Results show that CryptoMoE retains 99.2% of the original accuracy on average, while achieving a 2.8 ~ 3.5x speedup over dense baseline. Moreover, CryptoMoE achieves efficiency comparable to the insecure baseline that fully reveals routing information. We also adapt CipherPrune [22]'s pruning protocol to the MoE setting and construct a strong baseline. Compared to it, CryCttytoMoE achieves up to 4.3x communication reduction and 2.4x latency reduction. These results establish CryptoMoE. as the first framework to enable private, efficient, and accurate inference for MoE-based LLMs.

---

Table 1: Underlying Protocol and Description
Protocul
Description
Protocol
Description
Tmux
Hama [6] [x]],
b..
Maa
Haa[(2z] [y] S. a.y
Naua
: Hoqut
[x] [x] y 2 1(x == y)
: T (1r) s.l. soltmax
Husk
W]. [K] Top [ 8). s.L. W, K = Top-K(a, k)
[2] Hmamal [ D....... XY
Honete
F Hoseon
[2] c),
onehot(x, c), where 2
1
1). Vi c 10.c
Preliminaries
Notations. We use {x:)==d to denote set { 21.-. number of experts, tokens, and the number of experts to be activated, respectively, We use ((P) to denote the indicator function, which is I when P is true and 0 otherwise.
.n-1). We use n, m, to denote the
2.1 Mixture of Experts Layer
We present a brief introduction to the Mixture of Experts layer. The output of the MoE module for given input æ is determined by the weighted sum of the outputs of selected expert networks. The gate routing determines the weights and the selected experts:
W, K Top-K(G(z),k),
(I)
where & is the number of experts activate, G the gating network implemented by the softmax over linear layer, i.e., G(z) = Softmax Softmax (Linear(z)), K denotes the indices of selected experts and W = (Ge is the of the selected experts. The output of MoE layer then given by:
MoE(x)
Σ W. E,(z) KEK
E((x) == SwiGLU,(x))
Each expert network E; is a feed-forward network (FFN) implemented by SwiGLU [23]. For input with multiple tokens {x; ) tokens are routed to different experts based on the gating network. Then each expert network E& processes the tokens distributed it in parallel.
2.2 Cryptographic Primitives
Homomorphic Eneryption (HE). Following most hybrid HE/MPC schemes [24, 25, 26, 27, 28], CryptoMoE leverages the Brakerski-Fan Fareerrauteeeee (BFV) HE scheme [29] and mainly involves the following element-wise HE operations: ciphertext addition, sophertext-lainteer multiplication, and ciphertext rotation Rot(ct, s), which shifts the ciphertext et to the left by positions.
Secure Multi-Party Computation (MPC). We employ a 2-out-of-2 additive Secret Share (SS)-based MPC scheme [33] to keep the input data private throughout inference. We denote two parties by Po and Pi, where Po is the client and P1 is the server. We use [>:] to denote an additive share of a.. We
write [x]] = [[=]o. [x]]]) where Po holds [и]o and P1 holds [x]1. such that [æ]o- + [x]] = x. We
write [x]
denote the share of Boolean data. This work builds upon pre-existing MPC protocols
whose input and output are additive shares [30, 24, 31]. These protocols are summarized in Table 1.
Among them, mmmatmul is implemented using HE [24], while the rest are implemented using oblivious
transfer (OT) [30,3] 31].
Threat Model and Security Guarantee. CryptoMoE works in general private inference scenario that involves two parties, i.e., server Pi and client Po. The server holds the proprietary NN model, and the client owns private input [26, 28, 27, 32, 24, 31]. CryptoMoE enables the client to obtain the inference results while keeping the server's 's model weights and the client's 's input private. Consistent with previous works [26, 30, 33, 27, 24, 31], CryptoMoF adopts an honest-but-curious security model in which both parties follow the specification of the protocol but also try to learn more than allowed. CryptoMoE is built upon cryptographic primitives, including BFV and MPC protocols, the security can hence be guaranteed following [29, 34j.
2.3 Related Work
With the proliferation of ChatGPT, significant efforts have been made to enable private Transformer inference, including hybrid HE/MPC frameworks [32, 24, 31, 35, 36, 37, 38), Fully-HE frame- works [39, 40, 41] and FuP www.wworrrrrrr........... frameworks [42, 43, 44, 45]. However, these works only support

---

Useful taken with positvee score
Dummy token with zero score
Not computed dropped token
%
Comtine
Privecy
Accuracy
Erficenccy
Bp
Bs
Insecure
Dispatch
2 R Roteer
D0: Top-2 Router
Top-2 Router
11
Dense
X
Xp
Xg
(a) Insecure Baseline
(b) Dense Baseline
(c) CryptoMOE (Ours)
Figure 2: Toy examples and qualitative comparisons between baselines and CryptoMoE.
dense models like GPT-2 [6]. Recent work, CipherPrune [22], introduces dynamic token pruning for private Transformer inference. Nevertheless, applying its pruning protocol directly to MoE layers only leaks the number of tokens assigned to each expert but also incurs substantial communication overhead. Thus, how support private MoE-based model inference is still an open question.
3 Private MoE Inference and Baselines
We first establish general conceptual framework for private inference in MoE models, where all intermediate results are kept in secret shared form. The framework consists four steps:
Gate Routing. Given input tokens [[x;] 1 generating routing score [W] and indices
of selected experts [[]], as Equation @ Dispatch LLdispatch Given [W]. [4]. Iddispath is expected to securely determine the set of tokens assigned to each expert, denoted as {[<])=> where [X:] = [[=,] xy is routed to expert F.) Expert Compute. Given [[> each of the experts performs computation as [VE,] ([X.)) O Combine Lecmbiin-- After obtaining all expert outputs [[ Icombine protocol is intended to securely aggregate the outputs and produce the final token-wise results {Iu])=='
As mentioned in Figure I(b), even revealing the number of tokens assigned to each expert in step e may leak information about the types of input and the experts. Therefore, ensuring private inference for MoE models without disclosing any information about routing information W. K constitutes challenge. We first construct two baselines that serve benchmarks across three key dimensions: privacy, efficiency, and accuracy, as shown in Figure 2.
Insecure Baseline where [W], [K] is revealed in public, then the dispatch step can be executed in plaintext without extra cost. Expert computation remains encrypted. This baseline achieves the highest accuracy and efficiency, as its computational flow is identical to the plaintext counterpart and avoids extra overhead from Iddispath- However, leaks complete routing information and thus serves only upper bound reference for accuracy and efficiency.
Dense baseline. To protect routing information, an approach is follow the non-MoE models by evaluating all experts for every token, regardless of routing decisions, depicted in Figure 2 (b). This removes the need for Huispathh and the final output a weighted sum based on routing scores, with non-selected experts receiving zero weight. While this method protects routing privacy and maintains the same accuracy as insecure baseline, drastically increases computation. For example, in QWen-MoF. with 4-out-of-60 expert selection, it incurs a 15x increase in expert computation. Therefore, achieving private, efficient, accurate inference for MoE models remains an open question.
4 CryptoMoE Framework
4.1 Inference-Time Balanced Expert Routing
In this section, we introduce CryptoMoE, private, efficient, and accurate MoE inference framework. Building upon the dense baseline, CryptoMoB advances key idea: Inference- Time Balanced Expert Routing, where each expert processes exactly tokens, regardless of the routing outcome.
Figure shows the private inference workflow of MoE layer with CryptoMoF. Step 000 follows the procedure described in Section 3. The key differences is step @, which invokes a Ildispatch to

---

CI X
[26illi-g Xi (cX ()
Expert
Expertt2
OulputA
Taken
Routing
Expert Indicam)
[KT 1Sclcoted
Token A
Token B
Tokan
Expert
Quiput
Marshing
Output A Dutput
Token
Top-2 Router
Expert
Expert3
Compute
Expert
Token
(11 x d)
[v] [Scoms))
Expert
(m x
Drappedd
Dummy
(ya,lio yE, (1xx)
Figure 3: The workflow of private MoE layer inference in CryptoMoE.
produce {X)]=0 without leaking routing information. Notably, Tisputch enforces that each expert's input X, contains exactly Ł tokens. If fewer than í tokens are routed to an expert, dummy tokens are added for padding; if more than & are routed, excess tokens are dropped and not computed. By appropriately choosing t, for example by setting expert, we can maintain routing privacy without increasing computational cost. However, realizing such balanced expert routing in the private inference setting introduces three key challenges:
mk/nn the expected number of tokens per
Significant accuracy degradation. We observe that dropping tokens beyond the threshold t during dispatch can lead to up to 7% accuracy loss, as some discarded tokens are critical to the final output. Thus, minimizing the accuracy loss caused by token dropping is the first challenge.
Challenge 1:
Challenge 2: Construction of dispuich and combine. While step 1 can be implemented by existing protocol I[softmax MaMMMul and l proposed in Bolt [24] and Bumblebee [31], constructing IIdispatch and Lomomiiee is non-trivial. Idippath must securely assign tokens to experts based on routing information and select the & tokens for each expert. Similarly, constructing the lcombiie protocol to aggregate expert outputs into token-wise results is complex. Designing both protocols using MPC and HE protocols to preserve privacy without incurring significant overhead remains a challenge.
Challenge 3: High cost of linear layer computations. In expert computation, the three linear layers in SwiGLU dominate the overall latency. Since each expert receives only few tokens, the number of token dimensions that can be packed per ciphertext is limited, leading to an excessive number of costly HE rotations. Reducing this overhead is critical for improving efficiency.
To tackle these challenges, we propose: (i) a confidence-aware secure dispatch protocol that protects routing information while alleviating accuracy loss (Section 4.2); (ii) a lightweight and secure combine protocol (Section 4.3); and (iii) a batch matrix multiplication protocol that reduces HE rotations by factor of n, significantly accelerating expert computation (Section 4.4).
4.2
Confidence-Awaare Secure Dispatch Protocol
Token discarding occurs when more than & tokens are routed to the same expert. Uniformly selecting t tokens with equal probability can lead up to 7% accuracy loss. To mitigate this, we propose confidence-aware selection strategy: re-rank the tokens assigned to each expert based their routing confidence W = G(x) and retain the top-ř tokens. As shown Figure 3, Expert 2 receives two tokens and selects token B, which has higher confidence score. As demonstrated in Section 5, this approach consistently improves accuracy across different models and datasets.
Next, we construct our secure Idiipptch-- In private inference, dispatching the appropriate tokens to each expert is challenging, as we must keep the routing information W, K as secret shares. Figure illustrates our confidence-awant secure dispatch protocol. Specifically, ITaisparch takes secret-shared tokens {[=]+>=">> and routing information [[W], [K]} as inputs, and outputs the set of tokens assigned to each expert, {[XI)=-] where each expert receives exactly tokens. Our core idea is that after Top-k routing, there are km candidate tokens to be assigned to n experts. For each expert, rank the km tokens by their confidence scores and select the top & tokens. Tokens not assigned to the expert have zero scores. This design ensures each expert receives the desired tokens.
For each expert E,, the protocol contains three steps: Compute token priority scores. Use Neguul to evaluate whether selected expert indices [K] are equal to i, producing boolean mask [M:] E {0, fmmm that indicate which tokens are useful for this expert. Next, we employ Tm to combine the routing scores [W] with the masks [M,] resulting in token priority scores [S.]]
5

---

Routing
Dispatch tokens for each expert:
Exparto
Expert
Expert
Expert
in token embeddings
(Rt x
Example for expert
(t d)
Router
O
Compute Taken
Select tokens
Retrieve tokens
Priority Scores
Taken Priority Scores [S]]
r Selected taken IDs [4;] taken embaddings [x]]
Scores [IY]
Ulaquai
(len,
Blppa 4)
Daaaoor c= m)
1
Desosel
X1 (km..)
Expert
Boolean masks [M:
D DB
[x] (m x
Compute Token Prierity Scores
[w] (Amm
Select tokens
[K (t:)
Retrieve tokens
Secret shared taken A/B
Currently being dis pelt hed expert ID
Degusl Secure protocal (OT)
Dasmm Secure protocol (HE)
Figure 4: Secure dispatch protocol for 4, 2,
Useful tokens retain their original scores, while useless (dummy) tokens' scores are zero. To proceed, we must select the top tokens from the km candidates based on these scores. Prior work CipherPrune [22], introduces secure pruning protocol capable of achieving this. However, it incurs high communication cost of O (kmtd),, where d denotes the hidden dimension. This overhead arises because each comparison involves a secure swap of the whole token embeddings, making impractical for MoE inference. To address this limitation, we propose a novel protocol that decouples token scores from embeddings, thereby reducing the communication complexity Okkm log(km)). A detailed comparison with CipherPrune is provided in Appendix C. Our protocol comprises two steps: @ Select token indices. We apply Ip (k = t) to select the tokens with the highest priority scores. The resulting indices [K,] 2 correspond to token positions within the dispatched sequence of length km.. To map these indices back to the original mntoken input sequence, we perform an integer division by k to get [K]] [[ K. ]]. S Retrieve tokens. Using the selected indices, we retrieve the corresponding token embeddings. Specifically, we convert the selected indices [K]] E z into a one-hot matrix using Honehox ( [[ m) E Ztxm, which requires (tmm) calls to Nequul- We then perform a MaMaal with the input token embeddings [æ:] E 2m to obtain the desired token embeddings for expert i, i.e., [X]] E 2dx
4.3 Efficient Secure Combine Protocol
After the computation by each expert, we obtain the output [yE.] for expert i. A combination process necessary to aggregate results across all experts into the final token-wise outputs [v:]] This process must address the token reordering challenge: each [ye,] contains & tokens, ordered according to Haispatch, which differs from the original sequence order.
To address this, we propose lightweight see-hot-based reordering method, illustrated in Figure 5 (a). We reuse the one-hot matrix [onehot( (()l ztxm computed in step 2 of Iddispanh. and perform a local transpose to obtain [onehot( K mmx, Next, we perform a IaMMa with [yæ.] E Zixd reorder the tokens and compute the final token-wise result [ys] E mmx The complete combine protocol is illustrated in Figure 5 (b). Before reordering, we use IT mul to multiply the one-hot matrix with token scores [[S,], producing a scored one-hot matrix [R.] E n Subsequently, token reordering and weighted masking are performed simultancously using a Immammul on [R.] and [ve.] Finally, the outputs from all experts are summed to obtain the final result for the MoE layer. With this construction, yombine requires only one Im and one mm making it highly efficient.
Complexity Analysis. For single MoE layer, the proposed Hasspach and Lombinee introduce additional communication overhead of Onkkm log(km) - ntm)), where the first term stems from the 1ook protocol [46] and the second from Tonchoe- Experimental results show that our protocol is highly efficient, incurring only an 18% overhead while preserving privacy and leveraging the sparsity of MoE computation. Further implementation details of dispatch and IT cembine are provided in Appendix B.
4.4 Efficient Batch Sighertext-Plainteer MatMul (Batch MatMul) Protocol
Linear layer evaluation remains a major bottleneck HE for expert computation. Unlike dense models, MoE layers process groups of parallel tokens, each of size x di (t tokens, each with

---

selected taken indioes [K/]
dispatch
Combine expert
Secret shared
20
token A/B
output tokens expert [7c,]
Fq (+)
[snclotER,
Scared onehot
[xx.
0
Nagaal
Secure protocol (OT)
[5] (km..)
Top t scores [5,] (6)
Mussead Secure protocol (HE)
0
0
(ox d))
OM
Sum
0
F
matrix (ooshbot(
Transposed Onehot
Reardered tokens
Expert output
Expert output
Expert output
Expert output
MoE output
(a) One-Hot-Based Reordering
(b) Secure Token Combine Protocol
Figure 5: (a) An example for one-hot-basee token reordering. this case, m 6, t 3. (b) Secure
combine protocol for 4, m 2,
1.
Batch MatMul
(a) Origin Packing (BOLT)
(b) Batch MatMul Packing (CryptoMoE)
Asc
inpat
Complexity comparison
Methad
#Rot
((ed,)
CrystoMoc
0(4
Rotations
Rotations
Figure 6: Batch MatMul protocol reduces the number of HE rotations from O(nd1) ) to O(d)). This image illustrates an example MatMul with batch size n = 2. need to compute C x B and F = D x E, where A, D E Ztxda
ciphertext inputs, B, E zØ are plaintext expert weights. this example, t= 2, d1 = 4, d2 = 4, and each ciphertext can pack elements. Through batch MatMul packing in subfigure (b), we reduce the number of rotations from 6 to 2.
an embedding dimension dj). Existing packing schemes like BOLT [24] optimize for dense models by packing along the t-dimension, reducing the packed hidden size in ciphertext and thus minimizing expensive ciphertext rotations. However, MoE layers, each expert handles only a few tokens. Applying these schemes increases the packed hidden dimension, leading to more rotations and an extremely higher computation cost.
To address this, we introduce efficient batch ciphertext- plaintext MatMul protocol tailored for MoE. computation. The key idea is to pack partial token embeddings from all experts into a single ciphertext. Figure 6 shows toy example where n = 2, f 2, d1 4, with each ciphertext holding elements. the original packing scheme, each expert's input matrix of size f d1 is packed into ciphertext, requiring 3 rotations per MatMul to accumulate partial sums, resulting in 6 total tokens into single ciphertext with shape (nt 41) The weight matrices are adjusted accordingly in plaintext without additional overhead. As shown in Figure 6 (b), this reduces the number of rotations just 2, thanks to the smaller hidden dimension each ciphertext. Our batch MatMul protocol reduces HE rotations from ((nd) to O(d1). ). Our method is also compatible with the Baby-Step Giant-Step (BSGS) algorithm [24], which can further reduce the number of rotation operations. The Complexity analysis is provided in Appendix D.
rotations for two experts. In contrast, our method packs partial embeddings
Experiments
5.1 Experimental Setup
Implementation. We implement CryptoMoF. upon the SecretFlow-Ste framework [47], which is popular framework for preeccypres deep learning. We adopt secure two-party computation
7

---

Model
Method
Tatency (sloken)
Comm. (MBrtoken)
Insecure"
1.22
6.55
31.3
DeegSeckMoE 2
CryssMot.
Dene
4.43
1,89
20.52
310.9
182.0
CryptoMoR.
0.77 [SS-)
R.41 (24+)
39.1 (M8-)
CryptoMuE...
1.06 (4.1))
9.40 (.2x)
71,9 (4.3k)
60
1.386699
OL
CryeaMolE
3.38
1.09
11.00
232.0
93.0
CryptoMal:
(411))
(20))
(5.5x)
CryptoMoR..oo
(33-)
(1.7)
(MB-))
39
Insecure"
0.94
5.56
19.5
OWeaMaak
2.78/14.50
CrytoMoo
Dense
4.26
2.72
291.4
8 61.90
CryptoMul:
0.56 (76-1 6.93 (2.vv)
25.5 (1.33)
CryptoMoR.zo
(073 (5++) 7.09 (2A43
47.7 (633))
0.5K
1K
2K
4K
Irsecure with rouning information.
Sequence Length
OrynoMoeF baseline with CigherPrune's protocol.
Table 2: Cost comparison of single MoE layer. Insecure baseline cannot leverage batch MatMul optimization, and is therefore slower than CryptoMoE in some cases.
Figure 7: Effect of and sequence length average accuracy. 62. is the aver- age accuracy of the original model.
(2PC) setting without trusted third party. All the experiments are performed on machine with an Intel Xeon Platinum 8468 CPU (48 cores and 2. 2.H)..). We consider two network environments: 1) LAN setting with 3Gbps bandwidth and 0.2ms latency; 2) WAN setting with 400Mbps bandwidth and 40ms latency. We simulate network environment via Linux Traffic Control.
Datasets and Models. We consider three popular MoE models: 1) DeepSeckMoe
(6 of
64 experts) [8], 2) QWenMoF.-2.7B. 14.3 (4 of 60 experts) [20] and 3) OLMoF-11.3B/... (6 of 60 experts) (ICLR'25 Oral) [21]. Since MoEE models typically follow similar design, CryptoMoF also be applied to other MoF models. All the models are evaluated on eight famous zero-shot common sense reasoning tasks, including SIQA [13], OBQA [17], BoolQ [1 ARC-easy, ARC-challenge [16], HellaSwag [14], PIQA [12], and WinoGrande [15].
Baselines. Since CryptoMoE is the first framework enabling private MoE inference, compare with three baselines: 1) Insecure baseline, 2) Dense baseline, 3) CryptoMoE. baseline where Ilussptch and combine are constructed by CipherPrune [22]'s pruning protocol.
Selection of t. The token count assigned to each expert plays a critical role balancing accuracy and efficiency. A larger generally leads to higher accuracy but at the cost of reduced efficiency. We argue that the lower bound of t mk /n, which matches the number of tokens computed in the original MoE model without introducing additional computation cost. However, due to the inherent imbalance in token routing, this setting often results some tokens being discarded, leading to accuracy degradation. Empirically, setting 2mk/n achieves favorable balance, where mk/n is the expected number of tokens per expert. In subsequent experiments, we denote configurations with t mk/n /,, 2mk/ n,, etc., CryptoMoEx=1.oo CryptoMooE<22.0 and forth.
5.2 Cost Comparison of Single MoE Layer
In Table 2, we compare the latency and communication costs for a single MoE layer. We evaluate the prefill stage and report the amortized per-token latency and communication by dividing the total values by the input sequence length. The results show that: 1) With CryptoMofx-2oo we observe 1.7 5.9x latency reduction and a 3.1 6.1x communication reduction compared to dense baseline across three models. Additionally, CryptoMoE==1.0 achieves a 2 ~ 11x communication reduction over CipherPrune's protocol. 2) As increases, the latency of CryptoMoB grows slowly due to batch MatMul optimization. larger allows more tokens to be packed together, reducing expensive HE rotations and limiting lateney growth. 3) CryptoMoE matches the insecure baseline's 's performance in some cases. This is because the insecure baseline cannot leverage batched MatMul, experts receive varying token counts, making it complicated to apply batching in HE.
5.3 End-to-End Evaluation
We benchmark the accuracy, end-to-end amortized latency, and communication cost of different meth- ods in Table 3, using a batch size of 16 and CryptoMoE:=2200 An ablation study both batch size

---

Table 3: End-to and-to-end comparison with baselines.
Model
Method
Accuracy (%)
Lafnnyy (mimioken))
Comun. (GB).
SIQA ORQA
BuolQ
ARR--myy
ARC-challengy
HellaSwag
PIQA
WinoGrande
LAN
WAN
Dee
Inseerse"
Dense
72.5
73.0
77.2
80.3
2.33
3.48
10.0
9.16
1,33
2.8R/16-4F 4B
CryptoMol
5,96
CryttaMole
32.8
42.8
72.4
72.2
47.3
76.2
90.2
70.4
61.8
0.76
4.11 (23))
2,46 (27-)
Insocure"
1.02
OLMoE L3B/99
Desse
4.75
3.82
CryptoMaa
75.5
66.4
1.60
CryptuMofl
32,9
45.6
74.7
75.2
47.4
75.8
79.4
68.6
62.5 (03)
0,36
2,81 a33
1.31 (290
Insccraee
68.9
2.41
1.00
UWeaMolE 2,78/14.35 3B
Desse
1.98
7.96
7.61
CryptoMolB...
79.0
64.3
15.0
78.2
7,88
7.54
CryttMMoi
33.8
42.6
79.6
09.1
44.0
76.7
50.7
69.5
62.0 (ou)
(.56
3.54 (300
1.76 (630
brseline purlic muting
CrypioMeli beadine CipherPrnne' protecal llapuss oombine
Dynamic MoB
LAN
Latency Breakdown
WAN
Irsecure
227
11
955
Dense
638
2743
CryploMoE
314
1634
CryptaMoB w/o Ratch MatMul
330
1516
CryptoMoF
3x
1319
2x
0
72
120
182
516
640
0
116
629
1168
2389
A Attention Sharedd Expects
Gate Routing
Dispatch
Expert Compute (Linear)
Expect Compute (SILU)
Combine
Figure 8: Latency breakdown under LAN and WAN settings.
and will be presented in Section 5.4. The results demonstrate the following: 1) CryptoMoE retains 99. 19.2% of the accuracy of the insecure baseline on average. 2) With comparable accuracy, Crypto- MoE reduces LAN latency by 2.8 3.5x. WAN latency by 1.7 2.2x, and communication cost by 2.9 ~ 4.3x compared to the dense baseline. Moreover, it offers to 2.4x latency reduction over CipherPrune's protocol with higher accuracy thanks our efficient diisputch and gombbine protocols.
5.4 Ablation Study
Ablation Study on ł and Sequence Length. CryptoMoE benefits from balanced expert loads, as fewer tokens are discarded. Since both L and the input sequence length influence accuracy, we perform a two-dimensional ablation study on DeepSeekMoE; results for other models are provided in Appendix E. We vary the average sequence lengths from 0.5K to 4K by changing the batch size from 8 to 64. For each sequence length, adopt different values and report the average accuracy across all datasets in Figure 7. We observe that increasing either t or the sequence length improves accuracy, but gains become marginal beyond 2K tokens. This is likely due to inherent expert load imbalance in the dataset rather than input length limitations. Overall, setting t provides a robust trade-off across different configurations.
Ablation Study on Different Components. We demonstrate the effectiveness of the proposed tech- niques by adding them step by step. As shown in Ta- ble 4, we observe that: 1) Without confidence-aware selection, balanced expert routing reduces latency but harms accuracy a lot. 2) The batched MatMul optimization substantially reduces the computational overhead of expert linear layers, leading to a 2 re- duction in end-to-end latency.
Accuracy
Lalency (min/token)
Dense Daseline
62.
2.33
DespSeellMoai 2.87/16 4R
Balanced Expert Routing
57.9
1.20
C selection
6188
1.20
+Batch MalMai
61.3
0.76
Dense Baseline
63.0
0.99
OLMoE 1. 1.38S5 OR
Balanced Exper Routing
509
0.55
+C Conflallnaa-mnoo selettion
62.5
0.55
+Bacch MatMul
62.5
0,36
Roseline
1.98
QWenMoll 278/14.38
+Balanced Exper Routing
Coodladennooteeeee seleation
62.0
1.23
Latency Breakdown. To analyze the bottleneck of CryptoMoE and other baselines, we profiled a single Transformer block from DeepSeekMoE under LAN and WAN settings. The breakdown is shown Figure 8. Except for the first item,
+Baach MalMul
62.0
0.56
Table 4: Ablation study of accuracy and amor- tized latency (LAN) on different components.
Experts," all other components

---

related to the dynamic MoE layer. We draw the following conclusions: 1) In our scenario with short sequence lengths, the MoE layer dominates runtime, accounting for 68% and 91% of the total latency under LANN and WAN settings, respectively. This highlights the necessity for MoR layer optimization. 2) Within the MoF. layer, the expert linear layers are the primary bottleneck. Our batch MatMul optimization reduces their cost by 3 ~ 6x, yielding a 2 ~ 3x reduction in overall latency. 3) The dispatch and combine protocols contribute only 18% of LAN latency while ensuring routing privacy 4) Under WAN, gate routing and dispatch lateney increases significantly, mainly due to the top-k protocol, which involves many communication rounds. Developing round-efficient top-k protocols remains key direction for future improvement.
Scalability. Our balanced expert routing strategy is scalable larger models. Figure shows the average accuracy of naive selection strategy (i.e., uniform random selection) and our CryptoMoE on can be seen that CryptoMoE consistently
hixtral-13B/470 [48] and LLaMA4-Scour 17B/109B 190044-9 [1]. outperforms naive selection across all configurations. CryptoMoF maintains 100% accuracy on Mixtral even with accuracy of the original model.
1.0. On LLaMA4-Scout our CryptoMoF4=2.0 maintains 98.8%
For private inference, due the large model size, the memory usage of the SPU during execution exceeds the physical memory capacity of our machine. Reducing the memory overhead of private inference, especially for larger models, remains a challenging problem.
70
65
65
Accuracy (%)
60
55
55-
-.**
Original Model (66.87%)
Original Model (69.13%)
Naive token selection
50
Naive token selection
50
Moy
45
CryptoMoE
45
t= 1.0
1.5
t= 2.0
t=
1.0
1=5
t= 2.0
Mixtral-13B/47B
Figure 9: Accuracy of random selection and our CryptoMoE on Mixtral-13B/47B and LLaMA4- Scout-17B/109B.
Limitation and Future Work
Under WAN settings, Tepk becomes bottleneck due to its massive communication rounds, which could be optimized future work. Additionally, inference-time balanced expert routing is less effective for very short input sequences (e.g., length 64), it leads severe imbalance, which is another promising direction for improvement.
7 Conclusion
We propose CryptoMoE, the first framework enable private, accurate, and efficient inference for MoE-based LLMs. CryptoMoE preserves privacy through balanced expert routing and introduces novel secure dispatch and combine protocols tailored for MoE layers. also incorporates batch Mat- Mul protocol to boost computational efficiency. Experimental results show that CryptoMoE achieves 2.8 3.5x x reduction in end andtto-end latency compared to the dense baseline and an up 4.3 reduction in communication cost over CipherPrune, all with negligible accuracy loss.
Acknowledgements
This work was supported in part by NSFC under Grant 62495102, Grant 02464104, and Grant 62341407, in part by the National Key Research and Development Program under Grant 2024YFB4505004, in part by Beijing Municipal Science and Technology Program under Grant Z2411000042244 and in part by 111 Project under Grant B18001.
10

---

References
[1] Llama4.
April
2025. Meta.
[2] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19493 19437 2024.
[3] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprini arXiv:2412. 151 2024.
[4] Prodip Kumar Sarker, Qingjie Zhao, and Md Kamal Uddin. Transformer- person re- identification: a comprehensive review. IEEE Transactions on Intelligent Vehicles, 2024.
[5] Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu. Transformers in medical imaging: A survey. Medical Image Analysis, 88: 102802, 2023.
[6] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
[7] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2330 2023.
[8] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-expent language models. arXiv preprint arXiv:2401 S.kiv:2401.06000 1..6666,,,66,,, 2024.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1xXi:22110.14160 2021.
[10] Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedzionse Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. arXiv preprint arXiv:1905. 13319, 2019.
[11] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: /905. 10044, 2019.
[12] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1432-7439, 2020.
[13] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiga: Com- 2019.
monsense reasoning about social interactions. arXiv preprint
[14] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 1xXiv:995.008830 2019.
[15] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106.
2021.
[16] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint urXiv:1803.05455 2018.
[17] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 1809.02789, 2018.
11

---

[18] Xingyi Yang, Constantin Venhoff, Ashkan Khakzar, Christian Schroeder de Witt. Puneet K Dokania, Adel Bibi, and Philip Torr. Mixture of experts made intrinsically interpretable. arXiv preprini S5V:25503.07639, 2025.
[19] Jing Li, Zhijie Sun, Xuan He, Li Zeng, Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, and Xin Chen. Locmoe: A low-overhead moe for large language model training. arXiv preprint
arXi::2401.139200 2024.
[20] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[21] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02000 2024.
[22] Yancheng Zhang. Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang. Lei Jiang, and Qian Lou. Cipherprune: Efficient and scalable private transformer inference. arXiv preprint arXiv:2502.1678, 2025.
[23] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.052022 2020.
[24] Q. Pang, J. Zhu, H. Möllering, W. Zheng, and T. Schneider. Bolt: Privacy-preserving accurate and efficient inference for transformers. In 2024 IEEE Symposium on Security and Privacy (SP), pages 133-133, Los Alamitos, CA, USA, may 2024. IEEE Computer Society.
[25] Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. Cryptonets: Applying neural networks to enerypted data with high throughput and accuracy. In International conference on machine learning, pages 201-210. PMLR, 2016.
[26] Chiraag Juvekar, Vinod Vaikuntanathan, and AnanthaP. Chandrakasan. GAZELLE: A low latency framework for secure neural network inference, Jan 2018.
[27] Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding. Cheetah: Lean and fast secure (Two-Party) deep neural network inference. In 3/st USENIX Security Symposium (USENIX Security 22), pages 019-826,777 2022.
[28] Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and RalucaAda Popa. Delphi: A cryptographic inference service for neural networks, Jan 2020.
[29] Junfeng Fan and Frederik Vercauteren. Somewhat practical fully homomorphic encryption. Cryptology ePrint Archive, 2012.
[30] Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow2: Practical 2-party secure inference. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 325-342, 2020.
[31] Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Kui Ren, Cheng Hong. Tao Wei, and WenGuang Chen. Bumblebee: Secure two-part inference framework for large transformers. Network and Distributed System Security (NDSS), 2025.
[32] Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference transformers. In Advances in Neural Information Processing Systems, 2022.
[33] Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nishanth Chandran, and Aseem Rastogi. Simn: A math library for secure rnn inference. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1003-1020. IEEE, 2021.
[34] Oded Goldreich. Secure multi-party computation. Manuscript. Preliminary version, 78(110):1-
108, 1998.
[35] Hai Huang and Yongjian Wang. Secbert: Privacy-preserving pre-training based neural network inference system. Neural Networks, 172:106135, 2024.
[36] Jinglong Luo, Yehong Zhang, Zhuo Zhang, Jiaqi Zhang, Xin Mu, Hui Wang, Yue Yu, and Zenglin Xu. Secformer: Fast and accurate mrivacy-rreeevvng inference for transformer models via smpe. In Findings of the Association for Computational Linguisties ACL. 2024, pages 13333-13348, 2024.
12

---

[37] Chenkai Zeng, Debiao He, Qi Feng, Xiaolin Yang, and Qingcai Luo. Securegpt: A frame- work for multi party presevvii transformer inference in gpt. IEEE Transactions on Information Forensics and Security, 2024.
[38]
Tianshi Xu, Lemeng Wu, Runsheng Wang, and Meng Li. Privcimet: Efficient private inference via block circulant transformation. Neural Information Processing Systems (NeurIPS), 2024.
[39]
Jiawen Zhang. Xinpeng Yang, Lipeng He, Kejia Chen, Wen jie Lu, Yinghao Wang. Xiaoyang Hou, Jian Liu, Kui Ren, and Xiaohu Yang. Secure transformer inference made non-interactive. Network and Distributed System Security (NDSS), 2025.
[40] Dongjin Park, Eunsang Lee, and Joon-Woo Lee. Powerformer: Efficient srivacy-preserving transformer with batch rectifier-power max function and optimized homomorphic attention. Cryptology ePrint Archive, 2024.
[41] Jungho Moon, Dongwoo Yoo. Xiaoqian Jiang, and Miran Kim. THOR: Secure transformer inference with homomorphic encryption. Cryptology ePrint Archive, Paper 2024/1881, 2024.
[42] Kanav Gupta, Neha Jawalkar, Ananta Mukherjee, Nishanth Chandran, Divya Gupta, Ashish Pan- war, and Rahul Sharma. Sigma: Secure gpt inference with function secret sharing. Cryptology ePrint Archive, 2023.
[43] Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Chen. Puma: Secure inference of Ilama-7b in five minutes. arXiv preprint arXiv: 125333 2023.
[44] Yoshimasa Akimoto, Kazuto Fukuchi, Youhei Akimoto, and Jun Sakuma. Privformer: Privacy- preserving transformer with mpe. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P), pages 592-410. IEEE, 2023.
[45] Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P Xing, and Hao Zhang. Mpeformer:
fast, performant and private transformer inference with mpc. arXiv preprint arX1v22211.01452. 2022.
[46] Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren. Ciphergpt: Secure two-party gpt inference. Cryptology ePrint Archive, 2023.
[47] Junming Ma, Yancheng Zheng, Jun Feng. Derun Zhao, Haoqi Wu, Wenjing Fang, Jin Tan, Chaofan Yu, Benyu Zhang, and Lei Wang. SecretFlow SPU: A performant and Friendliy framework for Privacy-Preservicg machine learning. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 17-33, Boston, MA, July 2023. USENIX Association.
[48] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian. Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. ArXiv, abs/2401 .04088, 2024.
13

---

Expert Frequency Distriburion for Different Datasets
0.8
Frequency
0.44
Expert Index
Figure 10: Expert frequency distribution of the 10th layer of DeepSeekMoE-168 16B for text understand- ing tasks and mathematical reasoning tasks.
Expert Frequency Distribution for Different Datasets
mathga
arc challange
hallswoo
winogrande
social iqu
openbookop
Expert Index
Figure 11: Expert frequency distribution of the 10th layer of DeepSeckMoE-16B for 10 different datasets.
A Expert activation patterns
Figure 10 shows the expert activation patterns in the 10th layer of DeepSeekMoF--668 [8] when processing text understanding versus mathematical reasoning tasks. Text understanding tasks include SIQA [13]. OBQA [17]. BoolQ [11], ARC-easy, ARC-challenge [16], HellaSwag [14], PIQA [12] and WinoGrande [15]. Mathematical reasoning tasks comprise GSM8K [9] and MathQA [10]. Expert #2 is disproportionately activated for text understanding, while Experts #3 and #60 show higher activation for mathematical reasoning.
The detailed expert activation patterns of these datasets are shown in Figure 11.
B Secure Dispatch and Combination Protocols
We detail the secure token dispatch protocol Hddispathh Algorithm and secure combine protocol Ilmombine in Algorithm 2 in this section.
C Detailed Comparison with CipherPrune
CipherPrune proposes a secure pruning protocol that uses oblivious swaps to iteratively move pruned tokens to the end of the sequence, leveraging OT-based secure comparisons and binding masks to
14

---

Algorithm 1: Secure Dispatch Protocol Iddisptch
Input: Po. Pi hold secret shares of input token embeddings [x]] Zmxd and routing information (including routing expert indices and scores [K]] [W] E ==**), where m is the number of input tokens, d the hidden dimension, and is the number of activated experts per token,
Output: Po. Pi learn the secret shares of dispatched token embeddings for each expert
[ E gixd, where is the number of dispatched tokens for each expert, and is the number of experts.
1: for each expert i in [0, n 1] do
2: Flatten [A]. [W] into the shape kmm
3: Invoke squulal with imput [K] and and set output as boolean mask [M.] B E Z where
M, = 1 []] = i}_ Vij E [0, km 1].
4:
Invoke 1m with input [[,]] and [W]. and learn the token priority scores [S.]. where -. = - W []] Vj E km - 1].
5:
Invoke [[
= t) to obtain selected token indices and scores [ [ E Z'.
6:
Convert [K,] original token indices via integer division by k: [ |
7:
Invoke unchon [ m) to obtain one-hot matrix [[onehot( K']] E Z where
onchot(K3([i] = = = k} Vj E [0, - 1], & E [0,m 1].
8: Compute [[X]] Zx [onehot(K]]] x [x]] using HE protocol mammul to retrieve t
dispatched token embeddings for expert i.
9: end for
Algorithm 2: Secure Combine Protocolllcombin
Input:
Po. Pi hold secret shares of each expert's output [VE, E gtxd, selected top-t token scores for each expert [s]] E Z' and the computed one-hot matrix for selected token ID for each expert (onchot(K()] m- E 2 where m is the number of input tokens, d is the hidden dimension, and is the number of activated experts per token.
Output: Po. P1 learn the secret shares of MoE layer output [[v] E mmxd
do 2: Use mm ([onhhot K()] to get onehot matrix [onehot( K])J² E g
for each expert i in [0,
3: Invoke Immi with input (onchot(K<]]" and [S[] to compute scored onehot matrix
[R.] E gm where R, [.]]]]]] mehho((K))i]]e S](k),, Vj E [0, 1]. [0, 1].
4: Compute [vE,] E gmxd [R:]] [ye]] using HE protocol mm compute reorderd
scored tokens for expert i.
5: end for
6: Sum over expert contributions to the final MoE layer output [y] + C
Use the idea of CipherPrune for Taken Dispatch: Comm. Complexity: (n kmndd)
[X
Bind Scores
Secure Companes and Swaps
Truncate Tokens
(txd)
and Tokens
B
B
and Scores
token embedcling [k]
All tokens to be diepatched (km x
CryptoMoE Token Dispatch Protocol: Comm. Complexity: (nkm log(km) + mtm)
valuea comesponding token
Dupk
Tokan Priority Scores [Si]
Top-t Token ID [K,] Selected Taken IDe [K,]
(onchor(R())
token embeddings [X]]
protocol
(ms d:
Input Tokan Embeddings [x]
Secure protocol in HE
Figure 12: Comparison between CipherPrune and Our Protocol. This figure is an example for 4, m 2, = 1. We modified CipherPrune's 's mask binding strategy into score binding strategy to accommodate the requirements of MoE inference.
tokens for efficiency. Although this approach achieves secure pruning with linear complexity in the
15

---

62
58.81
59.80
60.18
60.42
62
59.60
60.24
59.95
61
61
1.5
61.40
61.83
62.01
62.13
60
61.18
61.45
61.56
61.51
60
59
61.92
62.44
62.56
62.84
61.56
62.01
61.76
58
59
57
62.76
62.80
62.85
63.01
62.15
62.00
62.06
62.00
58
56
0.5K
1K
2K
4K
0.5K
1K
2K
4K
Sequence Length
Sequence Length
OLMoE-13BB/6998
QWenMoE-2.7B/43338
Figure 13: Effect of and sequence length average accuracy for OLMoE and QWenMoB. The original average accuracy 62.96% for OLMoE and 62.00% for QWenMoE.
number of swaps (0mp) for p pruned tokens among m candidates), applying directly MoE token dispatch incurs significant overhead. Specifically, for each expert, dispatching t tokens to each of i experts from km candidate tokens requires (krmrt secure swaps on d-dimensional embeddings, resulting in (nkmmdd) communication. Figure 12 shows toy example of this process. Reordering in token combination after expert computation further doubles this cost. Experiments reveal that naively adopting CipherPrune introduces 82% latency overhead in privacy-presevving MoE inference.
In contrast, our protocol decouples token index selection from token embedding manipulation, eliminating expensive secure swaps on large dimensions. For token dispatch, protocol disspath computes Hopk to select t tokens per expert and Honchex encode selection masks, with complexity Onkmllog (km) + rtmr). Token embeddings are then aggregated via HE-based matrix multipicaation mmmmmal avoiding d-dimensional swaps. Immammll incur limited communication in HE-SS conversions, and the majority of computation is done by parallelizable HE operations.
For token combination, protocol TTmmbine reuses the selection masks from dispuich invert the dispatch process via another mammul. achieving reordering without extra secure comparisons or swaps. This reduces the latency of token combination to 1% of the total runtime.
Experiments demonstrate that our protocol introduces only 18% overhead. 4.7x improvement over CipherPrune, mainly due to replacing secure swaps with efficient HE-based linear operations. This design proves particularly advantagcous for values of d and make communication efficiency critical.
MoE model inference, where large
D Complexity Analysis of Batch MatMul Packing
Given sequence of ciphertext inputs (A))=1 Zixd and plaintext weights (B.)==) Zd xd2, 1- where C₁ A; B;. Each ciphertext pack N
aim to compute ciphertext results {C elements.
In the original BOLT [24] packing method, each matrix A, is packed column- with single N ciphertext holding т columns. To accumulate results across columns, N. = O ( rotations are required per ciphertext. For all n matrix multiplications, the total number of ciphertexts is O (mnd)) ) leading to overall rotation cost of O nedd) O (nd1).
contrast, our batch MatMul packing method packs columns from all A, matrices, allowing each ciphertext to store at columns (with each column containing nt entries from A,)-)) This reduces the required rotations per ciphertext to (AN While the total number of ciphertexts remains O (nid) O (d1).
the total rotation complexity drops to O ntdd N
16

---

Integrate stby-step Giant-step (BSGS) strategy into Batch MatMul packing, The BSGS algo- rithm is commonly used for ciphertext- MatMul reduce HE rotations [24], decomposing rotations into local (baby-step) and global (giant-step) phases. This reduces the number of rotations
per-MatMul from (d1) to O td N
The total number of rotations is O
dd.d2
By
N
combining BSGS with batch packing, the effective parallel token dimension increases from to vl, since columns from all n matrices are processed in parallel. This reduces the overall rota-
tion complexity O rttd, de
achieving vn x improvement compared applying BSGS
independently to each matrix multiplication.
E Ablation Study on and Sequence Length
Figure 13 presents the ablation study on the impact of t and input sequence length for both OLMoE and QWenMoE models. Consistent with Section 5.4, increasing either the sequence length improves accuracy. Additionally, QWenMoE exhibits better load balancing, resulting smaller accuracy degradation with shorter sequences.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: /
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
There doesn't 't exist theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experimental setup in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer:
[No]
Justification:
We are arranging our code and will be open-sourced once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt / assssnolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide detailed experimental setup in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiments focus inference, and as such, the results remain consistent across multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide detailed experimental setup in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpas//////../...///e//ee wttppppp.cccpppeeateeee httppp/bccethhccccueeeeeee..eeee ctwuidelelllllleeccccccc ines?
Answer: [Yes]
Justification: /
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: /
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: /
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: /
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: /
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: /
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: /
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: /
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https / 220 for what should or should not be described.
24


Reviews:
[{"review_id": "z19UR1mwSv", "reviewer": "Reviewer_NVGY", "summary": "This paper introduces CryptoMoE, the first cryptographic framework for private and efficient inference on Mixture-of-Experts (MoE) LLMs. MoE architectures dynamically route input tokens to different experts based on content, which can unintentionally leak sensitive information. CryptoMoE addresses this challenge by introducing Inference-Time Balanced Expert Routing—a privacy-preserving mechanism that ensures expert activation patterns are input-independent. To maintain accuracy, it uses a confidence-aware token selection and develops secure protocols for expert dispatch and combination. Additionally, it introduces a batch matrix multiplication protocol to improve the efficiency of homomorphic encryption operations. Evaluations on MoE models show 3× reduction in latency and communication cost, outperforming both dense and pruned baselines.", "questions": "1. Could you clarify how to implement the dense baseline and what protocols are used to achieve it?\n\n2. Could you explain why CryptoMoE outperforms the dense baseline, considering that multiple expensive non-linear operations like onehot and topk are involved in CryptoMoE?", "limitations": "no", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n+ Support private inference for MoE-based LLMs\n+ Protect expert routing information with private dispatch and combination\n+ Efficient protocols for batched matrix multiplications\n+ 2.8–3.5× latency reduction and 2.9–4.3× communication reduction\n\nWeaknesses\n\n- The protocols, including dispatch and combination, are designed using existing building blocks.\n- It is unclear how to implement the dense baseline and what protocols are used to achieve it.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your support and constructive comments, which have greatly helped us improve our work! See below for the answers to your questions and comments.\n\n-----\n\n**[To Q1 & W2: Details on the dense baseline implementation]**\n\nWe provide a detailed implementation of the **dense baseline** below, which complements the brief description in Section 3 of our paper. The dense baseline evaluates **all experts for each token**, thereby fully hiding the routing information. The core steps are as follows:\n\n- **Routing and Score Matrix Construction:** After the gate routing step, we obtain the expert indices and scores $K, W \\in \\mathbb{Z}^{m \\times k}$, where $m$ is the number of tokens and $k$ is the number of selected experts per token.  We then convert $K$ into a one-hot representation as follows: $K_{\\text{onehot}} = \\text{onehot}(K) \\in \\mathbb{Z}^{m \\times k \\times n}$, where $n$ is the total number of experts. Within the $n$-dimension of $K_{\\text{onehot}}$, only the entries corresponding to the selected expert indices are set to 1, and all others are 0. We then compute the expert score matrix: $S = W \\times K_{\\text{onehot}} \\in \\mathbb{Z}^{m \\times n}$, where the multiplication reduces along the $k$-dimension, and $m$ remains the parallel axis. Each row $S_i$ reflects the expert scores for the $i$-th token: the positions corresponding to selected experts contain their routing scores, while the rest are 0.\n- **Expert Computation:**  Each expert $E_i$ computes on **all** tokens: $y_{E_i} = E_i(x) \\in \\mathbb{Z}^{m \\times d}$, aggregating all expert outputs gives: $y_E \\in \\mathbb{Z}^{n \\times m \\times d}$. \n- **Combination:**  Finally, we compute the output of the MoE layer as: $y = S \\times y_E \\in \\mathbb{Z}^{m \\times d}$, where the multiplication reduces over the expert dimension $n$, and $m$ remains the parallel dimension. This operation essentially computes a weighted sum over expert outputs for each token.\n\nThe corresponding privacy-preserving inference algorithm is described below:\n\n**Algorithm: Dense Baseline**\n **Input:** Secret-shared token embeddings $[\\\\![x]\\\\!] \\in \\mathbb{Z}^{m \\times d}$, with $m$ tokens and $d$-dimensional embeddings; $k$ is the number of selected experts.\n **Output:** Secret shares of MoE output $[\\\\![y]\\\\!] \\in \\mathbb{Z}^{m \\times d}$.\n\n1. Use $\\Pi_{\\text{matmul}}$, $\\Pi_{\\text{softmax}}$, and $\\Pi_{\\text{topk}}$ to perform secure routing and obtain $[\\\\![K]\\\\!], [\\\\![W]\\\\!] \\in \\mathbb{Z}^{m \\times k}$.\n2. Compute outputs for all experts: $[\\\\![y_{E_i}]\\\\!] = E_i([\\\\![x]\\\\!]) \\in \\mathbb{Z}^{m \\times d}$ for $i \\in [0, n-1]$; reshape to $[\\\\![y_E]\\\\!] \\in \\mathbb{Z}^{n \\times m \\times d}$.\n3. Apply $\\Pi_{\\text{onehot}}([\\\\![K]\\\\!], c=n)$ to obtain one-hot routing matrix $[\\\\![\\text{onehot}(K)]\\\\!]^B \\in \\mathbb{Z}^{m \\times k \\times n}$.\n4. Compute expert score matrix:\n   $[\\\\![S]\\\\!] \\leftarrow \\Pi_{\\text{matmul}}([\\\\![W]\\\\!], [\\\\![\\text{onehot}(K)]\\\\!]^B) \\in \\mathbb{Z}^{m \\times n}$.\n5. Compute MoE output:\n   $[\\\\![y]\\\\!] \\leftarrow \\Pi_{\\text{matmul}}([\\\\![S]\\\\!], [\\\\![y_E]\\\\!]) \\in \\mathbb{Z}^{m \\times d}$.\n\n------\n\n**[To Q2: Why CryptoMoE outperforms the dense baseline]**\n\nAs shown in the ablation study in **Figure 8**, CryptoMoE significantly reduces the cost of expert computation compared to the dense baseline. In the dense baseline, each expert processes all $m$ tokens, leading to a high computational cost. In contrast, $\\text{CryptoMoE}\\_{t=2.0}$ reduces this to $2mk/n$ tokens per expert. Taking DeepSeekMoE-2.8B/16.4B as an example, with $m=128$, $k=6$, and $n=64$, each expert processes only $2 \\times 128 \\times 6 / 64 = 24$ tokens, whereas the dense baseline requires computing all $128$ tokens. CryptoMoE saves **over $500\\\\%$** of the expert computation cost, which is the main bottleneck in the MoE block. Although CryptoMoE introduces additional operations such as Top-K and one-hot encoding, these account for **only 18%** of the total LAN latency, as shown in Figure 8. Therefore, the savings in the expert layers far outweigh the cost in $\\Pi_{\\text{dispatch}}$ and $\\Pi_{\\text{combine}}$.\n\n------\n\n**[To W1: Dispatch and combination protocols are designed using existing building blocks.]**\n\nOur $\\Pi_{\\text{dispatch}}$ and $\\Pi_{\\text{combine}}$ protocols are built upon well-established cryptographic primitives [r46]. **Our key contributions lie in the token manipulation strategy designed specifically for secure MoE inference**, which enables secure token-to-expert assignment and result aggregation **without leaking any sensitive information**. The underlying cryptographic primitives, such as comparison protocols, are not our focus.\n\nTo highlight the novelty and effectiveness of our $\\Pi_{\\text{dispatch}}$ and $\\Pi_{\\text{combine}}$ protocols, we compare them with the protocol used in CipherPrune [r22]. Prior works [r22, r24] rely on MPC-based sorting that directly swaps **high-dimensional token embeddings**, resulting in significant communication and latency cost. This challenge is especially pronounced in MoE blocks, where sorting is required in both the dispatch and combine phases for each expert.\n\nIn contrast, **CryptoMoE eliminates the need for direct manipulation of embeddings** by decoupling token indices from embeddings. This design significantly reduces the communication cost. The table below compares CryptoMoE and CipherPrune on a single MoE block of DeepSeekMoE under a LAN setting, where $m$, $n$, $k$, $t$, and $d$ represent the number of input tokens, number of experts, number of selected experts, dispatched tokens per expert, and embedding dimension, respectively. The evaluation is conducted using the following parameters: $m=128$, $n=64$, $k=6$, $t=12$, $d=2048$.\n\n| Method      | Communication complexity | Dispatch Communication (GB) | Dispatch latency (s) | Combine Communication (GB) | Combine latency (s) |\n| ----------- | ------------------------ | --------------------------- | -------------------- | -------------------------- | ------------------- |\n| CipherPrune | $nkmtd$                  | 9.67                        | 90.8                 | 6.44                       | 52.6                |\n| Ours        | $nkm\\log(km)+ntm$        | 0.89                        | 34.4                 | 0.72                       | 3.0                 |\n\nMore detailed analysis is provided in **Appendix D** in our paper.\n\n------\n\n**References**\n\n- \\[r22, r24, r46] represent the 22th, 24th, 46th references cited in our original submission.\n\n----\n\n**We sincerely appreciate the professional, detailed advice from reviewer NVGY! We hope this response fully addresses your concerns.**"}, {"review_id": "iOWAlgsrIq", "reviewer": "Reviewer_SQLk", "summary": "This work propose an framework to securely and efficiently process the MoE layer in modern LLM models via HE+2PC. With proposed confidence-aware dispatching, it reduced the computation and communicated overhead compared to fully-protected baseline-dense. With multiple-token batched into single ciphertext, it reduces the rotation complexity.", "questions": "1.When performing the evaluation, at which point the data communicate back to client? I think this part should be added with some figures or algorithms to clearly show the overall flow. For example, I guess the highest communication overhead of baseline-dense comes from the   all-tokens sent to each expert, which led to tremendous intermediate results. It would be better to illustrate this in details.\n\n2.I think a detailed configurations for HE should be included in setup part. \n\n3.To better understanding the dispatch protocols, I think I need authors to help me understand the functions of m secret-shared tokens. By splitting the tokens into m shares, it seems that these share could compute with each expert in weighted-sum manner. Does this for accuracy improve? I am not sure why the tokens are secret-shared?\n\nI hope authors could address my concerns, thanks!", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This is an very interesting research problem and still remains un-explored, in which sensitive info requiring protection.\n2. With a index-obfuscating style method, it balance the privacy leakage and computation efficiency.\n\nWeakness:\n1. Missing some HE parameter setting, I think it is important for readers to know due to it is highly related with HE computation latency.\n2. Missing a detailed general HE computation protocol used in evaluation, including the computation and communication illustration together.", "was_revised": true, "final_justification": "Considering author's detailed rebuttal, I will keep my score for recommendation.", "rebuttal": "Thank you very much for your strong support and the thorough and creative comments! We also appreciate your evaluation of our work as “a very interesting research problem.” Please find our responses to your questions below.\n\n----\n\n**[To Q1: Details on the framework’s communication flow]**\n\nOur framework adopts the **hybrid HE/MPC scheme**, a widely used approach in recent secure ML research [r24, r25, r26, r27, r28]. In this scheme, Homomorphic Encryption (HE) is used to efficiently handle linear layers, while Multi-Party Computation (MPC)—including secret sharing (SS) and Oblivious Transfer (OT)—is employed for non-linear operations. This hybrid design involves communication in the following two main cases:\n\n- **HE–SS Conversion**: Communication arises at the interface between linear and non-linear layers, where HE ciphertexts must be converted to/from SS form.\n- **MPC Protocol Execution**: Communication is inherent to executing MPC protocols for non-linear functions. In particular, Oblivious Transfer (OT), a core building block for non-linear computation, requires interactive communication between the client and the server.\n\nFigures 4 and 5 in our manuscript illustrate how HE and MPC protocols are integrated into our dispatch and combine steps. During expert computation, the SiLU activation is performed via MPC, while the surrounding linear matrix multiplications are handled by HE.\n\nYour analysis of the dense baseline is correct. The dense baseline involves substantial communication cost because **each expert processes all $m$ tokens**, producing a large volume of intermediate results. These results then require **conversion between HE and SS representations**, causing significant HE ciphertext transmission. Additionally, the dense computation increases both linear and non-linear processing and communication costs.\n\n------\n\n**[To Q2 & W1: HE parameter settings]**\n\nWe follow the configuration of Bumblebee for HE parameters and set $N=8192$, $\\log q=114$, and $\\log p=64$, where $N$ is the polynomial degree, and $q$, $p$ are the ciphertext and plaintext moduli, respectively. For secret sharing, we adopt a 64-bit ring.\n\n------\n\n**[To Q3: Explanation on the dispatch protocol]**\n\nThe primary function of secret sharing in our protocol is to ensure data privacy: neither the client nor the server can access the full data independently. As shown in Figure 4, we process $m$ tokens of hidden dimension $d$, resulting in data of shape $m \\times d$. This data is converted into secret shares to maintain privacy. Throughout our protocol, all intermediate results are represented in secret-shared form.\n\nRegarding your suggestion to pre-split tokens into $m$ shares for weighted-sum computation at each expert—this is a very insightful idea. However, due to two critical constraints:\n\n- Each expert applies a SiLU non-linear activation, which makes pre-aggregation of experts' weights infeasible.\n- The expert weights must also remain private, and thus cannot be revealed or used to compute a public weighted sum.\n\nAs a result, we must design a secure $\\Pi_{\\text{dispatch}}$ and $\\Pi_{\\text{combine}}$ protocol based on secret sharing. Your idea would be well-suited to **purely linear pipelines**, where pre-aggregation of weights can be applied in a reparameterization-style approach as in [a1].\n\nWe hope this clarifies the necessity and role of secret-shared tokens in our protocol.\n\n------\n\n**[To W2: General HE computation protocol]**\n\nFor linear layer computations, we adopt the MatMul protocol described in Section 4.1.1 of BOLT [r24]. Furthermore, for matrix multiplication within the MoE block, we apply an additional **batch optimization** on top of the BOLT protocol. We will incorporate a more detailed explanation of the BOLT-based HE protocol in the appendix—thank you for the suggestion!\n\n------\n\n**[To formatting concerns]**\n\nThank you for pointing out the formatting of $\\bf{1}\\\\{\\mathcal{P}\\\\}$ in the first paragraph of Section 2. We use $\\bf{1}\\\\{\\mathcal{P}\\\\}$ to denote an indicator function, which returns 1 when condition $\\mathcal{P}$ is true and 0 otherwise. For instance, ${\\bf{1}}\\\\{x == 0\\\\}$ evaluates to 1 if $x = 0$, and 0 otherwise. We will revise the notation for better clarity and avoid confusion in the final version.\n\n------\n\n**References**\n\n- [r24, r25, r26, r27, r28] correspond to references 24 through 28 in our original submission.\n- [a1] Ding, Xiaohan, et al. *RepVGG: Making VGG-style ConvNets Great Again*. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.\n\n----\n\n**Thank you again for your strong support and the valuable comments, which have strengthened and made our paper more robust! We will add all of these discussions to our final version.**"}, {"review_id": "3vLO42U8Iu", "reviewer": "Reviewer_8zVq", "summary": "The authors mainly address the inefficiency of performing MoE securely. To reduce the number of tokens dispatched to experts, they select the top-k tokens for each expert based on confidence scores. For efficient execution, they propose new protocols for dispatching, combining, and batch matrix multiplication. In the top-k selection, they reduce complexity by avoiding secure swaps compared to the protocol used in CipherPrune. For the combine protocol, they efficiently aggregate the results by reusing the one-hot matrix. Finally, by applying column-packing to the combined input matrix, they reduce the number of required rotations in batch matrix multiplication. Experimental results demonstrate that the proposed method achieves a 2×–4× improvement in efficiency.", "questions": "Please refer to the Weaknesses section.", "limitations": "The authors adequately discuss the limitations in Section 6.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* This work is the first to identify the inefficiencies that can arise when performing MoE securely.\n* The authors achieved a 2×–4× speedup by using confidence-based selection to mitigate the inefficiencies in naively performing MoE securely.\n* The proposed protocol is well-designed for the given problem.\n* The effectiveness of the proposed method was validated on large language models.\n\n---\n\n### Weaknesses\n* In Section 4.4, Bolt utilizes the Baby-Step Giant-Step (BSGS) algorithm to reduce computational complexity when the polynomial dimension is large, but this aspect has not been considered in the paper. It is necessary to analyze whether the proposed method is also compatible with BSGS. If not, a comparison with Bolt incorporating BSGS should be provided.\n* Additionally, prior to the MoE operation, the ciphertexts are likely structured in a column-packed format without sample mixing. With the proposed method, repacking of column-packed ciphertexts would be necessary, but the analysis of this additional cost appears to be missing.\n* The description of the experimental setup for the end-to-end comparison is insufficient. The experiments were conducted on a very large model, yet the reported communication cost and speed seem relatively low. It is unclear whether only the MoE component was computed securely while the remaining parts were processed in plaintext. If the entire model was executed securely, it would be helpful to provide detailed explanations on how non-linear operations and matrix multiplications in the attention mechanism were handled, including which methods or protocols were employed.\n\nTypos\n* \"Threat Model\" is miswritten as \"Thread Model\" in line 99.", "was_revised": true, "final_justification": "The problem is well-specified, and the proposed method is novel. Therefore, I keep my score as 4: borderline accept.", "rebuttal": "We thank Reviewer 8zVq for your support and for the professional and detailed feedback! Below is our response.\n\n-----\n\n**[To W1: Compatibility with BOLT BSGS]** \n\nOur batch MatMul packing strategy is **compatible with the Baby-Step Giant-Step (BSGS)** algorithm. As detailed in **Appendix E** of our paper, we provide a complexity analysis of integrating BSGS with our packing. Specifically, BSGS combined with batch packing reduces the number of required rotations from $O(n\\sqrt{td_1 d_2/N})$ to $O(\\sqrt{ntd_1 d_2/N})$, resulting in a $\\sqrt{n} \\times$ improvement in rotation complexity.\n\n-----\n\n**[To W2: Cost of repacking for batch MatMul packing]** \n\nWe would like to clarify that both the input and output of our dispatch and combine protocols are in **secret share (SS) form**, not HE ciphertext. As shown in Algorithms 1 and 2 in Appendix C, under the SS form, the repacking operations can be performed **locally and in plaintext**, incurring **negligible cost**. Thus, our batch MatMul format does **not** introduce extra packing cost in the HE computation.\n\n-----\n\n**[To W3: Setup details for end-to-end evaluation]** \n\nAll components of our Transformer model—including RMSNorm, attention, and MoE blocks—are evaluated **securely under our HE/MPC hybrid framework**, ensuring **end-to-end privacy**. The reported inference time and communication cost in **Table 3** are **amortized per-token costs**, i.e., total cost divided by the number of input tokens. When considering this, our results are consistent with prior works such as BOLT [r24] and Bumblebee [r31]. We include a table below summarizing the per-token cost across BOLT and Bumblebee for clarity.\n\n| Work            | Model                  | Bandwidth (Gbps) | Ping latency (ms) | Latency (min/token) | Comm. (GB/token) |\n| --------------- | ---------------------- | ---------------- | ----------------- | ------------------- | ---------------- |\n| Ours            | DeepSeekMoE 2.8B/16.4B | 3                | 0.2               | 0.76                | 2.46             |\n| Ours            | OLMoE 1.3B/6.9B        | 3                | 0.2               | 0.36                | 1.31             |\n| Bolt [r24]      | BERT-base (0.1B)       | 3                | 0.8               | 0.024               | 0.47             |\n| BumbleBee [r31] | LLaMA-7B               | 1                | 0.5               | 1.73                | 0.85             |\n| BumbleBee [r31] | BERT-large (0.3B)      | 1                | 0.5               | 0.052               | 0.16             |\n\nWe would like to emphasize that prior works such as BOLT and Bumblebee focus on supporting dense models (e.g., BERT, LLaMA-1). In contrast, our core contribution lies in **proposing the first framework that supports privacy-preserving inference for MoE models**. For other components, such as RMSNorm and attention, we **directly adopt the protocols from prior works BOLT and Bumblebee**. Specifically, for matrix multiplications in linear layers, we leverage BOLT’s protocol, and for nonlinear operations, including RMSNorm, Softmax, and SiLU, we employ the protocols proposed in Bumblebee. We acknowledge that these aspects are not the primary focus of our work.\n\nFurthermore, as shown in **Figure 8**, when the total number of input tokens is relatively small (e.g., typically under 256 tokens), **the MoE layer becomes the dominant bottleneck** in both LAN and WAN settings. This observation further highlights our key contribution: **enabling efficient private inference for the MoE module**.\n\nWe will incorporate this clarification in the revised version to avoid potential confusion. We hope this addresses your concerns and provides a clearer understanding of the scope and focus of our design and performance contributions.\n\n**[To Typos]**\n\nThank you for pointing out that \"Threat Model\" was misspelled as \"Thread Model\".  We will correct this in our final version.\n\n**References**\n\n- \\[r24, r31] represent the 24th, 31th references cited in our original submission.\n\n----\n\n**Last, thanks so much for helping us improve this work through your professional perspective!**"}, {"review_id": "4bxPdpSqAu", "reviewer": "Reviewer_d2x4", "summary": "This paper introduces CryptoMoE, the first cryptographic framework that enables privacy-preserving inference for Mixture-of-Experts (MoE) large language models. While prior private inference systems focus on dense architectures, CryptoMoE innovatively addresses the privacy risk posed by dynamic expert routing in MoE layers, which are highly input-dependent and thus leak-prone. The authors propose a novel balanced expert routing protocol and corresponding secure dispatch and combine operations under hybrid HE/MPC.", "questions": "* Have you benchmarked your Batch MatMul protocol against Nexus or CipherGPT under the same threading and HE parameter settings?\n\n* Can you provide or commit to releasing code to allow independent verification?", "limitations": "* Given the significant performance improvements claimed over prior secure inference work, the lack of code release or detailed hardware configurations could hinder validation and adoption.\n\n* While the paper acknowledges WAN latency and short-sequence inefficiencies, it omits important concerns around reproducibility, assumptions on the adversarial model, and multi-party scalability.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* First to tackle private MoE inference under secure computation. The routing problem is well-motivated and practically significant.\n* Demonstrates strong empirical results, retaining accuracy while substantially improving efficiency compared to both insecure and dense baselines.\n\n**Weaknesses**\n* The matrix multiplication only compares the BOLT w/o BSGS version. Compared with the SOTA bumblebee[1] and the more suitable nexus [2] and ciphergpt [3] for batch matrix multiplications, there is no comparison, lacking novelty and effectiveness.\n* Under 48 threads, the inference performance shown in the paper far exceeds that of sota's work [1,2,3], and no open source link is provided. Perhaps I need to check the code to eliminate my concerns.\n\n**Reference**\n\n[1] BumbleBee: Secure Two-party Inference Framework for Large Transformers\n\nhttps://www.ndss-symposium.org/wp-content/uploads/2025-57-paper.pdf\n\n[2] Secure Transformer Inference Made Non-interactive\n\nhttps://www.ndss-symposium.org/wp-content/uploads/2025-868-paper.pdf\n\n[3] CipherGPT: Secure Two-Party GPT Inference\n\nhttps://eprint.iacr.org/2023/1147.pdf", "was_revised": true, "final_justification": "After reading the author's rebuttal, I gave this rating.", "rebuttal": "We thank Reviewer d2x4 for your valuable and detailed feedback!  Below we list our responses to each of the comments:\n\n-----\n\n**[To Q1 & W1: Comparison with BOLT (BSGS), Bumblebee, Nexus and CipherGPT on batch MatMul]**\n\nOur batch MatMul packing strategy is **fully compatible with the Baby-Step Giant-Step (BSGS)** algorithm. As analyzed in **Appendix E**, we provide a detailed complexity analysis for integrating BSGS with our batch packing method. Specifically, BSGS combined with batch packing reduces the number of required rotations from $O(n\\sqrt{td_1 d_2/N})$ to $O(\\sqrt{ntd_1 d_2/N})$, yielding a **$\\sqrt{n} \\times$ reduction** in rotation complexity. In our implementation, we **consistently apply the BSGS optimization**. For clarity, Figure 6 in our paper reports results **without** BSGS, but Appendix E covers both settings.\n\nIn the table below, we compare our batch MatMul protocol against existing approaches, including BOLT [r24], NEXUS [r37], and CipherGPT (**VOLE-based**) [r44]. All evaluations are conducted on a machine equipped with an **Intel Xeon Platinum 8468 CPU (48 cores @ 2.1GHz, 64 threads)** under a LAN setting. Given ciphertext matrices $\\\\{A_i\\\\}\\_{i=0}^{n-1} \\in \\mathbb{Z}^{t \\times d_1}$ and plaintext weights $\\\\{B_i\\\\}_{i=0}^{n-1} \\in \\mathbb{Z}^{d_1 \\times d_2}$, we compute $C_i = A_i \\times B_i$ for $n = 64$, $t = 24$, $d_1 = d_2 = 2048$. These dimensions are derived from the DeepSeekMoE-2.8B/16.4B models.\n\nWe follow the HE parameter settings used in Bumblebee, namely: $N = 8192$, $\\log q = 114$, and $\\log p = 64$, where $N$ is the polynomial degree, and $q$, $p$ are the ciphertext and plaintext moduli, respectively. For secret sharing, we adopt a 64-bit ring.\n\n| Method    | Comm. (GB) | \\#Rotation (Key-switch) | Latency(s) |\n| --------- | ---------- | ----------------------- | ---------- |\n| Bolt      | 0.482      | 15360                   | 62         |\n| Nexus     | 0.482      | 8388480                 | 3216       |\n| CipherGPT | 106.5      | /                       | 2421       |\n| Ours      | 0.482      | 1024                    | 21         |\n\nThe communication cost and computational latency for CipherGPT are estimated following their paper, as their code is not open-source. It is important to note that the MatMul protocols proposed by CipherGPT and Nexus are optimized for scenarios involving **different ciphertext input matrices $A_i$ multiplied by the same plaintext weight matrix $B$**. However, in the batch MatMul of the MoE block, **different experts process distinct ciphertext tokens with distinct weight matrices**. This means we cannot leverage the batch amortizable properties that CipherGPT and Nexus offer, resulting in their method incurring higher latency.\n\nThe results show that our batch matrix multiplication protocol, **specifically optimized for MoE layers**, performs significantly better than the methods discussed above.\n\n----\n\n**[To Q2 & W2 & Limitation 1: Concerns about performance and reproducibility]**\nWe clarify that the reported inference time and communication cost in **Table 3** are **amortized per-token costs**, i.e., the total cost divided by the number of input tokens. **When considering this**, our results are consistent with prior works such as BOLT [r24] and Bumblebee [r31]. We include a table below summarizing the per-token cost across BOLT, Bumblebee and NEXUS for clarity.\n\n| Work        | Model                  | Bandwidth (Gbps) | Ping latency (ms) | Latency (min/token) | Comm. (GB/token) |\n| ----------- | ---------------------- | ---------------- | ----------------- | ------------------- | ---------------- |\n| Ours        | DeepSeekMoE 2.8B/16.4B | 3                | 0.2               | 0.76                | 2.46             |\n| Ours        | OLMoE 1.3B/6.9B        | 3                | 0.2               | 0.36                | 1.31             |\n| Bolt        | BERT-base              | 3                | 0.8               | 0.024               | 0.47             |\n| BumbleBee   | LLaMA-7B               | 1                | 0.5               | 1.73                | 0.85             |\n| BumbleBee   | BERT-large             | 1                | 0.5               | 0.052               | 0.16             |\n| Nexus (CPU) | Llama-3-8B (batch32)   | 3                | 0.8               | 2.27                | /                |\n| Nexus (CPU) | BERT-base (batch32)    | 3                | 0.8               | 0.11                | 0.0013           |\n\nWe would like to emphasize that prior works such as BOLT and Bumblebee focus on supporting dense models (e.g., BERT, LLaMA-1). In contrast, our core contribution lies in **proposing the first framework that supports privacy-preserving inference for MoE models**. For other components, such as RMSNorm and attention, we **directly adopt the protocols from prior works BOLT and Bumblebee**. Specifically, for matrix multiplications in linear layers, we leverage BOLT’s protocol; for nonlinear operations, including RMSNorm, Softmax, and SiLU, we employ the protocols proposed in Bumblebee. We acknowledge that these aspects are not the primary focus of our work. In addition, the communication overhead of MoE models is relatively higher than that of dense models, primarily due to the additional communication introduced by the dispatch and combine protocols. However, MoE models reduce the overall computation per expert, leading to lower latency and thus offering better efficiency in terms of inference time.\n\n**As for our HE/MPC parameter configuration**, we follow the settings in Bumblebee: we use $N = 8192$, $\\log q = 114$, and $\\log p = 64$, where $N$ is the polynomial degree, and $q$, $p$ are the ciphertext and plaintext moduli, respectively. For secret sharing, we adopt a 64-bit ring. We will include these details in the revised version to improve clarity.\n\n**Regarding code availability**, we understand that an open-source release is vital for independent verification and fostering adoption. Our current code framework is built upon **Secretflow SPU**, and we are actively in the process of organizing and refining it for public release. We are fully committed to releasing a reference implementation of our code once our paper is accepted.\n\n-----\n\n**[To Limitation 2: Assumptions on the Adversarial Model and Multi-Party Scalability]**\n\nOur work focuses on the **two-party setting under the semi-honest threat model**, which is a common and practical assumption in privacy-preserving machine learning. Extending the framework to a malicious adversarial model or to a multi-party setting falls outside the current scope of our study. Nevertheless, both directions are important and promising. As noted in BOLT, **designing protocols that are secure against malicious adversaries is far from trivial**. Similarly, supporting multi-party computation would require **replacing the underlying MPC primitives we currently employ**, which we also consider a valuable avenue for future work. We will add a discussion of these points in the final version to enhance completeness.\n\n------\n\n**[To Paper Formatting Concerns]**\n\nThank you for your careful review and helpful suggestions.\n\nRegarding the notation inconsistency between \"top-t\" and \"top-k\", we clarify that our intention was to use **\"top-k\"** consistently to refer to the general protocol for selecting the top-$k$ elements (e.g., $\\Pi_{\\text{top}k}$). The occurrence of \"top-t\" in line 55 was meant to emphasize that **each expert receives $t$ tokens** during dispatch. We will standardize the notation to **\"top-k\"** throughout the paper and explicitly state **$k = t$** in relevant contexts to eliminate ambiguity.\n\nFor Figure 8, the **reported speedup factors (e.g., 3×, 2×)** refer to the performance gain of our proposed method compared to the **dense baseline**. The figure caption will be revised to use the plural form \"settings\" in the final version.\n\nWe will revise the final version to improve the consistency and clarity of notation and figures to avoid misunderstandings.\n\n-----\n\n**References**\n\n- \\[r24, r31, r37, r44] represent the 24th, 31th, 37th, 44th references cited in our original submission.\n\n----\n\n**We sincerely appreciate the detailed advice from reviewer d2x4! We take all your suggestions into account in our final version. Please let us know if you have any further questions. If you feel that the above improvements help clear up your doubts and further improve our work, you can kindly consider a re-evaluation, thank you!**"}]

Author rebuttal:

Meta-review: This paper introduces CryptoMoE, a novel framework for performing private and efficient inference on Mixture-of-Experts (MoE) Large Language Models. The core challenge addressed is the potential for information leakage from the dynamic routing mechanism in MoE models, where input tokens are sent to different "expert" sub-networks depending on their information. The authors propose a cryptographic system to hide which experts are being used for a given input. Their method includes a balanced expert routing strategy and new protocols for secure data handling. According to the experimental results, CryptoMoE achieves a multiple factor reduction in both latency and communication overhead compared to a standard dense model (without losing accuracy).

The reviews for this paper are borderline, with final ratings ranging from "Accept" to "Borderline reject". On the positive side, all reviewers agree that the paper tackles a novel and important problem, being the first to address private inference for MoE models. They also acknowledge the strong empirical results and the well-designed approach. However, significant concerns make the paper's acceptance questionable. Multiple reviewers pointed to a lack of clarity in the experimental setup, requesting more details on baselines and cryptographic parameters. There were also concerns about insufficient comparison to other state-of-the-art methods, which was partly resolved during the rebuttal period. There is one major remaining concern about the lack of publicly available code which hinders the community from reproducing the result (especially given the impressive performance reported).
Decision: Accept (poster)