MOAI: Module-Optimizing Architecture for Non-Interactive Secure Transformer Inference
Authors: Linru Zhang
Xiangning Wang
Jun Jie Sim
Zhicong Huang
Jiahao Zhong
HUAXIONG WANG
Pu Duan
Kwok-Yan Lam
Venue: ICLR 2026 Poster (2026)
Keywords: fully homomorphic encryption
secure transformer inference
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from:  (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2448 HE rotations in BERT-base inference.    (3) Column packing removes rotations in plaintext–ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext–ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, We achieve a 52.8\% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS'25). We then apply MOAI on the Powerformer's framework and achieve a 55.7\% reduction in evaluation time compared to Powerformer (Park et al., ACL'25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source.

Published as a conference paper at ICLR 2026
MOAI: MODULE-OPTIMIZING ARCHITECTURE FOR NON-INTERACTIVE SECURE TRANSFORMER INFER- ENCE
Linru Zhang', Xiangning Wang', Jun Jie Sim², Zhicong Huang", Jiahao Zhong', Huaxiong Wang'. Pu Duan', Kwok-Yan Lam! Nanyang Technological University, 2 *Ant International, 3 Ant Group
ABSTRACT
Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic en- cryption (HE) offers promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, moving 2448 HE rotations in BERT-base inference.
(3) Column packing removes rotations in plaintext-cipherrex matrix multiplications and interleaved batching further reduces the rotations in ciphertextt-ipheerexx matrix multiplications, MOAI uses least 1.7x fewer HE rotations compared to the state-of-the of-the-art works across all matrix multiplications of BERT-base. As a result, We achieve 52.8% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS'25). We then apply MOAI on the Powerformer's framework and achieve a 55.7% reduction evaluation time compared to Powerformer (Park et al., ACL'25), which approximates Softmax and LayerNorm with simpler functions transformer and proposes HE-based non-interactive transformer inference. We report amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI LLaMA-3-8B, Our implementation is publicly available as open source.
INTRODUCTION
The Transformer architecture Vaswani et al. (017)) powers many NLP tasks but requires heavy computation and memory, motivating deployment on cloud service providers (CSP). There are growing concerns about potential privacy leakage when clients upload sensitive information to CSP.
Fully homomorphic encryption (FHE) (FHE)Gentry (2099) enables direct, non-interactive computation over encrypted data, making it promising primitive for protecting clients' data privacy. In FHE--basdd Al-as-a-Service works Bourse et al. (2018): GGlad-Bachrach et al. (2016); Mishra et al. (2020) Lam al. 22244; Zhang al. (2024b). client encrypts its input data and uploads the ciphertexts to a cloud service provider (CSP), which performs model evaluation without any decryptions. The CSP then returns the encrypted inference results the client, ensuring that learns neither the client's 's inputs nor the inference outputs. However, the high computational overhead of homomorphic evaluation significantly limits the practicality of FHE-based inference, especially in scenarios that large number of inputs has to be processed such document classification, representative application of encoder-only models like BERT Devlin et al. (2019).
Related work. Some works Chen et al. 2022); Pang et al. (2024): Lu et al. (203); Dong et al. (002) Zeng et al. (2025) targeting interactive prvaacy-reserving transformer inference, usually

---

Published as a conference paper at ICLR 2026
BERT, relied on combination of FHE and MPC that splits the user's data into shares and distributes a share to the server.
The first non-interactive solution NEXUS, based purely on FHE, was proposed by Zhang et al. 2024a) (NDSS 225. They proposed several FHE- frienfriendly matrix multiplications and approximations for activation functions. However, NEXUS does not provide end-to-end inference solution. NEXUS only reports performance as an aggregation of microbenchmarks. In addition, due to inconsistent packing formats, its algorithms cannot be directly composed. As also observed by Lim al. (2025); Moon et al. (2024), the conversions between these formats during secure inference are not explicitly explained and the time cost of format conversions not included in the reported performance. matrix multiplication and functional bootstrapping for inference, but focuses only on the feedforward block rather than providing an end-to-end solution. The state-of-the- HE--aaed non-interactive solution THOR Moon et al. (2024) (CCS'25) achieved end-to-end secure transformer inference. THOR provided inference on 128 tokens with the BERT-base model and reported 10 minutes per input on a single GPU. Format conversions are required in THOR to complete ciphertext-cipherexx matrix multiplications. These solutions do not modify the transformer components such as Softmax and LayerNorm, and thus avoid the need for re-training or fine-tuning.
LEAF Zhang et al. (2025) leverages coefficient-
Another line of research focuses adapting transformers to FHE-friend]) functions. Zimerman et al. (2024b) replaced non-polynomial functions (e.g.. Softmax) with polynomial alternatives. Power- Softmax ZZieemann et al. (2024a) replaced the exponential function in Softmax with a power function and applied conventional fine-tuning, The state-of-the-ar solution in this direction, Powerformer Park let al. (2024b) (ACL'25) replaced Softmax and LayerNorm in the BERT-base model with power and linear functions, achieving 5.74 minutes per input on a single GPU, Although this line of work offers improved efficiency, requires re- re-training through knowledge distillation, which limits its generalizability to other transformer architectures.
Our contributions. We focus on designing a plug-and-play FHE framework that can be seamlessly integrated into existing transformer architectures. Similar NEXUS and THOR, we do not modify any transformer components to avoid the need for re-training through knowledge distillation.
In this paper, we present MOAL: Module-Optimizinn Architecture for non-interactive secure trans- former Inference. MOAI is FHEE-based. We apply MOAI on the BERT-base model and report an amortized time of 2.36 minutes per imput on a single GPU environment. We also extend our methods to decoder-only transformers, such as LLaMA-3-8B MOAI outperforms the state-of-the-art work THOR, achieving 52.8% reduction in evaluation time in the same environment. We also show the extendibility of MOAI by applying MOAI's evaluation flow to converted FHE-friendly transformer architectures. MOAI outperforms the state-of-the-ar work Powerformer, achieving 55.7% reduction in evaluation time when applying MOAI to Powerforer's FHE-friendly modifications. Our techniques can be summarized as follows:
(I) Consistent matrix packing strategies to avoid extra format conversion. The input ci- phertexts and the evaluation of Q, K are in column packing. Then the output of our FHE evaluation of QKT is in diagonal packing, which matches the input format of our FHE evaluation of Softmax. After evaluating Softmax in diagonal packing, a ciphertext- ciphertext matrix mul- tiplication (for softmax(QKT VVd'VV takes input of one matrix encrypted in diagonal packing (Xx /d))) and one matrix encrypted in column packing (Ene(V)) and the output matrix is encrypted in column packing. Resting parts such as feedforward layers are all in column packing, and are ready to be passed to the next transformer layer without format conversion. The details of the evaluation flow is in Figure Our evaluation flow ensures the consistent packing formats across all layers to avoid expensive format conversion between layers. We also remove the need of performing encrypted matrix transpose separately. In THOR, additional format conversions and transpose are necessary.
(2) Rotation-free free Softmax and LayerNorm evaluations. Evaluating Softmax and LayerNorm in NEXUS and THOR includes set of FHE rotations to compute summation of data packed in ciphertext. HE rotation is known as a computation intensive operation which affects the efficiency (c.g., Halevi & Shoup (2018)). We propose ootation-free Softmax and LayerNorm evaluation algorithms which eliminate rotation by putting the inputs to the summation in the same slot position of different ciphertexts. These algorithms save 2448 HE rotations in BERT-base inference.
2

---

Published as a conference paper at ICLR 2026
(3) Minimizing HE rotations in matrix multiplication. Column packing is adopted remove HE rotations in Saintext-cipherter matrix multiplication. Interleaved batching is applied to further reduces the rotations in pethertext-ciphertee matrix multiplications. MOAI achieves at least 1.7x fewer HE rotations than the state-of-the-ar works across all matrix multiplications of a transformer.
PRELIMINARIES
We use [n] to represent the set {0, 1,..,, 1}. We use bold lowercase letters for row vector R² where [vo, UI, "1y Un-1]. We [va]egin] to represent the vector v. Column vectors represented by vT. We define the scalar product for I and R", let We also define addition and Hadamard product (entry-wise multiplication) between two vectors with dimension as follows:
[cuskee[n]-
[t; uilke(n) & 0 < & < to denote the vector [ule...., Un Uo, UR- 1.. a left rolation of v. For -n k S 0, (v) denotes a right rotation.
[[]4]]e[[]] We use Rot, for
We use uppercase letters to denote matrix A R where A (ary) je(d- We use aj' to represent the j-th column of A, and
a1 || ... a)-1). Further, if A is square matrix with size X m, define Diag, (A) A's i-th (upper) diagonal for i E [m]] Diag, (A) = [00,s; +1 am 1m1,m- 1,00 Om- 0m-1,i-1]
write A (ao T
2.1 FULLY HOMOMORPHIC ENCRYPTION CKKS SCHRME
Fully Homomorphic Encryption (FHE) enables computation on encrypted data. The CKKS scheme Cheon al.] (2017) supports approximate arithmetic over complex vectors. We briefly describe CKKS and refer to Appendix and Cheon et al. (2017) ) for details.
Let N be the degree of a polynomial ring, whose value a power of Q be a product of distinct primes: Q = II 9t. where L is called the level of the ciphertext. fresh ciphertext starts from level L and drops L 1 after homomorphic multiplication. When remaining level is O. decryption fails. The bootstrapping process Cheon et al.) (2018) can refresh the ciphertext and restore the remaining level, enabling continued evaluations. The CKKS scheme supports SIMD- computation by encoding N/2 messages into a single ciphertext, enabling parallel homomorphic operations across all slots. Let v, v' be two vectors in N/2 and ct = Enc(v) and ct' = Enc(v'). The operations supported by the CKKS scheme
follows:
(dt et'). Return a ciphertext that encrypts the vector sum v + v Mult(ct, et'). Return a ciphertext that encrypts the Hadamard product 69 v'
PlainMult v'). Return a ciphertext that encrypts the Hadamard product v & v'
Roth (ct). Return a ciphertext that encrypts Rota(V) (v) = [Vk, UN/2-1 Co, U-1]
Many previous works state that HE rotation is of the slowest operations, and they optimize schemes by reducing the number of HE rotation (e.g.. |Halevi & Shoup (2018): Lu et al. (202I), Experimental results in (Yang et al. (2024); Agulló-Domingo al. (2025)) also indicate that HE rotation performs considerably slower, especially as the ciphertext modulus increases.
2.2 TRANSFORMERS AND FHE-BASED SECURE TRANSFORMER INFERENCE
The transformer architecture Vaswani et al. (011 follows an encoder-decoder design. We focus on the encoder (e.g., BERT), commonly used as a benchmark. Each encoder has layers with multi-head self-attention, normalization, and feed-forward blocks. Core computations include matrix multiplication, Softmax, LayerNorm, and GELU. Further details are in Appendix B
In FHE-based secure transformer inference, the model will take encrypted tokens input and output encrypted results. The model is evaluated over encrypted data homomorphically, where the FHE evaluations include plaintext-ciphertextt matrix multiplication (computing Q, K, V and feedforward layer), ciphertext- -iihertett--ipheetxtt matrix multiplication (computing QKT and softmax(-) and non-linear evaluations such as Softmx, LayerNorm and GELU.

---

Published as a conference paper at ICLR 2026
3 MATRIX PACKING, MULTIPLICATIONS AND INTERLEAVED BATCHING
3.1 MATRIX PACKING AND MULTIPLICATIONS
The SIMD capability of CKKS scheme allows multiple data to be packed into single ciphertext. which offers an array-like paradigm to encode matrices for encryption. There are mainly three ways to encode an encrypted matrix in Halevi & Shoup (2014), namely row, column, diagonal packing, The main difficulty in designing an encrypted matrix multiplication pipeline is that the encrypted matrix must be able to in and switch between various forms (e.g.. the transpose AT). To this end, we propose to use both the column packing and diagonal packing to realize seamless transition for encrypted matrix multiplication.
Definition 3.1 (Column Packing). Given a matrix X E RRxd the column packing of X is a set of d CKKS ciphertexts, denoted as Enceal (X) {{ Enc(xx], Enc(x[). .... .nnc)),,,,.,,( Enc(x]]1) where xj" is the j-th column of X. We say that the ciphertexi of X is in column packing.
Definition 3.2 (Diagonal Packing). Given a square matrix X E Rmxm, the (up- per) diagonal packing of X are m CKKS ciphertexts, denoted as Encaiag(X) == (Ene(Diag,(X)) Enc(Diag, (X)), (x)))).,,En(((aae,,, Ene(Diagmm((X)) where Diag,(X) the j-th (upper) diagonal of X. We use Diag, (X) as a row vector and say ciphertext of X is in diagonal packing.
Encrypted matrix- vector multiplication for both packing methods were proposed in HHlleii & Shoup (2014). We give a brief description of the algorithms for encrypted matrix-vector multiplication for both packing methods. Let be a square matrix of dimensions m m and e be a vector of length m.
For column packing, denote C' with its m columns as (Col((C), (C), (C)) Halevi & Shoup (2014) first proposed an auxiliary algorithm Replicate that takes the vector and return m-many vectors Vi, each encrypting Ut, that is {vi = [ea, Ui ....... us [i]].e The matrix- vector product is computed as Ct Σ (C) @ vi. We can extend the matrix-vector product to matrix-matrix product by iterating over all columns of the right hand side matrix.
For diagonal packing, denote C with its m diagonals as (Diago i (C), Diagm- (C)).
The matrix-vector product is computed as Cv = > Diag; Rot(v, i). A subsequent improvement in Halevi & Shoup [2021) reduced the number of rotations needed to O (vmm) by using the baby-step giant-step method (see Appendix C.T for details).
We describe our HE matrix multiplications in both column and diagonal packing:
Case I: Calculating Enc(Q), Ene(K), Enc(V) E Rm with Enc( (X) W:+1Tb; for E (Q,K,V) We encrypt X column packing while the weights and biases are not encrypted. We note that since the weights matrices are not encrypted, the matrix product can be interpreted as concatenations of linear combinations of X. XW = EErejar) W7 || Eie (r' Wt,1x] || ... Eieja' Wi.d' 1x,
The detailed algorithm is given as Algorithm 2 in Appendix C.2
Case I:: Calculating Enc(QKT). We remind the reader here that Q and K are encrypted in column packing as per the result of Case I. We then apply the following lemma that allows us to compute the matrix product QKT where both Q and K are encrypted in column packing, without transposing K.
Lemma 3.1. Suppose Q, K E Rmxdd so QKT is square matrix with size m x m. Then for j E [m]] d' Diag, (QKT) = Di q @ Rot,(k:), where qi and k; are the i-columa of Q and K respectively.
The result is a matrix encrypted in diagonal packing. The detailed algorithm is given as Algorithm in Appendix C22
Case III: Calculating Enc(softaxx(QKT /VXTTV/VVVV. The encrypted matrix V is encrypted in column packing, as per Case For now, we briefly mention that applying the Softmax function on a diagonally packed matrix will return a diagonally packed matrix. Thus, softmax(QK") / d') is encrypted in diagonal packing. For this matrix product. we can apply the matrix-vectur product for diagonally packed matrices described earlier Halevi & Shoup (2014) 202T) over all encrypted columns of V.
The detailed algorithm is given Algorithm in Appendix C.2
4

---

Published as a conference paper at ICLR 2026
3.2 INTERLEAVED BATCHING
In column packing (Definition 3.1 and diagonal packing (Definition 3.2). each ciphertext encrypts vector of dimension mi, i.e., the sequence length of input. Meanwhile, the number of slots N/2 in the CKKS scheme is usually 215 So batching multiple vectors into one ciphertext will reduce the amortized cost. Interleaved batching, previously discussed in Aharoni al. (2023 1 [Adir al. (2024) for evaluating convolutions in neural networks, is adopted in MOAI to further reduce the number of HE rotations in matrix multiplications.
Assume m ml)222. Let (2m) vectors (x(1) IR" JrE[N/(2m) be: ( = [ (0) (0)
(0) -m......, ( (N/(2m))1) = (x) (2/(m)) (N/(2m) y.. 2
(N/(2m))1)
interleaved batching is defined by packing the the first entry of each vector from (x(()
R [re])/(2m))- then the second entry, and at last the last entry. We use R/22 represent the interleaved batching of (2(mm) vectors in R
(0) (1) (2) 20 T0
(21(2))-1)
(N/(2m)-1) J
x
We use Lemma to address the its compatibility with rotation and include its proof Appendix [D.2] The naive batching method in existing works uses two HE rotations to do one "internal rotation" which only shifts slots within each vector rather than across all slots a ciphertext. MOAI achieves the same functionality by only one HE rotation, thereby reducing half of HE rotations.
Lemma 3.2. Let & E TR. N/2 be the interleaved batching vector of {x E R'm rE(N/(2m)) given by
Eq. 1 Then Rot:N//2n (3) the interleaved batching vector of (Roty (xot (eM)))))))))
Finally, we define interleaved column/diagonal packing by merging interleaved batching with our matrix packing methods directly. Please refer to Appendix D.3
3.3
COMPLEXITY COMPARISON
We provide a comparison of our matrix multiplications with matrix multiplications in prior FHE-based secure transformer inference works NEXUS Zhang al. (NDSSSS 25), THOR Moon al. (CCS and Powerformer Park et al. (ACL We focus on the total number of HE rotations, which is the most expensive underlying HE operations as discussed in Section 221
In Table 1 we report the total number of HE rotations in all matrix multiplications of transformer. g MOAT requires only 9648 HE rotations in all matrix multiplications BERT model, achieving 22.9x, 2.3x, and 1.7x fewer HE rotations than NEXUS, THOR, and Powerformer, respectively.
For consistency with prior work, we include comparison the total number of key switches in Ap- pendix D Key switch is the key component in the relinearization process of ciphertext multiplication and the HE rotation. MOAI requires only 834 key switches for all matrix multiplications a single multi-head attention block, MOAI achieves the least number of key switches.
4 NON-LINEAR FUNCTION EVALUATION
In this section we propose optimized evaluations of non-linear functions. Our novelty is that we do not require any rotations in evaluations of Softmax and LayerNorm. This greatly improves the speed of our algorithms. Table 2 gives a comparison between ours and NEXUS Zhang et al. (2024a) (NDSS'25) and THOR Moon et al. (2024) (CCS'25). When substitute BERT'S parameters into the equation, our rotation-fres Softmax and LayerNorm save 2448 HE rotations in total.
Softmax. The Softmax function appears in softmax(QKT///55 V, but we eliminate the vd' term by folding into WK, saving one CKKS level. The input of Softmax in diagonal packing, and we
For NEXUS, results are taken from Zhang al. (2024a) and Table 4 Moon et al. 2024), and normalized by batch size. NEXUS does not specify how it evaluates the third line, so we exclude this result for NEXUS.
For THOR, results are taken from Table 4 in Moon et al. (2024). include counts only from HE rotations. 3) For Powerformer, the attention block's 's HE rotation count our table is computed by Algorithm 14 in Park et al. 2024bb).
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of number of HE rotations in matrix multiplications. For short, we use a to denote Softmax and omit vd' is the intermediate size of the ford-forward layer. H is the number of heads. c = 16 is from THOR. & = 64 is from Powerformer.
Operation
Function
NEXUS #Ror"
THOR #Roe
Powerformer *Rot
Our MOAI *Rot"
Attention block
X 6 BMMd, W!" A [8]
Exx(X = Q.K.V
3(F 2
2V
Eneigro Bacc("
# +3)
14/ 32
Qth) gg4 R"
EueterQr9 (X (9y1) Basi (
(Q01(xih [T) e guAn, p(ù Raxa
N/A
millgge
3)/4
m 3
10
If
(X
X xurnyyW Woo
X(l è R Bu Wean Rird
2(7/2
2
PC1
EnclX)WW X R Wha ge
N7
v .
PC2
Ene(XW X R Wjy Rive
9 1 A
2 M72
Total (Sabstitute BPRU's parameters into the equation: N 2d = 128, = 768, of = 64, H 12, 12 layen)
221 184
22224
16740
9648
Table 2: Comparison with NEXUS Zhang et al. (2024a) THOR Moon et al. 2024 and MOAL Softmax and LayerNorm.
Function
NEXUS #koo!!:
THOO AiRe!"!
MOAI (ours) #itor"
sullaaax X R(ase)
[oga(we] Io gilneylm1 /N
[[ogsowe] An -gg Sp
LoyerNormiX X R(aae)
low, (4d 21 /N loga(d) (4r 21 /N
Total (Subsaitute BERT's parameters into the equation: N 2 128, == 768, H 12, 12 layers)
2448
2448
0
want the output of the Softmax function to be in diagonal packing, as the input to the next module. We introduce novel FHE Softmax algorithm (Algorithm I] Figure (T) that preserves this packing
I format and eliminates the need for rotations, based on the key insight in Lemma 4
Lemma 4.1. Let C E IR mm be a square matrix. The summation of columns equals to the summation of diagonals: Σ, c. = E. (C).
Darg WorrgOR
Factirg a/fosselg
GN7
Figure 1: Our rotation-free Softmax evaluation whose input and output are both in diagonal packing.
LayerNorm. Note that the input of LayerNorm is in column packing. We first give the SIMD style mean and variance algorithm, given input as Enc (X) where X E Rm For Enc(mean( (X)), compute the summation of all encrypted columns of X and then multiply 1/d: (Muli(1/d) E, (x.)) For Enc(van (xar(X)). First compute ct2² == Mult(Ene(x.) Enc(mean(XX) Enc(x,
(x)))) i E [d]. Then compute sMult(1/d, Exe(g ct2²). We can merge two scalar multiplication of 1/d to save one FHE level. Goldschmidt division algo- rithm Goldschmidt (1964): Qu & Xu is applied to obtain the inverse square root (1/ L2vz) of variance. Please refer to Algorithm for the detailed explanation in Appendix F
GELU. For the encrypted GELU algorithm using Equation 2 it suffices to approximate tanh(-) function by polynomial. The main challenge the balance between the size of effective input interval and the polynomial degree. By|Cheon et al. (2022). a degree-S2( R)) polynomial is required when using minimax approximation with lixed maximum error on the domain interval [-R, RR.. In our work, we approximate tanh(-) in Equation Jusiin degree-23 polynomial Paa (ar) on interval [-20, 10] which is enough in our experiment. If the model needs larger input interval, we apply the domain extension polynomial (z) := polynomial of tanh(z) on [-B, B]. Then Po (x) can take input from a larger interval [-LB, LB) for proper T.. (Theorem 4 in Cheon et al. (2022)) This can be further extended input interval
27.2135 3 from Cheon al. 2022). Let P(z) be the minimax
6

---

Published as a conference paper at ICLR 2026
Algorithm 1 HE Softmax evaluation: Encating (QKT) Enc,tiag (a (o(QKT))
Input: Encasag EuC Diagonal packing of encrypt matrix QK* E Rm
Output: Encateg softmax(QK") )): Diagonal packing of encrypt matrix o (QKT) E E mm
1: Write Encaiag (QKT) (Enc(Diago(QK)
Enc(Diag, (QKT)), ..., -
2: Evaluate Enc(exx((iia) (Q) by SIMD polynomial evaluation + 2/2")
3: Add ciphertexts obtain clyum Enc(E, exp(Diag, (QKT), (QKT)))-
Evaluate FHE inverse algorithm on claum using the Goldschmidt division algorithm Goldschmidi (1964.
Let the result be cI cum == Enc(1 / E, (XpgDiag, (QKT)))
5: For E [m)) return Mult (ut (eneeexp(Diagg (QKT)) ))))
as the i-th component.
L'B) We also point that using piecewise polynomial in NEXUS ZZang al. (2024a) to approximate GELU has precision problem. Please refer to Appendix F.2 for a detailed analysis.
5 MOAI: FULL FLOW
The name MOAI also comes from the monolithic human figures on Easter Island. They are huge and stable. They similar to scheme: large and stable end-to-end inference, and with very few "rotations". We show our full diagram in Figure 2 of the BERT-base-uncased model, which is implemented and test in Section 6.1 The proposed packing flow, evaluation algorithms and interleaved batching techniques are applicable to other transformer models and even decoder-only models such Llama3-8B.
We illustrate the dimensions and the packing formats of encrypted matrices in each module. The packing formats include column packing (Col-packing. Definition 3.1) and diagonal packing (Diag- packing, Definition 3.2). CPMM stands for ciphertext-plaintext matrix multiplication, while CCMM stands for seghertext-ciphertee matrix multiplications. As shown in Figure MOAI does not need conversion between the two packing methods.
Regarding the number HE rotations, Table and Table 8 show that MOAI requires only 9,648 HE rotations in the full flow, yielding 23.2x, 2.56x, and 1.46 x fewer HE rotations than NEXUS, THOR, and Powerformer, respectively. Unlike Powerformer, which bypasses HE rotations by replacing Softmax and LayerNorm, MOAI preserves both and eliminates all HE rotations with rotation-free evaluation algorithms.
Figure 2: The full diagram of MOAI. The parameters are from the BERT-base baRT-bes--uneased model, which has 12 transformer layers and 12 heads in each layer. is the number of classes.
126700
Transfomorr Layers
0.
Algor the poiking to Doag purking OrSe4
.WW !N
XW
NoveNo
Abprities Durltiig Out CUaM
12tere
12b
13hog0s
syechome Cut qestinng Cokgauking
CELVVXI
1302202
Clecefration
XIl sohimant))
XWa
A rm Kyuu
tures
g... pi
Xope
1200753
SWW
bout bootstrapping. Bootstrapping is an expensive operation that resets ciphertext noise, making its placement critical for performance. We make the following two observations: (1) A sufficiently accurate Softmax requires at least 10 iterations of the Goldschmidt division (line iin Algorithm based on experimental results in Figure 5), which together with exponentiation and final multiplication
7

---

Published as a conference paper at ICLR 2026
amounts at least 20 layers (8 for e", 11 for division, and for multiplication). (2) The runtime of matrix multiplication increases with noise level, as higher-level ciphertexts have larger moduli.
To address this, we perform bootstrapping on cleum after the third line of Algorithm refreshing it before Goldschmidt division. This reduces the Softmax depth from 20 to 10 layers, allowing the two preceding matrix multiplications to be executed lower level. As result, the attention block starts from lower layers, and a single bootstrapping operation on one ciphertext substantially decreases both multiplication and overall runtime. The optimized Softmax algorithm is given in Algorithm
EXPERIMENTAL RESULTS
Implementation details. We set the degree of the polynomial ring N - yielding 2 slots in a ciphertext. We chose a 1743-bit ciphertext modulus of in accordance to the Homomorphic Encryption Standard Albrecht et al. (02I) to achieve 128- bit security. We apply the scale propagation technique Bossuat et al. 2
) to better manage the noise propagation and set the scale to 2 in our implementation. Our codes available here
CPU implementation. We implement MOAI in C++, utilizzng the SEAL library SEAL for CKKS homomorphic encryption, FHE-MP-CNN for CKKS bootstrapping, and OpenMP Dagum & Menon (1998) for multi-thread programming. CPU results are evaluated on a Intel(R) Xeon(R) Platinum 8480+ CPU at 2.0 GHz with 56 cores.
GPU implementation. MOAI is implemented using the Phantom library for CKKS encryption and bootstrapping. GPU results are evaluated on a single NVIDIA H200 GPU (end-to-end evaluation in Table or a single NVIDIA A 100 GPU (comparison with SOTAs in Table Table 5p.
Model and Dataset. Same as SOTA works THOR Moon et al. 2024) and Powerformer Park et al. (2024b), we test MOAI's end-to-end performance on BERT-base model. It has 12 attention heads and 12 layers with an embedding dimension of 768. The input token length is 128. In evaluations, we use the pre-trained BERT-base-uncase model fine-tuned on SST-2, QNLI, and RTE tasks [
Microbenchmark. Our optimized algorithms achieve speedups of up to 22x for Softmax and 151x for LayerNorm compared to prior works. We report the microbenchmark results in Appendix H.I
6.1 END-TO-END INFERENCE EVALUATION
presents the end-to-end performance (amortized for 256 inputs). The results are obtained from the execution of the BERT model with 12 transformer layers. For evaluating BERT model with input contains up to 128 tokens, we report an amortized time of 9.6 minutes per input on CPU and 2.36 minutes per input on GPU, Beyond encoder-only models, we extend our methods to decoder-only transformers, demonstrating them on LLaMA-3-8B inference and reporting results in Appendix H.2
Table 3
6.2 COMPARISON WITH SOTAS.
We compare MOAI with state-of-the the--rt works on single NVIDIA A100 GPU, consistent with THOR |Moon et al. (024) and Powerformer Park et al. (2024b). For fair comparison, we also incorporate Powertormer's pooler and classification evaluation algorithm into our packing method (column packing) to account for total evaluation time.
Similar to THOR, MOAI is designed as plug-and-ple FHE framework that can be seamlessly integrated into existing transformer architectures. This line of work does not modify the transformer components such as Softmax and LayenNorm, and does not need re-training fine-tuning. Table 4
²CPU implementation: ht httttp::/eeee http:://tttttb com http://www.....m//dtt//a////////
GPU implementation: attps
GPU
gyttorch/tett-classee
"Similar to THOR and Powerformer, we do not compare end-to-end time with NEXUS Zhang et al. (024a). since it only provides microbenchmark results. Its inconsistent packing formats require costly conversions that are neither described in the paper nor implemented in its open-source code, making end-to-end reproduction
infcasible (see Moon al. (2024) Lim 20255)
8

---

Published as a conference paper at ICLR 2026
Table 3: Breakdown MOAI encrypted inference, amortized over 256 inputs
Operation
(Data dimensions) x
CPU
GPU time(s)
Component
Depth
Packing
in each layer
(Number of operations)
time(s)
Pt-ct MatrixMul
15 + 14
Col + Col
[R 128 (768664)
36
37.4
10.2
Ct-ct MatrixMul
14 + 13
Col-+ Diag
(R
[2288664 x R
x 12
40.3
8.55
Attention
Softmax
13 +
Diag Diag
R R 12
53.3
0.74
Ct-ct MatrixMul
Diag Col
RR
R 128884) 12
1.4
0.26
Pt-ct MatrixMul
2
Col +
(R 12
x 7688766
1.7
Sclf Output
Bootstrapping
1 11
Col +
E R 768
95.4
26.1
LayerNorm
Col
R128x 708
0.6
Bootstrapping
>
Col
1228 768
95.8
26.2
Pt-ct MatrixMul
-
Col + Col
(IR 128x x R70883072)
44.1
12.7
Intermediate
GELU
9
Col + Col
I 122xx0072
3.3
1.38
Pt-ct MatrixMul
2->1
Col Col
(2-28 12-28xx007) 3017 x R8072x708
7.1
2.5
Bootstrapping
1- 21
Col Col
F 11288 768
98.8
26.1
Final
LayerNorm
Col Col
R 1288
0.6
0.05
Bootstrapping
1- 15
Col Col
F 128x
94.8
26.07
Total
574.6
141.3
reports the layer-wise performance comparison between MOAI and THOR in the same environment with same ring dimension (N = 2 and slot size (n 2),). By reducing the number of HE rotations and adopting consistent packing strategies, MOAI achieves 52.8% reduction in total time.
Furthermore, we apply MOAT's packing methods and evaluation algorithms to Powerformer which replaces Softmax and LayerNorm with FHE-friendly functions (BPmax and Batch LN) and retrains the model accordingly. [Table rJpppots the layer-wise performance comparison between MOAI and Powerformer the same environment, following Powerformer's parameter setting, i.e., 2 slots in ciphertext and 11 multiplicative levels before bootstrapping. MOAI achieves 55.7% reduction total evaluation time, demonstrating that our packing methods and evaluation algorithms can significantly improve the efficiency of retrained, FHE-friendly transformer architectures.
Operation
Ours
THOR
Diff
Operation
Ours
Powerformer
Diff
Attention layer
16.54
49.77
33.23
Attention layer
14.29
37.65
43.36
Attention score
14.07
16.25
2.18
Attention score
6.61
28.76
22.15
Softmax
13.34
BPmax
0.06
0.75
Attention heads
0.29
13.08
12.79
Attention heads
0.39
18.95
18.56
Multi-heud attention
0.48
27.43
26.95
Multi-heud attention
4.05
22.54
18.49
LayerNorml
0.15
7.13
6.98
Batch LNI
0.004
FC1
16.18
49.8
33.62
PC1
4.79
59.21
54.42
GELU
3.3
29.42
26.94
GELU FC2 Batch LN2
5.2
8.31
5.83
FC2
2.88
46.31
13.79
43.77
29.98
LayerNNrm2
0.15
4.1
3,95
0.004
0.3
0.296
Pooler Classification
0.7
2.7
2
Pooler Classification
0.7
0.2
-0.5
Bootstrappings
227.84
337.86
110.02
Bootstrappings
102.5
103.72
1.22
Total
283.95
602.26
318.31
Total
152.4
344.52
192.12
Total w/o Pooler & Class.
283.25
599.56
316.31
Total wo Pooler & Class. 151.7
344.32
192.62
Table 4: Breakdown of execution time single A100 GPU compared to THOR
Table 5: Breakdown of execution time single A100 GPU compared to Powerformer
6.3 ACCURACY.
We analyze error propagation during encrypted evaluation by recording layer outputs and comparing them with plaintext results (Figure 3). The error grows over the first 6-7 layers but then stabilizes, indicating that it remains bounded. This shows that FHE evaluation does not cause significant accuracy loss. Further details error variation within layer provided in Appendix H.3] Table reports GLUE benchmark scores under both plaintext and encrypted inputs. The results indicate that the additional error introduced by FHE has only negligible effect inference accuracy.

---

Published as a conference paper at ICLR 2026
Dataset
SST-2
QNLI
RTE
Mican
Enor
005
a.re- 12705 Bed 1.0 257264 4.004 27 ba 2.70-4 2 1200
rang
Unenerypted input
Encrypted input
93.8
93.9
90.7
90.5
65.7
65.5
Match count
1818
5437
2959
Mismatch count
27
42
laye laye 2 lowr 3 layor laper lour layer? laper5 layer lour 10kaye Tllayer
Figure 3: Error analysis of 12 layers' output
Table 6: Scores on the GLUE benchmarks
7 CONCLUSION
In this work, we propose MOAI, an efficient and secure transformer inference framework on encrypted data. MOAI leverages consistent matrix packing strategies, rotation-free Softmax and LayerNorm evaluations, and interleaved batching, removing format conversion, minimizing HE rotations, achiev- ing substantial speedups and outperforming the state-of-the-art HE-based transformer inference work THOR Moon et al. (CCS 25). The proposed techniques are applicable to other revised transformer- based architectures, like Powerformer Park et al. (2024b) (ACL'25), to speedup the secure inference time.
REPRODUCIBILITY STATEMENT
Our implementation is publicly available via the anonymous GitHub repository linked in the submis- sion. All datasets used our experiments are public. Parameter settings, libraries, models, datasets, and hardware are detailed in Section
The execution time of THOR and Powerformer in Table 4 and Table 5 are taken from their paper, which are tested in the same hardware environment as ours (a single NVIDIA A100 GPU). More details can be found Section 66.2 The execution time of NEXUS in Table & are obtained by invoking its Softmax and LayerNorm in its open-source implementation. The GPU execution time of THOR in Table 8) are obtained from their paper, which is tested in the same hardware environment as ours (a single NVIDIA A100 GPU). THOR does not provide open-source CPU implementation report CPU results, so we omit the CPU time for THOR. More details can found in Appendix H.1
With these resources, all reported results in this work reproducible.
ACKNOWLEDGMENT
This work is a result of the joint lab collaboration between Ant International and Nanyang Tech- nological University, established part of five-year partnership advance digital trust and privacy-enhancing technologies (PETs), with funding provided by Ant International. This research supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or rec- ommendations expressed in this material are those of the author(s) and do reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.
REFERENCES
Allon Adir, Ehud Aharoni, Nir Drucker, Ronen Levy, Hayim Shaul, and Omri Soceanu. Homomorphic Encryption for Data Science (HE4DS). Springer, 2024.
Carlos Sgulló-Dominge Óscar Vera-López, Seyda Guzelhan, Lohit Daksha, Aymane El Jerari, Kaustubh Shivdikar, Rashmi Agrawal, David Kaeli, Ajay Joshi, and José L Abellán. Fideslib: A fllleef open-source fhe library for efficient ckks on gpus. In 2025 JEEE International Symposium Performance Analysis of Systems and Software (ISPASS), pp. 1-3. IEEE, 2025.
Ehud Aharoni, Allon Adir, Moran Baruch, Nir Drucker, Gilad Ezov, Ariel Farkash, Lev Greenberg, Ramy Masalha, Guy Moshkowich. Dov Murik, et al. Helayers: A tile tensors framework for large neural networks encrypted data. Proceedings Privacy Enhancing Technologies, 2023.
10

---

Published as a conference paper at ICLR 2026
Martin Albrecht, Melissa Chase, Hao Chen, Jintai Ding, Shafi Goldwasser, Sergey Gorbunov, Shai Halevi, Jeffrey Hoffstein, Kim Laine, Kristin Lauter, et al. Homomorphic encryption standard. Protecting privacy through homomorphic encryption, pp. 31-62, 2021.
Jean-Phillipe Bossuat, Christian Mouchet, Juan Troncoso-
and Jean-Pierte Hubaux. Effi- cient bootstrapping for approximate homomorphic encryption with non-sparse keys. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 587- 617. Springer, 2021.
Florian Bourse, Michele Minelli, Matthias Minihold, and Pascal Paillier. Fast homomorphic evaluation of deep discretized neural networks. In Annual International Cryptology Conference, pp. 483-512. Springer, 2018
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. THE-X: Privacy-preeeriing transformer inference with homomorphic encryption. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 3510-3520, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/ 00/663///1222222 w.........innns......mma ac1.277. URL
wwwpmthhoooooooog//2222000 (0lo:22..nninnggne.. findings-acl 2777
Jung Hee Cheon, Andrey Kim, Miran Kim, and Yongsoo Song. Homomorphic encryption for arithmetic of approximate numbers. In International Conference on the Theory and Application of Cryptology and Information Security, pp. 409 437. Springer, 2017.
Jung Hee Cheon, Kyoohyung Han, Andrey Kim, Miran Kim, and Yongsoo Song. Bootstrapping for approximate homomorphic encryption. In Advances in CryptoloyyEEUROCKE Annual International Conference on the Theory and Applications of Cryptographic Techniques, Tel Aviv, Israel, April 29-May 3, 2018 Proceedings, Part 37, pp. 360-384. Springer, 2018
2018: 37th
Jung Hee Cheon, Wootae Kim, and Jai Hyun Park. Efficient homomorphic evaluation on large intervals. IEEE Transactions on Information Forensics and Security, 17:2553-2568, 2022.
Leonardo Dagum and Ramesh Menon. Openmp: an industry standard api for shared- memoryy programming. IEEE compulational science and engineering, 5(1):46-55, 1998.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186 2019.
Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Chen. Puma: Secure inference of Ilama-7b in five minutes. arXiv preprint ar arXiv:2307.125333 2023.
Austin Ebel, Karthik Garimella, and Brandon Reagen. Orion: A fully homomorphic encryption framework for deep learning. In Proceedings of the 30th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 734-749, 2025.
Craig Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pp. 169-178, 2009.
Ran Gilld-Bachhache Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. Cryptonets: Applying neural networks encrypted data with high throughput and accuracy. In International conference on machine learning, pp. 201-210. PMLR, 2016.
Robert E Goldschmidt. Applications of division by convergence. PhD thesis, Massachusetts Institute of Technology, 1964.
Shai Halevi and Victor Shoup. Algorithms in helib. In Annual Cryptology Conference, pp. $54-571. Springer, 2014.
Shai Halevi and Victor Shoup. Faster homomorphic linear transformations in helib. Annual International Cryptology Conference, pp. 93-120. Springer, 2018.
11

---

Published as a conference paper at ICLR 2026
Shai Halevi and Victor Shoup. Bootstrapping for helib. Journal of Cryptology,
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in neural network training, In NeurIPS 2024, 2024.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 160iv/60060000 08466.08455 15, 2016.
Nandan Kumar Jha and Brandon Reagen. Entropy-guided attention for private llms. arXiv preprint arXiv:2501.03489, 2025.
Kwok-Yan Lam, Xianhui Lu, Linru Zhang, Xiangning Wang, Huaxiong Wang, and Si Qi Goh. Efficient fae-dased privacy-enhance neural network for trustworthy ai-as-a-service. IEEE Trans- actions on Dependable and Secure Computing. 21(5):4451-446 2024. doi: 10.1109/TDSC.2024 3353536.
Lawrence Lim, Vikas Kalagi, Divyakant Agrawal, and Amr El Abbadi. Tricycle: Private transformer inference with tricyclic encodings. Cryptology ePrint Archive, Paper 2025/1200, 2025. URL
https: /tpp:s://epritt.. lacr. org/2025/1s /200
Wen-jie Lu, Zhicong Huang, Cheng Hong, Yiping Ma, and Hunter Qu. Pegasus: Bridging polynomial and non-polynomial evaluations in homomorphic encryption. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 1057-1073. IEEE, 2021.
Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and WenGuang Chen. Bumblebee: Secure two-party inference framework for large transformers. Cryptology ePrint Archive, 2023.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: cryptographic inference service for neural networks. In 29th USENIX Security Sympo- sium (USENIX Security 20), pp. 2505-2522 2020.
Jungho Moon, Dongwoo Yoo, Xiaoqian Jiang, and Miran Kim. THOR: Secure transformer inference with homomorphic encryption. Cryptology ePrint Archive, Paper 2024/1881, 2024. URL https
7/ 7/eprint.. iacr. wttp///////00//3/
Qi Pang, Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy- preserving, accurate and efficient inference for transformers. In 2024 JEEE Symposium on Security and Privacy (SP), pp. 4753-4771. IEEE, 2024.
Dongjin Park, Eunsang Lee, and Joon-Woo Lee. Powerformer: Efficient privacy-preserving trans- former with batch rectifier-power creffeer-ppwerreeee max function and optimized homomorphic attention. Cryptology ePrint Archive, Paper 2024/1429, 2024a. httphh:t:// iacr, org.22///2/222
Dongjin Park, Eunsang Lee, and Joon-Woo Lee. Powerformer: Efficient and high-accuracy privacy- preserving language model with homomorphic encryption. Cryptology ePrint Archive, 2024b.
Jai Hyun Park. Ciphertext-ciphertext matrix multiplication: Fast for large matrices. Cryptology ePrint Archive, 2025.
Hongyuan Qu and Guangwu Xu. Improvements of homomorphic evaluation of inverse square root. Available at SSRN 4258571.
Eugene Y Remez. Sur la détermination des polynômes d' d'approximation de degré donnée. Comm. Soc. Math. Kharkov, 10(196):4--33 1934.
Donghwan Rho, Taeseong Kim, Minje Park, Jung Woo Kim, Hyunsik Chae, Ernest K Ryu, and Jung Hee Cheon. Encryption-friendly 1lm architecture. arXiv preprint arXiv:2410.024866 2024.
SEAL. Microsoft SEAL (release 4.1). 2023. Microsoft Research, Redmond, WA.
January
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint 17762 2024.
12

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Hao Yang,
Shiyu Shen, Wangchen Dai, Lu Zhou, Zhe Liu, and Yunlei Zhao. Phantom: cuda- accelerated word-wise homomorphic encryption library. IEEE Transactions Dependable and Secure Computing, (155:4895-4906 2024.
Wenxuan Zeng, Ye Dong, Jinjin Zhou, Junming Ma, Jin Tan, Runsheng Wang, and Meng Li. Mpcache: Mpe-friendly kv cache eviction for efficient private large language model inference. arXiv e-prints, pp. arXiv-2501 2025.
Jiawen Zhang, Xinpeng Yang, Lipeng He, Kejia Chen, Wen-jie Lu, Yinghao Wang, Xiaoyang Hou, Jian Liu, Kui Ren, and Xiaohu Yang. Secure transformer inference made non-interactive. Cryptology ePrint Archive, 2024a.
Linru Zhang, Xiangning Wang, Jiabo Wang, Rachael Pung, Huaxiong Wang, and Kwok- Yan Lam. An efficient fhe-enabled secure cloud-edge computing architecture for iomt data protection with its application to pandemic modeling. IEEE Internet of Things Journal, 1 1199:11272-15284, 2024b.
doi: 10110100.2202333338833
Linru Zhang, Xiangning Wang. Xianhui Lu, Huaxiong Wang, and Kwok Yan Lam. LEAF: A low- latency evaluation architecture for feedforward block in privacy-preserving transformer inference. Cryptology ePrint Archive, Paper 2025/956, 2025. URL
iaarriitttiit......o..m org/
2025/956
Yancheng Zhang, Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang, Lei Jiang, and Qian Lou. Cipherprune: Efficient and scalable private transformer inference. In The Thirteenth International Conference on Learning Representations.
Itamar Zimerman, Allon Adir, Ehud Aharoni, Matan Avitan, Moran Baruch, Nir Drucker, Jenny Lerner, Ramy Masalha, Reut Meiri, and Omri Soceanu. Power-softmax: Towards secure Ilm inference over encrypted data, 2024a. URU/tttps://aaxxv..rrr/////
Itamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov, Omri Soceanu, and Lior Wolf. Converting transformers to polynomial form for secure inference over homomorphic encryption. In Forty-first International Conference on Machine Learning, 2024b. Forum?id=9HPoJoal
net/
APPENDIX
A USAGE OF LARGE LANGUAGE MODELS
ChatGPT is used only for improving grammar and wording in draft preparation. Authors reviewed and revised all content and take full responsibility for the intellectual content of the submission.
B EXTENDED PRELIMINARIES: THE BERT MODEL
We define the mean and variance of vector as functions from R" + R. We denote the softmax function and layer normalization as functions from 1" + {0, 1)", specifically:
exp(x:) softmax(x) Erejn) exp( xx )E(M)
(x) := [ mean (x)
+ s).
var(x)
where 7.8 are model parameters. The functions softmax and LayerNorm can be directly extended to matrices, by applying the functions to each row of the input matrix.
Embedding. Let the input has m tokens. First it is embedded to m vectorized token embeddings, each of which has dimension d. We denote the matrix X, Rmxd d is also defined as the
13

---

Published as a conference paper at ICLR 2026
hidden size of the model. Let P E Rmnxd be the positional embeddings for the input tokens. Then, the embedding imput is obtained by computing
Xe + P.
Multi-Head Attention. Let X E Rmxd be the input, Wo, WK. Wy ig be weights ma- trices, and bq. bx. by be bias vectors. An attention block in head h [H] is defined as Attention(Q, K, V) softmax QKT va
V E Rdxd", where: Q XWQ + 1Tbq: K XWK + 17b,, V == XWy + 11by. The multi-head attention is the execution of H parallel attention blocks and we set the h-th result to be X where d' := d/ H. Then the H matrices are concatenated X XConeat := (X(() x ... || (())))) E R Rxd The attention block ends with a layer normalization procedure with residual connection: X := LayerNorm(XX + X Coneat).
Feed Forward Network. The feed forward network contains four components. A fully connected layer followed by the Gaussian Error Linear Units (GELU) Hendrycks & Gimpel (2016) acti- vation function, another fully connected layer and LayerNorm. The GELU function defined
as GELU(x)= := 1+ 3 Lo e e- dt). Since there is no dosed-form expression for GELU,
Hendrycks & Gimpel (016) suggested approximation by tanh:
GELU(x)) 0.5x tanh [22/ (a 0.044715x.)
Let WK1, Wiz be weights matrices, and bid1, biez be bias vectors. The feed forward network can be described as follows: X := LayerNorm(X
GELU(XWaa + 1 W + 1 E R The output feed forward block is set to be input of the attention block in the next transformer layer. The BERT-base- BERT-bass-uncased model Devlin et al. (2019) is using H 12 and it has 12 transformer layers, that is, it contains 12 attention blocks and 12 feed forward blocks.
After 12 transformer layers, the model applies linear layer with bias and tanh(-) () the first token ((CLS) as the aggregated sequence representation for classification tasks. Let this vector be X(CLS] E Rd The classification layer first computes tanh(x W + 1Tb) Rd, denoted as Finally let è is the number of classes, the output is xWa + 1T E RR
C DETAILS OF MATRIX MULTIPLICATIONS
C.1 THE BABY-STEP-GIANE SEATTEE--GAAT--SSE STEP METHOD
In Halevi & Shoup 2021 they optimized the algorithm by baby-step-giant-stee method, and the number of HE rotations is reduced to O(ym). The idea is to write m = x b. First for E [b], we compute Rot((v) for b times. To compute rotation Rot, (v) for is >= b, can write & = ab + with r < b, and
rv rotab (rot,,(v)) Combining the above, and with the diagonals of C, we will have
CvT=- = Σ rotas( Σ
rotm-ab (Diagottr(C)) @ rot,
(3)
rE(M
C.2 PSEUDO CODES SECTION
Algorithm describes the detailed implementation of CPMM when X is encrypted in column packing.
Algorithm describes the detailed implementation for encrypted inputs. For the sake of simplicity. we assume 7 N/2 Algorithm 4 now. Then the rotation of a vector with dimension m
can be directly evaluated using one homomorphic rotation. When < N/2 and m Halevi &
Shoup 2014] uses sparse packing, i.c., using m slots with index 0, (,2m) -0eNNNN (2mm). (m
2m). We will elaborate the our method of full packing in Section 3.2
C.3 FURTHER OPTIMIZATION BY HOISTING
Hoisting is technique to reduce the number of decompositions HE rotations where the same
ciphertext will be rotated multiple times. Halevi & Shoup (2018). More concretely, suppose
14

---

Published as a conference paper at ICLR 2026
Algorithm 2 piphertext-plaintettt matrix multiplication Multat, (Enceol (X), W, B).
Input: on (X): Column packing of encrypted X E mRx W: Weight matrix with size d x d'. P: Bias matrix with size m d'.
Output: Enc (XW + B): Column packing of encrypted XW + Rmxd"
Let Ans 0.
2: Parse Enc.ol as Enc(x}), Enc(x)) Enc(x
3: for E [d'] do
4: Let == (0, 0) R2 be naive ciphertext.
5: for E [d] do
6: ct, := Add (ctz, sMult (wij, Enc(xx]))
7:
end for
8: Let P. be the j-th column of P.
9: Ans == Ans U Add(ct,, p)))
end for
11: Return Ans.
Algorithm 3 Ciphertext-ciphertee matrix multiplication: Multcol.coo (Enceot (Q), Enceol (K)).
Input: EQ).at(Q), Enc.et (K): Columa packing of encrypted matrices Q, K E mxdd'
Output: Encaieg (QKT): Diagonal packing of encrypted matrix QKT E Rm
1: Let Ans == 0.
2: Parse Enc.al ()) as Enc(q]), Enc(q] Eno(qg-1)}
3: Parse Enc., (K) as
Enc(k]), Enc(k]).... .... Enc(k], }))
4: for E [m] do
5: ct. : (0, R2 be naive ciphertext.
6: for E [d"] do
7:
j 0 then
8:
temp == Ro (Enc(k?)).
9:
else
10:
temp:= = Enc(k?)
11:
end if
12:
ctj = Add (ct,, Mult(Enc(qa)) temp))
end for
14: Ans == Ans U ctj.
end for
16:
Return Ans.
15

---

Published as a conference paper at ICLR 2026
Algorithm 4 Ciphertext-cippeereext matrix multiplication: Mult,ding,ood (Enc (C), Enc (V)).
Input: Eacadag (C): Diagonal packing of encrypted matrices C' E Rmxm. Enceot Column packing of encrypted matrix V E Rmxx',
Output: Enc.ot (CV): Column packing of encrypted matrix CV e Rmmxd"
1: Let Ans : 0.
2: Parse Encaaiag (C) as EEnc(Diago (nn()ag (C)), Enc(Diagm-1(C))) (C))) Let c; Bnc(Diag,((") for i E [m]]
3: Parse Enc.ol (V) as Enc(v]), Enc(v]]cv... Enc(v]
4: Let b ;= [/m]. m g = [m/b].
5: for j E [d"] do
6: Baby step: For E [b], let B, == Rot fo (Enc(v])).
7: Giant step: Compute the ciphertext of Cv]:
ct
Σ aE19
Rot
Σ ME
(aat((Rot-alCCabtt))) B.))
8: Ans Ans Udj.
9: end for
10: Return Ans.
ciphertext is rotated N times and let m denote the length of the gadget vector. Without hoisting, performing N HE rotations requires N automorphism, N decomposition and N key switching operations. With hoisting, the decomposition is performed only once and reused for all rotations, replacing the remaining (N 1) decompositions with (Nm--V additional automorphisms. Because decomposition is more expensive than automorphism, hositing yields a efficiency improvement.
Now we report the efficiency improvements that can be obtained through hoisting. In summary, hoisting can accelerate the amortized time about at most 2.29 seconds out of the total amortized time cost of 141.3 seconds, i.e., an improvement of 1.62%. In MOAI, hoisting can accelerate rotations in ciphertext-ciphertext matrix multiplications. We tested the time cost of each operation in CKKS scheme using SEAL library SEAL The ratio of time for hwwp:ooooppsmmdddcoossistttoiiii switching roughly equal
40 80. Therefore the portion of the improvement from hoisting 41-m
(N+40N 4-0N) (m ( N400+000
41-m
40
< 121 In SEAL library, m is usually set in the
N+ 1-+40++800
121 121N
range of 10 20 in secure inference of transformers, which indicates a 16% 26% improvement
when the same ciphertext is rotated multiple times.
Further, for fair comparison with existing works Moon et al. (2024); Park et al. 2024b); Zhang et al. (2024a), which do not indicate using the hoisting technique, all comparisons (e.g.. Table Table
2 and Table 7 and experimental results (e.g., Table Table Table 5 Table 9 and Table do not
use hoisting.
D DETAILS OF INTERLEAVED BATCHING
D. NAIVE BATCHING AND INTERLEAVED BATCHING
Suppose we are given (/m2m) vectors (x I= }re[NN(2m)]
( = 1 .1 (0)
(0) (N/(2m)))) = [r] [N (N/(2m) (N/(2m)-)) mn-1
(N/(2m)
the naive idea of batching is to arrange them one by one (c.g.. in Zhang et al. (2024a)):
(0)
(N/(2m)-1) 1] (0) ..... To ..... I.
(1)
(N/(2m)-1) xm
(N/(2m) 11 N/2
We call x the naive batching of N/ (2(mm) vectors (x(r) R (re) For example, consider batching [av. 21. and [yu. //, 82]. The naive batching generates [zo, 21, 22, yo V1, y24. while the interleaved batching generates [z No. 31 V/, 22 y2l. This naive batching method compatible with the hophertext-plainter matrix multiplication (Algorithm 2), but it is compatible with the
16

---

Published as a conference paper at ICLR 2026
ciphertext-ciphertee matrix multiplications (Algorithm and Algorithm 4.. More concretely, this naive batching method is not compatible with rotation. When "rotate" the batched vector (e.g.. Line 8f Algorithm 3 the desired output should be ciphertext encrypting the following "internal
rotations":
[Rot, (k ), Rot, (kt)) Rot, (k(/((3))))))
In other words, it should be rotating each (()) "internally", which is different from directly rotating the batched vector by Rot, (k). Moon et al. (2024) proposes a method to obtain internal rotation, using 2 normal CKKS rotations and plaintxtt-ciphertee multiplications, As toy example, consider batching [To, 2,, x22] and [Jo- y1, 32]. To "internally" left rotate by 1, we should use the naive batching [ro, 21, 22, yo: 1/, V21 generate [21, 22, Tu. V1, y2, yo]- This
be done by Rot, (x) & 1, 1, = [ 22, O, -/2 0]. Then Rot 20t_)(x) @ [0, O, 1, 0, 0, 1] = [0, 0, Zo. 0, 0, yol. By
summing them together we obtain [21, 22; 2o y 1/2 vol.
D.2 PROOF OF LEMMA
Proof. On the one hand, by Eq. [ we have
(2m) (3) == xx
(1)
(1)
(2)
(N/(2))-1)
(0)
(2)
(2/(2))-1)
j+l
(2)
(
On the other hand, Vr [N/(2m)], we replace (r) ) Eq. by Rol, Rx),(x((1))
|0/ (r)
, 2211
Then we also obtain
Rot RolyN/(2m) (x) = [xj (M/(2m)-1)
(N/(2m)-1)
j+1 j+1
j+t
(N/(2m)-11)
This finishes the proof.
D.3 INTERLEAVED BATCHING FOR MATRICES
Similarly, suppose
given N/(2m) matrices (X(), IRmdd [r([ Let X E R be the interleaved batching matrix of (X(()) E R " [re(N/(2m)) The i-th column of X is XT e N/2
which is the interleaved batching vector of all the i-th column of each X(r), for E [N/(2m)]. according to Eq.
X = [1:100
(1)T
(2)T
(2/(2m)
F. Fi.0 .....
(O)T
(1)T (2)T
(N/(22))
1, 2 Ii,
O0T (1)T
(2)
(N/(2m)
i 1 m
x
2,m 1
Now we are can formally define the column packing and the diagonal packing for the interleaved batching.
Definition D.I (Column Packing for Interleaved Batching). Suppose we are given N/(2m) matrices (x(r) E B (re))/(2m)) the interleaved batching matrix of them is X E R xd whose i-th colunn given by Eq. for E [d]. Then the column packing for X a set of d CKKS ciphertexts, denoted as:
Enceot(88 (X) = [En((x]), Enc(x]), Enc(xx
where x E N/2 is the i-th column of X. We say that Enc X) is the interleaved column packing of encrypted (x(r) E pmmdd [ (re]N (r)T 'Here, use o to represent the j-th entry of vector x:
17

---

Published as a conference paper at ICLR 2026
Definition D.2 (Diagonal Packing for Interleaved Batching). Suppose given (2(2m) square
matrices (x(r) Rmxm [N/(M/() Define:
diag(X)= (X)s ERR (RN/2) [.E[m]]
where (iag((); the interleaved batching vector given by Eq. 1 using all the i-th (upper) diagonals of X for each [N/(2m))] The (upper) diagonal packing of set of m CKKS ciphertexts, denoted as:
Encaiog (d)ag(X)) = [ne(diig(XX)) (o)) ((w((nnnd(d)))))).). Enneeii ((m((dia(())mm))))
We use ding( X); 3; as a row vector. We say that Enncaaag (diag( (X) is the interleaved diagonal packing of encrypted {X(r) E [B" (2())
We conclude that the column/diagonal packing for interleaved batching is compatible with all the three homomorphic matrix multiplication algorithms (Algorithm 2 and 4 using Lemma (.2 Note that all the input and output ciphertexts of these algorithms will be encrypting the vectors in the form of the interleaved batching. Algorithm ( [ are the modified algorithms for homomorphic matrix multiplications using the interleaved batching.
For the HE evaluations of softmax (Algorithm [}. layer normalization (8) and GELU, assert that the algorithm with interleaved batching input remains the same. This is because in these algorithms, the rows of the input matrix independently evaluated. In the HE evaluations of softmax and layer normalization, we add permutation of rows of the input matrix, there will be the same permutation of rows of the output matrix. Note that interleaved batching is effectively a permutation of rows of the matrix. Therefore, Algorithm and 8 remains the same when the input is in interleaved batching.
Algorithm 5 pu (Encent (Enncea X), W, P) using interleaved batching.
Input: Eaceat ( (X): Interleaved column packing of encrypted (X(r) E IRddd " (E(N/(2m)) W: Weight matrix with size d x d'. P: Bias matrix with size m x d'.
Output: Enceos (XW + P): Interleaved column packing of encrypted {X + P e
Rmxd' [r([N/(2m)]-
1: Let Ans := 0.
2: Parse Enc.ot (X) as [Ec((x]) (nncxx)),,E,,(((3_)]
3: Let P be the interleaved batching matrix of N/(2m) repeated matrices P.
4: for E [d'] do
5: Let cly = (0, 0) Rt be naive ciphertext.
6: for E [d] do
7:
ct,- = Add (cty, swult Enc(x.)).
8:
end for
9:
Let DJJ E N/22 be j-th column of P, i.e., the interleaved batching vector of N/(2m) repeated
j-th column of P.
10: Ans Ans U Add(ct, p.).
11: end for
12: Return Ans.
E COMPLEXITY COMPARISON
We provide a detailed comparison of the complexity of our matrix multiplications with prior FHE- based secure transformer inference works: NEXUS Zhang al. (2024a (2024)(NDSS (NDSS'25) 25), THOR Moon ct al. (C24))CCS'25) and Powerformer Park et al. 2024')ACL'255
We focus the number of key switches (KS), which refers the total count of relinearizations in ciphertext-ciphereex multiplication, rotations, conjugations and any other operations requiring a KS. The number of KS is counted in one transformer layer in the BERT-base model.
Table 7 we provide a detailed comparison the number of key switches required in the BERT-base model inference. For NEXUS, results are taken from Zhang et al. (2024a) and Table 4 in Moon
18

---

Published as a conference paper at ICLR 2026
Algorithm Cott Enccol using interleaved batching.
Input: Enceol (Q), (n Interleaved column packing of encrypted (Q())
Rmxdd ]rE]N/(2m)]] and encrypted (K(r) E RR }rE[N/(2m)]- respectively.
Output: Encdiey (diag(ddaag)) Interleaved diagonal packing of encrypted (Q()((K()))
mmxm (rexx)r([///e [re[)/
1: Let Ans == 0.
2: Parse Enc (2) as Enc( Enc(a]), 1n
3: Parse Enc. () as
Enc(kg), Enc(k]],.... Enc(
4: for E [m] do
5: Let cty = (0,0) Rg be naive ciphertext.
6: for [d"] do
7:
j > 0 then
8:
temp == R (Enc(k]))
9:
else
10:
temp: = Enc(k!)
11: end if
12: ct, Add (cty, Mult(Enc((()), temp)).
13: end for
Ans Ans U Ul..
15:
end for
Return Ans.
Algorithm 7 Multatag,col (Encasag (Bacc (diag(C)). Enceot (V) using interleaved batching.
Input: Encarag (diog(()) (C)): Interleaved diagonal packing of encrypted {C(r) E Rm Ir((N/(2m))
Enceot(V) (V): Interleaved column packing of encrypted (V/(r e Rmmxd }re[N/(2m))
Output: Enceot (CVV): Interleaved column packing of encrypted {C(r) e gmmdd' )rE)N/(2m))
1: Let Ans := 0.
2: Parse Enc,iing (C)a as (Enc(diag(C)o) Enc(diag(C))) (indaagg@CC)))EE(daa(((()))).))) (nnedo)))).....ddddio(()nnnna.. Enc(diag(C)mm-1) Let
C; : Ein(diaa(C)) (C),) .) for E [m].
3: Parse Enc..t VV as Enc((] (VJ), Enc(VV), Enc(V]_1))
4: Let b : m 9 = [m/b].
5: for j E [d'] do
6: Baby step: For E [b], let B, == RotiT/(2m)(En(()))
7: Giant step:
Compute the encrypted interleaved batching vector of (C"(v("))Teeeveeo
cy = Σ
RotabN/(2m)
Mult a/em (Cabbr))
aE|el
re(b]
8: Ans == Ans U ctj.
9: end for
10: Return Ans.
19

---

Published as a conference paper at ICLR 2026
et al. ),024), and normalized by batch size. The results of THOR from Table 4 in Moon et al. 2024). For Powerformer, equations are derived from Section C.2 and Algorithm 14. with numbers from Tables 11 and 12 in Park al. (2024b). According to Algorithm 14 in Park al. (024b)) Powerformer requires several Block'l Trans, BlockSigma and Block Tau operations before each ciphertext-ciphertes matrix multiplication BlockMult'. All the operations involve (vvk) HE rotations (i.e., key switches this case). The execution time breakdown in Table 5 further validates the results in Table 7 To evaluate all matrix multiplications in a single layer of transformer model, MOAI achieves the least number of key switches.
Table 7:
Comparison of HE matrix multiplications in transformer architecture. The concrete parame- ters are based on BERT-base-uncased model: the CKKS ring dimension. 16 is from THOR. Ao = 64 is from Powerformer.
128, 768, d' 64, 3072, // 12. N is
Key switch
Function
Scheme
Equation
Number
(XW! (b) X E mmxd W. E Raxdd = Q.K.V
NEXUS
2md/
(2m)
768
[he[H]
THOR
2
3(H
2)
+
3d'
192
Powerformer
mdd 3d]
122
N/2
MOAI (ours)
NEXUS
mid' + H)N/(N////// (N/ (4m))
780
Q(h)) e Rmux
THOR
(2 + + m (log c 3)
460
R(h) E RRuxdd
Powerformer
2 (17 + 2) To 3k
486
MOAI (ours)
(mad m)/(N/ (2m))) H
390
[softmax (W)
NEXUS
N/A
N/A
softmax Q(1) (KW)TT) E Rmxm V(U) E Roaxd
THOR
m(logc + 3)/4 m
m+6
4c
492
Powerformer
23 (19 + 10 2) v 3
457
MOAT (ours)
d' d'Cm
(NNNN(22)) H
444
(X || ..... X De
NEXUS
2d 2(N/ ((
9216
X (b) E Bmaddd Wye Rddad
THOR
4 + 2 1)
122
Powerformer
V
2
mnd
75
N72
MOAI (ours)
NEXUS
(2md)) (NN(4m))
1536
XW X E pand Wia E Idxtt
THOR
2:
448
Powerformer
R
2
2
milt N/2
135
MOAI (ours)
0
NEXUS
(2mf) (onf)/(Ce (NN/ (4m))
6144
XWrea X E Roxt W fea E Rexd
THOR
2d
160
Powerformer
132
272
MOAI (ours)
NEXUS
18444
THOR
1874
Total (one transformer layer)
Powerformer
1407
MOAI (ours)
834
DETAILS OF NON-LINEAR EVALUATIONS
F.1 LAYERNORM DETAILS
In this section we give the detailed implementation of LayerNorm(X). The overview of the flow is shown in Figure 4 Recall that for matrix input, LayerNorm is a "row-to-row"" function applied to each row of the input. Given one row x = [m.]] E [d] from X, the i-th term of LayerNorm(x* Bi (x)a(x)) / var(x))
+.3. After computing mean() and var() we can obtain two different forms of layerNorm
NEXUS does not specify how it evaluates (Oftmax(((M so exclude this result for NEXUS.
20

---

Published as a conference paper at ICLR 2026
550
F31
PM
Calumn-Packing
IN meam([Ra (igaa (1.. s,a))
Packing Lager Norm (X)
SIMD
Pius Ruch Mia
Paai Paa
SIMD
LagerNorm Vid
Fap
Piu
SIMD 13
(ar( P1.
Figure 4: The flow of HE evaluation of LayerNorm.
where 8 =i Z44
Y dr; S
Vd Σ (dr; - 5) dd2 + +
(5)
dr, 5
d Σ, (dx; - 5)2 +
(6)
We give the detailed implementation in Algorithm The parameter {1,2] means to choose
Equation 5 or Equation 6 It is for the higher precision of the inverse square root 1/vz. After attention block, we choose = and compute LayerNorm according to Equation After feed forward block, we choose t 2 and compute LayerNorm according to Equation 6
Remark. We point out the problem of the HE LayerNorm implementation of NEXUS Zhang et al. (0024)). Comparing with our Equation and 6 they propose:
dx; (dx; s)z +
Vdy
Note that the input to the inverse square root evaluation is E, S)², which is about ((d)).
In the WERT-base--ncasee model d = 768, which makes Σ, (dz; 5)² extremely large and is out of the supported data range. Therefore their algorithm is unable to compute LayerNorm in BERR-bass-uncased model.
F.2 GELU APPROXIMATION
We approximate tanh(-) Equation 2 using degree-23 polynomial P2s(2:) on interval [-20, 10) which is enough in our experiment. The approximation method is the famous Remez algorithm Segorittm/Rmmzz (1934). It is noteworthy point out that all wyyyooial--pppoximeeee specific interval. As suggested by Cheon et al. (2022). degree- R) polynomial is required when using Remez or other minimax approximation with fixed maximum error on the domain interval RR, R). If the model needs larger input interval, we apply the domain extension polynomial (x)x) = LF 27521,3.2 x from Cheon et al, (2022). In their algorithm. let P(x) ( be the approximation polynomial of tanh(x) on [-B, B}. Then P B (z) take input from larger interval |-LB, LB
method only support
for proper L. (Theorem 4 in Cheon et al. 2022 ). This can be further extended to input interval
[-T. T. B), by using P B.-
(z())
21

---

Published as a conference paper at ICLR 2026
Algorithm 8 HE evaluation of LayerNorm.
Input: Enc X): Column packing of encrypted X Rmxdd model parameter Y, 8 e IR; algo-
rithm type E 21.
Output: Enc,ot (LayerNorm(X))) Column packing of encrypted LayerNorm(X)) E Rmxd
1: Let Ans 0.
2: Parse (X) as { Enexxx) Enc(x
3: Compute Enc(d me)n(X)). where mean(X) E I is a vector containing the mean of each
row in X:
Enc(d - mcan(X) = Enc(x.),
iE|d
i.e,, the i-th element in mean(X) is the mean the i-th row in X.
4:
Compute Enc(d var(X)), where var(X) Rm is vector containing the variance of each
row in X:
5: for j [d] do
6: Compute tmp = sMult(d, Enc(x{)) - Enc(d mean(())
7: Let tmp, == Mult(tmp,ttmg
8:
end forLet Enc(d var(X)) = E, tmpj-
9:
ift = 1 then
10:
Compute ct := sMult z2 Enc(d var(X))
11:
Use the Goldschmidt division algorithm Goldsehmidt (1944);
Qu & Xu to obtain the inverse
square root (1/ (z) of ct, denoted as Clsrroot-
Compute ct, :- sMult( Va clegroot).
end if
14: ift = 2 then
15:
Compute et == sMult( 7 Enc(d var(X)))
16:
Use the Goldschmidt division algorithm Goldschmidt (1944); Qu & Xu to obtain the inverse square root (1/ of ct, denoted as Ctsrroot-
Compute ct2 sMult( clagroot).
18:
end if
for E [d] do
20: tmp; Mult ct2, (sMult(d, Enc(x])) Enc(d mean(X)))) 3.
21:
Ans == Ans U tmp:-
22:
end for
23: Return Ans.
22

---

Published as a conference paper at ICLR 2026
Very recently, Rho et al. (2024) proposed another method to control the data range. They pre-fix the desired data range of the output of each component in the transformer. Then they design punish function accordingly and perform pre-training to the model weights.
We also point out that using the piecewise polynomial in NEXUS Zhang et al. (024a)) to approximate GELU has several problems. NEXUS uses different HE evaluation of GELU which is based the line of MPC-based solution Lu et al. (2023); Dong et al. (203)). More concretely, NEXUS calculate GELU (IR + R) by the following piecewise polynomial:
0
x S S--4
Pa(x)
-4 -4< < -195
GELU(x) =
Po(#)
1195 < 3
x
x > 3
Here Pz(x) and Po(x) are polynomials with degree and 6.
First, NEXUS reports the supported input range is [-8, 8]. We find that this range is not enough for BERT-base-uncased model, and our experiment the range is [-20, 10). Next, their piecewise polynomial relies heavily on determining which interval does the encrypted x fall in. This is done by HE evaluation of the an approximated sign function. For encrypted number æ, NEXUS uses encrypt bits b, (I), (2(x), (2))), (x:) that (x) if and only if belongs the i-th segment of Equation] 7 Ideally, GELU(r) from Equation 7 can be evaluated by the following for x: E [-8, 8]:
0 b.r) + (3(x))b2()) Ps(xr)bg(z) + 3ba(x).
But in fact, each b, approximated bits and it is small value instead of being 0 exactly. This will cause problems. For example, when x = -8, in general Pa(-8) is huge, and thus Po(-8)bg(-8) ePo(-8) is far from O. What is more, the actually data range we observed is [-20, 10), making Po( 20(-20) extremely huge. Theoretically one can choose extreme parameters of the CKKS evaluation of sign function so that the error is small, but this will increase the time cost dramatically.
F.3 REMARKS ON OTHER NON-LINEAR FUNCTIONS
As introduced in Sun et al. 2024) and He al. (024), we must choose approximations carefully, due to the presence of well-known large activations that are critical for maintaining model performance. Compared with existing works NEXUS Zhang et al. (2024a) and THOR Park (2025), we use a different approximation method based on Cheon et al. (2022) which requires less multiplicative levels. Cheon al. (2022) allows our solution to extended to much larger intervals (i.e., wider input range) with very few depth consumption. This also accelerate the non-linear evaluations by starting from lower ciphertext level, compared with NEXUS and THOR. In addition, our framework is designed in plug-and-play manner, and it is fully compatible with substitution of non-linear functions. Any activation function can be supported by our solution by simply adapting the corresponding approximation method.
Another line of works try to replace these non-linear functions in transformer to more FHE-friendly or MPC-friendly ones, such as Park et al. (2024b). A very recent work workDha & Reagen (2025) introduced a layerNorm free solution utilizing HE and MPC. As we can substitute the non-lineat functions, our framework can be easily adopted in Park et al. (024b)) and using ReLU in Tha & Reagen (2025).
G OPTIMIZED SOFTMAX EVALUATION ALGORITHM
In this section we elaborate the optimized softmax algorithm. Figure 5 shows the error of the Goldschmidt division algorithm. Curves in different colors represent different iterations in the Goldschmidt division algorithm. The y-axis is the absolute error while the x-axis is the real data. The absolute error grows rapidly if the input data is near 0. For stable solution and higher precision, have ensure that input data range is not close to O, and need at least 10 iterations in the Goldschmidt division algorithm.
Next, a cost of adding bootstrapping only ct.um after the third line of Algorithm I it can help to reduce the ciphertext modulus in previous matrix multiplications and can significantly reduce the time cost. The detailed optimized HE evaluation of softmax is shown in Algorithm
23

---

Published as a conference paper at ICLR 2026
250
180
33
0.00
A.222
0.00
5.00
0.00
0.012
0.00
1.00
www.realeeeeeooooooooo bort)
(a) Absolute error of Goldschmidt division algorithm in Softmax
(b) Details of the bottom left corner of the figure
Figure 5: Absolute error of Goldschmidt division algorithm in Softmax
Algorithm 9 HE Softmax evaluation: EnCatog (QKT) - ( (o(QKT)
Input: Encaiag (QKT): Diagonal packing of encrypt matrix QKT E Rmxm
Output: Encdiag (softmax(QK3) (T)))) Diagonal packing of encrypt matrix (QQKT) E Rmxm
1: Write Encaiog ( (QKT)& as (Enc(Diago (QKT)) Enc(Diag, (QKT)), Enee..)))
2: Evaluate Enc(exp(Diss, (QKT)) ((T))e)(Diag,((KTT)) by SIMD polynomial evaluation (1 +1+2/0 21+2/2")
3: Add ciphertexts to obtain Cleum = Enc( E, (Diag (Q.KT (QKT)))
4: Bootstrap Clyum to obtain refreshed ciphertext ct' = EEEeE E, exp(Diag, (QKT))
5: Evaluate HE inverse algorithm et' et' using the Goldschmidt division algorithm Goldschmidt
(1964). Let the result ct' su == (nc(1// exp(Diag, (opDiaa,(QKT)) (QKT)))
6: For [mn]. return Mult(ct' sum
Enc Enc(exp( (QKT)))))))) as the i-th component.
H DETAILS OF EXPERIMENTAL RESULTS
H.1 MICRO-BENCHMARK EVALUATION OF SOFTMAX AND LAYERNORM
We compare our implementation of rotation-fre SoftMax and LayerNorm evaluation algorithms with NEXUS et al. (2024a) (NDSS'25) 25) and THOR Park et al. (2024a) (CCS'25).
All experiments are evaluated with same polynomial ring dimension (N = 216) and slot size Intel(R) Xeon(R) Platinum 8480+ processor
(n = 2.) All CPU experiments conducted using a single-thread setting, and all GPU experiments are performed on single NVIDIA A100 GPU, We evaluate NEXUS using open-source implementation"" invoking its SoftMax and LayerNorm modules in both CPU and GPU environments. In contrast, THOR does not provide an open-source CPU implementation nor report CPU results; therefore, only GPU timings are included for THOR.
The amortized time taken for both functions are given in Table 8 evaluation, our SoftMax evaluation algorithm achieves least 4x speedup over NEXUS and 22x over THOR, while our LayerNorm evaluation algorithm achieves at least x and 151 x speedup, respectively.
By eliminating all rotations during
Table Comparison between Softmax and LayerNorm implementations
Softmax CPU time(s)
LayerNorm CPU time(s)
Scheme
Size
GPU time(s)
Size
GPU time(s)
MOAT
128
128
0.62
0.005
128
768
0.38
0.0039
NEXUS (5DSS'25)
25)
128 128
2.53
0.032
128
768
46.7
0.34
THOR (C)S'25)
128 x 128
0.11
128 x 768
0.59
https://githube http:://h.thhbbcccccc/////o///// http:/jjjuabbb/b//ee.aaa
24

---

Published as a conference paper at ICLR 2026
H.2 INFERENCE TIME ON LLAMA3-8B MODEL
In this section, we apply our methods in LLaMA-3-8B model. LLaMA-3-8B is 32-layer decoder- only Transformer with 4096 hidden size, 32 attentton heads (grouped-query attention), werw-fo dimension of 14,336 (SwiGLU), RoPE positional embeddings, RMSNorm normalization, and vocabulary size 128k tokens.
To evaluate LLaMA-3-8B model, we implement rotation-fire softmax with causal mask and R.SNorm evaluations by using the same idea of rotation-free SoftMax and LayerNorm evaluations. SiLU evaluation, similar to GELU evaluation, uses polynomial to approximate it according to Ebell et al. 2025). We incorporate NEXUS's Arg.aax evaluation algorithm Zhang et al.] (2024a) into MOAT total evaluation time. In the extract-token step, retain only the last token and repack multiple ciphertexts into one, producing row-packed ciphertexts consistent with NEXUS's s ArgMax evaluation. The results in Table aggregate microbenchmarks that cover all components required for LLaMA-3-B8 inference. The matrix packing strategies remain consistent, without requiring format conversions or transpositions. We use 8-token input for consistency with PUMA Dong et al.) (2023) and NEXUS Zhang et al. (2024a). Note that comparing our results with NEXUS is not appropriate because: (I) NEXUS's inconsistent packing formats require costly conversions, which are not reflected in its reported performance, as observed in Moon et al. (2024); [im et al. (2025); and (2) NEXUS does not implement all LLaMA-3-4 components, such as RoPE and SwiGLU.
Table 9: Amortized breakdown of MOAI encrypted inference of LLaMA-3-8B with input tokens
Operation in each layer
(Data dimensions) x
GPU time(s)
Description
Depth
(Number of operations)
Pt-ct MatrixMul
Q XW4
17
(R8x4006 R-406 4096)
20.2
Pt-ct MatrixMul
K XWx. V XWy RoPF(() RoPE(K)
-
(IR*x 1096
R40866 1024)
10.1
Pt-ct position-wise Mul
16 +
1006
0.11
Pt-ct position-wise Mul
16 + 15
1881024
0.025
Ct-ct MatrixMul
QKT d(QKT) d' d'QTVVV XW saltfoauu
15 + 14
(R88128 R 12288) x 32
8.6
Casual masked SoftMax
14
R x 32
0.6
Ct-ct MatrixMul
3 3
(R8x8
R 8x 32
0.2
Pt-ct MatrixMul
2 2-1
(R8x4006
x R x
1.3
Bootstrapping
20
R 8
8.8
RMSNorm
Normalization
R 4096
0.06
Rootstrapping
- -1
4096
8.8
Pt-ct MatrixMul
u XW.., V- XW,
14 13
(R 8x 4096
x 4096x16338
x
97.8
SiLU
h h Si)U(u) h h h hW.
13
R 14336
6.5
Position-wise Mul
3
8
0.02
Pt-ct MatrixMul
2-1
(R
x R 14336x4096)
48.8
Bootstrapping
11
8x J0m
8.7
RMSNorm
Only in last layer
21-+2
0.001
Extract token
Only in last layer
2-1 1
(R8x 4096
g1x4086
0.07
Bootstrapping
Only in the last layer
1 - 21 21 2
R 4006
0.015
Pt-ct Matrix-vector Mul
Only in the last layer
2 1
[IR1 4086 x R2 4096 12866128266
0.86
Bootstrapping
Only in the last layer
1 11
B xx1222256
0.47
ArgMax
Only in the last layer
11-11
R28256
2.5
Total
224.53
Accuracy Table repootss GLUE benchmark scores under both plaintext and encrypted inputs. The results indicate that the additional error introduced by FHE has only a negligible effect on inference accuracy in LLaMA-3-8B model.
Table 10: Scores the GLUE benchmarks
Model
Dataset
SST-2
QNLI
RTE
Unencrypted input
94.9
94.2
84.2
LLaMA-3-8B
Encrypted input
94.5
93.6
82.2
25

---

Published as a conference paper at ICLR 2026
Small input length We give experiment result of the small imput length, since the decoding usually comes with length 1. We list the time cost in Table [T] testing on LLaMA-3-8B model. For clarity, we omit operations whose runtime contributes only a small portion of the total computation time (e.g., less than 0.1 second).
Meanwhile, integrating some popular techniques from plaintext transformer inference (e.g., KV- cache, token pruning) with FHE is a useful and interesting direction for future work. A very recent work, Zeng al. (2025) in NeurlPS'25 has integrated secure multi-party computation (MPC) with V--ccche eviction. Zhang et al. introduced token pruning integrated with MPC, which can further reduce the number of tokens in MPC version inference. While up our knowledge, no prior work has explored FHE versions of the above techniques.
Table 11: Amortized breakdown of MOAI encrypted inference of LLaMA-3-8B with input token length =
Operation in each layer
(Data dimensions)
GPU time(s)
Description
(Number of operations)
Attention Heads
+ d'QKT)V
1096 > (RLx128 x 32)
1.57
Pt-ct MatrixMul
XW selfoutput
10966
IR 1096
0.18
Bootstrapping
R 1x-109
1.18
Bootstrapping after LayerNorm
R 1-466
1.15
Pt-ct MatrixMul
u XW.. V XW. h SiLU(u) y = hW,
4096
x R 4096x14336
x
12.22
SiLU
R
0.82
Pt-ct MatrixMul
Xx1336
x
R 143366x4966
5.93
Bootstrapping
R
4096
1.03
Arg.ax
Only in the last layer
128256
2.5
Total
26.94
H.3 ERROR ANALYSIS IN ONE TRANSFORMER LAYER
We divide one transformer layer into parts: (1) Row Row 4 in Table We call it attention layer; (2) Row 5 to Row 8 in Table 3 We call it self-output layer; (3) Row 9 to Row 10 in Table 3 We call intermediate layer; (4) Row 11 to Row 14 in (Table 3 We call it final output layer.
Figure dillusttraed the absolute errors of evaluating the first layer with 44 imput tokens. It shows that error grows slowly along the evaluation and the medians of them fall into [0.01, 0.03) in the final output of the layer, which is consistent with our error analysis of evaluating 12 layers.
I SET UP OPERATIONS OF THE CKKS SCHEME
Finally, we give more details of the CKKS scheme Cheon et al. 2017) The set up, encryption and decryption of the CKKS scheme can be summerized as follows.
Setup(1*). The input is security parameter The output a parameter set parms the scheme, including the ciphertext modulus Q, and polynomial degree N.
KeyGen(parmss The output includes secret key sk, public key pk and other auxiliary data called evaluation keys.
Enc,x(v). Given the vector v E CN/2, the output ciphertext is == (a. Rg. For convenience, we will use Enc(v) when the public key is clear from context. We will use naive ciphertext to refer to (0, 0) E R& which is ciphertext of O.
Deca: (ct). It outputs the vector E N/22 sk is omitted when it is clear from context.
26

---

Published as a conference paper at ICLR 2026
0.14
0.12
0.02
(a) Absolute etror of attention layer
(b) Absolute error of self-output layer
0.10
0.16
0.18
0.16
0.14
0.14
0.12
0.00
0.00
(c) Absolute error intermediate layer
(d) Absolute error final output layer
Figure 6: Box and whisker figure of absolute error in one layer
27


Reviews:
[{"review_id": "GeENj6dsTL", "reviewer": "Reviewer_qFBv", "summary": "This paper address the latency overhead of  homomorphic encryption (HE) in private LLM inference, in non-interactive setting (HE-only inference, instead of HE+MPC hybrid settings). The Authors have developed MOAI framework to reduce the rotation operation (the most-expensive operation in HE) in key  nonlinearities (e.g., Softmax and LayerNorms) and plaintext-ciphertext matrix multiplication, which reduce the end-to-end private inference latency. \n\nThe reduction in number of rotation operations is **substantial**, and also the methodology is very-well explained and it does not required any fine-tuning, and can be adapted to the existing open-sourced LLM.", "questions": "1. Why GELU latency shown in Table 4 is significantly lower than GELU in THOR, even when a 23-degree polynomial is used for their approximation?  \n\n2. If we substitute the GELU with ReLU in FFN, which is shown a promising solution in HE+MPC private inference on LayerNorm-free LLMs [1], then how it would impact the end-to-end performance in HE-only inference ?\n\n\n\n[1] Jha et al., AERO: Entropy-Guided Framework for Private LLM Inference", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Cryptographically secure private LLM inference is an emerging research area but suffers from high latency (in HE-only inference) and communication (in HE+MPC inference) overheads. In HE-only inference latency stems from rotation  and bootstrapping  operations, and the proposed frame work reduces the former to a significant extend. \n\n\n2. The paper has presented a very-detailed latency breakdown (Table 3) and operation-wise comparison with prior work (Table 1 and Table 2; and Table 4 and Table 5), which is **impressive and very useful** to understand the savings coming from reducing the rotations in linear as well as nonlinear operations). The authors also discuss the pitfalls and other shortcomings in the end-to-end implementation, and the assumptions made, in prior works such as NEXUS and THOR.  \n\n3. A detailed discussion and empirical analysis for Softmax approximation is provided in the Appendix G and H.3 (Figure 5 and Figure 6) \n\n\n4. The approach is simple and relatively straightforward to implement, in a plug-and-play manner. Also, the authors have provided the open-source implementation", "weaknesses": "The main limitation of HE-only inference lies in **accurately approximating nonlinear operations**, particularly when compared with hybrid HE+MPC approaches. Polynomial approximations in HE are highly sensitive to the data range over which they are defined. Within a narrow predefined range, the approximation error can remain low, but this restricts the *representational flexibility* of the model. Expanding the range to better capture the activation dynamics requires using *higher-degree polynomials*, which in turn **increases the number of bootstrapping operations** and thus end-to-end latency.\n\nIf we want to enforce a tighter data range for constraining activations to the region where the polynomial approximation remains valid, we need to add the regularization terms during training or we need to fine-tuning the model. For example, the polynomial approximation of LayerNorm in HE-only inference [1]\n\n\nMore importantly, this approximating with polynomial approach may not scale well to LLMs, due to the presence of well-known massive activation [2,3] which are critical to maintaining model performance. Consequently, achieving an accurate approximation would require an extremely wide range, which again necessitates a high-degree polynomial---defeating the efficiency goal of HE-only inference.\n\n\n[1]  Zimerman et al., Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption, ICML 2024\n\n[2] Sun et al., Massive Activations in Large Language Models, COLM 2024\n\n[3] He at al., Understanding and Minimising Outlier Features in Neural Network Training, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rYx6wBBTZ4", "reviewer": "Reviewer_XZwn", "summary": "This paper implements BERT inference using a GPU-accelerated implementation of CKKS. The main contributions of this paper are improved analysis of the data packing for both the linear and non-linear operations in encrypted transformer inference. The main benefit of this improved analysis is a reduction in the number of rotations needed for various operations in the transformer inference .", "questions": "Not all rotations are created equal. When counting rotations, the number of unique ciphertexts to be rotated often matters much more than the number of output rotations. Did you consider hoisting optimizations when counting rotations?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "I like that they provide their implementation. In general, the experimental results are well-documented.", "weaknesses": "The contributions all seem very minor/incremental. There is also no consideration of significant optimizations in the analysis of the rotation counts. These hoisting optimizations, where one ciphertext can be rotated many times at a cost similar to rotating the ciphertext once, can dramatically reduce the runtimes of FHE linear evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8XJXTJxhbX", "reviewer": "Reviewer_pF5R", "summary": "This work focuses on the secure inference of Transformers when FHE is adopted. The author provides a new flow that includes matrix multiplication and nonlinear functions, mainly to reduce the number of rotations. Results demonstrate the effectiveness compared with THOR and Powerformer.", "questions": "1. The authors use a 23-degree polynomial to approximate the GELU function but do not provide detailed coefficients of the polynomial. The high-degree terms usually have tiny coefficients, such as 10^{-20}. In this case, the fixed-point precision is not enough to express these coefficients and would set them as zero. Given that the high-degree terms are usually extremely large, this will cause a very large error.\n\n2. Could the authors provide the comparison of the #rotation with Nexus when only the QK^T and softmax are included? I believe this can better indicate the optimization of section 3.1. I am also curious about the comparison when batching is not applied.\n\n3. The performance on inputs with small input sequences is more interesting in the GPT model, since the decoding usually comes with an extremely small input length (only 1). Providing a comparison in such cases is important to demonstrate the effectiveness of the proposed method.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method that allows the output of QKT in diagonal packing is meaningful; this further makes the rotation in softmax removable. Its correctness is true after careful examination and is supposed to guarantee security.\n\n2. The paper has a clear definition and algorithm.", "weaknesses": "1. The assumption of batching in Section 3.2 is strange to me. Prior works seldom do this, and I only know the batching occurs during the offline stage. Since batching is not a trivial solution in HE, as one client usually queries only one sentence once and queries from different clients cannot be batched.\n\n2. The proposed method seems to rely on the input with a large sequence length. However, in the widely used GPT, the decoding only uses a sequence length of 1.\n\n3. As reported in prior works (Nexus), the time spent on bootstrapping takes around 2/3 of the total time. But this work does not focus on such a major bottleneck.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QNUqdr5CA8", "reviewer": "Reviewer_sUSa", "summary": "This paper presents MOAI, a non-interactive secure Transformer inference framework based on homomorphic encryption (HE). The authors propose a new packing strategy aimed at reducing the number of rotations and format conversions, thereby improving efficiency. To eliminate the overhead introduced by format conversion across layers, they propose a consistent matrix packing strategy. Furthermore, the paper presents a rotation-free evaluation scheme for SoftMax and LayerNorm, as well as a rotation-reduced approach for matrix multiplication. Experimental results demonstrate that MOAI improves evaluation time by 52.8% without sacrificing accuracy.", "questions": "1. Are there experimental results to support the authors' claim about GELU evaluation in Appendix F.2? Specifically, is there quantitative evidence showing accuracy degradation due to the piecewise low-degree polynomial approximation?\n\n2. Given that MOAI is completely non-interactive, could functional bootstrapping [2] be leveraged to reduce the overhead of non-linear functions?\n\n3. Since the Phantom library currently does not support CKKS bootstrapping, and MOAI's GPU implementation is based on Phantom, how is bootstrapping performed in practice?\n\n4. While this work focuses on optimizing FHE evaluation protocols, could techniques from the machine learning side be integrated as well? For example, work elimination [1], token pruning [3], and KV-cache eviction [4] have shown effectiveness in secure Transformer inference. A discussion on the potential integration of such techniques could be beneficial.\n\n[1] Pang, Qi, et al. \"Bolt: Privacy-preserving, accurate and efficient inference for transformers.\", IEEE S&P 2024.\n\n[2] Alexandru, et al. \"General functional bootstrapping using CKKS.\", Crypto, 2025.\n\n[3] Zhang, Yancheng, et al. \"Cipherprune: Efficient and scalable private transformer inference.\", ICLR 2025.\n\n[4] Zeng, Wenxuan, et al. \"MPCache: MPC-Friendly KV Cache Eviction for Efficient Private Large Language Model Inference.\", NeurIPS 2025.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The topic is relevant and timely. Non-interactive secure Transformer inference is important for privacy-preserving applications in communication-constrained scenarios.\n\n2. The paper is well-structured and easy to follow. Prior techniques such as packing strategies and interleaved batching are explained clearly.\n\n3. The evaluation is thorough, including a detailed breakdown of overheads for individual components. The release of the implementation further improves reproducibility and practical impact.", "weaknesses": "1. The technical novelty requires further clarification. The primary contribution appears to lie in the composition and refinement of existing packing strategies. However, several techniques adopted in this paper are not entirely new. For example, reducing the number of rotations via diagonal packing is a well-established approach (e.g., Section 4 in [1]). Applying such a strategy to SoftMax appears to be a straightforward extension.\n\n2. Certain methodological details lack clarity. For instance, in Algorithm 1, the authors describe the evaluation of the SoftMax function, but do not specify the degree of the polynomial used (line 301). Additionally, the authors apply the domain extension technique to approximate the GELU function, but it is unclear why this approach is not similarly applied to the sign function, which would validate the approximation at line 1095. The limitations of previous piecewise approximations are not clearly explained.\n\n3. Some notation and formulations require better consistency. For example, the asymptotic complexities of NEXUS and THOR are essentially the same, yet are presented in different forms. This inconsistency may confuse readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xItql7bEuz", "reviewer": "Reviewer_ycxg", "summary": "This paper proposes a new framework named MOAI, designed to address privacy concerns associated with LLM inference when deployed on Cloud Service Providers. The framework is based on Homomorphic Encryption, enabling non-interactive secure inference directly on encrypted data. MOAI introduces three optimizations: an evaluation flow combining column and diagonal packing; \"rotation-free\" algorithms for the non-linear functions Softmax and LayerNorm; and column packing and interleaved batching techniques. Experimental results claim that MOAI achieves a 52.8% reduction in inference time on the BERT-base model compared to THOR, the current SOTA non-retraining method. Furthermore, its techniques are shown to accelerate the retraining-based SOTA method Powerformer by 55.7%. The paper reports an amortized inference time of 2.36 minutes per input on BERT-base and demonstrates the feasibility of extending the framework to the LLaMA-3-8B model.", "questions": "While this work presents significant achievements, several key aspects require further clarification from the authors, particularly concerning the method's scalability and its reliability in real-world scenarios.\n\n1.  Scalability Regarding Sequence Length: The paper's introduction suggests application scenarios (e.g., protecting \"drafts\" or \"sensitive records\") that may involve long texts. However, all end-to-end experiments in the paper appear to be confined to fixed and relatively short sequence lengths (128 for BERT-base, and only 8 for LLaMA-3-8B).\n    \n       Question: The complexity and error propagation of HE operations are typically highly correlated with the sequence length ($m$). Could the authors provide experimental data or analysis illustrating how the MOAI framework's performance (including inference time) and accuracy (cumulative error) would change when processing longer sequences (e.g., $m=1024$ or $m=4096$)? It is currently unclear whether the performance advantages observed on short sequences can be sustained in long-document scenarios.\n2.  Accuracy and Task Scope of the LLaMA-3-8B Experiment: Extending the method to LLaMA-3-8B is a significant claim of this paper. However, two major questions surround this experiment:\n    \n       Question (Task Scope): The experiment described in Appendix H.2 appears to be a \"micro-benchmark\" of a single forward pass for an 8-token input, rather than a complete autoregressive generation task. Considering the non-interactive nature of the framework and the 224-second execution time for this single step, the feasibility of this method for practical generative chat applications is questionable.\n       Question (Lack of Accuracy Validation): More critically, the paper provides no accuracy evaluation for the LLaMA-3-8B experiment (e.g., Table 9 only reports execution time). All precision and error analyses (e.g., Figure 3 and Table 6) are limited to the 12-layer BERT-base model on classification tasks. Given that LLaMA-3-8B (32 layers) is much deeper than BERT-base (12 layers) and that errors can accumulate layer-by-layer (as shown in Figure 3), how can the authors ensure that the cumulative errors from 32 layers of approximate computations (e.g., for RMSNorm, SiLU, etc.) will not severely degrade the final output quality? We believe the effectiveness of the LLaMA experiment remains ambiguous without this accuracy validation.\n3.  Clarity of Application Scenarios (Amortization vs. Latency): The paper's key performance metric (2.36 minutes) is an \"amortized time\" based on a batch of 256 inputs. This is reasonable for offline analysis tasks.\n    \n       Question: Many privacy-sensitive applications (such as secure search) are interactive and more sensitive to \"latency\" than \"throughput.\" Could the authors provide the latency data for MOAI processing a single input? This would help reviewers more clearly define the practical application scope of the framework.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The paper introduces targeted optimizations for several critical bottlenecks in HE-based Transformer inference, including format conversions, rotations, and matrix multiplications. These optimizations collectively contribute to a significant improvement in the efficiency of secure HE-based inference.\n2.  Unlike approaches such as Powerformer, which necessitate model architecture modifications and retraining, MOAI is designed as a \"plug-and-play\" framework. It does not modify Transformer components, enabling its direct application to existing, pre-trained models and offering superior generality and practical utility.", "weaknesses": "Please refer to my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)