Secure Outlier-Aware Large Language Model Inference
Authors: Lifan Zhao
Zhixuan Fang
Venue: ICLR 2026 Poster (2026)
Keywords: Multiparty Computation
Privacy Perserving Machine Learning
Secure LLM Inference
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Secure multiparty computation allows the client to secretly inference their sensitive inputs without acquiring the proprietary machine learning model weights. As the decoder-only transformer-based large language model becomes the popular paradigm, the desire of applying MPC in large language models is increasing. However, such inference usually leads to great amount of latency, which is due to nonlinear operations in the Transformer architecture. Recent works either focus on improving cryptographic primitives or re-architecting and re-training to make LLM MPC-friendly. We, on the other hand, observe that properly addressing outlier phenomena, which are unique yet universal properties existing across different LLMs, can effectively reduce the input domain and thereby design faster protocols for non-linear operations. Hence, we propose Secure Outlier-Aware Large Language Model Inference framework (SOAL), which accelerates the RMSNorm operation by nearly 2 $\times$, SiLU by $2\times$, and Softmax by more than 5$\times$. SOAL maintains the same performance of the original model without any fine-tuning requirement.

Published as conference paper ICLR 2026
SECURE OUTLIER-AWARE LARGE LANGUAGE MODEL INFERENCE
Lifan Zhao
Zhixuan Fang Tsinghua University Shanghai Zhi Institute z Zang@mmii tsinguua edu. en
Shanghai Qi Zhi Institute zhaolf ac.
ABSTRACT
Secure multiparty computation allows the client to secretly inference their sen- sitive inputs without acquiring the proprictary machine learning model weights. As the decoder-only muusformer-based large language model becomes the popu- paradigm, the desire of applying MPC in large language models is increasing. However, such inference usually leads to great amount latency, which due to nonlincar operations in the Transformer architecture. Recent works either focus on improving cryptographic primitives or re re-architecting and re-training to make L.LM MPC-fricndly. We, on the other hand, observe that properly addressing out- lier phenomena, which are unique yet universal properties existing across different LLMs, effectively reduce the input domain and thereby design faster proto- cols for non-linear operations. Henee, propose Secure Outlier-A ware Large Language Model Inference framework (SOAL), which accelerates the RMSNorm operation by ncarly SiLUU by 2x, and Softmax by more than 5x. SOAL maintains similar performance as the original model without any fine-tuning re- quirement.
1 LNTRODUCTION
As large language models (Radford et al. 2019, Gupta et al. 2022 Jiang et al. 2023) have demon- strated remarkable capability in variety of aspects, the concemn leakage of private sensitive in- formation to cloud-based LLM inference services increasingly raised. On one hand, precious business data, and sensitive personal data processed by model providers' servers without any privacy protection. On the other hand, L.M model weights, which tokens and millions GPU-hours of training, are uploaded and stored on third-party servers, count- ing on their pledge. Secure multiparty computation (MPC) offers a solution where L.LM inference services can be offered without leaking any information to either users or model providers. How- ever, making the I.IM inference with an MPC protocol is notably slow. Inferencing 64-token input once using Llama2-7B (Gupta et al. 2022) via CrypTeen (Knott et al (202T) would take 169.76 76 seconds, while it only takes less than one second in plain text. Such latency hecomes even greater as the input sequences gets longer, i.c., would take more than 428.95 seconds for 512-token input. Such inefficiency stems from preprocessing-basee MPC frameworks' difficulty in handling nonlinear layers, including Softmax, Activation (SiL.U or GcI.U), and Normalization (Layernorm or RMSNorm).
the product of trillions of
The inherent complexity of non-lincar protocols comes from the disproportionate output growth across a large input domain. Meanwhile, activations during LLM inference may vary by many orders of magnitude. For example, activations for normalization layers Llama2-7B (Gupta al. [2022) vary from 10 5 to 10', as shown in Figure 5b For non-lineat protocols covering such a wide input domain, directly using a lookup table (LUTT) protocol would require a LUT with 32- 92-bitwidth. As
LUT protocols usually have a complexity of O(2"), it very expensive to compute, Other methods,
like Goldschmidt's 's method (Goldschmidı, 1964) would similarly consequent in more iterations.
Observing such flaws, previous works (Mohasscl & Zhang, 2017 Cheng al. 2023 Luo et al.
2024 have tried to substitute non- lonerr operations with
ones. SecureML (Mohassel
Conresponding uuthor.

---

Published as conference paper at ICLR 2026
& Zhang 2017) used Softmax(2) 5C Re. to avoid exponential computation. MPCFormer
ReLU(())
[Cheng et al. 2023) substituted (a) with Quad 0.125z² 0.25z + 0.5. Although they exploited the robustness of LLMs such that the model may still be functional after substitution, these methods face substantial accuracy degradation and require additional fine-tuning to compensate quality discrepancy. Such methods modify the original model, induce training overhead, and bring doubt regarding in the validity and quality of the new model.
Motivated by above challenges, we ask the following research question:
Can achieve efficient non-linear protocols specifically designed for MPC L.M inference?
critical insight: that LLMs, espccially decoder-only transformer- based LLMs, have many unique properties shared in common, which have been utilized in variety of domains. Dettmers et al. (Dettmers al, 2022) observed that most acti- vations and model weights have a skewed distribution, which means their values are concentrated in narrow range and only very few activations are significantly large. By handling outliers separately, they succeeded in making inference with FP8, one-byte data format for floating point number, rather than FP32. This method effectively lowers the requirement for GPU memory without sac- rificing much performance. Inspired by their observation, we study non- non-linear layers and find that similar outlier distribution phenomenon also persists. Figure [ demonstrates the layer-wise activa- tion distribution for normalization layers, where most activations aggregate around a certain value with small variances. Meanwhile, there ity. It the existence of such outliers that increases the complexity of MPC non-linear protocols.
Mcanwhile, another line of recent research offers
few outlier activations that deviate far from the major-
Our approach and contributions. We propose Secure Outlier-A Awre Large Language Model In- ference framework (SOAL). two-stage framework which addresses the unique outlier features in the decoder-only transformer-baeed L.MMs and optimizes non-linear protocols without compromis- ing LLM performance. In detail, our contribution is threefold:
We discover different outlier phenomena existing in activations for non-linear layers across various LLMs. We further address such phenomena by prefixing special tokens to users' input prompt to manipulate the occurrence of outliers. Our evaluation shows that such modification brings no performance degradation.
We design new protocols for normalization and activation layers, which benefit from the control of outliers. Based on collateral feature from prefixing tokens, we improve Softmax protocol as well. Our protocols can work with diffcrent cryptographic schemes.
3. Our SOAL achieves nearly 2x speedup when inferencing Llama2-78 and GPT-2 with 512 tokens under the additive secret sharing scheme. while maintaining the accuracy of the output. SOAL also obtain efficiency improvement both online time and total keysize under the function secret sharing scheme.
2 RELATED WORK
2.1 PRIVACY-PRESTERVING MACHINE LEARNING.
MPC originates from the Billionaire problem (Ya 1986 Shamir 1979) whose goal is to enable multiple untrusted parties to jointly compute a function while keeping the inputs private. As LLMs show their incredible ahility, recent works focus mitigating MPC with L.M inference only. They primarily follow two complementary directions. The first focuses accelerating secure inference through advanced cryptographic primitives. For instance, systems such SIGMA (Gupta et all 2024) and BOLT (Chen et all (024b) leverage function secret sharing (FSS) and vector oblivious linear evaluation (VOLE), respectively, reduce communication and latency. Others, like Iron (Hao et al. 2022h. PrivFormer (Dalskov et al.] 2023), and PUMA (Dong et al., 2025). optimize secure matri multiplications and nonlinear layers using a mix of secret sharing. homomorphic en- cryption, and polynomial approximations, demonstrating practical performance on full-scale trams- former models. Innovations like BumbleBee (Huang et al] 2025). SHAFT (Kei & Chow, (2025), and NEXUS (Zhang et al. 2025) further improve scalability by introducing protocol-level opu- mizations for core transtormer components, including attention, GELU, and Softmax. In parallel, second thread of research modifies transformer architectures better suit MPC. Works such as
2

---

Published as conference paper ICLR 2026
MPCFormer (Cheng et al. (2023 and SecFormer (Luo et al. (024). based on additive secret shar- ing (ASS) scheme, substitute costly nonlinearities (e-g.. Softmax and GELU) with MPC-friendly alternatives like low-degree polynomials and train these altered models using knowledge distilla- tion, achieving significant efficiency gains minimal accuracy loss. However, the requirement of knowledge distillation may change the model weights, thereby hindering its broader usage.
OUTLIERS IN LLM ACTIVATION
In this work, we specifically focus on decoder-only transformer-busee large language models (Rad- ford et al.] 2019; Gupta al. [2022] Jiang al. 2023). During L.L inference, the activations for each layer, no matter what input prompt is, always contain a few activations, known as out- liers, that stray far from others. On the other side of the coin, the rest of the activations cxhibit skewed pattern concentrated in a narrow range. Outliers in LLM inference have drawn much atten- tion recently [Bondarenko et al. [222 Wei al. (022) Puccetti al. [0222] Dettmers et al. 2022]
Xiao et al. 2023b [Sun et al. 2024). Some of these activations related particular features, 2022) showed empirically that occurrence of outlier features related the frequeney of tokens the training distribution.
which known outhier features. Puccetti al. (Puccetti
Dettmers et al. (Dettmers et al.] (022) showed that the scale of an L.M is related the emergent properties of these outlier features, and utilized such features to effectively quantize model weights. Other activations may jointly pertain to specific channels and tokens. Sun et al. (Sun et al. 2024] introduced the concept of "massive activations" - rare but extremely large activation values that appear consistently in special tokens (e.g., at begin of sentence (bos) or newline tokens). Chen et al. (Chen al. (2244)) further utilize such relationship propose prefixing method and make INT4 quantization comparable to other dynamic per-token quantization methods. Besides linear layers' activations, (Xiao al. 2023a 63 [Bondarenko al. 2023) reveal the attention sink phenomenon, where those related the bos token tend to be larger. this work, extend the study non- non-lincar layers and reveal such outliers not only persist, but also can be used to optimize MPC protocols.
3 OUTLIERS NON-LINEAR LAYERS
In this scction, we extend the LLM outlier study from the lincar layer to non-linear operations in transformer-base LLMs, SoftMax, Normalization and Activation. Given an input activation for non-linear operations, outliers are a small group of values which greatly deviate from the average. Outliers require larger input domain for non linear MPC protocols, which consequently need either more bits to handle more iterations to approximate.
We use RedPajama (Weher et al., 2024), a dataset released after the chosen LLMs, to run 128 inferences with different token lengths and collect activations for different non-linear operations. We conduct additional experiments using more LLMs and different datasets in Appendix A We identify different outlier phenomena corresponding different non-linear layers. Such outhier phenomena exist various datasets, showing that these ubiquitous features belonging to the decoder- only transformer-hsser L.I.s.
GPT-2
(b) Lluma2-7B
(c) Special Tokens Llama2-7B
Figure 1: The activations distribution for normalization operation in (a) GPT-2 and (b) 1lama2-7B. We use bold line for better visualizing the deviation from majority values. (c) Special tokens closely related to the outliers.
3

---

Published conference paper ICLR 2026
(a) Llama2-7B Origin
(b) Llama2-7B Prefixed
(a) GPT-2
(b) Llama2-7B
Figure 2: Activation Distribution for SiLU.
Figure 3: Activation Distribution for SoftMax.
Normalization We visualize the distribution of activations for each normalization layer in Figure M The long tails in most layers indicate that there exists small group of activations whose values gready depart from the majority. It is the existence of these outliers which greatly extend the domain requirement for reciprocal square root protocol design. Fortunately, the occurrence of these outliers shows strong connection with special tokens. As shown Figure [Te in Llama2-7B. outliers locates the positions where the token =. (period). "nn" (newline) and "(s)" (begin of sentence (BOS)) appear. Such observations align works (Sun et al.] 2024) Xiao et all, 2023a).
the outlier findings linear layers from previous
describe detail method to extract special tokens and address these Special-Tone Token Outlier later in Section 1.1
Activation We display the input distribution for Activation layer in Figure Values aggregate and form spikes around zero where the activation function most sensitive, while outliers uppear in first and last few layers. We additionally show that outliers in activation layers are also affected by the aforementioned special tokens in Figure [b
SoftMax We further explore layerwise activations distribution for SoftMax Higure S The majority of activation values concentrate around their layerwise average values, which also matches the observation in Decosiinc+ng+++ (Honn ct al. 20244 that >> 99.99% oy are within certain range. We will later demonstrate that the outliers can be effectively captured by Conformant Maxima- the local maxima discovered, which are derived from values guthered from predefined locations within attention logits.
In conclusion, these different yet consistent observations on non- non-linear layers point toward an answer to the question posed carlier, that if we can address these outliers properly, the skewed distribution of the rest activations be utilized to design better protocols for these non- non-linear operations.
METHOD
Preparation Stage
Inference Stage
Owneed LLM
GOAAL P Protocals
A
HRUP
Model Provider
User
Model Providor
Macel Weight
Cachaa of
Mare
Encrest
Random Dato
Special Toburs
H
Figure 4: SOAL Framework. In preparation stage, model providers collect activations by perform- ing extra inferences with their LLMs. They then obtain special tokens and auxiliary parameters yp from these activations. In the inference stage, model provider and user follow our new protocols to efficiently perform MPC LLM inference.
SOAL focuses the standard two party computation (2PC) with trusted dealer (Beaver [99] Bendlin al. 2011 Damgard & Zakarias 2013) that has received significant attention in privacy-
4

---

Published as conference paper ICLR 2026
preserving LLM inference Guppa al. 2022, Kottt et al. 2021 Jawalkar et al. 2024 Luo et al. 2024] Kei & Chow 2025. That is, there two semi-honest (honest but curious) parties, Po the who has the imput prompt z0 and as the model provider who offers the inference service with model weights #1. In the offline phase, a trusted dealer offers correlated randomness to accelerate online inference. Together,
2PC want to perform the computation public function = F(to, 21) without revealing anything more than the function output to each other. Detailed preliminaries about MPC and LLM are provided Appendix B
In general, SOAL a two-stage framework as shown Figure detailed in Section[4.] the model provider utilizes previous observations on outliers in non-linear layers and prepares model-specific auxiliary parameters used for secure inference. In the inference stage, which we describe fully in Section [4.2] the model provider and the user follow the new designed non-linear protocols to perform faster secure LLM inferences.
the preparation stage, as
4.1 PREPARATION STAGE: ADDRESS OUTLIER
4.1.1
SHECIAL-TOKEL OUTLIER
Ogg
Orga
34333
47
11
L1
135
125 12T 120 (2 DE 120
(a) GPT-2
(b) Llama2-7B
Figure 5: Comparison of layer-wise normalization activation distribution violin plot. Blue half represents original distribution, and green part is the distribution after prefixing special tokens.
As mentioned ahove, the outliers in normalization layers not only cxist, hut also occur along with special tokens. To identify the special tokens, we use the activations for pre-layer normalization and acquire the special-token outliers us follow:
Wc collect activations X E NxLx through N inference for cach layer. We then statistic token- wise maximum value, M E ITT,, where T is the tokenizer size. For i-th token whose ratio of its maximum value over the median maximum value of all other tokens exceeds threshold 1). such as 8 here since
Mr 7), record the corresponding token special token. We want to shrink the input domain as much as possible.
modian
Now, with the special tokens at hand, we can address these outliers normalization layers. Since the outliers appear where these special tokens first emerge, we prefix these tokens ahead of user's 's input prompts to manipulate such outliers appearing only at the prefixed positions. To visualize the effect, we inference twice with the same prompt, except that in the second time, prefix these special after prefixing the special tokens, the green
tokens ahead of that prompt. As shown in Figure parts longer have loig-tiiss in any layers, which means that the outliers have been limited within those prefixed positions. What's more, due to LLM''s 's autoregressive property, these special tokens can stored the format of KV-cache (Radford al. 2019) to avoid redundant computation, and use as the model-specific auxiliary parameters during inference stage. Compared to the overhead of fine-tuning the entire model for different tasks, such a loken-identiffing process can be easily done by the model provider offline within a few minutes.
4.1.2 CONFORMANT MAXIMA
Outliers in SoftMax conform with pattern that we can utilize. We observe that the maxima of softmax input present connection with special tokens. Given the collected softmax inputs e RNXLXL from N inferences of L-token length prompts, present heatmap with the shape L x L Each grid therefore represents for one position in We then the grayscale to
in Figure 6
5

---

Published as conference paper ICLR 2026
Offersree
Rizt
(a) Origin
(b) Prefixed
(c) Distributionof Maxima Positions
Figure 6: (a) Heatmap of origin maxima positions of Llama2-7B. Grayscale means the times maxima appear at that position. (b) Heatmap with prefixed special tokens. (c) Pic chart of the occurrence of exact maxima. For rest of maxima locating in other positions, conformant maxima close to them.
Algorithm 1: Secret
Reciprocal
Square
Algorithm 2: Secret
RMSNorm,
Root, Hrag
Input: For (0, 10,13,2 P. holds the shares
Input: For o E {0. (), Pa holds the
[x]
input share [=] and scale
Output: For a E 1). P. holds the shares
shure shure]pp.1 [pp.--
M where y RMSNorm(x).
Output: For a C 10, holds the
Ix'
[x].
shares [y] where
Txall [[quec([=']e]]
y= RMSNorm(x).
9 [agle :- I=ma(<='.. [+2]o))
Imma[[a [aa][[[@)a [.)
4 [0]. a b[zz]. claaall deailo
a [2 la == 11 Meggane [=
foreach lter < do
[d]
Luile- = 1ol- (8 Irl. WBy/
[d]]
Ioa [d]
[D] := Immal[d [T)
7 return Iy]o;
return [yle:
represent the number of times maxima locate at that position. Without prefixing, maxima may appear where special tokens first locate; therefore, there are
random vertical lines in Figure
G(a). Meanwhile, with special tokens prefixed. the maxima concentrate at the position of "(bus)" tokens and first one and last two tokens each row. Such phenomenon has been identified as attention sink, by StreamLLM (Xiao et all, (023b). They pointed out that certain token may receive more attention than others due to the occurrence of these tokens in the training datasets. Such phenomenon gives us a potential way to simplify maximum finding operation in SoftMax.
We record the exact maximum values Tmax E R N Wx softmax inputs from different infer- enees. We then collect the activations at (1) where the hos token locates, (2) the first token position, and (3) the last two positions of each row. The local maxima are selected from these activations, and we call them Conformant Maxima. As shown in Figure over 90% exact maxima's positions fall in these predefined positions. What's more, for those maxima mismatched, the differences be- tween the exact maxima and our conformant maxima are also small. Hence, by prefixing special tokens and collecting conformant maxima, we avoid the O(n log n) computation to find the exact maximum over n-taken length input.
4.2 INFERENCE STAGE: OPTIMIZE MPC PROTOCOLS
The bond between addressing outliers and optimizing MPC protocols lies in input domain reduction. In this subsection, we demonstrate our new non non-linear protocols under ASS scheme. The protocols under FSS scheme will be introduced in Appendix E
4.2.1 NORMALIZATION
Normalization is used between multi- head attention layers and feed-forver forward layers keep activa- tions from exploding. For the input X RExF, where F is the feature dimension. RMSNorm
6

---

Published as conference paper ICLR 2026
Algorithm Secret
Sigmoid for L.M..
Algorithm
Socret
Softmax
LLM.
Haymo
Inpert: For a E 10, P. holds the inpat
Input:
For o E (0. Ps holds the shares
share [=],. and [vdle.
[ai..
Output: For 0 (0. 13. P. holds the shares
Output: For o {0, 13, P. holds the shares [5]e where softmax(z).
[t]- where y =(x)).
[e]n = -Koga e- [=].
I+
I. (x") [xyl=
-
To ( [: Leglo [ogll)
Polo [volo 2- [6 Pih 2
[oa]= : Mem Iomee lollooleeei IolL
[e]. [=..)==11.)
[e [. [n....mm
10 [[.).
Tell- == [. 1o.[[[ll--ol--- [x]]-)
[mle. [. == (lsold)
[4. "1 11
[ol.:- le 16 1
[o] [lsa[[a'l.), where the initial guess
where
[c]- [ale [e
J. (oi-))
11.15 -0.5[d"]=- Two iteraiions.
bic That M1 [male)
M Heuu[]e],.Hoaa[(e [nde)).
return
[yla:
return jolei
of each row x is calculated with 7
Σ₁- (a,))t + 8, where B are fixed parameters. The
complexity lies in the reciprocal square root function which need multiple iterations to approximate.
We demonstrate our new protucol based on an approximation algorithm in Algorithm 2] As shown in Figure the activations concentrate closely around their layerwise centers after the special-token outliers have been constrained in the prefixed token of KV-cache. Such concentration centers shift according to specific layers, from scale of 10 to 10', To further shrink the input domain, scale the activations with layerwise secret values [y]].. and such scaling later be canceled Now the 7 V 2:f (
via the division. Thus, we change the RMSNorm(x)
input domain for the reciprocal square root has been reduced, we can use third-degree polynomial sentering at J = to get the initial guess. The coefficients a, b. d are acquired by BEGS (Broyden (1970) with MSE loss, and are set to = 0.913389 = -0.860195, c = 1.028723, d -0.359 in this work. With the proper initial guess, we can use only two Newton-Raphsor iterations to obtain the precise result.
4.2.2 ACTIVATION
Activation is designed to bring non-linearity LLM models. For the input X E Rx F, the SiLU function is calculated with SiLU(x)) (xx, where (xx) - is the sigmoid function.
mentioned above, the outliers of SiLU inputs exhibit connection the special tokens. shown in Figure 26 with prefixed special tokens, the range of input domain becomes acceptable for efficient MPC protocol design. Again, the calculation of prefixed outliers can be omitted by the 2--((1-6)
K KV-cache. We now reformulate the sigmoid as (xx) = 2- 1-=..-t:42 --5, 7.2-,. where z;; zy represent integer and fraction part of x and sign g = 1f2 03 if is less than zero, otherwise 0. Hence, we implement the sigmoid protocol Algorithm F
We first convert the exponential base to and local truncation protocol to get fraction [ and integer parts [= As the fraction part small, each party can use the owned share zy, calculate local exponential result 2 which can further re-shared secretly with the two predefined zero secret share [Po] and [r].. With one multiplication, we obtain the [2*] []. Now, we can further simply the equation as b, vy 5., where
2", As [a.]lo is an integer with small input domain, we can use LUT protocol to get Y1 and Vi. both which depend on the integer part. Then, in Lines 6-7, we obtain the denominator by adding U/ and U2 गy together, whose range lies between (1, 2.5). Given the denominator's 's domain is small, we use a second- degree polynomial approximation to get a decent initial guess of the reciprocal in Line 7, where 1.830181, f 1.078845.9 g 0.205-169. At last, we need one more round of Newton-Ruphson iteration and multiply oi back get the sigmoid result.
2 (1-x
2 and
4.2.3 SOFTMAX
Softmax is the key element in Transformer structure. Given X E RLXL where is the token length, the softmax of (2o, #1,,,, 2:), the i-th of X. is calculated with softmax(x)
7

---

Published as conference paper ICLR 2026
Algorithm Secret Exponetial. Hep
140
Gnpha
Input: For a C [0, holds the shares
[a]... and [Ro]o. [Mt
Secformer
5241
Output: For c 1), holds the shares
100)
[+] where exp(a).
[e'].. 1og2e-[x'] [='].-
[ode [luu:([=']
[syle 1= [0']e 1- [R.]
Pollu [Mu]a 2- [p [p]]
2%
[m]- == [on[[ooe.LLlee..
[+ [
120
10M
Tssaa [m]]-. [1]0]
Secuonce Length
return
[sle:
Figure 7: GPT-2 Inference Speed Compar-
ison
Model
Method
SotIMax
Normalization
Activation
Total
(G)
Comm.
Time
Comm.IDD
Comm/GGB
GPT-2
Cryplenn
SOAL
30.79
6,80
52.54
4,52
2.99
3.07
0.88
1.18
10.89
9.17
15.26
19.68
23.30
48.86
24.64
76.79
Hama2-78
Cryplen
199.99
375.03
27.86
12.25
87.30
169.31
428.95
702 44
16.01
Mixtral 8x78
Cryplen SOAL
242.98
39.66
376.01
32.45
65.36
31.25
24.31
32.25
266.52
104.44
161.11
441.00
1104.46
668.23
1611 222
964.50
Table Time and communication cost for secure inference of 512-token prompt. Average of 5 runs.
E)=1 exp(a, max(x))) Among the three obstacles in calculating Softmax under MPC setting max-
(xp[x-maxxx)
imum, expunential and reciprucal finding the maximum is the most unique one, its communi- cation cost grows (n log n.) log-linearly according to the actual input length n.
avoid numerical exploding problem caused by backpropagation during training, but, the maxima can mathematically he replaced by any arbitrary number during inference. exact maxima for most of the time, meanwhile the difference small when mismatched. It is proper candidate for 7 to avoid the log-linear complexity of exact maximum finding. Hence, our softmax protucol can now handle any length of input, always with fixed communication cost and rounds.
The common practice of subtracting the maxima
mentioned Section [.122] conformant maxima matches
We present our protocol for Softmax in Algorithm M Noticing that the number of reciprocal op- eration is much smaller than that of exponential, we directly get the most significant bits of the denominators. Therefore, by multiplying m with denominator d, we scale the denominator to small domain where the reciprocal can be obtained with two NR iterations.
As our conformant maxima no longer promises the non-positive property of the SoftMax input, we display new exponential protocol for the secret exponential function that can handle both positive and negative values in Algorithm S We first local truncate the [x>] by precision bits plus extra bits get [[s]]l and [zylor where
2 + xy. Due to the product of powers rule, each party locally calculate the exponential result of their share / hand. Afterward, they secretly multiplication,
reshare the result /= [Po]. or [P1]o- With one of 2",, Then, with the help of extra s bits, we can use a smaller look up table cover wider input domains. Additionally, since we have known remaining posilive inputs are fairly small, as shown in Figure(6c] we can shift the input range of the look up table to cover more negative input domain. Finally, by multiplying them together, we achieve the exponential results.
can get correct result
5 EXPERIMENT
EXPERIMENT SETUP
Setup. We choose L.Ms including GPT-2 (Radford et al. 2019), Llama2-7B (Touvron ct al. 2023). Mistral 8x7B (Jiang et al.] 2023), all of which are decoder-only transformer-bast based language models. We include Mistral 8x7B to demonstrate SOAL framework can work on LLMs with different struc- tures, i.c. Mixture of Expert. Note that, although BERT is commonly used in previous works, it is an decoder transformer that does not present the phenomena mentioned above.
8

---

Published as conference paper ICLR 2026
Tokens
Time
CommiGR)
Key Size (GB5
Signa
SOAL
Signa
SOAL
Sigma
SOAL
1.097
0.852
0.370
0.36
6.806
5,938
128
19366
1,493
0.824
0.783
14.292
11,915
256
4.166
2.9%
1,883
1819
33.191
25.873
Origin
SOAL
512
10.048
7.577
5.302
4.648
86.686
61.830
Are Challenge 1
0.4334
0.4343
1024
25.136
18.486
15,949
13.341
256.449
165.093
Are Easy
0.7635
0.7618
HellaSwag
0.5713
0.5730
Table 2: Comparison of time, communication cost and
PIQA f
0.7807
0,7769
size of generated key for Sigma and SOAL on GPT-2.
Winograde
0.6938
0.6993
GPT2
Llamn2-78
Mixtral 8x78
PPL/WikITeX) 4
5.55
5.58
Origin
Origin
SOAL
Oriinn
SOAL
Wiki
36.25
37.50
535
5,58
6,422
6.547
Table 4:
Evaluation of Llama2-
33.55
6.126
6.33
3930
7B performace using SOAL frame- work.
Table 3: Perplexity C4 and WikiText2 datasets across
different models.
As the goal of SOAL is to show the effectiveness of integrating these ubiquitous phenomena of LLM inference with MPC protocols. we employ ASS and FSS as foundational schemes. For ASS
scheme. mainly use Cryp'Ten (Knott al. (02T) and adopt truncation protocols from (Santos
al. 2024] 2024 We also include MPCFormer (Cheng al., 2023 and SecFormer (Luo al. 2024 with their codes yet changing the truncation method as well. For FS$ scheme, use the Sigma (uupta et al. (2024) protocol. Our experiments are conducted on two Nvidia A100 GPU with 10GB/s handwidth. the experiment, we use (869, 29901, 22550, 5167, 2073, 29928, 13, 1] for Llama2-7B, [17405, 628, 198, 50256] for GPT-2, and (28705, 13, 262, 1] for Mixtral 8x7B.
5.2 EFFICIENCY OPTIMIZATION
To evaluate the efficiency of SOAL, we record the time and communication cost to inference one to- ken given 512-token prompt in Ton different LLMss As the cost of non-lincar protocols dominates the total cost. With the help of SOAL,, the improvement in the non-linear operations significant. We gain about 2x speedup in SiLU/GeLU operations. We achieve more than 3x faster in SoftMax operation as SOAL can find conformant maxima within constant round. For normalization, although CrypTen only sets Newton- Raphooo iterations its reciprocal square root protocol, and iterations of Taylor Expansion for the exponential used for the initial guess. the corresponding input domain only covers (0.1, 1 15.6) with < 0.1% error, whereas the actual input domain always exceeds such a narrow range. Hence, we follow CrypTen's suggestion, modify the protocol's 's dcfault configuration to get precise result where we use 0.02 as the initial guess and use 11 Newton- Raphsson iterations. On the other hand, our SOAL framework elim- inates the outliers and uses the provided scaling secret [el.o to control the input domain. Therefore, only need two iterations. In total, SOAL shows nearly 2x efficiency than the original CrypTen protocol across different L.Mss.
see, in the original CrypTen,
As the non non-lines linear overhead increases quadratically when the input length gets longer, conduct the experiment show the input length's 's impact L.M inference in F As one can see, due to the benefit of softmax, the longer the input sequence is, the more time can be saved by SOAL. Meanwhile, our method achieves similar speedup as SecFormer (Luo et al. 2024), and becomes faster than MPCFormer (Cheng et al. [003) when making inference with longer prompt. is worth noting that we do not require any fine-tuning effort.
SOAL also works on other cryptographic primitives. Table 2 shows SOAL's inference result with Sigma. Although SOAL saves minor cost on the communication. it largely reduces the key size, which need to be transferred for each inference. We as well improve the online inference time.
5.3 PERFORMANCE EVALUATION
We use Huggingface's 's Evaluation library to evaluate SOAL (Llama2-7BB with different benchmarks, including Are Challenge, Are Easy (Clark all, 2018). HellaSwag [Zellers el al. 2019) PIQA Bisk
c al. 2020), Winograde (Sakaguchi et al. 2021 We also use PPL. on C4 (Raliel ct al, 2020 and
'The origin implementation of CrypTen uses local truncation, which has secure flaw pointed out by LLi 2023)
9

---

Published as conference paper ICLR 2026
WikiText-2 (Merity al. 2017 evaluate other models. Even though MPCFormer and SecFormer fine-tune their models for downstream tasks, their method primarily have 1-2% performance degra- dation reported in their pupers. We don't include their results as they only finetumed the BERT, and GLUE benchmark (Wang al. 2018) designed for encoder-decoder LLM.
The performance remains almost the sume the original Llama2-7B. since SOAL only slightly changes the user input instead of changing the entire model weight. Additionally, perplexities over different LLMs are provided in Table The perplexities are close the origin result, showing that our framework has the robustness to work with L.LMs different structures.
CONCLUSION
We present SOAL,, an efficient MPC framework optimized for decodcr-only L.MMs. We offer new direction to improve MPC LLM inference, which involves utilizing LLM's unique properties. lincar layer, to non-lincar layer.
We extend the study of outliers, well-known phenomenon SOAL addresses these outlier phenomena and reduces the corresponding input domain. Therefore, we design new non-linee non-linear protocols based on and gain nearly x speedup given long inputs on different LLMs. We believe there exist more unique features in LLM, waiting to be explored and exploited.
This work is supported by Shanghai Qi Zhi Institute Innovation Program. The work of Zhixuan Fang is also supported by Tsinghua University Dushi Program.
REFERENCES
Donald Beaver. Efficient multiparty protocols using circuit randomization. In Annual international cryptology conference, pp. 420-432. Springer, 1991.
Rikke Bendlin, Ivan Damgård, Claudio Orlandi, and Sarah Zakarias. Gemi-homomorphic encryption and multiparty computation. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 169-188. Springer, 2011.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artifciall Intelligence, volume 34, pp. 7432-7439, 2020.
Yelysci Bondarenko, Markus Nagel, and Tijmen Blankevoort, Understanding and overcoming the challenges of efficient transformer quamtization. In Proceedings of the 2021 Conference on Em- pirical Methods Natural Language Processing, pp. 7947-7969, 2021.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Remov- ing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems. 36:75067-75096, 2023.
Elette Boyle, Niv Gilboa, and Yuval Ishai. Function secret sharing. In Annual international confer- ence on the theory and applications of cryptographic techniques, pp. 337-367. Springer, 2015.
Elette Boyle, Niv Gilboa, and Yuval Ishai. Function secret sharing: Improvements and extensions. In Proceedings the 2016 ACM SIGSAC conference on computer and communications security, pp. 1292-1303, 2016.
Elette Boyle, Niv Gilboa, and Yuval Ishai. Sccure computation with preprocessing via function secret sharing. In Theory of Cryptography Conference, pp. 341-371. Springer, 2019.
Charles George Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1):76-90, 1970.
Mengzhao Chen, Yi Liu. Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers in 1lms. arXiv preprint 4100022410.05265, 2024a.
10

---

Published as conference paper ICLR 2026
Yihan Chen, Zhijian Luo, and Yuzhe Zhang. Bolt:
Scalable private inference for transformer models. In IEEE Symposium Security and Privacy (S&P), 2024b.
Yao Cheng, Sameer Wagh, al. Mpeformer: Fast privacy preserving inference for transformers. In International Conference Learning Representations (ICLR), 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 103.::01003055577 2018.
Anders Dalskov, Luca Kamm, and Marcel Keller. Privformer: Privacy- preserving transformer in- ference via 3-party computation. In IEEE European Symposium Security and Privacy (Euro S&P), 2023.
Ivan Damgård and Sarah Zakarias. Constant-overhead secure computation of boolean circuits ing preprocessing. Proceedings of the 10th theory of cryptography conference Theory of Cryptography, pp. 621-641, 2013.
Tim Deumers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers scale, Advances neural information processing systems, 35: 30318- 30332, 2022.
Ye Dong, Wen-jic Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong. Wei. Wen-Guang Chen, et al. Puma: Secure inference Ilama-7b five minutes. Security and Safety, 4:2025014, 2025.
Robert E Goldschmidt, Applications of division by convergence. PhD thesis, Massachusetts Institute of Technology, 1964.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandcy, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama herd models, arXiv preprint 24x7:2240.22788. 2024.
Kanav Gupta, Deepak Kumaraswamy, Nishanth Chandran, and Divya Gupta. Llama: low latency math library for secure inference. Proceedings on Privacy Enhancing Technologies, 4:274-294, 2022.
Kanav Gupta, Neha Jawalkar, Ananta Mukherjee, Nishanth Chandran, Divya Gupta, Ashish Panwar, and Rahul Sharma. Sigma: Secure gpt inference with function secret sharing. Proceedings on Privacy Enhancing Technologies. 4:61579, 2024.
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Irom: Private inference on transformers. Advances in neural information processing systems, 35:15718- 15731, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian enoor linear units (gelus), 2023. URL httpp::/
arxiv. wrx.v.org/abb/ 08415
Ke Hong. Guohao Dai. Jiaming Xu. Qiuli Mao. Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference with asyn- chronization, flat gemm optimization, and heuristics. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6. pp. 148 161, 2024. URLLbbtts://prrceeeee ngs svs wwwpp/a....prrre..eeae.aaaaaap.. files/ /ites//aperr//2e/ File/ 532 waw.arccceeeeeeet777322121333332
Ziqi Huang, Thomas Smith, and Vladimir Kolesnikov. Bumblebee: Efficient and scalable mpe for large transformers. In Network and Distributed System Security Symposium (NDSS), 2025.
Neha Jawalkar, Kanav Gupta, Arkaprava Basu, Nishanth Chandran, Divya Gupta, and Rahul Sharma. Orea: Fss-based secure training and inference with gpus. In 2024 IEEE Symposium on Security and Privacy (SP). pp. 597-616. IEEE, 2024.

---

Published as conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot. Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ame Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
//arxiv. wwwp:///rgg/aaaa/a//a/aaaa//a/// 2310.. 06825,
Andes Y. L. Kei and Sherman S. M. Chow. Shaft: Secure, handy, accurate, and fast transformer inference. In Network and Distributed System Security Symposiun (NDSS), 2025.
Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure mulli-pprry computation meets machine learning. Advances in Neural Information Processing Systems, 34:4961-4977 2021.
Yun Li, Yufci Duan, Zhicong Huang, Cheng Hong, Chao Zhang, and Yifan Song. Efficient {3PC} for binary circuits with application to [Maliciously-Secure]] (DNN) inference. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5377-5394, 2023.
Jinglong Luo, Yehong Zhang, Zhuo Zhang, Jiaqi Zhang. Xin Mu, Hui Wang. Yue Yu, and Zenglin Xu. Secformer: Fast and accurate privacy-preserving inference for transformer models via smpe. In Findings of the Association for Computational Linguistics: ACL 2024. pp. 13333-13349 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Payman Mohassel and Yupeng Zhang. Secureml: A system for scalable privacy-presse preserving machine learning. In 2017 IEEE symposium on security and privacy (SP), pp. 19-38. IKHE, 2017.
Giovanni Puccetti, Anna Rogers. Aleksandr Drozd, and Felice Orletta. Outlier dimensions that disrupt transformers are driven by frequency. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1304- 2022.
Alce Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nis- hanth Chandran, and Aseem Rastogi. Sirnn: A math library for secure rnn inference. In 2021 IEEE Symposiun on Security and Privacy (SP), pp. 1003-1020 IEEE, 2021.
Pranab Sahoo, Ayush Kumar Singh. Sripamn Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering large language models: Techniques and applica- tions, 2025. URt/https:: arx.v. orgg abs/bb//20 /20222222222 07927
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scalc. Communications of the ACM, 64(9)999 100999911666000 2021.
Manuel B Santos, Dimitris Mouris, Mchmet Ugurbil, Stanislaw Jarecki, José Reis, Shubho Sen- gupta. and Miguel de Vega. Curl: Private Ilms through wavelet-encoded look-up tables. Crypiol- ogy ePrint Archive, 2024.
Adi Shamir. How share secret. Communications the ACM, (2M,,22(1))6112-613111
Noam Shazeer. Glu variants improve transformer, 2020. URL https://arxiv.org/ abs/
2002.05202
Kyle Storrier, Adithya Vadapalli, Allan Lyons, and Ryan Henry. Grotto: Screaming fast (2+ 1) D)-pe or z2n via (2, 2)-dpfs. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security. pp. 2143-2157, 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Longuage Modeling, 2024. URL /ttps:///penrrvvee net/ Eorum? (amumii==Feee 7FF7aAifi
12

---

Published as conference paper ICLR 2026
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa. Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models practical size. arXiv preprint 4rXi::2488.001 18, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjud Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tune nune chat models. arXiv preprint arXiv:2307 230:2237.009888 2023.
Alex Wang. A Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: multi-task benchmark and analysis platform natural language understanding. Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pp. 333-355,222 2018.
Jean-Luc Watson, Sameer Wagh, and Raluca Ada Popa. Piranha: A (GPU) platform for secure computation. In s/ USENIX Security Symposium (USENIX Security 22). pp. 827-844, 2022.
Maurice Weher, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, ct al. Rodpajama: an open datasct for training large language models. Advances in nearal information processing systems, 37:1 16462- 116492, 2024.
Xiuying Wei, Yunchen Zhang. Xiangguo Zhang, Ruihao Gong, Shanghang Zhang. Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances Neural Information Processing Systems, 15:17402-17414 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2023b.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu. Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan. Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https:/ arxiv..... org/ abs /20../555... 2w..55055555 09388
Andrew Chi-Chih Yao. How generate and exchange secrets. In 27th annual symposium on foundations of computer science (Sfes 1986), pp. 162-167. IEEE, 1986.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 4791-4800) 2019.
Jiawen Zhang, Xinpeng Yang, Lipeng He, Kejia Chen, Wen-jie Lu,, Yinghao Wang. Xiaoyang Hou, Jian Liu, Kui Ren, Xiaohu Yang. Secure transformer inference made non- -on-interactive. In Network and Distributed System Security Symposiun (NDSS), 2025.
13


Reviews:
[{"review_id": "3TPBxba0mh", "reviewer": "Reviewer_RtFw", "summary": "The paper proposes SOAL (Secure Outlier-Aware Large Language Model Inference), a framework that accelerates privacy-preserving LLM inference under MPC (multi-party computation) by exploiting outlier distributions in non-linear layers. The authors observe that activations in normalization, activation (SiLU), and Softmax layers show heavy-tailed distributions where only a few “outlier” activations dominate. They propose to “control” these outliers by prefixing special tokens in the prompt, thereby reducing the input domain of non-linear MPC protocols. They then design optimized MPC protocols for RMSNorm, SiLU, and Softmax based on the narrowed input range and specific polynomial approximations. Experiments on GPT-2, Llama2-7B, and Mixtral 8×7B show about 2–3× latency reduction in secure inference with nearly no accuracy loss.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. SOAL requires no model retraining or architecture modification, increasing its compatibility with commercial LLM deployments.\n\n2. The paper demonstrates large latency reductions (≈2×–3× on major operations) while maintaining accuracy, which is significant for the secure inference community.", "weaknesses": "- The main idea, i.e. using activation range reduction to simplify MPC protocols, draws heavily from prior work in quantization and outlier suppression (e.g., SmoothQuant, Outlier Suppression). The contribution is more engineering-oriented integration than a new theoretical principle.\n\n- The prefix-based outlier control is validated on a fixed set of prompts; the paper does not analyze robustness to domain shift (e.g., multi-language, code, long context). The approach may overfit the profiled activation statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfY2eFHMIp", "reviewer": "Reviewer_DABD", "summary": "This paper focuses on the problem of secure LLM inference using MPC technologies, specifically 2PC setting. The authors studied the phenomenon of outlier activations during LLM inference and proposed tailored methods to handle outliers for different non-linear functions.  In this way, the input values to non-linear functions, which are typically the bottleneck of MPC-based approaches, can be limited to a narrow range. The authors then used simpler approximations or LUT to improve the efficiency of these non-linear functions. Particullarly, the introduction of conformant maxima avoid the heavy maximum finding. The authors also conducted some experiments against prior works on GPT-2 and LLama-7B models to showcase the effectiveness.", "questions": "- Could the author clarify whether the setting is exactly 2PC? I believe MPCFormer and Crypten using 2+1 setting, where a TTP is required to handle the generation of some correleated randomness.\n- Please refer to the weakness for more questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- Using MPC to secure LLM inference is an interesting and important direction.\n- The idea of handling activation by prefixing special tokens and using more efficient non-linear functions given a narrow input range is innovative. The introduction of conformant maxima is interesting.\n- Extensive experiments have been conducted.", "weaknesses": "- The authors focus on 2PC ASS and FSS, lacking the discussion of SOTA HE-based solutions.\n- The ASS baseline is mainly Crypten, and lacks comprehensive comparison against SOTA works. For example, MPCFormer, SecFormer and PUMA [1]\n- The MPC-side contribution is somewhat limited. The idea is similar to SecFormer and more recent work Nimbus [2], which also proposes distribution-aware non-linear function evaluaiton.\n- The description for special-token outlier collection is ambigous. Could the author provide the 'special-toekns' for GPT-2 and Mistral 8x7B as well? According to Figure 1(c), the token '.' (which is believe is not a commonly-defined special token) is regarded as outlier token. What if multile '.' occurs in the prompt? Besides, if the prompt is a long sentence, does that mean all the '.' should be prefixed? In this case, what is the impact to the utility?\n- How are the PPL figures in Table 3 calculated? Do the SOAL and original use the same prefixed prompt as inputs?\n\n\n[1]: PUMA: Secure inference of LLaMA-7B in five minutes. https://arxiv.org/abs/2307.12533\n\n[2]: Nimbus: Secure and Efficient Two-Party Inference for Transformers https://arxiv.org/pdf/2411.15707", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMvtMtE9cE", "reviewer": "Reviewer_8jL4", "summary": "SOAL focuses on improving the efficiency of LLM inference under MPC settings. In the past, the main reason for slow inference was the computational complexity of the nonlinear layer. SOAL leverages the statistical properties of activations, maintains the original model weights, and uses a designed trigger token to control the location of outliers, further narrowing the activation range.", "questions": "1. Is the post-prefix outlier anchor position stable in complex scenarios such as multilingual, long-context, and retrieval-enhanced or multimodal? Has performance been tested under different sampling parameters ($temperature, top-p$)?\n\n2. After incremental training/instruction fine-tuning, is it necessary to re-count special tokens? How does the cost of the Preparation phase versus retraining affect the quantitative benefits of model changes?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In the preparation stage, the distribution of nonlinear layer activations is pre-counted, outliers of special tokens are identified, and outliers are locked in predictable positions by prefixing; based on the previously shrunk input, the MPC protocol of RMSNorm, SiLU, and Softmax is reconstructed to achieve acceleration, and the advantages in time and communication volume are reported on gpt2, llama2-7b, and mixtral.", "weaknesses": "**1. Idealistic Assumptions:**\n\nThe evaluation is conducted in a semi-honest setting, lacks discussion of malicious actors, and provides no security proof.\n\n**2. Lack of Robustness Assessment:**\n\nThis method relies on a stable association between activation distributions and specific tokens, but the paper does not verify whether this relationship holds in **multilingual, long-context, or fine-tuned models** scenarios. It also lacks robustness tests against outlier distribution drift and prefix failure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VS6xfTrmhR", "reviewer": "Reviewer_Hgte", "summary": "This work introduces a secure inference system that smooths out outliers in order to reduce complexity (bit length) in underlying secure computation. They start from empirical investigation of the outlier issues and show the possibility to constrain their values within a small range by prefixing the prompt. They then propose optimized MPC protocols to evaluate the non-linear primitives on small bit-length inputs. They also propose the \"conformant maxima\" to compute approximate max values in softmax, which is based on the observation that the most maxima positions locate in a limited number of places.", "questions": "- Give reference for your claim: \"Inferencing a 64-token input once using Llama2-7B (Gupta et al., 2022) via CrypTen (Knott et al., 2021) would take 169.76 seconds.\" \n\n- Your claim \"Such inefficiency stems from MPC’s difficulty in handling nonlinear layers\" is not completely correct. It is only meaningful because you consider everything implemented in CrypTen, which is not a fair library to use for two-party computation (SOAL focuses on standard two-party computation as you claim). CrypTen is NOT a pure 2PC library because it doesn't include faithful ways of generating 2PC multiplication triples. It should be classified as a \"2+1\" library where triples are generated by a trusted third party. In pure 2PC secure inference system, the linear layers (mainly matrix multiplication) actually occupies at least half of the overhead (check BOLT[1] or BumbleBee [2] for reference). \n\n- In Figure 1 (c), how can I read from the graph that special tokens are related to outliers?\n\n- For ASS instantiation, how is your 8-bit LUT implemented?\n\n- As I mention above, please clarify in the text that the evaluation results with CrypTen is not pure 2PC. For example, in Table 1, if we evaluate with a \"real\" 2PC library, the linear layers should contribute much more in the total overhead.\n\n- Should include some recent schemes into the comparison, such as BOLT [1] and BumbleBee [2]. Though these schemes are not implemented with CrypTen, it doesn't affect a direct comparison by taking their results from the papers. For example, communication amount is a metric that normally doesn't change in different libraries, and would give readers a more clear picture on the advantage of your solution.\n\n\n[1] https://eprint.iacr.org/2023/1893\n[2] https://eprint.iacr.org/2023/1678", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ I appreciate their effort to identify origins of the outlier issues (and also maxima positions) and propose simple and effective way to reduce such issues in inference, consequently leading to more efficient secure computation on smaller bit length.\n\n+ The proposed protocols seem valid.\n\n+ The proposal is evaluated on both ASS and FSS schemes, which exceeds my expectation.", "weaknesses": "- Some claims have no reference support (or might even be invalid, see my questions below).\n\n- The usage of CrypTen is understandable, but it provides misleading results, which should be clearly identified and explained to the readers.\n\n- The experimental comparison is not complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)