CipherPrune:  Efficient and Scalable Private Transformer Inference
Authors: Yancheng Zhang
Jiaqi Xue
Mengxin Zheng
Mimi Xie
Mingzhe Zhang
Lei Jiang
Qian Lou
Venue: ICLR 2025 Poster (2025)
Keywords: Private Transformer Inference
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Private Transformer inference using cryptographic protocols offers promising solutions for privacy-preserving machine learning; however, it still faces significant runtime overhead (efficiency issues) and challenges in handling long-token inputs (scalability issues). We observe that the Transformer's operational complexity scales quadratically with the number of input tokens, making it essential to reduce the input token length. Notably, each token varies in importance, and many inputs contain redundant tokens. Additionally, prior private inference methods that rely on high-degree polynomial approximations for non-linear activations are computationally expensive. Therefore, reducing the polynomial degree for less important tokens can significantly accelerate private inference.  Building on these observations, we propose \textit{CipherPrune}, an efficient and scalable private inference framework that includes a secure encrypted token pruning protocol, a polynomial reduction protocol, and corresponding Transformer network optimizations. At the protocol level, encrypted token pruning adaptively removes unimportant tokens from encrypted inputs in a progressive, layer-wise manner. Additionally, encrypted polynomial reduction assigns lower-degree polynomials to less important tokens after pruning, enhancing efficiency without decryption. At the network level, we introduce protocol-aware network optimization via a gradient-based search to maximize pruning thresholds and polynomial reduction conditions while maintaining the desired accuracy. Our experiments demonstrate that CipherPrune reduces the execution overhead of private Transformer inference by approximately $6.1\times$ for 128-token inputs and $10.6\times$  for 512-token inputs, compared to previous methods, with only a marginal drop in accuracy. The code is publicly available at https://github.com/UCF-Lou-Lab-PET/cipher-prune-inference.

Published as a conference paper at ICLR 2025
CIPHERPRUNE: EFFICIENT AND SCALABLE PRIVATE TRANSFORMER INFERENCE
Yancheng Zhang', Jiaqi Xue', Menggin Zheng' Mimi Xie?, Mingzhe Zhang?, Lei Jiang'. Qian Lou
University of Central Florida University of Texas at San Antonio Ant Research *Indiana University Bloomington
ABSTRACT
Private Transformer inference using cryptographic protocols offers promising so- lutions for privacy-preserving machine learning; however, still faces significant runtime overhead (efficiency issues) and challenges in handling long-token inputs (scalability issues). We observe that the Transformer's operational complexity scales quadratically with the number of input tokens, making it essential to re- duce the input token length. Notably, each token varies in importance, and many inputs contain redundant tokens. Additionally, prior private inference methods that rely on high-degree polynomial approximations for non-linear activations are computationally expensive. Therefore. reducing the polynomial degree for less important tokens can significantly accelerate private inference. Building on these observations, propose CipherPrune, efficient and scalable private inference framework that includes a secure encrypted token pruning protocol, polynomial reduction protocol, and corresponding Transformer network optimizations. At the protocol level, encrypted token pruning adaptively removes unimportant tokens from encrypted inputs in progressive, layer-wise manner. Additionally, encrypted polynomial reduction assigns lower-degree polynomials to less important tokens after pruning, enhancing efficiency without decryption. the network level, we introduce protocol-aware network optimization via a gradient- gradient-based search to max- imize pruning thresholds and polynomial reduction conditions while maintaining the desired accuracy. Our experiments demonstrate that CipherPrune reduces the execution overhead of private Transformer inference by approximately 6.1x x for 128-token inputs and 10.6x for 51 512-token inputs, compared to previous meth- ods, with only a marginal drop in accuracy. The code is publicly available at Tttps//gtthub.com/UU---LL wt-.aa--ccciippee-rr-----eeeeeee
INTRODUCTION
Transformers Vaaswani et al. 2017) have become the predominant approach for tackling a wide range of machine learning tasks, spanning Natural Language Processing (NLP) (ue al. 2024 and Computer Vision (CV) (Zheng et al. [022) domains. Notably, Transformer-aa-a-Serviii ( (TaaS) (Rad ford et al. 2018) has emerged sophisticated and accurate Transformers deployed on cloud servers. Privacy has become a major concern, driving a growing demand for privacy-preserving TaaS solutions (Zheng et al. 2023 Hao
effective means for average users to harness the capabilities of
et al. 2022 Zeng et al., 2023 Santriaji et al. 2025 Lou et al. (021))
Homomorphic Encryption (HE) Gentry 2009) is a promising secure computing technology that
protects data privacy by enabling computations on encrypted data without decryption. However, applying HE continuously for deep computation tasks often results in prohibitively high latency (Lou
et al. 2019 Lou & [211g Zoinn et al. (2024) Zhang et al.) [223a Jiang al. [002)). To address
Lou
this, hybrid HE/Multi-parts Murtuy Computation (MPC)-based techniques (Chen et al. 2022 Zheng et al.
2023
Hao et al. 2022 Zeng et al. 2023 Zhang al. 2023b Lu et al.) 2025 Pang al. 2024 Xu
et al. 2024b) have been widely adopted for private Transformer inference, as illustrated in Figure
"Corresponding Author. Email: gian.lou@ucf.ccu..

---

Published as a conference paper at ICLR 2025
Output
4000
The movic great mix of action and comedy. Output: 99.3% positive
Input:
Different Inputs
f oTokeus: S2
Inpet
Rumtimm (b)
3000
Input ig
TTeliens 60
Non- San-Lincer (MPC)
Inpui
2000
Remove rodundancy
(1I))
Layer
w
1000
Pruned Input:
80%
1 TTokens: 90
movie was a great mix
30%
Layer
52 64 128 258 312
action and comedy.
68%
t TTokens: 128
Number of Input Tokens
Output: 99.2% positive
Redundancy Ratio
Clicet
Surver
Input
(4) Private Transformer Inference
(h) Issnes of Effieiency Scalability
(c) Input redundancy
(d) Input-speeifte Redundancy
(e) Prugressive Pruning
Figure I: (a) An illustration of HE/MPC- based private inference. (b) The high-latency and scalable challenge of private Transformer models over lengthy inputs. (c) An example of redundant input in sentiment analysis tasks. (d) Demonstration of varying levels of redundancy across different inputs.
(e) An example showcasing progressive redundancy pruning.
(a). This hybrid approach achieves state-of-the-an performance by using HE for linear operations and MPC for non-linear operations.
Unfortunately, prior private Transformer inferences (Lu et al. 2025; Pang et al. 2024 Zheng et al.
2023) still suffer from significant latency and poor scalability over long-token inputs. As Figure (b) shows, the prior private inference process for a GPT2 Transformer (Pang et al. (024) with 128 input tokens extends to 10 minutes. It necessitates the exchange of over 60 gigabytes of data between the server and the client. Furthermore, as the token length increases, the runtime overhead grows super-linearly, indicating poor scalability. This is primarily because the operational complexity of Transformers /Vaswani et al. 2017: Kim et al., 2022) scales quadratically with the number of input tokens. Reducing the number of input tokens without compromising accuracy essential.
We observe that most inputs contain redundant words/tokens, with varying levels of redundancy across different inputs. As illustrated in (c), in sentiment analysis task, an input that retains only the tokens movie and great while removing almost all others still maintains inference confidence and accuracy. We refer to such tokens that can be removed without significantly impacting accuracy as redundancy. Meanwhile, the different inputs have various levels of redundancy (Wang et al.) 2021 Kim et al., 2022; Yudha et al.) 2024). Figure (d) illustrates that some inputs exhibit greater redundancy, while others have less, with this variation being particularly evident across different tasks. Classification tasks typically have more redundancy compared to sequence-to-sequee tasks (Fu al.) 2024). To effectively prune more tokens from longer imputs and potentially reduce the Transformer's quadratic complexity to linear, pruning should be done progressively- -tht is, tokens should be pruned layer by layer over multiple stages, as illustrated in Figure I (e), rather than performing a one-time pruning at the first layer (Wang et al. 2021 Kim et al. 2022 Xu et al. 2024a)). Another key observation is that previous private Transformer inference methods, whether relying on precise non-linear activations (Hao et al.) 2022 Yudha et al. 2024) or using large-degree polynomial approximations (Lu et al. 2025] Pang et al., 2024) for these activations, continue to suffer from significant execution overhead for non-linear operations. Therefore, replacing non-linear activations or high-degree polynomials with lower-degree polynomials can beneficial. As shown Figure 2. a degree-d polynomial activation for tokens (Figure 2 (a)) (Figure (b)), where d; < d.
(a)) can be reduced to a degree-d, polynomial
Adopting existing plaintext-level pruning techniques (PlainPrume) (Wang al. 2021 Kim et al. 2022) to accelerate private Transformer inferences presents a formidable challenge. The primary reason is that the tokens are encrypted, and we need to calculate token importance scores layer by layer for specific encrypted inputs. This requires redesigning a new encrypted token pruning protocol. Meanwhile, the polynomial reduction in the encrypted domain poses similar challenge, and we need to design an encrypted polynomial reduction protocol for efficient private activation. Also, at the network level, need to learn the pruning and polynomial reduction thresholds while maximizing efficiency, ensuring privacy, and maintaining the desired level of accuracy.
To address these challenges, we introduce CipherPrune. a scalable and efficient frame- work for private inference that incorporates a secure encrypted token pruning protocol, a polynomial reduction protocol, and tailored Transformer network optimizations. At the protocol level, CipherPrune adaptively prunes unimportant tokens from encrypted inputs in a progressive, layer-by-layer manner. It also applies encrypted polynomial reduction
2

---

Published as a conference paper at ICLR 2025
by assigning lower-degree polynomials to less important tokens post-pruning, thereby improving efficiency without requiring decryption. At the network level, we implement protocol-aware optimization using a gradient-basee search, aiming to maximize pruning thresholds and polynomial reduction conditions while preserving the required accu- racy, Our experiments show that CipherPrune reduces the execution overhead of private Transformer inference by about 6.1 for 128-token inputs and 10.6 x for 512-token
Activate
Degree-d Polynomial P(:d)
Token
No Polynomial Reduction
Activate
Degree-d, Polynomial Pt; d, d)
Polynomial Reduction
Figure 2: Polynomial Reduction.
inputs compared to prior method (Pang et al. 2024). with only marginal drop in accuracy.
PRELIMINARIES
Token Pruning in Transformers. To reduce the computational overhead of plaintext Transformers, a range of strategies including efficient architecture design, knowledge distillation, quantization, and both model and token pruning have been developed. Among these, token pruning (Goyal et al.) Kim & Cho. 2020; Wang et al., (22T)) stands out for its ability to dynamically reduce token complexity. enhancing efficiency for scalable input lengths. Online token pruning progressively eliminates nonessential input tokens during inference, with recent techniques like learning- based token pruning assigning tunable thresholds to each Transformer layer. These thresholds are fine-tuned during training to maximize the removal of tokens from the sequence.
Given the attention map Att = SoftMaz(QK" /Vd) E R"**, QKT computes the dot product matrix. Dividing by vd scales down the values to excessively large dot products, especially when the dimensionality is large. The importance score S E Rn of tokens 2; E RR. in the input sequence can be computed as: S|i] = 1 7/4 Ex=o Σド= Atth [j (1) where H is the number of attention heads and Att" is the attention map in the h-th head. Importance score S is essentially computed by accumulating attention scores vertically, which indicates the importance of a token across all heads in one Transformer layer.
between the query and key vectors, resulting in
Cryptographic Primitives. Our protocol uses multiple cryptographic primitives including Additive Secret Sharing (ASS), Homomorphic Encryption (HE), and Oblivious Transfer (OT). We reuse partial existing protocols detailed in Appendix B
ASS. We employ 2-out-of-2 ASS scheme (Cramer et al. 2015) operating over the ring Ze, where l is the bitwidth of the input æ. ASS partitions x into two distinct random shares (x) (x=),- where
r (x), + (r)) mod Z, The parties, Po and P1, respectively hold (z) (x)))-))))))))))))) Importantly, it is guaranteed that neither Po nor Pi can discern the actual value of x (Cramer al., 2015). ASS lends itself to linear operations, i.e., addition and constant multiplication, without communications. HE. We leverage the BFV scheme (Brakerski) 2012 Fan & Vercauteren 2012 Deng et al. 2024), a leveled HE scheme, facilitate linear operations ciphertexts. The HE scheme has 4 functions: KeyGen, Enc, Dec, and Eval. KeyGen generates a public key ph and a secret key sk. Enc encrypts message with the public key pk to yield a ciphertext Dec, with the secret key sk and ciphertext e as inputs, decrypts the ciphertext to recover the message m. Finally, Eval, when given the public key pk. two ciphertexts e₁ and e2 encrypting messages mi and m2, along with linear function F. produces a new ciphertext c' encrypting the result of F(m, (m)))
OT. We use OT for non-linear operations (Rathee et al. 2020; 2021 Xue et al. (023) in network model. Specifically. we employ 1 ofo correlated OT (Asharov et al. 2013) (2-COT,) and I ofuut-of-k (Kolesnikov & Kumaresan 2013) (k:OT,). In 2-COT,, the protocol takes as inputs the sender's correlation E E, and receiver's bit choice i E (0, It then produces random element r E Z, for the sender and r + i x for the receiver. In k--TT,, the sender possesses i messages mo. -.. and the receiver holds an index i [k]. The protocol ensures the receiver learns as the output without learning any information about Tj, where j E [k] and j * i, while the sender learns nothing about the receiver's choice i.
Prior Private Transformer Inference. In response to the success of Transformers and the need to safeguard data privacy. various private Transformer Inferences (Chen et al./ 2002) Zheng et al. 2023
Hao et al. 2022. Li et al. 2022 Lu et al.) [025) Hou et al.) [202]] Luo et al. [224 Pang et al. 2024)
proposed. To efficiently run private Transformer inferences, multiple cryptographic primitives
3

---

Published as a conference paper at ICLR 2025
used in a popular hybrid HE/MPC method IRON (Hao et al. 2022). , i.e., in a Transformer, HE and SS are used for linear layers, and SS and OT are adopted for nonlinear layers. IRON and BumbleBee (Lu let al. 2025) focus on optimizing linear general matrix multiplicatioos: SecFormer (Luo al.. (024) improves non-linear operations, such as the exponential function, through polynomial approximation. BOLT (Pang al., 2024) introduces the baby-step giant-step (BSGS) algorithm to reduce the numher of HE rotations, proposes a word elimination (W.E.) technique, and uses polynomial approximation for non-linear operations, ultimately achieving state-of-the-a (SOTA) performance. It's worth noting that the W.E. technique in BOLT is not input-specific, as it uniformly prunes half the tokens regardless of the imput or task. This approach may fail to remove all redundancy when it exceeds half of the tokens and can harm accuracy when redundancy is less than half. Additionally, since W.E. performs one-time pruning at the first layer rather than progressive, layer-by-layer pruning, it is less effective at reducing tokens for longer inputs. Moreover, BOLT's W.E. protocol is computationally expensive due to its reliance on sorting, whereas our method achieves lower asymptotic complexity and faster concrete runtime. Specifically, the statc-of-the the--hheaat art BOLT still faces efficiency and scalability challenges with long-token inputs. For example, one private inference with a GPT2-Base model can take ~ 10 minutes for 1 128-token inputs and ~ 1 hour for 512-token inputs, requiring data exchanges of more than 60GB and 200GB, respectively. Besides the hybrid HE/MPC methods, a line of works has explored performing private Transformer inference with only HE (Zimerman al. 2023 Zhang et al. 2024). We leave a more detailed review of related works in Appendix G
Threat Model and Security Guarantee. CipherPrune operates in a common private inference scenario where server Po owns a proprictary Transformer-based model M with private weights w, and client P1 possesses private input data x. We assume the server and client are semi-honest, i.e. the server and client follow the designed protocols but are curious and attempt to learn extra information (e.g., x or w). This setting practical, as the server is incentivized to follow protocols and provide high-quality services for monetary gain, while the client is motivated to adhere to the protocol to receive those services. Consequently, this semi-honest setting is commonly adopted in existing works (Rathee et al., 2020; Huang et al. [022 Hao al.) 2022 Lu et al. 2025 Pang et al., 2024). In this semi-honest setting, our protocols prevent the server from learning the client's 's data and the inference result; meanwhile, these protocols also block the client from accessing the model's s parameters. In our protocols, we assume that both the server and client are aware of the number of pruned tokens. We argue that this information does not compromise the client's data or inference results, nor does it enable the client to access the model's weight parameters. Attacks that deviate from the semi-honest setting are beyond the scope of this work.
3 CIPHERPRUNE FRAMEWORK
3.1 MOTIVATION
Although plaintext pruning methods (Goyal et al. 2020 Kim & Cho 2020 Wang et al. 2021 enable efficient and scalable inference for standard plaintex--domin Transformers, integrating these techniques into private Transformers remains challenging. Additionally, encrypted polynomial reduction and its joint optimization with token pruning remain largely unexplored.
Lacking protocols for input-specific, progressive encrypted token pruning and encrypted polynomial reduction. The efficiency and scalability of plaintext pruning methods (Goyal eet al. 2020 Kim & Cho 2020; Wang et al. 2021) rely on two important features: (1) Input-specific pruning: This involves assigning an adaptive pruning ratio based on the dynamic importance of each input, as different inputs exhibit varying levels of redundancy. A fixed pruning ratio applied
Challenge 1:
universally can result in suboptimal pruning or excessive pruning, leading to catastrophic accuracy loss. (2) Progressive pruning: This approach prunes tokens layer by layer, rather than performing one-time early-layer pruning. Early pruning may fail to correctly identify redundant tokens, resulting in suboptimal or incorrect pruning. As mentioned earlier, prior W.E. in BOLT (Pang et al.) (024) is not input-specific or progressive. Furthermore, there is an absence of an encrypted protocol for polynomial production aimed reducing the overhead of non-linear approximations.
Lacking network optimization to support joint token pruning and polynomial reduction. In plaintext-domain token pruning (Goyal et al. 2020; Kim & Cho 2020 Wang et al. 2021). there is no strong motivation to use polynomials to approximate non- con-linear activations, as these operations are straightforward and inexpensive compute. However, in the ciphertext domain,
Challenge 2:
4

---

Published as a conference paper at ICLR 2025
large-degree polynomials are employed approximate non-linear functions for both efficiency and accuracy. We observed that polynomials of varying degrees can be assigned to different tokens the encrypted domain based on their importance scores, which also be optimized alongside token pruning. For instance, joint optimization of polynomial reduction and token pruning is essential, and network optimization involves searching for the optimal pruning and reduction thresholds
CipherPrune Overview. In this paper, introduce CipherPrune for an efficient and scalable private Transformer inference.
Embedded tokcus
Pruned
Sec.
tokens
Enerypted Tuken
Enerypted
dogree2
E2
Linear
Polynomial Reduction
Figure 3 shows the overview of Cipher- Prune. We first propose encrypted token pruning for both linear and non-linear ac- tivations in Section 3.2 Then, we develop an enerypted polynomial reduction for effi
Prusing
degrest
Sec. 3.4
Pruning support
Activation
for (MPC)
Network Optimization for Encrypted Prening
Figure 3: Overview of CipherPrune.
cient non-linear operations in Section 3.3
We also introduce network optimization for the joint
optimization of token pruning and polynomial reduction in Section 3.4
Figure 4 shows the workflow of private Transformer inference implementation with our CipherPrune. During the private inference, the client's 's input is encrypted and multiplied with the embedding matrix in the server by the MoMtMMul protocol. Then the result will be added to the positional encoding. @ The server performs the private attention computations, including linear projection via Mut and non-linear attention map via Mo respectively. After obtaining the tokens and attention maps, & our proposed encrypted token pruning method is performed to calculate token importance scores and compare these scores with the pruning threshold 0 and reduction threshold 3 to decide which tokens will be pruned or reduced in privacy-preservee discarded such that the following layers will have fewer operations. Low-degree polynomials are used to compute the GELU and SoftMax functions on reduced tokens. + Layernorm and Feedforward operations will be executed via the prior protocols, IT Leyerrormm MaMaMa and G.LU. We detail our proposed encrypted token pruning method in the subsequent subsections.
manner. The pruned tokens are
Server
Encoding
I Lineor
Transformers
Lincar
MatMal
Sultsas
MacMal
LayerNeeni
GELU
LayerNoeni
Output
Tokens
Prosed Tokuns
Embedding
Feed Forward
Private Inference
Keep
3, (SothMax)
Imput
0.3
Roluce
Xy (Ploy)
and Reduce
Prune
5x (Fruned)
Saftmax
LayerNura
GELU
Input
Output
Scerss 100
Tokeso
Threaboldde olll
Client
Dynamic Pruning and Polynomial
Cryplographic Protecals
Figure 4: The workflow of a private Transformer inference with CipherPrune.
3.2 ENCRYPTED TOKEN PRUNING
Pruning protocol II prune- In private inference, confidentially pruning tokens presents a challenge, the server and client must share attention maps and inputs without accessing their actual values. As depicted in Figure 5]] during inference, attention maps are protected using ASS, with each party only able view their respective shares.
the other party
Attention
map
shares
scure shares
Compare
Prune
Comparr with threshold
real mark
teken shores (y)
Bata protected
leeta pratocted
Unpexteted
take
Concunsiaatin
Receive from the other
token shares (x)
Figure 5: Illustration of mask generation and token pruning in Iorune with a non-sharing mask.
Specifically, the proposed secure token pruning protocol takes the secret-shared attention maps (Att) and tokens (x) as inputs, and outputs the pruned tokens (y) in secret-sharing format. The secret shares are held by server Po and client P,, respectively. First, Po and P1 compute the importance score their local secret shares, respectively. As depieted in Equation T the computation of
5

---

Published as a conference paper at ICLR 2025
the importance score involves only addition and constant multiplication, which can be performed efficiently via ASS. After Po and P1 acquire their respective shares of the importance score (S), they initiate comparison protocol H. This protocol contrasts the importance score against the threshold 0 learned offline, enabling them to determine the shares of the resultant mask (M), where M is if $ > 0, otherwise 0.
Possessing the shares (M) without access to their real values prevents the direct pruning of tokens (a)). A feasible solution involves reconstructing the non-shared mask, allowing both parties to independently prune their shares of the input sequence of n tokens (xr). This process then enables them to obtain the shares of the pruned output sequence of m tokens (y). Appendix A Figure 13 includes a more detailed and formal definition of Upruxe.
The overhead for our secure token pruning protocol is minimal. The importance score can be computed directly on shares, taking only 0.1 ms per attention module. This is efficient even for large models like BaRT-Large, which has 24 heads per layer. Additionally, our protocol only requires n invocations of the comparison protocol IIM each consistently completed within 5 ms, independent of the number of heads or embedding dimension. Thus, the total time complexity of our pruning protocol II prrne is linear, (n)) based on the input token sequence length
Keep Reyt
sharm
Bind Mask
Secure Soap
Truncate Tokens and MSR
3 mask shares (M)
and Tokens
Masked teken shares
Swapped token shares (8)
token shares (V)
3 token shares
Arthkaskk
(x)
valie
Beakon
x orjjl 0111
peill
(Mil)
selll 10111
shil aak
Sai
sum]
Badiase Mack
Ittly Yakeo connber Derive Pruning Token Number IV'
Ade
mamber
OT-basod Secure Swap
MEBD
5-kk Anlbeatk
(?ill - 19111
144
Artieatts share
4-30
A3s
by 01
Bind Mask and Tokens
Truncate tokens and MSB
Figure 6: Example of token pruning with a protected mask.
Pruning Mask Protocol Mmosk. To further safeguard the privacy of the binary pruning mask M, specifically to protect the locations of pruned tokens, we design an additional Pruning Mask Protocol, Imaas- The goal is to ensure that both parties, Po and Pi, obtain the pruned token sequence without knowing which specific tokens were pruned. One key observation is that the number of tokens after pruming, n', can be safely disclosed, as it is essential for subsequent processing and is not typically associated with significant security risks like adversarial attacks (Cui et al. (021]. Knowing n' is crucial because token pruning involves relocating the less- cess-ippottat m the end of the token sequence while maintaining the order of the remaining n' tokens. After this rearrangement, one can simply discard the m ASS tokens the end of the token list.
n' tokens to
[ the secure mask protocol mmask that is used to ensure the mask privacy in Uprune- The protocol takes secret-shared token sequence (x) and mask (M) as inputs, and generates the pruned tokens (v). O Bind Mask and Tokens. To swap tokens, their corresponding links with the mask will be disrupted. To preserve these links, there are two methods: one is to swap the masks and tokens respectively and simultaneously; the other is to bind the mask and tokens together so that they can be swapped as a unit. We adopt the second method, as binding the mask and tokens together proves more efficient than managing separate swaps for the mask and tokens. The bounded tokens (i) can be obtained via left-shifting (M) by / bits and adding to (x). where f is the bit width of a token in x. Figure Gillustrates using f= = 4 as an example; however, in practice, can be flexibly adjusted. @ Derive Pruning Token Number n'. We found that n' can be obtained by securely counting the number of 1s in M, which does not reveal the locations of 1s in M. Specifically, to determine n', both Po and P3 first convert their boolean mask shares (M) into a fixed-point format using 12 Each party then locally computes the sum of the arithmetic mask using ASS, yielding (n'). Finally, Po and P1 obtain n' by summing their respective shares, (n"), and (n²))-
b 3 + (1- b) =(i fffi+1 1]. b 2 +1++1++11111 b)- -
Swap [1) 1))
(2)
Secure Swap. This step aims to enable Po and P1 to iteratively move m tokens to the end of the token sequence via ba-based oblivious swap defined in Equation In each iteration, Po and P
perform an oblivious swap through the token sequence. To privately swap two tokens æ[i] and +[i+1], they first extract the MSB i from the bounded token / and perform four OT-based multiplications.

---

Published as a conference paper at ICLR 2025
! Truncate Tokens and MSB. Po and P1 can truncate the swapped token sequence (ï) and remove the MSB respectively to obtain the pruned token sequence. Figure 14 in the Appendix A details more about the mask protection protocol.
Analysis. The complexity of the proposed Imaask mainly depends on the number of oblivious swaps. To prune tokens out of input tokens, (mm) swaps are needed. Since token pruning is performed progressively, only a small number of tokens are pruned at each layer, which makes mask efficient during runtime. Specifically, for BERT-Base model with 128 input tokens, the pruning protocol only takes ~v 0.9s on average in each layer.
Time: 200
Time: 130
Bodutive wrehhead
tokens
dolcens
tokens
4 tolkees
Tikes Scars
Prent
Pranz & Appros
0.1x
0x
Prusc
youse
Redace
Keep
=: 0.45
Inggrttane Score
0.3
f B.G Reducing Threah.
Imsportance Score
Praning Thrmbold
Prening Thresh.
(a) Secure
(b) Token pruning
(c) Token pruning with polynomial reduction
computation overhead
Figure 7: Comparison of token pruning-oni method and pruning with polynomial reduction.
3.3 ENCRYPTED POLYNOMIAL REDUCTION
After pruning, retained tokens still require expensive operations, particularly for costly non-linear functions. These non-linear functions are computed via expensive high-degree polynomials (Lu ctt al. 2025; Pang et al., 2024). We notice that we can reduce the high-degree polynomials to their low-degree counterparts for the less important tokens. As demonstrated in Figure 7 (a), the cost of reduced polynomial can be 0.1x that of the high-degree polynomial. This moti- vates to accelerate the non-linear operations with low-degree polynomials while maintaining the desired accuracy. Similar to employing a threshold to prune tokens with importance scores below 0, use another reduction threshold (3 0) to identify tokens for reduction. As shown in Figure / (b)(c,, combining token pruning with polynomial reduction further reduces execution time compared
the pruning-only method. Importantly, we can optimize both
and B together during offline fine-tuning to enhance ef- ficiency. During the online inference phase, polynomial reduction occurs after token pruning (the pruning ratio greater than zero). This simplifies the reduction process: the tokens have already been pruned, and the locations of these tokens are rotated and concealed. Consequently, there's no need to safeguard the token mask for reduc- tion. Instead, we can simply modify the pruning protocol II Prune establish the reduction protocol. As illustrated in Figure g the pruned tokens are determined by executing protocols II prune and mask in tandem. A secure com- parison with the reduction threshold 3 then produces the reduction mask (M₂)- The location of this mask corresponds pruned tokens, not the original tokens, so revealing it does not compromise the location privacy of reduced tokens, provided that the privacy of pruned locations is maintained. Once the reduction mask M3 is known to each party, it can be used to guide the decision on whether to apply the high-degree polynomials or low-degree ones for the non-linear functions, where indicates using the high-degree ones and O signifies low-degree ones. The choice of these approximation polynomials can be flexible. We utilize prior non non-linear function approximation methods as detailed
00% remoiieed
00p token shares (31
shanes (8)
00p
Consasiate
Sccure Compare
reduection (Mp) musk starss
rerell reduction musks My
by OTF
Eara
Arthmuri:
valle:
Figure 8: Secure polynomial reduction.
in references such as (Kim et al. 2022 1021:Lo Lu et al. 2025 Pang et al. 2024). The specific configurations used are outlined in the Appendix
3.4 NETWORK OPTIMIZATION FOR ENCRYPTED PRUNING AND POLYNOMIAL REDUCTION
To support online token pruning and non-linear approximation, an offline fine-tuning method is needed to optimize pruning and approximation thresholds, and 3, to minimize inference overhead and achieve user-defined accuracy above This process is challenging because (1) previous studies have not incorporated the efficiency and accuracy of encrypted token pruming and non-linear approximation into the fine-tuning phase, and (2) the thresholds for different Transformer layers vary and are
7

---

Published as a conference paper at ICLR 2025
Algorithm 1 Crypto-aware Thresholds Learning
Input: pre-trained Transformer M,, training data 2,, initial thresholds 0,3
Set model M with layers, weights w. input tokens æ. accuracy requirement a, and hyperparameters T, A, a.
2. Search for optimal thresholds 0, 3 and weights w on data D.
(a) For I [L]. set soft masks for token 74, Mg (x:) = = ( and M3 (x,, = a 8th (0)))))")))
(b) For 1 E [71, integrate the erypto-friendly polynomial activation functions. In the 1-th layer, compute
GELU function as: GZ) 2 (ax) GELU(r:) + (1 , (x:)) ApproxGELU((z)) Except
for the first layer, compute SoftMax function as: SoftMax(x:) = M3 (x.)) SoftMax(z:) + (1
Ma (s.)) ApproxSoftMax(zs). For the input feature rin. compute the output feature Houe in the I-th
layer as: Zout = .M (x.n) -out
(c) Update w, 0 and /9 jointly to minimize the loss L, where L = Lrask + X(Cprune + aCCpproz.).
3. Finetune w on data ID with learned thresholds 9 and (3.
(a) Fix the threshold 0, 3. Binarize the mask M in every layer
if (z.)) g(1) and M is hinarized in the same way. otherwise.
(as) (m))
(b) Update w to minimize the loss Lrask. Derive the optimal fine-tuned transformer M'. 4. Output 0, 3and w if accuracy otherwise back to step 2.
challenging to pinpoint. To address these challenges, we introduce a crypto-aware fine-tuning method, outlined in Algorithm [ This method uses gradient search approach and proposes incorporate crypto-aware pruning and approximation into the training phase. Additionally, the loss functions are designed to optimize both efficiency and accuracy.
After initialization, we make the masks differentiable during fine-tuning to allow for trainable thresholds. as shown in step (. of Algorithm [ Here, T represents the temperature, and a is the Sigmoid function. This soft mask, a differentiable approximation of the binary mask, enables gradient-based updates to 0 and B. In step 2b(b) of the same algorithm, we introduce polynomial activation functions during the fine-tuning phase. ApproxSoftMax and ApProxGELU are low-degree polynomial approximations of the SoftMax and GELU functions. The ApproxSoftMax replaces the exponential function ex in original Soft with a Taylor series (1 + 2n
z )²" And the ApproxGELU
leverages simple polynomials such as (z) = 0..1 0.42x 2
0. - 0.01I to approximate
GELU. We defer the detailed polynomial in Equations 5 and ii the Appendix C If token x; 's importance score exceeds the threshold B3 it activates mainly through the original SoftMax or GELU functions; otherwise, through their polynomial approximations.
L-1
Cprane T,
(x)( Lappros. =
1 Σ MS (1))
(3)
The overall objective function is designed to minimize the loss function = Crask + X(Cprune a aappproz.) where L denotes the number of layers. Lusk optimizes accuracy for downstream tasks, while Eprune and Lupprox., defined by Md's 's and M3 lllssssl-nn In-mms respectively, target efficiency as detailed in Equation 3 The hyperparameters À and dictate the extent of pruning and approximation, with higher values leading to increased pruning or approximation. This structure introduces additional gradients, pushing 0 and /3 towards minimizing Lprue and Lappno. - Once the optimized and determined, we fix them and proceed to fine-tune the model weights to meet the accuracy requirement a as specified in Step For each Transformer layer, we binarize the masks Mj and Mi to select tokens for pruning or approximation. Subsequently, we update the model weights to minimize the downstream task loss Lrask-
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models and Datasets. We evaluated CipherPrune on the GPT2-Base and three BERT variants (Devlin) et al. 2018): BERT-Medium, BERT-Base, and BERT-Large. These models are commonly used in private Transformer frameworks. Similar to prior work (Pang et al. 2024), we fine-tune the BERT models four downstream NLP tasks in GLUE benchmarks (Wang al. 2018): the Multi-Genre Natural Language Inference Corpus (MNLI), the Stanford Question Answering Dataset (QNLI), the Stanford Sentiment Treebank (SST-2), and the Microsoft Research Paraphrase Corpus (MRPC).
8

---

Published as a conference paper at ICLR 2025
System Setup and Implementation. We encode floating-pon point parameters in Transformers into exed-point numbers and set the scale according to prior works et al. 2022 Lu et al. 2025
Pang
et al. 2024). CipherPrune uses the EzPC (EzP 2023) framework and the SEAL (SEA) 2023 hibrary.
EzPC compiles TensorFlow-base
deep neural networks into secure computation protocols running
on cryptographic backends. We simulate LAN with 3Ghps bandwidth and 0.8ms ping, and WAN with
200Mbps bandwidth and 40ms ping. following (Pang et al. (024). All experiments are conducted on an AMD Ryzen Threadripper PRO 3955WX (2.2GHz, 125GB RAM) and fine-tuning of the BERT model with threshold learning is done on NVIDIA GeForce RTX 3090 GPUs with CUDA 11.0.3.
4.2 RESULTS
Table 1: End-to-ens comparison of CipherPrune with prior works on BERT models. Time is in seconds. Comm. stands for communication in GB and Acc. for accuracy in percentage.
BERT Medium
BERT Base
BERT Large
Method
Time
Comm.
Acc.
Time
Comm.
Acc.
Time
Comm.
Acc.
IRON (Iao 2022
442.4
124.5
87.7.0.2
1087.8
281.0
90.41..1
2873.5
744.8
92.74t..
BOLT W.E. [Pan ct 2024
197.1
27.9
87.420.5
484,5
59.6
90.3
1279.8
142.6
92.620.2
BOLT (Pang et al. 2024
99.5
14.3
87.220.3
245.4
25.7
2..
624.3
67.9
92.4g0.2
CipherPrune
43.6
6.7
87.4
79.1
9.7
20.
157.6
18.4
92.5
End-to-end performance. In Table evaluate CipherPrune three BERT models, comparing it with previous private Trasnformer frameworks: IRON (Hao et al. 2022) and BOLT (Pang et al. 2024). CipherPrune achieves up ~ 18.2x speedup over IRON the BERT-Large model and 8.1x speedup over vanilla BOLT without W.E.. When compared with BOLT with the word elimination technique, CipherPrune is still 3.9x faster without compromising accuracy. Communication costs are also reduced by 2.3 ~ 40.4 x compared to prior works. Compared with BOLT, CipherPrune can remove more redundant tokens during inference thorough the adaptive and progressive pruning strat- egy. Moreover, CipherPrune also leverages low-degree polynomials to further reduce the computation and communication overhead. CipherPrune can easily extend other frameworks. Comparison with more related works like BumbleBee (Lu et al. 2025) MPCFormer (Li et al. 2022) and PUMA (Dong
al. 2023) can be found in Appendix D
Table 2: Accuracy and time comparisons of different methods.
GPT2-Baseling
CipherPrune stands for CipherPrune with token pruning only.
Runtime (min)
GPT2-BOLT
80
CipherPrune
Accuracy Metric Tasks (%)
Method
Time(Sec)
20
MNLI
QNLI
SST2
MPRC
BOLT w/o W.F.
84.75
90.32
91.74
90.53
484.5
BOLT
84.71
89.94
92.74
89.95
245.4
32
512
Number of Input Takens
CipherPrune"
84.74
90.17
92.75
90.45
115.3
CipherPrune
84.68
90.11
92.66
90.18
79.1
Figure 9: Runtime GPT2.
Token pruning and polynomial reduction. Table demonsttatss the the effects of the main design blocks in CipherPrune: adaptive token pruning and polynomial reduction. Our baseline is the vanilla BOLT framework without W.E.. BOLT's W.E. removes 50% of the input tokens and effectively cuts the overhead of cryptographic protocols by half. With fine-tuning, the W.E. incurs only marginal accuracy loss. Yet, the adaptive and progressive token pruning in CipherPrunef can further improve the utility-accuracy trade-off. Instead of setting the pruning ratio as 50% manually. CipherPrune! adaptively decides the pruning ratio based on both the input length and content. This contributes to up to 0.5% better accuracy. On the other hand, the progressive pruning CipherPrune allows to remove more redundant information, contributing to 2.1x runtime speed up over BOLT with W.E.. By incorporating polynomial reduction, CipherPrune can achieve up to 6.1x speed up over BOLT. While the accuracy drops slightly from CipherPrune' it is still comparable even higher than BOLT.
Scalability with the input length. In Figure 9 we compare the runtime of CipherPrune and BOLT with varying input token numbers GPT2. The baseline is BOLT without W.E.. The quadratic complexity of Transformer inference makes it challenging for BOLT to scale to long inputs. Although W.E. can reduce the overhead of private inference by half, BOLT with W.E. still scales quadratically

---

Published as a conference paper at ICLR 2025
with the number of input tokens. In contrast, CipherPrune demonstrates increasingly significant runtime savings as the input length grows, With 32 input tokens, CipherPrune achieves a 1.9x speedup. When the imput length reaches 512 tokens, CipherPrune is 10.6x faster than the baseline. The details about private inference on GPT2 models can be found in Appendix D
BOLT
CipherPrune
CipherPrune
BOLI Sort
Swap
LAN (3Gbps, 0.8ms)
CisherPrure
300
10
20*
Time(min)
200
90
Tima(Sec)
20"
100
80
20-1
Linear
SoftMax
LayerNorm
total
WAN (200Mbps, 40ms)
60
95
20-2
Time(Sec)
Time(min)
80
Act
900
Layer Number
500
20
Figure 11: Runtime com- parison of different prun- ing protocols.
Linear
SoftMax
GELU
LayerNorm
Prune
total
Figure 10: Runtime breakdown BERT-Base model.
Runtime breakdown. In Figure 10 break down the runtime for each protocol in the BERT-Base model with 128 input tokens. In the LAN setting, the communication is efficient and the main bottleneck is the HE-based linear operation. In contrast, the massive communication of the non-linear operations becomes the bottleneck in the WAN setting. Since pruned tokens are excluded from the computation in all subsequent layers, CipherPrune can effectively reduce the overhead of both linear and non-linear operations. This contributes to CipherPrune' 's efficiency in both LAN setting and WAN setting. As shown in Figure I0] the proposed pruning protocols in CipherPrune are lightweight, accounting for only 1.6% of the total runtime. This is because II prane leverages ASS to offload substantial computation the local side, such as accumulating the importance score. Additionally, Maskk utilizes the number of tokens in each layer to avoid sorting the whole token sequence.
we compare the efficiency of different pruning protocols. BOLT's W.E. uses Bitonic sort to sort the whole token sequence, which needs (n log" n) oblivious swaps. In CipherPrune, the client and server only need (mm) oblivious swaps to relocate and prune the less important tokens. Since only small number of tokens are removed each layer, CipherPrune has linear complexity to in general. By binding the mask with tokens on the MSB, CipherPrune can handle the token sequence and pruning mask in go and achieves 2.2 20.3x speed up.
Analysis on different pruming protocols. As shown in Figure IT
d-
0.1
Accuracy(55)
-605
0.01
90
40.07
88
87
90
270
Latency(Sec)
0.05
Study on the pruning parameters. In Figure 12 we show the accuracy-lateng, trade-off for the BERT-Base model under dif- ferent parameter settings. Larger X and a result in more tokens being pruned or reduced. With À less than 0.05, appropri- ate ratio of tokens is pruned, maintaining stable accuracy of around 90%. Smaller or leads more tokens being computed with high-degree polynomials, which increases accuracy but also latency. Notably, accuracy with a large is higher than with large X. This is because many tokens are reduced but not discarded, preserving necessary information for accurate inference.
Accura 19
91.0
90.5
0.0
2=0.1
90.0
89.5
--02
89.03
110
140
170
Latency(Sec)
Figure 12: Ablation study on hyper- parameters and a.
5 CONCLUSION
The proposed CipherPrune addresses the critical efficiency and scalability challenges of private Transformer inference by introducing novel approach that combines encrypted token pruning and polynomial reduction protocols. By progressively pruming redundant tokens and reducing the polynomial degree for less important tokens, CipherPrune significantly reduces runtime overhead while maintaining accuracy. Our experiments confirm its effectiveness, achieving substantial reduction in execution time compared to previous methods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ezpe.
2023.
Seal.
2023.
Gilad Asharov, Yehuda Lindell, Thomas Schneider, and Michael Zohner. More efficient oblivious transfer and extensions for faster secure computation. Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pp. 535-548, 2013.
Dan Bogdanov, Sven Laur, and Ríivo Talviste. A practical analysis of oblivious sorting algorithms for secure pulti-party computation. In Nordic Conference on Secure IT' Systems, pp. 59-74. Springer, 2014.
Zvika Brakerski. Fully homomorphic encryption without modulus switching from classical gapsvp. In CRYPTO, volume 7417, pp. 668-886. Springer, 2012.
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin preserving transformer inference with homomorphic encryption.
Li and Furu Wei. The-x: arXiv preprint arXiv:2206.002/11 2022.
Ronald Cramer, Ivan Bjerre Damgärd, et al. Secure multiparty computation. Cambridge University Press, 2015.
Jinming Cui, Chaochao Chen, Lingjuan Lyu, Carl Yang, and Wang Li. Exploiting data sparsity in secure cross-platform social recommendation. Advances in Neural Information Processing Systems, 34:10524-10534, 2021.
Xianglong Deng. Shengyu Fan, Zhicheng Hu, Zhuoyu Tian, Zihao Yang. Jiangrui Yu, Dingyuan Cao, Dan Meng, Rui Hou, Meng Li, Qian Lou, and Mingzhe Zhang. Trinity: A general purpose fhe accelerator. Accepted by MICRO 24, 2024.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 1810.04805 2018.
Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Chen. Puma: Secure inference of Ilama-7b five minutes. arXiv preprint ar arXiv:2307.12555
2023.
Junfeng Fan and Frederik Vercauteren. Somewhat practical fully homomorphic encryption. IACR Cryptol. ePrint Arch. pp. 144, 2012.
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Na- jibi. Lazyllm: Dynamic token pruning for efficient long context IIm inference. arXiv preprint arXiv:2407.14057 2024.
Craig Gentry. A fully homomorphic encryption scheme. Stanford university, 2009.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pp. 3690-3699, PMLR, 2020.
Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing. Guowen Xu, and Tianwei Zhang. Iron: Private inference on transformers. Advances Neural Information Processing Systems, 35:15718-15731, 2022.
Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren. Ciphergpt: Secure two-party gpt inference. Cryptology ePrint Archive, 2023.
Zhicong Huang. Wen-jie Lu, Cheng Hong, and Jiansheng Ding. Chectah: Lean and fast secure {two- party) } deep neural network inference. In 31st USENIX Security Symposium (USENIX Security 22), pp. 809, 2022.
Lei Jiang, Qian Lou, and Nrushad Joshi. Matcha: A fast and energy-efficient accelerator for fully homomorphic encryption over the torus. In The Design Automation Conference (DAC 2022), 2022.
11

---

Published as a conference paper at ICLR 2025
Gyuwan Kim and Kyunghyun Cho. Length-adaptive transformer: Train once with length drop, anytime with search. arXiv preprint arXiv:2010.07003, 2020.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 1-bert: Integer- only bert quantization. In International conference on machine learning, pp. 5506-55065 5518. PMLR, 2021.
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 784-794, 2022.
Vladimir Kolesnikov and Ranjit Kumaresan, Improved of extension for transferring short secrets, In Advances in Sypptology-CYPPOO 2013: 33rd Annual Cryptology Conference, Santa Barbara, CA, USA, August 18-22 22, 2013. Proceedings, Part II, pp. 54-70. Springer, 2013.
Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P Xing, and Hao Zhang. Mpcformer: fast, performant and private transformer inference with mpc. arXiv preprint arXiv:2211.01452. 2022.
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you looking for before generation. Advances in Neural Information Processing Systems, 37:22947-2299 -29700, 2024.
Qian Lou and Lei Jiang. She: A fast and accurate deep neural network for encrypted data. In Advances in Neural Information Processing Systems (NeurIPS) 2019, pp. 10035-10043, 2019.
Qian Lou, Bo Feng, Geoffrey C Fox, and Lei Jiang. Glyph: Fast and accurately training deep neural networks on encrypted data. NeurlPS 2020 (Advances in Neural Information Processing Systems), 2019.
Qian Lou, Yilin Shen, Hongxi Jin, and Lei Jiang. Safenet: A secure, accurate and fast neural network inference. 2021.
Wen-jie Lu, Zhicong Huang, Zhen Gu, Jingyu Li, Jian Liu, Cheng Hong, Kui Ren, Tao Wei, and Wenguang Chen. BumbleBee: Secure Two-party Inference Framework for Large Transformers. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025, The Internet Society, 2025.
Jinglong Luo, Yehong Zhang, Zhuo Zhang, Jiaqi Zhang, Xin Mu, Hui Wang, Yue Yu, and Zenglin Xu. Secformer: Towards fast and accurate sieeacyppreeerving inference for large language models. arXiv preprint arXiv:2401.0077 00793, 2024.
Qi Pang. Jinhao Zhu, Helen Möllering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy- preserving, accurate and efficient inference for transformers. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 4753-4771. IEEE, 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta. Aseem Rastogi, and Rahul Sharma. Cryptflow2: Practical 2-party secure inference. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. --2--42, 2020.
Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nishanth Chandran, and Aseem Rastogi. Simn: A math library for secure rnn inference. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 1003-1020. IEEE, 2021.
Muhammad Husni Santriaji, Jiaqi Xue, Yancheng Zhang, Qian Lou, and Yan Solihin. Dataseal: Ensuring the verifiability of private computation on encrypted data. In 2025 IEEE Symposium on Security and Privacy (SP), pp. 78-78. IEEE Computer Society, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
12

---

Published as a conference paper at ICLR 2025
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 0rXiv:180..07461, 2018.
Hanrui Wang,
Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In IEEE International Symposium on High-Perfforance Computer Architecture, pp. 97-110. IEEE, 2021.
Bingxin Xu, Yuzhang Shang, Yunhao Ge, Qian Lou, and Yan Yan. freepruner: training-free approach for large multimodal model acceleration. arXiv preprint arXiv:2411.1544 2024a.
Tianshi Xu, Lemeng Wu, Runsheng Wang, and Meng Li. Priveirmt:: Efficient private inference via block circulant transformation. arXiv preprint arXiv:2405. 14569, 2024b.
Jiaqi Xue, Yancheng Zhang, Yanshan Wang, Xueqiang Wang. Hao Zheng, and Qian Lou. Cryptotrain: Fast secure training on encrypted dataset. In Proceedings of the 1st ACM Workshop on Large Al Systems and Models with Privacy and Safety Analysis, pp. 97-104, 2023.
Jiaqi Xue, Mengxin Zheng, Ting Hua, Yilin Shen, Yepeng Liu, Ladislau Bölöni, and Qian Lou. Trojllm: black-box trojan prompt attack on large language models. Advances Neural Information Processing Systems, 36, 2024.
Ardhi Wiratama Baskara Yudha, Jiaqi Xue, Qian Lou, Huiyang Zhou, and Yan Solihin. Boostcom:
Towards efficient universal fully homomorphic encryption by boosting the word-wise comparisons. In (P The International Conference Parallel Architectures and Compilation Techniques (PACT), 2024.
Wenxuan Zeng. Meng Li, Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpevit: Searching for accurate and efficient mpe-friendly vision transformer with heterogeneous attention. In JEEE/CVF International Conference on Computer Vision, pp. 5052- 5063, 2023.
Jiawen Zhang, Xinpeng Yang. Lipeng He, Kejia Chen, Wen-jie Lu, Yinghao Wang, Xiaoyang Hou, Jian Liu, Kui Ren, and Xiaohu Yang. Secure transformer inference made non-interactive. Cryptology ePrint Archive, 2024.
Yancheng Zhang, Xun Chen, and Qian Lou. Hebridge: Connecting arithmetic and logic operations in st-style he schemes. In Proceedings of the 12th Workshop on Encrypted Computing and Applied Homomorphic Cryptography, pp. 23-35, 2023a.
Yuke Zhang, Dake Chen, Souvik Kundu, Chenghao Li, and Peter A Beerel. Sal-vit: Towards latency efficient private inference on vit using selective attention search with a learnable softmax approximation. In IEEE/CVF International Conference on Computer Vision, pp. 5116-5125, 2023b.
Mengxin Zheng, Qian Lou, and Lei Jiang. Trojvit: Trojan insertion in vision transformers. CVPR 2023, 2022.
Mengxin Zheng, Qian Lou, and Lei Jiang. Primer: Fast private transformer inference on encrypted data. DAC 2023, 2023.
Mengxin Zheng, Cheng Chu, Qian Lou, Nathan Youngblood, Mo Li, Sajjad Moazeni, and Lei Jiang. Ofhe: An electro-optical accelerator for discretized tfhe. In Proceedings of the 29th ACM/IEEE International Symposium Low Power Electronics and Design, pp. 1-6, 2024.
Itamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov, Omri Soceanu, and Lior Wolf. Converting transformers to polynomial form for secure inference over homomorphic encryption. arXiv preprint arXiv:2311.08011 2023.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A SECURE TOKEN PRUNING PROTOCOLS
We detail the encrypted token pruning protocols Пlrune in Figure 13 and massk in Figure 14 in this section.
Parties: Server Po. Client Pt.
Input: Po and P1 holds {(Att)) h (Att) HH-1 E EZ and (x) (2); E Ez respectively, where H is the number of heads, is the number of input tokens and D is the embedding dimension tokens. Additionally, P1 holds a threshold E Zge.
Output: Po and Pi get (W) (v), E Zyyx xD, respectively, where Prune(x) and n' is the number of remaining tokens.
Protocol:
I: For h E [H] Po and Pi compute locally with input (Att)", and learn the importance score in each head (s)" E Ziz,, where (s)* [}] = Σi=0 (Att)* [1
2: Po and Pi compute locally with input {(s) E Zige J and learn final importance score (S) Z for each token, where (5) [1] Σ (s) [t].
3: For i E [n]. Po and P1 invoke ΠMMP with inputs (S) and 0, and learn (M) E such that( (M) -lIMM(((S)) [i] - 0), where:
if STi] > 8,
[ 0 otherwise.
4: Po and P1 invoke mask with inputs (I) and pruning mask (M), and set outputs as (v).
Figure 13: Secure Token Pruning Protocol II prrnee
Parties: Server Po. Client Pt.
Input: Po and P1 hold (x) : (xe), E Zny. and (M) (M), E Zy. respectively, where n is the number of imput tokens and D is the embedding dimension of tokens.
Output: Po and P1 get (w) (w), zy'xD, respectively, where y Prune(x) and n' is the number of remaining tokens.
Protocol:
I: For E [m]. Po and Pı set (M) to the MSB of (x) and learn the masked tokens (T) E Z where (z(1)) (zl()) + ((M(i) {) and f is the fixed-point precision.
2: Po and P1 compute the sum of (TT24((M) [=)) and learn the number of remaining tokens n' and the number of tokens be pruned m, where m n n'.
3: For k E [m]] for E In A 1]. Po and P1 invoke 1lmab learn the highest bit of (z(i)). where = (J)).
With the highest bit of Z[i]. Po and Pı perform oblivious swap
[ = b. = (1 b) Fli +1
between [ and o(i + 1]:
Lii + 1] b øii + 1] +
Po and P1 learn the swapped token sequence (ā)
4: Po and Pi truncate (ū) locally by keeping the first n' tokens, clear current MSB (all remaining token has on the MSB), and set outputs as (y).
Figure 14: Secure Mask Protocol I Imask-
Complexity of mmakk-- The complexity of the proposed Imaak mainly depends the number of oblivious swaps. To prune m tokens out of n input tokens, ((m)) swaps are needed. Since token pruning performed progressively, only small number of tokens are pruned each layer, which
14

---

Published as a conference paper at ICLR 2025
makes makk efficient during runtime. Specifically, for BERT base model with 128 input tokens, the pruning protocol only takes ~ 0.9s on average in each layer. An alternative approach is to invoke an oblivious sort algorithm (Bogdanov et al. 2014 Pang et al., 2024) on (ž). However, this approach is less efficient because it blindly sort the whole token sequence without considering m. That is, even O(n²) oblivious swaps needed, where as the
if only token needs to be pruned, (nloggén) proposed msaak only need (nn) swaps. More generally, for an E-layer Transformer with a total of m tokens pruned, the overall time complexity using the sort strategy would be O(én²) while using the swap strategy remains an overall complexity of O (mm)). Specifically, using the sort strategy to prune tokens in one BERT Base model layer can take up to 3.8 4.5 s depending on the sorting algorithm used. In contrast, using the swap strategy only needs 0.5 s. Moreover, alternative to our MSB strategy, one can also swap the encrypted mask along with the enerypted token sequence. However, we find that this doubles the number of swaps needed, and thus is less efficient the our MSB strategy, as is shown in Figure
B EXISTING PROTOCOLS
Existing Protocols Used in Our Private Inference. In our private inference framework, we reuse several existing cryptographic protocols for basic computations. aa (Pang et al.) (024) processes two ASS matrices and outputs their product in SS form. For non-linear computations,
protocols Ma LIGELU, and Loaeer (Lu et al., 2025 Pang al. 2024) take a secret shared tensor and return the result of non-linear functrons in ASS. Basic protocols from (Rathee et al., 2020; 2021) also utilized. (CMp 2023). for example, inputs ASS values and outputs secret shared comparison result, while II EzP 2023) converts secret shared Boolean values into their corresponding arithmetic values.
C POLYNOMIAL REDUCTION FOR NON-LINEAR FUNCTIONS
The SoftMax and GELU functions can be approximated with polynomials. High-degree polynomi- als (Lu et al. 2025 Pang et al. 2024) can achieve the same accuracy as the LUT-based methods Hao ce al. (2022). While these polynomial approximations are more efficient than look-up tables, they can still incur considerable overheads. Reducing the high-degree polynomials to the low-degree ones for the less important tokens can imporve efficiency without compromising accuracy. The SoftMax function is applied to each row of an attention map. If a token is to be reduced, the corresponding row will be computed using the low-degree polynomial approximations. Otherwise, the corresponding row will be computed accurately via high-degree one. That is if M3 1. Po and P1 uses high-degree polynomials to compute the SoftMax function on token ar(i]:
SoftMax; Ejeld e
(4)
where x is a input vector of length and the exponential function is computed via a polynomial approximation. For the SoftMax protocol, we adopt similar strategy as (Kim et al. 2021 Hao et al. 2022), where we evaluate on the normalized inputs SoftMax(x maz;g]/];) Difterent from (Hao et al. 2022), we did not used the binary tree to find max value in the given vector. Instead, we traverse through the vector to find the max value. This is because each attention map computed independently and the binary tree cannot be re-used. If M 0, Po and P1 will approximate the SoftMax function with low-degree polynomial approximations. We detail how SoftMax can be approximated follows:
ApproxExp(x:))
ApproxSoftMax, (x) = ΣjE[d ApproxExp(2x)
(5)
((1 )1+ if E [T,
10
if S
ApproxExp(x)
(6)
where the 2" -egrree Taylor series is used to approximate the exponential function and T is the clipping boundary. The value and T determines the accuracy of above approximation. With 6
and T = 113, the approximation can achieve an average error within 2 2-10 (Lu et al. 2025). For low-degree polynomial approximation,
is used in the Taylor series.
15

---

Published as a conference paper at ICLR 2025
Similarly, Po P1 decide whether not approximate the GELU function for each token. If Ma i = =1, P. and P1 use high-degree polynomials (Lu et 2025 to compute the GELU function on token æ[i] with high-degree polynomial:
0
if S -5
(r),,, if - <5< 1197 1.97 23 i
ApproxGELU(r)
(1),,
2.
where (2x) and (ar) are degree-3 and degree-6 polynomials respectively. The detailed coefficient for the polynomial is:
(x) 0.50540312 0 0.42226581r 0. 0.11807613.
0.01103413.*
and p6 (x) = 0.00852632 + 0.5z + 0.36032927 0.03768820z + 0.00180675.
For BOLT baseline, use another high-degree polynomial compute the GELU function.
if x < -27
(p)r)
(*)x,,
if | < 2.7
88)
if x 2.7
We use the same coefficients for (z) as BOLT
Pang et al. 2024
IRON
BumbleBee
CipherPrune
10a
Time(Sec)
Comm. Comm.(GB) (GB)
102
102
G Large
20
20
100
2
Medium
Base BERT Architecture
Medium
Base BERT Architecture
Large
Figure 15: Comparison with prior works on the BERT model. The input has 128 tokens.
If Mg1 = 0. Po and P1 will use low-degree polynomial approximation to compute the GELU function instead. Encrypted polynomial reduction leverages low-degree polynomials to compute non-linear functions for less important tokens. For the GELU function, the following degree-2 polynomial Kim al. (021) is used:
0
if -1.762
ApproxGELU(2)
0.5r 0.28367x
r S [1.7626] if r > 1.7626
D COMPARISON WITH MORE RELATED WORKS.
Other 2PC frameworks. The primary focus of CipherPrune is to accelerate the private Transformer inference in the 2PC setting, As shown in Figure 15] CipherPrune can be casily extended other 2PC private inference frameworks like BumbleBee (Lu et al. 2025). We compare CipherPrune with BumbleBee and IRON BERT models. We test the performance in the same LAN setting as BumbleBee with Gbps bandwidth and 0.5 ms of ping time. CipherPrune achieves more than 60x speed up BOLT and 4.3x speed up over BumbleBee.
Extension to 3PC frameworks. Additionally, we highlight that CipherPrune can be also ex- tended to the 3PC frameworks like MPCFormer (L.i et al. 2022) and PUMA (Dong et al.
16

---

Published as a conference paper at ICLR 2025
MPCFormer
PUMA
CipherPrune
300
50
Comm. (GB)
40
200
30
Time
150
20
100
50
10
Medium
Base BERT Architecture
Large
Medium
Base
Large
BERT Architecture
Figure 16: Comparison with MPCFormer and PUMA on the BERT models. The input has 128 tokens.
MPCFormer
PUMA
CipherPrune
600
10a
Comm. (GB)
Time(Sec)
500
400
102
B Large
200
10
100
0
100
Base
Medium
Large
Base
Medium
GPT2 Architecture
GPT2 Architecture
Figure Comparison with MPCFormer and PUMA on the GPT2 models. The input has 128 tokens. The polynomial reduction is not used.
2023). This is because CipherPrune built upon basic primitives like comparison and Boolean- 1o-Arithmetic conversion. We compare CipherPrune with MPCFormer and PUMA on both the BERT and GPT2 models. CipherPrune has 6.6 9.4x speed up MPCFormer and 2.8 ~ 4.6x speed up over PUMA on
the BERT-Large and GPT2-Large models. For a fair comparison with prior works (Li al., 2022] Dong et al.) 2023), we report the runtime and communication cost mea- sured from the prefilling phase. As shown in Figure 18 perform private inference on GPT2 models to generate one token. Similar to PUMA, we do not consider the whole autoregressive decoding process. We note that the Key-Value (K cache and its
Inpus
Grnerated
taken
token
Ey
Prefilling
Decoding step
evalustiooo
Figure 18: Illustration of evaluation on GPT2 models.
compression (Li et al. 2024) can further improve the efficiency of the decoding phase for GPT2 models. We leave private KV cache compression for future work.
COMMUNICATION REDUCTION IN SOFTMAX AND GELU.
Table 3: Communication cost (in MB) of the SoftMax and GELU protocol in each Transformer layer.
Layer Index
2
3
7
10
11
Softmax
642.19
642.19
642.19
642.19
642.19
642.
642.
642.19
642.19
642.
642.
642.19
Pruned Softmax
642.19
129.58
127.89
119.73
97.04
71.52
43.92
21.50
10.67
6.16
4.03
GELU
698.84
698.84
84
698.84
84
698.84
698.84
698.84
698.84
698.8
84
698.84
698
84
698.84
698
Praned GELU
325.10
317.18 313.43
275.94
236.95
191.96
135.02
88.34
46.68
16.50
5.58
In Figure we illustrate why CipherPrune can reduce the communication overhead of both SoftMax and GELU. Suppose there are tokens layer, Then, the SoftMax protocol in the attention module has complexity of O(n?). CipherPrune's 's token pruning protocol invoked to select tokens n' is the number of tokens that removed. The overhead
out of all tokens, where
17

---

Published as a conference paper at ICLR 2025
n tokens
n'tokens
n" tokens
Attention
Prune de Reduce
Add Narm
Add & Norm
Attention
Pruno & Roduco
Add Narm
pp
innar
Linear
Linear
GELU
Lncar
Layer:
Layen++1
Toy example of two successive Transformer layers. In layer, the SoftMax and Prune protocol have n imput tokens. The number of input tokens is reduced to n' for the Linear layers, LayerNorm and GELU in layer, and SoftMax in layer
Figure 19:
of the GELU function in layer;, i.e., the current layer, has only O(n') complexity (which should be (n) without token pruning). The complexity of the SoftMax function in loyer,+1, i.e. the following layer, reduced On²²² (which should be O(n²) without token pruning). The SoftMax protocol has quadratic complexity with respect to the token number and the GELU protocol has linear complexity. Therefore, CipherPrune can reduce the overhead both GELU protocol and the SoftMax protocols by reducing the number of tokens. In Table 3 provide detailed layer-wise communication cost of the GELU and the SoftMax protocol. Compared to the unpruned baseline, CipherPrune can effectively reduce the overhead of the GELU and the SoftMax protocols layer by layer.
F ANALYSIS ON LAYER-WIE REDUNDANCY.
Pruning protocol runtime Number of pruned tokens
#Pruned tokens
Runtime(Sed
0.5
0
5 Layer index
10
11
Figure 20: The number of pruned tokens and pruning protocol runtime in different layers in the BERT Base model. The results are averaged across 128 QNLI samples.
In Figure 20 we present the number pruned tokens and the runtime of the pruning protocol for each layer in the BERT Base model. The number of pruned tokens per layer was averaged across 128 QNLI samples, while the pruning protocol runtime was measured over 10 independent runs. The mean token count for the QNLI samples is 48.5. During inference with BERT Base, input sequences with fewer tokens are padded to 128 tokens using padding tokens. Consistent with prior token pruning methods in plaintext (Goyal et al.) 2020), significant number of padding tokens are removed at layer layer 0, the number of pruned tokens is primarily influenced by the number of padding tokens rather than loken-level redundancy.
In CipherPrune, tokens are removed progressively, and once removed, they are excluded from computations in subsequent layers. Consequently, token pruning in earlier layers affects computations in later layers, whereas token pruning in later layers does not impact earlier layers. As a result, if layers 4 and 7 remove the same number of tokens, layer 7 processes fewer tokens overall, as illustrated in Figure 20 Specifically, tokens are removed in both layer 4 and layer 7, but the runtime of the pruning protocol in layer 4 is 2.4x longer than that in layer 7.
18

---

Published as a conference paper at ICLR 2025
G RELATED WORKS
In response to the success of Transformers and the need to safeguard data privacy, various private
Transformer Inferences (Chen al., 2022; Zheng et al. 2023 Hao et 2022 Li et al. 2022 Lu et al. 2025 Luo et 2024) Pang et al. 2024) are proposed. To efficiently run private Trans- former inferences, multiple cryptographic primitives are used popular hybrid HE/MPC method IRON (Hao et all] (022), i.e., in Transformer, HE and SS are used for linear layers, and SS and OT are adopted for nonlinear layers. IRON and BumbleBee (Lu al. 2025) focus on optimizing linear general matrix multiplications; SecFormer| Luo et al. (2024) improves the non-linear operations like exponential function with polynomial approximation; BOLT (Pang et al. 2024) introduces the baby-step giant-step (BSGS) algorithm to reduce the number of HE rotations, proposes word elimi- nation (W.E.) technique, and uses polynomial approximation for non-linear operations, ultimately achieving state-of-the-ard (SOTA) performance.
Other than above hybrid HE/MPC methods, there are also works exploring
Transformer inference using only HE (Zimerman et al., [023 Zhang et al. 2024). The first HE- based private Transformer inference work (Zimerman et 2023) replaces SoftMax function with scaled-Rel.U function. Since the scaled-Rel.U function be approximated with low-degree polynomials more easily, it can be computed more efficiently using only HE operations. A range-loss term needed during training reduce the polynomial degree while maintaining high accuracy. A training-five HE-based private Transformer inference was proposed (Zhang et al.) (2024), where non-linear operations are approximated by high-degree polynomials. The HE-based methods need frequent bootstrapping, especially when using high-degree polynomials, thus often incurring higher overhead than the hybrid HE/MPC methods in practice.
19


Reviews:
[{"review_id": "2YZX5MFhde", "reviewer": "Reviewer_Qed2", "summary": "CipherPrune effectively removes unnecessary tokens by combining input-adaptive pruning based on importance scores with progressive layer-wise pruning. For the remaining tokens, it applies high-order and low-order polynomials according to their importance to maximize computational efficiency. This approach achieved 6.1x and 10.6x faster performance on inputs of 128 and 512 tokens, respectively, significantly improving inference speed without accuracy loss.", "questions": "See the weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Achieving a substantial improvement in speed, with a 6.1x increase for 128 tokens and a 10.6x increase for 512 tokens, without any reduction in accuracy, is remarkable.\n\n- The application of different polynomial orders based on importance scores, rather than merely pruning, is an impressive idea.\n\n- The paper clearly differentiates itself from existing studies by detailing each step, explaining what previous research overlooked, why these aspects are necessary, and how their work contributes to these areas.\n\n- All experimental settings are well-documented, providing extensive end-to-end results and comparisons across multiple models.\n\nThis paper demonstrates advancements over BOLT in every aspect. The time complexity of pruning and the resulting speed have both improved, with no difference in accuracy compared to BOLT. The authors provided a thorough analysis of BOLT, detailing the shortcomings of its ideas and the specific areas they have enhanced. In addition to building on ideas from previous studies, they introduced an innovative approach by applying polynomials of different orders based on importance scores. Comparisons with various papers are well-presented in the appendix, and it’s impressive that this technique can be applied to most inference models, including 2PC and 3PC models. I believe this is an excellent paper.", "weaknesses": "- Although the paper claims no accuracy reduction, there is a minor decrease of around 0.2%.\n\n- In terms of pruning, the only prior work compared is BOLT. (I’m unsure if there are other relevant papers, so it may be possible that only BOLT exists in this context.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihc6nL3TFL", "reviewer": "Reviewer_V3E7", "summary": "This paper introduces *CipherPrune*, a framework designed to mitigate the communication and latency overheads associated with both nonlinear and linear operations (FLOPs) in private inference for transformer-based models. CipherPrune introduces a token pruning method that assesses input token redundancy, progressively removing unimportant tokens layer by layer across the network, thereby leveraging task-level token redundancy to achieve efficiency without sacrificing predictive performance.  \n\nAdditionally, for less important tokens CipherPrune employs low-degree polynomial approximation for GELU and Softmax, in contrast with the higher degree polynomial approximation for critical input tokens, further enhancing efficiency without compromising security guarantees or predictive performance.", "questions": "See the weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "$\\bullet$ This work contributed to both the front-- protocol and network level-- and demonstrated the potential of a synergistic approach that achieves significant speedups and substantial communication reductions. \n\n$\\bullet$ The authors identify limitations in plaintext-based token pruning for private inference and introduce ciphertext-specific pruning techniques that avoid the need for decryption at each network layer, addressing a key bottleneck in efficiency. \n\n$\\bullet$ They effectively tackle the shortcomings of task-agnostic token pruning (such as word elimination) seen in prior work, demonstrating the advantages of a task-specific, layer-wise progressive token pruning strategy that optimizes both performance and efficiency. \n\n$\\bullet$ The paper is well-written and clearly outlines the challenges, the limitations of previous approaches, and the authors' contributions, making it easy to follow and appreciate the impact of this work.", "weaknesses": "$\\bullet$ Step 3 (Figure 4), prune and reduce, would increase the computational burden on the client side. A discussion on the resource burden (computation and memory) on the client should have been included, assuming the client has a low-end mobile device with limited compute and memory. \n\n$\\bullet$ This work does not address the overhead introduced by LayerNorm, which can add substantial communication and latency costs depending on the protocols used. For instance, in CipherGPT [3], LayerNorm accounts for **22%** of the total latency and communication overhead. \n\n$\\bullet$ One significant drawback of this paper is the absence of a detailed characterization of layerwise token redundancy. Without this, readers miss out on key insights into how redundancy varies across layers, which could have helped guide more efficient network design and optimization.\n\n$\\bullet$ Overall, the CipherPrune framework appears quite complex to understand, implement, and deploy. Its successful application requires expertise in both protocol design and network architecture, making it challenging for those without specialized knowledge in these areas. \n\n\n\n## Correction in the draft\n\n$\\bullet$ L#355 Pruning ratio larger than zero --> Pruning ratio greater than zero\n\n$\\bullet$ Figure 4 depicts the 2PC threat model for Post-LN configuration, and in the FFN block diagram, one linear layer (after the GELU layer) is missing. \n\n$\\bullet$ Missing citations [1] and [2]\n\n1.  Zimerman et al., Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption,  ICML 2024\n\n2.  Luo et al. SecFormer: Towards Fast and Accurate Privacy-Preserving Inference for Large Language Models, ACL 2024. \n\n3. Hou et al., CipherGPT: Secure Two-Party GPT Inference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1Y3qq5cdK", "reviewer": "Reviewer_Mztk", "summary": "This paper introduces an innovative framework designed to address the efficiency and scalability challenges of private Transformer inference. The proposed CipherPrune framework incorporates secure encrypted token pruning and polynomial reduction protocols to optimize performance. By progressively pruning unimportant tokens and assigning lower-degree polynomial approximations to less important tokens, CipherPrune achieves notable reductions in execution time and communication costs without compromising accuracy.", "questions": "Can you specify which types of NLP tasks or applications are most suitable for the CipherPrune framework, and which tasks may see limited benefits from the proposed token pruning approach?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The combination of encrypted token pruning and polynomial reduction is novel and well-executed, contributing to reduce PI costs and latency while maintaining model performance.\n\n2. The proposed CipherPrune demonstrates substantial improvements in runtime and communication efficiency.", "weaknesses": "As token pruning may be less effective in tasks that rely heavily on retaining comprehensive token-level information, such as machine translation, I suggest that the authors explicitly outline which applications are most suitable for this method and where its benefits may be more limited. This clarification would enhance the practical understanding and applicability of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZBrZbgUKT", "reviewer": "Reviewer_Xg5m", "summary": "The paper introduces **CipherPrune**, a framework designed to enhance the efficiency and scalability of private inference in Transformer models, particularly BERT and GPT2, by implementing secure token pruning and polynomial reduction protocols. It addresses challenges related to runtime overhead and scalability by utilizing cryptographic techniques such as correlated oblivious transfer and secure multi-party computation, achieving significant speed improvements (up to 18.2x faster than existing methods) without compromising accuracy. Additionally, the framework incorporates crypto-aware fine-tuning to optimize pruning and approximation thresholds during training, ensuring efficient inference while maintaining data privacy.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Hybrid Cryptographic Approach**: The paper introduces a hybrid framework that combines homomorphic encryption for linear operations and secure multi-party computation (MPC) for non-linear operations, significantly improving the efficiency and scalability of private inference in Transformer models like BERT and GPT2.\n\n- **Secure Token Pruning Protocol**: CipherPrune implements a secure token pruning protocol that evaluates token importance against a learned threshold, allowing for efficient pruning of unimportant tokens while preserving data privacy during inference.\n\n- **Crypto-aware Fine-tuning**: The framework incorporates crypto-aware fine-tuning to optimize pruning and approximation thresholds during training, ensuring that the model maintains high performance and accuracy while facilitating efficient private inference.", "weaknesses": "I have a concern about the communication cost comparison between SoftMax and GELU here. \n\nFrom the manuscript, the pruned tokens are selected after attention so that the communication from SoftMax here is still significant because you need to do compare, multiply, and etc.\n\nHow does the communication cost from SoftMax compared to one that GELU introduce?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)