SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC (2024)

Abstract: With the growing use of Transformer models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for Transformer models often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and difficult to circumvent or optimize effectively. To address this concern, we introduce a comprehensive PPI framework called SecFormer to achieve fast and accurate PPI for Transformer models. We successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance and develop a suite of efficient SMPC protocols by employing suitable numerical computation methods to boost other complex nonlinear functions in PPI, including GeLU, LayerNorm, and a redesigned Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of 3.4% and 24.7% for BERTBASE and BERTLARGE, respectively. In terms of efficiency, SecFormer is 3.57 and 3.58 times faster than PUMA for BERTBASE and BERTLARGE, demonstrating its effectiveness and speed. The code is available by clicking here.

heading: 1 Introduction
text: Transformer models (Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020; Liu et al., 2019; Lewis et al., 2020; Zeng et al., 2021; Ouyang et al.,
*Corresponding author
2022; OpenAI, 2023) demonstrate exceptional performance across diverse downstream tasks (Chan et al., 2024; Jiayang et al., 2023; Lu et al., 2019; Zhang et al., 2020; Lu et al., 2020; Liu et al., 2020; Zhang et al., 2023c; Wang et al., 2023) and are extensively employed in a Model-as-a-Service (MaaS) paradigm to deliver high-quality inference services to clients. However, this MaaS framework poses a significant privacy risk (Li et al., 2023b,c,d, 2024; Zhang et al., 2024b; Zhang and Zhu, 2020) for inference data and model parameters. For instance, both Copilot1 and ChatGPT2, which are Transformer-based services, necessitate users to upload plaintext requests. This operational procedure not only poses a threat to users’ privacy but also probably contravenes relevant legal regulations such as the EU’s General Data Protection Regulation (GDPR)3.
Secure Multi-Party Computation (SMPC) (Shamir, 1979; Yao, 1986; Goldreich et al., 1987), has demonstrated great potential in safeguarding the privacy of both inference data and model weights (Gilad-Bachrach et al., 2016; Liu et al., 2017; Mishra et al., 2020; Rathee et al., 2021; Huang et al., 2022; Luo et al., 2023; Zhang et al.,
1https://github.com/features/copilot 2https://chat.openai.com 3https://gdpr-info.eu/
13333
2023a, 2024a). However, conducting PrivacyPreserving Inference (PPI)4 for Transformer models using SMPC proves to be notably slow. To illustrate, BERTBASE (Devlin et al., 2019) takes 71 seconds to inference per sample via SMPC, while plain-text inference takes less than 1 second.
This inefficiency stems from the limitations of current SMPC protocols in executing nonlinear operations in Transformer models (Section 2.2). As depicted in Fig. 1(a), we find that Softmax and GeLU, which account for a small share of the plaintext inference overhead, take up 77.03% of the time in PPI. This observation aligns with findings in Wang et al. (2022); Li et al. (2023a). In an effort to mitigate this, Li et al. (2023a) redesigned the Transformer model by substituting Softmax and GeLU with some SMPC friendly quadratics, bypassing the privacy-preserving computations of the nonlinear operations (i.e., erf, exponential, and maximum) in Softmax and GeLU. This aggressive substitution significantly enhances PPI efficiency, but unfortunately, substantially diminishes the model’s performance and is not scalable to larger models (Fig. 1(b)). Some other studies (Dong et al., 2023) tried to boost the PPI by designing more efficient SMPC protocols, which can preserve the model performance but still face expensive PPI overhead.
In this study, we present a comprehensive PPI framework named SecFormer, tailed to achieve fast and accurate PPI for Transformer models by exploiting the superiorities of both Transformer model and SMPC protocol designs. Our investigation reveals that preserving accurate computation of GeLU significantly improves PPI performance (Fig. 1(b)). Building on this insight, SecFormer implements model design to bypass the expensive nonlinear PPI operations such as exponential and maximum in Softmax (Section 3.1). This adaptation, coupled with the strategic use of knowledge distillation, allows SecFormer to construct a Transformer model that is both high-performing and compatible with SMPC. To further enhance the PPI performance, we turn to protocol design and develop a suite of efficient SMPC protocols that utilize suitable numerical calculation methods to handle other complex nonlinear functions in PPI, such as GeLU, LayerNorm, and the redesigned Softmax (Section 3.2). To be specific, SecFormer introduces a novel SMPC protocol for GeLU based
4Without confusion, we refer to SMPC-based PPI as PPI for short in this paper .
on segmented polynomials and Fourier series, facilitating efficient and accurate computation of GeLU. In addition, SecFormer deploys efficient privacypreserving square-root inverse and division calculation for LayerNorm and Softmax using the Goldschmidt method (Goldschmidt, 1964; Markstein, 2004), coupled with input deflation techniques to bypass the nonlinear initial-value computation.
We conducted extensive evaluations of SecFormer on various datasets using Transformer models BERTBASE and BERTLARGE. The experimental results reveal that SecFormer achieves an average performance improvement of 3.4% and 24.7% for BERTBASE and BERTLARGE, respectively, compared to the state-of-the-art approach based on pure model design (Section 4.2), while maintaining comparable efficiency. In comparison to the approach that only improves the SMPC protocols, SecFormer exhibits a speedup of 3.57 and 3.58 times in PPI (Section 4.3), while sustaining comparable PPI performance.

heading: 2 Background and Related Works

heading: 2.1 Workflow of SMPC-based Model Inference
text: Secure Multi-Party Computation (SMPC) is a cryptographic technique that offers a promising solution for model Privacy-Preserving Inference (PPI) among multiple participants (Gilad-Bachrach et al., 2016; Liu et al., 2017; Mishra et al., 2020; Rathee et al., 2021; Huang et al., 2022). Typically, participants adhere to cryptographic primitives like secret sharing (Shamir, 1979; Goldreich et al., 1987) to safeguard the model weights and inference data. This paper mainly introduces the 2-out-of-2 secret sharing scheme due to its efficiency and representativeness. Specifically, the 2-out-of-2 secret sharing divides a secret x in the ring of integers ZL into two random shares [[x]] = ([x]0, [x]1) with x = (([x]0 + [x]1) mod L), ensuring that neither share reveals information about x while allowing correct reconstruction of x when the two shares are combined. In constructing the SMPC protocols, the shares are owned by two distinct participants. They communicate the masked intermediate results to each other to accomplish the privacypreserving computation of different functions and get the shares of the computational results.
The PPI workflow leveraging 2-out-of-2 secret sharing is depicted in Fig. 2. It involves three essential stakeholders: a model inference service
provider that needs to protect model weights, a client that needs to protect inference data, and an SMPC engine that performs model PPI. The SMPC engine contains three non-colluding servers (i.e., participants): two computing servers Sj for j ∈ {0, 1} for shares computation of PPI and an assistant server T for generating random numbers needed to execute the SMPC protocols. Initially, the service provider and client securely transmit the shares of model weights and inference data to S0 and S1, respectively ( 1⃝ and 2⃝). Subsequently, the computing servers utilize these shares as input and complete PPI by executing the SMPC protocols through interactive computation with the assistance of T , yielding the shares of the inference results. ( 3⃝). These shares are then relayed to the client ( 4⃝), facilitating the local reconstruction of the inference result ( 5⃝). Since each participant has only one share of the inputs, outputs, or intermediate results, this PPI workflow can guarantee the privacy of model weights and inference data.

heading: 2.2 Main Bottlenecks of SMPC-based Transformer Model Inference
text: Although the above PPI workflow guarantees the privacy of model weights and inference data, it faces unacceptable communication overhead (Table 1) in implementing some of the nonlinear operations (i.e., Softmax, GeLU, and LayerNorm), which are abundantly present in Transformer models and become a main bottleneck in PPI.
Specifically, for a vector x = (x1, x2, . . . , xn), Softmax in Transformer converts it to an ndimensional probability distribution with
Softmax(x)[i] = exi−τ∑n
h=1 e xh−τ , (1)
where τ = maxnh=1 xh is used to ensure stable numerical computations. As indicated in Table 1, there are three obstacles to the SMPC of Softmax: exponential, division, and maximum. Note that the
calculation of maximum needs to call ΠLT operation logn times (Knott et al., 2021) and becomes the biggest obstacle.
The function of GeLU is defined as
GeLU(x) = x
2
( 1 + erf( x√ 2 ) ) , (2)
where erf(x) = 2√ π ∫ x 0 e
−t2dt. The GeLU function’s nonlinear component is derived from the erf and there is currently no SMPC protocol for its privacy-preserving computation.
Given a vector x = (x1, x2, . . . , xn), the LayerNorm function is defined as
LayerNorm(x) = γ · x− x̄√ var(x) + ϵ + β, (3)
where x̄ = ∑n h=1 xh/n, var(x) = ∑n
h=1(xh − x̄)2, γ and β are two learnable parameters, and ϵ is a very small decimal used to prevent the denominator from being zero. For SMPC, the main bottleneck in computing LayerNorm comes from the division and square root operations.

heading: 2.3 Efficient PPI for Transformer Models
text: To alleviate the aforementioned bottlenecks, existing works on PPI for Transformer models improve the efficiency through either model design or SMPC protocol design. The studies based on model design (Chen et al., 2022; Li et al., 2023a; Zeng et al., 2022; Zhang et al., 2023b; Liang et al., 2023) bypass the high overhead operations in PPI by replacing the SMPC-unfriendly nonlinear operations in Transformer. These schemes substantially increase efficiency but usually lead to a significant degradation in model performance. The studies that design more efficient SMPC protocols (Hao et al., 2022; Zheng et al., 2023a; Gupta et al., 2023; Dong et al., 2023; Hou et al., 2023; Ding et al.,
2023; Pang et al., 2023) improve the efficiency of PPI by customizing efficient SMPC protocols for the nonlinear operators in the Transformer. These schemes preserve the Transformer model’s performance but still face expensive computational and communication overheads.
As a representative work based on model design, Li et al. (2023a) improves the efficiency of PPI by replacing GeLU and Softmax with Quad = 0.125x2 + 0.25x+ 0.5 and
2Quad(x)[i] = (xi + c)
2
∑n h=1(xh + c) 2 , (4)
respectively, such that the privacy-preserving computation of erf, exponential, and maximum is bypassed. Following this, knowledge distillation is employed, with the fine-tuned Transformer model acting as the teacher and the approximate Transformer model as the student. Distillation is carried out on downstream task data, yielding a Transformer model compatible with SMPC. This approach is effective in improving the efficiency of PPI, however, it leads to a significant decrease in model performance. Our investigation reveals that preserving accurate computation of GeLU significantly improves PPI performance. Dong et al. (2023) achieves the first SMPC protocol for GeLU functions by utilizing segmented functions and polynomial fitting. However, the computation of segmented functions and polynomials requires multiple calls of ΠLT and ΠMul, making it inefficient.

heading: 3 SecFormer Framework
text: As discussed above, existing efficient PPI studies suffer from either performance degradation or high inference overhead. To resolve this issue, the SecFormer framework is proposed in this section. We begin with an overview of SecFormer in Section 3.1 and introduce the new efficient SMPC protocols for GeLU, LayerNorm, and Softmax in Section 3.2.

heading: 3.1 Overview
text: SecFormer enhances the efficiency of PPI for Transformer models, addressing both model and SMPC protocol design. The overall depiction of SecFormer is presented in Fig. 3.
In the model design phase, SecFormer implements new strategies to bypass the nonlinear operations with the high overhead in PPI, such as exponential and maximum in Softmax, while preserving model performance. Specifically, SecFormer
replaces Softmax with 2Quad while retaining the accurate computation of the GeLU. Inspired by (Li et al., 2023a), SecFormer further improves the performance of PPI inference by incorporating knowledge distillation techniques.
In the SMPC protocol design phase, SecFormer introduces a suite of efficient SMPC protocols by utilizing appropriate numerical computation methods. Specifically, SecFormer introduces a novel SMPC protocol for GeLU based on segmented polynomials and Fourier series, which facilitates the efficient and accurate computation of GeLU. Subsequently, SecFormer deploys streamlined privacy-preserving calculation for square-root inverse and division using the Goldschmidt method (Goldschmidt, 1964; Markstein, 2004), coupled with input deflation techniques to eliminate the need for nonlinear initial-value computation.

heading: 3.2 SMPC Protocols of SecFormer
text: We next present new efficient SMPC protocols of GeLU, LayerNorm, and the approximated Softmax designed in SecFormer. These algorithms’ security proofs and communication complexity analysis are presented in Appendix D.
Protocol for GeLU. To address the efficiency challenges of GeLU private computations (Section 2.2), some studies replaced GeLU in (2) with its SMPC-friendly alternatives such as ReLU (Zeng et al., 2022) or quadratics (Li et al., 2023a). Although this approach can enhance PPI efficiency, it may result in irreversible performance losses. Dong et al. (2023) introduces the first SMPC protocol for GeLU using segmented functions and polynomial fitting whose computation, however, entails multiple calls of ΠLT and ΠMul, rendering it inefficient.
To solve the above problems, we design an efficient SMPC protocol ΠGeLU based on segmented polynomials and Fourier series. As shown in Fig. 4, the erf function is an odd function symmetric about the origin with limx→∞ erf(x) = 1 and limx→−∞ erf(x) = −1. Therefore, we can convert it to the following segmented function
erf(x) ≈    − 1, x < −1.7 f(x), −1.7 ≤ x ≤ 1.7 1, x > 1.7 , (5)
where f(x) can be approximated through a Fourier series composed of sine functions with a period5 of
5The results of the sine function fitting for different periods are shown in Appendix F.
20. Although a greater number of terms enhances the accuracy of the fitting outcomes, it concurrently leads to increased communication overhead. Here, we employ the following 7-term Fourier series:
f(x) = β ⊙ sin(k ⊙ πx/10), (6)
where k = (1, 2, 3, 4, 5, 6, 7), β is the Fourier series coefficients and ⊙ denotes the element-wise multiplication. For i = 1, 2, . . . , 7,
βi = 1
10
∫ 10
−10 erf(x) sin(
kiπx
10 )dx . (7)
According to Eq. (7), we can compute the coefficients β = (1.25772, −0.0299154, 0.382155, −0.0519123, 0.196033,−0.0624557, 0.118029).
Based on (5), the computation of the erf function is converted into comparison and sine function. The precise calculation of GeLU can be accomplished by combining ΠMul with the erf function. The specific steps of the SMPC protocol for GeLU are shown in Algorithm 1. Specifically, in steps 1-5 of Algorithm 1, we determine in which interval of the segmented function the input x falls by calling the ΠLT . Then, in step 7, the privacy-preserving computation of f(x) is achieved by utilizing the Πsin presented in (Zheng et al., 2023b). The algorithm requires only 1 round of communication,
Algorithm 1: SMPC Protocol for ΠGeLU Input: For j ∈ {0, 1}, Sj holds the shares [x]j . Output: For j ∈ {0, 1}, Sj returns the shares [y]j
with y = GeLU(x). /* Determine the input interval */ 1 [[c0]] = ΠLT ([[x]],−1.7) // (x < −1.7) 2 [[c1]] = ΠLT ([[x]], 1.7) // (x < 1.7) 3 [[z0]] = [[c0]] // (x < −1.7) 4 [[z1]] = [[c1]]− [[z0]] // (−1.7 ≤ x ≤ 1.7) 5 [[z2]] = 1− [[c1]] // (x > 1.7) /* Compute f(
x√ 2 ) */
6 [[x̂]] = 1√ 2 [[x]] 7 [[f(x̂)]] = β ⊙Πsin(k ⊙ π[[x]]/10) /* Compute erf(x̂)) */ 8 [[erf(x̂)]] = [[z0]] + ΠMul([[z1]], [[f(x̂)]]) + [[z2]] /* Compute GeLU(x) */
9 [[y′]] = 1 + [[erf(x̂))]] 10 [[y]] = ΠMul([[ x 2 ]], [[y′]])
and the specific steps of it is in Appendix E.2. In steps 8-10, we compute the erf function and execute the GeLU calculation by invoking ΠMul.
Protocol for LayerNorm. Previous work (Knott et al., 2021) implements the privacy-preserving computation of LayerNorm in (3) by sequentially invoking ΠrSqrt and ΠDiv, resulting in expensive computational and communication overheads. Goldschmidt’s method enables the direct conversion of square root inverses (i.e., 1√. ) directly into multiple iterations of multiplications. However, achieving a broader convergence range often requires complex nonlinear initial value calculations, such as Look-up-table (LUT) (Rathee et al., 2021) or exponentiation (Knott et al., 2021), before the iteration begins. To resolve this issue, we propose to employ the deflation technique for bypassing these intricate nonlinear initial value calculations that are incompatible with SMPC. The detailed steps of the
SMPC protocol for LayerNorm are in Algorithm 2. Specifically, in steps 3-8, we use Goldschmid’s method to compute 1√q where q = (var(x)+ ϵ)/η. Through division by a constant η (A hyperparameter whose value is shown in Appendix G.), we initially deflate the denominator into the interval [0.001, 2.99] which ensures fast convergence for linear initial values. Then, we set the initial values p0 = 1, q0 = q, and compute mi = (3−qi−1)/2, pi = pi−1mi, qi = qi−1m2i at each iteration by calling ΠMul and ΠSquare. After t = 11 iteration, the value of 1√q is calculated.
Algorithm 2: SMPC Protocol for LayerNorm ΠLayerNorm
Input: For j ∈ {0, 1}, Sj holds the shares [x]j . Output: For j ∈ {0, 1}, Sj holds the shares [y]j with y = LayerNorm(x). /* Compute the mean and variance */
1 [[x̄]] = 1 n ·∑nh=1[[xh]]
2 [[var(x)]] = ∑n
h=1 ΠSquare([[xh]]− [[x̄]]) /* Goldschmidt’s method */
3 p0 = 1, q0 = 1 η ([[var(x)]] + ϵ) 4 for i← 1 to t do 5 [[mi]]← (3− qi−1)/2 6 [[qi]]← ΠMul([[qi−1]],ΠSquare([[mi]])) 7 [[pi]]← ΠMul([[pi−1]], [[mi]]) 8 end 9 /* Compute LayerNorm(x) */
10 [[y]] = γ · ( 1 η ([[x]]− [[x̄]]) · [[pt]]) + β
Protocol for Approximated Softmax. As mentioned in Section 3.1, we follow Li et al. (2023a) and bypass the privacy-preserving computations of exponential and maximum by substituting Softmax with 2Quad in (4). However, to preserve the normalized nature of Softmax, the division operations cannot be avoided and thus become a new obstacle.
To solve this problem, we again use the Goldschmidt’s method to convert the division operation to multiplications. Similar to the LayerNorm protocol, the complex calculation of initial values is avoided by effective deflation. The implementation of the SMPC protocol for the approximated Softmax (i.e., Π2Quad) is shown in Appendix B.

heading: 4 Experiments
text: This section showcases the effectiveness of SecFormer through extensive experiments. We begin with the experiment setup in Section 4.1 and then report the performance assessment results in Section 4.2 and efficiency evaluations in Section 4.3, respectively. We further provide more analysis for
SecFormer in Section 4.4, including an efficiency evaluation for ΠGeLU , ΠLayerNorm and Π2Quad.

heading: 4.1 Experimental Setup
text: Implementation. We implemented SecFormer using CrypTen6, a semi-honest privacy-preserving machine learning framework based on secret sharing. To simulate the experimental environment, we utilized three Tesla V100 servers with a 10GB/s bandwidth. The hyperparameters for model finetuning and distillation follow the settings in (Li et al., 2023a), see Appendix G for details.
Baselines. We compare SecFormer with state-ofthe-art works based on model design (MPCFormer (Li et al., 2023a)) and SMPC protocol design (PUMA (Dong et al., 2023)). Specifically, MPCFormer improves the efficiency of PPI by substituting Softmax and GeLU with some SMPC friendly quadratics. PUMA enhances PPI efficiency by designing more efficient SMPC protocols for GeLU, LayerNorm and Softmax. Following the setting in MPCFormer, we include the result of the finetuned redesigned model as the baseline, denoted as MPCFormerw/o and SecFormerw/o (i.e., fine-tuned without distillation). We also re-implement PUMA on CrypTen for consistency.
Models and Datasets. We followed MPCFormer using a representative transformer model BERT, see Appendix G for details. For the reliability of the experimental results, we use datasets with different evaluation metrics and sizes, including RTE, MRPC, CoLA, STS-B, and QNLI. In terms of evaluation metrics, MRPC uses F1 scores, STS-B employs the average of Person and Spearman correlations, CoLA uses Matthews correlations, and RTE and QNLI rely on accuracy.

heading: 4.2 Performance Comparison
text: We validate the performance of SecFormer and the main results are shown in Table 2. For the model design framework MPCFormer, SecFormer exhibits a significant performance improvement. Specifically, for BERTBASE, SecFormer outperforms MPCFormer across all tasks, resulting in a 3.4% average improvement. For BERTLARGE, MPCFormer faces significant performance degradation, including CoLA task failure. In contrast, even without data distillation, SecFormer outperforms
6https://github.com/facebookresearch/CrypTen
MPCFormer. After distillation, SecFormer demonstrates a substantial 24.7% performance improvement compared to MPCFormer. This is mainly because SecFormer implements an accurate computation of GeLU instead of replacing it aggressively with a quadratic polynomial.
For the protocol design framework PUMA, SecFormer incurs only a marginal 0.9% and 1.3% performance degradation. PUMA does not perform any model design and achieves PPI without performance loss. However, this results in PUMA facing unacceptable communication overheads, as detailed in Section 4.3.

heading: 4.3 Efficiency Comparison
text: We evaluate the efficiency by testing the time and communication volume required to perform single-sample inference across different frameworks. The main results are shown in Table 3. We can find that SecFormer is significantly more efficient than PUMA. Specifically, for BERTBASE and BERTLARGE, SecFormer performs 3.57 and 3.58 faster than PUMA on the total inference time. These advantages stem from that SecFormer utilizes model design to achieve efficient computation of Softmax, and design efficient SMPC protocols suitable for the Transformer models for other nonlinear operators (i.e., GeLU, LayerNorm) by using appropriate numerical computation techniques. The efficiency of each SMPC protocol is shown in Table 3 and will be discussed later in Section 4.4.
When considering the framework of model de-
sign, SecFormer is only 1.05 and 1.04 times slower than MPCFormer in the scenarios of BERTBASE and BERTLARGE, respectively. This result is based on the fact that SecFormer spends 41% of its time performing privacy-preserving calculations for GeLU, while MPCFormer spends only 0.01% of its time to implement the privacy-preserving calculations for Quad. However, the conclusions in Section 4.2 suggest that replacing GeLU with quadratic leads to dramatic degradation of model performance or even failure on some tasks (i.e., performance with 0 in Table 2).
In conclusion, experiments with SecFormer regarding performance and efficiency reveal its dual advantages, combining strengths from both protocol design and model design frameworks.

heading: 4.4 SMPC Protocols Evaluation
text: We compare ΠGeLU with PUMA in terms of time and communication overhead. The comparison results in Fig. 5 show that ΠGeLU is about 1.6 times
lower than PUMA in time and communication overhead. This is mainly due to the fact that it invokes fewer ΠLT relative to PUMA. In terms of accuracy, both ΠGeLU and PUMA meet the needs of PPI, while CrypTen can only maintain accuracy over a small range. See Appendix C for details.
We compare ΠLayerNorm with CrypTen (Knott et al., 2021) in terms of time and communication overhead. Fig. 6 shows that ΠLayerNorm is up to 4.5 times faster than CrypTen (Knott et al., 2021). This is due to the efficient privacypreserving square root inverse calculation proposed by SecFormer. As shown in Fig. 7, it is 4.2 times faster than CrypTen and reduces the communication volume by a factor of 2.5.
We compare Π2Quad with MPCFormer and
PUMA in terms of time and communication overhead. Fig. 8 shows that Π2Quad is 1.26 ∼ 2.09 times faster than MPCFormer and the communication overhead is reduced by 1.04 ∼ 1.12 times. These enhancements come from the efficient privacy-preserving division calculation proposed by SecFormer. As shown in Fig. 9, it is 3.2 times faster than CrypTen, and the communication overhead is reduced by 1.6 times.
Compared to PUMA, which achieves precise privacy-preserving Softmax, Π2Quad gets a drastic improvement in efficiency, i.e., 8.24 ∼ 16.8 times faster and 30.53 ∼ 36.2 times less communication. This is due to the fact that the model design performed by SecFormer avoids the calculation of exponential and maximum.

heading: 5 Conclusion
text: We present SecFormer, a synergistic PPI framework that strategically combines the strengths of both SMPC protocol design and Transformer model design. Extensive experiments reveal that SecFormer surpasses existing PPI methods, achieving fast and accurate PPI for Transformer models. It not only matches the performance of approaches that focus exclusively on SMPC protocols but also competes with the efficiency of methods dedicated solely to model design. SecFormer holds significant potential for enhancing large language models, offering an effective solution that promises to maintain high performance while ensuring stringent privacy and efficiency standards in increasingly complex and expansive linguistic landscapes.

heading: 6 Limitations
text: We summarize the limitations of SecFormer as follows: (1) SecFormer focuses on implementing PPI for the encoder-only Transformer model, such as BERT, without extending to other Transformer model families like the GPT series. We concentrate on the encoder-only Transformer model because of its continued prominence in real-world natural language understanding tasks, particularly within resource-constrained environments like edge computing. Prior efforts to implement the encoderonly Transformer model for PPI have encountered obstacles, including slow inference speeds and substantial performance degradation. Our work addresses these challenges and offers insights to guide future optimization efforts concerning PPI across diverse Transformer model families. The proposed protocols can be applied to implement PPI of other transformer-based models straightforwardly and we will consider PPI for decoder only Transformer models like GPT in the future. (2) Regarding SMPC protocols, SecFormer executes only on CrypTen and does not invoke the cutting-edge underlying SMPC protocols. We will try to exploit other privacy-preserving frameworks with more advanced SMPC protocols to further improve the inference efficiency of SecFomer in future work. (3) SecFormer only performs model design by replacing Softmax with 2Quad and does not incorporate other model lighting techniques. Other model lightweight techniques such as model quantization and pruning are compatible with the proposed SMPC protocols and can be combined into SecFormer to further improve the PPI effi-
ciency in the future.

heading: 7 Acknowledgements
text: This research is partially supported by the National Key R&D Program of China (No.2021ZD0112905), the National Natural Science Foundation of China (No. 62206139, 62106114), and the Major Key Project of PCL (PCL2023A09).

heading: B Protocol for the Approximated Privacy-Preserving Softmax
text: In this section, we give the specific implementation of the SMPC Protocol for the approximated Softmax (i.e., 2Quad) as mentioned in Section 3.1. In steps 3-8 of Algorithm 3, we first deflate the denominator q = ∑n h=1(x + c)
2 into the interval [0.001, 1.999], which ensures fast convergence for linear initial values, through division by a constant η. Subsequently, we set the initial values q0 = q, p0 = (x + c)2, and compute mi = 2− qi−1, pi = pi−1mi, qi = qi−1mi at each iteration by calling ΠMul. After t = 13 iterations, p q is computed.

heading: C Accuracy Comparison of Privacy-Preserving GeLU Algorithms
text: In this section we compare the performance of privacy-preserving GeLU with Puma and CrypTen.
Algorithm 3: SMPC Protocol for Softmax Π2Quad
Input: For j ∈ {0, 1}, Sj holds the shares [x]j . Output: For j ∈ {0, 1}, Sj holds the shares [y]j , where y = 2quad(x). /* Compute the numerator */
1 [[p]] = ΠSquare([[x+ c]]) /* Compute the denominator */
2 [[q]] = ∑n
h=1[[p[h]]] /* Goldschmidt’s method */
3 q0 = 1 η [[q]], [p0]] = 1 η [[p]] 4 for i← 1 to t do 5 [[mi]]← 2− [[qi−1]] 6 [[pi]]← ΠMul([[pi−1]], [[mi]]) 7 [[qi]]← ΠMul([[qi−1]], [[mi]]) 8 end 9 [[y]] = [[pt]]
The specific comparison results are shown in Table 4. Both SecFormer and Puma achieve privacypreserving computation within the entire interval of the GeLU function by using segmented polynomials. CrypTen, on the other hand, locally fits the erf function using a low-order Taylor expansion and thus can only achieve privacy-preserving computation of the GeLU function in a smaller interval.

heading: D Security Proof and Communication Complexity Analysis
text: D.1 Security Proof
SecFormer adheres to a semi-honest (also known as honest-but-curious) assumption similar to the works of Li et al. (2023a) and Dong et al. (2023), where honest participants constitute the majority. Under this assumption, the security of SecFormer can be formally proved within the simulation paradigm, specifically against static semihonest adversaries denoted as A, which can potentially corrupt one of the servers. The simulation paradigm delineates two distinct worlds: the real world and the ideal world. In the real world, the servers execute the protocols in the presence of semi-honest adversaries A. In contrast, the ideal world involves the servers transmitting inputs to a trusted dealer capable of correctly executing the protocol. The security of SecFormer necessitates that, for any semi-honest adversary A, the distribution of the real world remains indistinguishable from that of the ideal world. The definition of privacy-preserving inference protocols (Mishra et al., 2020; Huang et al., 2022; Hao et al., 2022) is as follows:
Definition 1 A protocol ΠP between the servers
who have the shares of the model weights and the inference data is a privacy-preserving protocol if it complies with the following criteria: (1) Correctness: For a model M with weights w and input samples x, the client’s output at the end of the protocol is the correct inference M (w, x); and (2) Security: For a computational server Sj , j ∈ {0, 1} that is corrupted by adversary A, there exists a probabilistic polynomial time simulator SimSj such that adversary A cannot distinguish V iewΠPSj (i.e., the view of Sj during the implementation of ΠP ) from SimSj . Similarly, for a corrupted server T , there exists an efficient simulator SimT such that V iewΠPT is indistinguishable from SimT .
SecFormer is constructed from the sub-protocols outlined in the works of Knott et al. (2021) and Zheng et al. (2023b). Leveraging the concept of universally composable security established by Canetti (2001), we can prove that SecFormer satisfies Definition 1 directly.
D.2 Communication Complexity Analysis The execution of ΠGeLU invokes two ΠLT , one ΠSin and one ΠMul. Thus the execution of the privacy-preserving GeLU algorithm takes a total of 2 logL+4 rounds of online communication and transmit 7210 bits.
The implementation of privacy-preserving LayerNorm requires calls to ΠMul, ΠSquare and privacy-preserving inverse of the square root. The inverse of the square root requires one call to ΠSquare and two calls to ΠMul in parallel per iteration, costing 2 rounds of communication and transferring 640 bits. Thus performing the square root inverse takes a total of 22 rounds of communication and transfers 7040 bits and the implementation of privacy-preserving LayerNorm takes a total of 24 rounds of communication and transfers 7424 bits.
The implementation of approximate privacypreserving Softmax requires calls to ΠMul and ΠDiv. The ΠDiv requires two call to ΠMul in parallel per iteration, costing 1 rounds of communication and transferring 512 bits. Thus performing the ΠDiv takes a total of 13 rounds of communication and transfers 6,656 bits and the implementation of
approximate privacy-preserving Softmax takes a total of 23 rounds of communication and transfers 6,784 bits.

heading: E Underlying SMPC Protocols
text: In this section, we provide a brief overview of the underlying protocols used and refer to the works of Knott et al. (2021) and Zheng et al. (2023b) for details. Let Sj with j ∈ {0, 1} be two parties that are used to execute the SMPC protocol. Each party Sj will be given one additive share ([u]j , [v]j) ∈ ZL of the operation inputs u and v for j ∈ {0, 1}.
E.1 Privacy-Preserving Linear Protocols
Privacy-preserving addition is implemented with [u+ v]j = [u]j + [v]j for j ∈ {0, 1}.
Privacy-preserving multiplication is implemented with Beaver-triples: (a, b, c) where a, b ∈ ZL are randomly sampled from ZL and c = a · b mod L. Specifically, for each j ∈ {0, 1}, Sj first calculates [d]j = [u]j − [a]j and [e]j = [v]j − [b]j . Then, they send the [d]j and [e]j to each other and reconstruct d = Rec([d]0, [d]1) and e = Rec([e]0, [e]1). Finally, the additive share of u · v can be computed using [u · v]j = −jd · e+ [u]j · e+ d · [v]j + [c]j . To complete the SS-based multiplication, both parties need to spend 1 round of two-way communication and transmit 256 bits.
E.2 Privacy-Preserving Non-Linear Protocols
Privacy-preserving comparison is implemented by the conversion between the additive shares and the binary shares. Specially, [[z]] = [[x− y]] is converted to the binary shares ⟨⟨z⟩⟩ through additive circuit with logL round of communication. Subsequently, the binary shares of z’s sign bit can be determined by ⟨⟨b⟩⟩ = ⟨⟨z⟩⟩ >> (l − 1)7. Finally, the additive shares of x < y can be derived by converting ⟨⟨b⟩⟩ to [[b]] with one round of communication. Thus, the implementation of privacy-preserving compare algorithm cost logL+1 round of communication and transmit 3456 bits.
7>> l denote shift l bit to the right.
Privacy-preserving maximum of the n-element vector x is implemented by calling logn privacypreserving comparisons using the tree reduction algorithm (Knott et al., 2021).
Privacy-preserving exponential is implemented using the repeated-squaring method
ex = limx→∞ ( 1 + x 2n )2n , (9)
which converts exponential calculations into addition and square operations. The number of iterations n is set to 8 in (Knott et al., 2021) by default.
Privacy-preserving reciprocal is implemented by Newton-Raphson method, which converts reciprocal calculations into addition and multiplication operations. The iterative formula is
yn+1 = yn(2− xyn). (10)
The initial value of the iteration is
y0 = 3e 1 2 −x + 0.003. (11)
The number of iterations is set to 10 in (Knott et al., 2021) by default.
Privacy-preserving square root is implemented by Newton-Raphson method, which converts exponential calculations into addition and multiplication operations. The iterative formula is
yn+1 = 1
2 yn(3− xy2n). (12)
The initial value of the iteration is
y0 = e −2.2(x 2 +0.2) + 0.198046875. (13)
The number of iterations is set to 3 in (Knott et al., 2021) by default.
Privacy-preserving sine is implemented on trigonometric identities. Specifically, sin(x) = sin(δ) cos(t) + cos(δ) sin(t), where δ = x − t. With the assistance of the server T , the random numbers t, sin(t), cos(t) are generated in the offline phase, and the share of sin(x) is computed in the online phase with only one round of communication and transmits 42 bit.8 See Algorithm 4 for an implementation of the privacy-preserving sine.
8CrypTen uses 16-bit computational precision.
Algorithm 4: Privacy-preserving sine Input: For j ∈ {0, 1}, Sj holds the shares [x]j ;
Same Pseudo-Random Function (PRF) and key kj .
Output: For j ∈ {0, 1}, Sj returns the shares [y]j , where y = sin(x).
/* Offline Phase */ 1 S0, T : [t]0, [u]0, [v]0 ← PRF (k0) 2 S1, T : [t]1 ← PRF (k1) 3 T : t = [t]0 + [t]1, [u]1 = sin(t)− [u]0, [v]1 =
cos(t)− [v]0 /* Online Phase */
4 [δ]j = ([x]j − [t]j) mod 20 5 δ = [δ]0 + [δ]1 // reconstruct δ by 1 round of communication 6 p = sin(δ), q = cos(δ) 7 [y]j = p[v]j + q[u]j

heading: F Fourier Series Fitting Results
text: In this section, we give the results of fitting erf(x) using Fourier series composed of different periodic sin functions. Specifically, we fit erf(x) using the 7-th order Fourier series composed of sin functions with periods of 10, 20, 30, and 40, respectively, and the specific fitting results are shown in Fig. 10.

heading: G Models and Hyper-parameter
text: Models. In this section, we provide a concise overview of the architecture of the experimental models. For more detailed information, we refer the readers to the HuggingFace Transformers library.
• BERTBASE: BERTBASE represents the foundational version of the Bert model, comprising 12 Transformer encoder layers, a hidden size
of 768, and 12 heads. With 110 million parameters, it undergoes training on a substantial corpus of unlabeled text data.
• BERTLARGE: BERTLARGE serves as an expanded iteration of BERTBASE, featuring 24 Transformer encoder layers, a hidden size of 1024, and 16 heads. Boasting approximately 340 million parameters, this version exhibits increased potency, enabling it to capture intricate language patterns.
Hyper-parameter. For LayerNorm and Softmax, we set the constants η as 2000 and 5000, respectively, to ensure that the value of the denominator can be deflated to a reasonable range of convergence. We follow the choice of hyperparameters for fine-tuning and distillation in MPCFormer (Li et al., 2023a). Specifically, in the fine-tuning phase, we use a learning rate of [1e−6, 5e−6, 1e−5, 1e−4], a batch size of [64, 256], and epochs of [10, 30, 100]. We fine-tuned each model with a combination of hyperparameters and selected the best performing model as teacher. In the distillation phase, we decide the number of epochs based on the MSE loss of the embedding and transformation layer distillations. For small datasets (CoLA, MRPC, RTE), the batch size is 8; for large datasets (QNLI, STS-B), the batch size is 32. Specifically, in the embedding and transform layer distillation phases, 10 epochs for QNLI, 20 epochs for MRPC, 50 epochs for STS-B, 50 epochs for CoLA, and 50 epochs for RTE.

heading: Future Work
text: Our work addresses these challenges and offers insights to guide future optimization efforts concerning PPI across diverse Transformer model families. The proposed protocols can be applied to implement PPI of other transformer-based models straightforwardly and we will consider PPI for decoder only Transformer models like GPT in the future. (2) Regarding SMPC protocols, SecFormer executes only on CrypTen and does not invoke the cutting-edge underlying SMPC protocols. We will try to exploit other privacy-preserving frameworks with more advanced SMPC protocols to further improve the inference efficiency of SecFomer in future work. (3) SecFormer only performs model design by replacing Softmax with 2Quad and does not incorporate other model lighting techniques. Other model lightweight techniques such as model quantization and pruning are compatible with the proposed SMPC protocols and can be combined into SecFormer to further improve the PPI effi-
ciency in the future