MPCFORMER: FAST, PERFORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC
Authors: Dacheng Li
Hongyi Wang
Rulin Shao
Han Guo
Eric Xing
Hao Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Secure Multiparty Computation
Privacy
Machine Learning
Transformer model

Abstract: Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60$\times$ or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model. On the IMDb dataset, it achieves similar performance to $\text{BERT}_\text{BASE}$, while being 5.3$\times$ faster. On the GLUE benchmark, it achieves 97% performance of $\text{BERT}_\text{BASE}$ with a 2.2$\times$ speedup. MPCFORMER remains effective with different trained Transformer weights such as $\text{ROBERTA}_\text{BASE}$ and larger models including $\text{BERT}_\text{LARGE}$. Code is available at https://github.com/MccRee177/MPCFormer.

Published as a conference paper at ICLR 2023
MPCFORMER: FAST, PERFORMANT AND PRIVATE TRANSFORMER INFERENCE WITH MPC
Dacheng Li Rulin Shao" Hongyi Wang",, Han Guo,, Eric Xing Hao Zhang c Carnegie Mellon University in Mohamed bin Zayed University of Artificial Intelligence
P Petuum Inc. University of California, Berkeley
ABSTRACT
Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60x or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar MI. performance to the input model. On the IMDb dataset, it achieves similar performance to BERT BASE. while being 5.3 faster. On the GLUE bench- mark, it achieves 97% performance of BERT with 2.2x speedup. MPC- FORMER remains effective with different trained Transformer weights such as ROBERTABASE and larger models including BERTLage Code is available www.:wwwwwwcccceeeeeeeeeecccee wwtp////eeeee///PP///eee... /MMP/PPPCPrr
INTRODUCTION
Pre-trained Transformer models can be easily fine-tuned on various downstream tasks with high performance and have been widely developed as model inference services (Bommasani et al.) 2021 Feng et al. 2020 Yang et al. 2019b Clark et al., 2020). However, these model inference services can pose privacy concerns. For instance, GitHub Copilot, a code-generating engine adapted from pre-trained GPT weights, requires either users reveal their code prompts the service provider, or the service provider to release the Copilot's trained weights, which are business proprietary, to
users (Chen al. 2021 Brown et al. 2020).
Secure Multi-Party Computation (MPC) offers a promising solution by keeping data and model weights private during inference (Evans al. 2018). However, the vanilla Transformer inference in MPC is unacceptably slow. For instance, BERT BASE inference takes <1 second without MPC, but 60 seconds with MPC (Figure 2). An intuitive way to accelerate MPC inference replaces compu- tational operations with their faster approximations and retrains the approximated model, which has been adopted on convolutional neural networks (CNNs) (Chou et al. 2018). Unfortunately. adapting this solution to Transformers drastically decreases the model's, 's performance ($ 5)
In this paper, we take the first step to pursue privacy-preserving Transformer model inference in MPC, while remaining fast and performant. We take inspiration from the approximation approact and attribute the performance degradation to two challenges. First, many MPC-friendly approx- imations toughen model training. For example, quadratic functions cause the gradient explosion problem in deep neural networks (Mishra et al. 2020). Second, downstream datasets used for Transformer fine-tuning usually contain insufficient data retrain an approximated Transformer with common task objectives (Zhang & Sabuncu 2018 Hinton et al. 2012).
To address these two challenges, resort to the knowledge distillation (KD) framework. KD can ease the model training by matching intermediate representations between the teacher and the stu- dent model (Romero et al. 2014); this intermediate supervision can alleviate the gradient explosion
Authors contributed equally. We will the term MPC-friendly approximations.

---

Published as a conference paper at ICLR 2023
Model Inference
MPCFormer
Sroge Pappenn MPC friendy
User
Teacher:
Approsmmatars
MPC Engine. Crypten
Service Provider
Fine-tuned
MPC Engine
Transformer
Secret
Stage2: lodge
Sharing
Pre-trained Transformer Weights, egg. CPT-3
Student
Distilled
Approximated
Approximated Transformer
Private Data
Transformer
Down-stream Datasets
Inference
Figure I: An illustration of our proposed MPCFORMER framework. MPCFORMER takes trained (or finetuned) Transformer model and adopts given MPC-friendly approximations, then uses KD on the downstream datasets to construct high-qualipy models. During inference time, MPCFORMER leverages an MPC engine to attain private model inference. For ease of illustration, we only show the service provider and the involve a trusted third party (TTP) to help with the joint computation.
MPC Systems such CrypTen (Knott al. 2021) may also
2015). At the same time, the KD objective is data-efficient and allows training problem (Lee et an approximated Transformer on small downstream datasets (Touvron et al. 2021).
Our approach and contributions. In this paper, build MPCFORMER, an casy-to-adopt frame- work for privacy-preservin Transformer inference. MPCFORMER takes in MPC-friendly ap- proximation and a trained Transformer. It returns Transformer with low inference latency MPC and high ML performance simultancously. To do so, MPCFORMER first replaces bottleneck func- tions in the input Transformer model with the given MPC-friendly approximations. The resulting approximated Transformer model has faster inference speed in MPC. Next, applies knowledge distillation to train the approximated Transformer with high performance, using teacher guidance from the original input Transformer. Finally, the model provider Transformer on top of an MPC engine, 6.g., CrypTen, for private model inference service. The overall workflow of MPCFORMER shown in Figure T
the distilled approximated
We implement MPCFORMER on an MPC system (Knott et al. 202T), with various MPC-friendly approximations. the process, also design a new and faster MPC-friendly approximation the Softmax function. We extensively evaluate our implementation with various Transformer models. On the IMDb benchmark, MPCFORMER achieves similar ML performance to BERT BASE with 5.3x speedup. It achieves similar ML performance to BERTLARGE with a 5.9x speedup. On the GLUE benchmark, achieves 97% performance BERT BASE with 2.2 speedup. MPCFORMER is also effective when given different trained Transformer models, e.g., RoBERTaRASF.
BACKGROUND
In this section, we first describe the Transformer model. Then we describe how functions in Trans- former models can be implemented in MPC, and analyze performance bottlenecks.
2.1 TRANSFORMER MODELS
An n-layer Transformer model consists three components: (1) The embedding layer. (2) A stack of Transformer layers. (3) The prediction layer. The embedding layer maps token (c.g.
word or an image patch) to a hidden representation (Devlin et al., 2018 Dosovitskiy et al.) 2020). One Transformer layer consists of attention module and several matrix multiplication modules
Bahdanau et al.) 2014). The prediction layer maps the output of the last Transformer layer to a task output (e.g., s. a probability distribution for classification task). A partial illustration of a Transformer model can be found in Figure
2.2 TRANSFORMER MODELS IN MPC
The Transformer model inference process can be formulated as 2-Parties Computation (2PC). In 2PC, the party inputs the data, and the model provider party inputs the Transformer model.
2

---

Published as a conference paper at ICLR 2023
They jointly compute inference result. Throughout the entire inference process, 2PC guarantees both parties only know information about their own inputs and the result (Yang et al. 2019a).
We describe the secret sharing scheme mean to preserve privacy during the inference pro- cess (Damgard et al. 2012 Goldreich et al. 2019). Assuming that the user provides a number æ its input, the secret sharing scheme splits into two numbers, 21 and 22- It then lets the user party hold 2 and distributes 22 to the model provider party. There are two properties of Dj and Z2- First, either 21 or 22 alone contains information about æ. This property allows the user hide the actual value x from the model provider. Second, together they reconstruct x. For instance, and z2 add up to x: 21 + ø2. The second property allows joint computation.
Table 1: The communication statistics explain the left figure. For instance, 50.3GB communica- tion in Softmax functions takes 34.1 seconds. In particular, the run-time in MPC is dominated by the communication as opposed to computation. Overall, the communication takes 46.4 seconds, which is 79% % of the whole inference process.
Softmax
67.8%
0.8%
Other
12.7%
18.6% MatMul
Routines Addition
Comm. (rounds)
GeLU
Multiplication
Comparison
Figure 2: BERTBASE model (12-layers, 512 tokens) run-time breakdown eval- uated on an MPC system. The overall runtime with MPC is 59.0 seconds, but < second without MPC.
functions
Comm. (volume)
Time (s)
MatMul
3.5GB
2.5
GeLU
14.8GB
9.6
Softmax
50.3GB
34.1
We take multiplication via Beaver triple as a joint computation example (Beaver 1991). In multi- plication, the user party provides x and the model provider provides y. and they secret share x and Thus, the user gets and y1; the model provider gets 22, and y/- Beaver triple assumes a triple ab has been generated" The triple also secret shared so that the user party gets c, a,, "q.a,bb, and the model provider gets 02, (22 bz. The user first computes éJ
01, 61 yi bi locally. The model provider similarly computes é2 I2 02 82 = 1/2 b2 locally. They communicate these four numbers and reconstruct compute 71 = 0j + ebi + 801 + de. The model provider computes 122 = C2 + cbअ + 802. At this point, the multiplication result xy can be reconstructed by xy m1 72-
é1 E2, 81 82. the user then use these two values to
There are two important observations in the multiplication example: (1) it does not leak information to the other party. For instance, the user does not send 21 to the model party. Instead, it sends It 0j where aj is a random mask; (2) it requires one extra round of communication compared the multiplication without MPC. This partially explains why vanilla Transformer models slow in MPC. In particular, functions in Transformers (e.g., nonlinear activation) can be mainly implemented by three routine i.e.. addition, multiplication, and comparison. Any computational operations composed by these routines would result in extra complexity in MPQ
Empirically, we show these complexities by running BERT (Figure 2) and reporting commu- nication statistics in Table [ with a secret-sharing-based MPC system (Knott et al.) 2021). We observe that GeLU functions and Softmax functions in Transformer layers are the major sources of bottlenecks, which echoes findings in concurrent study (Wagg et al.) 2022). GeLU(z) = t erf [
is slow because the Gaussian Error function erf(-) is evaluated by a high exp(x.)
order Taylor expansion, which requires many multiplication routines. Softmax
Σ, exp(x,)
slow because (1) The exponential function is evaluated by several iterations of squaring, which requires many multiplication routines; (2) the maximum operation over x is required for numerical stability (Paszke al.. 2019). which requires comparison routines.
Forr example, through oblivious transfer (Keller et al. 2016) homomorphic encryption Paillier 1999.
3 We only consider routines that take two secret share numbers for the case of illustration.
We provide details on implementing routines and functions in MPC at A.1
3

---

Published as a conference paper at ICLR 2023
3 RELATED WORK
MPC. Secure Multi-party Computation (MPC) enables joint computation between parties while keeping inputs private. The privacy feature and rich support of systems have made it suitable for Transformer inference (Mohassel Zhang 2017: Liu et al. 22017] Mohassel & Rindal 20188] Riazi et al. 2018: Juvekar et 2018 Wagh et al. 2019 Mishra al., 2020 Knott et al. (021). p.. In this paper, we do not aim to implement a new MPC system. Rather, we aim to develop an algorithmic solution to speed up Transformer inference that can be portable across many MPC systems.
Transformer models. Transformer models have achieved great success in language understanding
Yang et al. (2019b); Lan et al. ). Raffel et al. 12 (Clark et al. (2020), vision understanding
/
Dosovitskiy et al. (2020); et al. (021): Radford et al. (021), ) and beyond (Sharir et al. (021). In
particular, the two-stage training strategy for Transformer models has been shown to be effective in
extensive settings and has become the domincated paradigm (Liu et 2019 Radford et al. 2018
Ture et al. 2019). In this training strategy, Transformer models are first pre-trained on a large dataset for general understanding, and then fine-tuned on a small downstream dataset to learn task-specific features. In this work, we consider this paradigm as the default setting, where we assume that model providers use pre- pre-trained Transformer weights from elsewhere, and only have downstream data.
MPC-friendly approximations. Existing research has developed MPC-friendly approximations to speed up CNN computation in MPC. Chou et al. (2018) develops optimization framework that minimizes the approximation error of order 2 polynomial to ReLU: ReLU(z) 0.125 x x² + 0.25 x + 0.5. This introduces significant accuracy drop because the quadratic activation causes the Gradient Descent (GD) algorithm to diverge. Mishra et al. (2020) alleviates this problem by us- ing a set of carefully designed heuristics along with Neural Architecture Search (NAS). Mohassel & Zhang (2077) proposes approximation to softmax by replacing exponential with ReLU functions. We do not focus on developing heuristics for a single pair of bottleneck functions and approxima- tions. Rather, we focus on developing a general framework that can consistently output performant Transformer model with various approximations.
Knowledge Distillation (KD). KD transfers knowledge from the teacher model to the student model by matching their hidden representations (Hinton et al.] 2006). Several research has designed effective objectives for Transformer models (Sanh et al.] [2019] et al. (2019) Dosovitskiy et al. 2020) such matching the attention matrices. In particular, [anh al..(220 (2019) and et al. (2019) have a different goal than us and train on the pre-training dataset as well. However, we share the same assumption on the model providers' side they only have downstream datasets.
4 METHOD
In this section, we present the MPCFORMER framework. MPCFORMER allows the model provider to convert its Transformer model to a faster and performant one for private inference service. In 4. we introduce the workflow of MPCFORMER (Figure Tp. followed by the details of each step 4.2
4.1 HIGH-LEVEL WORKFLOW
In the inference service, a model provider holds a Transformer model T, and the user holds data X. They reach to an agreement on an MPC system perform private inference. In $2 we illustrate that using T to perform the inference in the MPC system is slow. Instead of using T, the model provider can use MPCFORMER to generate more suited one S. S runs much faster than T in the MPC setting while having similar ML performance compared to T.
To use MPCFORMER, the model provider needs to provide the trained Transformer model T, the downstream dataset D. and MPC-friendly approximations A. These MPC-friendly approximations A need to be fast in MPC and will be used to replace bottleneck functions T. The workflow can be concisely described as:
Convert: S = MPCFORMER(T, AD.A)
(I)
Inference: y (PCs(X)
4

---

Published as a conference paper at ICLR 2023
Accuracy Inference Speed
Accuracy Inference Speed
Accuracy Inference Speed
FFN Quad(=
FFN Quad(aW; + bi)
(aW bi)
Stage 1:
Slage Distillaionn
Approximation
MHA
MHA
MHA QK"
QX" V)
QK? v/d V)
softmaax
2Quad(
QQani
XN
xN
xN
Figure 3: The overview of the MPCFORMER framework. The first stage uses MPC-friende approx- imations and T to construct faster Transformer architecture S'. The second stage uses Knowledge Distillation on S' to learn performant and fast Transformer model S.
4.2 MPCFORMER
MPCFORMER is two-stage framework as shown in Figure The first stage leverages A and T to construct an MPC-friendly Transformer architecture S', which achieves fast inference the MPC system. The second stage applies knowledge distillation (KD) to S' to learn the output model S, which is fast in MPC and preserves the high performance of T.
4.2.1 STAGE 1: APPROXIMATION
In the first stage, MPCFORMER replaces bottleneck functions in T with given A construct MPC-friendly Transformer architecture S' (Figure 3). Below we show how construct A for our experiments ie. using the MPC system Knott et al. (2021).nr In $2, we identify the bottleneck to be GeLU and Softmax functions. We thus construct A for these two functions.
Approximating GeLU, Analysis in shows that multiplication in MPC requires extra com- munication. Thus, quadratics are the fastest nonlinear activation in MPC. Since GeLU and ReLU functions share similar function values, we simply take the quadratics designed for the ReLU func- tion ($3) to approximate the GeLU function: (eLU(() = 0.125x + 0.25x + 0.5. We denote this approximation "Quad".
Approximating Softmax. Prior works in CNNs have developed an MPC-friendly approximation to Softmax functions (Mohassel & Zhang 2017):
softmax(z) = ReLU(x)/
Σ ReLU(z)
(2)
We validate that this has faster inference speed than the Softmax function in setting (Figure 4p. We denote this approximation as "2ReLU". However, this is not yet satisfactory. Analysis in $ shows that evaluating the ReLU function requires heavy of comparison routines, which is very expensive. Thus, we propose a more aggressive approximation for the Softmax by replacing the ReLU in Eq. with quadratic function:
soflmax(()) R (a + c)2 / cc)²
(3)
We denote this as "2Quad"'. Importantly, "2Quad" and Softmax functions differ lot by numerical values, while prior works argue that similarity in numerical values is crucial to the model's per- formance (Chou et al. 2018). We are able to use this aggressive approximation because our next distillation stage effective enough to bridge the performance gap. Figure 4 shows the comparison between the running time of the original GeLU and softmax function against their approximations. In particular, 2Quad has a much faster inference speed than 2ReLU.
4.2.2 STAGE 2: DISTILLATION
In the second stage, we use KD to make the fast approximated Transformer model S' performant. The benefits of KD are two-fold. First, it allows as "2Quad", which leads to higher speedups. Second, its data efficiency allows us to effectively leam good S with the small downstream datasets. Coneretely, conduct layer-wise distillation
aggressive approximations such
5

---

Published as a conference paper at ICLR 2023
40
total time
10
9.6
35
34.1
comm. time
30
Time (seconds)
25
20
15
12.8
3.2
0.5
1.7
gelu
quad
softmax
2relu 2quad
GelUHikke Functions
Softmax-like Functions
Figure 4: Running time comparison of different approximations in $4.2.1 Blue areas represent the total running time. Orange areas represent the communication time. MPC-friendly approximations greatly reduce both the communication time and the total time. In particular, our proposed "2Quad" much faster than the original Softmax function, and the previous "2ReLU" approximation.
to transfer knowledge from the input model T to S' by matching representation at the following four positions: (1) the embedding layer, (2) the attention matrix in each Transformer layer, (3) the hidden states after each Transformer layer, and (4) the final prediction layer. These four positions have been shown store meaningful information in previous works (Hinton al. 2015 Tiao al.) 2019 Clark et al. 2019). We use the Mcan Square Error (MSE) loss to match the representations between 7 and for all positions. We follow the learning procedure of Jiao et al. (2019) to first distill the embedding and Transformer layers (including the attention matrix and the hidden states) and then distill the prediction layer.
Student initialization. An important component knowledge distillation the initialization of the student model (Sanh et al. 2019). Taking the advantage that S' and T share the same architec- ture, we initialize S" using weights in T. We find that this outperforms random weight initialization, especially smaller datasets (3.3)
EXPERIMENTS
We design the MPCFORMER framework to be compatible with many MPC-friendly approximations and trained Transformer models, so that model providers can conveniently plug in MPC-friendly approximations based on their MPC systems. Thus, we evaluate MPCFORMER with different MPC- friendly approximations under (1) Different datasets ($ (.I.. and (2) Different models (especially larger models) (:5.2). In the ablation study, we study (1) the effect of student initialization and, (2) the effect of the number of training examples in the distillation stage.
Experimental setup. We use two P3.2x AWS instances to simulate the inference service scenarios (one P3.2x for the model provider, and one for the user). Each instance is equipped with one Tesla V100 GPU, and 10GbE Ethernet bandwidth. We place instances in the same placement group to guarantee 10GbE bandwidth in AWS. Time breakdown measured with Cryp'Ten, which imple- ments secret sharing with semi-honest adversaries assumption (Section $2) (Knott et al. 2021). We train and evaluate models based on HuggingFace (Wolf et al. 2019). In particular, we find that the implementation of 2Quad requires careful of HuggingFace source code ($A.2)
Baselines. We identify three important properties during Transformer inference in $ I speed, performance, and privacy. In our workflow, privacy has been guaranteed by using MPC systems. Thus, we evaluate MPCFORMER by the other two aspects: speed and performance. Concretely, S shall run faster than T while matching the performance of T. Since there is a limited amount of work on Transformer inference in MPC, we seek baseline from CNN literature. In particu- lar, we choose the training strategy in (Chou et al.) 2018) and denote it MPCFORMER MPCFORMERwola! also constructs the approximated model S' but trains S' on D with the task- specific objective, i.e. without distillation. We note that S' is initialized with weights in 7 i.e with different functions, whose effect has not been studied. We thus propose second baseline MPC-

---

Published as a conference paper at ICLR 2023
FoRMERw/oppd) which trains S' on D without distillation, and random weight initialization. Below we compare the performance MPCFORMER with MPCFORMER and MPCFORMER with the same speedups.
We denote the output model of our framework with BARTBASE. Roberta-base, and BERT LARGE as MPCBert-B, MPCRoberta-B,, B, and MPCBert-L for short.
5.1 COMPARISON WITH BASELINES ON DIFFERENT BENCHMARKS
Settings. In this section, evaluate MPCFormer framework with different approximations and compare it with baselines on the IMDb dataset and the GLUE benchmark (Maas et al.) [011 BERT BERTBASE BASE as the base model. Accord-
Wang al. 2018). For all experiments in this section, ing to the dataset statistics, we use a sequence length of 512 for the IMDb dataset and a sequence length of 128 for GLUE datasets. We note that longer sequence length generally reflects higher speedup because the Softmax functions can be sped up more. Baselines are trained with learning rates tuned from le-6, 5e-6, le-5, and le-4, the number of epochs from 10, 30, and 100, the batch size 32 for IMDB, batch sizes 64 and 256 for GLUE. MPCBert-B is trained with learning rate 5e-5 for embedding and Transformer layer distillation, and le-5 for prediction layer distillation. Further details on hypee--arameters tuning can be found in A.4
Table 2: Performance and speedup the IMDB dataset and part of the GLUE benchmark (QNLI, CoLA and RTE) with different approximations and BERTBAR BASE as the backbone. The input model T is denoted with "*" "p" stands for using weights in T as initialization, "d" stands for applying knowledge distillation with as the teacher.
Method
Approximation
IMDb
GLUE
Speedup
Accuracy
Speedup
Avg. Score
Bert-B"
GeL.44offtmax
1x
94.1
1x
73.1
MPCBert Bulo(n))
87.5
40.8
MPCBert- (rCBBet--Biiod))
Quad+Softmax
1.24x
87.5
1.1xx
43.0
MPCBert-B
94.0
72.6
MPCBert-B UnCBert-Baiopp.l
Bulo(pa)
86.1
39.6
MPCCBert-Baioo))
GeI.U+RReL.U
1.76x
93.8
1.40x
71.8
MPCBert-B
94.0
72.0
APCBert-Baillfee
85.8
43.5
PrCBert-Buisga
Quad+2ReLUU
2.61
86.8
1.93x
48.2
MPCBert-B
94.0
69.8
MPCBert-Baisfpll
87.3
39.6
MPCBert-B.un Buw
GeLU+2Quad
2.65x
90.6
1.55
69.7
MPCBert-B
94.0
71.0
MPCBert-Ba.hhee
87.8
40.7
MPCBert-B.. BBibdd)
Quad+2Quad
5.26x
87.3
2.20x
40.8
MPCBert-B
93.9
68.4
Table 3: Performance and speedup the IMDB dataset and part of the GLUE benchmark (QNLI, CoLA and RTE) with different approximations and BERT BASE as the backbone. The input model T is denoted with "*" "p" stands for using weights in T as initialization, "d" stands for applying knowledge distillation with as the teacher.
Method
IMDb
GLUE
Speedup
Accuracy
Speedup
Avg. Score
Solution
94.1
Ix
73.1
Solution 2
87.8
40.7
5.26x
2.20x
MPCFormer
93.9
68.4
We show the accuracy and speedup the IMDb dataset in Table MPCBert-B achieves 5.26x speedup with "Quad+2Quad" approximation with almost no accuracy drop. In addition, we note that this holds for not only the fastest "Quad+2Quad" approximation but other approximations rang-

---

Published as a conference paper at ICLR 2023
Table 4: Performance on Glue benchmark with BERT BASE as the backbone. Fl score reported for QQP and MRPC. Average Pearson and Spearman correlation is reported for STS-B. Matthews correlation is reported for CoLA. Accuracy is reported for other datasets.
Method
Approx.
MNLI
QQP QNLI SST-2
CoLA
STS-B
MRPC
RTE
Avg.
Speed
393k
363k
108k
67k
8.5k
5.7k
3.5k
2.5k
-up
BiItBBB(GeUUSSottmax
84.7/85.0
88.1
91.7
93.1
57.8
89.1
90.3
69.7
82.8
1x
MPCBert-B Bio (pdd)
Quad
62.1/61.3
74.6
61.8
80.7
13.4
23.1
81.2
55.2
56.6
MPCBert-Bww Bul
+
73.1/72.5
82.9
75.5
83.4
16.4
41.3
81.2
52.7
63.3 1.93x
MPCBert-B
2ReLU
85.0/85.3
87.8
91.2
92.0
54.0
85.7
88.9
64.3
81.1
MPCBert-Bwa Bulo (pd)
Quad
63,5/62,4
78.6
59.8
81.1
9.9
19.5
81.4
52.7
55.7
MPCBert-Bwi (d)
70.6/70.5
83.4
69.8
83.3
0
36.1
81.2
52.7
60.9
2.2x
MPCBert-B
2Quad
84.9/85.1 88.1
90.6
92.0
52.6
80.3
88.7
80.3
ing from 1.24x to 2.65 speedups. Baselines have inferior performance for all approximations. For example, both baselines have at least an accuracy drop of 6.8% with 5,26 x speedup. Inter- estingly, MPCBert-B (d} has moderate accuracy drop for "GeI.U2RReLJU" approximation with 1.76x speedup. However, does not preserve accuracy with other approximations. In contrast, MPCFORMER consistently preserves accuracy with various kinds of approximation.
To validate the observation with more datasets, We evaluate MPCBert-B on the Glue benchmark (8 datasets) (Wang et al. 2018). As shown in Table 4 MPCBert-B achieves 1.93x speedup with 98% performance of BERT BASE on the GLUE benchmark, and 2.2x speedup with 97% performance of BART Both baselines introduce severe performance drop, i.e. 19.5 average score drop for MPCBert-Bwiafd and 26.2 average score drop for MPCBert- Bio(pa) with x speedup. Interest- ingly, we observe that the baseline MPC.eert-Bwott consistently outperforms MPCBert-Bwooppy This indicates that using weights in as initialization benefits the training of S' with a task-specific objective, using 12-layer Transformer backbone. Additionally, we evaluate MPCFORMER with more approximations using a subset of the GLUE benchmark. Results are shown in the right part of Table 3 We observe similar patterns as in the IMDb dataset. The baseline MPCBert-B, (oo per- forms well in "Gel "Gel.U+2Quad' and "GelJU+RReLU" approximations, but MPCFORMER achieves high MI. performance consistently under all approximations.
5.2 MORE COMPARISONS WITH DIFFERENT MODELS
We evaluate MPCFORMER with trained Transformer models other than BERT BASE, i.e., ROBERTABASE model (12 layers) (Liu al. 2019), and in particular larger BERTLARGE model (24 layers). Results are shown in Table 5 MPCRoberta-B preserves 98% average score of the input model ROBERTABASE- and outperforms baselines by large margins. Comparing the perfor- mance of baselines, again observe that initialization with weights in T helps training with S', ie., MPC (d} performs better than MPCRoberta-Ba. {p.d--
Table 5: The performance subset of Glue benchmark with Roberta-base backbone (denoted as "MPCRoberta-BB") MPCRoberta-B and baselines use "Quad+2Quad" approximations with 2.1 x speedup. MPCBert-L and baselines use "Quad+2ReLU" approximations with 2.0 speedup.
MNLI-m
MNLI-mm
QNLI
RTE
Avg
Speedup
Roberta-B"
87.4
87.0
92.6
76.5
85.8
1x
MPCRoberta-B (p4)
58.0
58.1
69.0
52.7
59.5
MPCRoberta-Baio (4)
73.1
72.7
81.6
52.7
70.0
2.1x
MPCRoberta-B
86.5
86.9
92.2
72.2
84.5
Beri-L.*
86.7
86.6
92.7
75.1
85.3
1x
MPCBert-Lyy (pd)
62.3
62.3
60.0
52.7
59.3
MPCBert-Lwwo (d)
35.4
35.2
50.5
52.7
43.5
2.0x
MPCBert-I.
86.5
86.7
92.8
72.2
84.6
To show our method can scale to different model sizes, we evaluate MPCFORMER with larger model BERT (24-layer). On the IMDb dataset, BERTRASE achieves 95.0% aceu-
8

---

Published as a conference paper at ICLR 2023
racy. MPCBert-L achieves 5.9x speedup with 94.5% accuracy, while baselines MPCBert-Lwwa [pd] achieves 87.1% accuracy, and MPCBert-Lwe (d) achieves 50.0% accuracy. We further select four datasets from the GLUE benchmark, where Bert-L* noticeably outperforms BerLL QNLI, CoLA, and RTE). Compared with B in table 4 Bert-I.* increases the average score from 82.8 85.3. MPCFORMER increases the average score from 81.5 to 84.6. This indicates that MPCFORMER can scale with the size of T. On the other hand, baselines do not scale with the input model: MPCBert-Loito (pd) decreases the average score from 60.1 59.3; MPCBert-Looo (d) decreases the average score from 68.5 to 43.5. In particular, observe that initializing S' with weights in T without distillation harms performance when the model is larger.
5.3 ABLATION STUDY
The first question we study is the effect different student initialization. This is of interest as we do not have a general knowledge of whether initializ-
1.02
Relative performans P
1:00
0.58
ing with weights in T will still benefit the training of S' after aggressive approximations. We design an experiment with random initialization (denoted as MPCCBrt-BB B, in the table). We train MPCBert-B, with 10x more epochs than MPCBert-B, and con- firm that its distillation objective has converged. We observe that on larger datasets(QNLI) and SST-2), the gap between using different initialization is small, but on smaller datasets(STS--A MRPC, and RTE), initializing with the weights in T is better.
0.34
0.92
MRPC
RTE
0.38
SST-2
0.86
QNLI
Training example ratio
Figure 5: Ratio of training examples versus performance (normalized by performance with ratio = 1.0) on MRPC, RTE, QNLI, and SST-2.
The second question we study the effect of the number of training examples in the distillation stage.
We perform studies on two small (RTE, MRPC) and two medium (SST-2, QNLI) datasets in the GLUE benchmark (Figure(5) 5). We find that roughly 5% of the small datasets and 2% of the medium datasets provides enough data to learn a good S. This shows that KD in our setting efficient enough to learn a good S in downstream tasks (using GLUE datasets representatives).
Table 6: Student model initialized with weights in T versus with random weights. Result for MPCBert-B, is tuned with embedding and Transformer layer distillation learning rates from 5e-5 and 3e-5. Results for both are obtained with the "Quad+2Quad" approximation.
QNLI
SST-2
STS-B
MRPC
RTE
Avg
MPCBert-B
90.6
92.0
80.8
88.7
84.0
MPCBert-Br B, 90.6
90.0
60.0
81.2
58.5
76.1
5.4 LIMITATION AND FUTURE DIRECTION We recognize two limitations in our paper. First, our speedups and performance are tested on single MPC system. We leave theoretical analysis or empirical study on more MPC systems as future work. Second, in design, T and S only differ by functions, i.e., they have the same model size. We leave extension to a smaller student model as future work.
CONCLUSION
In this paper, we propose framework to achieve fast and performant private Transformer model inference with MPC. Evaluations show that it is compatible with various MPC-friendly approxi- mations and trained Transformer models. We suggest two directions of interest: (1) Theoretical or empirical analysis more MPC systems, and (2) extension the problem formulation to allow smaller size of
LEKNOWLED
The authors thank Qirong Ho for allocating the computing resources. This research was sup- ported by NSF 11S1563887, NSF CCF1629559, NSF ISI111758, NGA HM04762010002,, NSF IIS I1S1955532, NSF CNS2008248, NSF IIS2123952 and NSF BCS2040381.

---

Published as a conference paper at ICLR 2023
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint wrXiv:/409.047.. 2014.
Donald Beaver. Efficient multiparty protocols using circuit randomization. In Annual International Cryptology Conference, pp. 420-432. Springer, 1991.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Edward Chou, Josh Beal, Daniel Levy, Serena Yeung. Albert Haque, and Li Fei-Fei. Faster cryp- tonets: Leveraging sparsity for real-world encrypted inference. arXiv preprint 1810://11.00993. 2018.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.045: 2019.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D) Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003. 10555, 2020.
Ronald Cramer, Ivan Damgård, and Yuval Ishai. Share conversion, pseudorandom secret-sharing and applications secure computation. In Theory of Cryptography Conference, pp. 342-362. Springer, 2005.
Ivan Damgård, Valerio Pastro, Nigel Smart, and Sarah Zakarias. Multiparty computation from somewhat homomorphic encryption. In Annual Cryptology Conference, pp. 643-662. Springer, 2012.
Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19299, 2020.
David Evans, Vladimir Kolesnikov, Mike Rosulek, et al. A pragmatic introduction secure multi- party computation. Foundations and Trends@ in Privacy and Security, 2(2-3):70-246, 2018.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre- are-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
Oded Goldreich, Silvio Micali, and Avi Wigderson. How to play any mental game, or completeness theorem for protocols with honest majority. In Providing Sound Foundations for Cryptogruphy: On the Work of Shafi Goldwasser and Silvio Micali, pp. 307-328. 2019.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: arXiv:1503.0253:1 2(7), 2015.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554, 2006.
10

---

Published as a conference paper at ICLR 2023
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0588, 2012.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 1IX1:1199.110511, 2019.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. {GAZELLE]: low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pp. 1651-1669, 2018.
Marcel Keller, Emmanuela Orsini, and Peter Scholl. Mascot: faster malicious arithmetic secure computation with oblivious transfer. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 830-842, 2016.
Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-pane party computation meets machine learning. Advances in Neural Information Processing Systems, $4:4961-4977 2021.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-superviser learning of language representations. arXiv preprint arXiv:1909, 1942, 2019.
Chen-Yu Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Artificial intelligence and statistics, pp. 562-570. PMLR, 2015.
Jian Liu, Mika Juuti, Yao Lu, and Nadarajah Asokan. Oblivious neural network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 619-631, 2017.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19xiv 16692, 2019.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
Andrew I.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http::
// w/www.ack www.acclweb...
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptographic inference service for neural networks, In 29th USENIX Security Sympo- sium (USENIX Security 20), pp. 2505-2522 2020.
Payman Mohassel and Peter Rindal. Aby3: A mixed protocol framework for machine learning. In Proceedings of 2018 ACM SIGSAC conference on computer and communications security, pp- 35-52,77 2018.
Payman Mohassel and Yupeng Zhang. Secureml: A system for scalable srivacy-preserving machine learning. 20/7 IEEE symposium on security and privacy (SP), pp. 19-38. IEFF, 2017.
Pascal Paillier. Public-key cryptosystems based on composite degree residuosity classes. In Inter- national conference on the theory and applications of cryptographic techniques, pp. 223-238. Springer, 1999.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2023
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre- vre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning,
pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn. Rex., 2 (1(140): 1-67, 2020.
M Sadegh Riazi, Christian Weinert, Oleksandr Tkachenko, Ebrahim M Songhori, Thomas Schnei- der, and Farinaz Koushanfar. Chameleon: A hybrid secure computation framework for machine learning applications. In Proceedings of the 2018 on Asia conference on computer and communi- cations security, pp. 707-721, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 2014.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100 2019.
Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor. An image is worth 16x16 words, what is a video worth? arXiv preprini arXiv:2 10..2003.3333 13915, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMIR, 2021.
Iulia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models. arXiv preprint arXi:: 2019.
Sameer Wagh, Divya Gupta, and Nishanth Chandran. Securenn: 3-party secure computation for neural network training. Proc. Priv. Enhancing Technol., 2019(3):26-9,, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804 074611, 2018
Yongqin Wang, G Edward Suh, Wenjie Xiong, Benjamin Lefaudeux, Brian Knott, Murali An- navaram, and Hsien-Hsii Lee. Characterization of mpe-based private inference for transformer- based models. In 2022 IEEE International Symposium Performance Analysis of Systems and Software (ISPASS). pp. 187-197. IEEE, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-a natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xinet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019b.
Guoyang Zeng. Fanchao Qi, Qianrui Zhou, Tingji Zhang, Zixian Ma, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. Openattack: An open-source textual adversarial attack toolkit. arXiv preprint urXiv:2009.09199, 2020.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.I A CONCRETE SYSTEM IMPLEMENTATION OF MPC: CRYPTEN
In this section, we provide how a concrete MPC system (CrypTen) implements routines and func- tions for Transformer models in detail (Knott al., (02I). We provide a portion of details here to help describe the complexity of Transformer inference in MPC. A more complete system overview and privacy proof are available in the CrypTen paper.
Threat model. Cryp'Ten follows Evans et al. (0188) to assume that parties are semi-honest. Under this assumption, parties are honest that they will follow the system protocols. However, each party is also curious (i.c., semi semi-honest), meaning it will try to infer the information about others' data based on the values it receives.
Secret shares CrypTen uses secret shares to implement private computation. floating point value zy is first scaled to integer of then secretly shared with both parties. Secret shares are of type arithmetic or binary. The arithmetic secret shares [x]] = { [=]2] is a set two numbers, constructed with a pair of zero-
where the first party holds [xr] and the second holds [z]]2. They
sum random maskings (Cramer et al., 2005) so that [z]] + [al] = x. Binary shares (x) are formed
by arithmetic secret shares of bits v, that the bitwise xor (ar) + (x))
Routines and functions In arithmetic secret shares, to privately evaluate addition [x] + [v] both parties simply compute: [z]:+ [y]s individually. Multiplication [x: is evaluated by using a Beaver triple generated off-line [[c] b)) where e ab (Beaver 1991). Both parties compute and reveal intermediate values [c] [ar] [a] and [ M 16.. The final result is computed by [=][y]] = [c] [bb] + [a]ō c6. Linear functions such as matrix multiplication can be evaluated by us- ing additions and multiplications. Non-lineat functions are evaluated by numerical approximations
cò.
using additions and multiplications, such as Taylor expansion.
Comparison requires conversions between arithmetic and binary secret shares. Conversion from [x] to (x:) first creates binary secret shares ([z]<) from arithmetic secret share [x:]; (i = 0, 1), then
computes (a) = ([x]]) + ([r]2) using an adder circuit. Conversion from (xx) [ar] is done by:
[x] = Σ> (x))",) where (z:) is the 6th bit of (z). The comparison function [z < 0] is then evaluated by: (1) convert [z] to (z) (2) compute the sign bit (b) (z) >> (L 1). (3) Convert (b) to [b].
We study on the standard setting where each tensor is represented in 64 bits (i.e. 1==4). Each multiplication requires one round communication for revealing the intermediate values e, 8. Each conversion from [z] to (x) requires log, L 6 rounds of communications for the adder circuit; each conversion from (x)) to [x:] requires one round for generating (=)))). Thus, each comparison requires 7 rounds communication. Each max(-) between N elements requires (loog, rounds of communications, assuming tree-reduction algorithm.
We provide a simple addition example here. The scaling factor and ring size Q are set to small for ease of understanding.
Table 7: Example of private addition computation. Suppose m is the actual message, then each party holds a share of m such that: [m]] [m]a m.
Action
Party
Party 2
Note
Declare x
2
random
X provided by party
Generate secrel sharing mask for secret share
[=
1=
sum sum to
[=]]
z [ag]]
[+]a
[sula
Declare
randiom
2
y provided by party sum
Gienerate a socret-shaalee sharing mask for
Fy = 50
[syla
30
share
1sh
Eull
50
[vle
u [=
Compute +
[x +
1wh
47
+ vle
Reveal
ala
both got correct results
1/ E z/QZ is required for privacy protocols, where 2/Q2 is ring with Q elements.
13

---

Published as a conference paper at ICLR 2023
In addition, we provide more complete breakdown in terms of communication and computer load for Figure 2 for a holistic view of execution pattern in the MPC setting.
Table 8: Functions computation versus communication breakdown (Unit: seconds).
functions
Comm.
Time
Comp.
Time
Total Time
MatMul
2.5
5.0
7.5
GeLU
9.6
1.4
11.0
Softmax
34.1
5.9
40.0
Others
0.3
0.2
0.5
Total
46.5
12.5
59.0
In particular, computation only takes 21 % of the running time and the communication takes 79% of the running time. Consequently, the number of floating point operations (FLOP), popular estimator for the running time in plain-text Transformer inference, is longer accurate in the MPC setting. The MPC system we use in the paper uses All-reduce to implement intermediate communication, where both parties have the same communication load. And they have similar computation load (see the multiplication example, where both parties are computing the same function locally with their own secret shares). Thus, the time breakdown is similar for both parties. the above table, we report the statistics from the model provider.
A.2 2QUAD IMPLEMENTATION DETAILS
We note that, implementing "2Quad" to replace the softmax requires attention to the effect brought by the masking operation. For example, the default implementation by Huggingface Wolf et al. (2019) would result in an exploding problem due to masking. Therefore, we would need to do different version of the implementation of masking. We describe it in detail below.
The default attention implementation by Huggingface
QK + MMoo-in/3)V
Allention (Q, K, V) = softmart
Vdh
GKT vuc
+M(o--in))
V.
QKT Vdd
+D inf)
Σド=1
we directly replace the ex with (x++)² as in 2Quad approximation, where x QK vd. + +Mto.-inf) will explode when being masked, causing problem in the forward pass. To solve this problem, could simply change the implementation of masking from "adding zero or negative infinite number in the exponent" to "multiplying
zero the exponential function". That is,
QR V4g
M(100
Attention(Q, V, =
V
QKT Vdg
Σ j=1 e
O M(100
QKT
+ M(L.))
Vdh
V.
Σj=1
K
QQT Vdd
+ +e),
M(1
It's just a different implementation the same masking purpose but avoids exploding at the masking positions.
In our experiments, we empirically tried c = 5 and worked pretty well, indicating the choice of the constant could be flexible.
14

---

Published as a conference paper at ICLR 2023
A.3 ROBUSTNESS OF THE STUDENT MODEL
Some approximations may increase the local Lipschitz constants, which decreases the robustness. We applied some empirical text adversarial attacks to evaluate the adversarial robustness of the BERT-baas model before and after approximations (Zeng et al.) 2020). As shown in Table []] the student model has a moderate increase in terms of attack success rate (ASR) over the three score- based attacks. But the student model has a lower ASR with the gradient-bass attack HotFlip. Considering these results, the effect on robustness by the approximations are empirically moderate.
Table 9: Sanity aceuracy (SA) and attack success rate (ASR) against various text attacks. The TextFooler, PWWS, and BERT-ATTACK are score-based attacks, and HotFlip is gradient-based attack. For ASR, lower is better.
SA
TextFooler
PWWS
BERT-Attack
HotFlip
Speedup 1.0x
Bert-base (teacher)
0.93
0.76
0.78
0.88
0.55
Bert-base (student)
0.92
0.78
0.82
0.91
0.51
2.2x
A.4
CHOICE
For baselines, We study the effect of hyper-parameters by running grid search over the STS-B
dataset 6 with learning rate from [le-6, 5e-6, le-5, 5e-5, le-4, 5e-4], batch size from [256, 128, 64,
32, 16), epoch from [3, 10, 30, 50, 80, 100, 200]. We show the grid search results with BERT BASE in figure
BASE
and a smaller grid search for BERT and ROBERTABASE in Figure g 9 empirically discover that the learning rates from le-6, 5e-6, le-5, batch size from 64 and 256, epoch from 10, 100 give good performance. To let baselines explore more hyper-parameters, we use learning rate from [1c-6, 5e-6, 1e-5, le-4], batch size from [64, 256], epochs from [10, 30, sequence length 512 for IMDB dataset, 32 to fit into our 16GB Tesla V100 GPU. We also empirically discover that (1) MPCBert-B not scale up when the base model scales to BERT i.e., MPCBert-Lwwofai (best 0.08). (2) baseline benefits from using the pre-trained weights, iee, MPCBert-Bwiaffe (best 0.42) performs better than MPCBert- d} (best 0.23). (3) MPCFormerwooodf benefits when the base model becomes better, i.e., MPCRoberta- (best 0.62) performs better than MPCBert-B (best 0.42).
We
100] for all Glue datasets. Since
batch size
(best 0.43)
For MER, we decide the number of epochs according to the MSE loss for embedding and Transformer layer distillation, 5 epochs for prediction layer distillation, and batch size 8 for small datasets (CoLA, MRPC, RTE) and 32 for larger ones (MNLI, QQP, SST2, STS-B B). We minimize the hyper-parameter tuning for MPCFORMER, since we would like the performance be an expectation for future researchers using MPCFORMER, who prefer not to tune syper-parameters. Specifically, we use epochs for MNLI, 5 epochs for QQP, 10 epochs for QNLI, 10 epochs for SST-2, 20 epochs for MRPC, 30 epochs for IMDB 50 epochs for STS-B, 50 epochs for CoLA,, 50 epoches for RTE, for the embedding and Transformer layer distillation stage.
12-123
bo-04
bn-32
21-36
ALL-
0.10
b10-
>33
0.09
0.315
100
230
3.0
390
350
240
Figure 6: Grid search results for MPCBert-Bwofpaj on STS-B dataset. X-axis is number of epochs, and Y-axis is correlation (unscaled).
We select STS-B because it is regression task, where performance varies in large range.
15

---

X-axis is number of Published as a conference paper at ICLR 2023
25-252
te-122
bo-04
bo-33
20-36
33
Figure 7: Grid search results for MPCBert-Bwoody on STS-B dataset. X-axis is number of epochs, and Y-axis is correlation (unscaled).
1a=228
bow04
13#20
-0.44
-0.00
55
75
Figure 8: Grid search results for MPCBert-LLwooog on STS-B dataset. X-axis number of epochs, and Y-axis is correlation (unscaled).
255
6.3
G1
-31
Figure 9: Grid search results for MPCRoberta-B.. Bwiofa) STS-B dataset. epochs, and Y-axis correlation (unscaled).
16


Reviews:
[{"review_id": "v6qcYd6DEN", "reviewer": "Reviewer_ENWC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a procedure for building better surrogate networks for transformer models fine-tuned to a specific downstream task that allows for faster MPC for private sharing with clients while preserving the model's performance (defined by model accuracy). To build a better surrogate, the authors propose a two-stage pipeline, $\\text{MPCFORMER}$: 1) Replace bottleneck layers (GeLU, Softmax) in the transformer model with their faster MPC-friendly counterparts (Quad, 2Quad) to form an intermediate model $\\mathcal{S}'$ 2) Retrain $\\mathcal{S}'$ on the downstream dataset using Knowledge Distillation from the original transformer model. Finally, the authors provide experimental evidence to show the procedure's benefit for various Bert-based transformer models.", "summary_of_the_review": "The paper addresses an important issue of improving the computation time of MPC for transformer models while preserving the model's performance. To this end, the authors propose a simple and elegant framework that they show to be useful in practice. I think this paper would be of great interest to the ML community.", "strength_and_weaknesses": "Strengths\n- Strong empirical performance in terms of accuracy of the final model. For most of the models trained by $\\text{MPCFORMER}$, the downstream accuracy is close to the accuracy of the original model. $\\textbf{Question:}$ Under this model, would it be possible to have a continual training framework where the model provider could continue the distillation procedure on the partial computations of the teacher model vs. those of the distilled model for randomly chosen client inputs?\n- The current models provide a 2x speed up while using a student model of the same size as the teacher. Smaller student models could give even better speedups.\n- This is a very general framework that would support various approximations that might be used in the future.\n\nWeakness\n- As the activation functions are quadratic, it is possible for the local Lipschitz constants to increase, which would decrease robustness. Current performance measures do not consider other aspects of the model. It would be better to have a more holistic view of model performance for the comparisons.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C1PX9JxZcKY", "reviewer": "Reviewer_BURu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to transfer a transformer model to a slightly different one that is cheaper to compute in multi-party computation.\n", "summary_of_the_review": "Interesting technique applied to multi-party computation\n", "strength_and_weaknesses": "The paper works out very well how the different parts of their approach contribute to their solution. I'm not aware of these techniques being used in the context of MPC optimization before. I'm positive about the achieved speedups with only a minor loss in accuracy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7SU9nA1Vk7L", "reviewer": "Reviewer_tn3E", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes MPCFormer, a novel two-stage approach for private transformer inference with MPC. The first stage replaces bottleneck functions in the pre-trained transformer (ie, functions with high computational/communication complexity under MPC), with MPC-friendly approximations. The second stage uses knowledge distillation to map the outputs of multiple layers of the MPC-friendly model to those of the original transformer. Additionally, a novel MPC-friendly Softmax approximation, dubbed \"2Quad\" is proposed, which provides a significant speed advantage compared to the existing \"2Relu\" Softmax approximation. Experimental evaluations on a range of language tasks with a range of transformer architectures validate the efficacy of the proposed approach relative to existing methods, both in terms of speed and performance. ", "summary_of_the_review": "Strong systems paper with limited algorithmic/theoretical novelty.", "strength_and_weaknesses": "# Strengths\n\n* The proposed approach is highly practical, as it can be easily adapted to fit a range of transformer architectures and tasks. \n\n* The paper is the first to tackle private transformer inference with MPC. \n\n* A novel MPC-friendly Softmax approximation, dubbed \"2Quad\" is proposed, which provides a significant speed advantage compared to the existing \"2Relu\" Softmax approximation.\n\n* Through evaluation and strong results. Experimental evaluations across a range of language tasks and a range of transformer architectures validate the efficacy of the proposed approach relative to existing methods, both in terms of speed and performance. Ablation study demonstrates the importance of the proposed knowledge distillation step, model initialization protocol, and Softmax approximation.\n\n# Weaknesses\n\n* Limited algorithmic and theoretical contributions.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4xkhzEBX0R", "reviewer": "Reviewer_dCKB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed MPCFORMER to enable fast inference under the privacy constraint of mutlit-party computation for transformer based models. Some speed-up is achieved as a result of the relaxation of the proposed method in the experiments. ", "summary_of_the_review": "This paper proposes an interesting problem about privacy preserving inference of transformer based models, however, the technique section is not clearly stated and there is a lack of theoretical analysis of the approximation of the computation. \n", "strength_and_weaknesses": "Strength:\n\n+ This paper proposes an interesting question for privacy-preserve inference of transformer models.\n\n\nWeaknesses:\n\n- This is a lack of theoretical analysis on the relaxation of the computation. \n\n- The performance breakdown is unclear without concrete Flops analysis under the original computation of transformers and the computation of the MPC setting.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%