Iron: Private Inference on Transformers
Authors: Meng Hao
Hongwei Li
Hanxiao Chen
Pengzhi Xing
Guowen Xu
Tianwei Zhang
Venue: NeurIPS 2022 Accept (2022)

Abstract: We initiate the study of private inference on Transformer-based models in the client-server setting, where clients have private inputs and servers hold proprietary models. Our main contribution is to provide several new secure protocols for matrix multiplication and complex non-linear functions like Softmax, GELU activations, and LayerNorm, which are critical components of Transformers. Specifically, we first propose a customized homomorphic encryption-based protocol for matrix multiplication that crucially relies on a novel compact packing technique. This design achieves $\sqrt{m} \times$ less communication ($m$ is the number of rows of the output matrix) over the most efficient work. Second, we design efficient protocols for three non-linear functions via integrating advanced underlying protocols and specialized optimizations. Compared to the state-of-the-art protocols, our recipes reduce about half of the communication and computation overhead. Furthermore, all protocols are numerically precise, which preserve the model accuracy of plaintext. These techniques together allow us to implement \Name, an efficient Transformer-based private inference framework. Experiments conducted on several real-world datasets and models demonstrate that \Name achieves $3 \sim 14\times$  less communication  and $3 \sim 11\times$ less runtime compared to the prior art.

Iron: Private Inference on Transformers
Meng Hao
Hongwei Li
Hanxiao Chen'
Pengzhi Xing'
Guowen Xu² Tianwei Zhang?
Universityy of Electronic Science and Technology of China
Nanyang Technological University {menghao, hanxiao chen, xing) wwweeedddduttt....eemm westu. en tongweili@uester edu. cn fguowen. xu, tianwei. zhang] @ntu. edu. sg
Abstract
We initiate the study of private inference Transformer-biae based models in the client-server setting, where clients have private inputs and servers hold proprictary models. Our main contribution is provide several new secure protocols for matrix multiplication and complex non-linear functions like Softmax, GELU activations, and LayerNorm, which are critical components of Transformers. Specifically, we first propose a customized homomorphic encryption- ppapid protocol for matrix multiplication that crucially relies novel compact packing technique. This design achieves / x less communication (m is the number of rows of the output matrix) over the most efficient work. Second, design efficient protocols for three non-linear functions via integrating advanced underlying protocols and specialized optimizations. Compared the state-of-the-art protocols, our recipes reduce about half of the communication and computation overhead. Furthermore, all protocols are numerically precise, which preserve the model accuracy of plaintext. These techniques together allow us to implement Iron, an efficient Transformer-basee private inference framework. Experiments conducted on several real-world datasets and models demonstrate that Iron achieves 3 ~ 14x less communication and less runtime compared to the prior art.
Introduction
Transformer-basee models [1-5] have realized tremendous success in the fields of natural language processing (NLP) and computer vision (CV) due to their strong representation capabilities. As new neural network architecture, the Transformer [1] mainly utilizes the olffaattntion mechanism to compute representations without sequence-aligned recurrence or convolution. Following this work. number of Transformer variants, such as BERT [2] and GPT [3] in NLP, VIT [4] and Swin Transformer [5] in CV, have achieved state-of-the-art performance on lots of real-world tasks.
The success of Transformers and other big models facilitates emerging inference services and applications [6, 7]. particular, a service provider trains a complex model based on the Transformer, a paid inference service, e.g., machine translation and question answering. A client queries this service with his input samples and obtains the desired responses. Unfortunately, current inference systems suffer from serious privacy concerns [8]. On the hand, clients need to send confidential inputs to the service provider, which could compromise the data privacy of these clients if the provider is untrusted. On the other hand, distribute the proprictary Transformer based model to clients, since he needs large amount of data and computation resources to construct the model [9]. Therefore, there exists gap between
and deploys
undesirable for the provider to
"This work was done at NTU a visiting student. Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

unprecedented performance and privacy constraints, which motivates our study of private Transformer inference.
Private inference aims to protect server's model weights from clients, while guaranteeing that the server learns no information about clients' private inputs. Recently, private inference traditional neural networks (e.g., convolutional neural networks) have been approached by using secure 2-party computation (2PC) techniques [10-14]. However, due to essentially different structures, private Transformer inference brings several new challenges. First, Transformer-baser models use lots of high-ding high-dimensional matrix multiplications, rather than matrix-vectoo multiplications widely studied in prior works. While we can straightforwardly extend prior matrix- multiplication protocols to our setting, unfortunately, even the most efficient design [14] incurs heavy communication because of interacting large amount of ciphertexts. Second, Transformer-based models use complex math functions like Softmax, GELU activations [15], and LayerNorm, in each block, rather than crypto- friendly non-linear functions such as ReLUU and Maxpool. Existing methods either use precision- impaired high-order polynomial approximations [16, 11] or only support limited math functions for specific scenarios [17]. Even worse, all of these approaches are computationally intensive and often require a large amount of communication (for more related works, please refer to Appendix A.5), To facilitate the widespread adoption of Transformer-basee inference services in privacy-critical scenarios, designing efficient protocols for the above complex operations is of paramount importance.
In this paper, we design Iron, an efficient hybrid cryptographic framework for private Transformer in- ference without revealing any sensitive information about the server's model weights or clients' inputs. Iron contributes several new specialized protocols for the complicated operations in Transformers to alleviate the performance overhead. Specifically, we first propose customized homomorphic encryption- based protocol for matrix multiplications. Our insight is to pack more plaintext inputs into a single ciphertext by devising a compact packing method, while preserving the functionality of ma- trix multiplication. Compared to the most efficient matrix-vector multiplication solution implemented in Cheetah [14], we can achieve vmax (m is the number of rows of the output matrix) improvement in terms of communication overhead, which is about 8x reduction for various Transformer models. Second, we carefully design efficient protocols for Softmax, GELU, and LayerNorm. These protocols are built upon SIRNN [17]. the state-of- offtthe-art cryptographic framework for private inference recurrent neural networks, and make several customized optimizations, such as reducing the overhead of exponentiation in Softmax and simplifying GELU and LayerNorm. These optimizations achieve 1.3 ~ 1.8x less runtime and 1.4 ~ 1.8x less communication on three non-linear functions. Further- more, these protocols are numerically precise, which preserve the model accuracy of plaintext. We also give a formal security proof for our designed protocols to demonstrate the security guarantee.
Based on the above efficient components, we implement a private Transformer inference framework, Iron, and conduct end-to-end experiments with various BERT architectures [2] (BERT-Tiny, BERT- Medium, BERT-Base and BERT-Large on GLUE benchmarks [18]. Note that Iron is readily extended to other Transformer- based models (c.g.. ViT) since they share very similar architectures and same operations. Experimental results show that Iron achieves 3 ~ 14x less communication and 3 ~ 1 less runtime costs over SIRNN on four BERT models. Moreover, compared with the general-purpose state-of-the-art framework MP-SPDZ [16], Iron has up to two orders of magnitude improvement in terms of both communication and computation efficiency.
A concurrent work [19] proposed a privacy-preserving Transformer inference with homomorphic encryption, called THE-X. Below, we illustrate some important differences in terms of protocol design and security. (1) Protocol design. Our work aims to design new efficient protocols for the complex operations of Transformer-bass with crypto- friendfy operations. For example, THE-X X replaces GELUs with simpler operations, i..., ReLUs, and Sofumax with the combination of ReLU and polynomials. (2) Security. Our work achieves more rigorous privacy protection than THE-X. Specifically, our work uses homomorphic encryption and secret sharing techniques to hide private information (including intermediate results) of all layers. Such rigorous privacy guarantee is in line with recent tfte-offthhe-rt private inference works [14, 17]. However, in THE-X, the inputs of each non-linear layer are leaked to the client, which may causes severe privacy leakages in real-world applications [13]. Therefore, our work may be used to enhance the security of THE-X.
models, while orthogonal to ours, THE-X replaces them
2

---

2 Preliminaries
2.1 Threat Model
As shown in the left part of Figure 1, Iron works in a general private inference scenario, where the server Po holds Transformer-based model M with private weights w, while the client P1 holds private input z. Our framework enables the client to query the server's inference service and learn the output of the model on its input, i.e., M (w, x).. Same as prior works [14, 17J, we consider an honest-but-curious adversary that passively corrupts either the server or the client, but not both. Such an adversary follows the protocol specification exactly, but may try to learn more information than allowed (e.g., the model's weights or inference inputs) via analyzing the data it receives. In Appendix A.1.2, we give more formal description of the threat model for security analysis.
2.2 Cryptographic Primitives
All our protocols are built on the 2-out-of-2 additive secret sharing (ASS) technique [21, 22] over the ring Eze, in which an é-bit input z is split into two random shares (z)o. (z)), held by Po and Pt. respectively, such that x = (z)) + (2)1 mod Zze. When L = 1, i..., over Zg. we use (x) to denote boolean shares. In protocols,
use (zr) to denote that P, holds (x); for b E {0.1}. The security guarantees that given a share (z)) or (z)) the value of x is perfectly hidden. This secret-sharing property is maintained throughout our private inference scheme. ASS naturally supports linear operations without communication. For instance, to compute cr + with the constant c and secret shares (ar) and (v), P. can locally compute (2)s- = c(x)) (y)), where (=)0 + (2)1 cr + y mod 2 To achieve more functionalities under shared inputs, we require to invoke advanced homomorphic encryption or oblivious transfer techniques [22].
Notations.
Let æ means x is not a divisor y. We use bold lower-case letters (e.g., x) to represent vectors, and bold upper-case letters (e.g.. X) to denote matrices. Like prior works [17, 14], we encode inputs with the fixed-point representation denoted by Fix (refer to Appendix A.1.1 for details). Let AN denote the set of integer polynomials AN.p Zp|æ:// (x + +1). We use the circumflex of lower-case letters (e.g., â) to represent a polynomial, and a[i] to denote the i-th coefficient of à. Given polynomials é, 5 E AN.ps the product& 2 = a. ÿ over A is defined as
2|1]
Σ OSI
Abbo -
Σ i<jKN
+DENN - j + mod p.
(1)
Additively Homomorphie Encryption (AHE) [23, 24]. This encryption scheme additionally enables linearly homomorphic operations on ciphertexts. Specifically, an AHE scheme is a tuple of algorithms AHE (KeyGen; Enc; Dec; Eval) with the parameter 9, q, p) and the following syntax: 1) KeyGen (1*) + (pk, sk): on imput a security parameter K, KeyGen is randomized algorithm that outputs public key pk AN.q and a secret key sk E AN.q. 2) Enc(pk, m) + è: the encryption algorithm Enc takes a plaintext polynomial m E AN.p and encrypts using pk into a ciphertext polynomial é E AN... 3) Dec(sk, é) + n: on input sk and a ciphertext è, the (deterministic) decryption algorithm Dec recovers the plaintext message m. 4) Eval(pk: ê, func) -> é: on input pk. two ciphertexts è è2 containing mh. m22 and linear function func, Eval outputs new ciphertext ē encrypting func (m², m2)) Let E. E, X denote homomorphic addition, homomorphic subtraction and homomorphic multiplication with a plaintext, respectively. Iron builds the matrix multiplication protocol on the Brakerski- Vercauteren (BFV) scheme [25, 26], which is one of the wfae--of-the-ar lattice-based homomorphic encryption solutions.
Oblivious Transfer. The l-out-of-k oblivious transfer (OT) [27] is denoted by k-OT,, where one party is the sender with k messages Zo.... Tk-1 {0, 1) and the other party is the receiver with an index i [k). The receiver learns x; as the output, and the sender learns nothing. Additionally, we also use the ofout-of-2 correlated OT, denoted by 2-COT, [28], which is defined as follows: the sender inputs a correlation E Zz,, the receiver inputs a choice bit i E {0, 1}. and the protocol outputs a random element r E Zyz to the sender and r + i x to the receiver. k-OT, and 2-COT, require 2X- kl and l bits of communication, respectively, and are executed in 2 rounds. The OT protocols are widely used to build the underlying protocols [13, 17] in Figure 3 we rely on.
'Like prior works [17, 14). Iron does not hide the information that can be indirectly extracted from the inference results. Study of mitigation solutions (e.g., differential privacy [20]) beyond the scope of this work.
3

---

Server
Ponkece Faceddng
Propristary model
Tranformr
Lines
Mathdul
Softmrae
Layanoom
Finer
LayeNoon
CELT
Private Inference
Enireding
Albertion
heed
Cheguts
Impats
0
liput:
Output
Cryptographic
Protocols
Alsurrox
Dayertiium
Nozuc
Cllent
Figure 1: Overview of private Transformer inference
Overview
3.1 Transformer Architecture
The Transformer is encoder-decoder architecture, where both parts have similar structure. Hence, we mainly focus on the encoder below. The encoder is composed of a stack of identical blocks, each with two sub-layers, i.., multi-head self-attention mechanism and a werd-fo network, as shown in Figure I. Besides, residual connection and layer normalization (LayerNorm) are employed around each of the two sub-layers. We describe a block of the encoder as follows.
Attention layer. An attention function can be described as mapping query XQ and set of key- value pairs (Xx:yv Xv) to a weighted sum of the values, where the weight is computed by metric of the query with the corresponding key [1]. This function can be formalized as below:
Attention(X., (XQ. XK. Xy) Softmax (XxX//V Xv.
(2)
where XQ, XK. Xy are different linear projections of the input X. i.c., XQ = XWQ. XW K. Xy XWv. , and d is the dimension of representations. Multi-head attention extends the above mechanism to H parallel attention layers and is illustrated in Appendix A.1.3.
Feed-forward layer. A fully connected feed-forwar layer consists of two linear transformations with a GELU activation in between, where GELU is the Gaussian Error Linear Unit function [15]. This layer can be represented as follows:
FeedForward(X) (X) GELU(XW, bb,) b,) b2.
(3)
In addition to the above encoder-decoder blocks, an embedding layer is employed at the beginning of the model to convert input tokens Xinput to continuous feature vector representations. This is formulated as X Xinpur W E. where W g is the embedding lookup table.
3.2 Private Transformer Inference
According to the required crypographic operations, the layers of Transformers can be broken into two categories linear and non-linear.
1) Linear layers: these include embedding, matrix multiplication Attention, and fully-connected layer. All protocols of these operations rely crucially the matrix multiplication protocol Ma in Section 4. It lies in the setting where Po and Pf take as input the matrices X and Y, and learn the sharings (Z) and (Z),, respectively, such that Z XY.
2) Non- Non-linear layers: these consist of Softmax, GELU and LayerNorm. The non-linear operations directly evaluated by exploiting our proposed protocols in Section 4.2, i.e., TGELU, TT LlaayeNNorm- All of these take the additive shares (X) as input, and output the additive shares (Y) = Func (X)), where Func E [GELU, LayerNorm).
Overview of private inferencce In the entire private inference process, we maintain the following invariant: Po and P1 begin with additive shares of the input to the layer, and end with additive shares (over the same ring Zzt) of the output of the layer after the protocol. This allows us to stitch protocols for arbitrary layers sequentially to obtain secure computation scheme for any Transformer. based models. To clearly understand our scheme, we take an example of private inference between Po and Pi on the embedding layer and the first encoder block, shown in the right part of Figure 1. Here, our example uses the designed protocols as a black box, i.e., MaatMul. TGGLL, So and Il LyyerrrNr and we discuss how to efficiently implement these protocols in Section 4.

---

Aneanpleovar
8= 2x 1xx 4x +xx²
Evaluation in MaMaa
x
[1
2
Y=
1
9
9 5x +6x"
+ 7xx 8x" 9xx 10x²
1
4
6
8
101
1
| (x20 +122
2 X-X-Y=
[17
17
21
23
291
mod 2
= 10x + +17x' + 20x² + 23x² + 26x* 29x* + 30x*
+ +xx + 14x + 21x + 28x 3x 30x mod (x16 1,29)
Figure 2: A toy example of our matrix multiplication protocol.
In the embedding layer, Po and Pi take as inputs WE E and Xinquts and invoke the matrix mul- tiplication protocol MMaaMua to compute the input embedding X. This is represented as (X) = (XMatuu(Xiipputt WE). In the attention layer, Ma is also invoked to generate the query, key and value matrices. For instance, (Xq) = IMMMMMu (X) We) (X)o Wo. where (X)o W can be computed locally by Po- The same idea is used in the generation of (Xx) and (Xv). Then, we compute XOx = X
K that requires two invocations of Ma The formulation is (Xox) MMMaMM ((Qq)u: (X x) + IlMaMMul ( (XQ)1, (Xx)o)= + (Xx)a(XX K + (Xo) (Xx)1. where the last two terms can be evaluated by Po and P1 locally. After that, we evaluate (Xok) Ilsoftmax (XQ /Vd,,, followed by computing (Z) = (XoxXv) by two invocations of Ma Then, the two parties invoke the LayerNorm protocol to evaluate (Z) +ILLayeNNorml ( (Z)). In the feed-forward layer, they compute (Yi) IlMatMui (()),, Wi) + (Z)o W followed by After that, they compute (Y2) = IlMatMu( (Y)), W1) + (Y)o W., where
(Y) IIGELL
YY) can be computed locally by Po. Finally, the evaluation of the encoder completes after computing (Y)
LayaerNorm ( (Y2)).
4 Supporting Protocols
4.1 Protocol for Matrix Multiplication
As shown in Section 1, naively extending well-studied matrix-vector multiplication protocols [29, 13, 14] to our matrix multiplication setting results significant communication overhead, mainly due to firqueetly interacting ciphertexts. In this work, we build specialized matrix multiplication protocol on top of the most efficient protocol, Cheetah [14]. Recall that the plaintext of an AHE scheme is polynomial, which can pack large number of inputs to amortize the overhead [29, 13]. The key contribution of our proposed protocol is a more compact input packing approach.
Our starting point is that polynomial multiplication implies vector inner product, if we arrange the coefficients properly [14]. As shown in Equation 1, when multiplying two polynomials of degree-(NNI)) the (N-1)-th coefficient of the resulting polynomial is the inner product of the two coefficient vectors in opposite orders. By using an appropriate arrangement of coefficients, this idea can be extended to matrix multiplications, since they consist of set of inner products. We give the definitions of two input packing functions T. Zn x - A and Z - N,2 as follows:
c = T (X), s.t., & [i n ++ (n - 1) - -]= -X for E [m].. E [m]
U - Re YY) s.L., v [j n + = Y(i for i E [m],j [K]]
where all other coefficients of å and á set to 0. Multiplication of polynomials 2 = f 9 directly gives the result of matrix multiplication Z - XY mod 2' in some g's coefficients. We formalize this process below, and give toy example in Figure 2.
Theorem 4.1. Assuming mark S N and given two polynomials å = (X) and = (Y),,, the matrix multiplication Z XY mod 2 can be evaluated via the product f. ÿ. where Z j] is computed in & k (j + 1). 11 for i E [m) and j E [k]].
We defer the proof to Appendix 1.2.1. When mork > N. we first partition the matrices X, Y into sub-matrices of m. Thw and no x ke elements, respectively, such that Bankyy N. Zero-padding is required when my { m, Rw n or kw f k. The protocol is shown in Algorithm 1.
Complexity and security analysis. For complexity, totally, two parties interact ciphertexts, and operate with O(mnk/N homomorphic additions and multiplications. We formalize the selection of parameters mas, Tayy kw as optimization problem, to minimize the communication
m__ + Thw
R mag
5

---

cost. Through our analysis in Appendix A.2.2, in general scenarios, our method theoretically achieves vm communication improvement over Cheetah [14]. Notably, when mnk .,, we reduce the communcation cost by a factor of m, since Cheetah encodes each row of the matrix into a ciphertext. Besides, the security proof shown in Appendix A.2.3.
Algorithm Secure Matrix Multiplication Protocol
Input: Po holds X Xn and P1 holds Y E Z.
Output: Po and P1 get (Z)o, (Z)1 E Znt respectively, where Z = XY.
I: Po. Pi compute the partition window size 0 my m. 0 nw and 0 Kw k such that Mw I N, and set n' = =[n/n/ [n//nw, m' = [m/mw]] and K' [ht/kw]-
2: Pi partitions the matrix Y into block matrices a E Th xhu for 3 E [n']] and Y E [k]]
3: Pi encodes the matrices to polynomials 03.7 = RR (Y (,n) for B E [n'] and y E [M]. Then P sends to Po the ciphertexts {CT g,y = Enc(ůs,,))
4: Po partitions the matrix X into block matrices Xx.3 E Znz 2muxn for [m''] and [n']. Po encodes the matrices to polynomials $a.9 H (X..9).
5: Po uniformly at random samples plaintext polynomials far for a E [m'] and y E [K]] Po decodes these polynomials to random mask R E Znyx according to Theorem 4.14.
6: On receiving the ciphertexts {CT from Pt. Po operates CT [sein M CCT pn)
fa,y for a E [m'] and ? [K]. Then Po sends P1 the ciphertexts {
7: Po outputs R mod 2 as the share (Z)o-
8: On receiving the ciphertexts {CT a.7J from Po. Pi computes (fan)) = Dec(CT,, a.y) that are decoded to (Z), using the method in Theorem 4.1.
4.2 Protocols for Non-linear Functions
Our non-linear protocols rely on several underlying protocols from the state-of-the-art works [13, 17]. In Figure 3, we enumerate the inputs and outputs of these protocols and then use them black box (See Appendix A.3.2 for details). On this basis, we provide efficient protocols for Softmax, GELU, and LayerNorm with specific optimizations. Additional details and security analysis are shown in Appendices A.3.3 and A.3.4.
Multiply Ilxulor
Compare CMMP
NegExp nxpp
RecipSqrt Sgrt
Recip leRepip
Input: Py: (x),. b),,
Imput: (x))
Input: Po: (x)n
Input:
Input:
Po: (x)
Pa: (x))
Output
Ouiput
Output:
Output:
Output
(2)
Pg: (z)8
Po: (z)o
Po: (2)
Py: (x),
st.z= xy
s.t.z 1(x> 0]
s.t. e5,x<0
s.t. z X>E
s.t. 1/x
Figure 3: Underlying protocols from [13, 17]
4.2.1 Softmax
To evaluate attention layers, we need an efficient protocol to compute Softmax on secret-shared values. In particular, for a vector x E Zd., the Softmax function denoted as Softmax; = eti "/Eeee e for i E [d]. The main challenge is to efficiently compute the underlying exponential function. Following the idea from [11, 30], we first normalize the input vector by x max,ejd Vi. which is always negative, and then invoke the existing exponential protocol II in Figure 3 that only evaluates exponentiation
negative inputs. A simple analysis shows softmax(x max,e[4] x;) is equal to softmax(x) We evaluate max using tree-reduee rreerredution protocol, denoted by Im Specifically, we arrange the vector X E Zd, into 2-ary tree with the depth of log d, and evaluate the tree in top-down fashion [31]. In each comparison of two secret-shared elements 2; and Zj, we reduce to the invocations of HCMP and I i.c., max(z; (2,, xz) = I (x; 2y, IICMP(;; xz)) + Zz- With the above insight, the Softmax protocol is detailed Algorithm 2.
SIRNN [17] also provides a solution for the generic exponential protocol by extending NnExp- The idea is that the exponential of x equals nExp(- x 2 0, and nExp(x) otherwise, where nExp is the
"This operation aims avoid extra information leakage from the output polynomial's coefficients. The details refer to [14] and their implementations.

---

same as the exponential function except for negative inputs. Compared to our solution, realizing the softmax function with this generic exponential protocol additionally requires calls to the reciprocal protocol Recip and multiplication protocol MUTT Besides, compared with the generic library MP-SPDZ. [16] that provides native support for exponentiation, our protocols achieve orders of magnitude improvement as shown in Section 5.2.
Algorithm/ 2 Secure Softmax Protocol
Input: Po. P1 hold (x)o Li (x)1 Zz respectively.
Output: Po. Pi get (y)o E Egj. (y) Egg respectively, where y Softmax(x)..
1: Po. Pi invoke 1m (x) to compute (max(x)). where max(x) ) = max, id] Zzz
2: For i E [d], Po. P1 invoke Napx on input (T+), and learn (e where I; = I; max(x).
3: Po. P1 invoke Reei with inputs (ieed edt) and learn (1/ //iE(a e²1).
4: For i E [d], Po. P1 invoke Motlor with inputs (1/Eiejo) eh)) and (ea*), and set outputs as (wi)-
4.2.2 GELU
Rather than crypto-friendly ReI.U [13], Transformer- based models utilize GELU activations [15), which can be represented as GELU(z) 0.5z (1 Tanh 22/T T (z + 0.044715x=))] The com- plete protocol is shown in Algorithm 3, in which we provide two insights to reduce the cost of GELU. First, we present an optimized protocol for the square of a secret-shared input (z). This relies on the observation: x² = (z) + (z) { + (z)((1)), where the first two terms can be locally computed by Po and P1. We only invoke OT protocols compute the cross term (z))) and the optimized overhead is half that of the multiplication protocol Muuor
Second, we further optimize the evaluation of Tanh(=) 2I+ We observe that the sign of is equal to the sign of Tanh(x), which allows us leverage the negative exponential protocol TTxp almost for free. At a high level, our Tanh protocol first learns the sign of the input x, and then evaluates Tanh on the negative input with the constraint | = |z]. Finally, the protocol generates the real output Tanh(z)) that equals Tanh (T) if x S O, and - Tanh( otherwise. Our leaner Tanh protocol is given in Algorithm 5 of Appendix 1.3.1. SIRNN [17] recently proposed the most efficient Tanh protocol with the insight of (z) = 2Sigmoid (2x) - 1, where Sigmoid(z) = Ite However, the evaluation of Sigmoid uses the same idea as the general exponential protocol, as shown in Section 1.2. Compared with the protocol in SIRNN, our recipe for Tanh saves one invocation of the reciprocal protocol RRcip and multiplication protocol IlMulor
Algorithm 3 Secure GELU Protocol
Input: Po. P1 hold (z)o E Lge, (2)1 E Zzz, respectively.
Output: Po. Pi get (y)o E Zoz, (y)1 Ezz, respectively, where y GELU(x)..
1: Po. Pj invoke MMattr with inputs (xr). and set their outputs as (2) = Fix(V2/x)((a)
((0.004411))())))
2: Po. Pi invoke Tanh with inputs (z), and set their outputs as (Tanh())).
3: Po. P1 invoke Muulor with inputs (Fix(0.5)a:) and (Fix(1) + Tanh(2)): (2)), and output (v)-
4.2.3 LayerNorm
For vector x E Zd., the LayerNorm function is denoted by LayerNorm, >(4; - p)/a / for E [d], where μ = -xejd x:/d and a ~ Eieig n²²². In contrast to batch normalization (BN) in CNNs that is evaluated for free [14], LayerNorm requires multiplication and reciprocal square root operations. In our implementation, we observe that the multiplication dominates the overhead of LayerNorm. To address this issue, adopt the same idea in GELU to compute the square of x; - μ. which saves half of the communication and computation costs.
Besides, inspired by the optimization in BN [14], we apply LayerNorm merge technique to further reduce the overhead. Specifically, we make the observation that the weights 7 and (3 of the LayerNorm layer are already known by Po. As result, Po first multiplies the scale factor 7 with the weights of
7

---

Table 1: Comparing the runtime (sec) and communication (MB) costs our matrix multiplication and non-linear protocols with SOTA
Matrix Multiplication
Non-Linear Protocols
Dims=(32, 8, 16)
(128, 64, 128)
(128, 768, 768)
Softmax
LayerNorm
Methods
Time
Comm.
Time
Comm.
Time
Comm.
Time
Comm.
Time
Comm.
Time
Comm.
0.006
0.066
1.74
1.71
15.45
Ours
4.78
206.265
102.435
0.30
10.07
0.16
2.79
0.77
14.78
6.10
134.37
7.95
347.71
4,16
184,42
0,38
14.07
Cheetah
SIRNN
(26x)
(11x)
(8x)
(3x)
(8x)
(1.7%)
(1.7x)
(1.8%
(1.8%)
(1.3x)
(1.4x)
0.04
1.34
1.59
70.08
110.33
4920.08
297.75
172,857
202.75
101,642
15.34
7,908.69
SIRNN
MP-SPDZ
(6x)
(12x)
(23x)
(40x)
(64x)
(318x)
(62x)
(837x)
(86x)
(992x)
(51x)
(785k)
the next linear layer. After invoking the linear layer protocol on the scaled weights. Po adds the shift weight a to his additive share. The optimized protocol for LayerNorm presented in Algorithm
Algorithm 4 Secure LayerNorm Protocol
Input: Po. P1 hold (x)o Ziges (x)1 E Z respectively.
Output: Po. Pi get (y)u. (y)1, respectively, where = LayerNorm(x)).
1: For E [d], Po. P1 invoke Mo to compute ((rr - ))2)) where μ = Σie id] sz/d.
2: Po. Pi invoke Ilss with inputs (Eeea (z; n)²} to learn output (t))
3:
For i E [4]. Po. P1 invoke MoMtlr with inputs (+) and (xx - p)), and set outputs as (y)).
5
Evaluation
5.1 Experimental Setup
Implementation. Iron is built top of the SEAL library [32] and the EMP toolkit [33] in C++. We also use the EzPC framework [34]. This framework compiles a high-level TensorFlow code to secure computation protocols, which then executed by our designed cryptographic backends. Like [17], we simulate a LAN network setting, where the bandwidth is 377 MBps and the echo latency is 0.8ms. All the following experiments are performed on AWS c5. c5.9xlarge instances with Intel Xeon 8000 series CPUs at 3.6GHz.
Datasets and Models. We evaluate Iron four NLP models from [35, 36]: BERT-Tiny, BERT- Medium, BERT-Base and BERT-Large. These models are parameterized by three hyper-parameters: the number of blocks, the dimension of representations and the number of input tokens (refer to Appendix A.4.1 for the hyper-parameters parameters of these models). We train the models for four NLP tasks over the datasets of the Stanford Sentiment Treebank (SST-2), the Microsoft Research Paraphrase Corpus (MRPC), the Multi-Genre Natural Language Inference Corpus (MNLJ) and the Stanford Question Answering Dataset (QNLI) from GLUE benchmarks [18].
5.2 Microbenchmark Evaluation
Matrix multiplication. the left part of Table 1, we compare the performance of the proposed matrix multiplication protocol with the ofate-of-the-art counterparts of Cheetah [14] and SIRNN [17]. For fairness, we follow Cheetah for the parameter setup of homomorphic encryption. Compared with Cheetah, our rumtime is 3 ~ 26x faster, and our communication cost is 8 ~ 25x lower, depending on the input size. Notably, for small-size matrices (c.g., ones with 32 x 8 and 8 x 16 elements), protocol only requires 0.11 MB communication and 6 ms runtime, while Cheetah achieves the computation with 2.79 MB and 160 ms. The reason is that our protocol encrypts the whole matrix into single ciphertext, while Cheetah could only encrypt each row into ciphertext, totally m ciphertexts (m is the number of rows of the output matrix). Moreover, compared with SIRNN that implements the most efficient OT-based matrix multiplication protocol, protocol incurs up two orders of magnitude less communication and an order of magnitude less runtime.
Non-linear functions. The right part of Table 1 shows the comparison of our Softmax, GELU and LayerNorm protocols with the generic MP-SPDZ framework [16) and the state-of-the-arr SIRNN library [17] for math functions. It is worth noting that implement some functions that these
8

---

10
105
Ourwark SRW
Our nert
x1111
Ourwor
100
Ours
Rurtime (min)
Compr untattttonnnn
100
SPAN
2100
Runtime (min)
304
NP-57D2
V57
Mn-SPoz
101
1or
20'
Soz
10
H642
x3.47
201
10'
10:
Model archibectuess of BEAT
Nodel rcchiteturns of BERT
Model architecturss
BERT
Model architectures BEAT
Figure 4: End-to-end comparisons with SIRNN and MP-SPDZ
frameworks did not provide before. In particular, GELU and LayerNorm are implemented in MP- SPDZ by calling its built-in functions for tanh, square root, and reciprocal, while we add Softmax and GELU protocols in SIRNN utilizing sigmoid and exponent functions. As shown in this table, our protocols are orders of magnitude better than those of MP-SPDZ, in terms of both runtime and communication. In particular, for the communication cost, our protocols achieve 785 x improvement. Moreover, while our protocols are built upon the underlying protocols from SIRNN, we also achieve 1.3 1.8x lower runtime and 1.4 1.8x lower communication due to our customized optimizations. Such an improvement gives us significant savings in communication complexity, since the non-linear layer protocols dominate the overall overhead, as described below.
5.3 End-to-end Inference Evaluation
Comparison with prior methods. In the left part of Figure 4, we evaluate our protocols on 4 BERT models compared with SIRNN [17]. is observed that our rumtime 3.3 11.83x faster than that of SIRNN, and our communication cost is 3.47 ~ 14.11x x lower over four models. Moreover, our performance gains scale up as the model size grows. This because our protocols achieve better amortized overhead when processing large-scale evaluations. We also compare the end-to-end private inference with MP- in Figure The results show that our protocols are orders of magnitude better, in terms of both time and communication costs. This is because specialized protocols are more communication efficient than generic alternatives, which is also observed by SIRNN.
Performance breakdown. In Figure 5, we present the runtime and communication breakdown of Iron on four BERT models. For clarity, we just report the result of one encoder. Recall that our private inference can be divided into linear protocols and non non-linear protocols. For linear protocols, we observe that the model size increases, the proportion of communication overhead remains approximately constant, accounting for 12 ~ 13%. This shows that our compact ciphertext encoding method effectively reduces the size of communication. For non-linear functions, as the model size increases, the proportion of computation overhead gradually decreases, from 84% in BERT-Tiny to 76% in BERT-Large. The savings come from amortizing the communication and computation costs by packing data from large tensors. Despite such advantages, the main bottleneck of our work the communication overhead of non-linear layers. However, it is an open problem to solve the communication issue while maintaining the model accuracy.
Accuracy comparison with plaintext. In the left part of Figure 6, we show the accuacy of plaintext (float-point) and Iron (fixed-point) on the BERT-Tiny model. We observe that the accuracy achieved by Iron matches the accuracy of the plaintext TensorFlow code. Specifically, the accuracy loss does not exceed 0.3% over all datasets, and surprisingly. Iron exceeds the plaintext baseline MNLI by 0.85%. Similar results also appear in private CNN inference [13]. Such accuracy advantages experimentally validate that our protocols are numerically precise. Moreover, in the right part of Figure 6, we also compare the accuracy with the plaintext baseline, as the fractional scale varies on MRPC. We observe that Iron with the scale of 12 exactly matches the accuracy of the plaintext model. The accuracy loss lower than 1 1% when the scale 6. This conclusion is in line with the prior work [13], namely that neural networks tolerate stochastic fault behavior [37].
6 Discussion
We discuss possible solutions to further improve the efficiency of private inference on Transformers. They are compatible with our proposed protocols and hence can be directly integrated into our framework.

---

Comprrer aalioo (M6B)
Unear
72
Hurtime (sec)
155
Ventinew
Non-lnner
-3
1.00
Accurs
art
20%
12%
12%
Marttst
Iiy
Medium Base Large
Thy
Hedlum Base
Large
60
MNU
MBurC QNU
5512
10 Scale
Model anchitectures of UERT
Model architectures BERT
Datasets
Figure 5: Performance breakdown on BERT.
Figure 6: Accuracy comparison with plaintext
Pushing expensive operations into an offline phase. Our framework is readily extended to the pre- processing model, including an offline client-input independent phase and an online input-dependent phase. This paradigm has been instantiated in private CNN inference [38, 8], and the results show that about 99% of the cryptographic overhead can be moved to the offline phase. We briefly outline how to build our protocols with this paradigm. For linear operations, we can generate in advance Beaver's s triple in the matrix form [22, 38] using our matrix multiplication protocol. In the online phase, these triples are consumed, and it additionally requires prograyptog operations and plaintext communication. For non- lineir layers, almost all operations crucially rely on the OT protocols, which can be pre- pre-generated in the offline phase [39, 14]. Like linear protocols, the online phase is cheap.
in private inference. Iron works with uniform bitwidth, which is required to be large enough to accommodate all intermediate values. While effectively avoiding integer overflows, our protocols may cause intensive communication, since the performance of non- non-linear operators depends critically on bitwidths. Taking inspiration from [17, 40], one possible optimization employ non-uniform (mixed) bitwidths, operate in low bitwidths and switch high bitwidths only when necessary. To this end, the mixed-precits precision model should be generated with proper compilers by using techiniques like quantization [41, 42]. With our protocols can be integrated seamlessly.
Using
fixed-point models,
Applying orthogonal model optimizations. We can improve performance by simplifying the model architecture such as model pruning and advanced neural architecture search [43, 44J, which have been commonly adopted in private CNN inference [12, 45, 37]. Like prior works [12, 45, 37] that find and tailor models to the requirements of private inference, we can define a proper search space with the goal of decreasing the overhead, e.g., substituting costly Softmax and GELU functions, or reducing the matrices' dimension in matrix multiplication. As mentioned in Section I, the concurrent work, THE-X [19], has explored replacing complex math functions with HE-friendly alternatives while achieving comparable accuracy.
Conclusion
We propose Iron, an efficient cryptographic framework for private Tramsformer inference. Specifi- cally, we carefully design new encoding method for optimizing homomorphic encryption- basedd matrix multiplication. Further, we devise several communication-effiiin Softmax, LayerNorm and GELU by integrating advanced secret-sharing primitives. Experimental results show that Iron outperforms prior art by up to one order of magnitude in terms of computation and communication overheads. We believe that our novel protocols would help advance the practical instantiations of private Transformer inference.
non-linear protocols like
Acknowledgment
The authors would like to thank the anonymous reviewers for their insightful comments. This work supported by the Key-Area Research and Development Program of Guangdong Province under National Natural Science Foundation of China under Grants 62020106013, 61972454, and 61 61200051, Sichuan Science and Technology Program under Grants 2020JDTD0007 and 2020YFG0298, the Fundamental Research Funds for Chinese Central Universities under Grant ZYGX20207BB277 and Singapore Ministry of Education (MOF) AcRF Tier 2 M-P-22PP20121- 0006.
Grant 2020B0101360000
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, 2017.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NeurlPS, 2018.
[3] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of ICLR,, 2020.
[5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings ICCV, pages 10012-10022. 2021.
[6] Clarifai general image embedding model. URL htttp:://www.caaiiaai..co general - immaall-mmme---e embeddinged....
[7] Openai URL hta.
[8] Ryan Lehmkuhl, Pratyush Mishra, Akshayaram Srinivasan, and Raluca Ada Popa. Muse: Secure inference resilient to malicious clients. Proceedings of USENIX Security, pages 2201-2218, 2021.
[9] Yupei Liu, Jinyuan Jia, Hongbin Liu, and Neil Zhenqiang Gong. Stolenencoder: Stealing pre-trained encoders. arXiv:2201.05880 2022.
[10] Ran Gilad-Bachrne Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of ICML., pages 201-210, 2016.
[11] Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-party computation meets machine learning. In Proceedings of NeurlPS, 2021.
[12] Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg. Cryptonas: Private inference relu budget. In Proceedings of NeurlPS, pages 16961-16971, 2020.
[13] Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow2: Practical 2-party secure inference. In Proceedings of ACM CCS, pages 325- 045-342,220 2020.
[14] Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding. Cheetah: Lean and fast secure two-party deep neural network inference. In Proceedings of USENIX Security, 2022.
[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 08415, 2016.
[16] Marcel Keller. Mp-spdz: A versatile framework for multi-party computation. In Proceedings of ACM CCS, pages 1575-1590, 2020.
[17] Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nishanth Chandran, and Aseem Rastogi. Sirnn: A math library for secure mnn inference. In Proceedings of IEEE S&P, pages 1003-1020, 2021.
[18] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR, pages 535-548, 2018.

---

[19] Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao. Daxin Jiang, Haoyi Zhou, transformer inference with homomorphic
Jianxin Li, and Furu Wei. The-x: encryption. In Findings of ACI, pages 35 1510-3520, 2022.
[20] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Zhang. Deep learning with differential privacy. Proceedings of ACM CCS, pages 308-318, 2016.
[21] Ronald Cramer, Ivan Bjerre Damgärd, al. Secure multiparty computation. Cambridge University Press, 2015.
[22] Daniel Demmler, Thomas Schneider, and Michael Zohner. Aby-a framework for efficient mixed-protocol secure two-party computation. In Proceedings of NDSS, 2015.
[23] Taher ElGamal. public key cryptosystem and signature scheme based on discrete logarithms. IEEE transactions on information theory, 31(4):469-472, 1985.
[24] Pascal Paillier. Public-key cryptosystems based on composite degree residuosity classes. In Proceedings of Eurocrypt, pages 223-238, 1999.
[25] Zvika Brakerski. Fully homomorphic encryption without modulus switching from classical gapsvp. In Proceedings of Crypto, pages 868-886, 2012.
[26] Junfeng Fan and Frederik Vercauteren. Somewhat practical fully homomorphic encryption. Cryptology ePrint Archive, 2012.
[27] Vladimir Kolesnikov and Ranjit Kumaresan. Improved ot extension for transferring short secrets. In Proceedings of Crypto, pages 54-70, 2013.
[28] Gilad Asharov, Yehuda Lindell, Thomas Schneider, and Michael Zohner. More efficient oblivious transfer and extensions for faster secure computation. In Proceedings of ACM CCS, pages 535-548, 2013.
[29] Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. Gazelle: low latency framework for secure neural network inference. In Proceedings of USENIX Security, pages 1651-1669, 2018
[30] Sijun Tan, Brian Knott, Yuan Tian, and David J Wu. Cryptgpu: Fast learning the gpu. In Proceedings of IEEE S&P, 2021.
machine
[31] Théo Ryffel, Pierre Tholoniat, David Pointcheval, and Francis Bach. Ariann: Low-interaction privacy-preserving deep learning via function secret sharing. In Proceedings of PETS, pages 291-316, 2022.
[32] Seal. URL httptg/s//ggttuu....///crrrt///t
[33]
emp-toolkit. URL URL.ttpp://gttaab..ccmmmmppppppp
[34] Ezpc. URL UKK.Atttee//gtaaab.cca/aaarrrr/r
[35] Bert. URL UBL.tttge//ggtab...ooogollleeeee
[36] Iulia Turc, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXi::1908.0800 2019.
[37] Zahra Ghodsi, Nandan Kumar Jha, Brandon Reagen, and Siddharth Garg. Circa: Stochastic relus for private deep learning. In Proceedings of NeurlPS, 2021.
[38] Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptographic inference service for neural networks. In Proceedings of USENIX Security, pages 2522, 2020.
[39] M Sadegh Riazi, Christian Weinert, Oleksandr Tkachenko, Ebrahim M Songhori, Thomas Schneider, and Farinaz Koushanfar. Chameleon: hybrid secure computation framework for machine learning applications. In Proceedings of AsiaCCS, pages 707-721, 2018. 2018.
12

---

[40] Siam Umar Hussain, Mojan Javaheripi. Mohammad Samragh, and Farinaz Koushanfar. Coinn: Crypto/ml codesign for oblivious inference via neural networks. In Proceedings of ACM CCS, pages 7266-3281, 2021.
[41] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer, I-bert: Integer-only bert quantization. In Proceedings of ICML. pages 5506-5518, 2021.
[42] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of AAAI, pages 8815-8821, 2020.
[43] François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. In Proceedings of EMNLP, 2021.
[44] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Proceedings of NeurlPS, 2019.
[45] Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen. Deepreduce: Relu reduction for fast private inference. In Proceedings of ICML, pages 4839-4849, 2021.
[46] Payman Mohassel and Yupeng Zhang. Secureml: system for scalable machine learning. In Proceedings of IEEE S&P. pages 19-38, 2017.
[47] Ran Canetti. Universally composable security: new paradigm for cryptographic protocols. In Proceedings of FOCS, pages 136-145, 2001.
[48] Robert E Goldschmidt. Applications of division by convergence. PhD) thesis, MIT, 1964.
[49] Xiaoqian Jiang, Miran Kim, Kristin Lauter, and Yongsoo Song, Secure outsourced matrix computation and application to neural networks. Proceedings ACM CCS, pages 1209-1222, 2018.
[50] Fabian Boemer, Anamaria Costache, Rosario Cammarota, and Casimir Wierzynski. ngraph-he2: A high-throughput framework for neural network inference on encrypted data. In Proceedings the 7th ACM Workshop Encrypted Computing & Applied Homomorphic Cryptography, pages 45-56, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uyXk9Ah6Qlk", "reviewer": "Reviewer_MFak", "summary": "This paper proposes a new secure cryptographic protocol for standard components in Transformers, such as matrix multiplications, Softmax, GeLU, and LayerNorm. The authors' method is the first implementation of secure inference on Transformer for 2PC setup. Furthermore, their new customized homomorphic encryptions allow faster and more efficient matrix multiplications over existing state-of-the-art methods. ", "questions": "1. In line 198, what are these symbols mean? Typo?\n2. Compared to the garbled circuit used in the ReLU protocol from CNN secured inference, is secure GELU protocol par regarding run-time and communication cost to the garbled circuit on ReLU? ", "limitations": "The authors provide detailed comments on their limitations and possible future directions further to accelerate the private inference on the transformer models. ", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Iron is the first private inference framework for the Transformer in secure 2-party computation technique.\n2. Iron's custom homomorphic encryptions allow fast linear and non-linear operations by a significant margin compared to the existing state-of-the-art methods such as Cheetah and SIRNN.\n3. Provides the protocols for essential components in the Transformer (e.g., Softmax, GeLU) instead of replacing them with easier operations like ReLU. \n\nWeaknesses\n1. The authors' submission does not include the official implementation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pJzkU1MePJb", "reviewer": "Reviewer_6r55", "summary": "The paper proposes to use homomorphic encryption for Transformer model inference.  Moreover, the proposed method improves the speed of matrix operations. The experiments show that the inference results are numerically comparable.", "questions": "- (L267) Does the proposed method only support CPUs? There is no experiment conducted on GPUs.\n\n- I did not find exact numbers of accuracy in the main content of the submission.\n\n- Only NLP tasks with short input texts are evaluated. How about the performance and speed for longer inputs?\n\n- How about the results on other tasks, such as vision problems?\n\n- The speed is not directly compared with standard Transformer inference.\n\n- How about the memory cost?\n\n- What are the costs of private inference?", "limitations": "The paper can explicitly include discussions about the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Speedup the inference speed of homomorphic encryption for Transformer models.\n\n- Because the Transformer network is hot now across research areas, how to host the models using private inference is worth exploring.\n\n- The submission will also release the code?\n\n==========\n\nWeaknesses:\n\n- The abstract claims \"the first Transformer-based private inference framework\". However, there was one paper https://aclanthology.org/2022.findings-acl.277/ that was published in ACL-22. The ACL-22 paper works on the same problem and also uses homomorphic encryption as the solution. The claim can be removed to make the claim more precise.\n\n- (L267) Does the proposed method only support CPUs? There is no experiment conducted on GPUs. It makes the work less practical.\n\n- (abstract) \"which preserve the model accuracy of plaintext\" The claim is not well supported by experiments. I did not find exact numbers of accuracy in the main content of the submission.\n\n- Only NLP tasks with short input texts are evaluated. More applications can be demonstrated, especially for longer texts. \n\n- The speed is not directly compared with standard Transformer inference. It seems that the inference is quite slow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d1YE6P2iw-N", "reviewer": "Reviewer_QBqi", "summary": "The paper proposes a solution based on LHE+Secret Sharing for private inference of transformer models. The key challenge is in adapting different transformer layers, namely large matrix-matrix multiplication, Softmax, layer normalization, and GELU non-linear function. \n\nThe main contribution of this paper is an encoding scheme that is designed to make matrix-matrix multiplication in LHE more efficient than existing schemes, and a protocol for performing secure matrix multiplication. There are minor contributions in improving the efficiency of evaluating Softmax, layer normalization, and GELU.\n\nExperiments are conducted on 4 NLP models, BERT-Tiny, BERTMedium, BERT-Base, and BERT-Large. Comparisons are performed against Cheetah, a baseline method in terms of runtime (latency) and communication efficiency. ", "questions": "Overall I am positive about the paper, and will revise the score based on the author’s rebuttal.\n\n- How does the proposed matrix-matrix multiplication scheme compare quantitatively and quantitatively to [1] above. And how do they all scale with matrix sizes since this is a key aspect of variation in transformers?\n\n- Line 323-324 states, \"For linear operations, we can generate in advance Beaver’s triple in the matrix form [21, 36] using our matrix multiplication protocol.\" Did you mean non-linear operations, unless I am missing something I am unsure why Beaver triples are needed for matrix multiplication?", "limitations": "There is no explicit discussion on the limitations of the current work. There is a separate section for discussing some orthogonal directions to augment the current work, in terms of mixed-bit widths and applying model optimizations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n+ The paper is very well written in terms of providing the right amount of background, and clarity of descriptions. I enjoyed reading the paper.\n\n+ Extending the concept of private inference for transformer-based models. Given the growing application of transformers, it is critical to design private inference-friendly versions of these models.\n\n+  The speedup over SIRNN is notable.\n\n+ The data encoding scheme and associated secure matrix multiplication protocol is effective.\n\n**Weaknesses:**\n- The primary contribution of the paper is a new data encoding scheme that is tailored for improving the efficiency of matrix-matrix multiplications. However, the paper does not compare (qualitatively or quantitatively) a key existing solution [1] for the same. The current paper only compares against the method in Cheetah, but it is not clear to this reviewer how it compares to [1]. I believe there is no single encoding scheme that is optimal for all matrix sizes.\n\n- The paper appears to be an incremental improvement over CHEETAH. The matrix multiplication is certainly better, but other protocols are minor adaptations of existing practice (e.g., subtracting max before computing softmax is very common practice even in applications that focus only on plaintext) and those implemented in CHEETAH.\n\n[1] Secure Outsourced Matrix Computation And Application to Neural Networks.\n\nThis is just a comment to make the authors aware of concurrent work [2] along the direction of adapting transformers for private inference with homomorphic encryption.\n[2] THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption, ACL 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept