Circa: Stochastic ReLUs for Private Deep Learning

1 Introduction . Today , Machine Learning as a Service ( MLaaS ) provides high-quality user experiences but comes at the cost of privacy—clients either share their personal data with the server or the server must disclose its model to the clients . Ideally , both the client and server would preserve the privacy of their inputs and model without sacrificing quality . A recent and growing body of work has focused on designing and optimizing cryptographic protocols for private inference ( PI ) . With PI , MLaaS computations are performed obliviously ; without the server seeing the client ’ s data nor the client learning the server ’ s model . PI protocols are built using cryptographic primitives including homomorphic encryption ( HE ) , Secret Sharing ( SS ) , and secure multiparty computation ( MPC ) . The challenge is that all known protocols for PI incur impractically high overheads , rendering them unusable . Existing PI frameworks [ 1 , 2 , 3 ] are based on hybrid protocols , where different cryptographic techniques are used to evaluate different network layers . Delphi [ 3 ] , a leading solution based on Gazelle [ 2 ] , uses additive secret sharing for convolution and fully-connected layers . Secret sharing supports fast addition and multiplication by moving large parts of the computation to an offline phase [ 4 ] . Thus , convolutions can be computed at near plaintext speed . Non-linear functions , notably ReLU , can not enjoy the same speedups . Most protocols ( including Delphi , Gazelle , and MiniONN [ 1 ] ) use Yao ’ s Garbled Circuits ( GC ) [ 5 ] to process ReLUs . GCs allow two parties to collaboratively and privately compute arbitrary Boolean functions . At a high-level , GCs represent functions as encrypted two-input truth tables . This means that computing a function with GCs requires the function be decomposed into a circuit of binary gates that processes inputs in a bit-wise fashion . Thus , evaluating ReLUs privately is extremely expensive , to the point that PI inference runtime is dominated by ReLUs [ 3 , 6 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Therefore , reducing ReLU cost is critical to realizing practical PI . There are two general approaches for minimizing the cost of ReLUs : designing new architectures that limit ReLU counts , and optimizing the cost per ReLU . Prior work has almost exclusively focused on minimizing ReLU counts . Work along this line includes replacing or approximating ReLUs with quadratics or polynomials ( e.g. , CryptoNets [ 7 ] , Delphi [ 3 ] , SAFENet [ 8 ] ) , designing new networks architectures to maximize accuracy per ReLU ( e.g. , CryptoNAS [ 6 ] ) , and more aggressive techniques that simply remove ReLUs from the network ( e.g. , DeepReDuce [ 9 ] ) . Relatively little attention has been given to minimizing the cost of the ReLU operation itself . In this paper we propose Circa1 , a novel method to reduce ReLU cost based on a new stochastic ReLU function . First , we refactor the ReLU as a sign computation followed by a multiplication , allowing us to push the multiplication from GC to SS , leaving only a sign computation in the GC . Next , we approximate the sign computation to further reduce GC cost . This approximation is not free ; it results in stochastic sign evaluation where the results are sometimes incorrect ( we call incorrect computations faults to differentiate from inference error/accuracy ) . Finally , we show that stochastic sign can be optimized even further by truncating its inputs ; truncation introduces new faults , but only for small positive or negative values . Our key insight is that deep networks are highly resilient to stochastic ReLU fault behavior , which provides significant opportunity for runtime benefit . The stochastic ReLU introduces two types of faults . First , the sign of a ReLU can be incorrectly computed with probability proportional to the magnitude of the input ( this probability is the ratio of the input magnitude over field prime . ) In practice we find this rarely occurs as most ReLU inputs are very small ( especially compared to the prime ) and thus the impact on accuracy is negligible . Second , truncation can cause either small positive or small negative values to fault . Circa allows users to choose between these two probabilistic fault modes . In NegPass , small negative numbers experience a fault with some probability and are incorrectly passed through the ReLU . Alternatively , PosZero incorrectly resolves small positive inputs to zero . Empirically , we find deep networks to be highly resilient against such faults , tolerating more than 10 % fault rate without sacrificing accuracy . Compared to Delphi , Circa-optimized networks run up to 3× times faster . We further show that Circa is orthogonal to the current best practice for ReLU count reduction [ 9 ] . When combined , we observe an additional 1.8× speedup . 2 Background . 2.1 Private Inference . We consider a client-server model where the client sends their input to the server for inference using the server ’ s model . The client and the server wish to keep both the input and model private during the inference computation . Our threat model is the same as prior work on private inference [ 1 , 2 , 3 ] . More specifically , we operate in a two-party setting ( i.e. , client and server ) where participants are honest-but-curious—they follow the protocol truthfully but may try to infer information about the other party ’ s input/model during execution . We take the Delphi protocol [ 3 ] as a baseline and implement our optimizations over it . Delphi uses HE and SS for linear layers , where computationally expensive HE operations are performed in an offline phase and online computations only require lightweight SS operations . For non-linear activations , Delphi uses ReLUs , evaluated using GCs , and polynomial activations ( x2 ) , evaluated using Beaver multiplication triples [ 4 ] . While Circa does not use polynomial activations , it uses Beaver triples in its stochastic ReLU implementation . Next , we introduce the necessary cryptographic primitives and provide an overview of the Delphi protocol . 2.2 Cryptographic Primitives . Finite Fields . The cryptographic primitives described subsequently operate on values in a finite field of integer modulo a prime p , Fp , i.e. , the set { 0 , 1 , . . . , p − 1 } . In practice , positive values will be represented with integers in range [ 0 , p−12 ) , and negative values will be integers in range [ p−1 2 , p ) . Additive Secret Sharing . Additive secret shares of a value x can be created for two parties by randomly sampling a value r and setting shares as 〈x〉1 = r and 〈x〉2 = x − r. The secret can be 1In Latin , “ circa '' means approximately . reconstructed by adding the shares x = 〈x〉1 + 〈x〉2 . Performing additions over two shared values is straightforward in this scheme , each party simply adds their respective shares to obtain an additive sharing of the result . Beaver Multiplication Triples . This protocol [ 4 ] is used to perform multiplications over two secret shared values . A set of multiplication triples are generated offline from random values a and b , such that the first party receives 〈a〉1 , 〈b〉1 , 〈ab〉1 , and the second party receives 〈a〉2 , 〈b〉2 , 〈ab〉2 . In the online phase x and y are secret shared among parties such that the first party holds 〈x〉1 , 〈y〉1 and the second party holds 〈x〉2 , 〈y〉2 . To perform multiplication they consume a set of triples generated offline and at the end of the protocol the first party obtains 〈xy〉1 and the second party obtains 〈xy〉2 . Homomorphic Encryption . HE [ 10 ] is a type of encryption that enables computation directly on encrypted data . Assuming a public key kpub and corresponding secret key ksec , an encryption function operates on a plaintext message to create a ciphertext c = E ( m , kpub ) , and a decryption function obtains the message from the ciphertext m = D ( c , ksec ) . An operation is homomorphic if for messages m1 , m2 we have a function ? such that decrypting the ciphertext E ( m1 , kpub ) ? E ( m2 , kpub ) , which we also write as HE ( m1 m2 ) , gives m1 m2 . Garbled Circuits . GCs [ 5 ] enable two parties to collaboratively compute a Boolean function on their private inputs . The function is first represented as a Boolean circuit C. One party ( the garbler ) encodes the circuit using procedure C̃ ← Garble ( C ) and sends it to the second party ( the evaluator ) . The evaluator also obtains labels of the inputs and is able to evaluate the circuit using procedure Eval ( C̃ ) without learning intermediate values . Finally , evaluator will share the output labels with the garbler , and both parties obtain the output in plaintext . The cost of GC is largely dependent on the size of the Boolean circuit being computed . 2.3 Delphi Protocol . We now briefly describe the linear and non-linear components of the Delphi protocol , depicted also in Figure 1 . For simplicity , we will focus on the ith layer , with input yi and output yi+1 . The layer performs linear computation xi = Wi.yi ( here Wi are the server ’ s weights ) followed by a non-linear activation yi+1 = ReLU ( xi ) . The protocol consists of an input independent offline phase , and an input dependent online phase . As a starting point , the client samples random vectors ri ∈ Fnp in the offline phase . For input y1 , the client computes y1 − r1 and sends it to the server . For the ith layer , the client and the server start with secret shares of the layer input , 〈yi〉c = ri and 〈yi〉s = yi−ri , and use the Delphi protocol to obtain shares of the output , 〈yi+1〉c = ri+1 and 〈yi+1〉s = yi+1 − ri+1 . Linear computation : In the offline phase , the server samples random vectors si ∈ Fnp . Using HE on the server side , as shown in Figure 1 , the client then obtains Wi.ri − si without learning the server ’ s randomness or weights and without the server learning the client ’ s randomness . In the online phase , the server computes Wi . ( yi − ri ) + si , at which point the client and the server hold additive secret shares of xi = Wi.yi . Circa uses the same protocol for linear layers as Delphi . Non-linear computation : In this description we will focus on ReLU computations ( for completeness , Figure 1 also illustrates how quadratic activations are computed ) . During the offline phase , the server creates a Boolean circuit C for each ReLU in the network , garbles the circuit and sends it to the client along with labels corresponding to the client ’ s input . In the online phase , the linear layer protocol produces the server ’ s share of the ReLUs input . The server sends labels corresponding to its share to the client . The client then evaluates the GC , which outputs the server ’ s share , yi+1 − ri+1 , for the next linear layer . Online GC evaluation is the most expensive component of Delphi ’ s online PI latency . Circa focuses on reducing this cost .

Summary: This paper focuses on minimizing the computation cost of the ReLU operation. Prior work uses Garbled Circuits (GC) to process ReLUs, which is computationally expensive. This paper proposes a novel method to reduce computation by decomposing ReLU operation into a sign computation and multiplication. The multiplication can be implemented by Secret Sharing and the cost of sign computation through GC can be reduced by approximation. This paper also provides adequate empirical evaluations to show the proposed methods can reduce runtime.