Circa: Stochastic ReLUs for Private Deep Learning

1 Introduction . Today , Machine Learning as a Service ( MLaaS ) provides high-quality user experiences but comes at the cost of privacy—clients either share their personal data with the server or the server must disclose its model to the clients . Ideally , both the client and server would preserve the privacy of their inputs and model without sacrificing quality . A recent and growing body of work has focused on designing and optimizing cryptographic protocols for private inference ( PI ) . With PI , MLaaS computations are performed obliviously ; without the server seeing the client ’ s data nor the client learning the server ’ s model . PI protocols are built using cryptographic primitives including homomorphic encryption ( HE ) , Secret Sharing ( SS ) , and secure multiparty computation ( MPC ) . The challenge is that all known protocols for PI incur impractically high overheads , rendering them unusable . Existing PI frameworks [ 1 , 2 , 3 ] are based on hybrid protocols , where different cryptographic techniques are used to evaluate different network layers . Delphi [ 3 ] , a leading solution based on Gazelle [ 2 ] , uses additive secret sharing for convolution and fully-connected layers . Secret sharing supports fast addition and multiplication by moving large parts of the computation to an offline phase [ 4 ] . Thus , convolutions can be computed at near plaintext speed . Non-linear functions , notably ReLU , can not enjoy the same speedups . Most protocols ( including Delphi , Gazelle , and MiniONN [ 1 ] ) use Yao ’ s Garbled Circuits ( GC ) [ 5 ] to process ReLUs . GCs allow two parties to collaboratively and privately compute arbitrary Boolean functions . At a high-level , GCs represent functions as encrypted two-input truth tables . This means that computing a function with GCs requires the function be decomposed into a circuit of binary gates that processes inputs in a bit-wise fashion . Thus , evaluating ReLUs privately is extremely expensive , to the point that PI inference runtime is dominated by ReLUs [ 3 , 6 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Therefore , reducing ReLU cost is critical to realizing practical PI . There are two general approaches for minimizing the cost of ReLUs : designing new architectures that limit ReLU counts , and optimizing the cost per ReLU . Prior work has almost exclusively focused on minimizing ReLU counts . Work along this line includes replacing or approximating ReLUs with quadratics or polynomials ( e.g. , CryptoNets [ 7 ] , Delphi [ 3 ] , SAFENet [ 8 ] ) , designing new networks architectures to maximize accuracy per ReLU ( e.g. , CryptoNAS [ 6 ] ) , and more aggressive techniques that simply remove ReLUs from the network ( e.g. , DeepReDuce [ 9 ] ) . Relatively little attention has been given to minimizing the cost of the ReLU operation itself . In this paper we propose Circa1 , a novel method to reduce ReLU cost based on a new stochastic ReLU function . First , we refactor the ReLU as a sign computation followed by a multiplication , allowing us to push the multiplication from GC to SS , leaving only a sign computation in the GC . Next , we approximate the sign computation to further reduce GC cost . This approximation is not free ; it results in stochastic sign evaluation where the results are sometimes incorrect ( we call incorrect computations faults to differentiate from inference error/accuracy ) . Finally , we show that stochastic sign can be optimized even further by truncating its inputs ; truncation introduces new faults , but only for small positive or negative values . Our key insight is that deep networks are highly resilient to stochastic ReLU fault behavior , which provides significant opportunity for runtime benefit . The stochastic ReLU introduces two types of faults . First , the sign of a ReLU can be incorrectly computed with probability proportional to the magnitude of the input ( this probability is the ratio of the input magnitude over field prime . ) In practice we find this rarely occurs as most ReLU inputs are very small ( especially compared to the prime ) and thus the impact on accuracy is negligible . Second , truncation can cause either small positive or small negative values to fault . Circa allows users to choose between these two probabilistic fault modes . In NegPass , small negative numbers experience a fault with some probability and are incorrectly passed through the ReLU . Alternatively , PosZero incorrectly resolves small positive inputs to zero . Empirically , we find deep networks to be highly resilient against such faults , tolerating more than 10 % fault rate without sacrificing accuracy . Compared to Delphi , Circa-optimized networks run up to 3× times faster . We further show that Circa is orthogonal to the current best practice for ReLU count reduction [ 9 ] . When combined , we observe an additional 1.8× speedup . 2 Background . 2.1 Private Inference . We consider a client-server model where the client sends their input to the server for inference using the server ’ s model . The client and the server wish to keep both the input and model private during the inference computation . Our threat model is the same as prior work on private inference [ 1 , 2 , 3 ] . More specifically , we operate in a two-party setting ( i.e. , client and server ) where participants are honest-but-curious—they follow the protocol truthfully but may try to infer information about the other party ’ s input/model during execution . We take the Delphi protocol [ 3 ] as a baseline and implement our optimizations over it . Delphi uses HE and SS for linear layers , where computationally expensive HE operations are performed in an offline phase and online computations only require lightweight SS operations . For non-linear activations , Delphi uses ReLUs , evaluated using GCs , and polynomial activations ( x2 ) , evaluated using Beaver multiplication triples [ 4 ] . While Circa does not use polynomial activations , it uses Beaver triples in its stochastic ReLU implementation . Next , we introduce the necessary cryptographic primitives and provide an overview of the Delphi protocol . 2.2 Cryptographic Primitives . Finite Fields . The cryptographic primitives described subsequently operate on values in a finite field of integer modulo a prime p , Fp , i.e. , the set { 0 , 1 , . . . , p − 1 } . In practice , positive values will be represented with integers in range [ 0 , p−12 ) , and negative values will be integers in range [ p−1 2 , p ) . Additive Secret Sharing . Additive secret shares of a value x can be created for two parties by randomly sampling a value r and setting shares as 〈x〉1 = r and 〈x〉2 = x − r. The secret can be 1In Latin , “ circa '' means approximately . reconstructed by adding the shares x = 〈x〉1 + 〈x〉2 . Performing additions over two shared values is straightforward in this scheme , each party simply adds their respective shares to obtain an additive sharing of the result . Beaver Multiplication Triples . This protocol [ 4 ] is used to perform multiplications over two secret shared values . A set of multiplication triples are generated offline from random values a and b , such that the first party receives 〈a〉1 , 〈b〉1 , 〈ab〉1 , and the second party receives 〈a〉2 , 〈b〉2 , 〈ab〉2 . In the online phase x and y are secret shared among parties such that the first party holds 〈x〉1 , 〈y〉1 and the second party holds 〈x〉2 , 〈y〉2 . To perform multiplication they consume a set of triples generated offline and at the end of the protocol the first party obtains 〈xy〉1 and the second party obtains 〈xy〉2 . Homomorphic Encryption . HE [ 10 ] is a type of encryption that enables computation directly on encrypted data . Assuming a public key kpub and corresponding secret key ksec , an encryption function operates on a plaintext message to create a ciphertext c = E ( m , kpub ) , and a decryption function obtains the message from the ciphertext m = D ( c , ksec ) . An operation is homomorphic if for messages m1 , m2 we have a function ? such that decrypting the ciphertext E ( m1 , kpub ) ? E ( m2 , kpub ) , which we also write as HE ( m1 m2 ) , gives m1 m2 . Garbled Circuits . GCs [ 5 ] enable two parties to collaboratively compute a Boolean function on their private inputs . The function is first represented as a Boolean circuit C. One party ( the garbler ) encodes the circuit using procedure C̃ ← Garble ( C ) and sends it to the second party ( the evaluator ) . The evaluator also obtains labels of the inputs and is able to evaluate the circuit using procedure Eval ( C̃ ) without learning intermediate values . Finally , evaluator will share the output labels with the garbler , and both parties obtain the output in plaintext . The cost of GC is largely dependent on the size of the Boolean circuit being computed . 2.3 Delphi Protocol . We now briefly describe the linear and non-linear components of the Delphi protocol , depicted also in Figure 1 . For simplicity , we will focus on the ith layer , with input yi and output yi+1 . The layer performs linear computation xi = Wi.yi ( here Wi are the server ’ s weights ) followed by a non-linear activation yi+1 = ReLU ( xi ) . The protocol consists of an input independent offline phase , and an input dependent online phase . As a starting point , the client samples random vectors ri ∈ Fnp in the offline phase . For input y1 , the client computes y1 − r1 and sends it to the server . For the ith layer , the client and the server start with secret shares of the layer input , 〈yi〉c = ri and 〈yi〉s = yi−ri , and use the Delphi protocol to obtain shares of the output , 〈yi+1〉c = ri+1 and 〈yi+1〉s = yi+1 − ri+1 . Linear computation : In the offline phase , the server samples random vectors si ∈ Fnp . Using HE on the server side , as shown in Figure 1 , the client then obtains Wi.ri − si without learning the server ’ s randomness or weights and without the server learning the client ’ s randomness . In the online phase , the server computes Wi . ( yi − ri ) + si , at which point the client and the server hold additive secret shares of xi = Wi.yi . Circa uses the same protocol for linear layers as Delphi . Non-linear computation : In this description we will focus on ReLU computations ( for completeness , Figure 1 also illustrates how quadratic activations are computed ) . During the offline phase , the server creates a Boolean circuit C for each ReLU in the network , garbles the circuit and sends it to the client along with labels corresponding to the client ’ s input . In the online phase , the linear layer protocol produces the server ’ s share of the ReLUs input . The server sends labels corresponding to its share to the client . The client then evaluates the GC , which outputs the server ’ s share , yi+1 − ri+1 , for the next linear layer . Online GC evaluation is the most expensive component of Delphi ’ s online PI latency . Circa focuses on reducing this cost .

Summary: The paper proposes several techniques to improve the trade-off between accuracy and storage/runtime for ReLU operations in private inference. The proposed techniques include refactoring ReLUs as a sign function plus a cheap multiplication, and replacing the sign function with a truncated stochastic sign that has runtime and storage beneﬁts (by sacrificing accuracy). Experiments on several datasets and network architectures show that the method has a better trade-off between accuracy and storage/runtime than prior state-of-the-art.