Circa: Stochastic ReLUs for Private Deep Learning
Authors: Zahra Ghodsi
Nandan Kumar Jha
Brandon Reagen
Siddharth Garg
Venue: NeurIPS 2021 Poster (2021)
Keywords: privacy preserving machine learning
private inference
stochastic ReLU

Abstract: The simultaneous rise of machine learning as a service and concerns over user privacy have increasingly motivated the need for private inference (PI). While recent work demonstrates PI is possible using cryptographic primitives, the computational overheads render it impractical. State-of-art deep networks are inadequate in this context because the source of slowdown in PI stems from the ReLU operations whereas optimizations for plaintext inference focus on reducing FLOPs. In this paper we re-think ReLU computations and propose optimizations for PI tailored to properties of neural networks. Specifically, we reformulate ReLU as an approximate sign test and introduce a novel truncation method for the sign test that significantly reduces the cost per ReLU. These optimizations result in a specific type of stochastic ReLU. The key observation is that the stochastic fault behavior is well suited for the fault-tolerant properties of neural network inference. Thus, we provide significant savings without impacting accuracy. We collectively call the optimizations Circa and demonstrate improvements of up to 4.7$\times$ storage and 3$\times$ runtime over baseline implementations; we further show that Circa can be used on top of recent PI optimizations to obtain 1.8$\times$ additional speedup.

Circa: Stochastic ReLUs for Private Deep Learning
Zahra Ghodsi', Nandan Kumar Jha?, Brandon Reagen', Siddharth Garg² "University of California San Diego "New York University zghodsi@ucsd. edu, {nj2049, bjr5, sg175)@nyu edu
Abstract
The simultaneous rise of machine learning as a service and concerns over user pri- vacy have increasingly motivated the need for private inference (PI). While recent work demonstrates Pl is possible using cryptographic primitives, the computational overheads render it impractical. State-of-art deep networks are inadequate in this context because the source of slowdown in PI stems from the ReLU operations whereas optimizations for plaintext inference focus on reducing FLOPS. In this paper we re-think ReLU computations and propose optimizations for PI tailored properties of neural networks. Specifically, we reformulate ReLU an ap- proximate sign test and introduce a novel truncation method for the sign test that significantly reduces the cost per ReLU. These optimizations result a specific type of stochastic ReLU. The key observation is that the stochastic fault behavior is well suited for the fault-tolerant properties of neural network inference. Thus, provide significant savings without impacting accuracy. We collectively call the optimizations Circa and demonstrate improvements of up to 4.7 x storage and 3x runtime over baseline implementations; we further show that Circa can be used on top of recent PI optimizations to obtain 1.8x additional speedup.
1 Introduction
Today, Machine Learning Service (MLaaS) provides high-quality user experiences but comes the cost of privacy-clients either share their personal data with the server or the server must disclose its model the clients. Ideally, both the client and server would preserve the privacy of their inputs and model without sacrificing quality. A recent and growing body of work has focused on designing and optimizing cryptographic protocols for private inference (PI). With PI, MLaaS computations are performed obliviously; without the server sceing the client's data nor the client learning the server's model. protocols are built using cryptographic primitives including homomorphic encryption (HE), Secret Sharing (SS), and secure multiparty computation (MPC). The challenge is that all known protocols for PI incur impractically high overheads, rendering them unusable.
Existing PI frameworks [
are based on hybrid protocols, where different cryptographic
techniques used evaluate different network layers. Delphi [ leading solution based on Gazelle (2) uses additive secret sharing for convolution and fully-connected layers. Secret sharing supports fast addition and multiplication by moving large parts of the computation
offline phase 4 Thus, convolutions can be computed at near plaintext speed. Non-linear functions, notably ReLU, cannot enjoy the same speedups. Most protocols (including Delphi, Gazelle, and MiniONN (I) use Yao's Garbled Circuits (GC) [5] to process ReLUs. GCs allow two parties to collaboratively and privately compute arbitrary Boolean functions. At high-level, GCs represent functions as encrypted two-input truth tables. This means that computing function with GCs requires the function be decomposed into a circuit of binary gates that processes inputs in bite fashion. Thus, evaluating ReLUs privately is extremely expensive, to the point that PI inference runtime dominated by ReL.Us
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Therefore, reducing ReLU cost is critical to realizing practical PI. There are two general approaches for minimizing the cost of ReLUs: designing new architectures that limit ReLU counts, and optimizing the cost per ReL.U. Prior work has almost exclusively focused on minimizing ReLU counts. Work along this line includes replacing or approximating ReLUS with quadratics or polynomials (e.g.. CryptoNets 77,, Delphi [3. SAFENet (8), designing new networks architectures to maximize accuracy per ReLU (c.g., CryptoNAS (6), and more aggressive techniques that simply remove ReLUs from the network (e.g., DeepReDuce (9]). Relatively little attention has been given to minimizing the cost of the ReLU operation itself.
In this paper we propose Cires a novel method to reduce ReLU cost based on new stochastic ReLU function. First, we refactor the ReLU as a sign computation followed by a multiplication. allowing us to push the multiplication from GC to SS, leaving only a sign computation in the GC. Next, we approximate the sign computation to further reduce GC cost. This approximation is not free; it results in stochastic sign evaluation where the results are sometimes incorrect (we call incorrect computations faults to differentiate from inference error/accuracy). Finally, we show that stochastic sign can be optimized even further by truncating its inputs; truncation introduces new faults, but only for small positive or negative values.
Our key insight is that deep networks are highly resilient to stochastic ReLU fault behavior, which provides significant opportunity for runtime benefit. The stochastic ReLU introduces two types of faults. First, the sign of a ReLU can be incorrectly computed with probability proportional to the magnitude of the input (this probability is the ratio of the input magnitude over field prime.) In practice we find this rarely occurs most ReLUU inputs are very small (especially compared to the prime) and thus the impact on accuracy is negligible. Second, truncation can cause either small positive or small negative values to fault. Circa allows users to choose between these two probabilistic fault modes. In NegPass, small negative numbers experience a fault with some probability and are incorrectly passed through the ReLU. Alternatively, PosZero incorrectly resolves small positive inputs to zero. Empirically, we find deep networks to be highly resilient against such faults, tolerating more than 10% fault rate without sacrificing accuracy. Compared to Delphi, Circa-optimized networks run up to x times faster. We further show that Circa is orthogonal to the current best practice for ReLU count reduction [9]. When combined, observe additional 1.8x speedup.
2 Background
2.1 Private Inference
We consider client-server model where the client sends their input to the server for inference using the server's model. The client and the server wish to keep both the input and model private during the inference computation. Our threat model is the same as prior work on private inference [ 2 Bl.
More specifically, we operate in a two-party setting (i.e., client and server) where participants are honest-but-cuuioou--tttt they follow the protocol truthfully but may try to infer information about the other party's 's input/model during execution.
We take the Delphi protocol 3 as baseline and implement optimizations over it. Delphi uses HE and SS for linear layers, where computationally expensive HE operations are performed in an offline phase and online computations only require lightweight SS operations. For non-linear activations, Delphi uses ReLUs, evaluated using GCs, and polynomial activations (x²), evaluated using Beaver multiplication triples (4]. While Circa does not use polynomial activations, it uses Beaver triples in its stochastic ReLU implementation. Next, we introduce the necessary cryptographic primitives and provide overview of the Delphi protocol.
2.2 Cryptographic Primitives
Finite Fields. The cryptographic primitives described subsequently operate on values in a finite field of integer modulo prime p. F i.e., the set (0, 1......., P 1}. In practice, positive values will be represented with integers in range jo, 2-1), and negative values will be integers in range (p))
Additive Secret Sharing. Additive secret shares of value x can be created for two parties by randomly sampling value and setting shares as (z)1 and (z)2 = x r. The secret can be
In Latin, "circa" means approximately.

---

Linear
Non-linear
client
server
client
server
offline
71 Fp
5 - F3
C client labels
GC
Garble(C,)
E(r;)
Win- - -S
HE(W -()
(a) (br)e (a,b))
Beaver Triples Precomoute
(a))s (be)s (apbi)s
server labels Eval(C)
(vi)c 72
(yi)s = yi -
GC
Vi+1 - 1i+1
online
client sets
server computes
(Vi+1)e
Beaver Triples
(Vi11)s
(x:) Win (xx)s = W - r;) + (1+41e T++1
V(+1 - M+11
Figure 1: An illustration of the Delphi protocol, which Circa uses. Delphi is a hybid protocol that uses HE (offline) and SS (online) for linear layers, and GC and Beaver multiplication triples for ReLU and polynomial non-linear layers respectively.
reconstructed by adding the shares x = (a)1 + (2)2. Performing additions over two shared values straightforward in this scheme, each party simply adds their respective shares to obtain additive sharing of the result.
Beaver Multiplication Triples. This protocol 4 is used to perform multiplications over two secret shared values. A of multiplication triples are generated offline from random values and such that the first party receives (a)1, (b)1, (ab) and the second party receives (a)2, (b)2, (ab)2. In the online phase x and y are secret shared among parties such that the first party holds (zr) (y) and the second party holds (z))., (y)2. To perform multiplication they consume a set of triples generated offline and at the end the protocol the first party obtains (zz)) and the second party obtains (zy)).
Homomorphic Eneryption. HE [0] is a type of encryption that enables computation directly on encrypted data. Assuming public key kipub and corresponding secret key ksecs encryption function operates on a plaintext message to create a ciphertext obtains the message from the ciphertext messages m2 we have a function such that decrypting the ciphertext E(m1, kyub) ) E (m2g, kpub), which we also write as HE(m, ④ m2), gives m. @ m2.
Em,, kpab), and a decryption function (c, keee). An operation C is homomorphic if for
Garbled Circuits. GCs [S] enable two parties to collaboratively compute a Boolean function on Boolean circuit C. One party (the garbler) encodes the circuit using procedure Ö +- Garble(C)) and sends it to the second party (the evaluator). The evaluator also obtains labels of the inputs and is able to evaluate the circuit using procedure Eval((") without learning intermediate values. Finally, evaluator will share the output labels with the garbler, and both parties obtain the output in plaintext. The cost of GC is largely dependent the size of the Boolean circuit being computed.
their private inputs. The function is first represented
2.3 Delphi Protocol
We now briefly describe the linear and non-linear components of the Delphi protocol, depicted also For simplicity, we will focus on the ith layer, with input yi and output y1+1- The layer performs linear computation X; Wi.yi (here W; are the server's weights) followed by a non-linear activation Yi+1 = Rx). The protocol consists of an input independent offline phase, and an input dependent online phase. As starting point, the client samples random vectors ri F in the offline phase. For input y1. the client computes y1 - Iı and sends it to the server. For the 7th layer, the client and the server start with secret shares of the layer input, (y:).- r; and (y:). y; r2, and use the Delphi protocol to obtain shares of the output, (yi+1)c c = ri+1 and (y++1)= = Yi+1 - ri+1-
in Figure I
Linear computation: In the offline phase, the server samples random vectors s; F Using HE on the server side, as shown in Figure[] the client then obtains W,.r; - s; without learning the server's randomness or weights and without the server learning the client's randomness. In the online phase, server computes W;-(y; - r;) + 8; at which point the client and the server hold additive secret shares of x4 Wi-yj. Circa uses the same protocol for lincar layers Delphi.

---

Non-linear computation: In this description we will focus on ReLU computations (for completeness, Figure [ also illustrates how quadratic activations are computed). During the offline phase, the server creates a Boolean circuit C' for each ReL.U in the network, garbles the circuit and sends it to the client along with labels corresponding to the client's input. In the online phase, the linear layer protocol produces the server's share of the ReLUs input. The server sends labels corresponding to its share to the client. The client then evaluates the GC, which outputs the server's share, Yi+1 - M1+1, for the next linear layer. Online GC evaluation is the most expensive component of Delphi's online PI latency. Circa focuses on reducing this cost.
3 Circa Methodology
We now describe Circa, beginning with a cost analysis of the GC design used in prior work. We then describe three optimizations to reduce GC size and latency that form the core of Circa.
3.1 Cost Analysis of ReLU GC
The inputs a conventional ReLU GC are the client's and server's shares of i.e. (x)) and (x).. and random value r from the client. Each is a value in the field Fy. implemented using m [log(p) bits.
Prior work 1I 1 3 implements ReLU with a circuit that performs several computations contributing to the GC cost as shown in Figure 2(a). First. x (I)) + (z). mod p is computed by obtaining (z)e + (a)), and p using two m-bit adder/subtractor modules (ADDISUB).
checked for overflow, and either 2 or 2 is selected using multiplexer (MUX). Then, x is compared with 5 using an m-bit comparator (>), and MUX outputs either 0 or x. Finally, the GC outputs the server's s share of ReLU(x)) by performing modulo subtraction of from the output of the previous MUX using two ADD/SUB modules and another MUX.
This design, used by Delphi and Gazelle, results in a GC size of 17.2KB per ReLU, Overall, the GCs for ResNet32, as implemented in Delphi, require close to 5GB of client-side storage per inference GC size directly correlates with PI latency, resulting in prohibitive online runtime.
3.2 Circa's Stochastic ReLU
Circa's Stochastic ReLU is built using three optimizations that work together to reduce GC size. We describe each optimization below.
Refactoring ReLUs. Our first observation is that ReLU(zz) can be refactored as xr),,g(()), where sign(z) equals 1 if æ 2 0 and 0 otherwise. Since multiplications can be evaluated cheaply online using Beaver triples, only (zgn(z) must be implemented in GC. Let sign(z); the GC computes the server's share v. (w) = sign( x)) - r, using shares of x and random value r provided by the client. The client will then set its share to (v) =r.
Figure shows our naive GC implementation for sign(x)) As in the ReLU(z)) GC (Figure 2 a)). we first compute = (z) + (x)). mod using two ADD/SUB modules and MUX, and use comparator to check x against G. By having the client pre-compute and provide both -r and 1 as inputs to the GC we save two ADD/SUB modules, since we no longer need to perform these
computations inside the GC. Note that the client selects r and can compute -r and by itself at plaintext speed. Formally, our GC for the sign function implements:
sign (x))) (c).: - r, 1r, ( -r if (x) + (x) mod p> > of
otherwise
The client and server now hold secret shares of both x (i.e., (#)e and (x)x), .). and v (i.e., (v), and (u).). We now use pre-computed Beaver multiplication triples, as described Section 2.2 to compute shares of = szgg((z)) This multiplication is cheap and the optimized ReLU is smaller and faster than standard GC implementation in Figure 2aa,, a). providing modest benefits.
Note that separate GC must be constructed for each ReLU in a network and GCs cannot be reused across inferences.

---

(x)
(x)
(x)
(x)
-
ADD
P
ADD
ADD
ADD
MUX
MUX
MUX
p/2
P/2
(2) (sign(*)))
(P), (x)
MUX
MUX
(x)
SUB
(1). (x)gn(x)))
(r), (x)g
p
(x),
(x))
ADD
Beaver Mult Triples
MUX
Beaver Mult Triples
(v). (Re).U(x))e
())a (R)LL(()))
())
(x.sign())),
(1) (x.sign(x)),
(y) = (x))
(y)x (x))
(c)
Figure Comparing implementations of the ReLU function for PI. (a) depicts the implementations in prior work M 2 3 where the ReLU function is implemented with GC, (b) shows a naive implemen- tation of the sign function followed by multiplication triples, and (c) describes the implementation in Circa with an optimized sign function followed by multiplication triples. The heaviest part of the computation each of these implementations relates to the GC which is shown in shaded blocks.
Stochastic Sign. Our second optimization further reduces the cost of the sign computation. As noted previously, the naive sign GC still uses high-cost components inside the GC because of the need to perform modulo additions to exactly reconstruct modulo additions require expensive checks for overflow and subsequent subtraction. Our next optimization only looks at the regime without overflow, greatly simplifying the GC at the cost of introducing occasional faults in sign computation.
Figure (c) shows our proposed stochastic sign optimization that reduces the logic inside the GC to only comparator and a MUX. We first formally define the stochastic sign function.
if (a) S p (z)
sign (p- - (x)es -r,1-)),,)
- otherwise
Note that in the stochastic sign GC, the client sends the negated value of its share (or (x)) instead of the share directly. This optimization avoids the need to compute p (x) inside the GC itself. We formalize the fault rates of the stochastic ReLU below.
Theorem 3.1. For any x E Ep, assuming shares (z), + mod and (x)c where t is picked uniformly at random from Ep.
pf sigg (()) - (x)c.-r,1 -r,1 + sign ((x)er (c) -r, -r) |>|
Proof. Consider the case where x is positive, i.e., x <
P. The wrong sign is assigned to x 2. + mod p < f. This is true when adding and incurs
(x). S p- (z)e. which can be rewritten an overflow, with 2 p r. Since t is drawn at random, the probability of error P E, A similar analysis for negative values of x shows that a wrong sign is assigned when x + t < which results an error probability of P [a|] where [x]|
x for x 5.
Truncated Stochastic Sign. Our third and most effective optimization builds on the observation that the (x),. S p (z)e (equivalently, (z), < t) check in the stochastic sign GC can be performed on

---

truncated values. We show that truncation introduces an additional fault mode; in particular, the check incorrect with some probability for small positive values of in the range 2%) (i.e. values that truncate to 0 with k-bit truncation) but is correct for all other values of x.
Let [ar represent truncation of the k least significant bits of x; i.e., only the bits of æ are retained. We define a truncated stochastic sign as:
most significant
signæ(p (z) (z)., -7, 1 - = sign (LD (2) [()).Jk. 1,00),,,,,11)) 1,,
(3)
where k represents the amount of truncation. We now prove that the truncated stochastic sign function incurs additional enrors (over the stochastic sign) only for small positive values.
Theorem 3.2. For any I E FT assuming shares (f)s = x + mod p and (x) = - t
where t is picked uniformly at random from Fp. and assuming sign (x)): (4) -T, 1 - r)
sign (x)) -r, wr,,-r), then:
r),
2 |r
P( signy ((z) (z) -r, (2) 11, *
(z) -r, -r,1-r))
Ve E (0,2
2k
and zero otherwise.
Proof. For negative x, the stochastic sign is error-free if (z), S p-- (x)e (equivalently (c), t). Consequently the truncated stochastic sign would not incur an error, since (x) = [t]x is assigned a negative sign. The additional error in this case is when [(x))]*
[t] for positive values of Therefore the error happens when [x + fJ [4]k, or x + / does not overflow to higher p - & bits.
So for [x|| > 2 there is no error. In other case, the error happens when x + 1 > 2 or equivalently
t 2k
lrl. Assuming a uniform distribution the error probability would be P 2
Putting it All Together: the Stochastic ReLU. We define Circa's stochastic ReLU as ReLU,(z)
a.sign, (z) with signy defined in Eq. Stochastic ReLUS incur two types of faults: (1) sign error,
independent of k with probability
(in practice |x|| < for typical choices of prime), and (2)
small positive values in the truncation range [0, 2 zeroed out with high probability "=]",
however, for small values we expect the impact on network accuracy to be low.
We note that Eq. 2 could have been defined such that sign outputs -r for (z) < (x) With this modification, truncation errors occur with the same probability but for small negative values in range [p- - 2A, p) that are passed through to the ReLU output. That is, our stochastic ReLU can operate in one of two modes: (1) PosZero, that zeros out small positive values, or (2) NegPass, that passes through small negative values.
4 Evaluation
In this section we evaluate Circa and validate our error model. We show that our optimizations have minimal effect network accuracy and show the runtime and storage benefits of Circa.
4.1
Experimental Setup
We perform experiments on ResNet18 I1. ResNet32 11 and VGG16 12 Since Circa can be used replace ReLU activations in any network, we also perform experiments on DeepReDuce- optimized models [9] that are the current state of the art for fast PI. We train these networks on CIFAR-10/100 113 and TinylmageNet 14] datasets in plaintext (Circa
involved). The training procedure uses stochastic gradient descent with step learning optimizer and 0.1 initial learning rate, 128 batch size, 0.0001 weight deacy. 0.9 momentum, and milestones at 100th and 150th epochs for 200 epochs. CIFAR- 1IFAR-10/100 (C10/100) datatset has 50k training and 10k test images (size 32 x 32) separated into 10/100 output classes. TinyImageNer (Tiny) consist of 200 output classes with 500 training and 50 test samples (size 64 x 64) per class. Prior work on PI typically evaluate on smaller datasets and lower resolution images because PI is prohibitively expensive for ImageNet or higher resolution inputs.

---

100
Total Fault (Stochastic ReLU)
2nd
80
Total Faull(Model))
1.0
Fault rate (%
Frequency
2 8 Probboo
2-1
Pos Fault (Stochastic ReL.U)
0.8
2
60
Pos Fault (Model)
0.6
30
0.4
2-30
20
2 12
0.0
21
2
2
2
2
17
Input (x)
-Tunccted-bitt
(a)
(b)
Figure 3: (a) Fault probability of stochastic ReLUU with 18-bit truncation in the PosZero mode, and a histogram of ResNet182 1'ss activations, (b) Validating Circa's fault model for ResNet18 CIFAR-100.
80
70
Accuracy/Faai Rate (%)
Accuracy/ Faaracc//Faltttte Rate (5)
60
60
-00
80
Accuracy (NegPass)
50)
Accuracy (NegPuss)
Accuracy (PoxZero)
30
Accursey (Po77en))
30
Fault Rate (NegPuss)
Fault (NegPass)
20
Faolt Rate (Pon7an))
20
Fault Rate (PooZetro
10
20 #Truncated-bits
20
# #Truncated
CIFAR-100 on ResNet18
TinyImageNet ResNet18
80
Accuracy/ Rate (%)
Accuracy/ Fauia Rate (%)
20
-0
Accuracy (NegPass)
Accuncy (NegPass)
Accuracy (PozZan)
Accuracy (ProZeroo
Pault Rate (NogPass) 1)
-0
(NegPass)
Fault Rate (PooZero)
to
Truncctedd-iit
#Trnncated-bin
(c) CIFAR- 100 on DecpReDuce Model
(d) TinylmageNer on DecpReDuce Model
Figure 4: Accuracy and fault rate variation with the increasing number of truncated-bits. Experiments performed CIFAR-100 and TinylmageNet with ResNet18 vanilla model (top row) and with DeepReDuce models 9 (bottom row).
Circa uses Delphi protocol as a base, but substantially modifies the way ReLUs are implemented. We use the SEAL library [15) for HE, and fancy-garbling library [16] for GC. We benchmark P1 runtime on an Intel 19-10900X CPU running at 3.70GHz with 64GB memory. The baseline accuracy of the models in PI is reported using an integer model with network values in a prime field. obtain an integer model, we scale and quantize model parameters and input to 15 bits (as in Delphi), and pick 31 bit prime field (p 2138816513) ensure that multiplication of two 15-bit values does not exceed the field. The baseline accuracy our integer models reported in Table
4.2 Experimental Results
Validating the Stochastic ReLU Fault Model. We begin by validating our model of stochastic Re- LUs described in Section 3.2 (Theorem 3.1 and 3.2) against Circa's 's stochastic ReLU implementation. Figure plots the fault probability of stochastic ReLU with 18-bit truncation in the PosZero mode against the histogram of ResNet18's activations after the first convolution layer. According to our fault model, small positive activations in truncation range < x < 218) incur a high fault probability, and we have P (218- 2 For values outside of this range, the fault probability is small and grows proportional activation absolute value, and we have = |al/pp Figure ((b) plots fault rates on ResNet18 8 trained on C100 for PosZero sotchastic ReLU mode. We plot the total fault rate for all
7

---

Table 1: Circa on CIFAR-10 10 (C10), CIFAR- 100 (C100), and TinylmageNet (Tiny).
Bascline Acc
Stochastic ReLU
Baseline Runtime (s)
Circa Runtime (8)
Runtime Speedup
Network-Datan
#ReL.Us (K)
NegPass (bits)
PosZero (bits)
ResNer32-C10
303.1
92.43%
91.47% (12)
91.85% (12)
6.32
2.47
2.6x
ResNet1 8-C10
557.1
94.66%
93.77% (11)
94.24% (11)
11.08
3.89
x
VGG16-C10
284.7
94.00%
93.77% (12)
93.61% (13)
5.89
2.25
2.6x
ResNec32-C100
303.1
67.32%
66.41% (14)
66.32% (13)
6,32
2.47
2.6x
ResNet18-C100
557.1
74.24%
73.80% (13)
73.76% (12)
11,05
4.15
2.7x
VGG16-C100
284.7
73.94%
73.25% (12)
73.19% (12)
5.89
2.25
2.6x
ResNer32- Tiny
1212.4
55.53%
55.15% (16)
54.56% (15)
24.24
9.04
2.7x
ResNet1 T-Tiy
2228.2
61.60%
60.60% (13)
60.65% (12)
44.55
14.28
3.1x
VGG16-Tiny 1114.1
50.85%
50.73% (12)
50.30% (12)
21.41
6.96
3.1x
activations and the fault rate for only positive activations. Points in the plot indicate measurements from the implementation and the lines show our estimates using the model. We observe that our model is consistent with the implementation across wide range of truncation values. As expected, as we increase the amount of truncation, the fault rates increase. With 28 bits of truncation, all positive activations are faulty. The total fault rate is 60%, which is lower than the positive fault rate because negative activations incur relatively few faults, as predicted by our model.
Fault Rates and Test Error vs. Truncation. Figure 4 shows the relationship between truncation, fault rates, and test accuracy. The experiments are done using the C100 and Tiny datsets with ResNet18 and DeepReDuce-optimzzz
networks. Each plot shows data for both NegPass and PosZero modes. We observe that in all cases, Circa is able to truncate 17-19 bits with negligible accuracy loss fault rates up 10%. We also find that the PosZero version of Circa is consistently slightly better than NegPass for these models, enabling between 1-2 extra bits of truncation. For all subsequent experiments, we pick the fault mode and truncation such that the resulting accuracy is within 1% of baseline.
Circa Accuracy and PI Runtime on Baseline Models. Table shows the accuracy and runtime of Circa for the C10/100 and Tiny datasets applied on top of standard ResNet18/33 8/32 and VGG16 models. Circa achieves 2.6 x to 3.1x PI runtime improvement, in each instance with less than 1% accuracy reduction. The runtime improvements are larger for Tiny because the bascline networks are different owing Tiny's 's higher spatial resolution.
Circa vs. State of the Art. Circa's 's baseline accuracy is obtained from plaintext training and can be different from other frameworks. For this reason, we compare relative accuracy drops from baseline using a 1% accuracy drop as a target. We also provide comparisons at similar baseline accuracy to other frameworks. Delphi, the baseline protocol on which we build Circa, reduces PI runtime by replacing selected ReL.Us with cheaper quadratic activations. For C100, Delphi reduces the number of ReLUs in ResNet32 by 1.2x with 1% accuracy loss compared to baseline which, at best, translates to an equal reduction in PI latency. For the same setting, Circa achieves a 2.6x reduction in PI latency. SAFENet [8] achieves 1.9x speedup over Delphi on ResNet32/C100, while Circa achieves 2.6x speedup. To fairly compare absolute accuracies, we also trained a ResNet32 model on C100 using cosine annealing learning rate for 90 epochs with a 68.15% baseline accuracy that is closer to the baseline in Delphi and SAFENet. Using this model, Circa achieves 67.76% accuracy (a 0.39% accuracy drop) for 2.6x speedup (17-bit truncation). while SAFENet and Delphi have 67.5% and 67.3% accuracies for 1.9x and 1.2x speedups, respectively.
Circa can be applied on any pre-trained ReLUU network. Table shows Circa's accuracy and PI runtime applied to DeepReDuce-optimizee networks, the current state of the art in Pl, across a range of ReLU counts on C100 and Tiny. Circa reduces DeepReDuce PI latency by 1.6x to 1.8x, with less than 1% accuracy loss. Moreover, Circa improves the set of Pareto optimal points. For example, Circa achieves 75.34% accuracy C100 with 1.65s PI runtime, while DeepReduce has both higher runtime (1.71s) and lower accuracy (74.7%). For Tiny, Circa increases DeepReduce's 's accuracy from 59.18% 61.63% effectively iso-runtime (3.18s vs. 3.21s).
Finally, NASS 17 builds on top of Gazelle and uses a reinforcement learning agent to search for optimal architectures and quantization parameters for each layer. The quantization approach in NASS primarily reduces the linear layer costs, which are already low for Cirea (heavy HE operations for linear layers moved offline per Delphi's protocol). On the other hand, the
8

---

Table Circa with Duep (ResNet18) models on CIFAR- 100 and TinylmageNet.
Baseline Acc
Stochastic ReLU
Bascline Runtime (s)
Circa Runtime (s)
Runtime Speedup
Network-Datase
#RcLUs (K)
NegPass (bits)
PosZero (bits)
DeepReD1-C100
229.4
76.22%
76.34% (13)
75.62% (12)
3.18
1.84
1.7x
DespReD2-C100
114.7
74.72%
73.47% (13)
73.64% (13)
1.71
1.05
1.6x
DeepReD3-C100
196.6
75.51%
75.13% (13)
75.34% (13)
2.76
1.7x
DeepReD4-C100
98.3
71.95%
71.45% (13)
71.65% (13)
1.48
0.903
1.6x
DeepReD1-Tiny
917.5
64.66%
64.62% (14)
64.53% (14)
12:27
6.68
1.8x
DeepReDD-Tinn
458.8
62.26%
61.28% (15)
61.26% (15)
6.50
3.94
1.6x
DeepReDS-Tiin Tiny
393.2
61.65%
61.63% (15)
61.66% (15)
5,38
3.21
1.7x
DeepReD6-Tiin 229.4
59.18%
58.65% (15)
58.61% (15)
3.18
2.01
1.6x
GC evaluation cost reduction of NASS is proportional to the reduction in number of ReLUs and is result of fewer ReLUs in the proposed architecture. Therefore, Cirea and NASS can be used in conjunction to further reduce costs in each framework. To demonstrate this, we trained the architecture from NASS and obtained a baseline 86.72% accuracy with 15-bit quantization. Applying Circa over this baseline yields 85.46% accuracy with 2.2x speedup. Reducing the quantization level to 10 bits would allow smaller 20-bit prime (further reducing GC costs). Apply- ing Circa on the 10-bit network yields 84.64% accuracy (compared to 84.6% reported in NASS) at 2.35x speedup over the 15-bit network.
GC Size (KB) ReLU
17.5
15.0
1.4x
12.5
1.9x
10.0
4.7x
Effectiveness of Circa Optimizations. Circa encom- passes three optimizations that build on top of each other, buying us multiplicative savings in GC size and PI run- time. F the GC size after each optimization. Replacing the baseline 31 bit ReLU GC with a 31-bit sign GC reduces GC size by 1.4x, x, (with no accuracy loss), a 31-bit stochastic sign GC is 1.9x smaller, and truncating the stochastic sign to 12-bits achieves 4.7x saving over the baseline. The runtime improvements from each of these optimizations are shown in Table in the Appendix.
7.5
5.0
2.5
0.0
ReLU
Sign StSign StSign,
(31 b)
(31 b) (31 b) (12 b)
Circuit
Figure 5: Garbled circuit size compari- son between baseline ReLU, naive sign and Circa stochastic ReLUs.
5 Related Work
PI Protocols. Over the past few years a series papers have proposed and optimized protocols for private machine learning. CryptoNets (7] demonstrates HE only protocol for inference using the MNIST dataset. SecureMI. [18 shows how secret sharing could used for MNIST inference and trains linear regression models [18] MiniONN [ combines secret sharing with multiplication triples and GCs, allowing them to run deeper networks, and forms the foundation for series of follow-on protocols. While MiniONN generates multiplication triples for each multiplication in linear layer, Gazelle 2 uses an efficent additive HE protocol to speed up linear layers. Delphi shows how significant speedup can be obtained over Gazelle by moving heavy cryptographic computations offline. XONN [19] enables private inference using only GCs for binarized neural networks and leverages the fact that XORs Another approach is to replace GCs with secure enclaves and process linear layers on GPUs for more performance (20). Some have also focused on privacy enhanced training 21 22 typically assuming a different threat model than this work.
be computed for free in the GC protocol to achieve speedups.
ReLU optimization. Prior work has also looked at designing ReLU optimized networks. A common approach is to replace ReLUs with quadratics 7 g 23 While effective in reducing GC cost, this complicates the training process and can degrade accuracy. Another approach is to design novel ReL Reot-nptde architectures. CryptoNAS (6) develops the idea of a ReLU budget and designs new architectures to maximize network accuracy per ReLU. DeepReDuce is recent work that proposes simply removing ReLUs from the network altogether ( Deee is the current state-of-the-ar solution for PI, and we demonstrated how Circa can be used on top of it for even more savings.
Fault tolerant inference. Many have previously shown neural inference resilient to faults even during inference. In the systems community, fault tolerant properties are often used to improve

---

Others have shown that networks can tolerate approximation reduce model size by pruning insignificant weights and activations and possibly compressing them 28
energy eeeeyeefic and runtimes 24
25
26
27
29
30
31
6 Conclusion
This paper presented Circa, new method to significantly speed up PI by lowering the high cost of ReLUS via approximation. Our overarching objective was to minimize the amount logic that must occur inside the expensive GC. To achieve this goal we reformulated ReLU as explicit sign test and mask, where only the sign test is evaluated with GCs and showed that we can truncate, or simply remove, many of the least significant bits to the sign test for even more savings. Though the sign test and truncation optimizations introduce error, we rigorously evaluated the effects and found negligible impact on accuracy. Compared to bascline protocol (Delphi) for PI, Circa provided over x (2.2x for quadratic Delphi) speedup. Furthermore, we showed how existing state-of-the-ar PI optimizations can be combined with Circa for even more savings, resulting in an additional speedup of 1.8x over a baseline protocol. In absolute terms, Circa can run Tiny ImageNet inferences within seconds, bringing Pl another step closer to real-time deployment.
Limitations and Societal Impact Circa applies only for inference and not for training, and addition- ally only applies to certain types of deep networks. Private inference seeks to protect individuals from having to reveal sensitive data, but might also allow unregulated misuse of deep learning services.
Acknowledgements
This work was supported in part by the Applications Driving Architectures (ADA) Research Center, JUMP Center co-sponsored by Semiconductor Research Consortium (SRC) and the Defense Ad- vanced Research Projects Agency (DARPA), by the DARPA Data Protection in Virtual Environments (DPRIVE) program (contract HR001 100011-21-9-0099 grants 1646671 and 1801495. The views, opinions and/or findings expressed are those the author and should not be interpreted as representing the official views policies of the Department of Defense or the U.S. Govermment.
and by the National Science Foundation (NSF)
References
[1] Jian Liu, Mika Juuti, Yao Lu, and N Asokan. Oblivious neural network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 619-631, 2017.
[2] Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. Gazelle: low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pages 1651-1669, 2018.
[3] Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: A cryptographic inference service for neural networks. In 29th USENIX Security Symposium (USENIX Security 20), 2020.
[4] Donald Beaver. Precomputing oblivious transfer. Annual International Cryptology Confer- ence, pages 97-109. Springer, 1995.
[5] Andrew Chi-Chih Yao. How to generate and exchange secrets. In 27th Annual Symposium on Foundations of Computer Science (sfes 1986), pages 162-167, 1986.
[6] Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg. CryptoNAS: Private inference on relu budget. In Advances in Neural Information Processing Systems, volume 33, pages 16961-16971, 2020.
[7] Ran Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International Conference on Machine Learning, pages 201-210, 2016.
Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John
10

---

[8] Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. SAFENet: Asecure, accurate and fast neu-ral network inference. International Conference on Learning Representations, 2021.
[9] Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen. DeepReDuce: Relu reduction for fast private inference. In International Conference on Machine Learning. 2021.
[10] Taher EIGamal. A public key cryptosystem and signature scheme based on discrete logarithms. IEEE transactions on information theory, (1(4):469 472, 1985.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[13] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar hrml, 5, 2010.
[14] Leon Yao and John Miller. Tiny imagenet classification with convolutional neural networks. CS 23/N, 2(5):8, 2015.
[15] Nathan Dowlin, Ran Gilad-Bachrack Bachrach, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. Manual for using homomorphic encryption for bioinformatics. 2015.
[16] Brent Carmer, Alex J. Malozemoff, and Marc Rosen, swanky: suite of rust libraries for secure multi-party computation, 2019. /tctps:/ggitnn wtt.taaww..o/aallllliia/a/aaaaaa
com
[17] Song Bian, Weiwen Jiang, Qing Lu, Yiyu Shi, and Takashi Sato. Nass: Optimizing secure inference via neural architecture search. arXiv preprint arXiv:2001.11854 2020.
[18] Payman Mohassel and Yupeng Zhang. SecureML: A system for scalable privacy-preservee machine learning. In 2017 IEEE Symposium on Security and Privacy (SP), pages 19-38, 2017.
[19] M Sadegh Riazi, Mohammad Samragh, Hao Chen, Kim Laine, Kristin Lauter, and Farinaz Koushanfar. XONN: XNOR-based oblivious deep neural network inference. 28th USENIX Security Symposium (USENIX Security 19), pages 1501-1518, 2019.
[20] Florian Tramèr and Dan Boneh. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. arXiv, 2019.
[21] Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek Mittal, and Tal Rabin. FALCON: Honest- Mojjettmaaority maliciously secure framework for private deep learning. arXiv, 2020.
[22] Sijun Tan, Brian Knott, Yuan Tian, and David J. Wu. CryptGPU: Fast privacy-preseevvin machine learning on the GPU. arXiv, 2021.
[23] Edward Chou, Josh Beal, Daniel Levy, Serena Yeung, Albert Haque, and Li Fei-Fei. Faster cryptonets: Leveraging sparsity for real-world encrypted inference. arXiv, 2018.
[24] Brandon Reagen, Paul Whatmough, Robert Adolf, Saketh Rama, Hyunkwang Lee, Sae Kyu Lee, José Miguel Hermández-Lobate low-power, highly- highla-accurate deep neural network accelerators. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pages 267-278, 2016
YeoY Wei, and David Brooks. Minerva: Enabling
[25] Brandon Reagen, Udit Gupta, Lillian Pentecost, Paul Whatmough, Sae Kyu Lee, Niamh Mulholland, David Brooks, and Gu-Yeon Wei. Ares: framework for quantifying the resilience of deep neural networks. In 20/8 55th ACM/ESDA/EEE Design Automation Conference (DAC), pages 1-6, 2018
[26] Jeff Zhang, Kartheek Rangineni, Zahra Ghodsi, and Siddharth Garg. Thundervolt: enabling aggressive voltage underscaling and timing error resilience for energy efficient deep learning accelerators. In Proceedings of the 55th Annual Design Automation Conference, pages 1 1-6, 2018.
11

---

[27] Lillian Pentecost, Marco Donato, Brandon Reagen, Udit Gupta, Siming Ma, Gu- Yeon Wei, and David Brooks. Maxnvm: Maximizing dnn storage density and inference efficiency with sparse encoding and emor mitigation. MICRO '52, New York, NY, USA, 2019. Association for Computing Machinery.
[28] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmman coding. arXiv, 2016.
[29] Brandon Reagen, Udit Gupta, Robert Adolf, Michael M. Mitzenmacher, Alexander M. Rush, Gu-Yeon Wei, and David Brooks. Weightless: Lossy weight encoding for deep neural network compression. In ICML, 2018.
[30] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pages 1058-1066, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[31] Udit Gupta, Brandon Reagen, Lillian Pentecost, Marco Donato, Thierry Tambe, Alexander M. Rush, Gu- Yeon Wei, and David Brooks. Masr; A modular accelerator for sparse mns. PACT, 2019.
12


Reviews:
[{"review_id": "xZYvLR6B7ts", "reviewer": "Reviewer_9jtr", "summary": "The authors propose to improve the speed of modern Homomorphic encryption (HE) approaches which leverage GC for RELUs by moving as much computations as possible outside of the GC. The optimizations introduced in Circa (i.e truncation and removing the modulo addition) come at the cost of stochastic faults. This study demonstrates that modern deep networks are highly resilient to RELU fault behavior and that Circa (the proposed) approach can offer up to 3x faster predictions than Delphi, a state-of-the-art framework. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThis study offers a novel and interesting direction for improving the speed of HE approaches by allowing stochastic faults. It observes that deep networks are resilient to stochastic RELU faults, and this property enables strong speed optimizations without significantly compromising accuracy. \n\nQuality / Significance: \nThe core claim of this paper, Circa can offer a significant speed boost without compromising accuracy, is well supported by the empirical results. This paper tackles a central and common challenge for HE, i.e estimating relus, and makes a meaningful step towards real time applications of HE. \n\nClarity: \nThis paper is clearly written.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately note their paper's limitations (applicable for inference only and only applicable to RELU networks) and the possible negative social impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "bOpJ7BvWLfy", "reviewer": "Reviewer_DEjr", "summary": "This paper proposes Circa to reduce the communicational overhead of ReLU-based privacy-preserving machine learning using three modifications. First, the ReLU circuit is refactorized into two parts including multiplication and sign function, so that a GC-based ReLU is converted into a GC-based Sign function with a Beaver’s Triple’s based multiplication. Secondly, GC-based modular operation is removed from GC-based circuits at the cost of precision. Thirdly, quantization or truncation is used to further speedup GC-based circuits at the cost of precision (potential decryption errors). The authors show that inference accuracy won't be decreased a lot by this decreased precision due to the network’s error-tolerance ability. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a clear ablation study for each of three techniques including sign, stochastic sign, and truncation. For example, readers are easy to find a take-home message that truncation on GC-based circuits plays a very important role in the last performance. \n\nHowever, there are other comments for the paper. \n\n1. The mechanism of the stochastic sign function is not clear. I knew that this paper tries to replace equation 1 with equation 2 since equation 2 does not have the operation of mod $p$. But it is confusing for the condition in equation 2: if $<x>_s$ <= $p$ - $<x>_c$. This condition may have a conflict with the condition in equation 1. From my understanding, $<x>_c$ +$<x>_s$ =$x$ whose range is from 0 to $2p$.    In equation 1, if $x$ mod $p$ > $p/2$, the output is $-r$, otherwise the output is $1- r$. Since $x$ is from 0 to $2p$, the output is $-r$ if $ x$ $\\in$ (0, p/2) $\\cup$ (p, 3p/2).  However, in equation 2, if $x$  $\\in$ (0, p), the output is $-r$ which is different from equation 1. It is not clear how the proposed work handles this conflict without a large error. The proof of the error probability is not clear.\n\n2. Unconvincing claims and assumptions. In line 198, this paper claims “in practice, $|x| << p$  for typical choice of prime”. However, if the input $x$ is 15 bits, the weights are also 15 bits, and $p$ is 32 bits, the intermediate results $x$ may be near to the $p$, especially when a convolutional layer, normalization layer, or the average pooling layer are used together. It is not clear how the paper controls the $|x|$ to be far less than p. Also, the error probability of stochastic ReLU shows that $p$ should be larger to maintain the small error probability, therefore introducing an overhead about HE parameters selection. For example, schemes without stochastic ReLU could use smaller p, but once using the proposed stochastic ReLU, one must use larger p which potentially introduces larger ciphertext and latency. This paper does not discuss this overhead.  \n\n3. Limited function and usage. The proposed work only supports the ReLU function. It is not clear how to use the proposed methods on the other non-linear operations like max-pool, sigmoid, and softmax function which are also very expensive based on GC. It is much more important to support these non-linear operations other than ReLU functions. \n\n4. Prior work. It is good to compare the truncation method with related work NASS [1] which also uses quantization on the network. \n\nI would like to see the paper's presentation in NeurIPS and will consider modifying my score if the authors clarify my concerns well. \n\n[1] Bian, S., Jiang, W., Lu, Q., Shi, Y., & Sato, T. (2020). NASS: Optimizing Secure Inference via Neural Architecture Search. ECAI.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "_mPU30GCX_f", "reviewer": "Reviewer_auVq", "summary": "This paper focuses on minimizing the computation cost of the ReLU operation. Prior work uses Garbled Circuits (GC) to process ReLUs, which is computationally expensive. This paper proposes a novel method to reduce computation by decomposing ReLU operation into a sign computation and multiplication. The multiplication can be implemented by Secret Sharing and the cost of sign computation through GC can be reduced by approximation. This paper also provides adequate empirical evaluations to show the proposed methods can reduce runtime.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of decomposing ReLU into multiplication and sign computation is nice. I also like the idea of approximating the sign computation to reduce the computation cost. The authors also provide the theoretical analysis of the fault rates of the stochastic approximation. The experiment evaluation is adequate. The authors evaluate Circa on various deep learning models and the results show the Circa requires much less runtime. This paper is also well-written and organized. \n\nMinor comments and questions:\n1.\tIn the experiments, could the authors elaborate on how those models are trained? Does training those models involves Circa or it is used only in evaluating the test accuracy?\n2.\tLine 163: Typo in ‘This multiplication is cheap and the ReLU optimized ReLU’.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they discuss them in Section 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "_9wnG7zEN-a", "reviewer": "Reviewer_3nef", "summary": "The paper proposes several techniques to improve the trade-off between accuracy and storage/runtime for ReLU operations in private inference. The proposed techniques include refactoring ReLUs as a sign function plus a cheap multiplication, and replacing the sign function with a truncated stochastic sign that has runtime and storage beneﬁts (by sacrificing accuracy). Experiments on several datasets and network architectures show that the method has a better trade-off between accuracy and storage/runtime than prior state-of-the-art.", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I do not work in this area and I am not familiar with related work, so I am not able to judge the originality and significance. \n\nDespite my unfamiliarity with the field, the paper is quite easy to follow. The first two sections provide concise while adequate preliminaries for me to understand the contexts. The approach and evaluation sections are well-organized and the takeaways are clear. Overall, the paper is well-written.\n\nI just have several minor questions/suggestions.\n* Line 253-260: It mentions that Circa achieves higher speedup than Delphi (2.6x v.s. 1.2x), but it is unclear what is the accuracy loss of Circa compared with Delphi (1%)?\nLine 269: Again, it mentioned that Circa achieves higher speedup than SAFENet (2.6x v.s. 1.9x), but what about the accuracies?\n* Line 172: itself.We -> itself. We\n* Section 4.2: increasing the spaces between paragraph titles and contents to make it easier to read.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations and potential negative societal impact of their work adequately.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission. The reviewers agree that this paper provides new techniques to reduce the computational overhead of ReLU-based private inference. During the discussion, the authors have addressed the questions raised by the reviewers. The authors should incorporate these clarifications during rebuttal into the next revision of the paper.
Decision: Accept (Poster)