SAFENet: A Secure, Accurate and Fast Neural Network Inference
Authors: Qian Lou
Yilin Shen
Hongxia Jin
Lei Jiang
Venue: ICLR 2021 Poster (2021)
Keywords: Cryptographic inference
Channel-Wise Approximated Activation
Hyper-Parameter Optimization
Garbled Circuits

Abstract: The advances in neural networks have driven many companies to provide prediction services to users in a wide range of applications. However, current prediction systems raise privacy concerns regarding the user's private data. A cryptographic neural network inference service is an efficient way to allow two parties to execute neural network inference without revealing either party’s data or model. Nevertheless, existing cryptographic neural network inference services suffer from huge running latency; in particular, the latency of communication-expensive cryptographic activation function is 3 orders of magnitude higher than plaintext-domain activation function. And activations are the necessary components of the modern neural networks. Therefore, slow cryptographic activation has become the primary obstacle of efficient cryptographic inference. 

In this paper, we propose a new technique, called SAFENet, to enable a Secure, Accurate and Fast nEural Network inference service. To speedup secure inference and guarantee inference accuracy, SAFENet includes channel-wise activation approximation with multiple-degree options. This is implemented by keeping the most useful activation channels and replacing the remaining, less useful, channels with various-degree polynomials. SAFENet also supports mixed-precision activation approximation by automatically assigning different replacement ratios to various layer; further increasing the approximation ratio and reducing inference latency. Our experimental results show SAFENet obtains the state-of-the-art inference latency and performance, reducing latency by $38\% \sim 61\%$ or improving accuracy by $1.8\% \sim 4\%$ over prior techniques on various encrypted datasets.

Published as a conference paper at ICLR 2021
SAFENET: A SECURE, ACCURATE AND FAST NEU- RAL NETWORK INFERENCE
Qian Lou', Yilin Shen², Hongxia Jin and Lei Jiang*
[(lougian, jiang60] eiu
" pyilinn.ar liinss shen, hongxia. jonngiia.jin@ssasuume @samsung. com
l Indiana University Bloomington, "Samsung Research America
ABSTRACT
The advances in neural networks have driven many companies to provide predic- tion services to users in wide range of applications. However, current prediction systems raise privacy concerns regarding the user's private data. A cryptographic neural network inference service is an efficient way to allow two parties to execute neural network inference without revealing either party's data or model. Nev- ertheless, existing cryptographic neural network inference services suffer from enormous running latency; in particular, the latency of communication-expensseee cryptographic activation function is 3 orders of magnitude higher than plaintext- domain activation function. And activations are the necessary components of the modern neural networks. Therefore, slow cryptographic activation has become the primary obstacle of efficient cryptographic inference. In this paper, we pro- pose a new technique, called SAFENet, enable a Secure, Accurate and Fast nEural Network inference service. To speedup secure inference and guarantee in- ference accuracy, SAFENet includes channel-wise activation approximation with multiple-degee options. This is implemented by keeping the most useful acti- vation channels and replacing the remaining, less useful, channels with various- degree polynomials. SAFENet also supports mixed-precision activation approx- imation by automatically assigning different replacement ratios to various layer; further increasing the approximation ratio and reducing inference latency. Our experimental results show SAFENet obtains the state-of-the-arr inference latency and performance, reducing latency by 38% 61% or improving accuracy by 1.8% ~ 4% over prior techniques on various encrypted datasets.
INTRODUCTION
Neural network inference service (NNaaS) is effective method for users acquire various intelligent services from powerful servers. NNaaS includes many emerging, intelligent, client-server applications such smart speakers, voice assistants, and image classifications Mishra et al.) (2020). However, to complete the intelligent service, the clients need to upload their raw data to the model holders. The network model holders in the server are able to access, process users' confidential data from the clients, and acquire the raw inference results, which potentially violates the privacy of clients. So there an urgent requirement to ensure the confidentiality of users' financial records, healthy-care data and other sensitive information during NNaaS.
Modern cryptography such as Homomorphic Encryption (HE) by Gentry al. (2009) and Multi- Party Computation (MPC) by Yao (1982) enables secure inference services that protect the user's s private data. During secure inference services, the provider's model is not released to any users and the user's private data is encrypted by HE or MPC. CryptoNets proposed by GilladBBahhaad ctt al. (2016) is the first HE-based secure neural network on encrypted data; however, its practicality limited by enormous computational overhead. For example, CryptoNets takes ~ 298 seconds to perform one secure MNIST image inference on a powerful server; its latency is orders of magnitude longer than the unenerypted inference. MiniONN by [Lu et al. (2017) and Gazelle by Juvekar et al. (2018) prove that using a hybrid of HE and MPC it is possible to design a low- latency, secure inference. Although Gazelle significantly reduces the MNIST inference latency of

---

Published as a conference paper at ICLR 2021
CryptoNets into ~ 0.3 seconds, it is still far from practical on larger dataset such CIFAR-10 and 100, due to heavy HE encryption protocol and expensive operations. For instance, Gazelle requires 240 seconds latency and ~ 8.3 GB communication to perform ResNet-32 on the CIFAR- 100 dataset. NASS by Biian et al. (200)) CONAD by [Shafran et al. (2019) and CryptoNAS neural network architectures, but they still suffer from heavy encryption protocol in the online phase. Delphi by MMshrr et al.
by Ghodsi et al. (020) are proposed to design cryptography--riende
(2020) significantly reduces the online latency by moving most heavy cryptography computations into the offline phase. Offline computations can be pre- processed in advance.
The State-of- thetthe-rt cryptographic inference service Delphi by Mishra et al. (2020) still suffers from enormous online latency; this is because a big communication overhead between the user and the service provider is required to support cryptographic Re activations. Our experiments show that the communication overhead is proportional to ReLU units in the whole neural network. Delphi attempts to reduce inference latency by replacing expensive ReLU with cheap polynomial approx- imation. Unfortunately, most ReLU units are found to be difficult to substitute without incurring a loss of accuracy. The accuracy will be dramatically decreased as more LU units are approx- imated by polynomials. Specifically, Delphi only replaces 42% ReLU numbers a CNN-7 network (detailed in Section 6.2 of MiniONN by Liu et al. (2017)) and ~ 20% ReLU numbers on ResNet-32 network, with < 1% accuracy decrease. When Delphi approximates more ReLU units, >> 3% inference accuracy will be lost compared to an all- model. If accuracy loss is constrained, non-linear layers still occupy almost 62% to 74% total lateney in CNN-7 and ResNet- 32 networks. Therefore, slow, non-linear layers are still the obstacle of fast and accurate secure inference.
Our contribution. One key observation is that the layer-wise activation approximation strategy in Delphi is too coarse-grained to replace the bottleneck layers in which the ReLU units are mainly lo- cated, e.g. the first layer in CNN-7 occupies > 58% ReLU units. The channels in bottleneck layers are difficult to completely replace without small accuracy loss. To meet accuracy constraints and speedup secure inference, SAFENet includes a more fine-grained channel- activation approx- imation to keep the most useful activation channels within each layer and replace the remaining. less important, activation channels by polynomials. In this way, only partial channels in each layer will be approximated, which is approximate- -riendlrr for bottleeeck layers. Another contribution of SAFENet is that automatic multiple-degee degree polynomial exploration in each layer is supported, compared to prior works using only degree-2 polynomials. Additionally, SAFENet enables mixed- precision activation approximation by assigning different approximation ratios to various layers, which further replaces more Re units with cheap polynomials. Our results show that under the same accuracy constraints, SAFENet obtains ofate-of-the-art inference latency, reducing latency by 38% ~ 61%. or improving accuracy by 1.8% ~ 4% over the prior techniques.
BACKGROUND AND RELATED WORK
Threat Model and Cryptographic Primitives. Our threat model is the same as previous work Delphi by Mishra et al. (2020). More specifically, we consider the service holder as a semi-hones cloud which attempts infer clients input information but follows the protocol. The server holds the Convolutional Neural Network (CNN) model and the client holds the input to the network. For linear computations, the client encrypts input and sends it to the server using a HE scheme by Mishra al. (2020). and then the server returns encrypted output to the client. The client decrypts and decodes the received output. The secret sharing (SS) in Delphi is used to protect the privacy of intermediate results in the hidden layers. Then garbled circuits (GC) guarantees the data privacy in the activation layers, and SS is used to securely combine HE and GC. Other than GC, Beaver's multiplicative Triples (BT) proposed by [Baaver (1995) is used to implement approximated activation using secure polynomials. BTT-baeed polynomial approximation for Le is 3-orders of magnitude cheaper than GC-based LU units on average, so it is used to design approximated secure activation function. At the end of secure inference, the server has learned nothing but the client learns the inference result. More details of cryptographic primitives can be found in Appendix A.1
2.1 CRYPTOGRAPHIC INFERENCE.
Modern neural networks usually consist of linear convolution layers and non-linear activation layers. As Figure Tassows, current state-of-the-art cryptographic inference, Delphi My/hhha et al. 2020
2

---

Published as a conference paper at ICLR 2021
Offiine-Linear Offiine-Act.
Online-Linear Online-Act.
150
86
Client
Server
Client
Server
Offiine:
[G]
Preprocess
latency(s)
120
85
Accuracy
F-WW u
[n]-We[[.
GC or BT
84
60
Online:
X---
(x- Wit
GC or BT
S.
OL (0%)
4L (32%)
5L (37%)
6L (42%)
7L (100%)
82
Linear layort
Non-linear layer
(a) The cryptographic inference in our baseline Delphi.
(b) Various approximate layers number (ratio).
Figure 1: (a) The secure inference scheme of CNN-7 on CIFAR-10. (b) Latency breakdown under
activation approximation at minimum of 84.5% accuracy. XL(Y% ()) means X Xllayer activation or Y%-channel activation are approximated, e.g. 4L(32%) means 4-layer activation approximated which equals 32% activation channels are approximated in the network.
has an offline phase and an online phase. The offline phase is independent of input data and used prepare data for the subsequent online phase. Each phase has linear and non-linear operations.
1. Offline linear layer. During the offline linear process, the client samples a random matrix r, that has the same shape with private input 2;, and then sends its encryption [re] to server. The server processes homomorphic convolutions and returns [ra] W - [ux] to client, where W, is the l-layer network weights and [ue] is ciphertext of sampled matrix by server. The last step in the offline linear layer is that client obtains C, = W2
which is one part of secret sharing of xę We.
Online linear layer. The online linear phase aims let the server obtain S, = (zt (zt-ri) rz) W.+ which is the other secret sharing part of x; W,. It is almost as fast as the unencrypted computation, since the online input is plaintext 2; Tt-
3. Offline Layer- Wise Activation layer. Delphi supports a layer-wise activation function where each activation layer either is ReLU based on GC the approximated polynomial based on BT. During the offline phase, GC needs to generate and share the garbled circuits. BT needs to generate and share the Beaver's triples.
4. Online Layer- either performed by GC or the approximated 2 polynomial. The latency of approximated activation implemented by BT is 192x smaller than ReLU based on GC.
Wise Activation layer. During the online phase of layer-wise activation, ReLU
Latency Bottleneck and Motivation. Figure Tb shows our baseline Delphi suffers from long tency and low accuracy under coarse-grained layer-wise activation approximation. Specifically, OL in Figure Figure[[]mean means none of the ReLU layers and 0% ReLU units are approximated by polynomials. The activation latency takes 72.5% of total latency. For only the online phase, activation latency cupies ~ 99% of the online latency. Therefore, activation layers are the performance bottleneck. OZ also shows the all- ReLU model achieves 85.1% accuracy. With increased approximation layer num- bers, Delphi is able to improve the approximation ratios, but the inference accuracy is decreased at the same time. With the 84.5% accuracy constraints, Delphi at most replaces 6-layer LU layers, with an approximation ratio of ~ 42% LU units. Figure shows the reason why the approxi- mation ratio is low (42%); it is difficult to totally replace the ReLU units by polynomials the bottleneck layer that has > 58% ReLU units without large decrease in accuracy. The ReLU units in modern networks are mainly located the first few layers, and the ReT. numbers are usu- ally decreased exponentially as shown in Figure 2 Especially for much deeper neural networks large dataset, replacing the first Le layer significantly decreases the accuracy. To solve the above problems, we propose more fine-grained channel-wise activation approximation method which is modelled as
optimization problem.
2.2 POPULATION BASED TRAINING (PBT).
Inspired from evolutionary algorithms, Population Based Training (PBT) proposed by Jaderberg et al. (2017) is more efficient method to jointly optimize model weights and user-specified hyper-parameter automatically during training. Many Reinforcement Learning (RL) based hyper- parameter optimization algorithms by Wang al.] (2019) and Lou et al. (2020) are not able to efficiently optimize apper-paramters, since they simply stop training prematurely and consider partially trained accuracy as the final accuracy reward. The details of PBT be seen in ap-
3

---

Published as a conference paper at ICLR 2021
pendix A.2 and PBT by Jaderberg et al. 2017). In this paper, we model the channel-wise activation approximation task as a parameer optimization problem.
Features
TAPAS Gazelle NASS
CONAD Delphi CryptoNAS InstaHide SAFEN
60
50
cennot be fully lppros.
Strong Encryption
x
5 40
Batched HEF
4
0 30
Optimized Activation
/
x
x
8
/
/
20
Channel Wise
K
x
K
10
Mixed- Precccsson
Multiple-Dggeen Degrees
x
x
0
0
4
Layer
Table I: Cryptographic inference works.
Figure 2: Neurons ratio.
2.3 COMPARISON WITH PRIOR WORKS.
Table T shows a comparison between prior works and SAFENet. TAPAS by Sanyal et al. (0188) XONN by Riazi et al.] (2019) and soteria by Aggarwal et al. (220)) focus on binary neural net- works which uses sign() function instead of the Re activation, thereby suffering from inference accuracy loss. And TAPAS by |anyya et al. (0118) and SHE by Lou & Jiang (2019) suffer from long-latency linear operations since they adapt a HE scheme called TFHE by Chillotti et al. (2018) that does not support ciphertext batching operations yet. For example. TAPAS and SHE take ~2 hour and 10 seconds respectively to perform one single MNIST inference with 99% accuracy. In contrast, our work SAFENet and Gazelle using the hybrid of batched HE and MPC are able to achieve < 1-second latency with > 99% accuracy. Gazelle by Juvekar et al. (2018) and MiniONN prove the feasibility of the hybrid use of GC and HE, but they both suffer from enormous latency. NASS by Bian et al.] (2020). CONAD by Shafran et al. (2019), and CryptoNAS by Ghodsi et al. (020) try to improve Gazelle and MiniONN's performance by the co-design of neural network chitectures and cryptographic protocol, but they all require a heavy, online, cryptographic phase. Delphi by Mishra et al. (2020) reduces online latency by moving some online operations into the offline phase and replacing layer-wise activation by degree-2 polynomials. Only our SAFENet sup- ports more fine-grained, channel-wise, activation approximation with multiple-degree polynomial exploration, shown in Table SAFENet also enables the mixed-precision approximation ratios for different layers. Other than MPC and HE based neural networks, InstaHide by Huang al (2020b) and TextHide by Huang et al. (2020a) use a class of subset-sum type encryption by Bhattacbaryya et al. (201 to protect the user's 's sensitive data in machine learning service with only < 5% com- putational overhead and little accuracy loss. However, an attack by Carlini et al. (2020) shows that there is potential security risk on the InstaHide. Compared to the light-weight methods InstaHide and TextHide, MPC and HE provide much stronger security guarantees.
3 SAFENET
Overview. In order to replace expensive non-linear activations by cheap linear polynomial approx- imations as much as possible, we mainly use three methods. Firstly, we use fine-grained activation approximation in channel-wise manner, instead of coarse-grained approximation in layer-wise manner. Secondly, since different layers neural networks have different contributions on the inference accuracy, propose automatic mixed-precio -ixeed-recision method to assign proper approxi- mation ratio for each layer. According to the assigned approximation ratio, SAFENet replaces the corresponding channels in the ascending order of their channel important factors. Mixed-precise further improves the total approximation ratio, given an accuracy threshold. Moreover, SAFENet's mixed-precision planner explores different polynomial approximations of various degrees.
SAFENet Methods Definition. Given an T-Layer CNN model, we define that (W (A are the model weights and activations, and let We R x x and A RSx X xh. be the t-th layer model weights and activations, where Cout is output channel number, Gin is input channel number, Wt, and h are kernels' width and height, Wa and h2 are activations' width and height. The 1-th layer output feature maps O Cour x xh, can be derived by Equa- tion II where Or....: is the output feature map of i-th layer i-th channel, Atji. is the 1-th layer and j-th channel activation, W Wa is the model parameters of t-th layer, i-th output channel and j-th input channel, w. and ho are output feature's width and weight, and * is the convolution operation.
T-1 and
Orian
Argas
W Weagin
(1)
4

---

Published as a conference paper at ICLR 2021
Then the channel-wist activation approximation can be defined as Equation where (t+1)-th t-th layer activation A4+1 of i-th channel is derived by expensive ReLU when the corresponding mask matrix, 1,+ 0. otherwise the approximated activation.
A441a ReLUU(Orss Approz (O,,a)))
if m441, O.
if mf41, = 1.
(0, if imp_order (A (A++1,) 2 Re+1 x (1 ac+1).
m4-1, 11,
(3)
Otherwise.
Equation defines the mask matrix M4+1,-- Here, imp order E [1, Cout) is the importance ascending order of i-th channel A4+1, in the layer (f+1), and cuspoodde((A41.,) can be derived
by sort ( Cin j=1 [W.,ad::::) where Σ/=1 [We.a.s. is the L-1 norm of i-th channel weight, and this L-1 norm is proven effective as the channel importance of Ac+1, by prior works by Li et al. (2017) and by Liu et al. (2019). 7 in Equation 3 is the neuron number in (f+1)-th layer and 1 the activation approximation ratio in the (t+1)- layer. Equation B shows that channel-wist activa- tion approximation in the (t+1)-th 1-th layer only approximate a1+1-ratio less important activations with polynomials by setting their corresponding mask matrix entry 1 = 1. Approz() activation function is enabled by multiple options including degree-3 polynomial a + a2x² + azæ + as, 2 polynomial b1 + by b3, and degree-O pruning, where aj a4 and 61 to 63 values derived from the model training. The reason why we choose these three options is that our experi- ments show larger degree (> 3) polynomial approximation is hardly convergent in training because of the uncontrollable gradient exploding, and smaller degree (< 2) polynomial approximation shows worse performance. However, the proper insertions of degree-O pruning may help on gradient ex- ploding in some layers. We use hyper-parameter 3r E {0, 2,3) 3) as the polynomial degree in the Appror() function. Channel- Wise Activation Approximation means T activation approximation ratios { IT-1 are designed in a hand-crafted manner with 3. 2. Peeced-rreisioo Activation Approximation automatically search the optimal combination {ar ) using our improved PBT algorithm called BTPBT. BTPBT is described in Algorithm Multiple-degree approximation means that approximation options (3:) T- are searched with a; by BTPBT algorithm.
Accuracy
ReLU
Compute
Layer 7%
Layer (+1 Approx.
Score
Approx.
Soore
raties
Layert 70%
Layert degree-2 poly.
Optioon
degree-3
Layer 60%
Layer 1-1
poly
Generate
degree-3 poly.
degree-2
Approx.
Approx.
poly
parameters
ratios a
Channel-Wise Activation
Mulfiple Approx. Options
degree-0
HixxedPreeision
poly.
Activation Planner
Approx options for each layer
PPuning))
Figure 3: The overview of SAFENet working scheme. SAFENet is used to search optimal or and 3r to get best score in the training. ar decides the approximation channels in the 1-th layer, 3 is the polynomial degree in the Approz() function.
3, {0, 2,3}
Working Flow. Figure 3 depicts the working flow of SAFENet. 1 At first, the mixed-precision activation planner randomly initiates approximation parameters, {az)T-] and {TT to control channel-wise activation approximation and approximation options for each layer. Approximation parameters including ] and { are received by the neural network model decide which activation channels are approximated first, and then to decide which approximation method is picked up. In each 1-th layer, n, a₁ unimportant channels are approximated by polynomials and n4 (1 04) more important channels are kept original activation function, e.g. ReLU, according to Equation 2 and Equation
The approximation methods for ni a unimportant LU units are decided. The approximation option consists of multiple-degree polynomials, including degree- 3e polynomials, where 3g E {0, 2, 3). The neural network model with channel wise activation approximation is performed for several times and outputs its inference accuracy. O The mixed- precision activation planner uses the BTPBT method to compute the optimizing score defined in Equation 4 aware of accuracy and reduced latency at the same time. The generated score is used
5

---

Published as a conference paper at ICLR 2021
to guide the approximation parameters in the next iteration. These steps repeat until cryptographic inference accuracy and latency threshold are satisfied.
Score. The goal of our SAFENet is to reduce cryptography latency by replacing expensive original activation with cheap approximated activation as much as possible, given the user-defined accu- racy threshold. Equation is used to assign score for the channel-wise activation approxima- tion parameters a and St. Σi=1 at ne is the number of ReLU units that are replaced. R(3)) means the reduced time ratio using a degree-Bt polynomial to replace ReLU units. For example, R(B, = 3) = 20.1 99.9%, since amortized degree-3 polynomial and ReLU cost 0.2 us and 20.1 us, respectively. Therefore, Σi= X n4 R(3,,) represents the reduced ratio of total latency. Our score keeps the balance of accuracy and efficiency, enabling users to replace more activation neurons with multiple approximated polynomial degrçes, while meeting the accuracy constraints.
score accuracy X (1 +1+)
x x R(3:))
(4)
Algorithm Binary Tree Population Based Training BTPBT(M D, Av, W)
Input: A T-layer neural network Mr with weight W, training data D, accuracy threshold A,-
Output: Activation approximation rate ax( T 1], polynomial degree B(0 T 1], score S, accuracy A and weight W.
1: Construct binary tree, the root node root contains layersi0 T 1] and all layers in the root_nd share the
and B8.
Search one single a and | for root nd using PBT(a,, W). Add root nd into an empty queue root-queee.
3: while root_queue is not Null do
4: parent_nd root www.eeuueeepppppeeeee
5:
left.child, right
BinaryDividdeppreett.dd
6:
11, T rool_queeee (le _cciild))))) if lefeffhhhldd has > layer; Same operation on right_child;
wwghttthild.......... 11, S, A, W) = PBT(left child. 3), right child.fo. fo. 81, D. W);
7:
8: Pick up the intermediate result (a:0 :::00:2 - T':0 1]0 : S, W)
as the best candidate BC that has the
the largest score S and accuracy A A..
9:
Perform (a[0 T 1]. 8[0 T S, A, W) -P : T- 30:30 T 1]}, D.W::
10: Return a0: T 1), 0 T S,
Layere:
3
PBT: <:
a(2) at
r (nt,
DRL
PBT
BTPBT(Ours)
Reduced Latency
BTPBT:
do
D.W.
100%
80%
a:
PET(aFF..0/Po,,D.M
60%
PBT
PBT
40%
Pickup the best candidate. e.g. BC [0,,0.00.00a]
fxee
20%
0%
0
40
160 200 Training Epochs
80
120
a: a[0 a(t) a[2] af3] PBTjO(0:3]-BC.D..)
(a) BTPBT example for 4-layer network.
(b) Learning curves about approximation ratio.
Figure 4: (a) Comparison of PBT and BTPBT. (b) The learning curve of CNN-7 on CIFAR-10.
Pixee-Precision Activation Planner. Finding appropriate parameters oa and 84 to achieve a higher score in Equation is a typical hyper-parameter optimization problem. Hand-crafted optimization for parameters is only practical to handle the search space of single pair of a and / for all layers, which confines the improvement effect on activation approximation. Therefore, mixed-precision activation approximation is proposed to enable each layer to have unique a, and B, However, applying byper-paramettr optimization methods automatically search mixed-precision activation approximation is non-trivial. This is because of the enormous search space of and 33, for each layer. Assume at [0, 1] and it is discretized into the times of 0.05, therefore 21 options; 3, {0, 2. has options. If neural network model has T' layers, the search space is (3 x 21)T. For this large search space, even PBT by Jaderberg et al. (2017) and Deep Reinforcement Learning (DRL) based methods by Wang et al. (2020) suffer from slow learning speed and low approximation ratio. Figure 4b shows PBT and DRL methods cannot effectively reduce latency by mixed-precision activation approximation.
We propose a Binary-Treee Tree PBT algorithm (BTPBT) in Algorithm ! to enable the search of oe and B, BTPBT in Algorithm [ takes the T-layer neural network Mr. training data D

---

Published as a conference paper at ICLR 2021
and accuracy threshold A, inputs, and returns all the searched aq and Br with updated weights W. accuracy A and score S. Instead of using PBT to search all a : T 1] and 310 T 1] directly, BTPBT starts from casier task that limits all a/0 T - 1] sharing the same value, e.g- o, and all 0 T - 1] sharing the same value, e.g. B. This searching process about root nd is shown in the line of Algorithm [ Then the root_nd with all T layers is equally divide into two child nodes using BinaryDivide()) function, so that the left child has the first-half layers of roof_ moof_ndaae and the right child has the remaining layers. We then perform PBT on both left and right child nodes, and limit all layers in each child share the same a and [.. Here, the total approximated activation channels of two child nodes should be larger than the approximated channels of the parent node. The child nodes will become new root nodes and PBT will be performed repeatedly on sibling nodes until the rool queue is null in Algorithm| [ The PBT operations in the binary tree can also be used for the warm-starting initialization for the joint search of all a(0 T 1] and 3|0 : T - 1] shown in line 8-99 of Algorithm
Figure 4a shows comparison of PBT and BTPBT for a T-layer network (T' = 4). PBT directly
performs a joint search on a<O0 T T 1] and 3 : T - 1] (We skip the B in Figure 44 due to space limitation). The enormous search space and random initialization of a/0 T - 1] and $10 : T 1] make PBT difficult to learn useful information in training. Therefore directly using PBT suffers from slow learning speed and inadequate searching results. To solve this issue, we use BTPBT to firstly search one single (rg for all layers and then search more a iteratively in a binary tree order. In this way, the intermediate results about a, and weights W' can be used for the final results or the candidates of the further search. For instance, the intermediate result with the largest score S can be considered as the best candidate for the warm-starting initialization for the joint search of a[0 T - 1). More details about B BTPBT can be seen at Appendix A.3 Figure 4b depicts the learning curves of PTB, DRL by Waag et al. (2020), and our SAFENet. All these methods are used to search syper--arameters Oi and B, for the same score shown in Equation After 200 training epochs, BTPBT is able to replace ~ 80% neurons, but PBT and DRL only replace < 30% neurons. In the beginning of training,
baselines and BTPBT are given the same pre-trained model. The DRL setting is specified in WWang et al. (2020). For PBT and our work, the evolution cycle between two exploitation is 20 iterations, and 50 workers ВТPВТ, each two sibling nodes and the root node require a PBT separately, therefore 21evels PBT is required. We set 20 epochs for each node's ' PBT. We use 40 epochs jointly search the activation approximation. The total 200 epochs with 50 workers take ~ 7 GPU hours.
used to simultaneously search parameters. For
Threat model of BTPBT planner. Our BTPBT takes the RU neural network model Mr and training data D as inputs in the server side, and outputs a well-trained allynomial-approximated neural network Pr. The Pr is used to provide privacy-preserving machine learning inference service that takes the client's encrypted sensitive data Dins as input, and outputs the encrypted prediction result to the client. So our BTPBT does NOT impact the privacy of user's data Dins- BTPBT shares the same threat model with our baseline Delphi by Mishra et al. (2020).
4 EXPERIMENTAL METHODOLOGY
Benchmarks and Dataset, Our secure inference experiments rely on CIFAR-10 and CIFAR-100 datasets. Both CIFAR-10 and CIFAR-100 contain 50000 training images and 10000 testing images, where each image size is 32 x2x33x3. Images in CIFAR-10 10 are classified into 10 classes, but CIFAR- 100 has 100 classes. Our baseline Delphi shows that CIFAR- is the most difficult dataset in the existing secure inference. We adopt CNN-7 specified in MiniONN by Liu et al. (017)) to evaluate CIFAR-10. This CNN-7 architecture is also used Delphi by Mishra et al. (2020). Moreover, a deeper network V6G-16 by Geiffann (0117) is evaluated to further improve inference accuracy on C0FAR-10. For CIFAR-100 ResNet-32 by He et al. (2016) evaluated.
Systems Setup. We ran the secure inference models on two instances. They are equipped with an Intel Xeon E7-4850 CPU and 16 GB DRAM. The communication links between these two in- stances are in the LAN setting, and each instance uses 4 threads, same the previous works. The prper-parameters optimization for activation replacement requires an NVIDIA Tesla V100 GPU. The SEAL library by SEAL and used to implement HE and garbled circuits functions. To avoid overflow, our experiments adopt 15- bit fixed point representation and the Least Significant Bits of intermediate results are also truncated
fancy-garbling library by Carmer et al. (2019) are
7

---

Published as a conference paper at ICLR 2021
to 15 bits. The hyper-parametery selection is implemented in Python, and the BTPBT is constructed based on regular PBT by Jaderberg et al. (001)) in Tune platform by Liaw et al. (2018)
5 RESULTS
Delphii (8%)
CW [Total: 40%)
Approx neurons ratio [%6
-D-Dipphi (Total: 415)
(Total: 71%]
CW (Total:: 50%)
CW-MP-MD (Total: 9136)
Approx. neurons ratio (16)
120
CW-MP (Total: 48%)
CW-MP-MD (Total: 56%)
100
1.00
80
60
40
26
20
6
7
2
8
10
11
12
13
Layers
(a) CNN-7.
(b) VGG-16.
Figure 5: Approximated ratio ablation study on CIFAR-10 10 at minimum of 84.5% accuracy.
Ablation Effects. Figure describes the ablation effects of proposed techniques on CIFAR-10 with CNN-7 and VGG-16. CW, MP,, and MD mean Channel- Wise, Hixee-Precisicion and Multiple Degrees, respectively. Square, triangle and cross marks Figure 5 represent degree-3, degree-2 polynomial approximation and pruning, respectively. For the CNN-7 in Figure Sa our baseline Delphi approximates ~ 41% % activation by replacing the ReLU in the last 6 layers with degree-2 polynomials. Channel-wise (CW) activation approximation using able to achieve the same inference accuracy, while replacing more expensive activation neurons. Channel-wise under mixed precision (CW-MP) further improves the approximation ratio of the reg- ular channel-wise method. Other than channel-wise approximation and mixed precision, Figure [5a shows that the approximation ratio also benefits from exploring multiple degrees in various layers (CW-MP-MD), Figure 5b) depicts the ablation study of deeper network VGG-16. Our SAFENet with CW-MP-MD improves the approximation ReLU numbers of Delphi by 28% under the same accuracy.
0.5 and 2 on all layers
CNN-7
Offline
Online
Total
Accuracy
HE- HE-Linear
MPC-Act
Total
HIE-Linear
MPC-Act
Total
Gazelle
Os
91s
91
37s
9s
46s
137s
85.1%
Delphi
37s
53.4s
90.4s
5.3s
5.3s
95.4s
84.6%
Delphi-Fast
37s
15.1s
52.1s
Os
1.69s
1.69s
53.7s
83.1%
SAFENet
33.3s
18.28
51.6s
Os
1.8s
1.8s
53.4s
85.1%
Table 2: The CNN-7 result CIFAR-10. Delphi-Fast approximates more activation than Delphi,
but losses more accuracy.
VGG-16
Offlie
Online
Total
Accuracy
HE-Linear
MPC-Act
Total
HE-Lincar MPC-Act
Total
Gazelle
Os
142s
142s
45s
16s
61s
203s
89.6%
Delphi
45s
114s
159s
13s
13s
172s
88.
Delphi-Fast Fast
45s
109s
154s
Os
12.48
12.4s
166s
84.9%
SAFENet
40.2s
56,8s
97s
Os
6.5s
6,5s
104s
88.9 %
Table 3: The VGG-16 result on CIFAR-10. Delphi-Fast approximates more activation than Delphi,
but losses more accuracy.
CIFAR-10. Table 2 shows the comparisons of CNN-7 implemented by prior techniques and SAFENet on CIFAR-10. Compared to Gazelle, Delphi not only moves the HE-based linear op- crations to the offline, but also reduces 41% online and offline activation latency without sig- nificant accuracy loss. Our work SAFENet has similar latency with Delphi in the linear layer, but SAFENet remarkably reduces latency at non-linear activation layers. More specifically, SAFENet eliminates ~ 66% online and offline latency in activation layers, and reduces 61% total latency with higher accuracy than Delphi. Delphi-Fass further reduces the latency of Delphi by trying to approximate more activation than Delphi, but it suffers from significantly accuracy decrease. For
8

---

Published as a conference paper at ICLR 2021
instance, Delphi-Fash decreases 2% accuracy when it has similar lateney with SAFENet. We also evaluated a deeper neural network VGG-16 on CIFAR-10 10 shown in Table When Gazelle and Del- phi have similar accuracy. Delphi only reduces 20% latency of Gazelle on non-linear activation layer. This shows that Delphi benefits less from deeper neural networks. Compared to Gazelle, our work SAFENet is able to reduce 59.4% online latency, 48.7% total latency. SAFENet has 50% less online latency and 39.5% less total latency with higher accuracy than Delphi, since SAFENet uses more find-grained channel-wise activation approximation. Compared Delphi-Fast, SAFENet achieves 4% accuracy improvement with less inference latency.
ResNet-32
Offline
Online
Total
Accuracy
Hi-T.inear
MPC-Act
Total
iiner
MPC-Act
Total
Gazelle
Os
158s
158s
64s
18s
82s
240s
67.9%
Delphi
64s
126.4s
190.4s
Os
14.4s
14.4s
204.8s
67.3%
Delphi-Fast
64s
121.1s
175.1s
Os
13.8s
13.8s
198.9s
65.7%
SAFENet
57.6s
63.2s
120.8s
Os
7.2s
7.2s
128s
67.5%
Table 4: CIFAR- results using ResNet-32. Delphi-Fast approximates more activation than Del- phi, but losses more accuracy.
CIFAR-100. Table 4 shows the results of ResNet-32 evaluated on CIFAR-100. Gazelle achieves 67.9% inference accuracy, and each image inference consists of a 158-second offline phase and a 8e-second online phase. Delphi is faster than Gazelle's online latency by 67.6 seconds with only a 0.4% accuracy decrease, but increases the offline latency by ~ 30 seconds. The offline phase is enlarged because Delphi moves most online HE operations offline phase and only reduces few GC operations offline. Compared to Gazelle, our work SAFENet able to reduce both online and offline latency since we significantly reduce the number of GC-based activations: SAFENet reduces online latency by 91.2% and offline latency by 23.5%. Compared to Delphi. SAFENet has 50% less online latency and 36.6% less offline latency. SAFENet reduces total latency by 46. 46.7% and 37.5% over Gazelle and Delphi, respectively. In addition, Delphi suffers from significant accuracy decrease when trying to approximate more activations. For example, Delphi-Fast approximates one more layer, losing 1.6% accuracy. Our work SAFENet improves 1.8% inference accuracy and reduces 35.8% latency over Delphi-Fast.
6 CONCLUSION
In this paper, we propose SAFENet to enable a Fast, Accurate and Secure neural Network inference service. SAFENet consists of three techniques including channel-wise activation, multiple-degree polynomial approximation, and mixed-precision approximation ratios. The channel-wise activa- tion approximation keeps the most useful activation channels and replaces the remaining less useful channels with the various-degree polynomials. The mixed-precisiie activation approximation is implemented by assigning various layers with different approximation ratios further increasing the approximation ratio and reducing inference latency. Our experimental results show SAFENet ob- tains the state-of-the-art inference latency and performance, decreasing latency by 38% 61% or improving accuracy by 1.8% ~ 4% over prior techniques.
REFERENCES
Anshul Aggarwal, Trevor Carlson, Reza Shokri, and Shruti Tople. Soteria: In search of efficient neural networks for private inference. arXiv preprint 1rXiv:2007.12934 2020.
Donald Beaver. Precomputing oblivious transfer. In Annual International Cryptology Conference, pp. 97-109. Springer, 1995.
Arnab Bhattacharyya, Piotr Indyk, David P Woodruff, and Ning Xie. The complexity of linear dependence problems in vector spaces. In ICS, pp. 496-508, 2011.
Song Bian, Weiwen Jiang, Qing Lu, Yiyu Shi, and Takashi Sato. Nass: Optimizing secure inference via neural architecture search. arXiv preprint arXiv:2001./ 2020.
N. Carlini, Samuel Deng, S. Garg, S. Jha, Saeed Mahloujifar, Mohammad Mahmoody, Shuang Song, Abhradeep Thakurta, and Florian Tramèr. An attack on instahide: Is private learning possible with instance encoding? 2020.

---

Published as a conference paper at ICLR 2021
Brent Carmer, Alex J. Malozemoff, and Mare Rosen. swanky; A suite of rust libraries for secure multi-party computation.
https//cc..aaaa...nn//aaaaa..a 2019.
llaria Chillotti et al. TFHE: Fast Fully Homomorphic Encryption over the Torus. IACR Cryptology, 2018.
Yonatan Geifman. Cifar vgg. httpss//g..tub...o///ttaaaaaa/
2017.
Craig Gentry et al. Fully homomorphic encryption using ideal lattices. In ACM Symposium on Theory of Computing, volume 9, pp. 170-178, 2009.
Zahra Ghodsi, Akshaj Veldanda, Brandon Reagen, and Siddharth Garg, Cryptonas: Private inference on a relu budget. arXiv preprint arXiv:2006.088 08733, 2020.
Ran Baaaadahachrch,, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Werns- ing. Cryptonets: Applying neural networks encrypted data with high throughput and accuracy. In International Conference on Machine Learning, pp. 201-210, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora. Texthide: Tackling data privacy in language understanding tasks. CoRR, abs/2010.06053, 2020a. URL https:
06053
Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. InstaHide: Instance-hiding schemes for private distributed learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4507-4518, PMLR, 13-18 Jul 2020b. URL http::/ppoceeeiinngg .ml.. mlr.
press/ press/v119/huagg001 html
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/1711.09846, 2017. /tLLittpp//arxiti
abs 111117111100 11711111111111
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. gazelle: A low latency frame- work for secure neural network inference. 27th USENIX Security Symposium (USENIX Secu- rity 18), pp. 1651-1669, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ArXiv, abs/ 0bs//660.085 0008008710, 2017.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Sto- ica. Tune: research platform for distributed model selection and training. arXiv preprint arXiv:
2018.
Jian Liu, Mika Juuti, Yao Lu, and N. Asokan. Oblivious neural network predictions via minionn transformations. In ACM SIGSAC Conference on Computer and Communications Security, pp- 619-631. 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019.
Qian Lou and Lei Jiang. She: A fast and accurate deep neural network for encrypted data. In Advances Neural Information Processing Systems 32, pp. 10035-10043. Curran Associates, Inc., 2019.
Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang. Autoq: Automated kernel-wise neural network quantization. In International Conference on Learning Representations, 2020.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan, Wenting Zheng, and Raluca Ada Popa. Delphi: cryptographic inference service for neural networks. In USENIX Security, 2020.
10

---

Published as a conference paper at ICLR 2021
M Sadegh Riazi, Mohammad Samragh, Hao Chen, Kim Laine, Kristin Lauter, and Farinaz Koushan- far. Xonn: Xnor- based oblivious deep neural network inference. In 28th USENIX Security Sym- posium (USENIX Security 19), pp. 1501-1518, 2019.
Amartya Sanyal et al. TAPAS: Tricks Accelerate (encrypted) Prediction As Service. arXiv, A 2018.
SEAL. Microsoft SEAL (release 3.2). httess//gthabb..cm//cccrott//// 2019. Microsoft Research, Redmond, WA.
February
Avital Shafran, Gil Segev, Shmuel Peleg, and Yedid Hoshen. Crypto-oriented neural architecture design. arXiv preprint arXiv:1911.12322, 2019.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Ning Wang, Yang Gao, Hao Chen, Peng Wang. Zhi Tian, Chunhua Shen, and Yanning Zhang. Nas- fcos: Fast neural architecture search for object detection. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Andrew Yao. Protocols for secure computations. In 23rd annual symposium on foundations of computer science (sfes 1982), pp. 160-164. IEEE, 1982.

---

Published as a conference paper at ICLR 2021
Layent:
1
2
[00) a[1 o(2) d
Directy search a(0:3)
8TPBT:
Layerk:
PBTio, Cot
Oo
Candidale]]]
Pickup the best candidate BC with the largesi score value S, e.g. BC (tt.0..
[ af1] a[3]
PBT|s((::3=:8..D..) o[0:3] candidate
Figure 6: An comparison of PBT and our BTPBT on the search of a 4-layer network.
A APPENDIX
A.1 CRYPTOGRAPHIC PRIMITIVES.
Homomorphic Encryption (HE). Homomorphic Encryption (HE) is a cryptosystem that supports computation on ciphertexts without decryption private key. Given public key pk: private key sk encryption function e(), and decryption function o(,, we can define an homomorphic operation if there is an operation @ such that (e(21, pk) c(x2, pk), sk) (<((11 Ø2 pk), sk), where T1 and 22 are sensitive plaintext from data holders, and only data holders have the private key sk.
Secret Sharing (SS) and Garbled Circuit (GC). Secret Sharing (SS) usually based on additive sharing private values between the parties. For example, one variable z can be shared into two parts: (x)) in P1 and (x)2 P2, and variable æ can be reconstructed by x = (x) 1 (x) In previous secure inference schemes by Juvekar et al. (2018), SS is used to protect the privacy of intermediate results the hidden layers. Garbled Circuit (GC) cryptographic protocol that enables two parties (Garbler and Evaluator) to jointly compute function over their private data without learning the other party's s data. To use GC, the computed function should be represented into Boolean circuit. The Garbler firstly garbles the Boolean circuit and generates the garbled table. The Evaluator receives the garbled table by the Oblivious Transfer by Juvekar et al. (2018)
and evaluates the table to the result. Previous secure inference schemes by Juvekar al. (0018) show that GC is used to process non-linear activation like Le function.
Beaver's multiplicative Triples (BT). Beaver's multiplicative Triples proposed by Beaver (1955)
can be considered as secure two-party computation protocol. Assume two parties Pi and P2 have
variables a and y. respectively. BT protocol enables two parties obtain the secret sharing of product zy without revealing x and More details can be seen in Delphi by Mishra et al. 2020
BT-based polynomial approximation for ReLU is 3-order magnitude cheaper than GC-based Le units on average, so it is used to design approximated secure activation function.
A.2 POPULATION BASED TPBTN.NG(PBT).
PBT computationally efficient because it adapts parallel scheme and weight sharing during the evolutionary process. Specifically, given pre-trained model, many workers are created and each worker shares the same pre-trained weights and has unique hyper-parameters. Multiple workers are then independently trained for several iterations and evaluated by the user-specified score function. In the exploitation, workers with better scores will keep their parameters, and copy their weights and parameersss the other workers. Workers with worse scores will perform exploration to randomly scale their until the score function is convergent.
The training, scoring, exploitation and exploration repeat
A.3 AN COMPARISON OF PBT AND BINARY-TRES TREE PBT(BTPBT).
Figure 6 shows a comparison PBT and BTPBT for a T-layer network (T = 4). PBT directly performs joint search a(0 3) and A[ 3). The enormous search space and random initializa-
12

---

Published as a conference paper at ICLR 2021
tion of a:O 3] and 3[0 3] make PBT difficult to learn useful information in training. Therefore directly using PBT suffers from slow learning speed and inadequate searching results. To solve this issue, use BTPBT to firstly search one single ag for all layers and then search more iteratively in binary tree order. The root node root nd of the binary tree is constructed by assigning a single ag all the 4 layers in the neural network. One PBT is performed to search the oo, given the dataset D and initialized weights W. The returning result including do, score S, accuracy and trained weights S is called candidate(OJ. After that, all layers in the root two parts: left.child and right_choe .child.hhd.............tl... The left_chill contains the first two layers which shares the same a1, and the right_chill includes the last two layers which shares the same 02- Then one single PBT is used to search and 02 with the score S, accuracy A and trained weights W. The returning, intermediate results are called candidate[1]. The search of left. child node is much more important since the former layers in the neural network contains more neuron activations than the latter layers. This BTPBT searching process is repeated until each node has only one single layer. The layer number of binary tree is equal to logzI' network. In this way, the intermediate results about a, B and weights W can be used for the final results or the candidates of the further search. For instance, the intermediate result with the largest score S can be considered as the best candidate BC for the warm-starting initialization for the joint search of a (10 3]. Our BTPBT is able to reduce the training time and improve the secure inference performance over the prior PBT.
equally divide into
where I is the layer number of neural
13


Reviews:
[{"review_id": "kYaDmq6cAO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe main contribution of this paper is a new heuristic for identifying \"less useful\" activation channels. The authors then propose using simple approximations for activation functions for these channels without compromising network accuracy. The main novelty in the approximation used by the authors is flexibility in the degree of the polynomial approximation. Additionally, the authors propose a new hyper-parameter search strategy (BTPBT) to efficiently search the hyper-parameter space for the optimal approximation parameters.\n\nScore Rationale:\n- The paper has three novel contributions:\n  - Channel-wise approximation of activations\n  - More flexibility in the form of the polynomial approximation\n  - BTPBT hyper-parameter search\n- As a result of these contributions the authors demonstrate a 40% reduction in total inference latency compared to state of art\n\nStrengths:\n- The authors present a thorough description of the experimental procedure that is sufficient to replicate their work\n- An informative ablation study is presented to summarize the relative contribution of the various techniques proposed\n- The baselines chosen are competitive and allow for a fair evaluation\n\nWeaknesses:\n- The main weakness of this work stems from the modest improvement in latency afforded by these techniques\n- In particular while secure inference is still orders of magnitude less efficient compared to the corresponding plaintext computation these techniques can only account for a factor of 2 compared with a baseline that implements no secure inference specific network tuning (e.g. Gazelle)\n\nAdditional Comments/Questions:\n- Is the accuracy of the three networks from Fig 4b comparable?\n- Fig. 5 caption has a typo. The (a) and (b) sub-captions should indicate the network name\n- What is the source of the ~10% improvement in the runtime of the linear layers when compared with Delphi?", "extra_scores": {}, "extra_text": {}}, {"review_id": "kMDBST2GFOp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n\nThis paper proposed a new method called SAFENet.  It is able to support a secure, accurate and fast neural network inference service. The best result presented in Table 2, is 3x faster than previous. The best result presented in Table 2, is 2x faster than previous. \n\nAlgorithm 1 is not very well written, it is hard to follow what is that algorithm trying to say.\n\nIn page 2, the 2nd line in Section 2.1, the sentence ``current state-of-the-art cryptographic inference, Delphi'' It should have a citation here. The section 2 is related work, but Section 2.1 doesn't cite any previous papers, this is a bit strange.\n\nIn the introduction, this paper discusses about cryptography but many important references are missing. For example [Yao] and [Gentry]\n\nAndrew Yao. Protocols for secure computations. FOCS 1982.\n\nCriag Gentry. Fully homomorphic encryption using ideal lattices. STOC 2009.\n\n\nIn related work, the following recent work should be discussed, since they had proposed a practical way to encrypt the image and text data. \n\nYangsibo Huang, Zhao Song, Kai Li, Sanjeev Arora. InstaHide: instance-hiding schemes for private distributed learning. ICML 2020\nYangsibo Huang, Zhao Song, Danqi Chen, Kai Li, Sanjeev Arora. TextHide: Tackling Data Privacy in Language Understanding Tasks. EMNLP 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aisF9dZc57", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an interesting idea where they seek to distinguish between activation channels that are crucial for preserving information flow and approximating the rest with low degree polynomials. The authors show that this doesn't result in decrease in accuracy but leads to speedup in performance.\n\nI think it is a neat trick to exploit the high-dimensional intermediate feature space given that the network only uses a low dimensional subspace of it. It is an interesting use of the idea which has been demonstrated in prior works - low dimensionality of the feature space or that the learning task can be done by randomly switching of a huge fraction of available channels.\n\nHowever, I think the paper has two main issues which prevents me from providing a more favourable decision.\n\n\n* Novelty and Comparison\n   * The key novelty of this paper is to figure out which channels can be approximated with polynomials and which needs to be retained in its original capacity. This approximation leads to a speedup. While it is a neat modification, I think it is very incremental in its novelty and the performance is not very large either.  The time for one ResNet32 inference on CIFAR100 with SafeNet is 0.62x of Delphi, which is not a very large improvement.\n   * It would also be helpful to compare with techniques other than gazelle and Delphi that modifies the neural networks prior to training (eg. binarizing the network, ternarizing the network, fixed point precision etc) so that the accuracy-latency of the trade-off can be put into perspective and compared easily.  When considering methods to speedup Encrypted Prediction as a Service, one should naturally talk about fixed precision networks and binary neural networks (BNNs). In the past, BNNs have shown remarkable speedups in computation without a major hit to accuracy (Sanyal et. al. 2018), which is in fact a major point of this paper. Similarly, the paper should also discuss and compare with techniques that uses modification of encryption schemes specifically meant to optimize activations in NNs (eg. Lou et. al. 2019)\n\nThere has been a large body of work in the past few works that claim to improve latency of encrypted prediction. It is getting hard to say whether a technique really provides a speedup unless a more comprehensive experimental survey is done with a series of papers using benchmark datasets and networks. i would encourage the authors to have more techniques in their experimental comparison section.\n\n* Clarity -  I have found the paper very hard to read in general and some major gaps when considering baselines and background discussion.\n    * I think the paper would be much easier to read if the citations were modified with something akin to \\citep and \\citet. \n    *  The figures (Fig 1, Fig 3, Fig 4) are very far from being self-contained. I think it is okay to omit some details from the figures But there should be proper explanations in the captions. It is very hard to understand to understand what the notations mean in Fig 1A and what 0L, 4L, and the percentages mean. Similarly Fig 4a, is very congested with overlaps between arrows and figures and it is very hard to read and understand what the different notations mean.\n\nI think, overall, the paper needs an overhaul to first make sure that the various components used in the pipeline are explained properly, make sure the figures are clear, the notations are clear especially in the earlier sections. I think further advantages can be gaining incites from papers that talk about importance of various layers (Zhang et. al. 2019), using low rank layers or representations etc and doing comparisons with other works that claim to provide an improvement in encrypted prediction latency.\n\nZhang, C., Bengio, S., & Singer, Y. (2019). Are all layers created equal?. arXiv preprint arXiv:1902.01996.\n\nLou, Qian, and Lei Jiang. \"SHE: A Fast and Accurate Deep Neural Network for Encrypted Data.\" Advances in Neural Information Processing Systems. 2019.\n\nSanyal, A., Kusner, M. J., Gascon, A., & Kanade, V. (2018). Tapas: Tricks to accelerate (encrypted) prediction as a service.  International Conference in Machine Learning, 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xjd8X9Fq46", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper present a system for two-party deep learning inference. The main contribution is activation layers that are more expensive in two-party computation are replaced by approximations dynamically based on the training data. To this end, the authors use a divide-and-conquer approach to gauge the impact of replacing activation functions of some layers by a version more amenable to secure computation. Furthermore, the algorithm also considers various degrees for approximation (0, 2, and 3). Experiments show that this reduces the latency by up to two thirds while maintaining a similar accuracy.\n\nPros:\n- The dynamic approach is more sophisticated.\n- The improvement in efficiency is clear.\n\nCons:\n- Deriving public meta-parameters from data might impact privacy, but no consideration is given to this aspect.\n- The further exploration of when to replace ReLU with an approximation seems somewhat expectable despite the novelty of the exact approach.\n\nMinor issues:\n- 1: \"huge\" (unscholarly language)\n- throughout: \"Delphi Mishra et al.\" - maybe \"Delphi by Mishra et al.\"?\n- References: curly brackets in several references such as {USENIX}\n\nConclusion:\nI recommend acceptance given the novelty.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors did a nice job of responding to the concerns of reviewers during the discussion phase which increased reviewer scores. Because of this I will vote to accept. 

The authors should carefully edit the paper for typos, grammatical errors, and style errors. Some examples:
- Abstract: Make this one paragraph without a line break
- End of 1st paragraph in Intro: "So there is an urge" -> "So there is an urgent"
- Start of 3rd paragraph in Intro: "State-of-the-art cryptographic" -> "The state-of-the-art cryptographic"
- Last paragraph of 2.1: "To solve above" -> "To solve the above"
- End of 2.3: "Compared to the light-weight InstaHide and TextHide, MPC and HE are of advantages in the security guarantees so far." -> "Compared to the light-weight methods InstaHide and TextHide, MPC and HE provide much stronger security guarantees."

I also urge the authors to please double check the reviewer comments when preparing a newer version to ensure all concerns are taken into account.
Decision: Accept (Poster)