SAFENet: A Secure, Accurate and Fast Neural Network Inference

1 INTRODUCTION . Neural network inference as a service ( NNaaS ) is an effective method for users to acquire various intelligent services from powerful servers . NNaaS includes many emerging , intelligent , client-server applications such as smart speakers , voice assistants , and image classifications Mishra et al . ( 2020 ) . However , to complete the intelligent service , the clients need to upload their raw data to the model holders . The network model holders in the server are able to access , process users ’ confidential data from the clients , and acquire the raw inference results , which potentially violates the privacy of clients . So there is an urgent requirement to ensure the confidentiality of users ’ financial records , healthy-care data and other sensitive information during NNaaS . Modern cryptography such as Homomorphic Encryption ( HE ) by Gentry et al . ( 2009 ) and MultiParty Computation ( MPC ) by Yao ( 1982 ) enables secure inference services that protect the user ’ s private data . During secure inference services , the provider ’ s model is not released to any users and the user ’ s private data is encrypted by HE or MPC . CryptoNets proposed by Gilad-Bachrach et al . ( 2016 ) is the first HE-based secure neural network on encrypted data ; however , its practicality is limited by enormous computational overhead . For example , CryptoNets takes ∼ 298 seconds to perform one secure MNIST image inference on a powerful server ; its latency is 6 orders of magnitude longer than the unencrypted inference . MiniONN by Liu et al . ( 2017 ) and Gazelle by Juvekar et al . ( 2018 ) prove that using a hybrid of HE and MPC it is possible to design a lowlatency , secure inference . Although Gazelle significantly reduces the MNIST inference latency of CryptoNets into ∼ 0.3 seconds , it is still far from practical on larger dataset such as CIFAR-10 and CIFAR-100 , due to heavy HE encryption protocol and expensive operations . For instance , Gazelle requires ∼ 240 seconds latency and ∼ 8.3 GB communication to perform ResNet-32 on the CIFAR-100 dataset . NASS by Bian et al . ( 2020 ) , CONAD by Shafran et al . ( 2019 ) and CryptoNAS by Ghodsi et al . ( 2020 ) are proposed to design cryptography-friendly neural network architectures , but they still suffer from heavy encryption protocol in the online phase . Delphi by Mishra et al . ( 2020 ) significantly reduces the online latency by moving most heavy cryptography computations into the offline phase . Offline computations can be pre-processed in advance . The State-of-the-art cryptographic inference service Delphi by Mishra et al . ( 2020 ) still suffers from enormous online latency ; this is because a big communication overhead between the user and the service provider is required to support cryptographic ReLU activations . Our experiments show that the communication overhead is proportional to ReLU units in the whole neural network . Delphi attempts to reduce inference latency by replacing expensive ReLU with cheap polynomial approximation . Unfortunately , most ReLU units are found to be difficult to substitute without incurring a loss of accuracy . The accuracy will be dramatically decreased as more ReLU units are approximated by polynomials . Specifically , Delphi only replaces ∼ 42 % ReLU numbers on a CNN-7 network ( detailed in Section 6.2 of MiniONN by Liu et al . ( 2017 ) ) and ∼ 20 % ReLU numbers on ResNet-32 network , with < 1 % accuracy decrease . When Delphi approximates more ReLU units , > 3 % inference accuracy will be lost compared to an all-ReLU model . If accuracy loss is constrained , non-linear layers still occupy almost 62 % to 74 % total latency in CNN-7 and ResNet32 networks . Therefore , slow , non-linear layers are still the obstacle of a fast and accurate secure inference . Our contribution . One key observation is that the layer-wise activation approximation strategy in Delphi is too coarse-grained to replace the bottleneck layers in which theReLU units are mainly located , e.g . the first layer in CNN-7 occupies > 58 % ReLU units . The channels in bottleneck layers are difficult to completely replace without a small accuracy loss . To meet accuracy constraints and speedup secure inference , SAFENet includes a more fine-grained channel-wise activation approximation to keep the most useful activation channels within each layer and replace the remaining , less important , activation channels by polynomials . In this way , only partial channels in each layer will be approximated , which is approximate-friendly for bottleneck layers . Another contribution of SAFENet is that automatic multiple-degree polynomial exploration in each layer is supported , compared to prior works using only degree-2 polynomials . Additionally , SAFENet enables mixedprecision activation approximation by assigning different approximation ratios to various layers , which further replaces more ReLU units with cheap polynomials . Our results show that under the same accuracy constraints , SAFENet obtains state-of-the-art inference latency , reducing latency by 38 % ∼ 61 % , or improving accuracy by 1.8 % ∼ 4 % over the prior techniques . 2 BACKGROUND AND RELATED WORK . Threat Model and Cryptographic Primitives . Our threat model is the same as previous work Delphi by Mishra et al . ( 2020 ) . More specifically , we consider the service holder as a semi-honest cloud which attempts to infer clients input information but follows the protocol . The server holds the Convolutional Neural Network ( CNN ) model and the client holds the input to the network . For linear computations , the client encrypts input and sends it to the server using a HE scheme by Mishra et al . ( 2020 ) , and then the server returns encrypted output to the client . The client decrypts and decodes the received output . The secret sharing ( SS ) in Delphi is used to protect the privacy of intermediate results in the hidden layers . Then garbled circuits ( GC ) guarantees the data privacy in the activation layers , and SS is used to securely combine HE and GC . Other than GC , Beaver ’ s multiplicative Triples ( BT ) proposed by Beaver ( 1995 ) is used to implement approximated activation using secure polynomials . BT-based polynomial approximation for ReLU is 3-orders of magnitude cheaper than GC-based ReLU units on average , so it is used to design approximated secure activation function . At the end of secure inference , the server has learned nothing but the client learns the inference result . More details of cryptographic primitives can be found in Appendix A.1 . 2.1 CRYPTOGRAPHIC INFERENCE .. Modern neural networks usually consist of linear convolution layers and non-linear activation layers . As Figure 1a shows , current state-of-the-art cryptographic inference , Delphi by Mishra et al . ( 2020 ) , Offline : . has an offline phase and an online phase . The offline phase is independent of input data and is used to prepare data for the subsequent online phase . Each phase has linear and non-linear operations . 1 . Offline linear layer . During the offline linear process , the client samples a random matrix rt that has the same shape with private input xt , and then sends its encryption [ rt ] to server . The server processes homomorphic convolutions and returns [ rt ] ∗Wt − [ ut ] to client , where Wt is the t-layer network weights and [ ut ] is a ciphertext of sampled matrix by server . The last step in the offline linear layer is that client obtains Ct = rt ·Wt − ut which is one part of secret sharing of xt ·Wt . 2 . Online linear layer . The online linear phase aims to let the server obtain St = ( xt−rt ) ·Wt+ut which is the other secret sharing part of xt ·Wt . It is almost as fast as the unencrypted computation , since the online input is a plaintext xt − rt . 3 . Offline Layer-Wise Activation layer . Delphi supports a layer-wise activation function where each activation layer either is ReLU based on GC or is the approximated polynomial based on BT . During the offline phase , GC needs to generate and share the garbled circuits . BT needs to generate and share the Beaver ’ s triples . 4 . Online Layer-Wise Activation layer . During the online phase of layer-wise activation , ReLU is either performed by GC or the approximated degree-2 polynomial . The latency of approximated activation implemented by BT is 192× smaller than ReLU based on GC . Latency Bottleneck and Motivation . Figure 1b shows our baseline Delphi suffers from long latency and low accuracy under coarse-grained layer-wise activation approximation . Specifically , 0L in Figure 1b means none of theReLU layers and 0 % ReLU units are approximated by polynomials . The activation latency takes 72.5 % of total latency . For only the online phase , activation latency occupies∼ 99 % of the online latency . Therefore , activation layers are the performance bottleneck . 0L also shows the all-ReLU model achieves 85.1 % accuracy . With increased approximation layer numbers , Delphi is able to improve the approximation ratios , but the inference accuracy is decreased at the same time . With the 84.5 % accuracy constraints , Delphi at most replaces 6-layer ReLU layers , with an approximation ratio of ∼ 42 % ReLU units . Figure 2 shows the reason why the approximation ratio is so low ( 42 % ) ; it is difficult to totally replace the ReLU units by polynomials in the bottleneck layer that has > 58 % ReLU units without a large decrease in accuracy . The ReLU units in modern networks are mainly located in the first few layers , and the ReLU numbers are usually decreased exponentially as shown in Figure 2 . Especially for much deeper neural networks on a large dataset , replacing the firstReLU layer significantly decreases the accuracy . To solve the above problems , we propose a more fine-grained channel-wise activation approximation method which is modelled as a hyper-parameter optimization problem . 2.2 POPULATION BASED TRAINING ( PBT ) .. Inspired from evolutionary algorithms , Population Based Training ( PBT ) proposed by Jaderberg et al . ( 2017 ) is a more efficient method to jointly optimize model weights and user-specified hyper-parameters automatically during training . Many Reinforcement Learning ( RL ) based hyperparameter optimization algorithms by Wang et al . ( 2019 ) and Lou et al . ( 2020 ) are not able to efficiently optimize hyper-parameters , since they simply stop training prematurely and consider partially trained accuracy as the final accuracy or reward . The details of PBT can be seen in ap- pendix A.2 and PBT by Jaderberg et al . ( 2017 ) . In this paper , we model the channel-wise activation approximation task as a hyper-parameter optimization problem . Features TAPAS Gazelle NASS CONAD Delphi CryptoNAS InstaHide SAFENet Strong Encryption 3 3 3 3 3 3 7 3 Batched HE 7 3 3 3 3 3 - 3 Optimized Activation 3 7 7 3 3 3 - 3 Channel-Wise 7 7 7 7 7 7 - 3 Mixed-Precision 7 7 7 7 7 7 - 3 Multiple-Degrees 7 7 7 7 7 7 - 3 Table 1 : Cryptographic inference works . 0 2 4 6 Layer 0 10 20 30 40 50 60 Ne ur on s ( % ) can not be fully approx . Figure 2 : Neurons ratio . 2.3 COMPARISON WITH PRIOR WORKS .. Table 1 shows a comparison between prior works and SAFENet . TAPAS by Sanyal et al . ( 2018 ) , XONN by Riazi et al . ( 2019 ) and soteria by Aggarwal et al . ( 2020 ) focus on binary neural networks which uses sign ( ) function instead of the ReLU activation , thereby suffering from inference accuracy loss . And TAPAS by Sanyal et al . ( 2018 ) and SHE by Lou & Jiang ( 2019 ) suffer from long-latency linear operations since they adapt a HE scheme called TFHE by Chillotti et al . ( 2018 ) that does not support ciphertext batching operations yet . For example , TAPAS and SHE take ∼2 hour and ∼10 seconds respectively to perform one single MNIST inference with 99 % accuracy . In contrast , our work SAFENet and Gazelle using the hybrid of batched HE and MPC are able to achieve < 1-second latency with > 99 % accuracy . Gazelle by Juvekar et al . ( 2018 ) and MiniONN prove the feasibility of the hybrid use of GC and HE , but they both suffer from enormous latency . NASS by Bian et al . ( 2020 ) , CONAD by Shafran et al . ( 2019 ) , and CryptoNAS by Ghodsi et al . ( 2020 ) try to improve Gazelle and MiniONN ’ s performance by the co-design of neural network architectures and cryptographic protocol , but they all require a heavy , online , cryptographic phase . Delphi by Mishra et al . ( 2020 ) reduces online latency by moving some online operations into the offline phase and replacing layer-wise activation by degree-2 polynomials . Only our SAFENet supports more fine-grained , channel-wise , activation approximation with multiple-degree polynomial exploration , shown in Table 1 . SAFENet also enables the mixed-precision approximation ratios for different layers . Other than MPC and HE based neural networks , InstaHide by Huang et al . ( 2020b ) and TextHide by Huang et al . ( 2020a ) use a class of subset-sum type encryption by Bhattacharyya et al . ( 2011 ) to protect the user ’ s sensitive data in machine learning service with only < 5 % computational overhead and little accuracy loss . However , an attack by Carlini et al . ( 2020 ) shows that there is a potential security risk on the InstaHide . Compared to the light-weight methods InstaHide and TextHide , MPC and HE provide much stronger security guarantees .

Summary: The main contribution of this paper is a new heuristic for identifying "less useful" activation channels. The authors then propose using simple approximations for activation functions for these channels without compromising network accuracy. The main novelty in the approximation used by the authors is flexibility in the degree of the polynomial approximation. Additionally, the authors propose a new hyper-parameter search strategy (BTPBT) to efficiently search the hyper-parameter space for the optimal approximation parameters.