Soteria: In search of efficient neural networks for private inference

1 INTRODUCTION . Machine learning models are susceptible to several security and privacy attacks throughout their training and inference pipelines . Defending each of these threats require different types of security mechanisms . One important requirement is that the sensitive input data as well as the trained model parameters remains confidential during the inference time . In this paper , we focus on private computation of inference over deep neural networks , which is the setting of machine learning-as-a-service . Consider a server that provides a machine learning service ( e.g. , classification ) , and a client who needs to use the service for an inference on her data record . The server is not willing to share the proprietary machine learning model , underpinning the service , with any client . The clients are also unwilling to share their sensitive private data with the server . In addition , we assume the two parties do not trust , nor include , any third entity in the protocol . In this setting , our first objective is to design a secure protocol that protects the confidentiality of client data as well as the prediction results against the server who runs the computation . The second objective is to preserve the confidentiality of the model parameters with respect to the client . These can be achieved using secure multi-party computations . We consider an honest-but-curious threat model . A number of techniques provide data confidentiality while computing and thereby allow private computation . The techniques include computation on trusted processors such as Intel SGX Hunt et al . ( 2018a ) ; Ohrimenko et al . ( 2016 ) , and computation on encrypted data , using homomorphic encryption , garbled circuits , secret sharing , and hybrid cryptographic approaches that jointly optimize the efficiency of private inference on neural networks Yao ( 1986 ) ; Gentry ( 2009b ) ; Yao ( 1982 ) ; Brakerski et al . ( 2011 ) ; Beaver ( 1992 ) . To provide private inference with minimal performance overhead and accuracy loss , the dominant line of research involves adapting cryptographic functions to ( an approximation of ) a given fixed model Liu et al . ( 2017 ) ; Mohassel & Zhang ( 2017 ) ; Mishra et al . ( 2020 ) ; Rouhani et al . ( 2018 ) ; Riazi et al . ( 2019 ) ; Chandran et al . ( 2019 ) ; Juvekar et al . ( 2018 ) ; Riazi et al . ( 2018 ) . However , the alternative approach of searching or designing a network architecture for a given set of efficient and known cryptographic primitives has not received much attention Ghodsi et al . ( 2020 ) . Note that secure multi-party computations protect the confidentiality of the inputs to the protocol , and its intermediate computations , and can not prevent inference attacks that exploit the output of the protocol to infer information about its inputs . We thus emphasize that the protection against the indirect inference attacks that aim at reconstructing model parameters Tramèr et al . ( 2016 ) or its training data Shokri et al . ( 2017 ) , by exploiting model predictions , is not our goal and out of scope for this work . Our Contributions . We approach the problem of private inference from a novel perspective where instead of modifying cryptographic schemes to support neural network computations , we advocate modification of the training algorithms for efficient cryptographic primitives . In a nutshell , we advocate co-designing and co-optimizing models and cryptographic operations for efficient private inference . Research has shown that training algorithms for deep learning are inherently flexible with respect to their neural network architecture . This means that different network configurations can achieve similar level of prediction accuracy . We exploit this fact about deep learning algorithms and investigate the problem of optimizing deep learning algorithms to ensure efficient private computation . To this end , we present SOTERIA — an approach for constructing deep neural networks optimized for performance , accuracy and confidentiality . Although SOTERIA could leverage any of the available cryptographic primitives or their combination , we select garbled circuits as its main building block to address the confidentiality concern . Garbled circuits ( GC ) are known to be efficient and allow generation of constant depth circuits even for non-linear functions . We show that neural network algorithms can be optimized to efficiently execute garbled circuits while achieving high accuracy guarantees . We observe that the efficiency of evaluating an inference circuit depends on two key factors : the model parameters and the network structure . With this observation , we design a regularized architecture search algorithm to construct neural networks . SOTERIA selects optimal parameter sparsity and network structure with the objective to guarantee a balance between performance and model accuracy . We summarize our contributions below : • We propose a neural architecture search based approach called SOTERIA for designing models that guarantee a balance between prediction utility and computation cost while maintaining confidentiality of the model parameters and the input during inference . • We build SOTERIA models using garbled circuits for both ternary neural networks as well as the model architecture identified using network architecture search approach . • We evaluate SOTERIA on the benchmark datasets used by all the prior work , and observe that our approach provides the flexibility to tune utility and efficiency parameters based on the requirement of the underlying scenario . Our complete anonymous source code is available on github.1 The two fields of research on neural architecture search Liu et al . ( 2019 ) ; Zoph & Le ( 2016 ) ; Elsken et al . ( 2018 ) and secure multi-party computation Yao ( 1986 ) ; Gentry ( 2009b ) ; Yao ( 1982 ) ; Brakerski et al . ( 2011 ) ; Beaver ( 1992 ) are rapidly growing . SOTERIA enables us to take advantage of the advancements in these fields to systematically design more efficient algorithms for complex machine learning tasks . 2 SOTERIA . We design SOTERIA to automatically learn the model architecture and its connections so as to optimize the cost of private inference in addition to optimizing accuracy . This approach is different than simply fine-tunning or compressing a model , as we aim to include the cost of private computation as part of the objective of architecture learning and parameter learning of the model . SOTERIA is built on top of two well-established classes of machine learning algorithms : neural architecture search algorithms , and ternary neural network algorithms . Neural architecture search for efficient private inference . Architecture search algorithms for neural networks are designed to replace the manual design of complex deep models . The objective is to learn a model structure that gives high accuracy when trained on the training set Elsken et al . 1Source code of SOTERIA : https : //github.com/SoteriaAnonymous/Soteria ( 2018 ) ; Zoph & Le ( 2016 ) ; Pham et al . ( 2018 ) ; Liu et al . ( 2019 ) . They automatically construct the model architecture by stacking a number of cells . Each cell is a directed acyclic graph , where each node is a neural operation ( e.g. , convolution with different dimensions , maxpool , identity ) . During the search algorithm , we use stochastic gradient descent to continuously update the scores associated with different candidate operations for each connection in the internal graph of a cell , as to maximize the accuracy of the model on some validation dataset . In SOTERIA , we regularize the computation of the connection scores over candidate operations , with factor λ . We penalize each operation proportional to its computation and communication overhead when garbled . Larger values of λ result in models that prefer efficiency of private inference over accuracy . As we balance the trade-off between accuracy and performance , SOTERIA can construct models which by design satisfy the requirements of our system . Ternary ( Sparse Binary ) Neural Network . To build a system that enables efficient private inference , we aim to reduce the number of parameters contained in the model . One approach to accomplish this is to first train the model , and then compress it afterwards . However , this method might not result in the highest accuracy that could be achieved if we were to construct the model for this purpose from the beginning . To better align with our approach of constructing model architectures , we aim to learn model structures which are accurate , sparse , and are efficient when implemented as garbled circuits . More specifically , in SOTERIA , we train models with ternary parameters ( −1 , 0 , +1 ) Li et al . ( 2016 ) and binary activation functions . This allows us to leverage the benefits for BNNs , as discussed in Section B , with the benefit of a potentially smaller circuit ( due to the model ’ s sparsity ) . We incorporate Ternary Neural Networks , or TNNs , into our regularized architecture search algorithm to find cells containing only ternary convolution and max-pooling layers that operate on binary inputs and ternary parameters . 3 EMPIRICAL EVALUATION . In this section , we evaluate the efficiency of our method in two main ways . First , we show how using ternary neural networks on fixed model architectures , as used in the prior work Riazi et al . ( 2019 ) ; Mohassel & Zhang ( 2017 ) ; Rouhani et al . ( 2018 ) ; Juvekar et al . ( 2018 ) , can reduce the overhead of secure inference on sparse neural networks . Second , we present the performance of SOTERIA architectures , in which model complexity is optimized along with the model accuracy . Implementation . We first build a representation of the model and parameters in SystemVerilog and then convert the model into a digital circuit supported by TinyGarble Songhori et al . ( 2015 ) , and implementation of the Garbled Circuit ( GC ) protocol . We synthesize and optimize our circuit ( using Synopsys Design Compiler ) to use circuit elements supported by TinyGarble . In the first step , we design a collection of parameterized components ( notably dot product , and maxpool ) to use as building blocks in our architecture search algorithm . Each component is flexible and can efficiently accept arbitrary size input and output , and is composed to form the complete model . In a general setting , hardware-level code is typically straight-forward to generate . However , to enable TNNs with SOTERIA , we have to dynamically define the sparsity of the modules depending on the result of model training and architecture search . Along with the parameter data , we define the sparsity information which is used during generate phases in SystemVerilog to build the sparse network in hardware ( taking advantage of the 0-valued parameters ) . Altogether , this allows us to build and evaluate SOTERIA models . 3.1 EXPERIMENTAL SETUP . We need to evaluate SOTERIA on datasets which are used in the prior work to be able to perform a comprehensive comparison between the methods ( see Table 3 for all the configurations ) . We evaluate our work on MNIST and CIFAR10 image classification datasets , as they are the only benchmark datasets which are extensively used in the literature to evaluate the performance of cryptographically secure neural network schemes . We run our experiments on an AWS c5.2xlarge instance , running Ubuntu 18.04 LTS on an Intel Xeon 8124M at 3.0 GHz . We use PyTorch 1.3 Pyt , a deep learning framework to implement our architecture search algorithm and train the ternary models . We use Synopsys Design Compiler Syn , version L-2016.03-SP5-2 , to synthesize SystemVerilog code into the gate-level netlist . Our synthesis runs the TinyGarble gate library infrastructure2 . We compute the number of non-XOR gates in the generated boolean circuit netlist as a measure of its complexity . We measure the exact performance of SOTERIA as its runtime during the offline and online phases of the protocol , and its communication cost . We present the experimental setup of prior work and SOTERIA , including their CPU specification and link to available software codes , in Table 7 . We also present the details of all neural network architectures which we evaluate in this paper in Table 6 .

Summary: The paper aims at developing an efficient neural network architecture with reduced computational cost under a cryptographic primitive where data on the client side and model on the server side are kept confidential. The motivation is that the existing works have focused on developing cryptographic techniques in a fixed network architecture and have not considered the neural network optimization perspective to enhance the efficiency of existing cryptographic computations. The paper then leverages the flexibility of the network structure while ensuring comparable accuracy and tries to find an architecture that provides a significant reduction in runtime while incurring negligible impact on prediction accuracy. Garbled circuits are chosen as the cryptographic primitive due to its compatibility with a wide range of computations, including non-linear functions. The paper then investigates how to reduce the private inference cost by incorporating two protocols: first one is neural architecture search for efficient private inference where a neural operation is penalized based on its computation and communication overhead, and the second one is to train models with ternary parameters which allows reducing the number of model parameters by introducing sparsity. Sparsity leads to a reduction in computation and communication overhead. However, to maintain a certain level of accuracy, the network needs to be scaled which incurs an increase in the runtime. Accordingly, a trade-off between efficiency and accuracy has been demonstrated with experiments on the CIFAR and MNIST datasets for varying regularization parameter and scaling factor.