HD-cos Networks: Efficient Neural Architechtures for Secure Multi-Party Computation

1 INTRODUCTION . Machine learning models are often trained with user data that may contain private information . For example , in healthcare patients diagnostics contain sensitive information and in financial sectors , user data contains potentially private information such as salaries and taxes . In these applications , storing the user data in plain text format at a centralized server can be privacy invasive . There have been several efforts to design secure and private ways of learning and inferring machine learning models . In this work , we focus on secure multi-party computation ( MPC ) , a branch of cryptography that allows parties to collaboratively perform computations on data sets without revealing the data they possess to each other ( Evans et al. , 2017 ) . Recently , there are several research papers that proposed to train and infer machine learning models in a secure fashion via MPC ( Gilad-Bachrach et al. , 2016 ; Graepel et al. , 2012 ; Obla et al. , 2020 ) . Loosely speaking , in the MPC setup , a piece of sensitive data is split into multiple shards called secret shares , and each secret share is stored with a different party . These parties are further chosen such that their fundamental interests are to protect sensitive data and hence can be viewed as non-colluding . The cryptography guarantee states that unless all the parties ( or at least k out of all parties , depending on the cryptographic algorithm design ) collude , sensitive data can not be reconstructed and revealed to anyone/anything . Training in the MPC setup is challenging due to several reasons . Firstly , only limited data types ( e.g . integer ) and/or operations ( e.g . integer addition and multiplication ) are natively supported in most MPC algorithms . This increases the complexity to support non-trivial operations over MPC . Secondly , since data is encrypted into multiple secret shares and stored in multiple parties , one share per party , directly training a machine learning model on this data can be expensive , both in terms of communication between the servers and computation at the individual servers . More concretely if simple operations like addition and multiplications take orders of nanoseconds in the normal computation scenarios , in the MPC setup they can take milliseconds or more , if the operation requires the parties in the MPC setup to communicate with each other . Furthermore , one of the key bottlenecks of secret sharing mechanisms is that most non-linear operations e.g. , ReLU ( x ) = max ( 0 , x ) , can not be efficiently computed . In this paper , we address these questions by proposing a general network construct that can be implemented in MPC setup efficiently . Our proposal consists of two parts : 1 ) use the cosine function as the activation function , and 2 ) use a structured weight matrix based on the Hadamard transform in place of the standard fully connected layer . We provide an algorithm to compute cosine under twoparty computation ( 2PC ) setup . Unlike ReLU which involves multiple rounds of communication , our proposed algorithm for cosine requires only two online rounds of communication between the two computation servers . The use of the proposed Hadamard transform for weight matrices means that the number of parameters in each dense layer scales linearly with the input dimenion , as opposed to the standard dense layer which scales quadratically . We demonstrate on a number of challenging datasets that the combination of these two constructs leads to a model that is as accurate as the commonly used ReLU-based model with fully connected layers . The rest of the paper is organized as follows . We first overview multiparty computation in Section 2 and then overview related works in Section 3 . We present the cosine activation function and structured matrix transformation with theoretical motivations and analysis of their computational efficiency in Section 4 and Section 5 . We then provide extensive experimental evaluations on several datasets in Section 6 . 2 MULTI-PARTY COMPUTATION . Secure multi-party computation ( MPC ) is a branch of cryptography that allows two or more parties to collaboratively perform computations on data sets without revealing the data they possess to each other ( Evans et al. , 2017 ) . Following earlier works ( Liu et al. , 2017 ; Mohassel & Zhang , 2017 ; Kelkar et al. , 2021 ) , we focus on the two party computation ( 2PC ) setup . Our results can be extended to multi-party setup by existing mechanisms . If the two parties are non-colluding , then 2PC setup guarantees that the two parties will not learn anything from the computation process and hence there is no data leak . During training in the 2PC setup , each party receives features and labels of the training dataset in the form of secret shares . They compute and temporarily store all intermediate results in the form of secret shares . Thus during training , both the servers collaboratively learn a machine learning model , which again is split between two parties . Upon completion of the training process , the final result , i.e . the ML model itself , composed of trained parameters , are in secret shares to be held by each party in the 2PC setup . At prediction time , each party receives features in secret shares , and performs the prediction where all intermediate results are in secret shares . The MPC cluster sends the prediction results in secret share back to the caller who provided the features for prediction . The caller can combine all shares of the secret prediction result into its plaintext representation . In this entire training/prediction process , the MPC cluster does not learn any sensitive data . While MPC may provide security/privacy guarantee and is Turing complete , it might be significantly slower than equivalent plaintext operation . The overall performance of MPC is determined by the computation and communication cost . The computation cost in MPC is typically higher than the cost of equivalent operations in cleartext . The bigger bottleneck is the communication cost among the parties in the MPC setup . The communication cost has three components . • Number of rounds : the number of times that parties in the MPC setup need to communicate/synchronize with each other to complete the MPC crypto protocol . For 2PC , this is often equivalent to the number of Remote Procedure Calls ( or RPCs ) that the two parties need per the crypto protocol design . Many MPC algorithms differentiate offline rounds vs. online rounds , where the former is input-independent and can be performed asynchronously in advance , and the latter is input-dependent , is on the critical path for the computation and must be performed synchronously . For example , addition of additive secret shares requires no online rounds , whereas multiplication of additive secret shares requires one online round . • Network bandwidth : the number of bytes that parties in the MPC setup need to send to each other to complete the MPC crypto protocol . For 2PC , each RPC between the two parties has a request and response . The bandwidth cost is the sum of all the bytes to be transmitted in the request and response per the crypto protocol . • Network latency : the network latency is a property of the network connecting all parties in the MPC setup . It depends on the network technology ( e.g . 10 Gigabit Ethernet or 10GE ) , network topology , as well as applicable network Quality of Service ( QoS ) settings and the network load . In this work , we propose neural networks which can be implemented with a few online rounds of communication and little network bandwidth . Following earlier works , We consider neural network architectures where the majority of the parameters are on the fully connected layers . • We propose and systematically study using cosine as the activation and demonstrate that it achieves comparable performance to existing activation and can be efficiently implemented with two online rounds . • We show that dense matrices in neural networks can be replaced by structured matrices , which have comparable performance to existing neural network architectures and can be efficiently implemented in MPC setup by reducing the bandwidth . We show that by using structured matrices , we can reduce the number of per-layer secure multiplications from O ( d2 ) to O ( d ) , where d is the layer width , thus reducing the memory bandwidth . 3 RELATED WORKS . Neural network inference under MPC . Barni et al . ( 2006 ) considered inference of neural networks in the MPC setup , where the linear computations are done at the servers in the encrypted field and the non-linear activations are computed at the clients directly in plaintext . The main caveat of this approach is that , to compute a L layer neural network , L rounds of communication is required between server and clients which can be prohibitive and furthermore intermediate results are leaked to the clients , which may not be desired . To overcome the information leakage , Orlandi et al . ( 2007 ) proposed methods to hide the results of the intermediate data from the clients ; the method still requires multiple rounds of communication . Liu et al . ( 2017 ) proposed algorithms that allows for evaluating arbitrary neural networks ; the intermediate computations ( e.g. , sign ( x ) ) are much more expensive than summations and multiplications . Efficient activation functions . Since inferring arbitrary neural networks can be inefficient , several papers have proposed different activation functions which are easy to compute . Gilad-Bachrach et al . ( 2016 ) proposed to use the simple square activation function . They also proposed to use mean-pooling instead of max-pooling . Chabanne et al . ( 2017 ) also noticed the limited accuracy guarantees of the square function and proposed to approximate ReLU with a low degree polynomial and added a batch normalization layer to improve accuracy . Wu et al . ( 2018 ) proposed to train a polynomial as activation to improve the performance . Obla et al . ( 2020 ) proposed a different algorithm for approximating activation methods and showed that it achieves superior performance on several image recognition datasets . Recently Knott et al . ( 2021 ) released a library for training and inference of machine learning models in the multi-party setup . Cosine as activation function . Related to using cosine as the activation function , Yu et al . ( 2015 ) proposed the Compact Nonlinear Maps method which is equivalent of using cosine as the activation function for neural networks with one hidden layer . Parascandolo et al . ( 2016 ) studied using the sine function as activation . Xie et al . ( 2019 ) used cosine activation in deep kernel learning . Noel et al . ( 2021 ) proposed a variant of the cosine function called Growing Cosine Unit x cos ( x ) and shows that it can speed up training and reduce parameters in convolutions neural networks . All the above works are not under the MPC setup . Training machine learning models under MPC . Graepel et al . ( 2012 ) proposed to use training algorithms that can be expressed as low degree polynomials , so that the training phase can be done over encrypted data . Aslett et al . ( 2015 ) proposed algorithms to train models such as random forests over training data . Mohassel & Zhang ( 2017 ) proposed a 2PC setup for training and inference various types of models including neural networks where data is distributed to two non-colluding servers . Kelkar et al . ( 2021 ) proposed an efficient algorithm for Poisson regression by adding a secure exponentiation primitive . Combining with differential privacy . We note that while the focus of this work is multi-party computation , it can be combined with other privacy preserving techniques such as differential privacy ( Dwork et al. , 2014 ) . There are some recent works which combine MPC with differential privacy ( Jayaraman & Wang , 2018 ) . Systematically evaluating performances with the combination of our proposed technique and that of differential privacy remains an interesting future direction .

Summary: MPC is a cryptographic technique to allow multiple party to jointly compute a protocol without leaking sensitive data, but building blocks in the neural network converted to MPC setup usually suffer from heavy communication overhead among parties, and jointly training ML models is also computationally expensive. So, in this work, authors propose an efficient MPC-based neural network. The network consists of cosine function as activation function under 2PC setup and linear transformations by Hadamard-Diagonal method.