HD-cos Networks: Efficient Neural Architechtures for Secure Multi-Party Computation

1 INTRODUCTION . Machine learning models are often trained with user data that may contain private information . For example , in healthcare patients diagnostics contain sensitive information and in financial sectors , user data contains potentially private information such as salaries and taxes . In these applications , storing the user data in plain text format at a centralized server can be privacy invasive . There have been several efforts to design secure and private ways of learning and inferring machine learning models . In this work , we focus on secure multi-party computation ( MPC ) , a branch of cryptography that allows parties to collaboratively perform computations on data sets without revealing the data they possess to each other ( Evans et al. , 2017 ) . Recently , there are several research papers that proposed to train and infer machine learning models in a secure fashion via MPC ( Gilad-Bachrach et al. , 2016 ; Graepel et al. , 2012 ; Obla et al. , 2020 ) . Loosely speaking , in the MPC setup , a piece of sensitive data is split into multiple shards called secret shares , and each secret share is stored with a different party . These parties are further chosen such that their fundamental interests are to protect sensitive data and hence can be viewed as non-colluding . The cryptography guarantee states that unless all the parties ( or at least k out of all parties , depending on the cryptographic algorithm design ) collude , sensitive data can not be reconstructed and revealed to anyone/anything . Training in the MPC setup is challenging due to several reasons . Firstly , only limited data types ( e.g . integer ) and/or operations ( e.g . integer addition and multiplication ) are natively supported in most MPC algorithms . This increases the complexity to support non-trivial operations over MPC . Secondly , since data is encrypted into multiple secret shares and stored in multiple parties , one share per party , directly training a machine learning model on this data can be expensive , both in terms of communication between the servers and computation at the individual servers . More concretely if simple operations like addition and multiplications take orders of nanoseconds in the normal computation scenarios , in the MPC setup they can take milliseconds or more , if the operation requires the parties in the MPC setup to communicate with each other . Furthermore , one of the key bottlenecks of secret sharing mechanisms is that most non-linear operations e.g. , ReLU ( x ) = max ( 0 , x ) , can not be efficiently computed . In this paper , we address these questions by proposing a general network construct that can be implemented in MPC setup efficiently . Our proposal consists of two parts : 1 ) use the cosine function as the activation function , and 2 ) use a structured weight matrix based on the Hadamard transform in place of the standard fully connected layer . We provide an algorithm to compute cosine under twoparty computation ( 2PC ) setup . Unlike ReLU which involves multiple rounds of communication , our proposed algorithm for cosine requires only two online rounds of communication between the two computation servers . The use of the proposed Hadamard transform for weight matrices means that the number of parameters in each dense layer scales linearly with the input dimenion , as opposed to the standard dense layer which scales quadratically . We demonstrate on a number of challenging datasets that the combination of these two constructs leads to a model that is as accurate as the commonly used ReLU-based model with fully connected layers . The rest of the paper is organized as follows . We first overview multiparty computation in Section 2 and then overview related works in Section 3 . We present the cosine activation function and structured matrix transformation with theoretical motivations and analysis of their computational efficiency in Section 4 and Section 5 . We then provide extensive experimental evaluations on several datasets in Section 6 . 2 MULTI-PARTY COMPUTATION . Secure multi-party computation ( MPC ) is a branch of cryptography that allows two or more parties to collaboratively perform computations on data sets without revealing the data they possess to each other ( Evans et al. , 2017 ) . Following earlier works ( Liu et al. , 2017 ; Mohassel & Zhang , 2017 ; Kelkar et al. , 2021 ) , we focus on the two party computation ( 2PC ) setup . Our results can be extended to multi-party setup by existing mechanisms . If the two parties are non-colluding , then 2PC setup guarantees that the two parties will not learn anything from the computation process and hence there is no data leak . During training in the 2PC setup , each party receives features and labels of the training dataset in the form of secret shares . They compute and temporarily store all intermediate results in the form of secret shares . Thus during training , both the servers collaboratively learn a machine learning model , which again is split between two parties . Upon completion of the training process , the final result , i.e . the ML model itself , composed of trained parameters , are in secret shares to be held by each party in the 2PC setup . At prediction time , each party receives features in secret shares , and performs the prediction where all intermediate results are in secret shares . The MPC cluster sends the prediction results in secret share back to the caller who provided the features for prediction . The caller can combine all shares of the secret prediction result into its plaintext representation . In this entire training/prediction process , the MPC cluster does not learn any sensitive data . While MPC may provide security/privacy guarantee and is Turing complete , it might be significantly slower than equivalent plaintext operation . The overall performance of MPC is determined by the computation and communication cost . The computation cost in MPC is typically higher than the cost of equivalent operations in cleartext . The bigger bottleneck is the communication cost among the parties in the MPC setup . The communication cost has three components . • Number of rounds : the number of times that parties in the MPC setup need to communicate/synchronize with each other to complete the MPC crypto protocol . For 2PC , this is often equivalent to the number of Remote Procedure Calls ( or RPCs ) that the two parties need per the crypto protocol design . Many MPC algorithms differentiate offline rounds vs. online rounds , where the former is input-independent and can be performed asynchronously in advance , and the latter is input-dependent , is on the critical path for the computation and must be performed synchronously . For example , addition of additive secret shares requires no online rounds , whereas multiplication of additive secret shares requires one online round . • Network bandwidth : the number of bytes that parties in the MPC setup need to send to each other to complete the MPC crypto protocol . For 2PC , each RPC between the two parties has a request and response . The bandwidth cost is the sum of all the bytes to be transmitted in the request and response per the crypto protocol . • Network latency : the network latency is a property of the network connecting all parties in the MPC setup . It depends on the network technology ( e.g . 10 Gigabit Ethernet or 10GE ) , network topology , as well as applicable network Quality of Service ( QoS ) settings and the network load . In this work , we propose neural networks which can be implemented with a few online rounds of communication and little network bandwidth . Following earlier works , We consider neural network architectures where the majority of the parameters are on the fully connected layers . • We propose and systematically study using cosine as the activation and demonstrate that it achieves comparable performance to existing activation and can be efficiently implemented with two online rounds . • We show that dense matrices in neural networks can be replaced by structured matrices , which have comparable performance to existing neural network architectures and can be efficiently implemented in MPC setup by reducing the bandwidth . We show that by using structured matrices , we can reduce the number of per-layer secure multiplications from O ( d2 ) to O ( d ) , where d is the layer width , thus reducing the memory bandwidth . 3 RELATED WORKS . Neural network inference under MPC . Barni et al . ( 2006 ) considered inference of neural networks in the MPC setup , where the linear computations are done at the servers in the encrypted field and the non-linear activations are computed at the clients directly in plaintext . The main caveat of this approach is that , to compute a L layer neural network , L rounds of communication is required between server and clients which can be prohibitive and furthermore intermediate results are leaked to the clients , which may not be desired . To overcome the information leakage , Orlandi et al . ( 2007 ) proposed methods to hide the results of the intermediate data from the clients ; the method still requires multiple rounds of communication . Liu et al . ( 2017 ) proposed algorithms that allows for evaluating arbitrary neural networks ; the intermediate computations ( e.g. , sign ( x ) ) are much more expensive than summations and multiplications . Efficient activation functions . Since inferring arbitrary neural networks can be inefficient , several papers have proposed different activation functions which are easy to compute . Gilad-Bachrach et al . ( 2016 ) proposed to use the simple square activation function . They also proposed to use mean-pooling instead of max-pooling . Chabanne et al . ( 2017 ) also noticed the limited accuracy guarantees of the square function and proposed to approximate ReLU with a low degree polynomial and added a batch normalization layer to improve accuracy . Wu et al . ( 2018 ) proposed to train a polynomial as activation to improve the performance . Obla et al . ( 2020 ) proposed a different algorithm for approximating activation methods and showed that it achieves superior performance on several image recognition datasets . Recently Knott et al . ( 2021 ) released a library for training and inference of machine learning models in the multi-party setup . Cosine as activation function . Related to using cosine as the activation function , Yu et al . ( 2015 ) proposed the Compact Nonlinear Maps method which is equivalent of using cosine as the activation function for neural networks with one hidden layer . Parascandolo et al . ( 2016 ) studied using the sine function as activation . Xie et al . ( 2019 ) used cosine activation in deep kernel learning . Noel et al . ( 2021 ) proposed a variant of the cosine function called Growing Cosine Unit x cos ( x ) and shows that it can speed up training and reduce parameters in convolutions neural networks . All the above works are not under the MPC setup . Training machine learning models under MPC . Graepel et al . ( 2012 ) proposed to use training algorithms that can be expressed as low degree polynomials , so that the training phase can be done over encrypted data . Aslett et al . ( 2015 ) proposed algorithms to train models such as random forests over training data . Mohassel & Zhang ( 2017 ) proposed a 2PC setup for training and inference various types of models including neural networks where data is distributed to two non-colluding servers . Kelkar et al . ( 2021 ) proposed an efficient algorithm for Poisson regression by adding a secure exponentiation primitive . Combining with differential privacy . We note that while the focus of this work is multi-party computation , it can be combined with other privacy preserving techniques such as differential privacy ( Dwork et al. , 2014 ) . There are some recent works which combine MPC with differential privacy ( Jayaraman & Wang , 2018 ) . Systematically evaluating performances with the combination of our proposed technique and that of differential privacy remains an interesting future direction .

Summary: The paper proposes using a cosine activation function and Hadamard-Diagonal transformation as a means to improve the efficiency of MPC for machine learning. The paper considers a two-server model, where the training computation are carried out by two non-colluding parties. Experiments are provided to demonstrate the improvement on the test accuracy of the proposed approach over other baselines that use different activation functions.