CrypTen: Secure Multi-Party Computation Meets Machine Learning

1 Introduction . Secure multi-party computation ( MPC ; [ 30 , 69 ] ) allows parties to collaboratively perform computations on their combined data sets without revealing the data they possess to each other . This capability of secure MPC has the potential to unlock a variety of machine-learning applications that are currently infeasible because of data privacy concerns . For example , secure MPC can allow medical research institutions to jointly train better diagnostic models without having to share their sensitive patient data [ 27 ] or allow social scientists to analyze gender wage gap statistics without companies having to share sensitive salary data [ 42 ] . The prospect of such applications of machine learning with rigorous privacy and security guarantees has spurred a number of studies on machine learning via secure MPC [ 38 , 41 , 48 , 58 , 63 , 66 , 67 ] . However , at present , adoption of secure MPC in machine learning is still relatively limited considering its wide-ranging potential . One of the main obstacles to widespread adoption is that the complexity of secure MPC techniques puts them out of reach for most machine-learning researchers , who frequently lack in-depth knowledge of cryptographic techniques . To foster the adoption of secure MPC techniques in machine learning , we present CRYPTEN : a flexible software framework that aims to make modern secure MPC techniques accessible to machinelearning researchers and developers without a background in cryptography . Specifically , CRYPTEN 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . provides a comprehensive tensor-computation library in which all computations are performed via secure MPC . CRYPTEN ’ s API closely follows the API of the popular PyTorch framework for machine learning [ 54 , 55 ] , which makes it easy to use for machine-learning practitioners . For example , it provides automatic differentiation and a modular neural-network package . CRYPTEN assumes an semi-honest threat model [ 30 , §2.3.2 ] and works for an arbitrary number of parties . To make private training and inference efficient , CRYPTEN off-loads computations to the GPU and uses high-performance communication libraries to implement interactions between parties . The paper presents : ( 1 ) an overview of CRYPTEN ’ s design principles ; ( 2 ) a description of the design of CRYPTEN and of the secure MPC protocols implemented ; ( 3 ) a collection of benchmark experiments using CRYPTEN to run private versions of state-of-the-art models for text classification , speech recognition , and image classification ; and ( 4 ) a discussion of open problems and a roadmap for the further development of CRYPTEN . Altogether , the paper demonstrates that CRYPTEN ’ s flexible , PyTorch-like API makes private inference and training of modern machine-learning models easy to implement and efficient . For example , CRYPTEN allows two parties to privately classify an image [ 26 , 35 ] in 2-3 seconds , or to securely make phoneme predictions for 16kHz speech recordings [ 17 ] faster than real-time . We hope that CRYPTEN ’ s promising performance and ease-ofuse will foster the adoption of secure MPC by the machine-learning community , and pave the way for a new generation of secure and private machine-learning systems . 2 Related Work . CRYPTEN is part of a large body of work that develops secure MPC protocols for machine learning ; see Appendix ? ? . Most closely related to our work is CryptGPU [ 63 ] , which implements an 2-out-of-3 replicated secret sharing protocol [ 4 , 37 ] on top of CRYPTEN . Like CRYPTEN , CryptGPU provides security against semi-honest corruption , but it is limited to the three-party setting . CryptGPU is one of several protocols optimized for the three-party setting . For example , Falcon [ 67 ] implements a maliciously secure three-party MPC protocol , combining techniques from SecureNN [ 66 ] and ABY3 [ 48 ] . Falcon allows evaluation and training of convolutional networks such as AlexNet [ 40 ] and VGG [ 62 ] . Other systems that work in this setting include Astra [ 16 ] , Blaze [ 56 ] , and CrypTFlow [ 41 ] . There also exists a family of two-party systems that , like CRYPTEN , assume a semi-honest threat model . These systems include Gazelle [ 38 ] , Chameleon [ 58 ] , EzPC [ 15 ] , MiniONN [ 45 ] , SecureML [ 49 ] , PySyft [ 60 ] , and Delphi [ 47 ] . XONN [ 59 ] also works in the two-party setting but provides malicious security . Compared to these systems , CRYPTEN provides a more flexible machinelearning focused API1 that supports reverse-mode automatic differentiation , implements a rich set of functions , and natively runs on GPUs . Moreover , CRYPTEN supports a wider range of use cases by working with an arbitrary number of parties , and make communication between parties efficient via communication primitives that were optimized for high-performance distributed computing . 3 Design Principles . In the development of CRYPTEN , we adopted the following two main design principles : Machine-learning first API . CRYPTEN has a general purpose , machine-learning first API design . Most other secure MPC frameworks [ 34 ] adopt an API that stays close to the underlying MPC protocols . This hampers adoption of these frameworks in machine learning , for example , because they do not natively support tensor operations ( but only scalar operations ) and because they lack features that machine-learning researchers have come to expect , such as automatic differentiation . Instead , CRYPTEN implements the tensor-computation API of the popular PyTorch machine-learning framework [ 54 ] , implements reverse-mode automatic differentiation , provides a modular neuralnetwork package with corresponding learning routines , and supports GPU computations . We aim to allow developers to transition code from PyTorch to CRYPTEN by changing a single Python import . Eager execution . CRYPTEN adopts an imperative programming model . This is different from existing MPC frameworks , which generally implement compilers for their own domain-specific languages [ 34 ] . While compiler approaches have potential performance benefits , they slow down the 1CrypTFlow [ 41 ] also provides such an API by integrating deeply with TensorFlow [ 1 ] , but unlike CRYPTEN , it does not support PyTorch ’ s eager execution model [ 55 ] or GPU support . development cycle , make debugging harder , and prevent users from using arbitrary host-language constructs [ 3 ] . Instead , CRYPTEN follows the recent trend in machine learning away from graph compilers [ 1 ] to frameworks that eagerly execute computations [ 3 , 55 ] , providing a better developer experience . Yet , CRYPTEN is performant because it implements state-of-the-art secure MPC protocols ( for settings with arbitrary number of parties ) , because it uses PyTorch ’ s highly optimized tensor library for most computations , because computations can be off-loaded to the GPU , and because it uses communication libraries that were optimized for high-performance distributed computing . 4 Design Overview Figure 1 gives an overview of CRYPTEN ’ s design . Parties perform computations using efficient PyTorch tensor operations . Because secure MPC computations are integer computations that are not natively supported on GPUs , CRYPTEN maps between integer and floatingpoint computations on GPUs ; see Section 5.3 . The multi-party computations are implemented on arithmetic and binary secret shares [ 22 , 32 ] ; see Section 5.1 . Whereas many computations can be performed directly on arithmetic secret shares , others require conversion between arithmetic and binary secret shares ( A2B ) and back ( B2A ) ; see Section 5.2 . Some multiparty computations require interaction between parties via a communicator that employs the high-performance communication primitives in Gloo [ 31 ] and NCCL [ 51 ] . Some multi-party computations require Beaver triples [ 7 ] , which are supplied by a trusted third party ( TTP ) .2 All secure computations are wrapped in a CrypTensor object that implements the PyTorch tensor API and that provides reverse-mode automatic differentiation ( autograd ) to enable gradient-based training of arbitrary ( deep ) learning models . Figure 2 illustrates CrypTensor creation , i.e. , how tensors are secret-shared and revealed , as well as a simple computation ( addition ) . Note that each party involved in the multi-party computation executes the same code . Whenever communication between the parties is required ( e.g. , as part of private multiplications ) , the communication acts as a synchronization point between the parties . The crypten.init ( ) call is required once to establish the communication channel . In the example , the input tensor for the creation of the arithmetic secret share is provided party src=0 , which indicates the rank3 of the party that supplies the data to be secret-shared ( the other parties executing this code may provide None as input ) . 2CRYPTEN adopts a trusted third party for generating Beaver triples for efficiency reasons , but we are planning to add TTP-free solutions based on additive homomorphic encryption [ 52 ] or oblivious transfer [ 39 ] . 3CRYPTEN relies on MPI primitives for communication : each party knows their rank and the world size . To enable deep-learning use cases , CRYPTEN allows implementing neural networks following PyTorch ’ s API . Figure 3 shows how to create and encrypt neural networks and how to use automatic differentiation in CRYPTEN . The example assumes that some training sample and the associated target label are provided by the party with rank 0 ( note the value of src ) . As illustrated by the example , CRYPTEN ’ s API closely follows that of PyTorch . Indeed , it is possible to write a single training loop that can be used to train models using CRYPTEN or PyTorch without code changes . This makes it easy to adapt PyTorch code to use secure MPC for its computations , and it also makes debugging easier . The appendix presents a table listing all tensor functions that CrypTensor implements . To enable interoperability with existing machinelearning platforms , neural networks can be imported into CRYPTEN via ONNX . Figure 4 shows how a PyTorch model is imported into CRYPTEN . The example illustrates how CRYPTEN makes private inference with a ResNet-18 easy . The example in the figure also demonstrates CRYPTEN ’ s GPU support . One caveat is that all parties must use the same type of device ( i.e. , CPU or GPU ) for computations . 5 Secure Computations . To facilitate secure computations , CRYPTEN implements arithmetic secret sharing [ 22 , 23 ] and binary secret sharing [ 32 ] , as well as conversions between these two types of sharing [ 24 ] . Arithmetic secret sharing is particularly wellsuited for operations that are common in modern machine-learning models , such as matrix multiplications and convolutions . Binary secret sharing is required for evaluating certain other common functions , such as rectified linear units . We provide a high-level overview of CRYPTEN ’ s secure computation protocol here ; a detailed description is presented in the appendix . 5.1 Secret Sharing . Arithmetic secret sharing shares a scalar value x ∈ Z/QZ , where Z/QZ denotes a ring with Q elements , across parties p ∈ P . We denote the sharing of x by [ x ] = { [ x ] p } p∈P , where [ x ] p ∈ Z/QZ indicates party p ’ s share of x . The shares are constructed such that their sum reconstructs the original value x , that is , x = ∑ p∈P [ x ] p mod Q . To share a value x , the parties generate a pseudorandom zero-share [ 18 ] with |P| random numbers that sum to 0 . The party that possesses the value x adds x to their share and discards x . We use a fixed-point encoding to obtain x from a floating-point value , xR . To do so , we multiply xR with a large scaling factor B and round to the nearest integer : x = bBxRe , where B = 2L for some precision of L bits . To decode a value , x , we compute xR ≈ x/B . Binary secret sharing is a special case of arithmetic secret sharing that operates within the binary field Z/2Z . A binary secret share , 〈x〉 , of a value x is formed by arithmetic secret shares of the bits of x , setting Q=2 . Each party p ∈ P holds a share , 〈x〉p , such that x = ⊕ p∈P〈x〉p is satisfied . Conversion from [ x ] to 〈x〉 is implemented by having the parties create a binary secret share of their [ x ] p shares , and summing the resulting binary shares . Specifically , the parties create a binary secret share , 〈 [ x ] p〉 , of all the bits in [ x ] p. Subsequently , the parties compute 〈x〉 = ∑ p∈P〈 [ x ] p〉 using a carry-lookahead adder in log2 ( |P| ) log2 ( L ) communication rounds [ 14 , 21 ] . Conversion from 〈x〉 to [ x ] is achieved by computing [ x ] = ∑B b=1 2 b [ 〈x〉 ( b ) ] , where 〈x〉 ( b ) denotes the b-th bit of the binary share 〈x〉 and B is the total number of bits in the shared secret , 〈x〉 . To create an arithmetic share of a bit , the parties use secret shares , ( [ r ( b ) ] , 〈r ( b ) 〉 ) , of random bits r ( b ) . The random bits are provided by the TTP , but we plan to add an implementation that generates them off-line via oblivious transfer [ 39 ] . The parties use 〈r ( b ) 〉 to mask 〈x〉 ( b ) and reveal the resulting masked bit z ( b ) . Subsequently , they compute [ 〈x〉 ( b ) ] = [ r ( b ) ] + z ( b ) − 2 [ r ( b ) ] z ( b ) .

Summary: The authors present an implementation for secure computation of deep learning inference and training. It is based on PyTorch, which makes its use easy for programmers familiar with the underlying framework. The secure computation is semi-honest and uses a trusted third party, which is assumed not to collaborate with any of the other parties.