CrypTen: Secure Multi-Party Computation Meets Machine Learning

1 Introduction . Secure multi-party computation ( MPC ; [ 30 , 69 ] ) allows parties to collaboratively perform computations on their combined data sets without revealing the data they possess to each other . This capability of secure MPC has the potential to unlock a variety of machine-learning applications that are currently infeasible because of data privacy concerns . For example , secure MPC can allow medical research institutions to jointly train better diagnostic models without having to share their sensitive patient data [ 27 ] or allow social scientists to analyze gender wage gap statistics without companies having to share sensitive salary data [ 42 ] . The prospect of such applications of machine learning with rigorous privacy and security guarantees has spurred a number of studies on machine learning via secure MPC [ 38 , 41 , 48 , 58 , 63 , 66 , 67 ] . However , at present , adoption of secure MPC in machine learning is still relatively limited considering its wide-ranging potential . One of the main obstacles to widespread adoption is that the complexity of secure MPC techniques puts them out of reach for most machine-learning researchers , who frequently lack in-depth knowledge of cryptographic techniques . To foster the adoption of secure MPC techniques in machine learning , we present CRYPTEN : a flexible software framework that aims to make modern secure MPC techniques accessible to machinelearning researchers and developers without a background in cryptography . Specifically , CRYPTEN 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . provides a comprehensive tensor-computation library in which all computations are performed via secure MPC . CRYPTEN ’ s API closely follows the API of the popular PyTorch framework for machine learning [ 54 , 55 ] , which makes it easy to use for machine-learning practitioners . For example , it provides automatic differentiation and a modular neural-network package . CRYPTEN assumes an semi-honest threat model [ 30 , §2.3.2 ] and works for an arbitrary number of parties . To make private training and inference efficient , CRYPTEN off-loads computations to the GPU and uses high-performance communication libraries to implement interactions between parties . The paper presents : ( 1 ) an overview of CRYPTEN ’ s design principles ; ( 2 ) a description of the design of CRYPTEN and of the secure MPC protocols implemented ; ( 3 ) a collection of benchmark experiments using CRYPTEN to run private versions of state-of-the-art models for text classification , speech recognition , and image classification ; and ( 4 ) a discussion of open problems and a roadmap for the further development of CRYPTEN . Altogether , the paper demonstrates that CRYPTEN ’ s flexible , PyTorch-like API makes private inference and training of modern machine-learning models easy to implement and efficient . For example , CRYPTEN allows two parties to privately classify an image [ 26 , 35 ] in 2-3 seconds , or to securely make phoneme predictions for 16kHz speech recordings [ 17 ] faster than real-time . We hope that CRYPTEN ’ s promising performance and ease-ofuse will foster the adoption of secure MPC by the machine-learning community , and pave the way for a new generation of secure and private machine-learning systems . 2 Related Work . CRYPTEN is part of a large body of work that develops secure MPC protocols for machine learning ; see Appendix ? ? . Most closely related to our work is CryptGPU [ 63 ] , which implements an 2-out-of-3 replicated secret sharing protocol [ 4 , 37 ] on top of CRYPTEN . Like CRYPTEN , CryptGPU provides security against semi-honest corruption , but it is limited to the three-party setting . CryptGPU is one of several protocols optimized for the three-party setting . For example , Falcon [ 67 ] implements a maliciously secure three-party MPC protocol , combining techniques from SecureNN [ 66 ] and ABY3 [ 48 ] . Falcon allows evaluation and training of convolutional networks such as AlexNet [ 40 ] and VGG [ 62 ] . Other systems that work in this setting include Astra [ 16 ] , Blaze [ 56 ] , and CrypTFlow [ 41 ] . There also exists a family of two-party systems that , like CRYPTEN , assume a semi-honest threat model . These systems include Gazelle [ 38 ] , Chameleon [ 58 ] , EzPC [ 15 ] , MiniONN [ 45 ] , SecureML [ 49 ] , PySyft [ 60 ] , and Delphi [ 47 ] . XONN [ 59 ] also works in the two-party setting but provides malicious security . Compared to these systems , CRYPTEN provides a more flexible machinelearning focused API1 that supports reverse-mode automatic differentiation , implements a rich set of functions , and natively runs on GPUs . Moreover , CRYPTEN supports a wider range of use cases by working with an arbitrary number of parties , and make communication between parties efficient via communication primitives that were optimized for high-performance distributed computing . 3 Design Principles . In the development of CRYPTEN , we adopted the following two main design principles : Machine-learning first API . CRYPTEN has a general purpose , machine-learning first API design . Most other secure MPC frameworks [ 34 ] adopt an API that stays close to the underlying MPC protocols . This hampers adoption of these frameworks in machine learning , for example , because they do not natively support tensor operations ( but only scalar operations ) and because they lack features that machine-learning researchers have come to expect , such as automatic differentiation . Instead , CRYPTEN implements the tensor-computation API of the popular PyTorch machine-learning framework [ 54 ] , implements reverse-mode automatic differentiation , provides a modular neuralnetwork package with corresponding learning routines , and supports GPU computations . We aim to allow developers to transition code from PyTorch to CRYPTEN by changing a single Python import . Eager execution . CRYPTEN adopts an imperative programming model . This is different from existing MPC frameworks , which generally implement compilers for their own domain-specific languages [ 34 ] . While compiler approaches have potential performance benefits , they slow down the 1CrypTFlow [ 41 ] also provides such an API by integrating deeply with TensorFlow [ 1 ] , but unlike CRYPTEN , it does not support PyTorch ’ s eager execution model [ 55 ] or GPU support . development cycle , make debugging harder , and prevent users from using arbitrary host-language constructs [ 3 ] . Instead , CRYPTEN follows the recent trend in machine learning away from graph compilers [ 1 ] to frameworks that eagerly execute computations [ 3 , 55 ] , providing a better developer experience . Yet , CRYPTEN is performant because it implements state-of-the-art secure MPC protocols ( for settings with arbitrary number of parties ) , because it uses PyTorch ’ s highly optimized tensor library for most computations , because computations can be off-loaded to the GPU , and because it uses communication libraries that were optimized for high-performance distributed computing . 4 Design Overview Figure 1 gives an overview of CRYPTEN ’ s design . Parties perform computations using efficient PyTorch tensor operations . Because secure MPC computations are integer computations that are not natively supported on GPUs , CRYPTEN maps between integer and floatingpoint computations on GPUs ; see Section 5.3 . The multi-party computations are implemented on arithmetic and binary secret shares [ 22 , 32 ] ; see Section 5.1 . Whereas many computations can be performed directly on arithmetic secret shares , others require conversion between arithmetic and binary secret shares ( A2B ) and back ( B2A ) ; see Section 5.2 . Some multiparty computations require interaction between parties via a communicator that employs the high-performance communication primitives in Gloo [ 31 ] and NCCL [ 51 ] . Some multi-party computations require Beaver triples [ 7 ] , which are supplied by a trusted third party ( TTP ) .2 All secure computations are wrapped in a CrypTensor object that implements the PyTorch tensor API and that provides reverse-mode automatic differentiation ( autograd ) to enable gradient-based training of arbitrary ( deep ) learning models . Figure 2 illustrates CrypTensor creation , i.e. , how tensors are secret-shared and revealed , as well as a simple computation ( addition ) . Note that each party involved in the multi-party computation executes the same code . Whenever communication between the parties is required ( e.g. , as part of private multiplications ) , the communication acts as a synchronization point between the parties . The crypten.init ( ) call is required once to establish the communication channel . In the example , the input tensor for the creation of the arithmetic secret share is provided party src=0 , which indicates the rank3 of the party that supplies the data to be secret-shared ( the other parties executing this code may provide None as input ) . 2CRYPTEN adopts a trusted third party for generating Beaver triples for efficiency reasons , but we are planning to add TTP-free solutions based on additive homomorphic encryption [ 52 ] or oblivious transfer [ 39 ] . 3CRYPTEN relies on MPI primitives for communication : each party knows their rank and the world size . To enable deep-learning use cases , CRYPTEN allows implementing neural networks following PyTorch ’ s API . Figure 3 shows how to create and encrypt neural networks and how to use automatic differentiation in CRYPTEN . The example assumes that some training sample and the associated target label are provided by the party with rank 0 ( note the value of src ) . As illustrated by the example , CRYPTEN ’ s API closely follows that of PyTorch . Indeed , it is possible to write a single training loop that can be used to train models using CRYPTEN or PyTorch without code changes . This makes it easy to adapt PyTorch code to use secure MPC for its computations , and it also makes debugging easier . The appendix presents a table listing all tensor functions that CrypTensor implements . To enable interoperability with existing machinelearning platforms , neural networks can be imported into CRYPTEN via ONNX . Figure 4 shows how a PyTorch model is imported into CRYPTEN . The example illustrates how CRYPTEN makes private inference with a ResNet-18 easy . The example in the figure also demonstrates CRYPTEN ’ s GPU support . One caveat is that all parties must use the same type of device ( i.e. , CPU or GPU ) for computations . 5 Secure Computations . To facilitate secure computations , CRYPTEN implements arithmetic secret sharing [ 22 , 23 ] and binary secret sharing [ 32 ] , as well as conversions between these two types of sharing [ 24 ] . Arithmetic secret sharing is particularly wellsuited for operations that are common in modern machine-learning models , such as matrix multiplications and convolutions . Binary secret sharing is required for evaluating certain other common functions , such as rectified linear units . We provide a high-level overview of CRYPTEN ’ s secure computation protocol here ; a detailed description is presented in the appendix . 5.1 Secret Sharing . Arithmetic secret sharing shares a scalar value x ∈ Z/QZ , where Z/QZ denotes a ring with Q elements , across parties p ∈ P . We denote the sharing of x by [ x ] = { [ x ] p } p∈P , where [ x ] p ∈ Z/QZ indicates party p ’ s share of x . The shares are constructed such that their sum reconstructs the original value x , that is , x = ∑ p∈P [ x ] p mod Q . To share a value x , the parties generate a pseudorandom zero-share [ 18 ] with |P| random numbers that sum to 0 . The party that possesses the value x adds x to their share and discards x . We use a fixed-point encoding to obtain x from a floating-point value , xR . To do so , we multiply xR with a large scaling factor B and round to the nearest integer : x = bBxRe , where B = 2L for some precision of L bits . To decode a value , x , we compute xR ≈ x/B . Binary secret sharing is a special case of arithmetic secret sharing that operates within the binary field Z/2Z . A binary secret share , 〈x〉 , of a value x is formed by arithmetic secret shares of the bits of x , setting Q=2 . Each party p ∈ P holds a share , 〈x〉p , such that x = ⊕ p∈P〈x〉p is satisfied . Conversion from [ x ] to 〈x〉 is implemented by having the parties create a binary secret share of their [ x ] p shares , and summing the resulting binary shares . Specifically , the parties create a binary secret share , 〈 [ x ] p〉 , of all the bits in [ x ] p. Subsequently , the parties compute 〈x〉 = ∑ p∈P〈 [ x ] p〉 using a carry-lookahead adder in log2 ( |P| ) log2 ( L ) communication rounds [ 14 , 21 ] . Conversion from 〈x〉 to [ x ] is achieved by computing [ x ] = ∑B b=1 2 b [ 〈x〉 ( b ) ] , where 〈x〉 ( b ) denotes the b-th bit of the binary share 〈x〉 and B is the total number of bits in the shared secret , 〈x〉 . To create an arithmetic share of a bit , the parties use secret shares , ( [ r ( b ) ] , 〈r ( b ) 〉 ) , of random bits r ( b ) . The random bits are provided by the TTP , but we plan to add an implementation that generates them off-line via oblivious transfer [ 39 ] . The parties use 〈r ( b ) 〉 to mask 〈x〉 ( b ) and reveal the resulting masked bit z ( b ) . Subsequently , they compute [ 〈x〉 ( b ) ] = [ r ( b ) ] + z ( b ) − 2 [ r ( b ) ] z ( b ) .

Summary: This paper presents a general software framework with GPU support for secure machine learning over multiple (arbitrary number of) parties: CrypTen. It provides a friendly way for people who don't have any cryptography background to implement the secure machine learning tasks, which could tolerate all-but-one corruptions in semi-honest case. It also demonstrates its capability and efficiency in realizing the state-of-the-art models for text classification, speech recognition, and image classification.