Goten: GPU-Outsourcing Trusted Execution of Neural Network Training and Prediction

1 INTRODUCTION . While deep neural networks ( DNN ) can produce predictive models with unparalleled performance , its training phase requires enormous data as input . A single data owner may not possess enough data to train a good DNN . Multiple data owners , say , financial institutions , may want to collaborate in training DNNs . Yet , they are often expected to protect the privacy of the data contributors . This discourages any collaborative training over global-scale data that is otherwise promising ( Cheng et al. , 2019 ) . Moreover , to perform prediction using a trained model , queriers need to submit their own private data ( e.g. , medical history ) . Meanwhile , the model owners want to protect the confidentiality of the trained model in the prediction phase as well . The exposure of the ( parameters of a ) model ( to queriers or a third-party cloud server ) may reveal information about its training data ( Fredrikson et al. , 2015 ) , deterring the participation of data contributors . Also , the model itself is of high commercial value . These concerns hinder the deployment of prediction as a service . An increasingly popular approach to ensure privacy is using a trusted execution environment ( TEE ) ( Cheng et al. , 2019 ; Tramèr & Boneh , 2019 ) and in particular , trusted processors , e.g. , Intel Software Guard Extension ( SGX ) . When a data provider sends some private data to a server equipped with SGX , it can initialize an enclave to receive the data in a confidential and authenticated way and subsequently operate on them . Even the untrusted server , who physically owns the enclave , can not read or tamper the data inside the enclave . This paper investigates the following questions : Can we support DNN training ( and prediction ) by using SGX and untrusted GPU while still preserving the privacy of all stakeholders ? If so , how much speedup do we gain by using GPU ? 1.1 OUR BASELINE APPROACH : CAFFESCONE . Arnautov et al . ( 2016 ) propose SCONE , a secure container mechanism that allows developers to directly run applications in an SGX enclave with almost zero code change1 . We combine SCONE with Caffe ( Jia et al. , 2014 ) , an efficient open-source DNN framework , to build our baseline privacypreserving DNN framework – CaffeSCONE . Beyond demonstrating what one can get by applying a generic solution that uses SGX ( SCONE ) for training ( not supported by Slalom ) , our CaffeSCONE implementation enables more benchmarking for insight in possible improvements , which are eventually achieved by our main result ( hence further optimizing it is not our goal ) . For one , we show ( in Section 4.2 ) that this baseline approach greatly suffers when the enclave ’ s memory limit is reached . Specifically , it invokes a native paging mechanism to swap data in and out , which further requires en/decryption . Also , we found that using more threads and cores can not improve performance . 1.2 OUR PROPOSED FRAMEWORK : GOTEN . Secure Outsourcing to GPU By using SGX solely , CaffeSCONE is already orders of magnitude faster than the state-of-the-art cryptographic solutions ( SecureML ( Mohassel & Zhang , 2017 ) , MiniONN ( Liu et al. , 2017 ) , Gazelle ( Juvekar et al. , 2018 ) , and DiNN ( Bourse et al. , 2018 ) , while only SecureML supports training ) . Nevertheless , in general , CPU ( with or without SGX ) is not optimized for costly operations in DNN such as matrix multiplication . Using specialized hardware such as GPU for such computation is a common practice . However , SGX-enclaves can not directly leverage GPU because its security guarantee is bounded within the CPU and fixed memory . It is unclear how CaffeSCONE ( and other works including TensorSCONE , Chiron ( Hunt et al. , 2018 ) , and MLCapsule ( Hanzlik et al. , 2018 ) ) can leverage GPU without trusting it ( or losing privacy ) . The SGX+GPU mode of our framework , which we call Goten , enables an even more efficient approach . To the best of our knowledge , no existing work ever explored this possibility on privacypreserving training . A recent work Slalom ( Tramèr & Boneh , 2019 ) also uses GPU but it only offers prediction privacy . We follow the common practice in the cryptographic privacy-preserving training literature ( SecureML , its subsequent work ( Wagh et al. , 2019 ) , and other prior works ( Nikolaenko et al. , 2013a ; b ) ) which employ non-colluding servers . Specifically , our framework uses three non-colluding GPU-enabled servers , two of them with a trusted processor . This setup appears to be necessary when the primary goal is to achieve privacy without heavyweight cryptographic tools . In practice , one can employ cloud service providers who are market competitors and value their reputations , or involve a government agency especially in healthcare/financial settings . Taking Full Advantage of the Servers We choose to exploit the server-aided setting fully and employ one additional server when compared with SecureML . What this server does is to “ bootstrap ” the triplets for secret sharing ( Beaver , 1991 ) across the two servers , which SecureML assumes such a bootstrap has been done in advance in an offline phase . Goten thus achieves a higher throughput without worrying that the offline preparation will be “ exhausted ” when the demand reaches its peak , which is also a hidden problem not addressed by Slalom . It also means Goten provides a “ true ” outsourcing solution – the time needed for securely outsourcing the job to the untrusted GPU is less than that for computing the job locally by the SGX plus any time needed for pre-computation . If desired , one may easily adapt our framework back to the two-server setting . ( See Section 2.2 . ) Dynamic Quantization Scheme We quantize the neural network parameters to fixed-point number format for efficient cryptographic operations ( cf. , static quantization in Slalom ) . This process needs to be implemented carefully for the following reasons . First , the many matrix multiplications in neural network may scale up the output values quickly , easily exceeding the numeric limit of the data type . Second , there are functions that map values to a small interval ( e.g. , softmax ( ) and sigmoid ( ) ) which require high precision . To avoid these potential accuracy problems , we developed a data-type conversion scheme , again , for enjoying “ the best of both worlds , ” i.e. , the benefit of accurate floating-point operations on trusted processors and efficient fixed-point operations on GPUs . Our experiment ( Section 4 ) confirms that our framework preserves high accuracy . 1TensorSCONE ( Kunkel et al. , 2019 ) employed SCONE with TensorFlow ( Abadi et al. , 2016a ) ( a DNN framework like Caffe we used ) ; unfortunately , it is not open source . Memory-aware Implementation A naı̈ve solution of overcoming the memory limit of SGX enclaves to rely on the Linux ’ s paging provided by Intel SGX SDK . However , it imposes much performance overhead ranging from 10× to 1000× comparing to unprotected programs ( Arnautov et al. , 2016 ) for exiting the enclave mode and switching back after processing the untrusted memory . Hence , in our framework , we take extra measures to reduce the memory footprints by looking into our specific DNN operations and handle any needed memory swapping by the enclave itself . 1.3 TECHNICAL CONTRIBUTIONS . Using both SGX and GPU for privacy-preserving training may sound straightforward , but we stress that we tackled a number of issues . To better understand the obstacles , here we revisit how Slalom performs privacy-preserving prediction and why it fails to support training . The core idea of Slalom can be described in simple terms : first apply static quantization on an input x to be protected , then outsource the job of computing f ( x+ r ) to GPU by hiding x with a blinding factor r in Zq ( where q is a large prime ) . Since it focuses on linear layers , f is linear and hence f ( x+ r ) = f ( x ) + f ( r ) . When SGX gets back f ( x + r ) , it performs “ unblinding ” using f ( r ) and obtains f ( x ) . For such outsourcing to be possible , f ( r ) should be precomputed . As simple as it may seem , Slalom needs to minimize the following three kinds of overheads – ( i ) computations over Zq performed by the untrusted GPU for the security of the blinding trick , ( ii ) the communication between TEE and the untrusted GPU , and ( iii ) loading the precomputed unblinding factor f ( r ) to TEE . Looking ahead , our outsourcing protocol faces even greater challenges regarding ( i ) and ( ii ) . Slalom addresses ( iii ) by assumption – it was done in an offline stage before the TEE needs to process any query . If we just ask the SGX to compute it , computing f ( r ) is of the same complexity as f ( x ) . Another way is to load them on-spot . It is again subjected to the memory limit and incurs the unwanted communication overhead . More importantly , it is insecure to ask the untrusted environment to compute f ( r ) . There are five conceptual challenges remain unsolved by Slalom regarding training . 1 ) Dynamic quantization : Slalom explicitly left it as one of the open challenges . 2 ) DNN weights are fixed at inference time , but it is not for training . This further complicates the dynamic quantization issue since the weights fluctuate . 3 ) The pre-computation technique does not apply for training . In more details , the training function is actually parameterized by a publicly-known weight W , i.e. , fW ( x ) multiples x withW . Moreover , the weight changes after ( a batch of ) operations are processed which makes fW ( r ) useless for a changing weight W ′ . 4 ) It is now apparent that Slalom does not protect the model weight W , which should be protected in private training ( and “ more private ” prediction ) . This is also one of the open challenges left explicitly by Slalom . 5 ) The last one is a challenge unique to our solution in addressing the other challenges . In their usage , TEE and GPU are colocated . However , in our settings , we need to propose an outsourcing solution which is efficient enough even we are subjected to an even higher communication overhead between the servers . Goten is the first framework that preserve the privacy of not only the prediction queries but the training data and model parameters with GPU and trusted environment . Our work achieves the highest efficiency of training and prediction in such privacy setting . This is the also first work which performs extensive experimental investigations of this possibility . Concretely , in our case study on VGG , we can speed up a linear layers up to 40× , and improve the performance of VGG11 by 8.64× . 2 SYSTEM MODEL . There are n mutually untrusted data providers who want to jointly train a DNN using their disjoint training data , but they are not willing to reveal their private data to others . They have already agreed on a specific DNN architecture . The corresponding code for the training algorithm is assumed to be genuine after manual or automated verification ( Sinha et al. , 2016 ) . After training , a querier can obtain prediction results from the resulting DNN and the results are only revealed to the querier .

Summary: The paper builds a privacy-preserving training framework within a Trusted Execution Environment (TEE) such as Intel SGX. The work is heavily inspired from Slalom, which does privacy-preserving inference in TEEs. The main drawbacks of Slalom when extending to training are (1) weight quantization needs to be dynamics as they change during training, and (2) pre-processing step of Slalom to compare u = f(r) isn't effective as the weights change, and running this within TEE is no better than running the full DNN within TEE. In addition, Goten also makes the weights private as opposed to Slalom. Overall, this is a very important contribution towards privacy preserving training and the paper takes a strong practical and implementation-focused approach by considering issues arising due to memory limitations in TEE and the performance implications of default Linux paging.