Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware

As Machine Learning ( ML ) gets applied to security-critical or sensitive domains , there is a growing need for integrity and privacy for outsourced ML computations . A pragmatic solution comes from Trusted Execution Environments ( TEEs ) , which use hardware and software protections to isolate sensitive computations from the untrusted software stack . However , these isolation guarantees come at a price in performance , compared to untrusted alternatives . This paper initiates the study of high performance execution of Deep Neural Networks ( DNNs ) in TEEs by efficiently partitioning DNN computations between trusted and untrusted devices . Building upon an efficient outsourcing scheme for matrix multiplication , we propose Slalom , a framework that securely delegates execution of all linear layers in a DNN from a TEE ( e.g. , Intel SGX or Sanctum ) to a faster , yet untrusted , co-located processor . We evaluate Slalom by running DNNs in an Intel SGX enclave , which selectively delegates work to an untrusted GPU . For canonical DNNs ( VGG16 , MobileNet and ResNet variants ) we obtain 6× to 20× increases in throughput for verifiable inference , and 4× to 11× for verifiable and private inference . 1 INTRODUCTION . Machine learning is increasingly used in sensitive decision making and security-critical settings . At the same time , the growth in both cloud offerings and software stack complexity widens the attack surface for ML applications . This raises the question of integrity and privacy guarantees for ML computations in untrusted environments , in particular for ML tasks outsourced by a client to a remote server . Prominent examples include cloud-based ML APIs ( e.g. , a speech-to-text application that consumes user-provided data ) or general ML-as-a-Service platforms . Trusted Execution Environments ( TEEs ) , e.g , Intel SGX ( McKeen et al. , 2013 ) , ARM TrustZone ( Alves & Felton , 2004 ) or Sanctum ( Costan et al. , 2016 ) offer a pragmatic solution to this problem . TEEs use hardware and software protections to isolate sensitive code from other applications , while attesting to its correct execution . Running outsourced ML computations in TEEs provides remote clients with strong privacy and integrity guarantees . For outsourced ML computations , TEEs outperform pure cryptographic approaches ( e.g , ( Gilad-Bachrach et al. , 2016 ; Mohassel & Zhang , 2017 ; Ghodsi et al. , 2017 ; Juvekar et al. , 2018 ) ) by multiple orders of magnitude . At the same time , the isolation guarantees of TEEs still come at a steep price in performance , compared to untrusted alternatives ( i.e. , running ML models on contemporary hardware with no security guarantees ) . For instance , Intel SGX ( Intel Corp. , 2015 ) incurs significant overhead for memory intensive tasks ( Orenbach et al. , 2017 ; Harnik & Tsfadia , 2017 ) , has difficulties exploiting multi-threading , and is currently limited to desktop CPUs that are outmatched by untrusted alternatives ( e.g. , GPUs or server CPUs ) . Thus , our thesis is that for modern ML workloads , TEEs will be at least an order of magnitude less efficient than the best available untrusted hardware . This leads us to the main question of this paper : How can we most efficiently leverage TEEs for secure machine learning ? This was posed by Stoica et al . ( 2017 ) as one of nine open research problems for system challenges in AI . A specific challenge they raised is that of appropriately splitting ML computations between trusted and untrusted components , to increase efficiency as well as security by minimizing the Trusted Computing Base . This paper explores a novel approach to this challenge , wherein a Deep Neural Network ( DNN ) execution is partially outsourced from a TEE to a co-located , untrusted but faster device . Our approach , inspired by the verifiable ASICs of Wahby et al . ( 2016 ) , differs from cryptographic ML outsourcing . In our case , work is delegated between two co-located parties , thus allowing for highly interactive—yet conceptually simpler— outsourcing protocols with orders-of-magnitude better efficiency . Our work also departs from prior systems that execute DNNs fully in a TEE ( Ohrimenko et al. , 2016 ; Hunt et al. , 2018 ; Cheng et al. , 2018 ; Hanzlik et al. , 2018 ) . The main observation that guides our approach is that matrix multiplication—the main bottleneck in DNNs— admits a concretely efficient verifiable outsourcing scheme known as Freivalds ’ algorithm ( Freivalds , 1977 ) , which can also be turned private in our setting . Our TEE selectively outsources these CPU intensive steps to a fast untrusted co-processor ( and runs the remaining steps itself ) therefore achieving much better performance than running the entire computation in the enclave , without compromising security . Contributions . We propose Slalom , a framework for efficient DNN inference in any trusted execution environment ( e.g. , SGX or Sanctum ) . To evaluate Slalom , we build a lightweight DNN library for Intel SGX , which may be of independent interest . Our library allows for outsourcing all linear layers to an untrusted GPU without compromising integrity or privacy . Our code is available at https : //github.com/ftramer/slalom . We formally prove Slalom ’ s security , and evaluate it on multiple canonical DNNs with a variety of computational costs—VGG16 ( Simonyan & Zisserman , 2014 ) , MobileNet ( Howard et al. , 2017 ) , and ResNets ( He et al. , 2016 ) . Compared to running all computations in SGX , outsourcing linear layers to an untrusted GPU increases throughput ( as well as energy efficiency ) by 6× to 20× for verifiable inference , and by 4× to 11× for verifiable and private inference . Finally , we discuss open challenges towards efficient verifiable training of DNNs in TEEs . 2 BACKGROUND . 2.1 PROBLEM SETTING . We consider an outsourcing scheme between a client C and a server S , where S executes a DNN F ( x ) : X → Y on data provided by C. The DNN can either belong to the user ( e.g. , as in some ML-as-a-service platforms ) , or to the server ( e.g. , as in a cloud-based ML API ) . Depending on the application , this scheme should satisfy one or more of the following security properties ( see Appendix B for formal definitions ) : • t-Integrity : For any S and input x , the probability that a user interacting with S does not abort ( i.e. , output ⊥ ) and outputs an incorrect value ỹ 6= F ( x ) is less than t. • Privacy : The server S learns no information about the user ’ s input x . • Model privacy : If the model F is provided by the user , S learns no information about F ( beyond e.g. , its approximate size ) . If F belongs to the server , C learns no more about F than what is revealed by y = F ( x ) .1 2.2 TRUSTED EXECUTION ENVIRONMENTS ( TEES ) , INTEL SGX , AND A STRONG BASELINE . Trusted Execution Environments ( TEE ) such as Intel SGX , ARM TrustZone or Sanctum ( Costan et al. , 2016 ) enable execution of programs in secure enclaves . Hardware protections isolate computations in enclaves from all 1For this zero-knowledge guarantee to be meaningful in our context , S would first commit to a specific DNN , and then convince C that this DNN was correctly evaluated on her input , without revealing anything else about the DNN . programs on the same host , including the operating system . Enclaves can produce remote attestations—digital signatures over an enclave ’ s code—that a remote party can verify using the manufacturer ’ s public key . Our experiments with Slalom use hardware enclaves provided by Intel SGX ( see Appendix A for details ) .2 TEEs offer an efficient solution for ML outsourcing : The server runs an enclave that initiates a secure communication with C and evaluates a model F on C ’ s input data . This simple scheme ( which we implemented in SGX , see Section 4 ) outperforms cryptographic ML outsourcing protocols by 2-3 orders of magnitude ( albeit under a different trust model ) . See Table 1 and Appendix C for a comparison to two representative works . Yet , SGX ’ s security comes at a performance cost , and there remains a large gap between TEEs and untrusted devices . For example , current SGX CPUs are limited to 128 MB of Processor Reserved Memory ( PRM ) ( Costan & Devadas , 2016 ) and incur severe paging overheads when exceeding this allowance ( Orenbach et al. , 2017 ) . We also failed to achieve noticeable speed ups for multi-threaded DNN evaluations in SGX enclaves ( see Appendix H ) . For DNN computations , current SGX enclaves thus can not compete—in terms of performance or energy efficiency ( see Appendix C ) —with contemporary untrusted hardware , such as a GPU or server CPU . In this work , we treat the above simple ( yet powerful ) TEE scheme as a baseline , and identify settings where we can still improve upon it . We will show that our system , Slalom , substantially outperforms this baseline when the server has access to the model F ( e.g. , F belongs to S as in cloud ML APIs , or F is public ) . Slalom performs best for verifiable inference ( the setting considered in SafetyNets ( Ghodsi et al. , 2017 ) ) . If the TEE can run some offline data-independent preprocessing ( e.g. , as in Gazelle ( Juvekar et al. , 2018 ) ) , Slalom also outperforms the baseline for private ( and verifiable ) outsourced computations in a later online phase . Such a two-stage approach is viable if user data is sent at irregular intervals yet has to be processed with high throughput when available . 2.3 OUTSOURCING OUTSOURCED DNNS AND FREIVALDS ’ ALGORITHM . Our idea for speeding up DNN inference in TEEs is to further outsource work from the TEE to a co-located faster untrusted processor . Improving upon the above baseline thus requires that the combined cost of doing work on the untrusted device and verifying it in the TEE be cheaper than evaluating the full DNN in the TEE . Wahby et al . ( 2016 ; 2017 ) aim at this goal for arbitrary computations outsourced between co-located ASICs . The generic non-interactive proofs they use for integrity are similar to those used in SafetyNets ( Ghodsi et al. , 2017 ) , which incur overheads that are too large to warrant outsourcing in our setting ( e.g. , Wahby et al . ( 2016 ) find that the technology gap between trusted and untrusted devices needs to be of over two decades for their scheme to break even ) . Similarly for privacy , standard cryptographic outsourcing protocols ( e.g. , ( Juvekar et al. , 2018 ) ) are unusable in our setting as simply running the computation in the TEE is much more efficient ( see Table 1 ) . 2SGX has recently come under several side-channel attacks ( Chen et al. , 2018 ; Van Bulck et al. , 2018 ) . Intel is making firmware and hardware updates to SGX with the goal of preventing these attacks . In time , it is likely that SGX can be made sufficiently secure to satisfy the requirements needed for Slalom . Even if not , other enclave architectures are available , such as Sanctum for RISC-V ( Costan et al. , 2016 ) or possibly a separate co-processor for security operations . To overcome this barrier , we design outsourcing protocols tailored to DNNs , leveraging two insights : 1 . In our setting , the TEE is co-located with the server ’ s faster untrusted processors , thus widening the design space to interactive outsourcing protocols with high communication but better efficiency . 2 . The TEE always has knowledge of the model and can selectively outsource part of the DNN evaluation and compute others—for which outsourcing is harder—itself . DNNs are a class of functions that are particularly well suited for selective outsourcing . Indeed , non-linearities— which are hard to securely outsource ( with integrity or privacy ) —represent a small fraction of the computation in a DNN so we can evaluate these in the TEE ( e.g. , for VGG16 inference on a single CPU thread , about 1.5 % of the computation is spent on non-linearities ) . In contrast , linear operators—the main computational bottleneck in DNNs—admit for a conceptually simple yet concretely efficient secure delegation scheme , described below . Integrity . We verify integrity of outsourced linear layers using variants of an algorithm by Freivalds ( 1977 ) . Lemma 2.1 ( Freivalds ) . Let A , B and C be n×n matrices over a field F and let s be a uniformly random vector in Sn , for S ⊆ F. Then , Pr [ Cs = A ( Bs ) | C 6= AB ] = Pr [ ( C −AB ) s = 0 | ( C −AB ) 6= 0 ] ≤ 1/|S| . The randomized check requires 3n2 multiplications , a significant reduction ( both in concrete terms and asymptotically ) over evaluating the product directly . The algorithm has no false negatives and trivially extends to rectangular matrices . Independently repeating the check k times yields soundness error 1/|S|k . Privacy . Input privacy for outsourced linear operators could be achieved with linearly homomorphic encryption , but the overhead ( see the micro-benchmarks in ( Juvekar et al. , 2018 ) ) is too high to compete with our baseline ( i.e. , computing the function directly in the TEE would be faster than outsourcing it over encrypted data ) . We instead propose a very efficient two-stage approach based on symmetric cryptography , i.e. , an additive stream cipher . Let f : Fm → Fn be a linear function over a field F. In an offline phase , the TEE generates a stream of one-time-use pseudorandom elements r ∈ Fm , and pre-computes u = f ( r ) . Then , in the online phase when the remote client sends an input x , the TEE computes Enc ( x ) = x+ r over Fm ( i.e. , a secure encryption of x with a stream cipher ) , and outsources the computation of f ( Enc ( x ) ) to the faster processor . Given the result f ( Enc ( x ) ) = f ( x+ r ) = f ( x ) + f ( r ) = f ( x ) + u , the TEE recovers f ( x ) using the pre-computed u. Communication . Using Freivalds ’ algorithm and symmetric encryption for each linear layer in a DNN incurs high interaction and communication between the TEE and untrusted co-processor ( e.g. , over 50MB per inference for VGG16 , see Table 3 ) . This would be prohibitive if they were not co-located . There are protocols with lower communication than repeatedly using Freivalds ’ ( ( Fiore & Gennaro , 2012 ; Thaler , 2013 ; Ghodsi et al. , 2017 ) ) . Yet , these incur a high overhead on the prover in practice and are thus not suitable in our setting .

Summary: The authors propose a new method of securely evaluating neural networks. The approach builds upon existing Trusted Execution Environments (TEE), a combination of hardware and software that isolates sensitive computations from the untrusted software stack. The downside of TEE is that it is expensive and slow to run. This paper proposes outsourcing the linear evaluation portions of the DNN to an untrusted stack that's co-located with the TEE. To achieve privacy (i.e., the input isn't revealed to the untrusted evaluator), the approach adds a random number r to the input vector x, evaluates f(x+r) on the untrusted stack, then subtracts off f(r) from the output. This limits the approach to be applicable to only linear functions. To achieve integrity (verify the correctness of the output), the paper proposes testing with random input vectors (an application of Freivalds theorem, which bounds the error probability). The techniques for integrity and privacy works only on integer evaluations, hence the network weights and inputs need to be quantized. The paper tries to minimize degradation in accuracy by quantizing as finely as numerically allowable, achieving <0.5% drop in accuracy on two example DNNs. Overall, compared to full evaluation in a TEE, this approach is 10x faster on one DNN, and 40x to 64x faster on another network (depending on how the network is formulated).