CrypTen: Secure Multi-Party Computation Meets Machine Learning
Authors: Brian Knott
Shobha Venkataraman
Awni Hannun
Shubhabrata Sengupta
Mark Ibrahim
Laurens van der Maaten
Venue: NeurIPS 2021 Poster (2021)
Keywords: Secure multi-party computation
deep-learning framework.

Abstract: Secure multi-party computation (MPC) allows parties to perform computations on data while keeping that data private. This capability has great potential for machine-learning applications: it facilitates training of machine-learning models on private data sets owned by different parties, evaluation of one party's private model using another party's private data, etc. Although a range of studies implement machine-learning models via secure MPC, such implementations are not yet mainstream. Adoption of secure MPC is hampered by the absence of flexible software frameworks that `"speak the language" of machine-learning researchers and engineers. To foster adoption of secure MPC in machine learning, we present CrypTen: a software framework that exposes popular secure MPC primitives via abstractions that are common in modern machine-learning frameworks, such as tensor computations, automatic differentiation, and modular neural networks. This paper describes the design of CrypTen and measure its performance on state-of-the-art models for text classification, speech recognition, and image classification. Our benchmarks show that CrypTen's GPU support and high-performance communication between (an arbitrary number of) parties allows it to perform efficient private evaluation of modern machine-learning models under a semi-honest threat model. For example, two parties using CrypTen can securely predict phonemes in speech recordings using Wav2Letter faster than real-time. We hope that CrypTen will spur adoption of secure MPC in the machine-learning community.

CRYPTEN: Secure Multi-Party Computation Meets Machine Learning
Brian Knott Shubho Sengupta
Shobha Venkataraman
Awni Hannun
Mark Ibrahim
Laurens van der Maaten
Facebook AI Research
(brianknote shobaaa
shobha, awni, ssengupta, bark
Abstract
Secure multi-party computation (MPC) allows parties to perform computations on data while keeping that data private. This capability has great potential for machine-learnigg applications: it facilitates training of machine-learning models on private data sets owned by different parties, evaluation of one party's 's private model using another party's s private data, Although range of studies imple- ment machine-learning models via secure MPC, such implementations are not yet mainstream. Adoption of secure MPC is hampered by the absence of flexible software frameworks that "speak the language" of machine-learning researchers and engineers. To foster adoption of secure MPC in machine learning, we present CRYPTEN: a software framework that exposes popular secure MPC primitives via abstractions that common in modern machine-learning frameworks, such tensor computations, automatic differentiation, and modular neural networks. This paper describes the design of CRYPTEN and measure its performance on state-of- of- the-art models for text classification, speech recognition, and image classification. Our benchmarks show that CRYPTEN'S GPU support and high-performance com- munication between (an arbitrary number of) parties allows it to perform efficient private evaluation of modern model. For example, two parties using CRYPTEN can securely predict phonemes in speech recordings using Wav2Letter [17] faster than real-time. We hope that CRYPTEN will spur adoption of secure MPC in the machine-learning community.
learning models under a semi-honest threat
1 Introduction
allows parties collaboratively perform computa- tions on their combined data sets without revealing the data they possess to each other. This capability of secure MPC has the potential to unlock variety of machine-learning applications that are currently infeasible because data privacy concerns. For example, secure MPC can allow medical research institutions to jointly train better diagnostic models without having share their sensitive patient data (27] or allow social scientists to analyze gender wage gap statistics without companies having to share sensitive salary data 42 The prospect of such applications of machine learning with rigorous privacy and security guarantees has spurred a number of studies on machine learning via secure MPC [8
Secure multi-party computation (MPC; 33 69
66 67]. However, at present, adoption of secure MPC in machine learning is still relatively limited considering its wide-ranging potential. One of the main obstacles to widespread adoption is that the complexity of secure MPC techniques puts them out of reach for most machine-learning researchers, who frequently lack in-depth knowledge of cryptographic techniques.
48 58
To foster the adoption of secure MPC techniques in machine learning, present CRYPTEN: flexible software framework that aims to make modern secure MPC techniques accessible to machine- learning researchers and developers without background in cryptography. Specifically, CRYPTEN
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

provides a comprehensive tensor-comppe computatinn library in which all computations are performed via secure MPC. CRYPTEN'S API closely follows the API of the popular Py PyTorch framework for machine which makes it easy to use for machine-learning practitioners. For example, provides automatic differentiation and modular neural-network package. CRYPTEN assumes an semi-honest threat model [30] $2.3.2] and works for an arbitrary number of parties. To make private training and inference efficient, CRYPTEN off-loads computations to the GPU and uses high-performance communication libraries to implement interactions between parties.
learning [54 55
The paper presents: (1) an overview of CRYPTEN'S design principles; (2) a description of the design of CRYPTEN and of the secure MPC protocols implemented; (3) collection of benchmark experiments using CRYPTEN to run private versions of ofate-of-the-arr models for text classification, speech recogmition, and image classification; and (4) a discussion of open problems and roadmap for the further development of CRYPTEN. Altogether, the paper demonstrates that CRYPTEN'S flexible, Py Torch-like API makes private inference and training of modern machine-learning models easy to implement and efficient. For example, CRYPTEN allows two parties to privately classify in 2-3 seconds, or to securely make phoneme predictions for 16kHz speech recordings [17] faster than real-time. We hope that CRYPTEN'S ' promising performance and ease-of- use will foster the adoption of secure MPC by the machine-learning community, and pave the way for a new generation of secure and private muchine-leaming systems.
an image 226 35
2 Related Work
CRYPTEN is part of a large body of work that develops secure MPC protocols for machine learning; Appendix ??. Most closely related
work is CryptGPU 633, which implements 2-out-of-3
replicated secret sharing protocol 4 377 on top of CRYPTEN Like CRYPTEN, CryptGPU provides security against semi-honest corruption, but it limited to the three-party setting. CryptGPU is one of several protocols optimized for the three-party setting. For example, Falcon [67] implements maliciously secure three-party MPC protocol, combining techniques from SecureNN
666 and
ABY3 48. Falcon allows evaluation and training of convolutional networks such as AlexNet
40 and
VGG Other systems that work in this setting include Astra [6]. Blaze (56). and CrypTFlow
There also exists a family of two-party systems that, like CRYPTEN, assume a semi-honest threat model. These systems include Gazelle (38. Chamelcon [58], EzPC (15), MiniONN 45,, Se- cureML. [49], PySyft [60], and Delphi (47. XONN (59] also works in the two-party setting but provides malicious security. Compared these systems, CRYPTEN provides more flexible machine- learning focused AP(T that supports reverse-mode automatic differentiation, implements a rich set of functions, and natively runs GPUs Moreover, CRYPTEN supports wider range of use cases by working with an arbitrary number of parties, and make communication between parties efficient via communication primitives that were optimized for high-performance distributed computing.
Design Principles
In the development of CRYPTEN, we adopted the following two main design principles:
Machine-learning first API. CRYPTEN has a general purpose, machine-lcarning first API design. Most other secure MPC frameworks 34 adopt API that stays close the underlying MPC protocols. This hampers adoption of these frameworks in machine learning, for example, because they do not natively support tensor operations (but only scalar operations) and because they lack features that machine-learning researchers have come to expect, such as automatic differentiation. Instead, CRYPTEN implements the lensor-computation API of the popular PyTorch machine-learning framework [54], implements reverse-mode automatic differentiation, provides a modular neural- network package with corresponding learning routines, and supports GPU computations. We aim to allow developers to transition code from PyTorch to CRYPTEN by changing single Python import.
Eager execution. CRYPTEN adopts imperative programming model. This is different from existing MPC frameworks, which generally implement compilers for their own domain-specific languages
34 While compiler approaches have potential performance benefits, they slow down the
CCrypTFlow [4] also provides such an API by integrating deeply with TensorFlow []] but unlike CRYPTEN, it does not support PyTorhh's 'ss eager execution model (55) or GPU support.

---

Communicctor
Trusted third party
ONNX Importer
Scllioeee communicaiinn
pehaitte Renwer siglie
Artthmetic secret share
PyTorch tensor GPU)
A2S conwarston
CrypTensor (wtt autograd
Neural network modules
Binary share
Computation Party
Figure 1: High-level overview of the design of CRYPTEN. See text in Section for details.
development cycle, make debugging harder, and prevent users from using arbitrary host-language constructs [3] Instead, CRYPTEN follows the recent trend in machine learning away from graph compilers D to frameworks that eagerly execute computations [ [55], providing a better developer experience. Yet, CRYPTEN is performant because implements state-of-the-art secure MPC protocols (for settings with arbitrary number of parties), because uses PyToorc''s 's highly optimized tensor library for most computations, because computations be off-loaded to the GPU, and because it uses communication libraries that were optimized for high-performance distributed computing.
Design Overview
Figure gives an overview of CRYPTEN'S de-
sign. Parties perform computations using effi- cient PyTorch tensor operations. Because se- cure MPC computations are integer computa- tions that are natively supported on GPUs, CRYPTEN maps between integer and floating- point computations on GPUs; see Section 5.3 The multi party computations are implemented on arithmetic and binary secret shares [22 32
inport crypten, torch
set communccttion and sync randon seeds: poppen.init())
secret share
torch. tensor ([1.0, 2.0, 3.8])
enc cyypten.cryptenso. (x, src=0)
reveal secret shared tensor: x_dec s_onc.gett.lain_teex O
see Section 5.1 Whereas many computations can be performed directly on arithmetic se- cret shares, others require conversion between arithmetic and binary secret shares (A2B) and back (B2A); Section 5.2] Some multi- party computations require interaction between parties via communicator that employs the high-performance communication primitives in Gloo 31 and NCCL (51). Some multi-party computations require Beaver triples 77). which are supplied by trusted third party (
assert torch.all cllch.ell_clooexx..d x)
# add secret shared tensors:
y = torch. tensor ([2.0, 3.0, 4.0])
y_enc = xy_enc x_enc y_enc
(y, sre=0)
xy_dec xy_ene. pet_plain__ext
assert alch.oll_coose dxy_dde,
Example of secret-sharing tensors, re- Figure 2:
vealing tensors, and private addition in CRYPTEN.
All secure computations are wrapped in a CrypTensor object that implements the PyTorch tensor API and that provides reverse-mode automatic differentiation (autograd) enable gradient-based training of arbitrary (deep) learning models. Figure 2 illustrates CrypTensor creation, i.e., how tensors secret-shared and revealed, well as simple computation (addition). Note that each party involved in the multi-party computation executes the same code. Whenever communication between the parties is required (e.g., as part of private multiplications). the communication acts synchronization point between the parties. The crypten. init () call is required once to establish the communication channel. In the example, the input tensor for the creation of the arithmetic secret share is provided party src=0, which indicates the rank of the party that supplies the data to be secret-shared (the other parties executing this code may provide None as input).
'CRyPTEN adopts trusted third party generating Beaver triples for efficiency reasons, we are planning TTP-free solutions based on additive homomorphic eneryption 52 or oblivious transfer (39). "CRYPTEn relies on MPI primitives for communication: each party knows their rank and the world size.

---

To enable deep-learning use cases, CRYPTEN allows implementing neural networks following PyTorch's API. Figure 3 shows how to create and encrypt neural networks and how to use automatic differentiation in CRYPTEN. The ex- ample assumes that some training sample and the associated target label are provided by the party with rank O (note the value of src), As illustrated by the example, CRYPTEN's API closely follows that of PyTocch. Indeed, it is possible to write single training loop that can be used to train models using CRYPTEN or Py- Torch without code changes. This makes it easy to adapt PyTorch code to use secure MPC for its computations, and it also makes debugging easier. The appendix presents table listing all tensor functions that CrypTensor implements.
import crypten. optinizer as optimizer import crypten. as nn
erypten.optinizer
create model. criterion, and optimizer: model_encc m. (equenttiall
nn Ln.Liner din, din), na. nn.ReLUO). Li.ein (hidden_din, nun_classes), encrypt()
criterion CrossEntroopyooss
optinizer aptimizer. SGD(
dadel_enc.parameters. 1r=0.1, nomentum=8.9,
perform prediction sanple:
target_onc = crypten. crypten.cryptensor (tarenttttttrre, (target, sre=d)
sample_enc enc = crypten, cryptensor (sample, sre=0)
output_enc model_enc model_encssample_ees (sample_enc)
perform backward pass and update paraneters:
beael_een.zero_raade loss_enc criterion target_enc) loss_enc. backward)) optimizer.step!
To enable interoperability with existing machine- learning platforms, neural networks can be im- ported into CRYPTEN via ONNX. Figure 4 shows how a PyTorch model is imported into CRYPTEN. The example illustrates how CRYPTEN makes private inference with ResNet- 18 easy. The example in the figure also demonstrates CRYPTEN'S GPU support. One caveat is that all parties must use the same type of device (i.e. CPU GPU) for computations.
Figure 3: Example using neural networks and au- tomatic differentiation in CRYPTEN.
import torchvision. datasets as datasets import torchvision models as models inport torchvision, transforms transforms
download and set ImageNet dataset: transform trams ToTensoro dataset = datasets. ImageNett imagenet folder, transfora-transfomm,
5 Secure Computations
To facilitate secure computations, CRYPTEN im- plements arithmetic secret sharing 22 23 and binary secret sharing [32), well as conver- sions between these two types of sharing (24).
secret share pre-trained ResNet-18 on GPU: model
model_enc crypten. crypten..m.froo.yyoooddd model, dataset[0),
Arithmetic secret sharing is particularly well- suited for operations that are common mod- machine-leamnnng models, such as matrix
encrypt()
perform inference on secret-shared inages: for image in dataset:
multiplications and convolutions. Binary se- cret sharing is required for evaluating certain other common functions, such as rectified lin- units. We provide high-level overview of CRYPTEN'S secure computation protocol here; detailed description is presented in the appendix.
image_enc crypten. cryptensor cuda
output_enc model_enc (inage_een))
output
plain text
Figure Private inference on shared images using a secret-shared ResNet- 18 model on GPU.
5.1 Secret Sharing
Arithmetic secret sharing shares scalar value Z/QZ, where Z/QZ denotes a ring with Q elements, across parties p P. We denote the sharing of x by [x] = [[<=pppp heEp, where [x]p E Z/QZ indicates party p's share of x. The shares are constructed such that their sum reconstructs the original value that is, x Eper(aip mod Q. To share value the parties generate pseudorandom zero-share [18] with |P] random numbers that sum to O. The party that possesses the value adds to their share and discards x. We use fixed-point encoding to obtain x from a floating point value, x To do so, multiply za with large scaling factor B and round to the nearest integer: where = 2 for some precision of bits. To decode a value, I, we compute 2R R a/B.
|BRR],
Binary secret sharing is special case of arithmetic secret sharing that operates within the binary field 2 22, binary secret share, (ar), value formed by arithmetic secret shares of the bits of x, setting Q = 2. Each party E P holds a share, (z)p. such that + (z), is satisfied.

---

Conversion from [a:] to (x) is implemented by having the parties create binary secret share of their [æ]p shares, and summing the resulting binary shares. Specifically, the parties create binary secret share, ((r)p), of all the bits in [x)p-- Subsequently, the parties compute (x) = (p)
using a carry-lookahead adder in logz ( (P)) (Lg communication rounds 14 21I
Conversion from (x) to [x]] is achieved by computing [x] = = 2 [(z) (),, where (z)(0) denotes the 6-th bit of the binary share (x) and B is the total number of bits in the shared secret, (x). To create an arithmetic share of a bit, the parties use secret shares, ([r(0)), (r(*))), of random bits r((), The random bits are provided by the TTP, but we plan to add an implementation that generates them off-line via oblivious transfer (39]. The parties use (r((t) to mask (xr) and reveal the resulting masked bit 2h) Subsequently, they compute ((x))()) [r(]] + 20) |r(0)] 2(h)
5.2 Secure Computation
Arithmetic and binary secret shares have homomorphic properties that can be used to implement secure computations. All computations in CRYPTEN are based on private addition and multiplication.
Private addition of two arithmetically secret shared values, [a] [x:] + [v]. is implemented by having each party p sum their shares of [a:] and [y]: each party p E P computes [z]p = [mlp + Wplp-
Private multiplication is implemented using random Beaver triples 77 [[a), [b], [c]) with c ab, that
are provided by the TTP. The parties compute [e] = [ac] [a] and [ [3]] [b]. and decrypt e and d
without information leakage due to the masking. They compute the result [<]] = [e(+ [a]]+++]+
using trivial implementations of addition and multiplication of secret shares with public values.
Linear functions are trivially implemented as combinations of private addition and multiplication. This allows CRYPTEN to compute dot products, outer products, matrix products, and convolutions.
Non-linear functions are implemented using standard approximations that only require private addi- tion and multiplication. Specifically, CRYPTEN evaluates exponentials using a limit approximation, logarithms using Householder iterations [36], and reciprocals using Newton-Rhapson iterations. This allows CRYPTEN to implement functions that are commonly used machine-learning models, including the sigmoid, softmax, and logistic-loss functions, as well as their gradients.
Comparators are implemented using function that evaluates [a 0] by: (1) converting [z] to a
binary secret-share (2); (2) computing its sign bit, (6) = (2) >> (L 1); and (3) converting the resulting bit to an arithmetic sharing [6]. This function allows CRYPTES to implement arbitrary
comparators. For example, it evaluates [x: < y] by computing [=] [x] [y] and evaluating [2 <<< 0).
Similarly. CRYPTEN can evaluate: (1) the sign function via sign [z) = 2[x >> 0] - 1: (2) the absolute
value function via [>>]| = [z] sign([z]); and (3) rectified linear units via ReLU((x))
[ar] [z > 0].
CRYPTEN also supports multiplexing: to do so, evaluates [c x y] [c][a] + (1 - [c)]
Lemma 1. The CRYPTEN secure-come computationnn protocol is secure against information leakage against any static passive adversary corrupting up to [P] - of the [P] parties involved the computation.
The proof of this lemma follows trivially from 9 11 21 24), and is given in the appendix. We adopt a protocol that provides security under a semi-honest threat model because it enables wide range of use cases of secure machine learning, whilst being more efficient than maliciously secure protocols.
5.3 Off-Loading Computations to the GPU
Hardware acceleration via GPUs is critical component for training and inference in modern machine- learning models. Akin to frameworks such as PyTorch (55) and TensorFlow M. CRYPTEN can off-load computations to the GPU. On the GPU, it uses highly-opimmized implementations for range of functions that are provided by CUDA librarics such as cuBLAS [99 and cuDNN 200.
Unfortunately, these libraries are designed for computations on seatingg-pinn numbers and do not support the integer types required to perform computations on L-bit fixed-point numbers. Akin to 633. we circumvent this problem by observing that for all integers a, b E Z n |-2",,221 we can compute the product ab using 64-bit floating-point representations and still recover the correct value over the integers. Specifically, CRYPTEN splits each 64-bit variable into four components, a = ag + 216a, + 232 022 + 248 03, where each a; represents a 16-bit integer component. We compute product ab of bi- integers by summing 10 pairwise products of their 16-bit components.
5

---

Batch
Com Communiiatannnnu mundd
Communtzztionn fin
parties
pa
Namber pa
Figure 5: Benchmarks for inference with text-sentiment classification model on GPUs in CRYPTEN and PyTorch. Left: Average wall-clock time per sample (in seconds). Middle: Number of bytes communicated per sample, per party (in GB). Right: Number communication rounds per sample.
The pairwise products of the 16-bit components are computed in parallel using highly optimized floating-point CUDA kemnlss. The same approach is used for matrix multiplications and convolutions. CRYPTEN further optimizes this approach by splitting into only components of 22-bits each when possible, which reduces the number of pairwise products required to 6 (see 663 Remark II.1]).
6 Benchmarks
To measure the performance of CRYPTEN, we performed experiments on three tasks: (1) text classification using a linear model that learns word embeddings: (2) speech recognition using the Wav2Letter model 17;; and (3) image classification using residual networks (35) and vision transformers 26). Because of space constraints, we focus on private inference using a secret-shared model on secret-shared data here, but our benchmark results with private training are very similar.
We performed benchmark experiments Skylake 18-core 1.6GHz) and GPUs (n Vidia P100). We set the number of OpenMP threads to in all benchmarks. All experiments were performed with the parties running in separate processes single machine. For GPU experiments, each party was assigned its own GPU. Although this setup is faster than scenario in which each party operates its provide a good sense of CRYPTEN'S performance. We average computation times over 30 batches, excluding the computation the first batch as that computation may include CuDNN benchmarking. Code reproducing the results of our experiments is available on htttps:/ccrypeen ai
proprictary cluster, testing inference both CPUs (Intel
machine[[we believe benchmark results
In our benchmarks, we focus on comparing (ciphertext) CRYPTEN computation with (plaintext) PyTorch computation. We refer the reader to [33] 633 for benchmarks that compare CRYPTEN to other secure MPC frameworks. Specifically, [33] finds CRYPTEN is 11-18x faster than PySyft [60] and approximately 3x faster than TF-Trusted in MNIST classification on CPU.
6.1 Text Classification
We performed text-sentiment classification experiments on the Yelp review dataset [70 using a model that consists of a linear layer operating on word embeddings. The embedding layer contains 32- dimensional embeddings of 519, 820 words, and the linear layer produces binary output indicating the sentiment the review. We evaluated the model on GPUs, varying the batch size and the number of parties participating. The normalized mean squared error (l)x [|xx-yll// / between the output of the CRYPTEN model and that of its PyTorch counterpart was smaller than 4 10 in all experiments.
Figure 5 presents the results of our experiments. The figure shows inference time per sample (in seconds) as a function of the number of parties involved in the computation for varying batch sizes (left); the amount of communication required per sample, per purty (in GB); and the number of communication rounds required per sample. We include results in which the number of parties herein, the CRYPTEN protocol but involve other parties, which implics that the single party is running the protocol on unencrypted data. One-party results allow us to bisect different sources of computational overhead: specifically, they separate overhead due communication from overhead due to fixed-point encoding, function approximations, and (lack of) sparse-matrix operations.
'Communication between GPUs in two machines connected via InfiniBand has approximately 20x lower throughput than communication between two GPUs in the same machine via NVLink (25GB/s versus 600GB/s).

---

Bach-40
Communicttinn eaardh
Communtcatioon Jin
10000
34"
parties
ounteral
Figure 6: Benchmarks for inference with Wav2Letter model on GPUs CRYPTEN and PyTorch. Left: Average wall-clock time per sample (in seconds). Middle: Number of bytes communicated per sample, per party (in GB). Right: Number communication rounds per sample.
Conpotaice
Corpuzatian
225
Caxeuniction
54
CPU
CPU
Namber parrttes
Nomterof partion
of partinn
Figure 7: Wall-clock time per sample (in sec.) for Wav2Letter inference CPUs and GPUs.
Figure 8: Average wall-clock time per sample (in seconds) for communication and computation during inference with Wav2Letter model on CPU (Ieft) and GPU (right).
The results in Figure 5 show that CRYPTEN is about 2.5-3 orders of magnitude slower than py Torch in text-sentiment classification, depending on the number of parties involved. Most computational overhead the word embedding layer: whereas PyTorch can evaluate this layer efficiently via sparse matrix multiplication, CRYPTEN cannot do sparse lookups as they would reveal information on the encrypted input. Instead, CRYPTEN performs a full matrix multiplication between the word- count vector and the embedding matrix. Yet, text sentiment predictions are quite fast in CRYPTEN: inference takes only 0.03 seconds per sample the two-party setting with batch size of 32.
The results also show that increasing the batch size is an effective way to reduce inference time and communication per sample. The number of communication rounds independent of the batch size, which means communication rounds can be amortized by using larger batch sizes. The number of bytes communicated is partly amortized as well because the size of weight tensors (e.g., in linear layers) does not depend on batch size. The results also show that whereas the number of communication rounds increases when moving from two-party to three-party computation, remains constant afterwards. The larger number of communication rounds for three-party paree- computation stems from the public division protocol, which requires additional communication rounds when more than two parties are involved to prevent wrap-around errors (see the appendix for details).
6.2 Speech Recognition
We performed speech-recognition experiments using Wav2Letter [17 the LibriSpeech dataset [3 The LibriSpeech dataset contains 16 kHz audio clips represented as a waveform (16, 000 samples per second). Because the audio clips vary in length, clip all of them to 1 second for the benchmark. Wav2Letter is a network with 13 convolutional layers using rectified linear unit (RcLU; (50]) activa- tions The network operates directly the waveform input, predicting plus special characters). The first two layers use filter size of 250 (with stride 160) and 48 (stride
of 29 labels (26 letters
2). The next seven layers use filter size 7, followed by two layers with filter size 32 and 1 (all with stride 1). All layers except the last two have 250 channels. The last two layers have 2, 000 channels.
The results in Figure 6 show that CRYPTEN is about 2.5-3 orders of magnitude slower than PyTorch depending on the number parties involved. For Wav2Letter, the overhead is largely due to ReLU layers in the network: evaluating ReLUU function requires a comparison, which involves
We used the reference implementation of Wav2Letter in torchaudio.
7

---

Datch
Ratch-D0
Batchabze
Commurtattee rease
Pance
Communtcationinn
5000
Oopoien
9000
Namber ofmbersfpartics
Namber of partics
Humberalpurtio
100030
Eetchvize
Batchsice
2230
Comsuat r0048
390030
1.44
Copplles
2230
200830
20"
Pytorth
000800
parties
Numberol partles
parties
Figure 9: Benchmarks for inference with image-classffication models on GPUs in CRYPTEN and PyTorch. Top: Results for ResNet-18 model. Bottom: Results for VÝT-B/16 vision transformer. Left: Average wall-clock time per sample (in seconds). Middle: Number of bytes communicated per sample, per party (in GB). Right: Number communication rounds per sample.
conversion between arithmetic and binary secret sharing and back (see the appendix). The number of communication rounds increases when the number of parties grows beyond 4: CRYPTEN uses a tree reduction for the summation the comparator protocol, which implies that the number of communication rounds grows whenever the number of parties increases from 2 to 2*+1
Figure 7 also presents results comparing Wav2Letter inference time between CPUs and GPUs. The results in the figure show that CRYPTEN is 1-2 orders of magnitude faster on GPUs than on CPUs. In real-world settings, this speedup can make the difference between secure MPC use case being practical or not. Figure shows how much wall-clock time is spent on communication and computation, respectively, when performing inference with Wav2Letter (using batch size 32). The results suggest that, whereas multi-pan -arty evaluation is compute- bound on CPU, it is communication- bound on GPU, On GPUs, 63% the time is spent on communication eight-party computation.
6.3 Image Classification
We performed image-classification experiments on the ImageNet dataset using residual networks (ResNets: (35)) and vision transformers (VIT; (26) 6 We experimented with a ResNet-18 with 18 convolutional layers and with a ViT-B/16 model that has 12 multi-head self-attention layers with 12 heads each, operating on image patches of 16 x 16 pixels. Following common practice (35) we preprocess images by rescaling them to size 256 x 256 and taking a center crop of size 224 x 224.
Figure 9 presents the results of our image- classifcciin benchmarks, which show that two parties can securely evaluate a ResNet- 18 model in 2.49 seconds and a ViT-B/16 model in 8.47 seconds. notable difference compared the prior results is that the number of bytes communicated per sample is no longer reduced by increasing the batch size. The reason for this is that the vast majority of communication involves tensors that have the same size intermediate activation functions: activation tensors are much larger than weight tensors in image-lassification models. The amount of communication required to evaluate the ViT-B/16 model particularly high due the repeated evaluation of the softmax function in the attention layer of Transformers [65] We also observe that in ResNet-18, 18, the number of communication rounds grows faster than expected for larger batch sizes. The reason for this is that the carry- larry-lookhhead adder (21 used in the conversion from [æ]] to (z) is very memory-intensive intensiive When CRYPTEN runs out of GPU memory, it replaces the adder by an implementation that requires O(P)) communication rounds (compared to (log, (Pl) for the carry-lookahead adder) but that requires less memory.
WW adopted the ResNet implementation from torchvision and the ViT implementation from https //github. com/ hwtp://atgggaa/g//ee////aaaaaart pytorch- image- image-models VIT's normalized mean squared error is larger than for other models because Gaussian error function approximation converges slowly; see Section ??.

---

7 Conclusion and Future Work
In this paper, we have introduced and benchmarked CRYPTEN. We hope that CRYPTEN'S flexible, machine-learning first API design and performance can help foster adoption of secure MPC in machine learning. We see the following directions for future research and development of CRYPTEN.
Numerical issues are substantially more common in CRYPTEN implementations of machine-lcarning algorithms than in their Py PoTcc counterparts. In particular, the tixed-poim representation with L bits of precision (L 16 by default) is more prone to numerical overflow or underflow than floating-point representations. Moreover, arithmetic secret shares are prone to wrap-around errors in which the sum of the shares [elp exceeds the size of the ring, Q= 264 Wrap-around errors can be difficult to debug because they may only arise in the multi-party setting, in which no individual party can detect them. We plan to implement tools in CRYPTEN that assist users in debugging such numerical issues.
Q=264.
End-to-end privacy requires seamless integration between data-processing frameworks, such as secure SQL. implementations [ and data-modeling frameworks like CRYPTEN In "plaintext" software, such frameworks are developed independently and combined via "glue code" or platforms that facilitate the construction of processing and modeling pipelines. Real- world use cases of machine learning via secure MPC require the development of a platform that makes the integration of private data processing and modeling seamless, both from an implementation and security point-of-view.
Differential privacy mechanisms may be required in real-world applications of CRYPTEN in order to provide rigorous guarantees the information leakage that inevitably occurs when the results of private computation are publicly revealed [28]. CRYPTEN implements sampling algorithms for the Bernoulli, Laplace, and Gaussian distributions (see appendix), which allows for the implementation of randomized response [68), the Laplace mechanism [29). and the Gaussian mechanism [6 28
(although care must be taken when implementing these mechanisms [12 46).. ). In future work, we
aim to use these mechanisms, for example, to do secure MPC implementation of DP-SGD (2].
Threat models may vary per use case. Specifically, some use cases may require malicious security or may not provide a TTP. Possible extensions may include support for malicious security via message authentication codes [22] as well as support for Beaver triple generation via additive homomorphic encryption [52], oblivious transfer [39] or more recent methods [10] to eliminate the need for a TTP.
Model architecture design for secure MPC is another important direction for future research. Following prior work in this research area, this study has focused on implementing existing machine- learning models in secure MPC framework. However, these models were designed based on computational considerations in "plaintext" implementations of the models on modern GPU or TPU hardware. The results of our benchmarks suggest that this may be suboptimal because those considerations are very different a secure MPC environment. For example, the evaluation of softmax functions over large numbers of values requires a lot of communication in secure MPC, which makes attention layers very slow. This implies that multilayer perceptron models [64] are likely much more efficient than vision transformers 26 65) for image classification. We hope that CRYPTEN'S machine-learning API and ease of use will spur studies that design model architectures specifically optimized for secure MPC environment, for example, via neural architecture search 44 47
8 Broader Impact
Although we believe that the adoption of secure MPC in machine learning can lead to the development of Al systems that are substantially more private and secure, we note that there are also potential downsides to such adoption. In particular, because the computations in secure MPC are performed on encrypted data, it can be harder to do quality control of Al systems implemented in CRYPTEN, For example, it is impossible to inspect the values of intermediate activations (or even model outputs) unless all parties agree to reveal those values. This may make it harder to explain why model makes a certain decision (5 or to detect data-noisn attacks [8. Indeed, there exist fundamental trade-offs between privacy and utility [57] and those trade-offs apply to CRYPTEN users, too.
also worth noting that, although the protocols implemented in CRYPTEN come with rigorous cryptographic guarantees, practical implementations of these protocols may be broken by other means. For example, we have no reason to assume that CRYPTEN would not be susceptible to side-channel attacks [61]. Hence, good data stewardship remains essential even when using secure computation.

---

Acknowledgments. We thank Joe Spisak, Sijun Tan, Gregory Chanan, Igor Fedan, and the Py PyTorch team for their support. We thank Mark Tygert, Anderson Nascimento, Amrita Roy Chowdhury, and anonymous reviewers for helpful discussions and feedback on early versions of this paper.
References
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp. G. Irving. M. Isard, Y. Jia, R. Joze- fowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Va- sudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Large-scale machine learning on heterogeneous systems, 2015.
TensorFlow:
[2] M. Abadi, A. Chu, I. Goodfellow, H. McMahan, Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the CCS, pages 308-318, 2016.
[3] A. Agrawal, A. N. Modi, A. Passos, A. Lavoie, A. Agarwal, Shankar, I. Ganichev, J. Leven- berg, M. Hong, R. Monga, and S. Cai. TensorFlow Eager: A multi-stage, python- eebbeddd dsl for machine learning. arXiv: Netiv:/903..00 /903.09833..55. 2019.
[4] T. Araki, J. Furukawa, Y. Lindell, A. Nof, and K. Ohara, High throughput semi-honest secure three-party computation with an honest majority. In ACM CCS, pages 905-817, 2016.
[5] D. W. Archer, D. Bogdanov, L. Kamm, Y. Lindell, K. Nielsen, J. I. Pagter, N. P. Smart, and R. N. Wright. From keys to databases real-world applications of secure multi-party computation. Computer Journal,
2018.
[6] B. Balle and Y.-X. Wang. Improving the Gaussian mechanism for differential privacy. In Proceedings of International Conference on Machine Learning, 2018.
[7] D. Beaver. Efficient multiparty protocols using eircuit randomization. In Annual International Cryptology Conference, pages 420-432 432. Springer, 1991.
[8] B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning (ICML), pages 1467-1474, 2012.
[9] D. Bogdanov, S. Laur, and J. Willemson. Sharemind: A framework for fast computations. In S. Jajodia and J. Lopez, editors, Computer Security ESORICS 2008, pages 192-206. Springer Berlin Heidelberg, 2008.
[10] E. Boyle, G. Couteau, N. Gilboa, Y. Ishai, L. Kohl, and P. Scholl. Correlated pseudorandom functions from variable-density LPN. In Cryptology ePrint Archive: Report 2020/1417, 2020.
[11] R. Canetti. Universally composable security: new paradigm for cryptographic protocols, In Proceedings of the Annual Symposium on Foundations of Computer Science, pages 136-145, 2001.
[12] C. Canonne, G. Kamath, and T. Steinke. The discrete Gaussian for differential privacy. In arXiv 2004.00010 2020.
[13] Cape Privacy. Trusted..d. URL
[14] O. Catrina and S. De Hoogh. Improved primitives for secure multiparty integer computation. In International Conference Security and Cryptography for Networks, pages 182-199, Springer, 2010.
[15] N. Chandran, D. Gupta, A. Rastogi, R. Sharma, and S. Tripathi. EzPC: Programmable, efficient, and scalable secure two-party computation for machine learning. In IEEE European Symposium on Security and Privacy, 2019.
[16] H. Chaudhari, A. Choudhury, A. Patra, and A. Suresh. Astra: High throughput 3pc over rings with application secure prediction. In ACM SIGSAC Conference on Cloud Computing Security Workshop, 2019.
10

---

[17] R. Collobert, C. Puhrsch, and G. Synnaeve. Wav2letter: An end-to-end convnet- basee speech recognition system. In arXiv:1609.03193, 2016.
[18] R. Cramer, I. Damgård, and Y. Ishai. Share conversion, pseudorandom secret-sharing and applications to secure computation. In Lecture Notes in Computer Science, volume 3378, pages 342-362 2005.
[19] cuBLAS, http:://eerllppp...vviiaaa...oc
[20] cuDNN. https://developerr
[21] I. Damgård, M. Fitzi, E. Kiltz, J. B. Niclsen, and T. Toft. Unconditionally secure constant-rounds multi-party computation for equality, comparison, bits and exponentiation. In TCC, 2005.
[22] Damgård, V. Pastro, N. Smart, and S. Zakarias. Multiparty computation from somewhat homomorphic encryption. Cryptology ePrint Archive, Report 201 1/535, 2011. https:// eprint. iacr. oroe/r////201115555
[23] I. Damgård, Escudero, T. Frederiksen, M. Keller, P. Scholl, and N. Volgushev. New primitives for actively-secure mpe over rings with applications to private machine learning. In Proceedings of the IEEE Symposium on Security and Privacy, 2019.
[24] D. Demmler, Schneider, and M. Zohner. ABY framework for efficient mixed pereddproooool secure two-party computation. In NDSS, 2015.
[25] F. Doshi-Velee and B. Kim. Towards a rigorous science of interpretable machine learning, In 17Xiv:1702.08608, 2017.
[26] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[27] T. Dugan and X. Zou. A survey of secure multiparty computation protocols for privacy preserving genetic tests. In Proceedings of the International Symposium on Biomedical Imaging, 2016.
(28) C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-25 2014.
[29] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[30] D. Evans, V. Kolesnikov, and M. Rosulek. A Pragmatic Introduction to Secure Multi-Paryy Computation. NOW Publishers, 2018.
[31] Gloo. ttttps://githann com httpa////.ccccccoooocaacattooo// gloo
[32] O. Goldreich, S. Micali, and A. Wigderson. How play any mental game completeness theorem for protocols with honest majority. In STOC, pages 218-229 1987.
[33] V. Haralampieva, D. Rueckert, and J. Posserat-Pallbach A systematic comparison of encrypted machine learning solutions for image classification. In Proceedings of the Workshop on Privacy- Preserving Machine Learning in Practice, pages 55-59, 2020.
[34] M. Hastings, B. Hemenway, D. Noble, and S. Zdancewic. SoK: general-purpose compilers for secure multi-party computation. In 2019 IEEE Symposium Security and Privacy (SP), 2019.
[35] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[36] A. S. Householder. The Numerical Treatment of a Single Nonlinear Equation. 1970.
[37] M. Ito, A. Saito, and T. Nishizeki. Secret sharing scheme realizing general access structure. Flectronics and Communications in Japan (Purt III: Fundamental Electronic Science), 72: 56-64, 1989.
11

---

[38] C. Juvekar, V. Vaikuntanathan, and A. Chandrakasan. Gazelle: A low latency framework for secure neural network inference. In arXiv 1801.05507, 2018.
[39] M. Keller, E. Orsini, and P. Scholl. MASCOT: Faster malicious arithmetic secure computation with oblivious transfer. In Proceedings of the ACM SIGSAC Conference Computer and Communications Security. pages 830-842, 2016.
[40] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
[41] N. Kumar, M. Rathee, N. Chandran, D. Gupta, A. Rastogi, and R. Sharma. Cryp'TFlow: Secure TensorFlow inference. In IEEE Symposium on Security and Privacy, pages 336-353 0563,,3, 2020.
[42] A. Lapets, N. Volgushev, A. Bestavros, F. Jansen, and M. Varia. Secure multi-party computation for analytics deployed as a lightweight web application. Technical Report -U-CS-TR 2016-008, Boston University, 2016.
[43] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE,
1998.
[44] H. Liu, K. Simonyan, and Y. Yang. 17Xiv
DARTS: Differentiable architecture search. In
2018.
[45] J. Liu, M. J. Y. Lu, and N. Asokan. Oblivious neural network predictions via MiniONN transformations. In ACM Conference on Computer and Communications Security (CCS), 2017.
[46] I. Mironov. On significance of the least significant bits for differential privacy. In Proceedings ACM Conference on Computer and Communications Security (CCS), pages 650-66) 2012.
[47] Mishra, R. Lehmkuhl, A. Srinivasan, W. Zheng, and R. Popa. Delphi: A cryptographic inference service for neural networks. In USENIX Security Symposium, 2020.
[48] Mohassel and P. Rindal. ABY3: mixed protocol framework for machine learning. In ACM Conference on Computer and Communications Security (CCS), 2018.
[49] P. Mohassel and Y. Zhang. SecureML: A system for scalable privacy-preservin machine learning. In JEEE Symposium on Security and Privacy, 2017.
[50] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the International Conference on Machine Learning (ICML), 2010.
[51] NCCL. https:://evellpper.nr nvidia. com/ncll
[52] P. Paillier. Public-key cryptosystems based on composite degree residuosity classes. In EUROCRYPT, pages 223-238, 1999.
[53] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. LibriSpeech: An ASR corpus based on public domain audio books. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015.
[54] A. Paszke, S. Gross, S. Chintala, and G. Chanan. PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration, 2017.
[55] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, Bai, and S. Chintala. PyTorrch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[56] A. Patra and A. Suresh. Blaze: Blazing fast privacy preserving machine learning. In Symposium on Network and Distributed System Security (NDSS), 2020.
[57] I. S. Reed. Information theory and privacy data banks. Proceedings of the June 4-8, 1973, National Computer Conference and Exposition, 1973.
12

---

[58] M. S. Riazi, C. Weinert, O. Tkachenko, E. M. Songhori, T. Schneider, and F. Koushanfar. Chameleon: hybrid secure computation framework for machine learning applications. In Cryptology ePrint Archive, volume 2017/1 164, 2017.
[59] M. S. Riazi, M. Samragh, H. Chen, K. Laine, K. Lauter, and F. Koushanfar. Xonn: Xnor-based oblivious deep neural network inference. USENIX Security, 2019.
[60] T. Ryffel, A. Trask, M. Dahl, B. Wagner, J. Mancuso, D. Rueckert, and J. Basserat-Palmbachh A generic framework for privacy preserving deep learning. In 1rX 2018.
[61] O. Seker, S. Berndt, L. Wilke, and Eisenbarth. SNI-in-hhe-head: Protecting MPC-in-the-heae protocols against side-channel analysis. In Cryptology ePrint Archive: Report 2020/544, 2020.
[62] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[63] Tan, B. Knott, Y. Tian, and D. J. Wu. CryptGPU: Fast privacy-preserving machine learning on the GPU. In arXiv 2104. 2021.
[64] I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy. MLP-Mixer: An all-MLP architecture for vision. In arXiv:2105.0106
2021.
[65] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[66] Wagh, D. Gupta, and N. Chandran. SecureNN: Efficient and private neural network training. In Cryptology ePrint Archive, volume 201 2014/4 2018.
[67] S. Wagh, S. Tople. F. Benhamouda, E. Kushilevitz, P. Mittal, and T. Rabin. FALCON: Honest- majority maliciously secure framework for private deep learning. Proc. Priv. Enhancing Technol., 2021.
[68] S. L. Warner. Randomized response: survey technique for eliminating evasive bias. Journal of the American Statistical Association, 60(309):63-69, 1965.
[69] A. C.-C. Yao. How to generate and exchange secrets. In FOCS, pages 162-167, 1986.
[70] Yelp. Yelp Review Dataset. URL hoRllattpee//ww..elpccoo///a//e
[71] B. Zoph and Q V. Le. Neural architecture search with reinforcement learning. In axX1v:161.01177,,2 2016.
13


Reviews:
[{"review_id": "vPtF5DWJ-C", "reviewer": "Reviewer_ye4J", "summary": "This paper presents a general software framework with GPU support for secure machine learning over multiple (arbitrary number of) parties: CrypTen. It provides a friendly way for people who don't have any cryptography background to implement the secure machine learning tasks, which could tolerate all-but-one corruptions in semi-honest case. It also demonstrates its capability and efficiency in realizing the state-of-the-art models for text classification, speech recognition, and image classification.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly written and technically sound. The experiments well demonstrates the performance of this software framework. \n\nStrength: the software framework is easy to use and requires no knowledge of cryptography. It's the first public framework to realize secure modern machine learning tasks among arbitrary number of parties. And it shows a competitive performance for those tasks.\n\nWeakness: \n - As stated in the paper, it only provides secure against semi-honest adversaries. While other works presents malicious security, but for 2 to 4 parties. \n - In the current implementation, the security relies on a Trusted Third Party.\n - Most of the techniques were already presented here: https://eprint.iacr.org/2019/599.pdf   I think the authors should also cite this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Those are properly addressed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "pcYb-2W7Sah", "reviewer": "Reviewer_PL5J", "summary": "The authors present an implementation for secure computation of deep learning inference and training. It is based on PyTorch, which makes its use easy for programmers familiar with the underlying framework. The secure computation is semi-honest and uses a trusted third party, which is assumed not to collaborate with any of the other parties.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the effort towards making secure computation more accessible deserves attention, and I appreciate the work the authors have done in this direction.\n\nThe authors hint at MPC compilers being potentially faster. What order of magnitude would that be? The authors refer to numerous works, but don't mention any results. For example, CrypTFlow takes 25.9 seconds and 6.9 GB for ResNet-50. In particular, the communication figure seems to be much better than the about 50 GB for ResNet-18 shown in this work.\n\nThe paper says that using a ripple-carry adder takes O(log(L)) rounds for L being the number of bits, which is wrong. Ripple-carry addition refers to the textbook addition that O(L) rounds. An adder taking O(log(L)) rounds would be a carry-lookahead adder.\n\nI'm missing an argument in Section 5.3 for how much time using the GPU saves. What is the improvement per clock cycle or second?\n\nThe authors say that the results for private training are very similar to inference. I would expect that, in the context MPC, training fares better due to implicit parallelization. Is that what the authors mean here?\n\nI consider the frequent announcement of future features (\"we plan to add/support...\") out of place in an academic paper, and I would prefer a more neutral style, say \"a possible extension would be...\".\n\nMinor:\nl244: Please move the URL to a footnote or the appendix.\nl317: should be Newton-Raphson\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "od8jkBtIw_-", "reviewer": "Reviewer_oKEp", "summary": "This work is to fill the gaps of software framework for MPC used in Machine Learning. It provides many MPC primitives and fits them into ML scenario.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. A great contribution to allow ML researcher, who may be lack of solid cryptography knowledge, have a chance to work on MPC-based privacy of ML with relatively easy access to implementation.\n2. This work does more than basic ML by providing tensor computation, automatic differentiation and modular neural network to facilitate works deep learning with PyTorch-friendly APIs for state-of-the-art CV and NLP works.\n3. With ability to support CrypTen on GPU, it achieves great performance on multiple benchmarks, such as image recognition and speech recognition.\n\nCons:\n1. Although CrypTen did great jobs on passive security allowing an arbitrary number of parties, active security is also crucial to protect ML privacy, and it may helps more on real-world applications, since malicious adversary deviate cryptographic protocols with any arbitrary action.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact has seen.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "kyhDSz-amm0", "reviewer": "Reviewer_aZn5", "summary": "The authors proposed crypten, a new deep learning framework for secure multi-party computation (MPC) based on the PyTorch API. One of the important obstacle in adoption of MPC for deep learning is that using the existing MPC libraries requires extensive knowledge about MPC. A big advantage of crypten is that it follows the same familiar API of PyTorch and hides the MPC complexity behind the API. The provided code snippets demonstrate that using crypten is certainly easy for non-experts. They also support GPU acceleration for arithmetic computation.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- The authors use existing well-known MPC techniques and optimization to create the framework.\n- Following the exact API of a well-known deep learning framework, like PyTorch, for a MPC library is novel.\n- The authors could've done a better job in reviewing and comparing the existing MPC frameworks for deep learning, for example, a table comparing the features of different frameworks. \n\n\nQuality:\n- The code snippets and the experiments on different application sufficiently demonstrated crypten's ease-of-use.\n- The shortcomings of the framework are also presented and explained.  \n\n\nClarity:\n- The paper is well written and organized and it is easy to follow.\n\n\nSignificance: \n- The researchers and practitioners will find the work interesting and can probably build on top of the framework.\n- The paper is the first to present a framework which follows a well-known deep learning API.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitation and shortcoming of the paper and their plan on how to extend the work. They also mentioned the possible broader impact of their work on the society and the community. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is an agreement in the reviews that the paper’s secure multi-party computation (SMPC) framework for deep learning training and inference provides a valuable contribution. While most of the techniques are already known and the protection is only against honest-but-curious (as opposed to malicious) adversaries, the paper seems to fill an important gap and the contributions are a step forward towards broader deployment of SMPC protocols for ML primitives, and could stimulate future work in the area. I therefore lean towards acceptance.
Decision: Accept (Poster)