Secure Quantized Training for Deep Learning
Authors: Marcel Keller
Ke Sun
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Federated learning
secure multi-party computation
quantization
deep learning

Abstract: We have implemented training of neural networks in secure multi-party
computation (MPC) using quantization commonly used in said setting. To
the best of our knowledge, we are the first to present training of
MNIST purely implemented in MPC that comes within one percent of
accuracy of training using plaintext computation.  We found that
training with MPC is possible, but it takes more epochs and achieves a
lower accuracy than the usual CPU/GPU computation.  More concretely,
we have trained a network with two convolution and two dense layers to
98.5% accuracy in 150 epochs. This took a day in our MPC
implementation.


Secure Quantized Training for Deep Learning
Anonymous Author(s) Affiliation Address email
Abstract
We have implemented training of neural networks in secure multi-party computa- tion (MPC) using quantization commonly used in said setting. To the best of our knowledge,
2
the first present training of MNIST purely implemented in MPC that comes within one percent of accuracy of training using plaintext compu- tation. We found that training with MPC is possible, but it takes more epochs and achieves a lower accuracy than the usual CPU/GPU computation. More concretely, we have trained network with two convolution and two dense layers 98.5% accuracy in 150 epochs. This took day in our MPC implementation.
Introduction
10 Secure multi-party computation (MPC) is a cryptographic technique that allows a set of parties to the inputs any intermediate results.
compute public output private inputs without revealing This makes it a potential solution to federated learning where the sample data stays private and only the model
12
only inference results revealed.
14 Imagine a set of healthcare providers holding sensitive patient data. MPC allows them to collabo- ratively train model. This model could then either be released or even kept private for inference using MPC again. See Figure for an illustration. millionaires' problem where two people want to find out who is richer without revealing their wealth. There is clearly a difference between the one bit information desired and the full figures.
15
16
more conceptual example is the well-known
18
19
There has been sustained interest in applying secure computation to machine learning and neural networks going back at least (Brni et al. [2006). More recent advantages practical MPC have led to an increased effort in implementing both inference and training.
20
21
number of works such Mohassel and Zhang [2017], Mohassel and Rindal 2018 Wagh et al. 2019), Wagh et al.] [2021] implement neural network training with MPC at least in parts. However, they either give accuracy figures below 95% figures that have been obtained using plaintext training.
29
25 For the latter case, the works do not clarify how close the computation for plaintext training matches
26 the lower precision and other differences in the MPC setting. Agrawal al. 2019 claim higher accuracy in a comparable setting for a convolutional neural network with more channels than we
27
28 use. However, they have only implemented dense layers, and we achieve comparable accuracy to them with only dense layers. All works use quantization in the sense that fractional number is represented as 2 $1. This makes addition considerably faster in the secure computation setting
29
30
31 because it reduces to integer addition. Furthermore, some of the works suggest to replace the softmax
32 function that uses exponentiation with a Rel.U-based replacement. Keller and Sun (2020) have found
33 that this softmax replacement deteriorates accuracy in dense neural networks to the extent that it does not justify the performance gains.
34
35 The concurrent work of Tan et al. 2021 gives some figures on the learning curve when run using
36 secure computation. However, they stop at five epochs for MNIST training where they achieve 94% accuracy whereas we present the figures up to 150 epochs and 98.5% accuracy. Furthermore, their
37
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Figure 1: Outsourced computation: Data holders (on the left) secret-share their data to number of computing parties (on the right), who then return the desired the result (e.g., a model or inference results on further queries). All communication except outputs are secret-shared and thus secure if no two computing parties collude.
38 choice of fixed- meed-point precision 20 considerably below 32, which we found to be optimal. We also
39 found that our implementation is 40% faster than theirs. Note that we use the CPU of one AWS c5. 9xlarge instance per party whereas Tan et al. use one NVIDIA Tesla V100 GPU per party. We believe this somewhat counter-intuitive result comes from MPC heavily relying communication, which is an aspect where GPUs do not have an advantage over CPUs.
40
41
42
43
In this paper, we present an extensible framework for implementing deep learning training based on MP-SPDZ by Keller 22020). a framework for multi-party computation Similar TensorFlow and Py Torch, our approach allows representing deep learning models as succession of layers. We then use this implementation to obtain accuracy figures for MNIST training by utilizing the MP-SPDZ emulator, which allows to run the plaintext equivalent of secure computation, that is, the same algorithms with the same precision. Finally, we run one of the most promising instantiation in real secure computation in order to benchmark it confirm the result from the plaintext emulator.
44
45
48
47
48
49
50
There are a number of projects that integrate secure computation directly into popular machine learning frameworks such as Cryp'Ten by Gunning et al. [2019). PySyft by [Ryffel et al. [2018. and TF Encrypted by Daal et al. [2018]. Our approach differs from all of them by running the protocol as native CPU code (implemented using C++). This allows for much faster execution. For example, CrypTen provides an MNIST training example ampc_autograd. cnn) that takes over one minute
51
53
54
55
to run one epoch with 100 samples on one machine. In comparison, our implementation takes minutes to run one epoch with the full dataset of 60,000 samples.
56
57
Another line of work (e.g.. Quoe et al. [2021]) computation outside the reach of the operating system. This is different security model to multi party computation that works with distributing the information among several entities.
|) uses trusted execution environments that provide
59
60
The paper is structured as follows: After introducing the basics of the protocol we use in Section 2 we will explain the mathematical building blocks in Section 3 and their use in the context of deep learning in Section 4 Finally, we will present implementation in Section 5 and our experimental results for MNIST in Section
61
63
64
2
An Efficient Secure Multi-Party Computation Protocol
65
There a wide range of MPC protocols with a variety of security properties (see Keller (2020) for an overview). In this paper we focus on the setting of three-pary computation with one semi-honest corruption. This means that out of the three parties two are expected to behave honestly, i.e., they follow the protocol and keep their view of the protocol secret, and one party is expected to follow the protocol but might try extract information from their view. The reason for choosing this setting is that it allows efficient MPC protocol while still allowing secure outsourced computation. The concrete protocol we use goes back to Benaloh and Leichter [190]] with further aspects by Araki et al. [2016). Mohassel and Rindal 2018). and Ecrikson et al. 2020 We summarize the core protocol below. The mathematical building blocks in the next section mostly use the aspects below.
66
67
68
69
71
73
We are committed to publishing code open source.

---

74 Secret sharing All intermediate values in our protocol are stored using replicated secret sharing. A
75 secret value x is a represented as a random sum z Zi + 21 + 22. and party P; holds (I;-1, 21++)
76 where the indices are computed modulo three. Clearly, each party is missing one value to compute
77 the sum, On the other hand, each pair of parties hold all necessary to reconstruct the secret. For a
78 uniformly random generation of shares, the computation domain has to be finite. Most commonly,
79 this domain is defined by integer computation modulo a number. We use 2 for & being a multiple 64 and 2 as the moduli. The first case corresponds to extension of 64-bit arithmetic found on most
81 processors. We will refer to the two settings as arithmetic and binary secret sharing throughout the paper.
85
Input sharing The secret sharing scheme implies a protocol to share inputs where the inputting
84
party samples the shares and distributes them accordingly. [Erikson al. [2020) have proposed more efficient protocol where the inputting party only needs to send one value instead of two pairs of 85
86
values. If P. would like to input I, Ij is set to zero, and z;-1 is generated with pseudo-random
87
generator using key previously shared between P; and P.+1. P; can compute T;+1 = x 2;-1 and send it to P.-1- While the resulting secret sharing is not entirely random, the fact that P. already knows E makes randomizing I; obsolete.
89
The commutative nature of addition allows to add secret sharings without communication. More concretely, secret sharings x = Ho + #1 + 22 and y = yo + y1 + 1/2 imply the secret sharing y = (ro + vo) + (21 + y1) + (z2 + 922)
Addition
91
95
Multiplication The product of x = x + 2n + 22 and y = yo + y + 1/2 is
x y = (aro + 21 + 22) (yo + y + 12)
= (roWOo + zo8i + 2 00))) + (#19)) + x192 + x11)) + (x23/2 + 0230 + x00/2)
94 Each of the brackets only contains shares known by one of the parties. They can thus compute an additive secret sharing (one summand per party) of the product. However, every party only holding one share does not satisfy the replication requirement for further multiplications. It is not secure for every party to pass their value on to another party because the summands are not distributed randomly. This can be fixed by rerandomization: Let xy 20 + ži + 22 where z; is know to P.. Every party P. computes =/ = 2; + Ti,++1 1i-1, where Ti,++1 is generated with a pseudo-random generator using key pre-shared between P. and P,+1. The resulting sum zy = + 2 + zl is aacudo-randoms and it is thus secure for P. to send 2/ to P.+1 in order to create a replicated secret sharing (2y))+1 (2 241))
95
96
97
99
101
102
3 Secure Computation Building Blocks
103
104
In this section, we will discuss how to implement computation with MPC with a focus on how it differs from computation on CPUs GPUs Most of the techniques below are already known individually. To the best of our knowledge however, we are the first to put them together in an efficient and extensible framework for secure computation of deep learning training.
105
106
107
Domain conversion Recall we that we use computation modulo 2 for being a multiple of 64 well as 1. Given that the main operations are just addition and multiplication in the respective domain, it is desirable to compute integer arithmetic in the large domain but operations with a straight-forwad binary circuit modulo two. There has been long-running interest in this going back to least Kolessnkko et al. 2013) We mainly rely on the approach proposed by Mohassel and
108
110
111
112
113
Rindal (2018) and Araki et al. 2018 Recall that E 2 is shared as x
27g + 21 + 22. Now let k-1 the bit decomposition of 2p. that is, 2.(i) {0, 1} and To = Σ i= zo (0)2 is self-evident (x)
k-
114
===
(6)
115
that = = x0 +0+ 0 is a valid secret sharing modulo two (albeit not a random one). Furthermore,
every party holding 2o can generate (4) It is therefore possible for the parties to generate secret sharing modulo two of a single share modulo 2 Repeating this for all shares and the computing
116
117
118
the addition as binary circuit allows the parties to generate a secret sharing modulo two from a secret sharing modulo 2*, Conversion in the other direction can be achieved using a similar technique
119
120
or using daBits as described by Rotaru and Wood [219]. In the following we will use the term computation for any technique that works over both computation domains.
3

---

122
Quantization While Aliasgrr et al. [2033] showed that it is possible to implement floating-pons computation, the cost is far higher than integer computation. It is therefore common to represent fractional numbers using quantization (also called fixed-point representation) as suggested by Catrina and Saxena 2210]] A real number x is represented as 7 = [ 2/ where f is an integer specifying the precision. The linearity of the representation allows to compute addition by simply adding the representing integers. Multiplication however requires to adjust the result because it will have twice the precision: (x 2 (y 2 = ry 2 There are two ways to rectify this:
123
124
125
126
127
An obvious correction would be to shift the result by f bits after adding 2 to the integer representation. This ensures rounding to the nearest number possible in the representation, with the tie being broken by rounding up. Dalskov et al. [202]] presented an efficient implementation of the truncation using mixed-circuit computation.
130
132
139
However, Catrina and Saxena have found that in the context of secure computation it is more efficient to use probabilistic truncation. This method rounds up or down probabilistically depending on the input. For example, probabilistically rounding 0.75 to an integer would see it rounded of up with probability 0.75 and down with probability 0.25. The probabilistic truncation is an effect of the fact that the operation involves the truncation of a randomized
134
135
138
137
value, that is the computation of [xx + r)/2m for a random m-bit value r. It is easy to see that
138
139
|x/2"|] ] [(x + +)/ = | + 1
(x mod 2nd + r) < 2m
(x mod 2nd + r) 2 2mh
140
Therefore, the larger (z mod 2nt)) is, the more likely the latter condition is true. Dalskov et al. 2020] present an efficient protocol in our security model.
142
Our quantization scheme is related to quantized neural networks (see e.g. Hubara et al. 2016) ) However, our consideration is not to compress the model, but to improve the computational speed and save communication cost.
143
144
145
Dut products Dot products are an essential building block of linear computation such as matrix multiplication. In the light of quantization, it is possible reduce the usage of truncation by deferring after the summation. In other words, the dot product in the integer representations is computed before truncating. This not only reduces the truncation error, it is also more efficient because the truncation is the most expensive part in quantized secure multiplication. Similarly, our protocol allows to defer the communication needed for multiplication. Let o and ÿ be two vectors where the elements are and similarly for y The inner product then is + x
146
147
148
149
150
+ + secret shared, that is, (x( = (0(0)
(i)
151
(0) yl = Σ (1 + 1
(1)
(0)
(1)
+ I22
+
+ U/2
Σ
(8) (6)
(0) (1)
(1) )+ + Σ (of ( vi
(6) ( + +21 122 + +
( yo + 00
9/
+ 1 No
16) 18) y2 12 90
(4)
(8) (
+ 2 y2
152
The three sums in the last term can be compute locally by one party each before applying the same protocol as for single multiplication.
154
Comparisons Arithmetic secret sharing does not allow to access the individual bits directly. It therefore not straightforward to compute comparisons such as "less than". There is a long line of literature on how to achieve this going back to at least Damgard et al. [2006). More recently, most attention has been given to combine the power of arithmetic and binary secret sharing in order to combine the best of worlds. One possibility to do so is to plainly convert to the binary domain and compute the comparison circuit there. In our concrete implementation we use the more efficient approach by Mohaassl and Rindal (2018). It starts by taking the difference of the two
155
156
157
158
159
160
161
inputs. Computing the comparison then reduces to comparing to zero, which turn is equivalent
162
to extracting the most significant bit as it indicates the sign. The latter is achieved by converting
163
the shares locally to bit-wise sharing of the arithmetic shares, which sum up to the secret value.
remains to compute the sum of the binary shares in order to come up with the most significant bit.

---

185
Oblivious Selection Plain secure computation does not allow branching because the parties would need be aware which branch is followed. Conditional assignment can be implemented follows however. If b E {0. 1} denotes the condition, + b- (y x) is either æ or y depending on b. If the condition is available in binary secret sharing but x and y in arithmetic secret sharing, b has to be converted to the latter. This can be done using a daBit as introduced by |Rotaru and Wood (2019). which is secret random bit shared both in arithmetic and binary. It allows to mask a bit in one world by XORing it. The result is then revealed and the masking is undone in the other world.
186
167
168
169
170
172
Division Catrina and Saxena [010] have shown how to implement quantized division using the algorithm by Goldschmidt [194]. It mainly uses arithmetic and the probabilistic truncation already explained. In addition, the initial approximation requires a full bit decomposition as described above. The error of the output depends on the error in the multiplications used for Goldschmidt's s iteration, which compounds in particular when using probabilistic truncation. Due to the nature of secure computation, the result of division by zero is undefined. One could obtain a secret failure bit by testing the divisor to zero. However, we found that not to be necessary in our algorithm. This is because we only use division by secret value only for the softmax function where the it is guaranteed to strictly positive.
173
174
176
178
180
Logarithm Computation logarithm with any public base can be reduced logarithm to base two using log, y = log, y log, 2. Aly and Smart [2019] have proposed computing y = a 2 where E 00.5, 1) and b E Z. This then allows to compute log, y = logg a + b. Given the restricted range of a, log, a can be approximated using a division of polynomials. Numerical stability and input range control are less of issue here because we only use logarithm for the loss computation, which does
182
184
185
186
not influence the training.
Exponentiation By using xº 2 log, #, any exponentiation be reduced to exponentiation
with base two. Aly and Smart [2019] have shown how to compute 2n = -2(a] 20- by computing
189
the two exponents using bit decomposition and the second factor using a polynomial apprxxiatioo.
190
Regarding the first factor, if b Eizo 6,2º is an integer with b, {0, 1),
2 2022a 4,2' = II 24:2'
[[ + by (22' - 1)). 12:
120
191
As with division the numerical stability depends on the truncation used for multiplication.
192
2019] have proposed to compute square root using Gold- schmidt and Raphson-Noe Newton iterations. We could combine this with the division above. However, Lu
Inverse square root Aly and Smart
193
194
ct al. 2020) have proposed a more direct computation that avoids rumning two successive iterations.
195
Uniformly random fractional number Limiting ourselves to intervals of the form [x., x + 2n] for a potentially negative integer 6, we can reduce the problem to generate a random (ʃ + e)-bit number where f is the fixed-point precision. Recall that we represent a fractional number x as [x 2-/1. Generating a random ri-it number is straight-forwar using random bits, which in our protocol can be generated as presented by Damgard al. (2019). In the context of our protocol however, Dalskov
196
197
198
199
200
et al. [221 have presented a more efficient approach that involves mixed-circute computation.
201
Communication cost Table show the total communication cost of some of the building blocks in our protocol for f = This setting mandates the modulus 2128 because the division protocol requires bit length 4f.
202
203
4 Machine Learning Building Blocks
204
205
In this section, we will use the building blocks in Section to construct high-level computational modules for deep learning.
206
Fully connected layers Both forward and back-propagen of fully connected layers can be seen 208 as matrix multiplications and thus can be implemented using dot products. A particular challenge in secure computation is to compute number of outputs in parallel in order to save communication
5

---

Table I: Communication cost of select computation for 32 and integer modulus 2
Bits
Integer multiplication
384
Probabilistic truncation
1,536
Nearest truncation
4,462
Comparison
1,369
Division (prob. truncation)
29,866
Division (nearest truncation)
57,798
Exponentiation (prob, truncation)
77,684
Exponentiation (nearest truncation)
171,638
Invert square root (prob. truncation)
20,073
Invert square root (nearest truncation)
27,699
210
rounds. We solve this by having a dedicated infrastructure in our implementation that computes all dot products for matrix multiplication in single batch, thus combining all necessary communication.
212
2D convolution layers Similar to fully connected layers, 2D convolution and its corresponding gradient can be implemented using only dot products, and we again compute several output values in parallel.
214
215
Rectified Linear Unit (ReLU) ReLU Nair and Hinton 2010 is defined as follows:
ReLU(z) = xr if x
00. otherwise
216
can thus be implemented as a comparison followed by an oblivious selection. For back- preck--pppggttinn is advantageous to reuse the comparison results from forward propagation due the relatively high cost in secure computation. Note that the comparison results are stored in secret-shared form and thus there is reduction in security.
217
218
219
220
Max pooling Similar to ReLU, max pooling can be reduced to comparison and oblivious selection. In secure computation, it saves communication rounds if the process uses balanced rather than iterating over all input values of one maximum computation. For back bock-propagation it again pays off the store intermediate results from forward propagation, again secret-shared form.
222
223
224
Softmax and cross entropy loss 225
This combination requires computing the following gradient for
back-propa
08
--E Uk 2t + log Σ c -- Yi
(1)
3r:
Or;
Σ, c
226 where yi denotes an element of the ground truth as one-hot vector, and x; denotes the output of the last layer.
the right hand side of eq. T the values in the denominator are potentially large due to the use of the exponential. This is prone numerical overflow in our quantized representation because the latter puts relatively strict limits on the values. We therefore optimize the computation by computing the maximum of the input values:
228
230
m = max ({xz)) j
Then compute
t e
Σ
e
E, e )
Σ, E,
All the exponents on the left-most term most and thus the dividend is at most one and the 234 divisor is at most the number of labels (which is 10 MNIST), The same technique can be used to
235 compute the sigmoid activation function, as sigmoid(
exp(0) 1+xp((->) exp(O)+exx(zz) is special
case of softmax.

---

237 Stochastic gradient descent
The model parameter update in SGD only involves basic arithmetic: 0; 6 02 - मे Σ Vis where 8j is the parameter indexed by j. B is the mini-batch size, y 0 is 239 the learning rate, and Vay is the gradient the loss with respect to the i'th sample in the mini-batch and the parameter f.. In order to tackle the limited precision with quantization, we defer dividing by the batch size to the model update. This means that we do not divide the gradient value by the batch divide the model
241
size when computing them as described in the previous paragraph. Instead,
243
update by the batch size. Since we use a batch size that is a power of two (128), it is sufficient to use probabilistic truncation instead of full-blown division. This saves both time and decreases the error.
244
245
The main difference to SGD in terms of basic computational operations is the additional use of an inverse square root. We again defer the division by the batch size to just before the model update.
Adam
247
248
Parameter initialization We use the Glorot initialization by Glorot and Bengio 2010]. Besides basic operations, it mainly involves generating uniformly random fractional value in given interval.
249
5
Implementation
250
251
MP-SPDZ. not only implements a range of MPC protocols, it also comes with high-level library containing the building blocks in Section 3 MP-SPDZ already featured capabilities to train dense neural networks as well as inference for convolutional neural networks. We have added backward propagation for a number of layer types, including 2D convolution. Furthermore, we have corrected a bug in the backward propagation for
We built our implementaiion on MP-SPDZ. by Keller 2020
252
253
254
255
256
dense layers.
257
MP-SPDZ allows implementing the computation in Python code, which is then compiled a specific bytecode. This code can be execute by virtual machine executing the actual secure computation. The process allows to optimize the computation in the context of MPC.
259
The framework also features an emulator that executes the exact computation that could be done securely in the clear. This allowed us to collect the accuracy figures the next section at a lower cost.
261
282
It is licensed under BSD-style license, which allows to extend the code.
MNIST
263
264
For concrete measurement of accuracy and running times, we have implemented training for the well-known MNIST dataset by LeCun et al. [2010]. We work mainly with the models that have been used by WWahh al. [2019] with secure computation, and we will reuse their numbering (A-D). The models contain up to four linear layers. Network C is a convolutional neural network going back
285
286
287
268
to the seminal work by LeCun et al. 1998 whereas the others are simpler networks that have been
269
proposed by works on secure computation such as Mohassel and Zhang 2017 Liu et al. [2017] and
270
Riazi et al. [0118. We present the networks as Keras code in the supplemental material.
271 Fig Figure 2 shows the results for various quantization precisions and and the two rounding options. We have used SGD with learning rate 0.01, batch size 128, and the usual MNIST training/test split. = 64 is the best option with probabilistic rounding, improving on both f = 16 and f = 32. Furthermore, nearest rounding performs worse that probabilistic for f 16 and f 32. Due the high cost, we only ran f = 32 with probabilistic rounding several times. The range is indicate by the shaded area. We focus on f 32 because it offers the faster convergence.
272
273
274
275
276
277
Figure then shows the result with a variety of optimizers. While increasing the learning rate for SGD leads to lower stability, Adam exposes smoother learning learning curve albeit not faster process.
279
Finally, Figures 4 shows our results for all networks used by Wagh et al. 2019 As one would expect, the most sophisticated network performs best. Somewhat surprisingly, however, Network A (without convolutional layers) performs better than the simpler networks containing convolutional layers.
281
7

---

100
16. prob.
0.02
32, prob.
64. prob.
16, nearest
10
0.04
32, nearest
loss
0.06
10
0.08
10
0.1
0
50
100
150
#epochs
Figure 2: Loss and accuracy for network and precision options when running SGD with rate 0.01.
SGD, rate 0.01
SGD, rate 0.05
100
Adam, rate 0,001
Adam, rate 0,01
0.1
Adam, rate 0.1
10-1
loss
error
10
0.2
10
0.3
20
40
60
80
100
#epochs
Figure 3: Loss and accuracy for network C with various optimizer options, f = 32, and probabilistic truncation.
283 Resources
We ran the emulator on AWS c5. .xx arga instances. One epoch takes few second to several minutes depending on the model. Overall, we estimate that have used few weeks worth of computing time including experiments not included here because of bugs in the code.
285
6.1
Secure computation
In order to verify emulation results, have Network C with precision f 32 and probabilistic rounding in our actual multi-party computation protocol. We could verify that converges 98.5% accuracy 150 epochs, taking 20 hours. Table compares our result previous
288
289
280
works in a LAN setting. Note that Wagh et al. [2019] and Wagh et al.) [2021) give accuracy figures.
From personal communication with the authors and the fact that the source repository for the latter w says that their "code has not run end-to-end training", we derive our assessment that their
292
293
figures do not reflect the secure computation.

---

Network A
100
0.02
Network
Network C
0.04
Network D
10
loss
0.06
еттог
10
0.08
10-3
0.12
0
50
100
150
#epochs
Figure Loss and accuracy for various networks, 32, and probabilistic truncation.
Table 2: Comparison previous work in the LAN setting. (*) Mohassel and Zhang 2017 and Agrawal et al. 2019) use a different security model and are thus incomparable. We include them for completeness. Two numbers refer online and offline time. Accuracy N/A means that the accuracy figures were not given or computed in a way that does not reflect the secure computation.
Network
Epoch time (s)
Acc. (# epochs)
Precision (f)
Mohassel and Zhang
2017
283/19333*
93.4% (15)
13
Mohassel and Rindal
2018
180
94.0% (15)
N/A
A
Agrawal et al. 2019
31392*
95.0% (10)
N/A
Wagh et al. 2019
247
N/A
Wagh et al.
202T
41
N/A
13
Ours
97.9%
(15)
Ours
50
97.7%
(15)
32
Wagh et
2019
4176
N/A
13
B
Wagh ct al
2021
101
N/A
13
Ours
144
93.6%
(15)
16
Ours
249
94.7%
(15)
32
Wagh et al.
2019
7188
N/A
13
Wagh et al.
2021
891
N/A
13
Tan al. 2021
1036
94.0% (5)
20
Ours
344
94.9% (5)
Ours
643
93.8% (5)
32
Mohassel and Rindal [2018)
234
N/A
N/A
D
Ours
41
96.8% (15)
16
Ours
96.8% (15)
294
7
Conclusions
295
We have presented an implementation of deep learning training purely in multi-party computation with extensive results on the accuracy. have found that the lower precision of MPC computation increases the error considerably. We only have considered one particular implementation of more complex computation such as division and exponentiation, which are crucial to the learning process part of softmax. Future work might consider different approximations of these building blocks.
296
298

---

References
300
301
N. Agrawal, A. S. Shamsabadi, M. Kusner, and A. Gascón. QUOTIENT: Two-party secure neural network training and prediction. In L. Cavallaro, J. Kinder, X. Wang, and Katz, editors, ACM
302
303
CCS 2019, pages 123 1231- ACM Press, Nov. 2019, doi: 10.1 00.1145/3119555333388
304
M. Aliasgari, M. Blanton, Y. Zhang, and A. Steele. Secure computation on floating point numbers. In NDSS 2013. The Internet Society, Feb. 2013.
305
306
A. Aly and N. Smart. Benchmarking privacy preserving scientific operations. In R. H. Deng. V. Mauthier-Umaña M. Ochoa, and M. Yung. editors, ACNS 19, volume 1 11464 of LNCS, pages
307
308
509 529. Springer, Heidelberg, June 2019. doi: 10.1007/978-3-00000 000-9778---000-566--6-- 25.
T. Araki, J. Furukawa, Y. Lindell, A. Nof, and K. Ohara. High-throughput semi-honest secure three-party computation with an honest majority. In R. Weippl, S. Katzenbeisser, C. Kruegel, A. C. Myers, and S. Halevi, editors, ACM CCS 2016, pages 805-817. ACM Press, Oct. 2016. doi:
310
312
02.144//976744.297833
T. Araki, A. Barak, J. Furukawa, M. Keller, Y. Lindell, K. Ohara, and H. Tsuchida. Generalizing the SPDZ compiler for other protocols. In D. Lie, M. Mannan, M. Backes, and X. Wang, editors, ACM CCS 2018, pages 880-895. ACM Press, Oct. 2018. doi: 10. 00.1445/244333432444544
314
316
M. Barni, C. Orlandi, and A. Piva. A tion. In Proceedings of the 8th workshop Multimedia and security, pages 146-151, 2006.
protocol for neural-network- computa-
318
J. C. Benaloh and J. Leichter. Generalized secret sharing and monotone functions. In S. Goldwasser, editor, CRYPTO 188, volume 403 of LNCS, pages 27-35. Springer, Heidelherg, Aug. 1990. doi: 10.1007/0-387-347999
320
O. Catrina and A. Saxena. Secure computation with fixed-point numbers. In R. Sion, editor, FC 2010, volume 6052 of LNCS, pages 35-50. Springer, Heidelberg, Jan. 2010.
322
323
M. Dahl, J. Mancuso, Y. Dupis, B. Decoste, M. Giraud, I. Livingstone, J. Patriquin, and G. Uhma. Private machine learning in tensorflow using secure computation. CoRR, abs/18 11.0000160 2018.
324
325
URL hattp://arxiv...
wwwpaasss............ 08130
326
A. Dalskov, D. Escudero, and M. Keller. Fantastic four: Honest-majority four-party secure com- putation with malicious security. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
327
329
A. P. K. Dalskov, D. Escudero, and M. Keller. Secure evaluation of quantized neural networks. PoPETs, 2020(4):35-375,, Oct. 2020. doi: 2224778/ppett-2222--000
331 I. Damgärd, M. Fitzi, E. Kiltz, J. B. Nielsen, and T. Toft. Unconditionally secure constant-rounds multi-pan culti-partty computation for equality, comparison, bits and exponentiation. In S. Halevi and T. Rabin, editors, TCC 2006, volume 3876 of LNCS, pages 285-304. Springer, Heidelberg, Mar. 2006. doi: 10. 1007/ 168 116811111..
333
334
335 I. Damgård, D. Escudero, T. K. Frederiksen, M. Keller, P. Scholl, and N. Volgushev. New primitives for actively-secure MPC over rings with applications private machine learning. In 2019 IEEE Symposium on Security and Privacy, pages 102-1120. IEEE Computer Society Press, May 2019. doi: 10.1100//P.001.00000
336
337
338
339
H. Eerikson, M. Keller, C. Orlandi, P. Pullonen, J. Puura, and M. Simkin. Use your brain! Arithmetic 3PC for any modulus with active security. Y. T. Kalai, 2020, pages 5:1-4:24. Schloss Dagstuhl, June 2020. doi: 10.42330
340
Smith, and D. Wichs, editors, ITC
342
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 149-2556. JMLR Workshop and Conference Proceedings. 2010.
344
R. E. Goldschmidt. Applications of division by convergence. Master's thesis, MIT, 1964.
10

---

348
D. Gunning, A. Hannun, M. Ibrahim, B. Knott, L. van der Maaten, V. Reis, S. Sengupta, S. Venkatara- man, and X. Zhou. Crypten: new research tool for secure machine learning with pytorch. 2019.
347
349
I. Hubara, M. Courbariaux, D. Soudry, R. El- Yamiv, and Y. Bengio. Binarized neural networks, In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
351
M. Keller. MP-SPDZ: A versatile framework for multi multi-party computation. In J. Ligatti, X. Ou, J. Katz, and G. Vigna, editors, ACM CCS 20, pages 1575-1590. ACM Press, Nov. 2020. doi: 10.1145/3372297.44777
352
353
354
M. Keller and K. Sun. Effectiveness of MPC-friendly softmax replacement. 2020.
355 V. Kolesnikov, A.-R. Sadeghi, and T. Schneider. systematic approach to practically efficient general two-party secure function evaluation protocols and their modular design. Journal of Computer Security, 21(2):283-315, 2013.
358
357
358 Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86( (1:21112:7822311 1998.
359
380
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Avail- able: http: //t///aaaa yann. lecun. com/ eadb/ mistt 2. 2010. Creative Commons Attribution-Shan Alike 3.0 license, http:: attps:://creatiiecoome. www.::onnno..oooooooooeeoeoooooo
381
382
org/ httgg//leesess/yyeee
383
J. Liu, M. Juuti, Y. Lu, and N. Asokan. Oblivious neural network predictions via MiniONN transformations. In B. M. Thuraisingham, D. Evans, T. Malkin, and D. Xu, editors, ACM CCS 2017, pages 619-631. ACM Press, Oct. / Nov. 2017. doi: 10. 1 1
384
365
388
W.-j. Lu, Y. Fang, Z. Huang, C. Hong, C. Chen, H. Qu, Y. Zhou, and K. Ren. Faster secure multiparty computation of adaptive gradient descent. In Proceedings of the 2020 Workshop on Privacy- Preserving Machine Learning in Practice, PPMLP"20, 20, page 47-49, New York, NY, USA, 2020. doi: 10.1145/3411511.1 13314477.
387
368
369
Association for Computing Machinery. ISBN 9781450380881.
370
URL https: :/do:::///o.... org 10. 1145/3411555 3419427
Mohassel and P. Rindal. ABY3: A mixed protocol framework for machine learning. In D. Lie, M. Mannan, M. Backes, and X. Wang, editors, ACM CCS 2018, pages 35-52. ACM Press, Oct.
371
372
373
2018. doi: 10. 1...144/2433333324377777
374 P. Mohassel and Y. Zhang. SecureML: A system for scalable
machine learning. In 2017 IEEE Symposium on Security and Privacy, pages 19-38. IEEE Computer Society Press, May 2017. doi: 10. 109/SP.2017.1..
375
377 V. Nair and G. E. Hinton. Rectified linear units improve Restricted Boltzmann machines. In Pro- ceedings of the 27th International Conference on International Conference on Machine Learning. ICML' 10, pages 807-814, 2010.
378
379
380
D. L. Quoc, F. Gregor, S. Arnautov, R. Kunkel, P. Bhatotia, and C. Fetzer. securetf: A secure tensorflow framework. CoRR, 200//2201.0004,, 2021. URL settps://arxiv.oot/ httgg/bb//210111
381
08204
383
M. S. Riazi, C. Weinert, O. Tkachenko, E. M. Songhori, T. Schneider, and Koushanfar. Chameleon: A hybrid secure computation framework for machine learning applications. In J. Kim, G.-J. Ahn, S. Kim, Y. Kim, J. López, and T. Kim, editors, ASIACCS 18, pages 707-721. ACM Press, Apr. 2018
385
387
D. Rotaru and T. Wood. MArBled circuits: Mixing arithmetic and Boolean circuits with active security. In F. Hao, S. Ruj. and S. Sen Gupta, editors, INDOCRYPT 2019, volume 1 11898 of LNCS, pages 227-249. Springer, Heidelberg, Dec. 2019. doi:
389
T. Ryffel, A. Trask, M. Dahl, B. Wagner, J. Mancuso, D. Rueckert, and J. Passerat-Palmbach A generic framework for privacy preserving deep learning. CoRR, abs/181 1.04017, 2018. URL http://arxiss org/ httgg/ab/18811/ 08017
391
11

---

393
S. Tan. B. Knott, Y. Tian, and D. Wu. CryptGPU: Fast GPU, 2021.
machine learning on the
394
395
S. Wagh, D. Gupta, and N. Chandran. SecureNN: 3-party secure computation for neural network training, PoPETs, 2019(3):2-49, July 2019. doi: 10. 10.2478/popets-20199 200100000.
S. Wagh, S. Tople, F. Benhamouda, E. Kushilevitz, P. Mittal, and T. Rabin. Falcon: Honest-majority maliciously secure framework for private deep learning. PoPET's (021(1):188-2000 Jan. 2021. doi: 10. 10.2478/popets- 2001-0001.
398
Checklist
401
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
403
contributions and scope? [Yes]
404
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [No] We only
406
work with MNIST, and we believe that the potential negative impacts associated are very low.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
409
410
2. If you are including theoretical results...
411
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
412
(b) Did you include complete proofs of all theoretical results? [N/A]
413
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include all
415
416
code in the supplemental material, and the references include the URL for MNIST.
417
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section (
418
(c) Did you report error bars (c.g., with respect to the random seed after running experi-
420
ments multiple times)? [Yes] Only in Figure 2
421
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 6
422
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
424
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section G
(b) Did you mention the license of the assets? [Yes] See the references and and Section 5
(c) Did you include any new assets either in the supplemental material as URL? Yes)
427
Our code is included the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
429
430
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
432
5. If you used crowdsourcing or conducted research with human subjects...
433
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
434
435
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
436
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
438
12


Reviews:
[{"review_id": "oujCs-CY8bW", "reviewer": "Reviewer_U3fp", "summary": "The paper presents a secure multi-party computation (MPC) implementation of training several neural network architectures on MNIST. A 3-party semi-honest setup is considered with every individual party being honest-but-curious about the inputs of the other two parties. For a neural network with two convolution and two fully-connected layers, the paper reports the accuracy of 98.5% in 150 epochs of secure MPC training. The main contribution of the paper is to present an interesting combination of known MPC building blocks that are carefully selected from the literature. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality:* The MPC building blocks used in the paper are taken from the literature (and this is clearly stated by the authors). In other words, this work is an interesting combination of known MPC building blocks that are carefully selected from the literature. My main concern is that the comparison with the prior work is mentioned in a terse manner, and needs to be fleshed out. \n\n1. As the authors mention, the main contribution of the paper is to put together secure computation building blocks in an efficient framework. Would it be possible to comment which building blocks are significantly different than the prior work ([Wagh et al. 2019, 2021], [Mohassel and Rindal 2018], [Tan et al. 2021])? Were some MPC components more critical than others in terms of accuracy/speed? Currently, it is not clear which components enable the authors to obtain superior results.\n\n2. The authors mention that their implementation, which uses CPU, is 40% faster than [Tan et al. 2021], which uses GPU, speculating that communication costs could be the reason for this. Can the authors give more details on the communication cost comparison? The supplemental material has a table on communication per epoch, however, in some cases the proposed methods seem to use more communication than [Tan et al. 2021]. It will be helpful to include a detail comparison and a short discussion. \n\n3. The authors mention that their implementation is built on MP-SPDZ [Keller 2020]. They also mention that “MP-SPDZ already featured capabilities to train dense neural networks…”. How does the proposed implementation differ? Which secure MPC building blocks are changed by the paper? It will be helpful to add more details.\n\n*Quality:* The submission looks to be technically sound, but there are several avenues for improvement. \n\n1. My main concern here is that the machine learning setup and the security model are not precisely defined. How is the dataset partitioned across parties? Is the trained model need to be kept secure as well (i.e., is the goal to compute secret shares of the trained model)? \n\n2. Fig. 1 does not seem to convey sufficient technical details other than the high level idea. It is important to give more details.\n\n3. The abstract claims the accuracy of 98.5% in 150 epochs for a specific neural network architecture. It will be helpful to include this case in Table 2 with more details to substantiate the claims.\n\n*Clarity:* The paper can include more details on the building blocks and their implementation. \n\n1. Instead of giving potential candidates (e.g., [Lu et al. 2020] is more direct than [Aly and Smart 2019] for inverse square root), it will be helpful to give more details/insights on the chosen component. \n\n2. The current description of the security model is a bit confusing in my view. It is stated that “once party is expected to follow the protocol but might try extract information from their view”. Each of the three parties individually can do this, right? It will be helpful to add a formal description of the security requirements.\n\n*Significance:* It is well-known that the problem of training neural networks using secure MPC is technically challenging. The paper seems to make progress on this problem. However, more details on the comparison with the prior work is necessary in order to understand the significance of the contributions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It will be helpful to discuss the limitations of the proposed MPC framework. Can it be scaled to larger number of parties and/or larger sized datasets? Which MPC components would be key bottlenecks to scalability in terms of computation or communication?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "SAX-_ku-Ksu", "reviewer": "Reviewer_Sgcg", "summary": "Summary: This paper proposes an implementation and evaluation of an MPC protocol for training an MNIST model. The trained model achieves accuracy above 98% in 1 day.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to push the boundaries of practical neural network training with MPC.\nThe paper does not present any new techniques, but rather implements a collection of standard approaches from prior work and then evaluates a small number of model architectures and parameter configurations on MNIST.\n\nFrom the introduction, it is hard to extract what this paper claims as its main contribution.\nAfter defining MPC, the introduction starts discussing some intricate details about prior work without giving much context. It then follows with some low-level implementation details (e.g., the type of CPU or GPU used in prior work).\nAfter that, we seem to get to the real contribution: a framework for implemnting MPC models.\nBut here again, it isn't very clear what this framework presumably does (or allows to do) that prior work didn't.\n\nA very large part of the rest of the paper (p.3-6) presents known results from the literature, at times in overly detailed manner. In particular, it wouldn't be unnatural to assume that any NeurIPS reader is intimately familiar with the entirety of Section 4.\n\nThe framework that is claimed as the main contribution is never described in detail. What exactly does this paper propose that couldn't be done with prior tools? Or is the main contribution the engineering effort of porting existing techniques into a common library? If so, it would be worth arguing what the research contribution is.\n\nThe experiments consider 4 different neural networks from prior work that are evaluated on MNIST.\nThere are some experiments comparing different types of quantization, or different optimizers, but again it isn't very clear what to learn from this.\nGiven that the paper proposes a unified framework for various MPC techniques, I was expecting to see a much richer experimental section comparing different approaches. The paper claims that performing a wider range of comparisons might be very expensive. But I don't understand why these experiments have to be performed in \"full\" MPC. The impact of various MPC blocks on model accuracy could probably be easily assessed without performing an actual distributed training?\n\nOverall, while the main experimental result in the paper (an MPC with >98% accuracy on MNIST) is quite neat, it appears that achieving this result mainly required the engineering effort of putting together the techniques from past works and actually running the experiments for long enough.\nIt is certainly cool to showcase this, but further details on the benefits of the proposed framework and additional experiments with it seem necessary before publishing this paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "MeEBu3Tt4zT", "reviewer": "Reviewer_MTzN", "summary": "The main contribution of this paper is an implementation of MNIST training using a three-party secure multi-party computation (MPC) protocol based on the MP-SPDZ library. The core claim is that full end-to-end training for 150 epochs can be completed in about 20 hours.", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors themselves admit that all the secure computation building blocks described in Section 3 are already known individually. In fact, they also admit that all these building blocks are available in the high-level library that comes with the MP-SPDZ framework. Even on the machine learning side, the paper merely exploits existing network architectures reported in earlier papers without any modification or optimization. So the only contribution is to put the various pieces together to implement MNIST training and benchmarking the time taken with some previous publications. While this is indeed an useful effort that could be beneficial to the privacy-preserving machine learning community, the lack of any pretense of originality makes the paper unsuitable for publication. Unfortunately, the paper does not even attempt to analyze why the proposed implementation works faster, what are the bottlenecks, and what are the avenues for improvement.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None of the limitations of the proposed implementation are discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GlzL0Gsdb-", "reviewer": "Reviewer_VTuC", "summary": "The authors train neural networks completely under an MPC setting and get accuracy results that are close to training in the clear. They utilize many techniques and tricks in a clever way, such as replicated secret sharing, domain conversion, quantization, oblivious selection and so on. They run experiments on MNIST and Fashion-MNIST datasets to find that training under an MPC setting slows down the training considerably by increasing the number of iterations required for convergence of the model with an accuracy of 98.5%. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting combination of many well-known techniques and achieve secure neural network training with high accuracy. Authors compare their work with recent benchmarks and clearly explain how their work and results differ from theirs. The main limitation is that the protocol is implemented for three party computation and does not generalize to more parties. This might be a limitation as the main motivation for outsourced computations is to distribute the computation load. Moreover, no collusions between the parties are allowed. There is also the high runtime for high accuracy, but this is expected for an MPC-based setting. One related comment would be to have a dedicated section for the protocol and have a pseudocode of the algorithm in that section. The way the algorithm is presented is scattered and hard to follow. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations about high runtime are mentioned in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work demonstrates an implementation of MPC to train a convolutional neural network for MNIST that achieves 98.5 (now updated to 99.2) percent accuracy.

The reviewers were unconvinced that the contribution of the work is well-explained in the paper. The primary contribution is to bring together known techniques to demonstrate an efficient and high-accuracy implementation of neural network training with MPC. While I find this to be an impressive contribution, I would expect to see a more thorough empirical section comparing different approaches (potentially in simulation, as one of the reviewers said). In the current form, I do not find this paper to be a good fit for this conference.
Decision: Reject