Secure Quantized Training for Deep Learning

We have implemented training of neural networks in secure multi-party computa-1 tion ( MPC ) using quantization commonly used in said setting . To the best of our2 knowledge , we are the first to present training of MNIST purely implemented in3 MPC that comes within one percent of accuracy of training using plaintext compu-4 tation . We found that training with MPC is possible , but it takes more epochs and5 achieves a lower accuracy than the usual CPU/GPU computation . More concretely,6 we have trained a network with two convolution and two dense layers to 98.5 % 7 accuracy in 150 epochs . This took a day in our MPC implementation.8 1 Introduction9 Secure multi-party computation ( MPC ) is a cryptographic technique that allows a set of parties to10 compute a public output on private inputs without revealing the inputs or any intermediate results.11 This makes it a potential solution to federated learning where the sample data stays private and only12 the model or even only inference results are revealed.13 Imagine a set of healthcare providers holding sensitive patient data . MPC allows them to collabo-14 ratively train a model . This model could then either be released or even kept private for inference15 using MPC again . See Figure for an illustration . A more conceptual example is the well-known16 millionaires ’ problem where two people want to find out who is richer without revealing their wealth.17 There is clearly a difference between the one bit of information desired and the full figures.18 There has been a sustained interest in applying secure computation to machine learning and neural19 networks going back to at least Barni et al . [ 2006 ] . More recent advantages in practical MPC have20 led to an increased effort in implementing both inference and training.21 A number of works such as Mohassel and Zhang [ 2017 ] , Mohassel and Rindal [ 2018 ] , Wagh et al.22 [ 2019 ] , Wagh et al . [ 2021 ] implement neural network training with MPC at least in parts . However,23 they either give accuracy figures below 95 % or figures that have been obtained using plaintext training.24 For the latter case , the works do not clarify how close the computation for plaintext training matches25 the lower precision and other differences in the MPC setting . Agrawal et al . [ 2019 ] claim a higher26 accuracy in a comparable setting for a convolutional neural network with more channels than we27 use . However , they have only implemented dense layers , and we achieve comparable accuracy to28 them with only dense layers . All works use quantization in the sense that a fractional number x is29 represented as bx · 2−fe . This makes addition considerably faster in the secure computation setting30 because it reduces to integer addition . Furthermore , some of the works suggest to replace the softmax31 function that uses exponentiation with a ReLU-based replacement . Keller and Sun [ 2020 ] have found32 that this softmax replacement deteriorates accuracy in dense neural networks to the extent that it does33 not justify the performance gains.34 The concurrent work of Tan et al . [ 2021 ] gives some figures on the learning curve when run using35 secure computation . However , they stop at five epochs for MNIST training where they achieve 94 % 36 accuracy whereas we present the figures up to 150 epochs and 98.5 % accuracy . Furthermore , their37 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . choice of fixed-point precision 20 is considerably below 32 , which we found to be optimal . We also38 found that our implementation is 40 % faster than theirs . Note that we use the CPU of one AWS39 c5.9xlarge instance per party whereas Tan et al . use one NVIDIA Tesla V100 GPU per party . We40 believe this somewhat counter-intuitive result comes from MPC heavily relying on communication,41 which is an aspect where GPUs do not have an advantage over CPUs.42 In this paper , we present an extensible framework for implementing deep learning training based on43 MP-SPDZ by Keller [ 2020 ] , a framework for multi-party computation.1 Similar to TensorFlow and44 PyTorch , our approach allows representing deep learning models as succession of layers . We then45 use this implementation to obtain accuracy figures for MNIST training by utilizing the MP-SPDZ46 emulator , which allows to run the plaintext equivalent of secure computation , that is , the same47 algorithms with the same precision . Finally , we run one of the most promising instantiation in real48 secure computation in order to benchmark it confirm the result from the plaintext emulator.49 There are a number of projects that integrate secure computation directly into popular machine50 learning frameworks such as CrypTen by Gunning et al . [ 2019 ] , PySyft by Ryffel et al . [ 2018 ] , and51 TF Encrypted by Dahl et al . [ 2018 ] . Our approach differs from all of them by running the protocol as52 native CPU code ( implemented using C++ ) . This allows for much faster execution . For example,53 CrypTen provides an MNIST training example ( mpc_autograd_cnn ) that takes over one minute54 to run one epoch with 100 samples on one machine . In comparison , our implementation takes 1155 minutes to run one epoch with the full dataset of 60,000 samples.56 Another line of work ( e.g. , Quoc et al . [ 2021 ] ) uses trusted execution environments that provide57 computation outside the reach of the operating system . This is a different security model to multi-party58 computation that works with distributing the information among several entities.59 The paper is structured as follows : After introducing the basics of the protocol we use in Section 2,60 we will explain the mathematical building blocks in Section 3 and their use in the context of deep61 learning in Section 4 . Finally , we will present our implementation in Section 5 and our experimental62 results for MNIST in Section 6.63 2 An Efficient Secure Multi-Party Computation Protocol64 There is a wide range of MPC protocols with a variety of security properties ( see Keller [ 2020 ] for an65 overview ) . In this paper we focus on the setting of three-party computation with one semi-honest66 corruption . This means that out of the three parties two are expected to behave honestly , i.e. , they67 follow the protocol and keep their view of the protocol secret , and one party is expected to follow the68 protocol but might try extract information from their view . The reason for choosing this setting is that69 it allows an efficient MPC protocol while still allowing secure outsourced computation . The concrete70 protocol we use goes back to Benaloh and Leichter [ 1990 ] with further aspects by Araki et al . [ 2016 ] ,71 Mohassel and Rindal [ 2018 ] , and Eerikson et al . [ 2020 ] . We summarize the core protocol below . The72 mathematical building blocks in the next section mostly use the aspects below.73 1We are committed to publishing our code as open source . Secret sharing All intermediate values in our protocol are stored using replicated secret sharing . A74 secret value x is a represented as a random sum x = x0 + x1 + x2 , and party Pi holds ( xi−1 , xi+1 ) 75 where the indices are computed modulo three . Clearly , each party is missing one value to compute76 the sum . On the other hand , each pair of parties hold all necessary to reconstruct the secret . For a77 uniformly random generation of shares , the computation domain has to be finite . Most commonly,78 this domain is defined by integer computation modulo a number . We use 2k for k being a multiple 6479 and 2 as the moduli . The first case corresponds to an extension of 64-bit arithmetic found on most80 processors . We will refer to the two settings as arithmetic and binary secret sharing throughout the81 paper.82 Input sharing The secret sharing scheme implies a protocol to share inputs where the inputting83 party samples the shares and distributes them accordingly . Eerikson et al . [ 2020 ] have proposed a84 more efficient protocol where the inputting party only needs to send one value instead of two pairs of85 values . If Pi would like to input x , xi is set to zero , and xi−1 is generated with a pseudo-random86 generator using a key previously shared between Pi and Pi+1 . Pi can compute xi+1 = x− xi−1 and87 send it to Pi−1 . While the resulting secret sharing is not entirely random , the fact that Pi already88 knows x makes randomizing xi obsolete.89 Addition The commutative nature of addition allows to add secret sharings without communication.90 More concretely , secret sharings x = x0 + x1 + x2 and y = y0 + y1 + y2 imply the secret sharing91 x+ y = ( x0 + y0 ) + ( x1 + y1 ) + ( x2 + y2 ) .92 Multiplication The product of x = x0 + x1 + x2 and y = y0 + y1 + y2 is93 x · y = ( x0 + x1 + x2 ) · ( y0 + y1 + y2 ) = ( x0y0 + x0y1 + x1y0 ) + ( x1y1 + x1y2 + x1y1 ) + ( x2y2 + x2y0 + x0y2 ) . Each of the brackets only contains shares known by one of the parties . They can thus compute an94 additive secret sharing ( one summand per party ) of the product . However , every party only holding95 one share does not satisfy the replication requirement for further multiplications . It is not secure96 for every party to pass their value on to another party because the summands are not distributed97 randomly . This can be fixed by rerandomization : Let xy = z0 + z1 + z2 where zi is know to Pi.98 Every party Pi computes z′i = zi + ri , i+1 − ri−1 , i where ri , i+1 is generated with a pseudo-random99 generator using a key pre-shared between Pi and Pi+1 . The resulting sum xy = z′0 + z ′ 1 + z ′ 2 is100 pseudo-random , and it is thus secure for Pi to send z′i to Pi+1 in order to create a replicated secret101 sharing ( ( xy ) i−1 , ( xy ) i+1 ) = ( z′i , z ′ i−1 ) .102 3 Secure Computation Building Blocks103 In this section , we will discuss how to implement computation with MPC with a focus on how104 it differs from computation on CPUs or GPUs . Most of the techniques below are already known105 individually . To the best of our knowledge however , we are the first to put them together in an efficient106 and extensible framework for secure computation of deep learning training.107 Domain conversion Recall we that we use computation modulo 2k for k being a multiple of 64108 as well as 1 . Given that the main operations are just addition and multiplication in the respective109 domain , it is desirable to compute integer arithmetic in the large domain but operations with a110 straight-forward binary circuit modulo two . There has been a long-running interest in this going111 back to least Kolesnikov et al . [ 2013 ] . We mainly rely on the approach proposed by Mohassel and112 Rindal [ 2018 ] and Araki et al . [ 2018 ] . Recall that x ∈ 2k is shared as x = x0 + x1 + x2 . Now let113 { x ( i ) 0 } k−1 j=0 the bit decomposition of x0 , that is , x ( i ) 0 ∈ { 0 , 1 } and x0 = ∑k−1 i=0 x ( i ) 0 2 i . It is self-evident114 that x ( i ) 0 = x ( i ) 0 + 0 + 0 is a valid secret sharing modulo two ( albeit not a random one ) . Furthermore,115 every party holding x0 can generate x ( i ) 0 . It is therefore possible for the parties to generate a secret116 sharing modulo two of a single share modulo 2k . Repeating this for all shares and the computing117 the addition as a binary circuit allows the parties to generate a secret sharing modulo two from a118 secret sharing modulo 2k . Conversion in the other direction can be achieved using a similar technique119 or using daBits as described by Rotaru and Wood [ 2019 ] . In the following we will use the term120 mixed-circuit computation for any technique that works over both computation domains.121 Quantization While Aliasgari et al . [ 2013 ] showed that it is possible to implement floating-point122 computation , the cost is far higher than integer computation . It is therefore common to represent123 fractional numbers using quantization ( also called fixed-point representation ) as suggested by Catrina124 and Saxena [ 2010 ] . A real number x is represented as x̃ = bx · 2fe where f is an integer specifying125 the precision . The linearity of the representation allows to compute addition by simply adding the126 representing integers . Multiplication however requires to adjust the result because it will have twice127 the precision : ( x · 2f ) · ( y · 2f ) = xy · 22f . There are two ways to rectify this:128 • An obvious correction would be to shift the result by f bits after adding 2f−1 to the integer129 representation . This ensures rounding to the nearest number possible in the representation,130 with the tie being broken by rounding up . Dalskov et al . [ 2021 ] presented an efficient131 implementation of the truncation using mixed-circuit computation.132 • However , Catrina and Saxena have found that in the context of secure computation it is more133 efficient to use probabilistic truncation . This method rounds up or down probabilistically134 depending on the input . For example , probabilistically rounding 0.75 to an integer would135 see it rounded of up with probability 0.75 and down with probability 0.25 . The probabilistic136 truncation is an effect of the fact that the operation involves the truncation of a randomized137 value , that is the computation of b ( x+ r ) /2mc for a random m-bit value r. It is easy to see138 that139 b ( x+ r ) /2mc = { bx/2mc ( x mod 2m + r ) < 2m bx/2mc+ 1 ( x mod 2m + r ) ≥ 2m . Therefore , the larger ( x mod 2m ) is , the more likely the latter condition is true . Dalskov140 et al . [ 2020 ] present an efficient protocol in our security model.141 Our quantization scheme is related to quantized neural networks ( see e.g . Hubara et al . [ 2016 ] ) .142 However , our consideration is not to compress the model , but to improve the computational speed143 and save communication cost.144 Dot products Dot products are an essential building block of linear computation such as matrix145 multiplication . In the light of quantization , it is possible to reduce the usage of truncation by deferring146 after the summation . In other words , the dot product in the integer representations is computed before147 truncating . This not only reduces the truncation error , it is also more efficient because the truncation148 is the most expensive part in quantized secure multiplication . Similarly , our protocol allows to defer149 the communication needed for multiplication . Let ~x and ~y be two vectors where the elements are150 secret shared , that is , { x ( i ) } = x ( i ) 0 + x ( i ) 1 + x ( i ) 2 and similarly for y ( i ) . The inner product then is151 ∑ i x ( i ) · y ( i ) = ∑ i ( x ( i ) 0 + x ( i ) 1 + x ( i ) 2 ) · ( y ( i ) 0 + y ( i ) 1 + y ( i ) 2 ) = ∑ i ( x ( i ) 0 y ( i ) 0 + x ( i ) 0 y ( i ) 1 + x ( i ) 1 y ( i ) 0 ) + ∑ i ( x ( i ) 1 y ( i ) 1 + x ( i ) 1 y ( i ) 2 + x ( i ) 1 y ( i ) 1 ) + ∑ i ( x ( i ) 2 y ( i ) 2 + x ( i ) 2 y ( i ) 0 + x ( i ) 0 y ( i ) 2 ) . The three sums in the last term can be compute locally by one party each before applying the same152 protocol as for a single multiplication.153 Comparisons Arithmetic secret sharing does not allow to access the individual bits directly . It154 is therefore not straightforward to compute comparisons such as “ less than ” . There is a long line155 of literature on how to achieve this going back to at least Damgård et al . [ 2006 ] . More recently,156 most attention has been given to combine the power of arithmetic and binary secret sharing in157 order to combine the best of worlds . One possibility to do so is to plainly convert to the binary158 domain and compute the comparison circuit there . In our concrete implementation we use the more159 efficient approach by Mohassel and Rindal [ 2018 ] . It starts by taking the difference of the two160 inputs . Computing the comparison then reduces to comparing to zero , which in turn is equivalent161 to extracting the most significant bit as it indicates the sign . The latter is achieved by converting162 the shares locally to bit-wise sharing of the arithmetic shares , which sum up to the secret value . It163 remains to compute the sum of the binary shares in order to come up with the most significant bit.164 Oblivious Selection Plain secure computation does not allow branching because the parties would165 need to be aware which branch is followed . Conditional assignment can be implemented as follows166 however . If b ∈ { 0 , 1 } denotes the condition , x+ b · ( y − x ) is either x or y depending on b . If the167 condition is available in binary secret sharing but x and y in arithmetic secret sharing , b has to be168 converted to the latter . This can be done using a daBit as introduced by Rotaru and Wood [ 2019 ] ,169 which is a secret random bit shared both in arithmetic and binary . It allows to mask a bit in one world170 by XORing it . The result is then revealed and the masking is undone in the other world.171 Division Catrina and Saxena [ 2010 ] have shown how to implement quantized division using the172 algorithm by Goldschmidt [ 1964 ] . It mainly uses arithmetic and the probabilistic truncation already173 explained . In addition , the initial approximation requires a full bit decomposition as described above.174 The error of the output depends on the error in the multiplications used for Goldschmidt ’ s iteration,175 which compounds in particular when using probabilistic truncation . Due to the nature of secure176 computation , the result of division by zero is undefined . One could obtain a secret failure bit by177 testing the divisor to zero . However , we found that not to be necessary in our algorithm . This is178 because we only use division by secret value only for the softmax function where the it is guaranteed179 to strictly positive.180 Logarithm Computation logarithm with any public base can be reduced to logarithm to base two181 using logx y = log2 y · logx 2 . Aly and Smart [ 2019 ] have proposed computing y = a · 2b where182 a ∈ [ 0.5 , 1 ) and b ∈ Z . This then allows to compute log2 y = log2 a+ b . Given the restricted range183 of a , log2 a can be approximated using a division of polynomials . Numerical stability and input range184 control are less of an issue here because we only use logarithm for the loss computation , which does185 not influence the training.186 Exponentiation By using xy = 2y log2 x , any exponentiation can be reduced to exponentiation187 with base two . Aly and Smart [ 2019 ] have shown how to compute 2a = 2bac · 2a−bac by computing188 the two exponents using bit decomposition and the second factor using a polynomial approximation.189 Regarding the first factor , if b = ∑ i≥0 bi2 i is an integer with bi ∈ { 0 , 1 } ,190 2b = 2 ∑ i≥0 bi2 i = ∏ i≥0 2bi2 i = ∏ i≥0 ( 1 + bi · ( 22 i − 1 ) ) . As with division the numerical stability depends on the truncation used for multiplication.191 Inverse square root Aly and Smart [ 2019 ] have proposed to compute square root using Gold-192 schmidt and Raphson-Newton iterations . We could combine this with the division above . However , Lu193 et al . [ 2020 ] have proposed a more direct computation that avoids running two successive iterations.194 Uniformly random fractional number Limiting ourselves to intervals of the form [ x , x+ 2e ] for195 a potentially negative integer e , we can reduce the problem to generate a random ( f + e ) -bit number196 where f is the fixed-point precision . Recall that we represent a fractional number x as bx · 2−fe.197 Generating a random n-bit number is straight-forward using random bits , which in our protocol can198 be generated as presented by Damgård et al . [ 2019 ] . In the context of our protocol however , Dalskov199 et al . [ 2021 ] have a presented a more efficient approach that involves mixed-circuit computation.200 Communication cost Table 1 show the total communication cost of some of the building blocks201 in our protocol for f = 32 . This setting mandates the modulus 2128 because the division protocol202 requires bit length 4f .203 4 Machine Learning Building Blocks204 In this section , we will use the building blocks in Section 3 to construct high-level computational205 modules for deep learning.206 Fully connected layers Both forward and back-propagation of fully connected layers can be seen207 as matrix multiplications and thus can be implemented using dot products . A particular challenge in208 secure computation is to compute a number of outputs in parallel in order to save communication209 rounds . We solve this by having a dedicated infrastructure in our implementation that computes all dot210 products for a matrix multiplication in a single batch , thus combining all necessary communication.211 2D convolution layers Similar to fully connected layers , 2D convolution and its corresponding212 gradient can be implemented using only dot products , and we again compute several output values in213 parallel.214 Rectified Linear Unit ( ReLU ) ReLU Nair and Hinton [ 2010 ] is defined as follows:215 ReLU ( x ) : = { x , if x > 0 0. otherwise It can thus be implemented as a comparison followed by an oblivious selection . For back-propagation,216 it is advantageous to reuse the comparison results from forward propagation due to the relatively high217 cost in secure computation . Note that the comparison results are stored in secret-shared form and218 thus there is no reduction in security.219 Max pooling Similar to ReLU , max pooling can be reduced to comparison and oblivious selection.220 In secure computation , it saves communication rounds if the process uses a balanced tree rather than221 iterating over all input values of one maximum computation . For back-propagation it again pays off222 to the store intermediate results from forward propagation , again in secret-shared form.223 Softmax and cross entropy loss This combination requires computing the following gradient for224 back-propagation:225 5i : = ∂ ` ∂xi = ∂ ∂xi ( − ∑ k yk · xk + log ∑ j exj ) = −yi + exi∑ j e xj , ( 1 ) where yi denotes an element of the ground truth as a one-hot vector , and xi denotes the output of the226 last layer.227 On the right hand side of eq . ( 1 ) , the values in the denominator are potentially large due to the use228 of the exponential . This is prone to numerical overflow in our quantized representation because the229 latter puts relatively strict limits on the values . We therefore optimize the computation by computing230 the maximum of the input values:231 m = max j ( { xj } ) . Then we compute232 exi−m∑ j e xj−m = exie−m ( ∑ j e xj ) e−m = exi∑ j e xj . All the exponents on the left-most term are at most zero , and thus the dividend is at most one and the233 divisor is at most the number of labels ( which is 10 in MNIST ) . The same technique can be used to234 compute the sigmoid activation function , as sigmoid ( x ) = 11+exp ( −x ) = exp ( 0 ) exp ( 0 ) +exp ( x ) is a special235 case of softmax.236 Stochastic gradient descent The model parameter update in SGD only involves basic arithmetic:237 θj ← θj − γB ∑B i=15ij where θj is the parameter indexed by j , B is the mini-batch size , γ > 0 is238 the learning rate , and5ij is the gradient of the loss with respect to the i ’ th sample in the mini-batch239 and the parameter θj . In order to tackle the limited precision with quantization , we defer dividing by240 the batch size to the model update . This means that we do not divide the gradient value by the batch241 size when computing them as described in the previous paragraph . Instead , we divide the model242 update by the batch size . Since we use a batch size that is a power of two ( 128 ) , it is sufficient to use243 probabilistic truncation instead of full-blown division . This saves both time and decreases the error.244 Adam The main difference to SGD in terms of basic computational operations is the additional245 use of an inverse square root . We again defer the division by the batch size to just before the model246 update.247 Parameter initialization We use the Glorot initialization by Glorot and Bengio [ 2010 ] . Besides248 basic operations , it mainly involves generating a uniformly random fractional value in a given interval.249 5 Implementation250 We built our implementation on MP-SPDZ by Keller [ 2020 ] . MP-SPDZ not only implements a251 range of MPC protocols , it also comes with a high-level library containing the building blocks in252 Section 3 . MP-SPDZ already featured capabilities to train dense neural networks as well as inference253 for convolutional neural networks . We have added backward propagation for a number of layer types,254 including 2D convolution . Furthermore , we have corrected a bug in the backward propagation for255 dense layers.256 MP-SPDZ allows implementing the computation in Python code , which is then compiled a specific257 bytecode . This code can be execute by a virtual machine executing the actual secure computation.258 The process allows to optimize the computation in the context of MPC.259 The framework also features an emulator that executes the exact computation that could be done260 securely in the clear . This allowed us to collect the accuracy figures in the next section at a lower cost.261 It is licensed under a BSD-style license , which allows to extend the code.262 6 MNIST263 For a concrete measurement of accuracy and running times , we have implemented training for the264 well-known MNIST dataset by LeCun et al . [ 2010 ] . We work mainly with the models that have been265 used by Wagh et al . [ 2019 ] with secure computation , and we will reuse their numbering ( A–D ) . The266 models contain up to four linear layers . Network C is a convolutional neural network going back267 to the seminal work by LeCun et al . [ 1998 ] whereas the others are simpler networks that have been268 proposed by works on secure computation such as Mohassel and Zhang [ 2017 ] , Liu et al . [ 2017 ] , and269 Riazi et al . [ 2018 ] . We present the networks as Keras code in the supplemental material.270 Figure 2 shows the results for various quantization precisions and and the two rounding options.271 We have used SGD with learning rate 0.01 , batch size 128 , and the usual MNIST training/test split.272 f = 64 is the best option with probabilistic rounding , improving on both f = 16 and f = 32.273 Furthermore , nearest rounding performs worse that probabilistic for f = 16 and f = 32 . Due to the274 high cost , we only ran f = 32 with probabilistic rounding several times . The range is indicate by the275 shaded area . We focus on f = 32 because it offers the faster convergence.276 Figure 3 then shows the result with a variety of optimizers . While increasing the learning rate for277 SGD leads to a lower stability , Adam exposes a smoother learning learning curve albeit not a faster278 process.279 Finally , Figures 4 shows our results for all networks used by Wagh et al . [ 2019 ] . As one would expect,280 the most sophisticated network performs best . Somewhat surprisingly , however , Network A ( without281 convolutional layers ) performs better than the simpler networks containing convolutional layers.282 Resources We ran the emulator on AWS c5.9xlarge instances . One epoch takes a few second to283 several minutes depending on the model . Overall , we estimate that we have used a few weeks worth284 of computing time including experiments not included here because of bugs in the code.285 6.1 Secure computation286 In order to verify our emulation results , we have run Network C with precision f = 32 and287 probabilistic rounding in our actual multi-party computation protocol . We could verify that it288 converges on 98.5 % accuracy at 150 epochs , taking 20 hours . Table 2 compares our result to previous289 works in a LAN setting . Note that Wagh et al . [ 2019 ] and Wagh et al . [ 2021 ] give accuracy figures.290 From personal communication with the authors and the fact that the source repository for the latter291 work2 says that their “ code has not run end-to-end training ” , we derive our assessment that their292 figures do not reflect the secure computation.293 2https : //github.com/snwagh/falcon-public 7 Conclusions294 We have presented an implementation of deep learning training purely in multi-party computation295 with extensive results on the accuracy . We have found that the lower precision of MPC computation296 increases the error considerably . We only have considered one particular implementation of more297 complex computation such as division and exponentiation , which are crucial to the learning process298 as part of softmax . Future work might consider different approximations of these building blocks.299 References300 N. Agrawal , A. S. Shamsabadi , M. J. Kusner , and A. Gascón . QUOTIENT : Two-party secure neural301 network training and prediction . In L. Cavallaro , J. Kinder , X. Wang , and J. Katz , editors , ACM302 CCS 2019 , pages 1231–1247 . ACM Press , Nov. 2019. doi : 10.1145/3319535.3339819.303 M. Aliasgari , M. Blanton , Y. Zhang , and A. Steele . Secure computation on floating point numbers.304 In NDSS 2013 . The Internet Society , Feb. 2013.305 A. Aly and N. P. Smart . Benchmarking privacy preserving scientific operations . In R. H. Deng,306 V. Gauthier-Umaña , M. Ochoa , and M. Yung , editors , ACNS 19 , volume 11464 of LNCS , pages307 509–529 . Springer , Heidelberg , June 2019. doi : 10.1007/978-3-030-21568-2_25.308 T. Araki , J. Furukawa , Y. Lindell , A. Nof , and K. Ohara . High-throughput semi-honest secure309 three-party computation with an honest majority . In E. R. Weippl , S. Katzenbeisser , C. Kruegel,310 A. C. Myers , and S. Halevi , editors , ACM CCS 2016 , pages 805–817 . ACM Press , Oct. 2016. doi:311 10.1145/2976749.2978331.312 T. Araki , A. Barak , J. Furukawa , M. Keller , Y. Lindell , K. Ohara , and H. Tsuchida . Generalizing313 the SPDZ compiler for other protocols . In D. Lie , M. Mannan , M. Backes , and X. Wang , editors,314 ACM CCS 2018 , pages 880–895 . ACM Press , Oct. 2018. doi : 10.1145/3243734.3243854.315 M. Barni , C. Orlandi , and A. Piva . A privacy-preserving protocol for neural-network-based computa-316 tion . In Proceedings of the 8th workshop on Multimedia and security , pages 146–151 , 2006.317 J. C. Benaloh and J. Leichter . Generalized secret sharing and monotone functions . In S. Goldwasser,318 editor , CRYPTO ’ 88 , volume 403 of LNCS , pages 27–35 . Springer , Heidelberg , Aug. 1990. doi:319 10.1007/0-387-34799-2_3.320 O. Catrina and A. Saxena . Secure computation with fixed-point numbers . In R. Sion , editor , FC321 2010 , volume 6052 of LNCS , pages 35–50 . Springer , Heidelberg , Jan. 2010.322 M. Dahl , J. Mancuso , Y. Dupis , B. Decoste , M. Giraud , I. Livingstone , J. Patriquin , and G. Uhma.323 Private machine learning in tensorflow using secure computation . CoRR , abs/1810.08130 , 2018.324 URL http : //arxiv.org/abs/1810.08130.325 A. Dalskov , D. Escudero , and M. Keller . Fantastic four : Honest-majority four-party secure com-326 putation with malicious security . In 30th USENIX Security Symposium ( USENIX Security 21 ) ,327 2021.328 A. P. K. Dalskov , D. Escudero , and M. Keller . Secure evaluation of quantized neural networks.329 PoPETs , 2020 ( 4 ) :355–375 , Oct. 2020. doi : 10.2478/popets-2020-0077.330 I. Damgård , M. Fitzi , E. Kiltz , J . B. Nielsen , and T. Toft . Unconditionally secure constant-rounds331 multi-party computation for equality , comparison , bits and exponentiation . In S. Halevi and332 T. Rabin , editors , TCC 2006 , volume 3876 of LNCS , pages 285–304 . Springer , Heidelberg , Mar.333 2006. doi : 10.1007/11681878_15.334 I. Damgård , D. Escudero , T. K. Frederiksen , M. Keller , P. Scholl , and N. Volgushev . New primitives335 for actively-secure MPC over rings with applications to private machine learning . In 2019 IEEE336 Symposium on Security and Privacy , pages 1102–1120 . IEEE Computer Society Press , May 2019.337 doi : 10.1109/SP.2019.00078.338 H. Eerikson , M. Keller , C. Orlandi , P. Pullonen , J. Puura , and M. Simkin . Use your brain ! Arithmetic339 3PC for any modulus with active security . In Y. T. Kalai , A. D. Smith , and D. Wichs , editors , ITC340 2020 , pages 5:1–5:24 . Schloss Dagstuhl , June 2020. doi : 10.4230/LIPIcs.ITC.2020.5.341 X. Glorot and Y. Bengio . Understanding the difficulty of training deep feedforward neural networks.342 In Proceedings of the thirteenth international conference on artificial intelligence and statistics,343 pages 249–256 . JMLR Workshop and Conference Proceedings , 2010.344 R. E. Goldschmidt . Applications of division by convergence . Master ’ s thesis , MIT , 1964.345 D. Gunning , A. Hannun , M. Ibrahim , B. Knott , L. van der Maaten , V. Reis , S. Sengupta , S. Venkatara-346 man , and X. Zhou . Crypten : A new research tool for secure machine learning with pytorch,347 2019.348 I. Hubara , M. Courbariaux , D. Soudry , R. El-Yaniv , and Y. Bengio . Binarized neural networks . In349 Advances in Neural Information Processing Systems , volume 29 . Curran Associates , Inc. , 2016.350 M. Keller . MP-SPDZ : A versatile framework for multi-party computation . In J. Ligatti , X. Ou,351 J. Katz , and G. Vigna , editors , ACM CCS 20 , pages 1575–1590 . ACM Press , Nov. 2020. doi:352 10.1145/3372297.3417872.353 M. Keller and K. Sun . Effectiveness of MPC-friendly softmax replacement , 2020.354 V. Kolesnikov , A.-R. Sadeghi , and T. Schneider . A systematic approach to practically efficient general355 two-party secure function evaluation protocols and their modular design . Journal of Computer356 Security , 21 ( 2 ) :283–315 , 2013.357 Y. LeCun , L. Bottou , Y. Bengio , and P. Haffner . Gradient-based learning applied to document358 recognition . Proceedings of the IEEE , 86 ( 11 ) :2278–2324 , 1998.359 Y. LeCun , C. Cortes , and C. Burges . Mnist handwritten digit database . ATT Labs [ Online ] . Avail-360 able : http : // yann . lecun . com/ exdb/ mnist , 2 , 2010 . Creative Commons Attribution-Share361 Alike 3.0 license , https : //creativecommons.org/licenses/by-sa/3.0/.362 J. Liu , M. Juuti , Y. Lu , and N. Asokan . Oblivious neural network predictions via MiniONN363 transformations . In B. M. Thuraisingham , D. Evans , T. Malkin , and D. Xu , editors , ACM CCS364 2017 , pages 619–631 . ACM Press , Oct. / Nov. 2017. doi : 10.1145/3133956.3134056.365 W.-j . Lu , Y. Fang , Z. Huang , C. Hong , C. Chen , H. Qu , Y. Zhou , and K. Ren . Faster secure multiparty366 computation of adaptive gradient descent . In Proceedings of the 2020 Workshop on Privacy-367 Preserving Machine Learning in Practice , PPMLP ’ 20 , page 47–49 , New York , NY , USA , 2020.368 Association for Computing Machinery . ISBN 9781450380881. doi : 10.1145/3411501.3419427.369 URL https : //doi.org/10.1145/3411501.3419427.370 P. Mohassel and P. Rindal . ABY3 : A mixed protocol framework for machine learning . In D. Lie,371 M. Mannan , M. Backes , and X. Wang , editors , ACM CCS 2018 , pages 35–52 . ACM Press , Oct.372 2018. doi : 10.1145/3243734.3243760.373 P. Mohassel and Y. Zhang . SecureML : A system for scalable privacy-preserving machine learning.374 In 2017 IEEE Symposium on Security and Privacy , pages 19–38 . IEEE Computer Society Press,375 May 2017. doi : 10.1109/SP.2017.12.376 V. Nair and G. E. Hinton . Rectified linear units improve Restricted Boltzmann machines . In Pro-377 ceedings of the 27th International Conference on International Conference on Machine Learning,378 ICML ’ 10 , pages 807–814 , 2010.379 D. L. Quoc , F. Gregor , S. Arnautov , R. Kunkel , P. Bhatotia , and C. Fetzer . securetf : A secure380 tensorflow framework . CoRR , abs/2101.08204 , 2021 . URL https : //arxiv.org/abs/2101.381 08204.382 M. S. Riazi , C. Weinert , O. Tkachenko , E. M. Songhori , T. Schneider , and F. Koushanfar . Chameleon:383 A hybrid secure computation framework for machine learning applications . In J. Kim , G.-J . Ahn,384 S. Kim , Y. Kim , J. López , and T. Kim , editors , ASIACCS 18 , pages 707–721 . ACM Press , Apr.385 2018.386 D. Rotaru and T. Wood . MArBled circuits : Mixing arithmetic and Boolean circuits with active387 security . In F. Hao , S. Ruj , and S. Sen Gupta , editors , INDOCRYPT 2019 , volume 11898 of LNCS,388 pages 227–249 . Springer , Heidelberg , Dec. 2019. doi : 10.1007/978-3-030-35423-7_12.389 T. Ryffel , A. Trask , M. Dahl , B. Wagner , J. Mancuso , D. Rueckert , and J. Passerat-Palmbach . A390 generic framework for privacy preserving deep learning . CoRR , abs/1811.04017 , 2018 . URL391 http : //arxiv.org/abs/1811.04017.392 S. Tan , B. Knott , Y. Tian , and D. J. Wu . CryptGPU : Fast privacy-preserving machine learning on the393 GPU , 2021.394 S. Wagh , D. Gupta , and N. Chandran . SecureNN : 3-party secure computation for neural network395 training . PoPETs , 2019 ( 3 ) :26–49 , July 2019. doi : 10.2478/popets-2019-0035.396 S. Wagh , S. Tople , F. Benhamouda , E. Kushilevitz , P. Mittal , and T. Rabin . Falcon : Honest-majority397 maliciously secure framework for private deep learning . PoPETs , 2021 ( 1 ) :188–208 , Jan. 2021.398 doi : 10.2478/popets-2021-0011.399 Checklist400 1 . For all authors ... 401 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s402 contributions and scope ? [ Yes ] 403 ( b ) Did you describe the limitations of your work ? [ Yes ] See Section 7.404 ( c ) Did you discuss any potential negative societal impacts of your work ? [ No ] We only405 work with MNIST , and we believe that the potential negative impacts associated to it406 are very low.407 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to408 them ? [ Yes ] 409 2 . If you are including theoretical results ... 410 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 411 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 412 3 . If you ran experiments ... 413 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-414 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] We include all415 code in the supplemental material , and the references include the URL for MNIST.416 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they417 were chosen ) ? [ Yes ] See Section 6.418 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-419 ments multiple times ) ? [ Yes ] Only in Figure 2.420 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type421 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Section 6.422 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 423 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See Section 6.424 ( b ) Did you mention the license of the assets ? [ Yes ] See the references and and Section 5.425 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 426 Our code is included the supplemental material.427 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re428 using/curating ? [ N/A ] 429 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable430 information or offensive content ? [ N/A ] 431 5 . If you used crowdsourcing or conducted research with human subjects ... 432 ( a ) Did you include the full text of instructions given to participants and screenshots , if433 applicable ? [ N/A ] 434 ( b ) Did you describe any potential participant risks , with links to Institutional Review435 Board ( IRB ) approvals , if applicable ? [ N/A ] 436 ( c ) Did you include the estimated hourly wage paid to participants and the total amount437 spent on participant compensation ? [ N/A ] 438

Summary: The paper presents a secure multi-party computation (MPC) implementation of training several neural network architectures on MNIST. A 3-party semi-honest setup is considered with every individual party being honest-but-curious about the inputs of the other two parties. For a neural network with two convolution and two fully-connected layers, the paper reports the accuracy of 98.5% in 150 epochs of secure MPC training. The main contribution of the paper is to present an interesting combination of known MPC building blocks that are carefully selected from the literature.