Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors train neural networks completely under an MPC setting and get accuracy results that are close to training in the clear. They utilize many techniques and tricks in a clever way, such as replicated secret sharing, domain conversion, quantization, oblivious selection and so on. They run experiments on MNIST and Fashion-MNIST datasets to find that training under an MPC setting slows down the training considerably by increasing the number of iterations required for convergence of the model with an accuracy of 98.5%. 
main_review: This paper presents an interesting combination of many well-known techniques and achieve secure neural network training with high accuracy. Authors compare their work with recent benchmarks and clearly explain how their work and results differ from theirs. The main limitation is that the protocol is implemented for three party computation and does not generalize to more parties. This might be a limitation as the main motivation for outsourced computations is to distribute the computation load. Moreover, no collusions between the parties are allowed. There is also the high runtime for high accuracy, but this is expected for an MPC-based setting. One related comment would be to have a dedicated section for the protocol and have a pseudocode of the algorithm in that section. The way the algorithm is presented is scattered and hard to follow. 

limitations_and_societal_impact: Limitations about high runtime are mentioned in the paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a secure multi-party computation (MPC) implementation of training several neural network architectures on MNIST. A 3-party semi-honest setup is considered with every individual party being honest-but-curious about the inputs of the other two parties. For a neural network with two convolution and two fully-connected layers, the paper reports the accuracy of 98.5% in 150 epochs of secure MPC training. The main contribution of the paper is to present an interesting combination of known MPC building blocks that are carefully selected from the literature. 
main_review: *Originality:* The MPC building blocks used in the paper are taken from the literature (and this is clearly stated by the authors). In other words, this work is an interesting combination of known MPC building blocks that are carefully selected from the literature. My main concern is that the comparison with the prior work is mentioned in a terse manner, and needs to be fleshed out. 

1. As the authors mention, the main contribution of the paper is to put together secure computation building blocks in an efficient framework. Would it be possible to comment which building blocks are significantly different than the prior work ([Wagh et al. 2019, 2021], [Mohassel and Rindal 2018], [Tan et al. 2021])? Were some MPC components more critical than others in terms of accuracy/speed? Currently, it is not clear which components enable the authors to obtain superior results.

2. The authors mention that their implementation, which uses CPU, is 40% faster than [Tan et al. 2021], which uses GPU, speculating that communication costs could be the reason for this. Can the authors give more details on the communication cost comparison? The supplemental material has a table on communication per epoch, however, in some cases the proposed methods seem to use more communication than [Tan et al. 2021]. It will be helpful to include a detail comparison and a short discussion. 

3. The authors mention that their implementation is built on MP-SPDZ [Keller 2020]. They also mention that “MP-SPDZ already featured capabilities to train dense neural networks…”. How does the proposed implementation differ? Which secure MPC building blocks are changed by the paper? It will be helpful to add more details.

*Quality:* The submission looks to be technically sound, but there are several avenues for improvement. 

1. My main concern here is that the machine learning setup and the security model are not precisely defined. How is the dataset partitioned across parties? Is the trained model need to be kept secure as well (i.e., is the goal to compute secret shares of the trained model)? 

2. Fig. 1 does not seem to convey sufficient technical details other than the high level idea. It is important to give more details.

3. The abstract claims the accuracy of 98.5% in 150 epochs for a specific neural network architecture. It will be helpful to include this case in Table 2 with more details to substantiate the claims.

*Clarity:* The paper can include more details on the building blocks and their implementation. 

1. Instead of giving potential candidates (e.g., [Lu et al. 2020] is more direct than [Aly and Smart 2019] for inverse square root), it will be helpful to give more details/insights on the chosen component. 

2. The current description of the security model is a bit confusing in my view. It is stated that “once party is expected to follow the protocol but might try extract information from their view”. Each of the three parties individually can do this, right? It will be helpful to add a formal description of the security requirements.

*Significance:* It is well-known that the problem of training neural networks using secure MPC is technically challenging. The paper seems to make progress on this problem. However, more details on the comparison with the prior work is necessary in order to understand the significance of the contributions.
limitations_and_societal_impact: It will be helpful to discuss the limitations of the proposed MPC framework. Can it be scaled to larger number of parties and/or larger sized datasets? Which MPC components would be key bottlenecks to scalability in terms of computation or communication?
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: The main contribution of this paper is an implementation of MNIST training using a three-party secure multi-party computation (MPC) protocol based on the MP-SPDZ library. The core claim is that full end-to-end training for 150 epochs can be completed in about 20 hours.
main_review: The authors themselves admit that all the secure computation building blocks described in Section 3 are already known individually. In fact, they also admit that all these building blocks are available in the high-level library that comes with the MP-SPDZ framework. Even on the machine learning side, the paper merely exploits existing network architectures reported in earlier papers without any modification or optimization. So the only contribution is to put the various pieces together to implement MNIST training and benchmarking the time taken with some previous publications. While this is indeed an useful effort that could be beneficial to the privacy-preserving machine learning community, the lack of any pretense of originality makes the paper unsuitable for publication. Unfortunately, the paper does not even attempt to analyze why the proposed implementation works faster, what are the bottlenecks, and what are the avenues for improvement.


limitations_and_societal_impact: None of the limitations of the proposed implementation are discussed.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: Summary: This paper proposes an implementation and evaluation of an MPC protocol for training an MNIST model. The trained model achieves accuracy above 98% in 1 day.
main_review: This paper aims to push the boundaries of practical neural network training with MPC.
The paper does not present any new techniques, but rather implements a collection of standard approaches from prior work and then evaluates a small number of model architectures and parameter configurations on MNIST.

From the introduction, it is hard to extract what this paper claims as its main contribution.
After defining MPC, the introduction starts discussing some intricate details about prior work without giving much context. It then follows with some low-level implementation details (e.g., the type of CPU or GPU used in prior work).
After that, we seem to get to the real contribution: a framework for implemnting MPC models.
But here again, it isn't very clear what this framework presumably does (or allows to do) that prior work didn't.

A very large part of the rest of the paper (p.3-6) presents known results from the literature, at times in overly detailed manner. In particular, it wouldn't be unnatural to assume that any NeurIPS reader is intimately familiar with the entirety of Section 4.

The framework that is claimed as the main contribution is never described in detail. What exactly does this paper propose that couldn't be done with prior tools? Or is the main contribution the engineering effort of porting existing techniques into a common library? If so, it would be worth arguing what the research contribution is.

The experiments consider 4 different neural networks from prior work that are evaluated on MNIST.
There are some experiments comparing different types of quantization, or different optimizers, but again it isn't very clear what to learn from this.
Given that the paper proposes a unified framework for various MPC techniques, I was expecting to see a much richer experimental section comparing different approaches. The paper claims that performing a wider range of comparisons might be very expensive. But I don't understand why these experiments have to be performed in "full" MPC. The impact of various MPC blocks on model accuracy could probably be easily assessed without performing an actual distributed training?

Overall, while the main experimental result in the paper (an MPC with >98% accuracy on MNIST) is quite neat, it appears that achieving this result mainly required the engineering effort of putting together the techniques from past works and actually running the experiments for long enough.
It is certainly cool to showcase this, but further details on the benefits of the proposed framework and additional experiments with it seem necessary before publishing this paper. 
limitations_and_societal_impact: -
review_title: null

Meta-review: This work demonstrates an implementation of MPC to train a convolutional neural network for MNIST that achieves 98.5 (now updated to 99.2) percent accuracy.

The reviewers were unconvinced that the contribution of the work is well-explained in the paper. The primary contribution is to bring together known techniques to demonstrate an efficient and high-accuracy implementation of neural network training with MPC. While I find this to be an impressive contribution, I would expect to see a more thorough empirical section comparing different approaches (potentially in simulation, as one of the reviewers said). In the current form, I do not find this paper to be a good fit for this conference.
Decision: Reject