Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the problem of secure and privacy-preserving distributed ML training (specifically logistic regression model). It does so by combining MPC and DP. The approach by normalizing the data and then adding the noise to the parameters proportional to sensitivity of how this data would change the parameters. The protocol is completely distributed and does not require a trusted party.
The paper seems to describe the method that was used in iDash competition last year.
strength_and_weaknesses: Strengths:
- a really well-written paper, extremely easy to follow
- the problem of cross-silo learning is addressed completely by considering security, privacy and integrity
- related work is very well-explained (recommendations on expanding are below)

Weaknesses:
- the novelty of the work compared to existing MPC+DP approaches is not clear
- DP guarantees need further investigation (e.g., why delta is missing and why not use Gaussian discrete noise)
- the method seem to work only on LR models
- baselines method (no DP and no MPC is missing)
- comparison with other MPC+DP approaches is missing
- how parties combine their data in vertical setting is not explained
clarity,_quality,_novelty_and_reproducibility: Clarity: The authors explain the method extremely well.
Quality:
- DP mechanism needs to be explained more thoroughly. For example: (1) why not use discrete Gaussian [1] (2) Does your method provide DP guarantees even if it is using Box-Muller on not real values that have been shown to be vulnerable to floating point attacks (3) does the method provide epsilon or epsilon, delta guarantees.
- how do parties match their features when data is vertically split? would not they need to run an MPC protocol for that as well?
- what is the baseline of an approach without DP and MPC and use of fixed point arithmetic? Please also consider reporting communication overhead.
Novelty:
There is a bit of work in this space and it seems that this paper combines existing techniques. Is it the performance that is the main contribution? If so how does it compare with other works (SecureML, [2]) or with trusted-execution based works?

Related work:
[1] The Discrete Gaussian for Differential Privacy by Canonne et al.
[2] CaPC Learning: Confidential and Private Collaborative Learning by Choquette-Choo et al.
SecureML by Mohassel and Zhang.

summary_of_the_review: This is a really nicely written paper. However contributions seem minor theoretically. Many details that need to be in main body (e.g., other datasets, generalisability, communication overhead appear in appendix) and it is hard to determine what is the generalizability of the methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of training a logistic regression (LR) model with differential privacy over distributed data. In order to improve over local DP approaches, it relies on multiparty computation to train a pooled-equivalent model on secret shares, adding differential privacy on the final model thanks to the output perturbation method of Chaudhuri et al. Experiments are performed on several datasets and demonstrate the improved performance, at the expense of a large communication cost (which can be mitigated depending on the protocol used and the threat model)

To sum up, the main contributions of the paper are:
1) Proposing an MPC protocol to run the output perturbation of Chaudhuri et al. for logistic regression, working for both horizontal and vertical FL
2) Extending MP-SPDZ for logistic regression training with L2 regularization
3) Empirical study of the benefits of the proposed approach in terms of performance (for a given privacy budget) and communication, mainly on the iDASH 2021 track III competition dataset but also on TCGA and GSE2034 in the supplementary
strength_and_weaknesses: # Strengths

1. The paper is very well written, which was not a given given the very technical nature of the topics tackled (MPC *and* DP). I really thank the authors for their writing efforts
2. The idea proposed is general enough to be applied to any model amenable to the output perturbation method of Chaudhuri et al., which includes e.g. SVMs
3. The method seems to reach the best performance on the current benchmark
4. The method should yield results equivalent to centralized training (up to discretization issues), which is great for healthcare applications where there is high heterogeneity and little data per center

# Weaknesses

5. The paper is still limited to the output perturbation method, and only covers the logistic regression
  - In particular, I think the abstract is misleading and should reflect more the focus on logistic regression
6. The structure of the paper is unbalanced between the main paper and the supplementary material. The supplementary material addresses many points that are lacking in the main paper, including extension to other datasets, more interesting baselines, issues with finite precision. Several appendices are not even referred to in the main paper.
7. The choice of baselines (even after the extension in the appendix) is not entirely fair:
  - DP-SGD is performed with a batch size of 1, which is a worst-case approach.
  - Further, DP-SGD is only used with local trainings followed by ensembling. The authors do not consider a FedAvg + DP-SGD approach which would be natural in this setting. Such an approach was already used e.g. in the iDASH 2020 competition (Beguier et al. Differentially Private Federated Learning for Cancer Prediction)
  - For the method of Jayaraman et al. 2018 in table 6, the variant with output perturbation is used, although this paper shows that the gradient perturbation yields better results
8. Regarding the experimental setup :
  - It would be better to add error bars (where applicable) to understand the significance of the results. In particular, this could be possible for the cross-validation results of Table 2 and Table 6
9. For vertical FL, one of the main challenges is to align record across parties. Which method do the authors use to tackle this question?


Minor comments:
- How do you explain the fact that in Table 7, central learning yields a better performance than the proposed MPC method? Is it due to finite precision?
- how much does the initial secret sharing transfer (encrypted data points) weigh with respect to the total communication cost?
- For L2 data normalization, for horizontal FL, it is probably simpler to apply it prior to creating shared secrets. When computing the total runtime, is it contained in it?
clarity,_quality,_novelty_and_reproducibility: # Clarity

The paper is very well written and clear.

# Quality

I think the paper and its appendices contain material for a good paper, up to some restructuring and maybe adding a few more baselines (cf above)

# Novelty

The contributions of the paper, although limited in scope, are novel to the best of my knowledge

# Reproducibility

Code is provided by the authors. I have not checked it exactly to understand how easy it would be to reproduce it.
summary_of_the_review: This paper proposes a novel protocol for logistic regression with differential privacy in a distributed setting, which seems to reach good results. Although I think the paper would benefit from a restructuring and adding some baselines, I am bending in favor of acceptance, hence the current score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: the paper studies a method which combines MPC with DP that bypasses utility challenges of federated learning with local DP (with small datasets). authors show that the method achieves good performance by reporting their submission entry being 1st place for an iDASH 2021 challenge, at the cost of substantially more compute time. 
strength_and_weaknesses: strength: the paper is well-written and the empirical results are solid. 

weaknesses: 
- the main method in the paper is a combination of existing protocols in MPC and differentially private statistics release, and thus the paper lacks methodological novelty. but this perhaps isn't a super serious issue, given that authors demonstrated their point and showed good performance. 
- the paper only considers a single dataset / benchmark on which results are reported. this makes understanding the robustness of the method and results a bit difficult. 
- the method is likely so far most practical for small (to perhaps moderate) size problems, given the large compute overhead. 
clarity,_quality,_novelty_and_reproducibility: quality: the paper has good technical and empirical quality. 
clarity: the paper is well-written. 
originality: the paper lacks a bit of methodological novelty, but the fact that it got the different components together working well together is new. 
reproducibility: the results are likely reproducible, given the authors attach a link to their code.
summary_of_the_review: the paper studies a method which combines MPC with DP that bypasses utility challenges of federated learning with local DP (with small datasets). authors show that the method achieves good performance by reporting their submission entry being 1st place for an iDASH 2021 challenge, at the cost of substantially more compute time. the paper is well-written, is of high quality, but lacks a little in methodological novelty.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of differentially private federated learning. More specifically, in order to improve the utility of the differentially private model in federated learning, the authors propose to combine secure multiparty computation (MPC) with differential privacy (DP) when solving the regularized logistic regression. The main contribution of the paper is a new framework that allows private training for vertically distributed data.
strength_and_weaknesses: The strength of the paper:
1. The proposed method enables private training for vertically distributed data, which is an important setting in federated learning.
2. The authors propose a new method for secure sampling of a vector from a Gaussian distribution inside MPC.
3. Empirical results validate the advantage of the proposed method.

The Weaknesses of the paper:
1. It seems that the proposed method requires communication between computing parties at each iteration, which could be the key bottleneck in federated learning. Compared with the output perturbation-based method, which only requires one communication at the end, the proposed method suffers from high communication complexity.
2. There are no theoretical guarantees of the proposed method.
3. The presentation is not clear. For example, the learning problem is not defined. There are lots of undefined notations. More importantly, the random noise magnitudes for several baselines are unclear.
4. It seems to miss an important baseline, i.e., Gu et al. (2021), in the experiments.
clarity,_quality,_novelty_and_reproducibility: The presentation of the current paper is not clear:
1. It is unclear how to get the update rule in line 8 in Protocol 4. If we choose $C$ and $\Lambda$ to be zero, should it recover the original update rule?
2. There is no formal privacy guarantee of the proposed method.
3. For the baseline methods in Table 1 and Table 5, what are their noise magnitudes to achieve DP? For example, how does the variance of BASELINE-DPSGD in Table 5 scale with the number of samples and computing parties?
4. Why the accuracy of nonprivate method can be worse than the private method with privacy budget 1 in Table 4?
5. It is unclear how to get the hyperparameters of the proposed method.

The main contribution of the proposed method is to use secure computing parties, which enables the training of vertically distributed data. However, such a method seems to be very easy to implement in existing methods and thus allows them to handle the vertically distributed data. Furthermore, what is the advantage of the proposed noise generation method inside MPC compared with existing methods?


summary_of_the_review: The proposed method seems to be interesting since it can deal with vertically distributed data in federated learning. However, there are some problems that need to be addressed in the current paper (see weaknesses and clarity sections).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an MPC solution for training differentially private models, which achieves higher accuracy than a pure DP approach while providing the same formal privacy guarantees. 

Strengths:

+The paper is well written

Weaknesses:

-The novelty is limited because MPC has been used for training differential private models in many prior works

-Lack of thorough experimental evaluation

The main concern about this paper is the novelty of the algorithm. Even after the author's response, it does not gather sufficient support. 

justification_for_why_not_higher_score: The main concern about this paper is the novelty of the algorithm.
Decision: Reject