Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper proposes chi-square test in the federated setting where data sharing is not possible.
main_review: Paper proposes Fed-Chi square,a  secure chi-square test in the federated setting. The proposed method is practical in that it has many real-world applications as discussed by the authors. Overall, I like the paper, I have couple questions for the authors:

1. For security, although the paper states that the method does not guarantee differential privacy, is it possible to show the gain in accuracy after not considering DP?

2. Also, how does this compare to standard pooling of test statistics? i.e. everyone does the test on their own data and the final estimates are pooled. Similar to how one does for multiple imputation.
summary_of_the_review: It is a promising paper, I have couple minor questions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses the problem of conducting correlation tests with sensitive data separately collected from $n$ clients, where centralizing data collection is risky but the available secure multiparty computation is costly in computation. The proposed test adjusts the classical chi-square test by first rewriting it as a second frequency moments estimation using $n$ vectors, each from a client. The proposed test prevent data recovering when performing the test by letting each client project its vector to a lower dimension. The higher the dimension is, the more accurate the test is, yet the higher the computation costs.

main_review: Strengths:
1. The algorithm is intuitive and easy to implement, and several possible applications are discussed. 
2. It provides the relationship of the reduced dimension with the test accuracy and the computation cost.

Weaknesses:
1. In evaluation section 4.1, what is the accuracy of the original chi-square test? If the original test has power 1 in the presented simulation, would we expect a similar comparison with the proposed test when the original test has lower power, such as 0.5?
2. I don’t fully understand Figure 3. What is an epoch? When computing the accuracy, what is the ground truth?
3. In the case study of online FDR, could you also show the power (#correctly rejected/ #hypotheses should be rejected)? Although unlikely, I would want to know whether lower FDR as $l$ increase is because the test is conservative and the power is also lower. Could you also include the original chi-square test in this case?
4. Theoretically, do we know if the FED chi-square test statistics is an unbiased estimation given the original chi-square statistics, or systematically larger or smaller?
5. As a reader not in the expertise of stable random projection and geometric mean estimator, I don't fully understand why the lower dimensional $e_k$ gets to the approximation.

summary_of_the_review: Overall I think the proposed test is intuitive, but I am a little confused about the experiments and theory as described above. I would feel more confident to judge the paper if my confusion is resolved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a federated analytics technique for computing the Pearson correlation for two random variables. The method is based on repeated uses of secure aggregation, as well as stable random projections. The protocol is argued to be secure in a semi-honest security model. 
main_review: The algorithm presented in the paper is pretty simple. First, it uses secure aggregation to compute marginal statistics. Each participant applies a stable random projection (with particular parameters) on their (normalized) frequency distribution. The results are then aggregated again using secure aggregation. Due to the nature of the stable distribution, the aggregated result follows a stable distribution as well, and its scale can be estimated. This scale ends up being equal to the squared sum in the correlation coefficient. 

One concern I have is that the protocol actually reveals some significant amount of information to the server in addition to just the result. The authors explain in a remark that this is not a concern, because the leaked information constitutes a linear system (over a finite field) with a large solution space. However, I don't understand if all of these solutions are supposed to be equally realistic or equally likely. For example, realistic solutions may be particularly short vectors, or have some other properties that may make identifying them possible. Does this make it easier to find a solution to the system?
summary_of_the_review: This paper presents a nice new federated analytics technique for computing correlations. I have some concerns about the security of the protocol, but it's possible I'm misunderstanding something.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a federated $\chi^{2}$-test protocol. This is proposed as a technique that is computation- and communication-efficient and leaks much less information than other alternatives. The claim is that the proposed technique can tolerate up to 20\% of clients dropout with minor accuracy drop. The proposed technique does not include differential privacy guarantee.

main_review: Strengths: 

1) The problem is important and complex. 

2) The numeric experimental results are somewhat convincing.

Weakness:

1) I could not find any plausible proof or development of the theoretical claims. This may be a matter of presentation, but Sections~3.5-3.7 seem to be a bunch of statements without justification, explanation, mutual linkages or proofs. 

2) The paper is hard to follow and seems hastily written without proper development. For example, Section~3.2 seems to end abruptly. It describes a challenge, but not how the paper's proposal addresses the challenge. Such details may be elsewhere, but is unclear from the narrative. 

3) Not considering differential privacy seems a major deficiency. 



summary_of_the_review: The problem is important. However, technical concerns remain and the paper does not seem to be well-developed and is hard to read. 


Revised comments: Upon reading the authors' comments, I understand their perspective on proofs (not one that I completely agree with) and I thank them for pointing out the places where proofs may be found. However, I am still not convinced that the writing is transparent and clear, or that the issue of differential privacy may be entirely waved off. That being said, I will revise my rating and downgrade my confidence/expertise score, so if the paper is appealing to other reviewers, I will not stand in the way of its acceptance.
review_title: null

Meta-review: This work proposes a federated version of the classical $\chi^2$ correlation test. The key new step is the use of stable projection to reduce computational overheads associated with the use of secure multi-party protocols. Overall while the contribution is of interest the novelty is rather limited. I also consider the work to be somewhat outside of scope for ICLR. It would be more suitable for a security or statistics focused venue. Therefore I do not recommend acceptance.
Decision: Reject