Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper designs differentially private hypothesis tests in the generalized linear model (GLM) for the problems of testing a linear relationship (where the null hypothesis is that the slope of the linear model is (say) beta=0) and testing for the presence of mixtures. 
The paper studies a DP version of the F-statistic and shows that converges asymptotically to the distribution of its non-private counterpart, as do the regression coefficients. Using this statistic, in combination with the Monte Carlo bootstrapping framework as used in Gaboardi et al (2016), the authors construct DP hypothesis tests both for testing a linear relationship and for the presence of mixtures.
Three datasets are used to evaluate these testers experimentally. The first tester is being compared to its non-private counterpart and to a test based on DP parametric bootstrap method for confidence interval  and seems to perform better. For testing for the presence of mixtures, the authors provide additionally e DP nonparametric method based on Couch et al. which performs better than the one based on the F-statistic for smaller datasets.
strengths_and_weaknesses: Strengths:
- The paper clearly states and supports its claims and positions itself with respect to prior work.
-  The convergence properties of the proposed DP version of the F-statistic are fully analysed. The authors show that the linear regression coefficients converge to their non-private counterparts with optimal rate and the power of the differentially private F-statistic converges to the statistical power of the non-private F-statistic.
- The paper includes experimental evaluations that compare the performance of the tests to the non-private tests but also to other approaches (one different approach for each problem), which creates a more complete picture. 

Weaknesses:
- I would say that there are not clear techniques that emerge from this work but analyzing the convergence of the statistic is not trivial and in general this is a minor point. It is a solid paper overall.


questions: One thing that I missed is some discussion on the choice of the F-statistic for these problems. I think some references on how widely-used this statistic is or why it is a better choice than other approaches in the literature might be useful to a statistics non-expert.

Small typo:
In line 8 of algorithm 2, the assignment to theta_0,theta_1 should presumably not be there.

limitations: Both adequately discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper provides a framework for differentially private hypothesis tests for testing a linear relationship (when also uncertainty estimates are reported) and testing for the presence of mixtures. The proposed tests for detecting presence of mixtures are the first DP tests tailored for this particular task. There is both strong experimental and theoretical evidence (DP F-statistics converges to non-DP one).
strengths_and_weaknesses: Strengths: 

- Strong theory, showing that the DP F-statistics provided by these test converge in distribution to those of the non-DP ones.
- Strong experimental evidence
- An extensive literature review on this line of research (DP hypothesis testing)

Weaknesses:

- I think the paper would need some polishing (see details below)
- Difficult to digest for a non-expert in this topic like me. Considering that problem is of quite general nature (DP hypothesis testing, linear models), I would imagine the presentation could be made more easily accessible to a wider audience. I am afraid that the results would be appreciated by only expert in this line of research.
- On a related note, I think it is good that there is a short introduction to hypothesis testing in the appendix, but I think it needs also polishing: for example F-statistics is not explained anywhere.
questions: 

The introduction says: "For the mixture model tester, we additionally adapt and evaluate a nonparametric method, a DP analogue of the Kruskal-Wallis test due to Couch, Kazan, Shi, Bray, and Groce [21]..". It remains very unclear without looking to the appendix what is meant by this.

Being non-expert in this topic, it sounds to me that there are absolutely no better fit among DP hypothesis testing methods you could reasonably compare against, am I correct?

One of first things coming to my mind: How can I trust that your adaptation of this median testing method is the best possible method to compare against, how can I trust this particular adaptation is reasonable? Why not have an adaption of some other DP hypothesis testing algorithm? In any case I think it would be good to have some more explicit motivation for these choices. 

Also, you mention that you compare to the DP parametric bootstrap method for confidence intervals but the corresponding experiments are only in the appendix (without reference).

Few remarks:

Perhaps a typo in abstract: "which are uniformly most powerful unbiased in the non-private setting" ?

Figures 3a and 3b: some dashed lines missing? (non-private baselines)

The notation for clipping used in Algorithm 2 is not explained anywhere.

Notice: The abbreviation KW that you use in Experiments - section is not explained anywhere in the main text. 
limitations: Yes (the methods are widely applicable: hypothesis testing for linear regression)
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Provides a GLRT & Monte-carlo based styled DP test for two kinds of hypothesis tests in the setting of linear regression: a) Testing a Linear Relationship, b) Testing for Mixtures. They take a holistic approach covering various aspects such as asymptotic  convergence in distribution results for the test-statistic. They provide empirical power analysis and significance analysis results. They compare against non-parametric tests such as the Kruskal-Wallis test.

Typically, a classical way has been to understand the asymptotic distribution of a test-statistic conditioned on null hypothesis being true, followed by verifying its departure from that case based on observed test-statistic to have a guarantee on the performance of the test both in terms of its significance, power and worst-case characterizations. Note: power of a test depends on effect-size, sample size, significance level and so forth
strengths_and_weaknesses: Strengths:  The submission does quite a thorough job in covering various aspects of DP hyp-testing for linear regression. They provide DP-tests, study convergence of test-statistic, draw similarities to GLRT, study the power, provide parametric bootstrap based computational test and so forth.

Weakness: It is bit unfair to expect this, given the page limit for the main paper: That said, the organization of the paper's approach & results in the appendix could have been better covered in the main paper. This is purely a comment in terms of providing clarity in the reading. 


questions: 1.) line 269: is the notation a typo or intended in terms of a constant in using O(1) in log 2n^O(1)?

2.) To this Q, the authors mention the conclusion section covers limitations-although it does not.: (Did I miss it?)
     Did you describe the limitations of your work? [Yes] See conclusion and experiments 
     425 section (our DP tests do not perform well for certain data distributions). 
limitations: -some details in expt section on how the empirical power was computed would be useful.
-the word appendix is referred to only once in the main paper. it instead refers to section A or C etc.
Please say 'Section A in appendix' and so forth.
-The section on Overview of Techniques seems a bit too packed in a short space. It should ideally better help the navigation of the paper.

-reference to paper on test statistic introduced in eqn 1 is needed or it needs to be clarified that it was introduced in this submission. it is unclear-which case it is in the beginning of the paper to the reader.

-minor: The word general linear model (GLM), has been used multiple times in this submission to refer to linear regression with Gaussian assumptions. GLM is typically used for generalized linear models of which linear regression is just a special case. please change the terminology to avoid this overloading. 
-A table of main notations would be quite helpful.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of hypothesis testing for two problems in the generalised linear model: testing a linear relationship and testing for the presence of mixtures. Most of the techniques involves are based on privatising the $F$-statistic for the general linear model. These techniques are used through a Monte Carlo parametric bootstrap. Other techniques for testing presence of mixtures in this model are based on the DP non-parametric tests of a prior work by Couch et al. 2019.

The main results are that the DP $F$-statistic converges to the asymptotic distribution of its non-private counterpart, which implies the convergence of statistical power of DP $F$-statistic to that of the non-private $F$-statistic. The Monte Carlo parametric bootstrap is used to ensure an $\alpha$ target significance.

Asymptotic results for these problems are obtained in this paper. Also, experiments are performed to compare these results with either the non-private counterparts of these tests or other available techniques for this problem.

Edit: Updated score.
strengths_and_weaknesses: Strengths:
1. Given that this problem has not been studied that extensively under DP constraints before (except in Sheffet 2017), I would say that this is a great starting point for this.
2. The work is also fairly technical and rigorous. The analyses of the significance levels theoretically is very thorough even if it is dealing with just asymptotic guarantees.
3. The convergence of statistical power of the DP $F$-statistic to that of its non-private counter part is a significant contribution. This means that the loss in accuracy due to privacy isn't much, which is usually the main target in most work on DP.
4. This work also addresses one of the main issues in the prior work on this: the prior work only starts to reject the null hypothesis when the sample size is very large. In other words, their tests could give accurate results even on "smaller" datasets.
5. One more thing I appreciate that the experiments have been performed both on synthetic and real-world datasets, as opposed to just on the former. It gives the impression that these tests are quite versatile in terms of the underlying data distributions.

Weaknesses:
1. I don't see this as a huge weakness (and is dependent on personal preferences), but getting non-asymptotic results would have made the paper a bit stronger. There are different schools of thoughts on this, of course. As I said though, this is not a big concern for me.
2. I have some issue with the writing of the paper. It looks like it was a beautifully written paper, but chopping it down to nine pages made it a little hard to read in the first shot. I'll be more specific about this. (A) The fluff text for Algorithm 1 didn't do much for me. I agree that a text version of the algorithms is supposed to help, but an intuition behind why certain decisions and steps are taken within them is probably a more helpful discussion. (B) Saying which algorithms have been restated in the appendices would have been nice. It gets confusing when algorithms are stated in the main body, but the discussions refer to the ones in the appendices.
questions: I don't have a lot of questions, but just something for clarification.

1. The results in this paper are valid for the case when the ambient dimension is $2$, with the claim that they could be extended to arbitrary dimensions. Is that extension straightforward? Or would you require techniques that are significantly different mathematically from the ones being currently used in this paper?

2. Also, while generalising to higher dimensions, does the accuracy of the private tests still converge to that of the non-private ones?

3. In Line 12 of Algorithm 1, why do we need to use DPStats? I thought those points are artificial points, so do we need privacy here?
limitations: I think the authors have been honest about wanting to have non-asymptotic results for this problem. Even though I have mentioned that as a weakness above, I don't think it's a big deal.

I have mentioned my concerns about the writing above already. So, I don't have any other concerns. This results are quite self-contained and "optimal" already in the sense that the private convergence and the non-private convergence match.

Also, one more thing I would recommend is motivating the problem a bit more here. We know hypothesis testing and linear regression independently, but why is testing for this problem important? More like, why is privatising this is an important problem to solve? There are lot of problems that could need private solutions, but why is this an important problem to solve under DP constraints?
review_title: null

Meta-review: metareview: The reviewers were all quite positive on the results, and agreed the paper should be accepted. The only point that came up in the discussion is a slight miscommunication between the reviewers and authors: the reviewers asked for better motivation for some aspects (e.g., the F-statistic), and it seemed like the authors said they would motivate linear regression better. All the reviewers understand linear regression is very important, so the authors are recommended to instead revise based on their specific comments.
confidence: Certain
Decision: Accept