Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed a modified version of Hilbert-Schmidt Independence Criterion (HSIC) to test the dependence between two multivariate variables. The method is suitable for data that is not iid, i.e. correlation exists between different data points. The paper first demonstrate the asymptotic distributions of HSIC under the null hypothesis (independent) and the condition for the alternative in Theorem 3.2 and 3.3. The approximation of the asymptotic HSIC distribution is provided in proposition 3.4. In the simulation, the proposed method HSICcl shows a better performance than earlier versions of HSICorig both in terms of Type I error and empirical power. In real data, HSICcl identifies more associated pathways than previous versions. 

main_review: I have not be able to go through the theoretical part and just comment on the real experiment part. It will be nice to justify the identified pathways are biologically meaningful, especially the newly identified pathways. In line 284, it says “The original study found no additional benefit of estradiol or moisturiser over placebo.” How is this conclusion linked with results provided in this paper? Also, it says “In a second study …..”, what is the main conclusion of that paper? Is the main conclusion supported by this paper’s results?

I think this test will be interesting if its predictions could be justified in the biological setting.

---------------------------
I am convinced by the authors' reply that the extra detected pathways are biologically sensible and I will raise my rating for this paper. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper derives an approximation of the asymptotic null distribution of the Hilbert-Schmidt Independence Criterion under a matched and complete clustering. The null distribution corresponds to independence between the variables.  The samples consist of equally sized clusters each with matched within cluster correlation. The distribution is described as a weighted sum of chi-squared variables, where the weights are approximated by the eigenvalues of a matrix composed from the eigenstructure of the marginal kernel matrices grouped by cluster. Simulations confirm the increased power of the method. Specifically, simulation shows the test is conservative in lower sample size, but increases in power versus other methods (concatenating the samples longitudinally).  Experimental results demonstrate the increased power is able to identify additional metabolite pathways are associated with the vaginal microbiomes composition. 
main_review: **Originality** This is a useful tool for dependence testing in cases of non-i.i.d. instances. The paper's contribution is useful. The paper builds from Zhang et al. taking the assumption of equivalent correlation within each cluster structure to derive the novel asymptotic distribution.

**Quality** The technical description and experiment are statistically sound. The discussion regarding small sample methods (non-asymptotic) could be expanded beyond the reference to Zhan et al. (2017). 

In terms of comparisons, it seems permutation based tests could derive surrogate null distributions to handle the cases of missing data and unequal cluster size.  Especially for the simulated data, I would like to see a comparison versus permutation based methods. On the real data, it seems that the  stringent significance threshold (based on Bonferroni correction) would require a large number of permutations, but it could be useful to compare.

**Clarity** The paper is well-organized and well-written. 

**Significance** A solid statistical test for dependence has the potential to impact scientific discovery in exploratory work such as biomarker discovery. Longitudinally, clustered data is increasing in importance in biomedical health applications.  A properly documented and released code may positively impact biostatistical practice. 
 
### After authors' response.
I have read the authors' response and the other reviews. I maintain my overall score. I hope the authors can report computational complexity (cubic in the sample size to determine asymptotic distribution for the Gaussian kernel) could be mentioned in addition to the wall clock time. 

limitations_and_societal_impact: A statement of the computational complexity and its comparison to the computation required to run permutation tests with HSIC would be useful.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This submission introduces a calibrated testing procedure for the
independence between two variables under non-iid sampling. The
sampling is assumed to follow a clustered structure, where samples
within clusters are correlated but the clusters are independent from
each others. The procedure relies on the Hilbert-Schmidt Independence
Criterion (HSIC), and a first contribution is an asymptotic null
distribution under the clustered sampling assumption. The test is also
shown to be consistent. Experiments on simulated and real data show
that the proposed test guarantees a control on the type I error, and
has higher power than existing alternatives in a variety of scenarios.

main_review: The submission is clear and well written. The problem of non-iid data
is an important and (in my opinion) under-addressed problem in
statistics and machine learning, making this work a useful and
interesting contribution. This contribution is significant as the
proposed results are non-trivial. The experiments are thorough and
illustrate the proposed results well.

I would appreciate some clarification on the effect of d (the size of
the clusters) on the proposed results. The sampling assumption
constrains all clusters to be of equal size d, so that the sample size
n is m*d where m is the number of clusters. All asymptotics are given
in m->inf (and therefore n->inf) and these asymptotics don't seem to
depend on d. Is it possible to say something about non-asymptotic
properties of the statistic that would vary with the cluster size?
Does this size affect the convergence speed? Experiments with varying
cluster sizes would also be welcome. With d=1 one recovers the iid
setting, and all experiments are done with d=3. Larger d will always
increase the dependence. Would the performance (calibration, power) of
the proposed procedure be affected in this case?

I also have a few minor comments:

- I think that the statement of Proposition 3.4 could be made more
precise. It currently states that under some assumptions, "we can
compare the statistics [...] against the (1-alpha) quatile of the
distribution [...]". I would find it more clear if the convergence
result was stated in the Proposition (and the consequence on practical
testing could be discussed after the Proposition).

- line 129, the "centered kernel functions" could be defined more
  clearly. The only centering defined so far has been on empirical nxn
  Gram matrices, not on functions.

- lines 234-236, I couldn't understand how X** differed from X. Both
  are defined as (x_11, x_12, x_13, ..., x_p1, x_p2n x_p3). If they
  are identical, this should be stated clearly. And in this case,
  HSIC_cat is equivalent to HSIC_orig?

- line 254, it could be useful to provide intuition as to why HSIC_cl
  with Gaussian kernel tends to be conservative.

- line 254, it is not clear from the table that the type I error rate
  gets closer to the nominal alpha when m=1000, the difference with
  m=500 is tiny. Did you observe a stronger improvement with larger m?

- Figure 1: power is provided at nominal level for all methods. Are
  the baseline methods correctly calibrated?

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a test of independence for cluster correlated data. The test statistic is still HSIC. However they derive a different asymptotic distribution under cluster correlation.
main_review: The paper seems well wrtitten. I don't have many comments about the writing  or the structure.

Unfortunately I believe the problem/contribution to be rather narrow. I am not sure if cluster correlated independence tests are impactful/ how an important this problem is.

In the real data experiment, could you write explictly what is X, what is Y, and what are their dimensions.

I DID NOT CHECK PROOFS IN THE APPENDIX. I think 14 pages of technical appendix is challenging to completely verify. I believe it would be stronger if most of the proof could be given in the main text, or some insightful high-level discussion/sketch could be given that suggests the proof is correct.

Experimental evidence that the results are correct are currently not very strong I believe. For the Gaussian kernel the p-values are not that close to alpha (although they are smaller so in terms of type 1 error it seems ok). Could you experimentally demonstrate the p-values are uniformly distributed under the null? Or take larger samples?

While it is nice there is a check fo the type 1 error experimentally in table 1, I'm not sure how convincing one gaussian distribution is to conclude from this that the type 1 error rate is correct. 

Overall view: 

The paper is generally well written and structured. The experiments are well organized and described. I'm a bit in doubt between weak accept/weak reject. The proofs are all in the appendix and I was not able to check them due to length. More insightful proof sketch that suggests the analysis is correct/ more extensive type 1 experiments could help in that respect. Main question for me is if cluster correlated independence tests are relevant/impactful enough for neurips.


limitations_and_societal_impact: -
review_title: null

Meta-review: The paper proposes a test of independence for cluster correlated data based on HSIC and derive an asymptotic distribution for HSIC under cluster correlation.
In evaluations, the test is slightly conservative.
The test is applied to a microbiome study.
The paper is well organized and well written.
After discussion with the authors, only minor issues remain.

However, the reviewers agree that this is a good contribution and should be accepted.
Decision: Accept (Poster)