Title: AMDP: An Adaptive Detection Procedure for False Discovery Rate Control in High-Dimensional Mediation Analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Mediation analysis

Composite null hypothesis

Local false discovery rate

Optimal ranking rule

High-dimensional
Abstract: High-dimensional mediation analysis is often associated with a multiple testing problem for detecting significant mediators. Assessing the uncertainty of this detecting process via false discovery rate (FDR) has garnered great interest. To control the FDR in multiple testing, two essential steps are involved: ranking and selection. Existing approaches either construct p-values without calibration or disregard the joint information across tests, leading to conservation in FDR control or non-optimal ranking rules for multiple hypotheses. In this paper, we develop an adaptive mediation detection procedure (referred to as "AMDP") to identify relevant mediators while asymptotically controlling the FDR in high-dimensional mediation analysis. AMDP produces the optimal rule for ranking hypotheses and proposes a data-driven strategy to determine the threshold for mediator selection. This novel method captures information from the proportions of composite null hypotheses and the distribution of p-values, which turns the high dimensionality into an advantage instead of a limitation. The numerical studies on synthetic and real data sets illustrate the performances of AMDP compared with existing approaches.


Peer reviews:
comments: SUMMARY:
This paper proposes a high-dimensional mediation analysis detection procedure with false discovery control. Classic FDR control for multiple testing does not use information across tests making it overly conservative. The authors propose a local FDR-based procedure for identifying mediators and a data-driven approach for estimating the null densities and determining thresholds. Experiments show the proposed approach enjoys high power while still maintaining good FDR control.

STRENGTHS:
The application of multiple testing on high-dimensional mediation analysis is interesting. 

The local FDR uses across-dimensional information, and the test statistics and thresholds can be estimated from data. 

The proposed algorithm maximises power while maintaining FDR control. 

Applying a local FDR-based method mediation analysis problem is not trivial. The data-driven approach proposed in Section 2.2 could be useful in other FDR control applications (e.g. how the null densities are obtained).

WEAKNESSES:
This paper is very technical. I would appreciate some more introduction and motivation: 

What is local FDR? 
Why do we want to use local FDR in mediation analysis? 
What may be the obstacles when using local FDR in mediation analysis? 

If the authors could present answers to these questions at the beginning, it would make the paper feel more motivated.

QUESTIONS:
None.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a method to control for false discoveries in high-dimensional mediation analysis. This is the problem of inferring whether any of a large set of possible mediator variables act as a mediator between an exposure variable and an outcome variable. This work is particularly interested in the application where the exposure is a single-nucleotide polymorphism (SNP), the exposure is the expression level of a gene, and the mediators are the methylation levels at a set of CpGs. It is known that methylation is involved in the regulation of gene expression, so in this setting one wishes to discover whether the SNP is affecting the methylation at a CpG and in turn this is affecting the level of gene expression of a gene. This is an interesting problem and one of interest when seeking to uncover the functional effect of genetic variants. In this setting, there will typically be many CpGs as possible mediators, so we are in the realm of high-dimensional statistics and need to account for the fact that we are running a large number of tests in parallel. Previous methods for this problem are deemed to be too conversatve, sacrificing power for the suppression of false discoveries, and the authors seek to identify an approach which has improved power.\n\nThe method is verified on simulated data and illustrated on a prostate cancer data-set, where it is shown to identify more (SNP, CpG, gene) triples than existing methods\n\nSTRENGTHS:\n- The paper tackles a problem of practical interest\n- The method is justified theoretically (I did not check the proofs)\n- The method is justified empirically on simulated data and illustrated on a real-world data set\n\nWEAKNESSES:\nThe presentation felt a bit compressed with many relevant details deferred to the appendix, such as the examples described in Remark 2 and many of the plots relating to the prostate cancer data-set.\n\nThe authors provide limited information on the context, assuming a reader will be familiar with genetic and epigenetic analysis.\n\nQUESTIONS:\nNo questions\n\nLIMITATIONS:\nI do not believe that the method has any obvious negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
The paper proposes and analyses a new method for FDR control.

STRENGTHS:
The paper is clear on the method, the assumptions and the proofs. The empirical section is promising and adequate.

WEAKNESSES:
Some of the assumptions and the writing are not clear, or seem too strong to me. For example the assumption about "no confounding" which is not listed as a formal assumption but is mentioned in the text, and attributed to another paper. It would be helpful is the authors could provide an explanation about why this is not a limiting assumption. 

The assumption 2 seems reasonable, but assumption 1 again seems limiting, with a few distributions that could satisfy it. It is not a paper breaking assumption, but it would be useful if the authors are upfront about it, and discuss which possible distributions satisfy it. Similarly, about assumption 2 it would be helpful if the authors could write a couple of sentences discussing what the assumption means and implies. 

The abstract and the intro makes a big deal about the curse of dimensionality becoming a blessing, but this was not obvious to me from the paper's text. could the authors explain what they mean by this exactly ? could you point out where this is discussed in the paper? 

I thought the "limitations" section was required this year for neurips submissions. but this is missing, and the discussions section at the end is not comprehensive about this.

QUESTIONS:
Please see the weakness section.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUPDATE \n\nIn light of the revisions the authors made I have raised my score and now support acceptance of this work. Thank you very much.\n\n-----------------------------------------\nThe authors propose a procedure to increase statistical power to identify mediators while controlling the FDR in high-dimensional data sets. Their method leverages the proportions of composite null hypotheses and the distribution of p-values under the alternative to derive an algorithm to estimate p-values for mediator variables while controlling the FDR. They perform a theoretical analysis suggesting that their method is asymptotically optimal and showcase that it controls FDR more consistently than other methods (DACT, JC, JS-mixture) in a simulation study and identifies more mediators than JS-mixture in an empirical study suggesting higher statistical power.\n\nSTRENGTHS:\n- The problem is very relevant as high-dimensional data are frequent not only in genetics, but also in other domains like neuroscience and user data, which all potentially suffer from similar problems.\n- The work is very extensive comprising theoretical analysis, simulations and an empirical application of the methods.\n- Related methods are extensively discussed in the introduction.\n\nWEAKNESSES:\n- There is no real discussion of the limitations/breakpoints of this method.\n- The effect sizes chosen for the simulation analysis seem to be unreasonably large (at least showing how the method behaves for smaller effect sizes, which are more commonly observed empirically, would be informative).\n- The paper is occasionally quite dense and would benefit from focusing on some key aspects.\n\nQUESTIONS:\nThis is very interesting work with broad applications to many empirical studies. I am willing to support acceptance of this paper, if the following concerns/questions are addressed appropriately. I hope you find my comments helpful and constructive.\n\n**Major points**\n- Can you provide a motivation for the choice of simulation parameters (i.e., sample sizes, number of mediators, FDR alpha and effect sizes). Current genetic studies tend to have larger sample sizes ( in the order of 20.000) while older studies had smaller sample sizes so including a wider range of samples e.g. 100, 500, 1000, 10000, 20000 may better cover empirical scenarios. Moreover, I don’t understand why you picked alpha = 0.1 as reference, since most studies choose alpha=0.05, for genetic studies this might even be smaller.\n- In Figure 1, it appears that AMDP deviates more from the 0.1 FDR line when the effect sizes get smaller. Could you speculate on why that is? On that note, the effect sizes chosen are rather large. What happens for smaller effect sizes (up to .2 or .3). Does this deviation get larger, is this a breakpoint for the method?\n- Similarly, in the empirical analysis (Figure 3), JS-mixtures identifies more triplets for alpha=0.01. Could you explain why that is, does this trend hold for alphas that are even smaller like 0.001?\n- What are the conditions under which the method is not appropriate? When does it break? I am missing a clear limitation section.\n\n\n**Minor point**\n- p.8 ll. 288 ff. essentially contains a figure caption of a figure that is presented in the supplement, which is not very helpful for the reader. I would suggest to either summarize the take-aways from this supplementary analysis here and move the caption to the figure where it belongs or spend this space on discussion the implications including break points and limitations of the analyses you present in the main manuscript. \n- Just out of curiosity, can you elaborate on how this method could be used for image analysis? Can this be applied to neuroimaging (e.g., fMRI) as well?\n\nLIMITATIONS:\n- The paper does not have a clear limitation section (see Questions).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}