SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the previous section we have seen how to reach FDP control via Knockoffs. As explained above, aggregation is needed to mitigate the randomness of the Knockoff generation process. Therefore, we aim to extend the previous result to the case of aggregated Knockoffs. Let us first define aggregation:
Definition 7. For  $D$  draws of Knockoffs, an aggregation procedure is a function  $f:\mathbb{R}^{D}\mapsto \mathbb{R}$  that maps a vector of  $(\pi^d)_{d\in [[D]]}$  statistics to an aggregated statistic  $\overline{\pi}$ .
In practice, since we have  $p$  variables, aggregation is performed for each variable, i.e.:
$$
\forall j \in \llbracket p \rrbracket , \quad f \left(\pi_ {j} ^ {1}, \dots , \pi_ {j} ^ {D}\right) = \overline {{\pi_ {j}}}.
$$
Then, inference is performed on the vector of aggregated statistics  $(\overline{\pi}_1,\dots ,\overline{\pi}_p)$
For a fixed aggregation scheme  $f$ , we can naturally extend the calibration procedure of the preceding section. Instead of drawing a single  $B \times p$  matrix of  $\pi^0$  statistics containing  $\pi_b^0 \in \mathbb{R}^p$  for each  $b \in [[B]]$ , we draw  $D$  such matrices. Given  $d \in [[D]]$ , each matrix contains  $\pi_b^{0,d} \in \mathbb{R}^p$  for each  $[[B]]$ .
Then, for each  $b \in [[B]]$ , we perform aggregation:  $\overline{\pi}_b^0 = f\left((\pi_b^{0,d})_{d \in [[D]]}\right)$ . The JER in the aggregated case is defined as:
$$
\overline {{\operatorname {J E R}}} (\mathbf {t}) = \mathbb {P} \left(\exists j \in \llbracket k _ {\max } \wedge p _ {0} \rrbracket : \bar {\pi} _ {(j: \mathcal {H} _ {0})} <   t _ {j}\right).
$$
We obtain the aggregated template following the same procedure, i.e. drawing  $D$  templates and aggregating them. For each  $b' \in [[B'))$ , the aggregated threshold family is written:
$$
\overline {{\mathbf {T}}} \left(\frac {b ^ {\prime}}{B ^ {\prime}}\right) = f \left(\left(\mathbf {T} ^ {d} \left(\frac {b ^ {\prime}}{B ^ {\prime}}\right)\right) _ {d \in [ D ]}\right).
$$
We can then write the empirical JER in the aggregated case as:
$$
\widehat {\mathrm {J E R}} \left(\overline {{\mathbf {T}}} \left(\frac {b ^ {\prime}}{B ^ {\prime}}\right)\right) = \frac {1}{B} \sum_ {b = 1} ^ {B} 1 \left\{\exists j \in \llbracket k _ {m a x} \rrbracket : \bar {\pi} _ {b (j)} ^ {0} <   \overline {{\mathbf {T}}} _ {j} \left(\frac {b ^ {\prime}}{B ^ {\prime}}\right) \right\}.
$$
Calibration can be performed in the same way as in the non-aggregated case. Note that we perform calibration after aggregating; therefore, JER control is ensured directly on aggregated statistics and is not a result of aggregating JER controlling families. Importantly, this approach holds without additional assumptions on the aggregation scheme  $f$ . We consider the threshold family  $\overline{\mathbf{t}}_{\alpha}^{B} = \overline{\mathbf{T}} (\lambda_{B}(\alpha))$ , where
$$
\lambda_ {B} (\alpha) = \frac {1}{B ^ {\prime}} \max  \left\{b ^ {\prime} \in [   [ B ^ {\prime} ]   ] \quad s. t. \quad \widehat {\mathrm {J E R}} _ {B} ^ {0} \left(\overline {{\mathbf {T}}} \left(\frac {b ^ {\prime}}{B ^ {\prime}}\right)\right) \leq \alpha \right\}.
$$
With  $\overline{\mathbf{T}}^0$  a template composed of  $B^{\prime}$  candidate curves that match quantiles of the distribution of  $\overline{\pi}^0$  statistics, we obtain the following result:
Theorem 3 (JER control for aggregated  $\pi$ -statistics). Consider the threshold family defined by  $\overline{\mathbf{t}}_{\alpha}^{B} = \overline{\mathbf{T}}^{0}(\lambda_{B}(\alpha))$ . Then, as  $B \to +\infty$ ,
$$
\overline {{\mathrm {J E R}}} (\bar {\mathbf {t}} _ {\alpha} ^ {B}) \leq \alpha + O _ {P} (1 / \sqrt {B}).
$$
Proof. The proof is identical to that of Theorem 2 using the empirical aggregated JER.
The calibrated aggregated threshold family yields valid FDP upper bounds via Proposition 1. The proposed KOPI (Knockoffs -  $\pi$ ) method therefore achieves FDP control on aggregated Knockoffs.
Methods considered. In our implementation of KOPI, we rely on the harmonic mean [28] as the aggregation scheme  $f$ . Additionally, we set  $k_{max} = \lfloor p / 50\rfloor$  following the approach of [3]. We also consider both state-of-the-art Knockoffs aggregation schemes: AKO (Aggregation of Multiple Knockoffs, 17) and e-values based aggregation [18]. Additionally, we consider Vanilla Knockoffs, i.e. [6] and FDP control via Closed Testing [13]. In simulated data experiments, we generate Knockoffs assuming a Gaussian distribution for  $\mathbf{X}$ , with all variables centered. For methods that support aggregation, we use  $D = 50$  Knockoff draws.
Setup. At each simulation run, we generate Gaussian data  $\mathbf{X} \in \mathbb{R}^{n \times p}$  with a Toeplitz correlation matrix corresponding to a first-order auto-regressive model with parameter  $\rho$ , i.e.  $\pmb{\Sigma}_{i,j} = \rho^{|i - j|}$ .
Then, we draw the true support  $\beta^{*} \in \{0,1\}^{p}$ . The number of non-null coefficients of  $\beta^{*}$  is controlled by the sparsity parameter  $s_{p}$ , i.e.  $s_{p} = \| \boldsymbol{\beta}^{*}\|_{0} / p$ . The target variable  $\mathbf{y}$  is built using a linear model:
$$
\mathbf {y} = \mathbf {X} \boldsymbol {\beta} ^ {*} + \sigma \boldsymbol {\epsilon},
$$
Figure 1: FDP bound coverage at level  $\alpha$  and empirical Power for 50 simulation runs and five different methods: Vanilla Knockoffs, aggregated Knockoffs using e-values, aggregated Knockoffs using quantile-aggregation, KOPI and Knockoff inference via Closed Testing. We use  $D = 50$  Knockoffs draws and the following simulation settings:  $\alpha = 0.1$ ,  $q = 0.1$ ,  $p = 500$ . Each column represents a varying parameter with the first row displaying FDP coverage and the second row displaying power. The red line and associated error bands represent the acceptable limits for FDP bound coverage. KOPI consistently outperforms all other methods while retaining FDP control.
with  $\sigma$  controlling the amplitude of the noise:  $\sigma = \| \mathbf{X}\boldsymbol{\beta}^{*}\|_{2} / (\mathrm{SNR}\|\boldsymbol{\epsilon}\|_{2})$ , SNR being the signal-to-noise ratio. We choose the central setting  $n = 500, p = 500, \rho = 0.5, s_{p} = 0.1$ ,  $\mathrm{SNR} = 2$ . For each parameter, we explore a range of possible values to benchmark the methods across varied settings.
To select variables using FDP upper bounds, we retain the largest possible set of variables  $S$  such that  $V(S) \leq q|S|$  (Algorithm 4). For each of the  $N$  simulations and each method, we compute the empirical FDP and True Positive Proportion (TPP):
$$
\widehat {F D P} (S) = \frac {| S \cap \mathcal {H} _ {0} |}{| S |} \quad \text {a n d} \quad \widehat {T P P} (S) = \frac {| S \cap \mathcal {H} _ {1} |}{| \mathcal {H} _ {1} |}.
$$
If the FDP is controlled at level  $\alpha$ ,  $|\{k \in [N] : \widehat{FDP}(S_k) > q\}| \sim \mathcal{B}(N, \alpha)$ . Then, we can compute error bands on the  $\alpha$ -level using std  $(\mathcal{B}(N, \alpha)/N) = \sqrt{\alpha(1 - \alpha)/N}$ . The second row of Fig. 1 represents the empirical power achieved by each method, which corresponds to the average of TPPs defined above for  $N$  runs i.e. Power  $= \sum_{k=1}^{N} \widehat{TPP}(S_k)/N$ . Fig. 1 shows that across all different settings, KOPI retains FDP control. We can also see that FDR control does not imply FDP control, as Vanilla Knockoffs are consistently outside of FDP bound coverage intervals. However, the two existing aggregation schemes (AKO and e-values) that formally guarantee FDR control are generally conservative and achieve FDP control empirically. This is consistent with the findings of [18]. The Closed Testing procedure of [13] achieves FDP control as announced but suffers from a lack of power.
Interestingly, KOPI achieves FDP control while offering power gains compared to FDR-controlling Knockoffs aggregation methods. Yet FDP control is a much stronger guarantee than FDR control, as discussed previously. These gains are especially noticeable in challenging inference settings where most methods exhibit a clear decrease in power or even catastrophic behavior (i.e. zero power).
Moreover, Fig. 3 (in appendix) shows that when using  $q = 0.05$  rather than  $q = 0.1$  as in Fig. 1, the robustness of KOPI with regards to difficult inference settings is even more salient. More precisely, for  $q = 0.05$ , AKO and Closed Testing are always powerless. E-values aggregation yields good power in easier settings such as  $\rho \leq 0.6$ ,  $\mathrm{SNR} \geq 2.5$  or  $n > 750$  but exhibits catastrophic behavior in harder settings. Overall, apart from KOPI, only Vanilla Knockoffs exhibit non-zero power, but this method fails to control the FDP as it is intended to control FDR. KOPI preserves FDP control in all settings while yielding superior power compared to all other methods.
KOPI Closed Testing e-values AKO Vanilla KO
Figure 2: Empirical FDP and power on semi-simulated data for 42 contrast pairs. We use 7 HCP contrasts C0: "Motor Hand", C1: "Motor Foot", C2: "Gambling", C3: "Relational", C4: "Emotion", C5: "Social", C6: "Working Memory". We consider all 42 possible train/test pairs: the train contrast is used to obtain a ground truth, while the test contrast is used to generate the response. Inference is performed using the 5 methods considered in the paper and the empirical FDP is reported in the left box plot, while power is reported in the right box plot. Notice (right figure) that KOPI yields superior power compared to all other Knockoffs-based methods while controlling the FDP (left Fig.).
5.2 Brain data application
The goal of human brain mapping is to associate cognitive tasks with relevant brain regions. This problem is tackled using functional Magnetic Resonance Imaging (fMRI), which consists in recording the blood oxygenation level dependent signal via an MRI scanner. The importance of conditional inference for this problem has been outlined in [26]. We use the Human Connectome Project (HCP900) dataset that contains brain images of healthy young adults performing different tasks while inside an MRI scanner. Details about this dataset and empirical results can be found in Appendix E.
While these results demonstrate the face validity of the approach, FDP control and power cannot be evaluated. Therefore, following [16], we consider an additional experiment that consists in using semi-simulated data. We consider a first fMRI dataset  $(\mathbf{X}_1,\mathbf{y}_1)$  on which we perform inference using a Lasso estimator; this yields  $\beta_1^* \in \mathbb{R}^p$  that we will use as our ground truth. Then, we consider a separate fMRI dataset  $(\mathbf{X}_2,\mathbf{y}_2)$  for data generation. The point of using a separate dataset is to avoid circularity between the ground truth definition and the inference procedure. Concretely, we discard the original response vector  $\mathbf{y}_2$  for this dataset and build a simulated response  $\mathbf{y}_2^{sim}$  using a linear model, with the same notation as previously (we set  $\sigma$  so that  $SNR = 4$ ):  $\mathbf{y}_2^{sim} = \mathbf{X}_2\boldsymbol{\beta}_1^* + \sigma \boldsymbol{\epsilon}$ .
Then, inference is performed using Knockoffs-based methods on  $(\mathbf{X}_2,\mathbf{y}_2^{sim})$ . Since we consider  $\beta_1^*$  as the ground truth, the FDP and TPP can be computed for each method. As can be seen in Fig. 2, KOPI is the most powerful method among those that control the FDP.
5.3 Genomic data application
In addition to the brain data application, we compared KOPI to other Knockoffs-based methods on gene-expression data [5] containing 79 samples and 90 genes. KOPI yields a non trivial selection for all runs, with 3 genes selected in  $100\%$  of all 50 runs of the experiment. Across all runs, only 8 different genes are selected by KOPI. Vanilla Knockoffs select 24 different genes across all runs and no gene exceeds a selection frequency of  $70\%$ . All other methods are powerless in all runs. Details and results of this experiment can be found in Appendix D.
In this paper, we have proposed a novel method that reaches FDP control on aggregated Knockoffs. It combines the benefits of aggregation, i.e. improving the stability of the inference, in addition to providing a probabilistic control of the FDP, rather than controlling only its expectation, the FDR.
Simulation results support that KOPI indeed controls the FDP. Furthermore, while FDP control is a stricter guarantee than FDR control, KOPI actually offers power gains compared to state-of-the-art aggregation-based Knockoffs methods. This sensitivity gain is a direct benefit from the JER approach and its adaptivity to arbitrary aggregation schemes. While the latter has been formulated and used so far in mass univariate settings [3], the present work presents a first use of this approach in the context of multiple regression. Moreover, KOPI does not require any assumption on the data at hand or on the law of Knockoff statistics under the null.
The computation time of the proposed approach is comparable to existing aggregation schemes for Knockoffs: sampling  $\pi$  statistics under the null using Algorithm 1 can be done once and for all for a given value of  $p$ . JER estimation via Algorithm 2 and calibration can be performed via binary search of complexity  $\mathcal{O}(\log (B'))$ . Finding the rejection set  $\hat{S}$  after performing calibration is done in linear time via [7]. In practice, the computation time is the same as for classical knockoff aggregation [19] and is in minutes for the brain imaging datasets considered. Avenues for future work include a theoretical analysis of the False Negative Proportion (FNP) [8] of KOPI and developing a step-down version of the method to further improve power.
We provide a Python package containing the code for KOPI available at https://github.com/alexblnn/KOPI.
7 Acknowledgments and disclosure of funding
This project was funded by a UDOPIA PhD grant from Université Paris-Saclay and also supported by the FastBig ANR project (ANR-17-CE23-0011), the KARAIB AI chair (ANR-20-CHIA-0025-01), the H2020 Research Infrastructures Grant EBRAIN-Health 101058516 and the SansSouci ANR project (ANR-16-CE40-0019). The authors thank Binh Nguyen for his precious help on the code base and Samuel Davenport for useful discussions about this work.
[1] Barber, R. F. and Candès, E. J. (2015). Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055-2085.
[2] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300.
[3] Blain, A., Thirion, B., and Neuvial, P. (2022). Notip: Non-parametric true discovery proportion control for brain imaging. NeuroImage, 260:119492.
[4] Blanchard, G., Neuvial, P., Roquain, E., et al. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics, 48(3):1281-1303.
[5] Bourgon, R., Gentleman, R., and Huber, W. (2010). Independent filtering increases detection power for high-throughput experiments. Proceedings of the National Academy of Sciences, 107(21):9546-9551.
[6] Candès, E., Fan, Y., Janson, L., and Lv, J. (2018). Panning for gold: 'model-x' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(3):551-577.
[7] Enjalbert-Courrech, N. and Neuvial, P. (2022). Powerful and interpretable control of false discoveries in two-group differential expression studies. Bioinformatics, 38(23):5214-5221.
[8] Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. Journal of the Royal Statistical Society Series B: Statistical Methodology, 64(3):499-517.
[9] Gimenez, J. R. and Zou, J. (2019). Improving the stability of the knockoff procedure: Multiple simultaneous knockoffs and entropy maximization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2184-2192. PMLR.
[10] Goeman, J. J. and Solari, A. (2011). Multiple testing for exploratory research. Statistical Science, 26(4):584-597.
[11] Janson, L. and Su, W. (2016). Familywise error rate control via knockoffs. Electronic Journal of Statistics, 10(1):960-975.
[12] Korn, E. L., Troendle, J. F., McShane, L. M., and Simon, R. (2004). Controlling the number of false discoveries: application to high-dimensional genomic data. Journal of Statistical Planning and Inference, 124(2):379-398.
[13] Li, J., Maathuis, M. H., and Goeman, J. J. (2022). Simultaneous false discovery proportion bounds via knockoffs and closed testing. arXiv preprint arXiv:2212.12822.
[14] Luo, Y., Fithian, W., and Lei, L. (2022). Improving knockoffs with conditional calibration.
[15] Meinshausen, N., Meier, L., and Buhlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association, 104(488):1671-1681.
[16] Nguyen, B. T., Thirion, B., and Arlot, S. (2022). A Conditional Randomization Test for Sparse Logistic Regression in High-Dimension. In NeurIPS 2022, volume 35 of Advances in Neural Information Processing Systems, New Orleans, United States.
[17] Nguyen, T.-B., Chevalier, J.-A., Thirion, B., and Arlot, S. (2020). Aggregation of multiple knockoffs. In International Conference on Machine Learning, pages 7283-7293. PMLR.
[18] Ren, Z. and Barber, R. F. (2022). Derandomized knockoffs: leveraging e-values for false discovery rate control. arXiv preprint arXiv:2205.15461.
[19] Ren, Z., Wei, Y., and Candès, E. (2021). Derandomizing knockoffs. Journal of the American Statistical Association, pages 1-11.
[20] Stiglic, G. and Kokol, P. (2010). Stability of ranked gene lists in large microarray analysis studies. Journal of biomedicine and biotechnology, 2010.
[21] Thirion, B., Varoquaux, G., Dohmatob, E., and Poline, J.-B. (2014). Which fMRI clustering gives good brain parcellations? Frontiers in Neuroscience, 8(167):13.
[22] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288.
[23] Van Essen, D. C., Ugurbil, K., Auerbach, E. J., Barch, D. M., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., Della Penna, S., Feinberg, D. A., Glasser, M. F., Harel, N., Heath, A. C., Larson-Prior, L. J., Marcus, D. S., Michalareas, G., Moeller, S., Oostenveld, R., Petersen, S. E., Prior, F. W., Schlaggar, B. L., Smith, S. M., Snyder, A. Z., Xu, J., and Yacoub, E. (2012). The Human Connectome Project: a data acquisition perspective. Neuroimage, 62(4):2222-2231.
[24] Vovk, V. and Wang, R. (2021). E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736-1754.
[25] Wang, R. and Ramdas, A. (2022). False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):822-852.
[26] Weichwald, S., Meyer, T., Özdenizci, O., Schölkopf, B., Ball, T., and Grosse-Wentrup, M. (2015). Causal interpretation rules for encoding and decoding models in neuroimaging. NeuroImage, 110:48-59.
[27] Weinstein, A., Su, W. J., Bogdan, M., Barber, R. F., and Candès, E. J. (2020). A power analysis for knockoffs with the lasso coefficient-difference statistic. arXiv preprint arXiv:2007.15346.
[28] Wilson, D. J. (2019). The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences, 116(4):1195-1200.