SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Université Paris-Saclay alexandre.blain@inria.fr
Bertrand Thirion
bertrand.thirion@inria.fr
Olivier Grisel
olivier.grisel@inria.fr
Pierre Neuvial
Institut de Mathématiques de Toulouse  
Université de Toulouse  
rre.neuvial@math.univ-toulouse.fr
Controlled variable selection is an important analytical step in various scientific fields, such as brain imaging or genomics. In these high-dimensional data settings, considering too many variables leads to poor models and high costs, hence the need for statistical guarantees on false positives. Knockoffs are a popular statistical tool for conditional variable selection in high dimension. However, they control for the expected proportion of false discoveries (FDR) and not their actual proportion (FDP). We present a new method, KOPI, that controls the proportion of false discoveries for Knockoff-based inference. The proposed method also relies on a new type of aggregation to address the undesirable randomness associated with classical Knockoff inference. We demonstrate FDP control and substantial power gains over existing Knockoff-based methods in various simulation settings and achieve good sensitivity/specificity tradeoffs on brain imaging and genomic data.
Statistically controlled variable selection arises in many different application fields, when the aim is to identify variables that are important for predicting an outcome of interest. For instance, in the context of brain imaging, practitioners are interested in finding which brain areas are relevant for predicting behavior or brain diseases. Such problems also appear in genomics, where practitioners wish to select genes associated with disease outcomes.
More precisely, we consider here conditional variable selection, meaning that we wish to select variables that are relevant to predict an outcome given the other variables. This type of inference is substantially more challenging than marginal inference, especially in high-dimensional settings, where the number of variables exceeds the number of samples. This is typically the case for brain mapping studies that comprise at most a few hundred subjects (hence, samples), while modern functional Magnetic Resonance Imaging (MRI) scans consist of more than 100k voxels. In the context of conditional inference, those are typically reduced to a few hundreds of brain regions, still possibly more than the number of samples.
Importantly, statistical guarantees are needed to ensure that the inference is reliable - i.e. that the proportion of false discoveries made by the variable selection procedure is controlled.
In the Knockoffs framework [1, 6], this problem is tackled by building noisy copies of the original variables. These copies are then compared to their original counterpart to perform variable selection.
The intuition underlying Knockoffs is that irrelevant variables do not get a larger weight than their Knockoff, while relevant variables do. Crucially, the Model-X Knockoffs procedure [6] controls the False Discovery Rate [2] which is the expected proportion of false discoveries.
A major caveat with this procedure is the random nature of the Knockoffs generation process: for two runs of the Knockoffs procedure on the same data, different Knockoffs will be built and subsequently different variables may be selected. This undesirable behavior hinders reproducibility. A second caveat is that False Discovery Rate (FDR) control does not imply False Discovery Proportion (FDP) control [12]. This leads to potentially unreliable inference: single runs of the method can produce a much higher proportion of False Discoveries than the chosen FDR level.
In this work, we propose a novel Knockoff-based inference procedure that addresses both concerns while offering power gains over existing methods, for no significant computation cost. The paper is organized as follows. After a refresher on Knockoff inference and aggregation, we consider the  $\pi$  statistic introduced in [17] to rank variables by relevance. Using the symmetry of knockoffs under the null hypothesis, we construct explicit upper bounds on the Joint Error Rate (JER; 4) of these statistics, leading to FDP control. We then use the calibration principle of [4] to obtain sharper bounds. Finally, we obtain a robust version of this method using harmonic mean aggregation of the  $\pi$  statistics across multiple Knockoffs draws. We demonstrate empirical power gains in various simulation settings and show the practical benefits of the proposed method for conditionally important region identification on fMRI and genomic datasets.
There has been much effort in the statistical community to achieve derandomized Knockoff-based inference. [19] introduced the idea of running Model-X Knockoffs [6] multiple times and computing for each the proportion of runs for which it was selected. [9] explore the idea of sampling multiple Knockoffs simultaneously. This induces a massive computational cost, which is prohibitive compared to methods that can support parallel computing. [17] introduced an aggregation method that relies on viewing Model-X Knockoffs as a Benjamini-Hochberg (BH) procedure [2] on so-called intermediate  $p$ -values. Such  $p$ -values can be computed on different Knockoff runs and aggregated using quantile aggregation [15] – then, BH is performed on the aggregated  $p$ -values to select variables. This approach relies on the heavy assumption that Knockoff statistics are i.i.d. under the null. Additionally, it is penalized by the conservativeness of the quantile aggregation scheme. Alternative aggregation schemes such as the harmonic mean [28] can be used but do not yield valid  $p$ -values.
[18] introduced an alternative aggregation procedure where Model-X Knockoffs are viewed as an e-BH procedure [25] on well-defined e-values [24]. Since the mean of two e-values remains an e-value, aggregation is done by averaging e-values across different Knockoffs draws. Then, e-BH is performed on the aggregated e-values to select variables. FDR control on aggregated Knockoffs is achieved without any additional assumption compared to Model-X Knockoffs. However, this method requires the difficult setting of a hyperparameter related to the chosen risk level, which highly impacts power in practice. Other recent developments in Knockoffs include the conditional calibration framework of Luo et al. [14] which aims at improving the power of Knockoffs-based methods.
There have been a few attempts at controlling other type 1 errors than the FDR using Knockoffs. [11] achieves k-FWER control and proposes that FDP control can be obtained by using a procedure that leverages joint k-FWER control. Recently, [13] introduced such a procedure to reach FDP control based on the k-FWER control introduced in [11]. In summary, the KOPI approach is the first one that aims at controlling the FDP of knockoffs-based inference for any aggregation scheme, leading to both accurate FDP control and increased sensitivity.
3 Refresher on Knockoffs
Notation. We denote vectors by bold lowercase letters. A vector  $\mathbf{x} = \{x_1, \ldots, x_p\}$  from which we removed the  $j^{th}$  coordinate is denoted by  $\mathbf{x}_{-j}$ , i.e.  $\mathbf{x} \setminus \{x_j\}$ . Independence between two random vectors  $\mathbf{x}$  and  $\mathbf{y}$  is denoted by  $\mathbf{x} \perp \mathbf{y}$ . For two vectors  $\mathbf{x}$  and  $\tilde{\mathbf{x}}$  and a subset  $S$  of indices,  $(\mathbf{x}, \tilde{\mathbf{x}})_{swap(S)}$  denotes the vector obtained from  $(\mathbf{x}, \tilde{\mathbf{x}})$  by swapping the entries  $x_j$  and  $\tilde{x}_j$  for each  $j \in S$ . Matrices are denoted by bold uppercase letters, the only exception being the vector of Knockoff statistics
that we denote by  $\mathbf{W}$  as in [1, 6]. For any set  $S$ ,  $|S|$  denotes the cardinality of  $S$ . For a vector  $\mathbf{z} = (z_j)_{1 \leq j \leq p}$  and  $S \subset [[p]]$ , we denote by  $z_{(j; S)}$  (or  $z_{(j)}$  when there is no ambiguity) the  $j^{th}$  smallest value in the sub-vector  $(\mathbf{z}_s)_{s \in S}$ . For an integer  $k$ ,  $[[k]]$  denotes the set  $\{1, \ldots, k\}$ . Equality in distribution is denoted by  $\stackrel{d}{=}$ .
Problem setup. The input data are denoted by  $\mathbf{X} \in \mathbb{R}^{n \times p}$ , where  $n$  is the number of samples and  $p$  the number of variables. The outcome of interest is denoted by  $\mathbf{y} \in \mathbb{R}^n$ . The goal is to select variables that are relevant with regards to the outcome conditionally on all others. Formally, we test simultaneously for all  $j \in [p]$ :
$$
H _ {0, j}: y \perp x _ {j} | \mathbf {x} _ {- j} \quad \text {v e r s u s} \quad H _ {1, j}: y \not \perp x _ {j} | \mathbf {x} _ {- j}.
$$
The output of a variable selection method is a rejection set  $\hat{S} \subset [[p]]$  that estimates the true unknown support  $\mathcal{H}_1 = \{j : y \not\ll x_j | \mathbf{x}_{-j}\}$ . Its complement is the set of true null hypotheses  $\mathcal{H}_0 = \{j : y \perp x_j | \mathbf{x}_{-j}\}$ . Its cardinality  $|\mathcal{H}_0|$  is denoted by  $p_0$ . To ensure reliable inference, our aim is to provide a statistical guarantee on the proportion of False Discoveries in  $\hat{S}$ . The False Discovery Proportion (FDP) and the False Discovery Rate (FDR) [2] are defined as:
$$
\operatorname {F D P} (\hat {S}) = \frac {| \hat {S} \cap \mathcal {H} _ {0} |}{| \hat {S} | \vee 1}, \quad \operatorname {F D R} (\hat {S}) = \mathbb {E} [ \operatorname {F D P} (\hat {S}) ] = \mathbb {E} \left[ \frac {| \hat {S} \cap \mathcal {H} _ {0} |}{| \hat {S} | \vee 1} \right].
$$
An  $\alpha$ -level post-hoc FDP upper bound [10] is a function  $V$  that verifies:
$$
\mathbb {P} \left(\forall S \subset [   [ p ]   ], \mathrm {F D P} (S) \leq V (S) / | S |\right) \geq 1 - \alpha .
$$
Knockoffs. The Knockoff filter is a variable selection technique introduced by [1] and refined by [6] which controls the FDR. This procedure relies on building noisy copies of the original variables called Knockoff variables, that are designed to serve as controls for variable selection.
Definition 1 (Model-X Knockoffs, 6). For the family of random variables  $\mathbf{x} = (x_{1},\ldots ,x_{p})$  Knockoffs are a new family of random variables  $\tilde{\mathbf{x}} = (\tilde{x}_1,\dots ,\tilde{x}_p)$  satisfying:
1. for any  $S\subset [[p]]$ $(\mathbf{x},\tilde{\mathbf{x}})_{swap(S)}\stackrel {d}{=}(\mathbf{x},\tilde{\mathbf{x}})$
2.  $\tilde{\mathbf{x}}\perp \mathbf{y}|\mathbf{x}$
Once we have such variables at our disposal, we quantify their importance relative to the original ones. This is done by computing Knockoff statistics  $\mathbf{W} = (W_{1},\dots ,W_{p})$  that are defined as follows.
Definition 2 (Knockoff Statistic, 6). A knockoff statistic  $\mathbf{W} = (W_{1},\dots ,W_{p})$  is a measure of feature importance that satisfies:
1.  $\mathbf{W}$  depends only on  $\mathbf{X},\tilde{\mathbf{X}}$  and  $\mathbf{y}$ :  $\mathbf{W} = g(\mathbf{X},\tilde{\mathbf{X}},\mathbf{y})$
2. Swapping column  $\mathbf{x}_j$  and its knockoff column  $\tilde{\mathbf{x}}_j$  switches the sign of  $W_{j}$
$$
W _ {j} ([ \mathbf {X}, \tilde {\mathbf {X}} ] _ {s w a p (S)}, \mathbf {y}) = \left\{ \begin{array}{l l} W _ {j} ([ \mathbf {X}, \tilde {\mathbf {X}} ], \mathbf {y}) \text {i f} j \in S ^ {c} \\ - W _ {j} ([ \mathbf {X}, \tilde {\mathbf {X}} ], \mathbf {y}) \text {i f} j \in S. \end{array} \right.
$$
The most commonly used Knockoff statistic is the Lasso-coefficient difference (LCD) [27]. This statistic is obtained by fitting a Lasso estimator [22] on  $[\mathbf{X},\tilde{\mathbf{X}}]\in \mathbb{R}^{n\times 2p}$ , which yields  $\widehat{\beta}\in \mathbb{R}^{2p}$ . Then, the Knockoff statistic can be computed using  $\widehat{\beta}$ :
$$
\forall j \in [   [ p ]   ], \quad W _ {j} = | \widehat {\beta} _ {j} | - | \widehat {\beta} _ {j + p} |.
$$
This coefficient summarizes the importance of the original  $j^{th}$  variable relative to its own Knockoff:  $W_{j} > 0$  indicates that the original variable is more more important for fitting  $y$  than the Knockoff variable, meaning that the  $j^{th}$  variable is likely relevant. Conversely,  $W_{j} < 0$  indicates that the  $j^{th}$  variable is probably irrelevant. We thus wish to select variables corresponding to large and positive  $W_{j}$ . Formally, the rejection set  $\hat{S}$  can be written  $\hat{S} = \{j : W_{j} > T_{q}\}$ , where  $T_{q}$  is chosen to provably control the FDR at level  $q$  [6].
Aggregation schemes. Due to the randomness in the knockoff generation process, different variables may be selected for two different runs of the method, which is undesirable. To mitigate this, aggregation of multiple Knockoffs runs is needed. Ren and Barber [18] introduced an aggregation scheme which relies defining Knockoffs  $e$ -values.
$$
e _ {j} = \frac {p}{1 + | \{k : W _ {k} \leq - T _ {q} \} |} 1 _ {\{W _ {j} \geq T _ {q} \}}.
$$
Such e-values can be averaged across  $D$  draws and e-BH [25] is performed for variable selection. Alternatively, [17] defines the following  $\pi$ -statistic, that quantifies the evidence against a variable:
$$
\pi_ {j} = \left\{ \begin{array}{l l} \frac {1 + | \{k : W _ {k} \leq - W _ {j} \} |}{p} \text {i f} & W _ {j} > 0 \\ 1 \quad \text {i f} \quad W _ {j} \leq 0. \end{array} \right. \tag {1}
$$
In [17]  $\pi$  statistics are treated as  $p$ -values and aggregated using quantile aggregation [15]. However, they can only be considered  $p$ -values under restrictive assumptions that are hard to check. In the next section, these statistics are used as a building block to reach FDP control. The KOPI framework does not require  $\pi$  statistics to be valid  $p$ -values.
4 Main contribution: FDP control for aggregated Knockoffs
4.1 Post hoc FDP control for  $\pi$  statistics
To obtain FDP control, we rely on Joint Error Rate control as introduced in [4]. For  $k_{max} \in [[p]]$ , we define a threshold family of size  $k_{max}$  as a vector  $\mathbf{t} = (t_j)_{j \in [[k_{max}]]}$  such that  $0 \leq t_1 \leq \dots \leq t_{k_{max}} \leq 1$ .
Definition 3 (Joint Error Rate, 4). Denote by  $\pi_{(j;\mathcal{H}_0)}$  the  $j^{th}$  smallest value  $\pi_j$  amongst all null hypotheses. The JER associated with  $\mathbf{t} = (t_j)_{j\in [[k_{max}]]}$  is:
$$
\operatorname {J E R} (\mathbf {t}) = \mathbb {P} \left(\exists j \in \llbracket k _ {\max } \wedge p _ {0} \rrbracket : \pi_ {(j: \mathcal {H} _ {0})} <   t _ {j}\right). \tag {2}
$$
The threshold family  $\mathbf{t}$  is said to control the JER at level  $\alpha$  iff  $\mathrm{JER}(\mathbf{t})\leq \alpha$
An  $\alpha$ -level FDP upper bound can be derived from JER control via the following result:
Proposition 1 (FDP control via JER control 4). If  $\mathbf{t}$  is a threshold family of length  $k_{max}$  that controls the JER at level  $\alpha$ , then,  $V^{\mathbf{t}}(S) / |S|$  is an  $\alpha$ -level FDP upper bound, with:
$$
V ^ {\mathbf {t}} (S) = \min  _ {1 \leq k \leq k _ {m a x}} (k - 1) + \sum_ {i \in S} 1 _ {\left\{\pi_ {i} > t _ {k} \right\}}. \tag {3}
$$
The proof of this result - originally included in [4] - can be found in appendix A.1 for self-containedness. In the remainder of this section, we show how to obtain JER control for  $\pi$  statistics.
4.2 Joint distribution of  $\pi$  statistics under the null
By Definition 3,  $\mathrm{JER}(\mathbf{t})$  of a given threshold family only depends on the joint null distribution of the  $\pi$  statistics. As for earlier FDR control [1] or k-FWER control [11] results, the key idea to obtain JER control for  $\pi$  statistics is to prove that the relevant part of this distribution is in fact known, thanks to the properties of knockoff statistics. We use the same notation as in [11]. Letting  $Z_{j} = |\{k\in [[p]]:W_{k}\leq -W_{j}\} |$  and  $\chi_j = sign(W_j)$ , the  $\pi$  statistics  $(\pi_j)_{j = [[p]]}$  are given by:
$$
\pi_ {j} = \frac {1 + Z _ {j}}{p} 1 _ {\{\chi_ {j} = 1 \}} + 1 _ {\{\chi_ {j} = - 1 \}}.
$$
For a given  $\mathbf{W}$ , let  $\sigma(\mathbf{W})$  be a permutation of  $[p]$  that sorts  $\mathbf{W}$  by decreasing modulus:  $\sigma(\mathbf{W}) = (\sigma_1, \ldots, \sigma_p)$  such that  $|W_{\sigma_1}| \geq |W_{\sigma_2}| \cdots \geq |W_{\sigma_p}|$ . We start by proving that the  $Z$  statistics can be expressed as a function of the vector of  $\chi$  statistics:
Lemma 1. For  $j \in [[p]]$  such that  $\chi_{\sigma_j} = 1$ ,  $Z_{\sigma_j} = \sum_{k=1}^{j-1} 1_{\{\chi_{\sigma_k} = -1\}}$ .
Proof of Lemma 1. Since  $\chi_{\sigma_j} = 1$ , we have:
$$
\begin{array}{l} Z _ {\sigma_ {j}} = \left| \left\{k \in [   [ p ]   ]: W _ {\sigma_ {k}} \leq - W _ {\sigma_ {j}} \right\} \right| \\ = \left| \left\{k \in [   [ p ]   ]: W _ {\sigma_ {k}} <   0 \text {a n d} W _ {\sigma_ {k}} \leq - W _ {\sigma_ {j}} \right\} \right| \\ = | \left\{k \in [   [ p ]   ]: W _ {\sigma_ {k}} <   0 \text {a n d} \left| W _ {\sigma_ {k}} \right| \leq \left| W _ {\sigma_ {j}} \right| \right\} | \\ = \left| \left\{k \in [   [ p ]   ]: W _ {\sigma_ {k}} <   0 \text {a n d} k \leq j \right\} \right| \\ = \sum_ {k = 1} ^ {j - 1} 1 _ {\left\{\chi_ {\sigma_ {k}} = - 1 \right\}}. \\ \end{array}
$$
Lemma 1 implies that the distribution of order statistics of  $\pi |\sigma (\mathbf{W})$  is entirely determined by that of  $\chi |\sigma (\mathbf{W})$ . To formalize this, we introduce  $\pi^0$  statistics.
Definition 4 ( $\pi^0$  statistics). Let  $\chi^0 = (\chi_j^0)_{1\leq j\leq p}$  be a collection of  $p$  i.i.d. Rademacher random variables, that is, for all  $j$ ,  $\mathbb{P}(\chi_j^0 = 1) = \mathbb{P}(\chi_j^0 = -1) = 1 / 2$ . The associated  $\pi^0$  statistics are defined for  $j\in [[p]]$  by
$$
\pi_ {j} ^ {0} = \frac {1 + Z _ {j} ^ {0}}{p} 1 _ {\left\{\chi_ {j} ^ {0} = 1 \right\}} + 1 _ {\left\{\chi_ {j} ^ {0} = - 1 \right\}}, \text {w h e r e} Z _ {j} ^ {0} = \sum_ {k = 1} ^ {j - 1} 1 _ {\left\{\chi_ {k} ^ {0} = - 1 \right\}}. \tag {4}
$$
Theorem 1. Let  $\mathbf{t}$  be a threshold family of length  $k_{max}$ . Then, for  $\pi^0 = (\pi_j^0)_{j\in [[p]]}$  as in (4),
$$
\operatorname {J E R} (\mathbf {t}) \leq \operatorname {J E R} ^ {0} (\mathbf {t}): = \mathbb {P} \left(\exists k \in \llbracket k _ {\max } \rrbracket : \pi_ {(k)} ^ {0} <   t _ {k}\right). \tag {5}
$$
Proof of Theorem 1. Let  $k \in [[k_{max}]]$ . Since  $t_k \leq 1$ , we have  $\pi_{(k:\mathcal{H}_0)} < t_k$  if and only if  $N_k \geq k$ , where
$$
N _ {k} = \left| \left\{j \in \mathcal {H} _ {0}, \chi_ {j} = 1 \text {a n d} \frac {1 + Z _ {j}}{p} <   t _ {k} \right\} \right|.
$$
With the notation of Definition 4, we define the random variable
$$
N _ {k} ^ {0} = \left| \left\{j \in \mathcal {H} _ {0}, \chi_ {j} ^ {0} = 1 \text {a n d} \frac {1 + Z _ {j} ^ {0}}{p} <   t _ {k} \right\} \right|.
$$
If  $\mathcal{H}_0 = [[p]]$ , then Lemma 1 implies that conditional on  $\sigma(\mathbf{W})$ ,  $N_k$  and  $N_k^0$  have the same distribution. Indeed, the vectors  $(W_j)_{j/\chi_j=1}$  and  $(Z_j)_{j/\chi_j=1}$  have the same ordering, and conditional on  $\sigma(\mathbf{W})$ ,  $(\chi_j)_{j \in \mathcal{H}_0}$  are jointly independent and uniformly distributed on  $\{-1,1\}$  (Lemma 2.1 in 11; 1). Using the same argument as in the proof of Lemma 3.1 in Janson and Su [11], in the case where  $\mathcal{H}_0 \subsetneq [[p]]$ , false null  $\chi_j$  will insert  $-1$ 's into the process on the nulls, implying that  $N_k$  is stochastically dominated by  $N_k^0$ . Noting that  $N_k^0 \geq k$  if and only if  $\pi_{(k)}^0 < t_k$ , we obtain that
$$
\begin{array}{l} \mathbb {P} \left(\exists k \in [   [ k _ {m a x} \wedge p _ {0} ]   ], \pi_ {(k: \mathcal {H} _ {0})} <   t _ {k} | \sigma (\mathbf {W})\right) \leq \mathbb {P} \left(\exists k \in [   [ k _ {m a x} \wedge p _ {0} ]   ], \pi_ {(k)} ^ {0} <   t _ {k}\right) \\ \leq \mathbb {P} \left(\exists k \in [   [ k _ {m a x} ], \pi_ {(k)} ^ {0} <   t _ {k}\right). \\ \end{array}
$$
Taking the expectation with respect to  $\sigma (\mathbf{W})$  yields the desired result.
Theorem 1 is related to Lemma 3.1 of Janson and Su [11] and Lemma 3.1 of Li et al. [13], that rely on the sign-flip property of Knockoff statistics under the null [1]. The interest of Theorem 1 is that the upper bound  $\mathrm{JER}^0 (\mathbf{t})$  only depends on the  $\pi^0$  statistics and the threshold family  $\mathbf{t}$ , and not on the original data. Therefore, it can be estimated with arbitrary precision for any given  $\mathbf{t}$  using Monte-Carlo simulation, as explained in the next section and described in Algorithm 1 in Supp. Mat.
4.3 Joint Error Rate control for  $\pi$  statistics via calibration
To approximate the JER upper bound derived in Theorem 1, we draw  $B$  Monte-Carlo samples using Algorithm 1. This yields a set of  $B$  vectors of  $\pi^0$  statistics denoted by  $\pi_b^0\in \mathbb{R}^p$  for each  $b\in [[B]]$ . This allows us to evaluate the empirical JER, which estimates the upper bound of interest.
Definition 5 (Empirical JER). For  $B$  vectors of  $\pi^0$  statistics and a threshold family  $\mathbf{t}$ , the empirical JER is defined as:
$$
\widehat {\mathrm {J E R}} _ {B} ^ {0} (\mathbf {t}) = \frac {1}{B} \sum_ {b = 1} ^ {B} 1 \left\{\exists k \in \llbracket k _ {\max } \rrbracket : \pi_ {b (k)} ^ {0} <   t _ {k} \right\}, \tag {6}
$$
where for each  $b\in \llbracket B\rrbracket$ $\pi_{b(1)}^{0}\leq \dots \leq \pi_{b(p)}^{0}$
Since  $\widehat{\mathrm{JER}}_B^0 (\mathbf{t})$  can be made arbitrarily close (by choosing  $B$  large enough) to  $\widehat{\mathrm{JER}}^0 (\mathbf{t})$  for any given threshold family  $\mathbf{t}$ , it remains to choose  $\mathbf{t}$  such that  $\widehat{\mathrm{JER}}^0 (\mathbf{t})\leq \alpha$  in order to ensure JER control. To this end, we consider a sorted set of candidate threshold families called a template:
Definition 6 (Template [4]). A template is a component-wise non-decreasing function  $\mathbf{T}:[0,1]\mapsto \mathbb{R}^p$  that maps a parameter  $\lambda \in [0,1]$  to a threshold family  $\mathbf{T}(\lambda)\in \mathbb{R}^p$ .
This definition is naturally extended to the case of templates containing a finite number of threshold families. The template corresponding to  $B'$  threshold families is then denoted by  $(\mathbf{T}(b'/B'))_{b' \in [B']}$ .
Once a template is specified, the calibration procedure [4] can be performed; this consists in finding the least conservative threshold family  $\mathbf{t}$  amongst the template that controls the empirical JER at level  $\alpha$ . Formally, we consider the threshold family defined  $\mathbf{t}_{\alpha}^{B} = \mathbf{T}(\lambda_{B}(\alpha))$ , where
$$
\lambda_ {B} (\alpha) = \frac {1}{B ^ {\prime}} \max  \left\{b ^ {\prime} \in [   [ B ^ {\prime} ]   ] \quad s. t. \quad \widehat {\mathrm {J E R}} _ {B} ^ {0} \left(\mathbf {T} \left(\frac {b ^ {\prime}}{B ^ {\prime}}\right)\right) \leq \alpha \right\}.
$$
As observed by Blain et al. [3], optimal power is reached when the candidate families match the shape of the distribution of the null statistics. We define a template based on the distribution of the  $\pi^0$  statistics appearing in Theorem 1. In practice, we draw  $B^{\prime}$  samples from this distribution independently from the  $B$  Monte Carlo samples to avoid circularity biases. Since a template has to be component-wise non-decreasing, i.e. the set of candidate threshold families has to be sorted, we extract empirical quantiles from these  $B^{\prime}$  sorted vectors. This yields a template  $\mathbf{T}^0$  composed of  $B^{\prime}$  candidate curves that match quantiles of the distribution of  $\pi^0$  statistics. The  $\frac{b'}{B'}$ -quantile curve defines the threshold family  $\mathbf{T}^0(b'/B')$ . We obtain the following result:
Theorem 2 (JER control for  $\pi$ -statistics). Consider the threshold family defined by  $\mathbf{t}_{\alpha}^{B} = \mathbf{T}^{0}(\lambda_{B}(\alpha))$ . Then, as  $B \to +\infty$ ,
$$
\operatorname {J E R} \left(\mathbf {t} _ {\alpha} ^ {B}\right) \leq \alpha + O _ {P} (1 / \sqrt {B}).
$$
The number  $B$  of Monte-Carlo samples in Theorem 2 can be chosen arbitrarily large to obtain JER control, leading to valid FDP bounds via Equation 3. This result is proved in Appendix A.2.
4.4 False Discovery Proportion control for aggregated Knockoffs