where the first inequality follows from definition (26) and the second inequality follows from ignoring the fact that $i \in S$. This entails

$$ \begin{aligned} \mathbb{E}(\text{FCP}(\mathcal{R}_\alpha, Y)) &\le \sum_{i \in [m]} \mathbb{E} \left( \frac{\mathbf{1}\{p_i^{(Y_{n+i})} \le \alpha s_i^{\min}/m\}}{s_i^{\min}} \right) \\ &= \sum_{i \in [m]} \mathbb{E} \left( \mathbb{E} \left[ \frac{\mathbf{1}\{p_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\mathbf{p}_{-i})/m\}}{s_i^{\min}(\mathbf{p}_{-i})} \middle| W_i \right] \right), \end{aligned} $$

by using the random vector $W = (W_i, i \in [m])$ defined in Assumption 4. Now, combining Assumption 4 (i) with Assumption 5, we have that

$$ s_i^{\min}(\mathbf{p}_{-i}) = s_i^{\min}(\Psi_i(p_i^{(Y_{n+i})}, W_i)) $$

is a nonincreasing function of $p_i^{(Y_{n+i})}$. By Assumption 4 (ii) and applying Lemma F.1 (conditionally on $W_i$), we obtain

$$ \mathbb{E} \left[ \frac{\mathbf{1}\{p_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\mathbf{p}_{-i})/m\}}{s_i^{\min}(\mathbf{p}_{-i})} \middle| W_i \right] \le \frac{\alpha}{m}, $$

Putting this back into the FCR bound implies the result.

### C.3. Examining Assumption 4

We show here that the full-calibrated and class-calibrated p-value families satisfy Assumption 4 in the iid and conditional models, respectively. For this, it is interesting to relax Assumption 1 by assuming that the score functions can use the covariates of the calibration+test samples in an exchangeable way, as suggested in Marandon et al. (2024); Gazin et al. (2023). This is useful for instance when the learning sample and the calibration+test samples are not based on the same distribution, so that the scores may be improved by using transfer learning; we refer to Gazin et al. (2023) for more details on this.

**ASSUMPTION 6.** For any $y \in \mathcal{Y}$; $S_y(\cdot)$ is of the form $S_y(\cdot; \mathcal{D}_{\text{train}}, (X_i)_{i \in [n+m]})$ for an independent training data sample $\mathcal{D}_{\text{train}}$ and is invariant by permutation of the elements of $(X_i)_{i \in [n+m]}$. In addition, the scores have no ties and the score function is regular in the sense of Assumption 1.

**p-value family $\bar{p}$** The p-value family $\bar{p}$ given by (7) satisfies Assumption 4 in the iid model.

**PROPOSITION C.2.** Let us consider a model where the variables $(X_i, Y_i)$, $i \in [n+m]$, are exchangeable conditionally on $\mathcal{D}_{\text{train}}$ and score functions satisfying Assumptions 6. Then the p-value family $\bar{p}$ given by (7) satisfies Assumption 4.
which is clearly nonincreasing in each p-value. For instance, examples include $f(p_i^{(y)}, y \in \mathcal{Y}) = \sum_{y \in \mathcal{Y}} \log(p_i^{(y)})$ (Fisher's combination) in the classification case and $f(p_i^{(y)}, y \in \mathcal{Y}) = \int_{y \in \mathbb{R}} p_i^{(y)} e^{-y^2} dy$ in the regression case.

Finally, we mention that Assumption 5 is not satisfied in our context for top-k type selection rules because of ties. Indeed, let us consider the simple case where $\mathcal{S}(\mathbf{p})$ selects the 2-smallest p-values among the p-values $p_i := p_i^{(y_0)}$, $i \in [m]$, for some pre-specified $y_0 \in \mathcal{Y}$ and $m = 3$. When the p-values are at the minimum value $1/(n+1)$, the selection rule should make a unique choice and let us say that it selects $\{1, 2\}$ when $(p_1, p_2, p_3) = (1/(n+1), 1/(n+1), 1/(n+1))$. Then, for $i = 3$, we easily obtain $s_i^{\min}(1/(n+1), 1/(n+1), 1/(n+1)) = 0$ (because $i = 3$ is not selected), while $s_i^{\min}(1/(n+1), 2/(n+1), 1/(n+1)) = 2$ (because $i = 3$ is selected). This contradicts the concordance assumption.

Nevertheless, by using a classical randomization trick for the conformal p-values Vovk et al. (2005); Bates et al. (2023); Jin and Candes (2023) the p-value family has almost surely distinct values and top-k type selection rules are concordant (under light additional assumptions), which concurs with the observation made in Benjamini and Bogomolov (2013). More specifically, for $k \in [m]$, assume $\mathcal{S}(\mathbf{p})$ selects the k-smallest $f_i = f(p_i^{(y)}, y \in \mathcal{Y})$ where $f: (0, 1)^\mathcal{Y} \to (0, 1)$ is some measurable coordinate-wise nondecreasing surjective function and that the $f_i$'s have almost surely no ties. Then $\mathcal{A}(\mathbf{p}_{-i}) = \{z \in [0, 1]^\mathcal{Y} : i \in \mathcal{S}(z, \mathbf{p}_{-i})\}$ is not empty, because $z$ such that $f(z) < \min_{j \in [m]} f_j$ always belong to it. Hence, by definition $s_i^{\min}(\mathbf{p}_{-i}) = k$ for all values of $\mathbf{p}_{-i}$, which establishes concordance. For instance, the concordance holds when using randomized conformal p-values and selecting the k-smallest $\mathcal{I}$-adjusted p-values, with $f(p_i^{(y)}, y \in \mathcal{Y}) = q_i$ being equal to $p_i^{(y_0)}$ (excluding $y_0$), $\max_{y \in [a, b]} p_i^{(y)}$ (excluding $[a, b]$) or $\min_{k \in [K]} p_i^{(k)}$ (non-trivial classification).

## Appendix D: Procedures using weighted class-calibrated p-values

Procedure InfoSP does not take into account the proportion of labels in each class in the test sample. However, in the classification case, these proportions are estimable from the data, and the estimates can aid inference.

Let $\pi_k = \sum_{i=1}^m \mathbf{1}\{Y_{n+i} = k\}/m$ the true proportion of examples with label $k$ in the test sample, $k \in [K]$, and consider the following possible estimates:

* Calibration-based estimator: $\hat{\pi}_k^{\text{cal}} = (|\mathcal{D}_{\text{cal}}^{(k)}| + 1)/(n+1) = (1 + \sum_{j \in [n]} \mathbf{1}\{Y_j = k\})/(n+1)$, $k \in [K]$;

* Storey-$\lambda$ estimator: $\hat{\pi}_k^{\text{Storey}} = (1 + \sum_{i=1}^m \mathbf{1}\{p_i^{(k)} > \lambda\})/(m(1 - \lambda))$, $k \in [K]$. It is similar to the classical estimator of true null hypotheses proportion in multiple testing (Storey, 2002).

Given the class-calibrated p-value family $\tilde{\mathbf{p}}$ and one of the estimators $\hat{\pi}_k \in \{\hat{\pi}_k^{\text{cal}}, \hat{\pi}_k^{\text{Storey}}\}$, we define the corresponding adaptive (weighted) p-value collection $\tilde{\mathbf{p}}_{\text{adapt}} = (\tilde{p}_{i, \text{adapt}}^{(k)}, k \in [K], i \in [m])$ by

$$ \tilde{p}_{i, \text{adapt}}^{(k)} = \frac{\hat{\pi}_k}{w_k} \tilde{p}_i^{(k)}, \quad k \in [K], i \in [m], \qquad (28) $$

where $(w_k, k \in [K])$ are deterministic nonnegative weights such that $\sum_{k \in [K]} w_k = 1$. The ratio-nale behind (28) is that the term $\hat{\pi}_k$ balances the false coverage errors between classes by trying to decrease p-values related to labels which do not appear much in the test sample. The weights $w_k$ are additional parameter that add flexibility, but they have to sum to one. If we use equal
PROOF. Let us first establish Assumption 4 (i) by following an argument similar to the one of Bates et al. (2023); Marandon et al. (2024). By Assumption 6, we can work on an event where the elements of the sets $A_i = \{S_{Y_k}(X_k), k \in [n]\} \cup \{S_{Y_{n+i}}(X_{n+i})\}$ are all distinct. For any $i \in [m]$, we have by (7) that for all $j \in [m]\setminus\{i\}$ and $y \in \mathcal{Y}$,

$$ \begin{aligned} \bar{p}_j^{(y)} &= \frac{1}{n+1} \left( 1 + \sum_{k=1}^{n} \mathbf{1}\{S_{Y_k}(X_k) \ge S_y(X_{n+j})\} \right) \\ &= \frac{1}{n+1} \left( \mathbf{1}\{S_{Y_{n+i}}(X_{n+i}) < S_y(X_{n+j})\} + \sum_{s \in A_i} \mathbf{1}\{s \ge S_y(X_{n+j})\} \right). \end{aligned} $$

Denoting $A_i = \{a_{i,(1)}, \dots, a_{i,(n+1)}\}$ with $a_{i,(1)} > \dots > a_{i,(n+1)}$, and noting that $S_{Y_{n+i}}(X_{n+i}) = a_{i,(\ell)}$ with $\bar{p}_i^{(y)} = \ell/(n+1)$, we may write

$$ \bar{\mathbf{p}}_{-i} := (\bar{p}_j^{(y)})_{j \in [m] \setminus \{i\}, y \in \mathcal{Y}} = \Psi(\bar{p}_i^{(Y_{n+i})}, \bar{W}_i), $$

by letting for $u \in (0, 1]$,

$$ \begin{aligned} \overline{W}_i &:= (A_i, (S_y(X_{n+j}))_{j \in [m] \setminus \{i\}, y \in \mathcal{Y}}) \\ \Psi(u, \overline{W}_i) &:= \left( \frac{1}{n+1} \left( \mathbf{1}\{a_{i,([u(n+1)])} < S_y(X_{n+j})\} + \sum_{s \in A_i} \mathbf{1}\{s \ge S_y(X_{n+j})\} \right) \right)_{j \in [m] \setminus \{i\}, y \in \mathcal{Y}}. \end{aligned} $$

Clearly, each of the elements inside $\Psi(u, \overline{W}_i)$ is nondecreasing in $u$, which gives Assumption 4 (i) (note that $\Psi$ does not depend on $i$ in this context).

Next, we establish Assumption 4 (ii). Since the variables $(X_i, Y_i)$, $i \in [n+m]$, are exchangeable and since the scores functions, the set $A_i$, and $(S_y(X_{n+j}))_{j \in [m] \setminus \{i\}, y \in \mathcal{Y}}$ are invariant by permutations of $(X_1, Y_1), \dots, (X_n, Y_n), (X_{n+i}, Y_{n+i})$ (by using Assumptions 6), we have that the random vector $(S_{Y_1}(X_1), \dots, S_{Y_n}(X_n), S_{Y_{n+i}}(X_{n+i}))$ is exchangeable conditionally on $\overline{W}_i$. Since there are no ties in the vector, it follows that $(n+1)\bar{p}_i^{(Y_{n+i})}$ (i.e., the rank of $S_{Y_{n+i}}(X_{n+i})$ in $A_i$) is uniformly distributed in $[n+1]$ conditionally on $\overline{W}_i$. Thus Assumption 4 (ii) is satisfied (note that the conditional probability is well defined thanks to the regularity condition in Assumption 6).

p-value family $\tilde{\mathbf{p}}$ The p-value family $\tilde{\mathbf{p}}$ given by (8) satisfies Assumption 4 in the conditional model.

PROPOSITION C.3. In the case $\mathcal{Y} = [K]$, let us consider score functions satisfying Assumptions 6 and a model for the variables $(X_i, Y_i)$, $i \in [n+m]$, for which $(Y_i, i \in [n+m])$ is a deterministic vector and, for each $y \in \mathcal{Y}$, the variables $(X_i)_{i \in [n+m]: Y_i=y}$, are exchangeable. Then the p-value family $\tilde{\mathbf{p}}$ given by (8) satisfies Assumption 4.

The proof is similar to the proof of Proposition C.2. We provide it below for completeness.
weights than the class-calibrated p-value is multiplied by K × ˆπ$_{k}$ which will be less than one only if ˆπ$_{k}$ < 1 /K .

Applying InfoSP with these adaptive p-values gives rise to a new procedure R InfoSP α ( ˜ p$_{adapt}$ ) that we denote by Adapt-InfoSP .

**Proposition D.1.** Consider an informative subset collection I satisfying Assumption 2, score functions satisfying Assumption 1 and the p-value collection ˜ p$_{adapt-cal}$ = ( ˜ p ( k ) i , adapt-cal , k ∈ [ K ] , i ∈ [ m ]) defined as in (28) with the calibration-based estimator ˆπ cal k = ( |D ( k ) cal | + 1) / ( n + 1) , k ∈ [ K ] . Then the corresponding Adapt-InfoSP procedure R InfoSP α ( ˜ p$_{adapt-cal}$ ) satisfies the following:

(i) in the class-conditional model,

$$
\sup_{P_{X|Y,Y}} \mathrm{FCR}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\tilde{\mathbf{p}}_{\mathrm{adapt-cal}}), P_{X|Y}, Y) \le \alpha \sum_{y \in [K]} w_y \frac{n+1}{m} \frac{\sum_{i \in [m]} \mathbf{1}\{Y_{n+i} = y\}}{\sum_{j \in [n]} \mathbf{1}\{Y_j = y\} + 1}. \quad (29)
$$

(ii) in the iid model, sup$_{P}$$_{X,Y}$ FCR( R InfoSP α ( ˜ p$_{adapt-cal}$ ) , P$_{X,Y}$ ) ≤ α , that is, Adapt-InfoSP satisfies the FCR control (15).

Proposition D.1 is proved in § E.3. The bound (29) is only sharp when the labels are generated in the same way in the calibration and test sample (which implies the correct control in (ii)), so Adapt-InfoSP should not be used if the label proportions are expected to be (very) different between calibration and test samples.

We illustrate in Figure 6 the performance of the adaptive procedures for nonnull selection and non-trivial selection, respectively, in the set-up of unbalanced classes described in § 5.2. For the adaptive versions, w$_{k}$ = 1 /K for all k ∈ [ K ], and λ = 1 / 2. We consider two settings for the class-conditional model: the test sample has class probabilities 0 . 1 , 0 . 1 , 0 . 8 (i.e., a small label shift), and class probabilities 0 . 4 , 0 . 4 , 0 . 2 (i.e., a large label shift).

The only procedure with a theoretical class-conditional FCR guarantee is InfoSP on class calibrated p-values. The adaptive procedure with ˆπ cal k , k ∈ [ K ] violates FCR control only when the label-shift is large for non-trivial selection. Interestingly, this procedure has excellent power when the label shift is small. The adaptive procedure with ˆπ Storey k , k ∈ [ K ] is a close second in this case, but when the label shift is large it is no better than InfoSP in the settings considered. The fact that the adaptive procedure with ˆπ cal k , k ∈ [ K ] tends to control the FCR (or inflate it only by a little), suggests (arguably) that for power purposes it may be reasonable to use it if the label shift is small.

In the simulations we carried out for the iid settings, Adapt-InfoSP on class-conditional p- values had worse power than InfoSP (omitted for brevity).

Appendix E: Proofs

E.1. Proof of Theorem 3.1

The proof is straightforward from the theory developed in § C. Namely, we apply Theorem C.1 by checking the two required assumptions: Assumption 4 holds for the two considered p-value collections (§ C.3); Assumption 5 holds for the considered BH( q ) selection (§ C.4). The result thus follows from the fact that by Proposition C.4 we have

$$
\mathcal{R}_{\alpha} = (C_{n+i}^{\alpha(1 \vee s_{\text{min}}^{\text{min}})/m}(\mathbf{p}))_{i \in S} = (C_{n+i}^{\alpha|\text{BH}(\mathbf{q})|/m}(\mathbf{p}))_{i \in \text{BH}(\mathbf{q})} = \mathcal{R}_{\alpha}^{\text{InfoSP}}(\mathbf{p}).
$$
PROOF. For any $i \in [m]$, we have by (8) that for all $j \in [m]\setminus\{i\}$ and $y \in \mathcal{Y}$,

$$ \begin{aligned} \tilde{p}_j^{(y)} &= \frac{1}{|\mathcal{D}_{\text{cal}}^{(y)}| + 1} \left( 1 + \sum_{k \in \mathcal{D}_{\text{cal}}^{(y)}} \mathbf{1}\{S_y(X_k) \ge S_y(X_{n+j})\} \right) \\ &= \frac{1}{n_i^{(y)}} \left( \mathbf{1}\{S_y(X_{n+i}) < S_y(X_{n+j})\} + \sum_{s \in A_i^{(y)}} \mathbf{1}\{s \ge S_y(X_{n+j})\} \right), \end{aligned} $$

by letting $n_i^{(y)} = |A_i^{(y)}|$ and $A_i^{(y)} = \{S_y(X_k), k \in \mathcal{D}_{\text{cal}}^{(y)}\} \cup \{S_y(X_{n+i})\}$ whose elements can be assumed to be all distinct by Assumptions 6 (strictly, this is only true for labels $y \in \mathcal{Y}$ that appears at least once in the fixed sample $(Y_i, i \in [n+m])$, but the labels $y$ not appearing in $(Y_i, i \in [n+m])$ can be trivially handled because they correspond to $p$-values all equal to 1). Denoting $A_i^{(y)} = \{a_{i,(1)}^{(y)}, \dots, a_{i,(n_i^{(y)})}^{(y)}\}$ with $a_{i,(1)}^{(y)} > \dots > a_{i,(n_i^{(y)})}^{(y)}$, and noting that $S_y(X_{n+i}) = a_{i,(n_i^{(y)}\tilde{p}_i^{(y)})}^{(y)}$, we may write

$$ \tilde{\mathbf{p}}_{-i} := (\tilde{p}_j^{(y)})_{j \in [m] \setminus \{i\}, y \in \mathcal{Y}} = \Psi_i(\tilde{p}_i^{(Y_{n+i})}, \tilde{W}_i), $$

by letting

$$ \tilde{W}_i := ((A_i^{(Y_{n+i})}, (S_{Y_{n+i}}(X_{n+j}))_{j \in [m] \setminus \{i\}}); (X_j, Y_j)_{j \in [n+m]: Y_j \neq Y_{n+i}}) \quad (27) $$

and for $u \in (0,1]$, $\Psi_i(u, \tilde{W}_i) := (\Psi_i^{(y)}(u, \tilde{W}_i))_{y \in \mathcal{Y}}$ where

$$ \Psi_i^{(y)}(u, \tilde{W}_i) := \begin{cases} \left( \frac{1}{n_i^{(y)}} \left( \mathbf{1}\{a_{i,(\lceil un_i^{(y)} \rceil)} < S_y(X_{n+j})\} + \sum_{s \in A_i^{(y)}} \mathbf{1}\{s \ge S_y(X_{n+j})\} \right) \right)_{j \in [m] \setminus \{i\}} & \text{if } y = Y_{n+i}; \\ (\tilde{p}_j^{(y)})_{j \in [m] \setminus \{i\}} & \text{if } y \neq Y_{n+i}. \end{cases} $$

Clearly, the elements inside $\Psi_i(u, \tilde{W}_i)$ are nondecreasing in $u$ which gives Assumption 4 (i).

Next, to establish Assumption 4 (ii), we use that by assumption the vector

$$ (S_{Y_{n+i}}(X_{n+i}), S_{Y_{n+i}}(X_k), k \in \mathcal{D}_{\text{cal}}^{(Y_{n+i})}) $$

is exchangeable conditionally on $\tilde{W}_i$ (by permutation invariance of $\tilde{W}_i$, which also comes from Assumptions 6). Since there are no ties in the vector, it follows that $n_i^{(Y_{n+i})} \tilde{p}_i^{(Y_{n+i})}$ (i.e., the rank of $S_{Y_{n+i}}(X_{n+i})$ in $A_i^{(Y_{n+i})}$) is uniformly distributed in $[n_i^{(Y_{n+i})}]$ conditionally on $\tilde{W}_i$. Thus Assumption 4 (ii) is satisfied.

### C.4. Examining Assumption 5

PROPOSITION C.4. Assumption 5 holds for the informative selection rule $\mathcal{S}(\mathbf{p}) = \text{BH}(\mathbf{q})$ with $s_i^{\min}(\mathbf{p}_{-i}) = |\mathcal{S}(\mathbf{p})|$ whenever $i \in \mathcal{S}(\mathbf{p})$.

Note that the above result is also true for selection rule of the type $\mathcal{S}(\mathbf{p}) = \{i \in [m] : q_i \le \tau\}$ for some fixed threshold $\tau$.
<figure><img src="image_9.png" /><figcaption>Fig 6: Selecting informative prediction sets in the class-conditional setting. FCR (left column), resolution-adjusted power (middle column), and the expected fraction of covering prediction sets (right column) versus SNR in a classification setting where prediction sets excluding a null class are of interest (top two rows) and when prediction sets excluding a trivial class are of interest (bottom two rows). The class probabilities in the calibration sample are 0.15, 0.1, and 0.75; in the test sample, we have a small label shift (rows 1 and 3) and a large label shift (rows 2 and 4). The number of data generations was 2000, and *n* = *m* = 500.</figcaption></figure>
PROOF. First, a classical property of BH procedure is the following leave-one-out property: for all $i \in [m]$, $i \in \text{BH}(\mathbf{q})$ if and only if $\text{BH}(\mathbf{q}) = \text{BH}(\mathbf{q}^{0,i})$ where $\mathbf{q}^{0,i}$ is the vector $\mathbf{q}$ where the $i$-th coordinate has been replaced by 0, see for instance Ferreira and Zwinderman (2006); Sarkar (2008); Roquain and Villers (2011); Ramdas et al. (2019). This implies

$$s_i^{\min}(\mathbf{p}_{-i}) = \min_{z \in [0,1]^Y : i \in S(z, \mathbf{p}_{-i})} |S(z, \mathbf{p}_{-i})| = |\text{BH}(\mathbf{q}^{0,i})|.$$

Hence, the result is proved as soon as $\mathbf{q}$ is proved to be coordinate-wise nondecreasing in each $p$-value trajectory. This holds by Lemma C.5.

LEMMA C.5. Suppose Assumption 2, then, almost surely, $\mathbf{q}$ defined by (11) is a non-decreasing function of each $p$-value $p_i^{(y)}$, $i \in [m]$ and $y \in \mathcal{Y}$.

PROOF. Let $\mathbf{p}$ and $\mathbf{p}'$ be two $p$-value collections with $p_i^{(y)} \le p_i^{(y)}$, for all $i \in [m]$ and $y \in \mathcal{Y}$, with corresponding values $\mathbf{q}$ and $\mathbf{q}'$. Let $i \in [m]$. By definition, $C_{n+i}^{q_i}(\mathbf{p}') \in \mathcal{I}$ and $C_{n+i}^{q_i}(\mathbf{p}) \subset C_{n+i}^{q_i}(\mathbf{p}')$. By Assumption 2 (i) (iii), we have $C_{n+i}^{q_i}(\mathbf{p}) \in \mathcal{I}$, which in turn implies $q_i \le q_i'$ by definition of $q_i$.

D. Procedures using weighted class-calibrated $p$-values

Procedure InfoSP does not take into account the proportion of labels in each class in the test sample. However, in the classification case, these proportions are estimable from the data, and the estimates can aid inference.

Let $\pi_k = \sum_{i=1}^m \mathbf{1}\{Y_{n+i} = k\}/m$ the true proportion of examples with label $k$ in the test sample, $k \in [K]$, and consider the following possible estimates:

* Calibration-based estimator: $\hat{\pi}_k^{\text{cal}} = (|\mathcal{D}_{\text{cal}}^{(k)}|+1)/(n+1) = (1+\sum_{j \in [n]} \mathbf{1}\{Y_j = k\})/(n+1)$, $k \in [K]$;

* Storey-$\lambda$ estimator: $\hat{\pi}_k^{\text{Storey}} = (1+\sum_{i=1}^m \mathbf{1}\{p_i^{(k)} > \lambda\})/(m(1-\lambda))$, $k \in [K]$. It is similar to the classical estimator of true null hypotheses proportion in multiple testing (Storey, 2002).

Given the class-calibrated $p$-value family $\tilde{\mathbf{p}}$ and one of the estimators $\hat{\pi}_k \in \{\hat{\pi}_k^{\text{cal}}, \hat{\pi}_k^{\text{Storey}}\}$, we define the corresponding adaptive (weighted) $p$-value collection $\tilde{\mathbf{p}}_{\text{adapt}} = (\tilde{p}_{i,\text{adapt}}^{(k)}, k \in [K], i \in [m]}$ by

$$\tilde{p}_{i,\text{adapt}}^{(k)} = \frac{\hat{\pi}_k}{w_k} \tilde{p}_i^{(k)}, \quad k \in [K], i \in [m], \qquad (28)$$

where $(w_k, k \in [K])$ are deterministic nonnegative weights such that $\sum_{k \in [K]} w_k = 1$. The rationale behind (28) is that the term $\hat{\pi}_k$ balances the false coverage errors between classes by trying to decrease $p$-values related to labels which do not appear much in the test sample. The weights $w_k$ are additional parameter that add flexibility, but they have to sum to one. If we use equal weights than the class-calibrated $p$-value is multiplied by $K \times \hat{\pi}_k$ which will be less than one only if $\hat{\pi}_k < 1/K$.
## E.2. Proof of Theorem 3.3

The proof is a consequence of Lemma H.5 applied with the FCR criterion: condition (i) in Lemma H.5 is satisfied from Theorem 3.1 (which is true more broadly in the case of exchangeable samples, see Theorem C.1 and Proposition C.2); condition (ii) in Lemma H.5 follows from the assumed permutation preserving property of $S^{(0)}$. Hence, the conclusion of Lemma H.5 applies which gives the FCR control of InfoSCOP.

Finally, the FDR control is a consequence of the FCR control by applying Lemma 2.1.

## E.3. Proof of Proposition D.1

Let us first prove (i) by considering the class-conditional model. We follow the proof of Theorem C.1 (see § C.2) and we use that the p-value collection $\tilde{\mathbf{p}}$ satisfies Assumption 4 (see Proposition C.3), where the probability in the super-uniform property (25) holds in the class-conditional model with $\tilde{W}_i$, $i \in [m]$ given by (27). We also use that the informative selection rule $S(\cdot)$ satisfies Assumption 5 (see Proposition C.4). Hence, following the same approach as in § C.2, and denoting $\tilde{\mathbf{p}}_{\text{cal}} := \tilde{\mathbf{p}}_{\text{adapt-cal}}$ and $\tilde{\mathbf{p}}_{-i, \text{cal}} := (\tilde{p}_{j, \text{cal}}^{(y)})_{j \neq i, y \in \mathcal{Y}}$, we obtain

$$ \begin{aligned} \text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{cal}}), P_{X|Y}, Y) &\le \sum_{i \in [m]} \mathbb{E}_{P_{X|Y}} \left( \mathbb{E} \left[ \frac{\mathbf{1}\{\tilde{p}_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})/m\}}{s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})} \middle| \tilde{W}_i \right] \right) \\ &= \sum_{i \in [m]} \mathbb{E}_{P_{X|Y}} \left( \mathbb{E} \left[ \frac{\mathbf{1}\{\tilde{p}_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})w_{Y_{n+i}}/(m\hat{\pi}_{Y_{n+i}}^{\text{cal}})\}}{s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})} \middle| \tilde{W}_i \right] \right). \end{aligned} $$

Using now the super-uniform property (25), the fact that $\tilde{\mathbf{p}}_{-i, \text{cal}} = (\frac{\hat{\pi}_k}{w_k} \tilde{p}_j^{(k)})_{k \neq i, y \in \mathcal{Y}} = \Phi(Y, \tilde{\mathbf{p}}_{-i})$, with $\Phi(Y, \cdot)$ coordinate-wise nondecreasing, and Assumption 4 (i) entail

$$ s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}}) = s_i^{\min}(\tilde{\mathbf{p}}_{-i}) = s_i^{\min}(\Phi(Y, \Psi_i(p_i^{(Y_{n+i})}, W_i))). $$

Hence, $s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})$ can be written as a function $g(p_i^{(Y_{n+i})})$, with $g: u \mapsto s_i^{\min}(\Phi(Y, \Psi_i(u, W_i)))$ nonincreasing and only depending on $Y$ and $W_i$. Applying Lemma H.1 for $c = w_{Y_{n+i}}/(m\hat{\pi}_{Y_{n+i}}^{\text{cal}})$, we obtain

$$ \mathbb{E}_{P_{X|Y}} \left[ \frac{\mathbf{1}\{\tilde{p}_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})w_{Y_{n+i}}/(m\hat{\pi}_{Y_{n+i}}^{\text{cal}})\}}{s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})} \middle| \tilde{W}_i \right] \le \alpha \frac{w_{Y_{n+i}}}{m\hat{\pi}_{Y_{n+i}}^{\text{cal}}}. $$

As a consequence, we derive

$$ \text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{cal}}), P_{X|Y}, Y) \le \alpha \sum_{k \in [K]} w_k \left( \frac{\sum_{i \in [m]} \mathbf{1}\{Y_{n+i} = k\}}{m} \frac{1}{\hat{\pi}_k^{\text{cal}}} \right), $$
Applying InfoSP with these adaptive p-values gives rise to a new procedure $\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{adapt}})$ that we denote by Adapt-InfoSP.

**PROPOSITION D.1.** Consider an informative subset collection $\mathcal{I}$ satisfying Assumption 2, score functions satisfying Assumption 1 and the p-value collection $\tilde{\mathbf{p}}_{\text{adapt-cal}} = (\tilde{p}_{i,\text{adapt-cal}}^{(k)}, k \in [K], i \in [m]}$ defined as in (28) with the calibration-based estimator $\hat{\pi}_{k}^{\text{cal}} = (|\mathcal{D}_{\text{cal}}^{(k)}|+1)/(n+1), k \in [K]$. Then the corresponding Adapt-InfoSP procedure $\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{adapt-cal}})$ satisfies the following:

(i) in the class-conditional model,

$$ \sup_{P_{X|Y}, Y} \text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{adapt-cal}}), P_{X|Y}, Y) \le \alpha \sum_{y \in [K]} w_y \frac{n+1}{m} \frac{\sum_{i \in [m]} \mathbf{1}\{Y_{n+i} = y\}}{\sum_{j \in [n]} \mathbf{1}\{Y_j = y\} + 1}. \quad (29) $$

(ii) in the iid model, $\sup_{P_{X,Y}} \text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{adapt-cal}}), P_{X,Y}) \le \alpha$, that is, Adapt-InfoSP satisfies the FCR control (15).

Proposition D.1 is proved in § E.3. The bound (29) is only sharp when the labels are generated in the same way in the calibration and test sample (which implies the correct control in (ii)), so Adapt-InfoSP should not be used if the label proportions are expected to be (very) different between calibration and test samples.

We illustrate in Figure 6 the performance of the adaptive procedures for nonnull selection and non-trivial selection, respectively, in the set-up of unbalanced classes described in § 5.2. For the adaptive versions, $w_k = 1/K$ for all $k \in [K]$, and $\lambda = 1/2$. We consider two settings for the class-conditional model: the test sample has class probabilities 0.1,0.1, 0.8 (i.e., a small label shift), and class probabilities 0.4,0.4,0.2 (i.e., a large label shift).

The only procedure with a theoretical class-conditional FCR guarantee is InfoSP on class calibrated p-values. The adaptive procedure with $\hat{\pi}_{k}^{\text{cal}}, k \in [K]$ violates FCR control only when the label-shift is large for non-trivial selection. Interestingly, this procedure has excellent power when the label shift is small. The adaptive procedure with $\hat{\pi}_{k}^{\text{Storey}}, k \in [K]$ is a close second in this case, but when the label shift is large it is no better than InfoSP in the settings considered. The fact that the adaptive procedure with $\hat{\pi}_{k}^{\text{cal}}, k \in [K]$ tends to control the FCR (or inflate it only by a little), suggests (arguably) that for power purposes it may be reasonable to use it if the label shift is small.

In the simulations we carried out for the iid settings, Adapt-InfoSP on class-conditional p-values had worse power than InfoSP (omitted for brevity).

## E. Proofs

### E.1. Proof of Theorem 3.1

The proof is straightforward from the theory developed in § C. Namely, we apply Theorem C.1 by checking the two required assumptions: Assumption 4 holds for the two considered p-value collections (§ C.3); Assumption 5 holds for the considered $\text{BH}(\mathbf{q})$ selection (§ C.4).
which proves (i). We deduce (ii) by a simple integration:

$$
\begin{align*}
\text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{cal}}), P_{X,Y}) &\le \alpha \sum_{k \in [K]} w_k \mathbb{E} \left( \frac{\sum_{i \in [m]} \mathbf{1}\{Y_{n+i} = k\} n + 1}{1 + \sum_{j \in [m]} \mathbf{1}\{Y_j = k\} m} \right) \\
&= \alpha \sum_{k \in [K]} w_k(n+1) \mathbb{P}(Y_1 = k) \mathbb{E} \left( \frac{1}{1 + \sum_{j \in [m]} \mathbf{1}\{Y_j = k\}} \right) \\
&\le \alpha \sum_{k \in [K]} w_k(n+1) \mathbb{P}(Y_1 = k) \frac{1}{(n+1)\mathbb{P}(Y_1 = k)} = \alpha \sum_{k \in [K]} w_k = \alpha,
\end{align*}
$$

by using Lemma H.3 for the last inequality.

E.4. Proof of Proposition 5.1

First define $R_j$ the rank of $S_{Y_j}(X_j)$ in $\{S_k(X_j), k \in [K]\}$ (ordered in increasing order) for $j \in [n+m]$, and consider the slightly smaller conformal $p$-values

$$
\tilde{p}_i^{(k)} = \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{R_j > 1\} \mathbf{1}\{S_{Y_j}(X_j) \ge S_k(X_{n+i})\} \right) \le \bar{p}_i^{(k)}, \quad i \in [m], k \in [K], \quad (30)
$$

which means that the calibration is only made with examples having a label not minimizing
the score function. The rationale behind using this *p*-value rather than $\bar{p}_i^{(k)}$ is that, due to the
post-processing, the elements $X_{n+i}$ of the test sample cannot produce an error provided that
$R_{n+i} = 1$ so that we can restrict the test sample to those with $R_{n+i} > 1$ when computing the
FCR.

Assume $K = 2$. We first prove that $\mathcal{R}_{\alpha}^{\text{InfoSP}}(\bar{\mathbf{p}}) = \mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}})$. It is enough to prove that the adjusted $p$-values $q_i$ obtained from $\bar{\mathbf{p}}$ and $\tilde{\mathbf{p}}$ are the same for this non-trivial selection (since for the selected $i$, the procedure always chooses argmin$_{k \in [K]} S_k(X_{n+i})$ due to the post-processing). Letting $S_{\min}(x) = \min_{k \in [K]} S_k(x)$, $S_{\max}(x) = \max_{k \in [K]} S_k(x)$, this comes from

$$
\begin{align*}
\min_{k \in [K]} \{\tilde{p}_i^{(k)}\} &= \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{R_j > 1\} \mathbf{1}\{S_{Y_j}(X_j) \ge S_{\max}(X_{n+i})\} \right) \\
&= \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) \ge S_{\max}(X_{n+i})\} \right) = \min_{k \in [K]} \{\bar{p}_i^{(k)}\},
\end{align*}
$$

where the second equality holds because $S_{Y_j}(X_j) \ge S_{\max}(X_{n+i})$ is impossible for $R_j = 1$, that is, when $S_{Y_j}(X_j)$ is a minimum ($S_{\min}(X_j) < 1/2 < S_{\max}(X_j)$ almost surely by the assumptions on the score function).

Now prove Proposition 5.1, by showing the equality for $\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}})$ and by carefully modifying
the proof of Theorem C.1 (§ C.2) in order to get only equalities, by using that we consider the
case of the non-trivial selection, that is, $S = \text{BH}(\mathbf{q})$, with $q_i = \min_{k \in [K]} \tilde{p}_i^{(k)}$. By definition of the
FCP (2), we have (remember also that the prediction set always includes $\arg \min_{k \in [K]} S_k(X_{n+i})$
## E.2. Proof of Theorem 3.3

The proof is a consequence of Lemma F.4 applied with the FCR criterion: condition (i) in Lemma F.4 is satisfied from Theorem 3.1 (which is true more broadly in the case of exchangeable samples, see Theorem C.1 and Proposition C.2); condition (ii) in Lemma F.4 follows from the assumed permutation preserving property of $S^{(0)}$. Hence, the conclusion of Lemma F.4 applies which gives the FCR control of InfoSCOP.

Finally, the FDR control is a consequence of the FCR control by applying Lemma 2.1.

## E.3. Proof of Proposition D.1

Let us first prove (i) by considering the class-conditional model. We follow the proof of Theorem C.1 (see § C.2) and we use that the p-value collection $\tilde{p}$ satisfies Assumption 4 (see Proposition C.3), where the probability in the super-uniform property (25) holds in the class-conditional model with $\tilde{W}_i$, $i \in [m]$ given by (27). We also use that the informative selection rule $S(\cdot)$ satisfies Assumption 5 (see Proposition C.4). Hence, following the same approach as in § C.2, and denoting $\tilde{\mathbf{p}}_{\text{cal}} := \tilde{\mathbf{p}}_{\text{adapt-cal}}$ and $\tilde{\mathbf{p}}_{-i, \text{cal}} := (\tilde{p}_{j, \text{cal}}^{(y)})_{j \neq i, y \in \mathcal{Y}}$, we obtain

$$ \begin{aligned} \text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{cal}}), P_{X|Y}, Y) &\le \sum_{i \in [m]} \mathbb{E}_{P_{X|Y}} \left( \mathbb{E} \left[ \frac{\mathbf{1}\{\tilde{p}_{i, \text{cal}}^{(Y_{n+i})} \le \alpha s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})/m\}}{s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})} \middle| \tilde{W}_i \right] \right) \\ &= \sum_{i \in [m]} \mathbb{E}_{P_{X|Y}} \left( \mathbb{E} \left[ \frac{\mathbf{1}\{\tilde{p}_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}}) w_{Y_{n+i}} / (m \hat{\pi}_{Y_{n+i}}^{\text{cal}})\}}{s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})} \middle| \tilde{W}_i \right] \right). \end{aligned} $$

Using now the super-uniform property (25), the fact that $\tilde{\mathbf{p}}_{-i, \text{cal}} = (\frac{\hat{\pi}_k}{w_k} \tilde{p}_j^{(k)})_{k \neq i, y \in \mathcal{Y}} = \Phi(Y, \tilde{\mathbf{p}}_{-i})$, with $\Phi(Y, \cdot)$ coordinate-wise nondecreasing, and Assumption 4 (i) entail

$$ s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}}) = s_i^{\min}(\tilde{\mathbf{p}}_{-i}) = s_i^{\min}(\Phi(Y, \Psi_i(p_i^{(Y_{n+i})}, W_i))). $$

Hence, $s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})$ can be written as a function $g(p_i^{(Y_{n+i})})$, with $g: u \mapsto s_i^{\min}(\Phi(Y, \Psi_i(u, W_i)))$ nonincreasing and only depending on $Y$ and $W_i$. Applying Lemma F.1 for $c = w_{Y_{n+i}} / (m \hat{\pi}_{Y_{n+i}}^{\text{cal}})$, we obtain

$$ \mathbb{E}_{P_{X|Y}} \left[ \frac{\mathbf{1}\{\tilde{p}_i^{(Y_{n+i})} \le \alpha s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}}) w_{Y_{n+i}} / (m \hat{\pi}_{Y_{n+i}}^{\text{cal}})\}}{s_i^{\min}(\tilde{\mathbf{p}}_{-i, \text{cal}})} \middle| \tilde{W}_i \right] \le \alpha \frac{w_{Y_{n+i}}}{m \hat{\pi}_{Y_{n+i}}^{\text{cal}}}. $$

As a consequence, we derive

$$ \text{FCR}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\tilde{\mathbf{p}}_{\text{cal}}), P_{X|Y}, Y) \le \alpha \sum_{k \in [K]} w_k \left( \frac{\sum_{i \in [m]} \mathbf{1}\{Y_{n+i} = k\}}{m} \frac{1}{\hat{\pi}_k^{\text{cal}}} \right), $$
due to the post-processing)

$$
\begin{align*}
\mathrm{FCP}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\check{\mathbf{p}}), Y) &= \frac{\sum_{i \in \mathcal{S}} \mathbf{1}\{Y_{n+i} \notin C_{n+i}^{\alpha |\mathcal{S}|/m}(\check{\mathbf{p}})\}}{1 \lor |\mathcal{S}|} \\
&= \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\} \frac{\mathbf{1}\{i \in \mathcal{S}, \check{p}_i^{(Y_{n+i})} \le \alpha |\mathcal{S}|/m\}}{1 \lor |\mathcal{S}|},
\end{align*}
$$

because no error can occur when $R_{n+i} = 1$. Now, since $\mathcal{S} = \mathrm{BH}(\mathbf{q})$ and $q_i \le \check{p}_i^{(Y_{n+i})}$, $\mathbf{1}\{i \in \mathcal{S}, \check{p}_i^{(Y_{n+i})} \le \alpha |\mathcal{S}|/m\} = \mathbf{1}\{\check{p}_i^{(Y_{n+i})} \le \alpha |\mathcal{S}|/m\}$. This entails

$$
\mathrm{FCP}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\check{\mathbf{p}}), Y) = \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\} \frac{\mathbf{1}\{\check{p}_i^{(Y_{n+i})} \le \alpha |\mathcal{S}|/m\}}{1 \lor |\mathcal{S}|}.
$$

Let $\xi = (\mathbf{1}\{R_j > 1\})_{j \in [n+m]}$ and now prove

$$
\mathbb{E}[\mathrm{FCP}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\check{\mathbf{p}}), Y) | \xi] = \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\} \mathbb{E} \left[ \frac{[(n+1)\alpha K_i / m] / (n' + 1)}{K_i} \middle| \xi \right], \quad (31)
$$

with $n' = \sum_{j \in [n]} \mathbf{1}\{R_j > 1\}$ for some random variables $K_i \ge 1$, $i \in [m]$. This implies the result because the last display is at most

$$
\alpha \frac{(n + 1) \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\}}{m(\sum_{j \in [n]} \mathbf{1}\{R_j > 1\} + 1)}
$$

(with equality if $(n+1)\alpha/m$ is an integer). By Lemma H.3, the expectation of the latter is equal to $\alpha(1 - (1 - \mathbb{P}(R_1 > 1))^{n+1})$.

Let us now prove (31). For this, fix $i \in [m]$ with $R_{n+i} > 1$ and note that for all $j \in [m]$, $j \neq i$,

$$
\begin{align*}
q_j &= \frac{1}{n+1} \left( 1 + \sum_{k=1}^{n} \mathbf{1}\{R_k > 1\} \mathbf{1}\{S_{Y_k}(X_k) \ge S_{\max}(X_{n+j})\} \right) \\
&= \frac{1}{n+1} \left( \mathbf{1}\{S_{Y_{n+i}}(X_{n+i}) < S_{\max}(X_{n+j})\} + \sum_{s \in A_i} \mathbf{1}\{s \ge S_{\max}(X_{n+j})\} \right) \\
&= \frac{1}{n+1} \left( \mathbf{1}\{S_{\max}(X_{n+i}) < S_{\max}(X_{n+j})\} + \sum_{s \in A_i} \mathbf{1}\{s \ge S_{\max}(X_{n+j})\} \right) \\
&\ge \frac{1}{n+1} \sum_{s \in A_i} \mathbf{1}\{s \ge S_{\max}(X_{n+j})\} =: q'_j,
\end{align*}
$$

by letting $A_i = \{S_{Y_k}(X_j), j \in [n] : R_k > 1\} \cup \{S_{Y_{n+i}}(X_{n+i})\}$ (all distinct almost surely). The third equality above is true because $K=2$ and $R_{n+i} > 1$ and thus $S_{Y_{n+i}}(X_{n+i}) = S_{\max}(X_{n+i})$. We apply now Lemma H.4 because $\mathbf{q} = (q_j, 1 \le j \le m)$ and $\mathbf{q}' = (q'_j, 1 \le j \le m)$ (defined as above and with $q'_i = 1/(n+1)$) satisfy (32). Indeed, if $q_j > q_i$ for $j \ne i$, then $S_{\max}(X_{n+j}) \le S_{\max}(X_{n+i})$ and $q_j = q'_j$. Hence, we obtain, by letting $K_i = |\mathrm{BH}(\mathbf{q}')|$ (note that $\mathbf{q}'$ depends on $i$)

$$
\{q_i \le \alpha |\mathrm{BH}(\mathbf{q}')|/m\} = \{q_i \le \alpha K_i/m\} \subset \{|\mathrm{BH}(\mathbf{q}')| = K_i\}.
$$

In addition, $\mathbf{q}' = (q'_j, 1 \le j \le m)$ is a vector measurable w.r.t. the variable $W_i = (A_i, (S_{\max}(X_{n+j}))_{j \in [m]\setminus\{i\}})$.
Now, we use that by exchangeability of the elements of $A_i$ conditionally on $\xi$, $W_i$, and $R_{n+i} > 1$)
(with no ties),

$$
\mathbb{P}(\check{p}_i^{(Y_{n+i})} \le t | W_i, \xi, R_{n+i} > 1) = \frac{[(n+1)t]}{n' + 1}.
$$
which proves (i). We deduce (ii) by a simple integration:

$$
\begin{align*}
\mathrm{FCR}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\tilde{\mathbf{p}}_{\mathrm{cal}}), P_{X,Y}) &\le \alpha \sum_{k \in [K]} w_k \mathbb{E} \left( \frac{\sum_{i \in [m]} \mathbf{1}\{Y_{n+i} = k\}}{1 + \sum_{j \in [m]} \mathbf{1}\{Y_j = k\}} \frac{n+1}{m} \right) \\
&= \alpha \sum_{k \in [K]} w_k (n+1) \mathbb{P}(Y_1 = k) \mathbb{E} \left( \frac{1}{1 + \sum_{j \in [m]} \mathbf{1}\{Y_j = k\}} \right) \\
&\le \alpha \sum_{k \in [K]} w_k (n+1) \mathbb{P}(Y_1 = k) \frac{1}{(n+1)\mathbb{P}(Y_1 = k)} = \alpha \sum_{k \in [K]} w_k = \alpha,
\end{align*}
$$

by using Lemma F.2 for the last inequality.

E.4. Proof of Proposition 5.1

First define $R_j$ the rank of $S_{Y_j}(X_j)$ in $\{S_k(X_j), k \in [K]\}$ (ordered in increasing order) for $j \in [n+m]$, and consider the slightly smaller conformal $p$-values

$$
\tilde{p}_i^{(k)} = \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{R_j > 1\} \mathbf{1}\{S_{Y_j}(X_j) \ge S_k(X_{n+i})\} \right) \le \bar{p}_i^{(k)}, \quad i \in [m], k \in [K], \tag{30}
$$

which means that the calibration is only made with examples having a label not min-
imizing the score function. The rationale behind using this *p*-value rather than $\bar{p}_i^{(k)}$ is
that, due to the post-processing, the elements $X_{n+i}$ of the test sample cannot produce
an error provided that $R_{n+i} = 1$ so that we can restrict the test sample to those with
$R_{n+i} > 1$ when computing the FCR.

Assume $K=2$. We first prove that $\mathcal{R}_{\alpha}^{\text{InfoSP}}(\bar{\mathbf{p}}) = \mathcal{R}_{\alpha}^{\text{InfoSP}}(\check{\mathbf{p}})$. It is enough to prove that the adjusted $p$-values $q_i$ obtained from $\bar{\mathbf{p}}$ and $\check{\mathbf{p}}$ are the same for this non-trivial selection (since for the selected $i$, the procedure always chooses $\arg\min_{k \in [K]} S_k(X_{n+i})$ due to the post-processing). Letting $S_{\min}(x) = \min_{k \in [K]} S_k(x)$, $S_{\max}(x) = \max_{k \in [K]} S_k(x)$, this comes from

$$
\begin{align*}
\min_{k \in [K]} \{\tilde{p}_i^{(k)}\} &= \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{R_j > 1\} \mathbf{1}\{S_{Y_j}(X_j) \ge S_{\max}(X_{n+i})\} \right) \\
&= \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) \ge S_{\max}(X_{n+i})\} \right) = \min_{k \in [K]} \{\bar{p}_i^{(k)}\},
\end{align*}
$$

where the second equality holds because $S_{Y_j}(X_j) \ge S_{\max}(X_{n+i})$ is impossible for $R_j = 1$,
that is, when $S_{Y_j}(X_j)$ is a minimum ($S_{\min}(X_j) < 1/2 < S_{\max}(X_j)$ almost surely by the
assumptions on the score function).

Now prove Proposition 5.1, by showing the equality for $\mathcal{R}_{\alpha}^{\text{InfoSP}}(\check{\mathbf{p}})$ and by carefully
modifying the proof of Theorem C.1 (§ C.2) in order to get only equalities, by using
that we consider the case of the non-trivial selection, that is, $\mathcal{S} = \text{BH}(\mathbf{q})$, with $q_i =$
Applying this with $t = \alpha K_i/m$ entails (31).

## Appendix F: Computing I-adjusted p-values

We gather here details for computing the $q_i$'s (11) for different informative set collection $\mathcal{I}$. In the regression case, recall that $p_i^{(y)}$ necessarily corresponds to the full-calibrated p-values (7).

**Regression and excluding $\mathcal{Y}_0 \subset \mathcal{Y}$**. The informative set collection is given by $\mathcal{I} = \{I \text{ interval of } \mathbb{R}: I \cap \mathcal{Y}_0 = \emptyset\}$.

* The general expression is $q_i = \sup_{y \in \mathcal{Y}_0} p_i^{(y)}$. Indeed, by definition, $C_{n+i}^{\alpha}(\mathbf{p}) \in \mathcal{I}$ iff $C_{n+i}^{\alpha}(\mathbf{p}) \cap \mathcal{Y}_0 = \emptyset$ iff $\forall y \in \mathcal{Y}_0, p_i^{(y)} \le \alpha$, which means $\sup_{y \in \mathcal{Y}_0} p_i^{(y)} \le \alpha$.

* Another expression for the latter is given as follows:

$$ q_i = \frac{1}{n+1} \sup_{y \in \mathcal{Y}_0} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) \ge S_y(X_{n+i})\} \right) \\ = \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) \ge \inf_{y \in \mathcal{Y}_0} S_y(X_{n+i})\} \right). $$

* If $\mathcal{Y}_0 = [a, b]$ and $S_y(x) = |y - \mu(x)|/\sigma(x)$ the above expression gives

$$ q_i = p_i^{(a)} \mathbf{1}\{\mu(X_{n+i}) < a\} + p_i^{(b)} \mathbf{1}\{\mu(X_{n+i}) > b\} + \mathbf{1}\{a \le \mu(X_{n+i}) \le b\}. $$

This is because $\inf_{y \in [a, b]} S_y(x) = 0$ if $\mu(x) \in [a, b]$, $S_a(x)$ if $\mu(x) < a$, and $S_b(x)$ if $\mu(x) > b$.

* If $\mathcal{Y}_0 = [a, b]$ and $S_y(x) = \max(q_{\beta_0}(x) - y, y - q_{\beta_1}(x))$, we have

$$ q_i = p_i^{(a)} \mathbf{1}\{\mu(X_{n+i}) < a\} + p_i^{(b)} \mathbf{1}\{\mu(X_{n+i}) > b\} + p_i^{(\mu(X_{n+i}))} \mathbf{1}\{a \le \mu(X_{n+i}) \le b\}, $$

where $\mu(x) = (q_{\beta_0}(x) + q_{\beta_1}(x))/2$. This is because $\inf_{y \in [a, b]} S_y(x) = S_{\mu(x)}(x)$ if $\mu(x) \in [a, b]$, $S_a(x)$ if $\mu(x) < a$, and $S_b(x)$ if $\mu(x) > b$.

**Regression and length-restriction.** The informative set collection is given by $\mathcal{I} = \{I \text{ interval of } \mathbb{R}: |I| \le 2\lambda_0\}$, $\lambda_0 > 0$.

* By definition, $C_{n+i}^{\alpha}(\mathbf{p}) \in \mathcal{I}$ iff $|\{y \in \mathbb{R}: p_i^{(y)} > \alpha\}| \le 2\lambda_0$, that is, $|\{y \in \mathbb{R}: S_y(X_{n+i}) \le S_{(\lceil(1-\alpha)(n+1)\rceil)}\}| \le 2\lambda_0$. We then obtain the expression of $q_i$ by inverting the condition $S_y(X_{n+i}) \le S_{(\lceil(1-\alpha)(n+1)\rceil)}$ with respect to $y$. This expression depends on the considered score function.

* For $S_y(x) = |y - \mu(x)|/\sigma(x)$, we have $S_y(X_{n+i}) \le S_{(\lceil(1-\alpha)(n+1)\rceil)}$ iff $|y - \mu(X_{n+i})| \le \sigma(X_{n+i})S_{(\lceil(1-\alpha)(n+1)\rceil)}$ which gives $q_i \le \alpha$ iff $\sigma(X_{n+i})S_{(\lceil(1-\alpha)(n+1)\rceil)} \le \lambda_0$ (see Remark 2.1). Inverting the latter with respect to $\alpha$ gives

$$ q_i = (n+1)^{-1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) > \lambda_0/\sigma(X_{n+i})\} \right). $$

* For $S_y(x) = \max(q_{\beta_0}(x) - y, y - q_{\beta_1}(x))$, we have $S_y(X_{n+i}) \le S_{(\lceil(1-\alpha)(n+1)\rceil)}$ iff $q_{\beta_0}(x) - S_{(\lceil(1-\alpha)(n+1)\rceil)} \le y \le S_{(\lceil(1-\alpha)(n+1)\rceil)} + q_{\beta_1}(x)$. Hence, $q_i \le \alpha$ iff $S_{(\lceil(1-\alpha)(n+1)\rceil)} + (q_{\beta_1}(X_{n+i}) - q_{\beta_0}(X_{n+i}))/2 \le \lambda_0$, which gives

$$ q_i = (n+1)^{-1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) > \lambda_0 - (q_{\beta_1}(X_{n+i}) - q_{\beta_0}(X_{n+i}))/2\} \right). $$
$\min_{k \in [K]} \check{p}_i^{(k)}$. By definition of the FCP (2), we have (remember also that the prediction set always includes $\arg \min_{k \in [K]} S_k(X_{n+i})$ due to the post-processing)

$$
\begin{aligned}
\mathrm{FCP}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\check{\mathbf{p}}), Y) &= \frac{\sum_{i \in S} \mathbf{1}\{Y_{n+i} \notin C_{n+i}^{\alpha|S|/m}(\check{\mathbf{p}})\}}{1 \vee |S|} \\
&= \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\} \frac{\mathbf{1}\{i \in S, \check{p}_i^{(Y_{n+i})} \le \alpha|S|/m\}}{1 \vee |S|},
\end{aligned}
$$

because no error can occur when $R_{n+i} = 1$. Now, since $S = \mathrm{BH}(\mathbf{q})$ and $q_i \le \check{p}_i^{(Y_{n+i})}$, $\mathbf{1}\{i \in S, \check{p}_i^{(Y_{n+i})} \le \alpha|S|/m\} = \mathbf{1}\{\check{p}_i^{(Y_{n+i})} \le \alpha|S|/m\}$. This entails

$$
\mathrm{FCP}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\check{\mathbf{p}}), Y) = \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\} \frac{\mathbf{1}\{\check{p}_i^{(Y_{n+i})} \le \alpha|S|/m\}}{1 \vee |S|}.
$$

Let $\xi = (\mathbf{1}\{R_j > 1\})_{j \in [n+m]}$ and now prove

$$
\mathbb{E}[\mathrm{FCP}(\mathcal{R}_{\alpha}^{\mathrm{InfoSP}}(\check{\mathbf{p}}), Y) | \xi] = \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\} \mathbb{E}\left[ \frac{\left\lfloor \frac{(n+1)\alpha K_i/m}{n' + 1} \right\rfloor}{K_i} \mid \xi \right], \quad (31)
$$

with $n' = \sum_{j \in [n]} \mathbf{1}\{R_j > 1\}$ for some random variables $K_i \ge 1$, $i \in [m]$. This implies
the result because the last display is at most

$$
\alpha \frac{(n + 1) \sum_{i \in [m]} \mathbf{1}\{R_{n+i} > 1\}}{m(\sum_{j \in [n]} \mathbf{1}\{R_j > 1\} + 1)}
$$

(with equality if $(n+1)\alpha/m$ is an integer). By Lemma F.2, the expectation of the latter is equal to $\alpha(1 - (1 - \mathbb{P}(R_1 > 1))^{n+1})$.

Let us now prove (31). For this, fix $i \in [m]$ with $R_{n+i} > 1$ and note that for all
$j \in [m], j \neq i$,

$$
\begin{align*}
q_j &= \frac{1}{n+1} \left( 1 + \sum_{k=1}^{n} \mathbf{1}\{R_k > 1\} \mathbf{1}\{S_{Y_k}(X_k) \ge S_{\max}(X_{n+j})\} \right) \\
&= \frac{1}{n+1} \left( \mathbf{1}\{S_{Y_{n+i}}(X_{n+i}) < S_{\max}(X_{n+j})\} + \sum_{s \in A_i} \mathbf{1}\{s \ge S_{\max}(X_{n+j})\} \right) \\
&= \frac{1}{n+1} \left( \mathbf{1}\{S_{\max}(X_{n+i}) < S_{\max}(X_{n+j})\} + \sum_{s \in A_i} \mathbf{1}\{s \ge S_{\max}(X_{n+j})\} \right) \\
&\ge \frac{1}{n+1} \sum_{s \in A_i} \mathbf{1}\{s \ge S_{\max}(X_{n+j})\} =: q'_j,
\end{align*}
$$

by letting $A_i = \{S_{Y_k}(X_j), j \in [n] : R_k > 1\} \cup \{S_{Y_{n+i}}(X_{n+i})\}$ (all distinct almost surely).
The third equality above is true because $K=2$ and $R_{n+i} > 1$ and thus $S_{Y_{n+i}}(X_{n+i}) =$
$S_{\max}(X_{n+i})$. We apply now Lemma F.3 because $\mathbf{q} = (q_j, 1 \le j \le m)$ and $\mathbf{q}' = (q'_j, 1 \le$
**Classification.** The following informative collections are considered:

* For excluding $\mathcal{Y}_0 \subset [K]$, $\mathcal{I} = \{C \subset [K] : C \cap \mathcal{Y}_0 = \emptyset\}$. The general expression is $q_i = \max_{y \in \mathcal{Y}_0} p_i^{(y)}$, with the same reasoning as in the regression case.

* For non-trivial classification, $\mathcal{I} = \{C \subset [K] : |C| \le K - 1\}$ and $q_i = \min_{y \in [K]} p_i^{(y)}$. Indeed, $C_{n+i}^{\alpha}(\mathbf{p}) \in \mathcal{I}$ iff $|\{y \in [K] : p_i^{(y)} > \alpha\}| \le K - 1$ iff there exists $y \in [K]$ such that $p_i^{(y)} \le \alpha$, that is, $\min_{y \in [K]} p_i^{(y)} \le \alpha$.

* For at most $k_0$-sized classification, $\mathcal{I} = \{C \subset [K] : |C| \le k_0\}$. A similar reasoning leads to $q_i$ being the $(K - k_0)$-th smallest element in the set $\{p_i^{(y)}, y \in [K]\}$.

## Appendix G: Alternative proof of Theorem 3.1 using PRDS

Proving Theorem 3.1 does not go through the PRDS property (Benjamini and Yekutieli, 2001) but rather uses the monotonicity introduced in Assumption 4, see Section E.1. We propose in this section an alternative proof that explicitly relies on the PRDS property, and which shares similarities with the techniques developed in Jin and Candes (2023).

### G.1. Identifying a PRDS property

Recall that a random variable family $(T_i)_{i \in [m]}$ (valued in $[0, 1]$) is said to be PRDS on $\mathcal{H} \subset [m]$ if for any measurable nondecreasing$^5$ set $D \subset [0, 1]^m$, for all $i \in \mathcal{H}$, the function $u \in [0, 1] \mapsto \mathbb{P}((T_j)_{j \in [m]} \in D | T_i = u)$ is nondecreasing.

The following results can be seen as an extension of Lemma 5 in Jin and Candes (2023) to our $\mathcal{I}$-informative context.

**Theorem G.1.** Consider the setting and assumptions of Theorem 3.1 and $(q_i)_{i \in [m]}$ the $\mathcal{I}$-adjusted p-value vector defined by (11). Then the family $(q_1, \dots, q_{i-1}, p_i^{(Y_{n+i})}, q_{i+1}, \dots, q_m)$ is PRDS on $\{i\}$ for any $i \in [m]$.

*Proof.* Let us denote $\mathbf{p}_{-i} = (p_j^{(y)})_{j \neq i, y \in \mathcal{Y}}$ and $\mathbf{q}_{-i} := (q_j)_{j \neq i}$. By Section C.3, Assumption 4 is satisfied with a random variable $W_i$ which is independent of $p_i^{(Y_{n+i})}$. In particular, $\mathbf{p}_{-i} = \Psi_i(p_i^{(Y_{n+i})}, W_i)$ where $u \in [0, 1] \mapsto \Psi_i(u, W_i) \in \mathbb{R}^{[m-1] \times \mathcal{Y}}$ is a nondecreasing function. In addition, by Section C.4, $\mathbf{q}_{-i}$ is a coordinate-wise nondecreasing function of $\mathbf{p}_{-i}$. Combining these facts, we obtain that

$$ \mathbf{q}_{-i} = \phi_i(p_i^{(Y_{n+i})}, W_i), $$

where $u \in [0, 1] \mapsto \phi_i(u, W_i) \in \mathbb{R}^{[m-1]}$ is a nondecreasing function. As a result, for any measurable nondecreasing set $D \subset [0, 1]^m$,

$$ \begin{aligned} \mathbb{P}((p_i^{(Y_{n+i})}, \mathbf{q}_{-i}) \in D | p_i^{(Y_{n+i})} = u) &= \mathbb{P}((p_i^{(Y_{n+i})}, \phi_i(p_i^{(Y_{n+i})}, W_i)) \in D | p_i^{(Y_{n+i})} = u) \\ &= \mathbb{P}((u, \phi_i(u, W_i)) \in D), \end{aligned} $$

where we denoted $(p_i^{(Y_{n+i})}, \mathbf{q}_{-i}) = (q_1, \dots, q_{i-1}, p_i^{(Y_{n+i})}, q_{i+1}, \dots, q_m)$ with some notation abuse and where we used independence between $W_i$ and $p_i^{(Y_{n+i})}$. Since for a fixed $W_i$, $u \mapsto \mathbf{1}\{(u, \phi_i(u, W_i)) \in D\}$ is nondecreasing (since $D$ is nondecreasing), the last display is nondecreasing in $u$. $\square$

<footnote>$^5$A subset $D \subset [0, 1]^m$ is said to be nondecreasing if for all $(x_i)_{i \in [m]} \in D$ and all $(y_i)_{i \in [m]} \in [0, 1]^m$, $(\forall i \in [m], x_i \le y_i)$ implies $(y_i)_{i \in [m]} \in D$.</footnote>
$j \le m$) (defined as above and with $q'_i = 1/(n+1)$) satisfy (32). Indeed, if $q_j > q_i$ for $j \ne i$, then $S_{\max}(X_{n+j}) \le S_{\max}(X_{n+i})$ and $q_j = q'_j$. Hence, we obtain, by letting $K_i = |\text{BH}(\mathbf{q}')|$ (note that $\mathbf{q}'$ depends on $i$)

$$ \{q_i \le \alpha |\text{BH}(\mathbf{q}')|/m\} = \{q_i \le \alpha K_i/m\} \subset \{|\text{BH}(\mathbf{q}')| = K_i\}. $$

In addition, $\mathbf{q}' = (q'_j, 1 \le j \le m)$ is a vector measurable w.r.t. the variable $W_i = (A_i, (S_{\max}(X_{n+j}))_{j \in [m]\setminus\{i\}})$. Now, we use that by exchangeability of the elements of $A_i$ conditionally on $\xi$, $W_i$, and $R_{n+i} > 1$) (with no ties),

$$ \mathbb{P}(\tilde{p}_i^{(Y_{n+i})} \le t | W_i, \xi, R_{n+i} > 1) = \frac{\lfloor (n+1)t \rfloor}{n' + 1}. $$

Applying this with $t = \alpha K_i/m$ entails (31).

## F. Auxiliary results

LEMMA F.1 (LEMMA 3.2 IN BLANCHARD AND ROQUAIN (2008)). Let $g: [0, 1] \to (0, \infty)$ be a nonincreasing function and $U$ be a random variable which is super-uniform, that is, $\forall u \in [0, 1]$, $\mathbb{P}(U \le u) \le u$. Then, for any $c > 0$, we have

$$ \mathbb{E} \left[ \frac{1\{U \le cg(U)\}}{g(U)} \right] \le c. $$

LEMMA F.2 (LEMMA 1 OF BENJAMINI ET AL. (2006)). If $T$ is a Binomial variable with parameter $N - 1 \ge 0$ and $t \in (0, 1]$, we have

$$ \mathbb{E}[1/(T+1)] = (1 - (1-t)^N)/(Nt) \le 1/(Nt). $$

LEMMA F.3 (LEMMA D.6 OF MARANDON ET AL. (2024)). Write $\hat{\ell} = \hat{\ell}(\mathbf{q}) = |\text{BH}(\mathbf{q})|$ for the number of rejections of $\text{BH}(\mathbf{q})$ (21). Fix any $i \in \{1, \dots, m\}$ and consider two collections $\mathbf{q} = (q_j, 1 \le j \le m)$ and $\mathbf{q}' = (q'_j, 1 \le j \le m)$ which satisfy almost surely that

$$ \forall j \in \{1, \dots, m\}, \quad \begin{cases} q'_j \le q_j & \text{if } q_j \le q_i; \\ q'_j = q_j & \text{if } q_j > q_i. \end{cases} \qquad (32) $$

$$ \text{Then } \{q_i \le \alpha \hat{\ell}(\mathbf{q})/m\} = \{q_i \le \alpha \hat{\ell}(\mathbf{q}')/m\} \subset \{\hat{\ell}(\mathbf{q}) = \hat{\ell}(\mathbf{q}')\}. $$

The calibration splitting trick can be seen as a way to obtain statistical guarantees in conformal inference when making a data-driven choice regarding the inference. In Marandon et al. (2024); Gazin et al. (2023), the choices are about adaptive score functions. The next lemma presents this trick in the case that the choice is about which examples to select for (potentially more efficient and powerful) further inference $||$.

LEMMA F.4 (CALIBRATION SPLITTING TRICK FOR SELECTIVE CONFORMAL PREDICTION SETS). Assume that for any training sample $\mathcal{D}_{\text{train}}$, calibration sample $\mathcal{D}_{\text{cal}} = ((X_j, Y_j))_{j \in [n_{\text{cal}}]}$ and test sample $\mathcal{D}_{\text{test}} = (X_j, Y_j)_{j \in [n_{\text{cal}}+1, n_{\text{cal}}+n_{\text{test}}]}$, such that $(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{cal}}, \mathcal{D}_{\text{test}})$ has some distribution $Q$, the procedure (that is, a prediction set collection on a selection) and initial selection rule are as follows:

$||$This idea was sketched in version 3 of the arXiv version Bao et al. (2023) of the work Bao et al. (2024)
## G.2. Alternative proof of Theorem 3.1

By Definition 1, we have

$$ 
\begin{aligned}
\text{FCP}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\mathbf{p}), Y) &= \frac{\sum_{i \in \text{BH}(\mathbf{q})} \mathbf{1}\{Y_{n+i} \notin C_{n+i}^{\alpha|\text{BH}(\mathbf{q})|/m}(\mathbf{p})\}}{1 \lor |\text{BH}(\mathbf{q})|} \\
&= \frac{\sum_{i \in \text{BH}(\mathbf{q})} \mathbf{1}\{Y_{n+i} \notin C_{n+i}^{\alpha|\text{BH}(\mathbf{q}^{0,i})|/m}(\mathbf{p})\}}{1 \lor |\text{BH}(\mathbf{q}^{0,i})|} \\
&\leq \sum_{i \in [m]} \frac{\mathbf{1}\{p_i^{(Y_{n+i})} \leq \alpha|\text{BH}(\mathbf{q}^{0,i})|/m\}}{1 \lor |\text{BH}(\mathbf{q}^{0,i})|},
\end{aligned}
$$

where the second equality follows from $\text{BH}(\mathbf{q}^{0,i}) = \text{BH}(\mathbf{q})$ when $i \in \text{BH}(\mathbf{q})$, by denoting $\mathbf{q}^{0,i}$ the vector $\mathbf{q}$ where the $i$-th coordinate has been replaced by 0, see for instance Ramdas et al. (2019). This entails

$$ \mathbb{E}(\text{FCP}(\mathcal{R}_{\alpha}^{\text{InfoSP}}(\mathbf{p}), Y)) \leq \sum_{i \in [m]} \mathbb{E} \left( \frac{\mathbf{1}\{p_i^{(Y_{n+i})} \leq \alpha|\text{BH}(\mathbf{q}^{0,i})|/m\}}{1 \lor |\text{BH}(\mathbf{q}^{0,i})|} \right). $$

Now, by Theorem G.1, for any $r > 0$, the function $u \mapsto \mathbb{P}(|\text{BH}(\mathbf{q}^{0,i})| < r | p_i^{(Y_{n+i})} = u)$ is nondecreasing and thus so is $u \mapsto \mathbb{P}(|\text{BH}(\mathbf{q}^{0,i})| < r | p_i^{(Y_{n+i})} \leq u)$. Applying Lemma H.2 gives the result.

## Appendix H: Auxiliary results

**Lemma H.1 (Lemma 3.2 (i) in Blanchard and Roquain (2008)).** Let $g: [0, 1] \to (0, \infty)$ be a nonincreasing function and $U$ be a random variable which is super-uniform, that is, $\forall u \in [0, 1]$, $\mathbb{P}(U \leq u) \leq u$. Then, for any $c > 0$, we have

$$ \mathbb{E} \left[ \frac{\mathbf{1}\{U \leq cg(U)\}}{g(U)} \right] \leq c. $$

**Lemma H.2 (Lemma 3.2 (ii) in Blanchard and Roquain (2008)).** Let $U, V$ be nonnegative random variables such that $U$ is super-uniform, that is, $\forall u \in [0, 1]$, $\mathbb{P}(U \leq u) \leq u$ and $V$ is such that for any $r > 0$, the function $u \mapsto \mathbb{P}(V < r | U \leq u)$ is nondecreasing. Then, for any $c > 0$, we have

$$ \mathbb{E} \left[ \frac{\mathbf{1}\{U \leq cV\}}{V} \right] \leq c. $$

**Lemma H.3 (Lemma 1 of Benjamini et al. (2006)).** If $T$ is a Binomial variable with parameter $N - 1 \geq 0$ and $t \in (0, 1]$, we have

$$ \mathbb{E}[1/(T+1)] = (1 - (1-t)^N)/(Nt) \leq 1/(Nt). $$

**Lemma H.4 (Lemma D.6 of Marandon et al. (2024)).** Write $\hat{\ell} = \hat{\ell}(\mathbf{q}) = |\text{BH}(\mathbf{q})|$ for the number of rejections of $\text{BH}(\mathbf{q})$ (21). Fix any $i \in \{1, \dots, m\}$ and consider two collections $\mathbf{q} = (q_j, 1 \leq j \leq m)$ and $\mathbf{q}' = (q'_j, 1 \leq j \leq m)$ which satisfy almost surely that

$$ \forall j \in \{1, \dots, m\}, \begin{cases} q'_j \leq q_j & \text{if } q_j \leq q_i; \\ q'_j = q_j & \text{if } q_j > q_i. \end{cases} \qquad (32) $$

$$ \text{Then } \{q_i \leq \alpha \hat{\ell}(\mathbf{q})/m\} = \{q_i \leq \alpha \hat{\ell}(\mathbf{q}')/m\} \subset \{\hat{\ell}(\mathbf{q}) = \hat{\ell}(\mathbf{q}')\}. $$
(i) the procedure $\mathcal{R} = (C_{n_{cal}+i})_{i \in S}$, $C_{n_{cal}+i} \subset \mathcal{Y}$, $S \subset [n_{test}]$, with $\mathcal{R} = \mathcal{R}(\mathcal{D}_{cal}, \mathcal{D}_{test}^X, \mathcal{D}_{train})$ built upon $\mathcal{D}_{train}$, $\mathcal{D}_{cal}$ and $\mathcal{D}_{test}^X := (X_j)_{j \in [n_{cal}+1, n_{cal}+n_{test}]}$ such that it controls a criterion $\mathbb{E}_Q(\mathcal{E}(\mathcal{R}, Y)) \le \alpha$ if the entries of $\mathcal{D}_{cal} \cup \mathcal{D}_{test}$ are exchangeable conditionally on $\mathcal{D}_{train}$;

(ii) an initial selection rule $S^{(0)} = S^{(0)}(\mathcal{D}_{cal}, \mathcal{D}_{test}^X; \mathcal{D}_{train}) \subset [n_{cal} + 1, n_{cal} + n_{test}]$ built upon $\mathcal{D}_{train}$, $\mathcal{D}_{cal}$ and $\mathcal{D}_{test}^X$ which is permutation preserving in the latter, that is, for any permutation $\sigma$ of $[n_{cal} + 1, n_{cal} + n_{test}]$, we have $\sigma(S^{(0)}(\mathcal{D}_{cal}, \mathcal{D}_{test}^X; \mathcal{D}_{train})) = S^{(0)}(\mathcal{D}_{cal}, \sigma(\mathcal{D}_{test}^X); \mathcal{D}_{train})$.

Let us consider samples $(\mathcal{D}_{train}, \mathcal{D}_{cal}, \mathcal{D}_{test})$ as above such that the entries of $\mathcal{D}_{cal} \cup \mathcal{D}_{test}$ are exchangeable conditionally on $\mathcal{D}_{train}$ and split the calibration set $\mathcal{D}_{cal}$ into two samples $\mathcal{D}_{cal}^{(1)}$ and $\mathcal{D}_{cal}^{(2)}$ of respective sizes $n_{cal}^{(1)}$ and $n_{cal}^{(2)}$. Let $\mathcal{D}_{cal}^{(2),X} = (X_j)_{j \in [n_{cal}^{(1)}+1, n_{cal}]}$. Then for $S^{(0)} = S^{(0)}(\mathcal{D}_{cal}^{(1)}, \mathcal{D}_{cal}^{(2),X} \cup \mathcal{D}_{test}^X; \mathcal{D}_{train}) \subset [n_{cal}^{(1)} + 1, n_{cal} + n_{test}]$, the procedure $\mathcal{R}^{split} = \mathcal{R}((\mathcal{D}_{cal}^{(2)})_{S^{(0)}}, (\mathcal{D}_{test}^X)_{S^{(0)}}, \mathcal{D}_{train})$ is a procedure achieving $\mathbb{E}_Q(\mathcal{E}(\mathcal{R}^{split}, Y) | S^{(0)}) \le \alpha$ and thus also $\mathbb{E}_Q(\mathcal{E}(\mathcal{R}^{split}, Y)) \le \alpha$.

For instance, Lemma F.4 can be used for the criterion $\mathcal{E}(\mathcal{R}, Y) = \text{FCP}(\mathcal{R}, Y)$ in which case it provides FCR control. For instance, since classical marginal prediction intervals always satisfy (i) with $S = [m]$ and the FCR criterion, Lemma F.4 shows that sample splitting can offer FCR control for any (permutation preserving) selection rule. This is an alternative method to the swapping approach of Jin and Ren (2023).

**PROOF.** For convenience, let us write $\mathcal{D}_{cal} \cup \mathcal{D}_{test} = ((X_j, Y_j))_{j \in [n_{cal}+n_{test}]} = (Z_j)_{j \in [n_{cal}+n_{test}]}$.

By (i), it is enough to prove that the entries of $(Z_j)_{j \in S}$ are exchangeable conditionally on $S^{(0)} = S$, $\mathcal{D}_{cal}^{(1)}$ and $\mathcal{D}_{train}$, for any possible realisation $S$ of $S^{(0)}$. For any $\sigma$ permutation of $[n_{cal}^{(1)} + 1, n_{cal} + n_{test}]$ which only affects the indexes of $S$, we have

$$ \begin{aligned} \mathcal{D}((Z_{\sigma(j)})_{j \in S, j \ge n_{cal}^{(1)}+1} | S^{(0)} &= S, \mathcal{D}_{cal}^{(1)}, \mathcal{D}_{train}) \\ &= \mathcal{D}((Z_{\sigma(j)})_{j \in S, j \ge n_{cal}^{(1)}+1} | S(\mathcal{D}_{cal}^{(1)}, \mathcal{D}_{cal}^{(2),X} \cup \mathcal{D}_{test}^X; \mathcal{D}_{train}) = S, \mathcal{D}_{cal}^{(1)}, \mathcal{D}_{train}) \\ &= \mathcal{D}((Z_{\sigma(j)})_{j \in S, j \ge n_{cal}^{(1)}+1} | S(\mathcal{D}_{cal}^{(1)}, (Z_{\sigma(j)})_{j \in [n_{cal}^{(1)}+1, n_{cal}+n_{test}]}; \mathcal{D}_{train}) = S, \mathcal{D}_{cal}^{(1)}, \mathcal{D}_{train}), \end{aligned} $$

by using the permutation preserving property (ii) and because $\sigma(S) = S$ by definition of $\sigma$. Now using that the distribution of $(Z_{\sigma(j)})_{j \in [n_{cal}^{(1)}+1, n_{cal}+n_{test}]}$ is the same as $(Z_j)_{j \in [n_{cal}^{(1)}+1, n_{cal}+n_{test}]}$ conditionally on $\mathcal{D}_{train}$ and $\mathcal{D}_{cal}^{(1)} = (Z_j)_{j \in [n_{cal}^{(1)}]}$, the last display is equal to

$$ \mathcal{D}((Z_j)_{j \in S, j \ge n_{cal}^{(1)}+1} | S^{(0)} = S, \mathcal{D}_{cal}^{(1)}, \mathcal{D}_{train}), $$

which provides the desired exchangeability for valid inference on $(\mathcal{D}_{test}^X)_{S^{(0)}}$ using $(\mathcal{D}_{cal}^{(2)})_{S^{(0)}}$.

## G. Additional illustrations in the regression case

### G.1. Excluding a null value $y_0 \in \mathbb{R}$

We consider here $\mathcal{I} = \{I \text{ interval of } \mathbb{R} : y_0 \notin I\}$, which is useful in situation where the value $y_0$ corresponds to some “normal” value and the user wants to report only prediction
The calibration splitting trick can be seen as a way to obtain statistical guarantees in conformal inference when making a data-driven choice regarding the inference. In Marandon et al. (2024); Gazin et al. (2024), the choices are about adaptive score functions. The next lemma presents this trick in the case that the choice is about which examples to select for (potentially more efficient and powerful) further inference<fnref n="6" />.

**Lemma H.5** (Calibration splitting trick for selective conformal prediction sets). Assume that for any training sample $\mathcal{D}_{\text{train}}$, calibration sample $\mathcal{D}_{\text{cal}} = ((X_j, Y_j))_{j \in [n_{\text{cal}}]}$ and test sample $\mathcal{D}_{\text{test}} = (X_j, Y_j)_{j \in [n_{\text{cal}}+1, n_{\text{cal}}+n_{\text{test}}]}$, such that $(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{cal}}, \mathcal{D}_{\text{test}})$ has some distribution $Q$, the procedure (that is, a prediction set collection on a selection) and initial selection rule are as follows:

(i) the procedure $\mathcal{R} = (\mathcal{C}_{n_{\text{cal}}+i})_{i \in S}$, $\mathcal{C}_{n_{\text{cal}}+i} \subset \mathcal{Y}$, $S \subset [n_{\text{test}}]$, with $\mathcal{R} = \mathcal{R}(\mathcal{D}_{\text{cal}}, \mathcal{D}_{\text{test}}; \mathcal{D}_{\text{train}})$ built upon $\mathcal{D}_{\text{train}}$, $\mathcal{D}_{\text{cal}}$ and $\mathcal{D}_{\text{test}}^X := (X_j)_{j \in [n_{\text{cal}}+1, n_{\text{cal}}+n_{\text{test}}]}$ such that it controls a criterion $\mathbb{E}_Q(\mathcal{E}(\mathcal{R}, Y)) \le \alpha$ if the entries of $\mathcal{D}_{\text{cal}} \cup \mathcal{D}_{\text{test}}$ are exchangeable conditionally on $\mathcal{D}_{\text{train}}$;

(ii) an initial selection rule $S^{(0)} = S^{(0)}(\mathcal{D}_{\text{cal}}, \mathcal{D}_{\text{test}}^X; \mathcal{D}_{\text{train}}) \subset [n_{\text{cal}}+1, n_{\text{cal}}+n_{\text{test}}]$ built upon $\mathcal{D}_{\text{train}}$, $\mathcal{D}_{\text{cal}}$ and $\mathcal{D}_{\text{test}}^X$ which is permutation preserving in the latter, that is, for any permutation $\sigma$ of $[n_{\text{cal}}+1, n_{\text{cal}}+n_{\text{test}}]$, we have $\sigma(S^{(0)}(\mathcal{D}_{\text{cal}}, \mathcal{D}_{\text{test}}^X; \mathcal{D}_{\text{train}})) = S^{(0)}(\mathcal{D}_{\text{cal}}, \sigma(\mathcal{D}_{\text{test}}^X; \mathcal{D}_{\text{train}}))$.

Let us consider samples $(\mathcal{D}_{\text{train}}, \mathcal{D}_{\text{cal}}, \mathcal{D}_{\text{test}})$ as above such that the entries of $\mathcal{D}_{\text{cal}} \cup \mathcal{D}_{\text{test}}$ are exchangeable conditionally on $\mathcal{D}_{\text{train}}$ and split the calibration set $\mathcal{D}_{\text{cal}}$ into two samples $\mathcal{D}_{\text{cal}}^{(1)}$ and $\mathcal{D}_{\text{cal}}^{(2)}$ of respective sizes $n_{\text{cal}}^{(1)}$ and $n_{\text{cal}}^{(2)}$. Let $\mathcal{D}_{\text{cal}}^{(2),X} = (X_j)_{j \in [n_{\text{cal}}^{(1)}, n_{\text{cal}}^{(1)}]}$. Then for $S^{(0)} = S^{(0)}(\mathcal{D}_{\text{cal}}^{(1)}, \mathcal{D}_{\text{cal}}^{(2),X} \cup \mathcal{D}_{\text{test}}^X; \mathcal{D}_{\text{train}}) \subset [n_{\text{cal}}^{(1)} + 1, n_{\text{cal}}^{(1)} + n_{\text{test}}]$, the procedure $\mathcal{R}^{\text{split}} = \mathcal{R}((\mathcal{D}_{\text{cal}}^{(2)})_{S^{(0)}}, (\mathcal{D}_{\text{test}}^X)_{S^{(0)}}; \mathcal{D}_{\text{train}})$ is a procedure achieving $\mathbb{E}_Q(\mathcal{E}(\mathcal{R}^{\text{split}}, Y) | S^{(0)}) \le \alpha$ and thus also $\mathbb{E}_Q(\mathcal{E}(\mathcal{R}^{\text{split}}, Y)) \le \alpha$.

For instance, Lemma H.5 can be used for the criterion $\mathcal{E}(\mathcal{R}, Y) = \text{FCP}(\mathcal{R}, Y)$ in which case it provides FCR control. For instance, since classical marginal prediction intervals always satisfy (i) with $S = [m]$ and the FCR criterion, Lemma H.5 shows that sample splitting can offer FCR control for any (permutation preserving) selection rule. This is an alternative method to the swapping approach of Jin and Ren (2024).

**Proof.** For convenience, let us write $\mathcal{D}_{\text{cal}} \cup \mathcal{D}_{\text{test}} = ((X_j, Y_j))_{j \in [n_{\text{cal}}+n_{\text{test}}]} = (Z_j)_{j \in [n_{\text{cal}}+n_{\text{test}}]}$. By (i), it is enough to prove that the entries of $(Z_j)_{j \in S}$ are exchangeable conditionally on $S^{(0)} = S$, $\mathcal{D}_{\text{cal}}^{(1)}$ and $\mathcal{D}_{\text{train}}$, for any possible realisation $S$ of $S^{(0)}$. For any $\sigma$ permutation of $[n_{\text{cal}}^{(1)}+1, n_{\text{cal}}+n_{\text{test}}]$ which only affects the indexes of $S$, we have

$$ \begin{aligned} & \mathcal{D}((Z_{\sigma(j)})_{j \in S, j \ge n_{\text{cal}}^{(1)}+1} | S^{(0)} = S, \mathcal{D}_{\text{cal}}^{(1)}, \mathcal{D}_{\text{train}}) \\ &= \mathcal{D}((Z_{\sigma(j)})_{j \in S, j \ge n_{\text{cal}}^{(1)}+1} | S(\mathcal{D}_{\text{cal}}^{(1)}, \mathcal{D}_{\text{cal}}^{(2),X} \cup \mathcal{D}_{\text{test}}^X; \mathcal{D}_{\text{train}}) = S, \mathcal{D}_{\text{cal}}^{(1)}, \mathcal{D}_{\text{train}}) \\ &= \mathcal{D}((Z_{\sigma(j)})_{j \in S, j \ge n_{\text{cal}}^{(1)}+1} | S(\mathcal{D}_{\text{cal}}^{(1)}, (Z_{\sigma(j)})_{j \in [n_{\text{cal}}^{(1)}+1, n_{\text{cal}}+n_{\text{test}}]}; \mathcal{D}_{\text{train}}) = S, \mathcal{D}_{\text{cal}}^{(1)}, \mathcal{D}_{\text{train}}), \end{aligned} $$

by using the permutation preserving property (ii) and because $\sigma(S) = S$ by definition of $\sigma$. Now using that the distribution of $(Z_{\sigma(j)})_{j \in [n_{\text{cal}}^{(1)}+1, n_{\text{cal}}+n_{\text{test}}]}$ is the same as $(Z_j)_{j \in [n_{\text{cal}}^{(1)}+1, n_{\text{cal}}+n_{\text{test}}]}$ conditionally on $\mathcal{D}_{\text{train}}$ and $\mathcal{D}_{\text{cal}}^{(1)} = (Z_j)_{j \in [n_{\text{cal}}^{(1)}]}$, the last display is equal to

$$ \mathcal{D}((Z_j)_{j \in S, j \ge n_{\text{cal}}^{(1)}+1} | S^{(0)} = S, \mathcal{D}_{\text{cal}}^{(1)}, \mathcal{D}_{\text{train}}), $$

which provides the desired exchangeability for valid inference on $(\mathcal{D}_{\text{test}}^X)_{S^{(0)}}$ using $(\mathcal{D}_{\text{cal}}^{(2)})_{S^{(0)}}$. $\square$

<footnote n="6">⁶This idea was sketched in version 3 of the arXiv version Bao et al. (2024) of the work Bao et al. (2024)</footnote>
intervals for “abnormal” individuals, that is, when the outcome value deviates from this reference value. The score considered here is $S_y(x) = |y - \mu(x)|/\sigma(x)$, where $\mu$, $\sigma$ are predictors of the conditional mean and variance, respectively.

An illustration is provided in Figure 7 in the non-parametric regression Gaussian model for different situations: Rows 1,2 correspond to an homoscedastic model with perfect prediction of the variance and a mean predictor with various accuracy (less accurate in the middle for row 1, more accuracy in the middle for row 2). Rows 3,4 correspond to an heteroscedastic model with perfect prediction of the mean and a the variance predictor under-estimating the variance in the middle for row 3, and over-estimating the variance in the middle for row 4). The marginal prediction interval (no selection) is displayed in light blue while the prediction interval after selection is displayed in dark red. The quantities reported at the top of each panel are the FCR and adjusted power averaged over 1000 repetitions (while each panel only displays the last experiment as a typical realisation of the sample).

First, as expected, the naive classical conformal selection does not provide FCR control in all cases, while InfoSP and InfoSCOP do control the FCR in any case. Second, InfoSCOP always improves InfoSP in terms of power, and the range of improvement depends on where non-covering errors are likely to happen: when errors are more likely to arise on the selection (rows 1,3), the behavior of InfoSP and InfoSCOP are similar; when errors are less likely to arise on the selection (rows 2,4), InfoSCOP improves over InfoSP in a striking manner (and is even better than classical conformal). This is both because of the reduction of the selection effect and because the pre-processed p-values are calibrated with much lower residuals and thus are much more efficient than the original p-values.

## G.2. Prediction intervals for Jin and Candes (2023) selection

We focus here on the case where the user wants to build prediction sets only for outcomes $Y_{n+i} > y_0$, which corresponds to excluding the set $\mathcal{Y}_0 = (-\infty, y_0]$ and is related to the selection proposed in Jin and Candes (2023).

We consider here the aim of finding one-sided prediction intervals on the selected. Let us assume that the score function is monotone in the following sense (Jin and Candes, 2023): for all $x \in \mathbb{R}^d$, for $y \le y'$, $S_y(x) \ge S_{y'}(x)$. A classical example of monotonic score function is given by $S_y(x) = (\mu(x) - y)/\sigma(x)$. The following result summarizes our finding in this case.

COROLLARY G.1. Consider the iid model in the regression case and assume that the score function is monotone (see above) and such that Assumption 2 (ii) (iii) and Assumption 1 hold. Then the following holds for InfoSP with informative collection $\mathcal{I} = \{I \text{ interval of } \mathbb{R} : I \cap (-\infty, y_0] = \emptyset\}$ and p-value collection $\bar{p}$ (7):

(i) InfoSP selects $S = \text{BH}(\mathbf{q})$ the rejected set of BH procedure at level $\alpha$ applied with the p-values

$$q_i = \bar{p}_i^{(y_0)} = \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} \mathbf{1}\{S_{Y_j}(X_j) \ge S_{y_0}(X_{n+i})\} \right),$$
# Appendix I: Additional illustrations in the regression case

## I.1. Excluding a null value $y_0 \in \mathbb{R}$

We consider here $\mathcal{I} = \{I \text{ interval of } \mathbb{R} : y_0 \notin I\}$, which is useful in situation where the value $y_0$ corresponds to some “normal” value and the user wants to report only prediction intervals for “abnormal” individuals, that is, when the outcome value deviates from this reference value. The score considered here is $S_y(x) = |y - \mu(x)|/\sigma(x)$, where $\mu, \sigma$ are predictors of the conditional mean and variance, respectively.

An illustration is provided in Figure 7 in the non-parametric regression Gaussian model for different situations: Rows 1,2 correspond to an homoscedastic model with perfect prediction of the variance and a mean predictor with various accuracy (less accurate in the middle for row 1, more accuracy in the middle for row 2). Rows 3,4 correspond to an heteroscedastic model with perfect prediction of the mean and a the variance predictor under-estimating the variance in the middle for row 3, and over-estimating the variance in the middle for row 4). The marginal prediction interval (no selection) is displayed in light blue while the prediction interval after selection is displayed in dark red. The quantities reported at the top of each panel are the FCR and adjusted power averaged over 1000 repetitions (while each panel only displays the last experiment as a typical realisation of the sample).

First, as expected, the naive classical conformal selection does not provide FCR control in all cases, while InfoSP and InfoSCOP do control the FCR in any case. Second, InfoSCOP always improves InfoSP in terms of power, and the range of improvement depends on where non-covering errors are likely to happen: when errors are more likely to arise on the selection (rows 1,3), the behavior of InfoSP and InfoSCOP are similar; when errors are less likely to arise on the selection (rows 2,4), InfoSCOP improves over InfoSP in a striking manner (and is even better than classical conformal). This is both because of the reduction of the selection effect and because the pre-processed p-values are calibrated with much lower residuals and thus are much more efficient than the original p-values.

## I.2. Prediction intervals for Jin and Candes (2023) selection

We focus here on the case where the user wants to build prediction sets only for outcomes $Y_{n+i} > y_0$, which corresponds to excluding the set $\mathcal{Y}_0 = (-\infty, y_0]$ and is related to the selection proposed in Jin and Candes (2023).

We consider here the aim of finding one-sided prediction intervals on the selected. Let us assume that the score function is monotone in the following sense (Jin and Candes, 2023): for all $x \in \mathbb{R}^d$, for $y \le y'$, $S_y(x) \ge S_{y'}(x)$. A classical example of monotonic score function is given by $S_y(x) = (\mu(x) - y)/\sigma(x)$. The following result summarizes our finding in this case.

**Corollary I.1.** Consider the iid model in the regression case and assume that the score function is monotone (see above) and such that Assumption 2 (ii) (iii) and Assumption 1 hold. Then the following holds for InfoSP with informative collection $\mathcal{I} = \{I \text{ interval of } \mathbb{R} : I \cap (-\infty, y_0] = \emptyset\}$ and p-value collection $\bar{p}$ (7):

(i) InfoSP selects $S = \text{BH}(q)$ the rejected set of BH procedure at level $\alpha$ applied with the p-values

$$q_i = \bar{p}_i^{(y_0)} = \frac{1}{n+1} \left( 1 + \sum_{j=1}^{n} 1 \{ S_{Y_j}(X_j) \ge S_{y_0}(X_{n+i}) \} \right),$$

which coincides with the rejection set of the procedure proposed in Jin and Candes (2023).
which coincides with the rejection set of the procedure proposed in Jin and Candes (2023).

(ii) The selection $\mathcal{S} = \text{BH}(\mathbf{q})$ of InfoSP controls the FDR at level $\alpha$ in the following sense:

$$ \sup_{P_{XY}} \mathbb{E}_{(X,Y)\sim P_{XY}} \left[ \frac{\sum_{i \in \mathcal{S}} \mathbf{1}\{Y_{n+i} \le y_0\}}{1 \vee |\mathcal{S}|} \right] \le \alpha. $$

(iii) The prediction intervals are of the form $C_{n+i} = \{y > y_0 : S_y(X_{n+i}) \le S_{(n_\alpha)}\}$, $i \in \mathcal{S}$, where $S_{(1)} \le \dots \le S_{(n)}$ are the ordered calibration scores $S_{Y_j}(X_j)$, $1 \le j \le n$ (with $S_{(n+1)} = +\infty$), and $n_\alpha = \lceil (1 - \alpha|\mathcal{S}(\mathbf{p})|/m)(n+1) \rceil$.

(iv) These prediction intervals control the FCR at level $\alpha$ in the sense of (15).

In other words, the above result provides complements the multiple testing procedure of Jin and Candes (2023), by providing in addition FCR controlling informative prediction sets on the selected outcomes.

**PROOF.** The proof is direct with Example 2.1 and monotonicity (for (i)), Lemma 2.1 (for (ii)), Remark 2.1 (for (iii)), Theorem 3.1 (for (iv)).

Obviously, a similar result holds for InfoSCOP, for any initial selection step $\mathcal{S}^{(0)} \subset [r+1, n+m]$ that satisfies the permutation preserving Assumption 3. Corollary G.1 is illustrated on Figure 8 when $\mathcal{S}^{(0)}$ is taken here has $\text{BH}(\mathbf{q})$ at level $2\alpha$ with the score $S_y(x) = (\mu(x) - y)/\sigma(x)$. The comments are qualitatively similar to those of Figure 7: when covering errors are less likely on the selection, the improvement of InfoSCOP over InfoSP is substantial.

### G.3. Comparison with existing selective prediction sets

Figure 9 is useful to visualize the difference between the selection proposed by Bao et al. (2024) and our informative selection. We display selective prediction intervals for two FCR controlling selections proposed in Bao et al. (2024): SCOPa uses a thresholding rule $\mathcal{S} = \{i \in [m] : \mu(X_{n+i}) \ge y_0\}$, while SCOPb selects the largest $\mu(X_{n+i})$'s. Each time, a suitable conditional conformal prediction set is built in Bao et al. (2024) and reported in Figure 9 in green and purple. As one can see, either the selection is too conservative, or the prediction interval could include the nominal $b_0$. This not the case of InfoSCOP which always exclude $y_0$ by essence.

## H. Additional illustrations in the classification case

### H.1. Three classes, each bivariate normal

For the iid model, as described in § 5.2, we demonstrate the initial selection step in infoSCOP for excluding a null class in one realization of the data generation in Figure 10.

For non-trivial classification, as described in § 5.3, we provide numerical results in Figure 11.
<figure><img src="image_10.png" /><figcaption>Fig 7: Informative selection by excluding the null value $y_0$ in the (homoscedastic or heteroscedastic) regression case, see § I.1.</figcaption></figure>
### H.2. *Three classes, each a mixture of bivariate normals*

For the iid model, as in § 5.2, we consider the balanced and unbalanced cases of 3 classes. However, here we consider that each class comes from a mixture of two bivariate normals, where one component of the mixture is identical in all three classes, and given the class there is probability half of coming from that common mixture component. Thus the overlap between the classes is much greater than in the settings considered in § 5.2.

In Figure 12 we show that the qualitative conclusions with regard to the respective procedures are as in § 5.2. Interestingly, in the hardest setting at the bottom row we see that the power of InfoSCOP is even larger than classic conformal. This is because the initial selection step not only prevents paying too much for selection, but it can also improve the calibration set for computing the conformal *p*-values after selection. In this hard setting, there is a large improvement of the tail probability that matters for inference. We demonstrate this in one realization of the data generation in Figure 13, where we can see that the 0.95 quantile of the original calibration set is much larger than the 0.95 quantile of examples that remain in the calibration set after the initial selection step. This is because the data generation is such that examples in the central cloud tend to have larger nonconformity scores $S_{Y_i}(X_i)$ than the examples that are not in the central cloud, and most of the examples from the central cloud were eliminated from the calibration sample with the initial selection step. So the remaining nonconformity scores after the initial selection step tend to be smaller. Since the nonconformity scores of the examples from the test sample are unchanged, the improved calibration set after the initial selection step results in *p*-values that tend to be smaller when testing the null group, and in particular there are many more *p*-values that are below the 0.05 level.

### H.3. *Three classes of animals*

For the class-conditional model with label shift described in § 6, Figure 14 provides the resulting FCR and power measures.
(ii) The selection $\mathcal{S} = \text{BH}(\mathbf{q})$ of InfoSP controls the FDR at level $\alpha$ in the following sense:

$$ \sup_{P_{XY}} \mathbb{E}_{(X,Y)\sim P_{XY}} \left[ \frac{\sum_{i \in \mathcal{S}} 1\{Y_{n+i} \le y_0\}}{1 \vee |\mathcal{S}|} \right] \le \alpha. $$

(iii) The prediction intervals are of the form $C_{n+i} = \{y > y_0 : S_y(X_{n+i}) \le S_{(n_\alpha)}\}, i \in \mathcal{S}$, where $S_{(1)} \le \dots \le S_{(n)}$ are the ordered calibration scores $S_{Y_j}(X_j)$, $1 \le j \le n$ (with $S_{(n+1)} = +\infty$), and $n_\alpha = \lceil (1 - \alpha|\mathcal{S}(\mathbf{p})|/m)(n+1) \rceil$.

(iv) These prediction intervals control the FCR at level $\alpha$ in the sense of (15).

In other words, the above result complements the multiple testing procedure of Jin and Candes (2023), by providing in addition FCR controlling informative prediction sets on the selected outcomes.

*Proof.* The proof is direct with Example 2.1 and monotonicity (for (i)), Lemma 2.1 (for (ii)), Remark 2.1 (for (iii)), Theorem 3.1 (for (iv)). □

Obviously, a similar result holds for InfoSCOP, for any initial selection step $\mathcal{S}^{(0)} \subset [r+1, n+m]$ that satisfies the permutation preserving Assumption 3. Corollary 1.1 is illustrated on Figure 8 when $\mathcal{S}^{(0)}$ is taken here has BH($\mathbf{q}$) at level $2\alpha$ with the score $S_y(x) = (\mu(x) - y)/\sigma(x)$. The comments are qualitatively similar to those of Figure 7: when covering errors are less likely on the selection, the improvement of InfoSCOP over InfoSP is substantial.

<figure><img src="image_11.png" /><figcaption>Fig 8: Same as Figure 7 with Jin and Candes (2023) type selection and one-sided prediction intervals, see § I.2.</figcaption></figure>
<figure><img src="image_20.png" /><figcaption>Fig. 6: Selecting informative prediction sets in the class-conditional setting. FCR (left column), resolution-adjusted power (middle column), and the expected fraction of covering prediction sets (right column) versus SNR in a classification setting where prediction sets excluding a null class are of interest (top two rows) and when prediction sets excluding a trivial class are of interest (bottom two rows). The class probabilities in the calibration sample are 0.15, 0.1, and 0.75; in the test sample, we have a small label shift (rows 1 and 3) and a large label shift (rows 2 and 4). The number of data generations was 2000, and $n = m = 500$.</figcaption></figure>
<table><caption>TABLE 1</caption><thead><tr><th></th><th>CC</th><th>InfoSP</th><th>InfoSCOP</th></tr></thead><tbody><tr><th>FCR</th><td>0.150</td><td>0.030</td><td>0.081</td></tr><tr><th>Power</th><td>16.9</td><td>12.6</td><td>15.8</td></tr><tr><th>Res-adjusted Power</th><td>2.81</td><td>1.73</td><td>2.37</td></tr></tbody></table>

The FCR, power, and resolution-adjusted power, for the three procedures considered for the yeast application (see text). The resolution adjusted power is the expected number of discoveries divided by the length of the prediction interval (since the length is fixed in this analysis). Based on $B = 10000$ repetitions.

## 1.3. Predicting gene expression from promoter sequences

Vaishnav et al. (2022) fit a convolutional neural network model on tens of millions of random promoter sequences with the goal of predicting the expression level of a particular gene induced by a promoter sequence. They then used the model’s predictions to study the effects of promoters. They evaluated the performance on 61150 labeled sequences that were not used in training the model. The 61150 pairs of predicted and measured expressions will be used as a basis for a simulation to evaluate the performance of classic conformal, InfoSP, and InfoSCOP.

Suppose an analyst considers $m = 50$ new random promoter sequences. From these, the analyst is interested in the prediction intervals for promoter sequences that are under-expressed. Importantly, the range of plausible values (i.e., a two-sided prediction interval) are of interest to the analyst, since it is not enough to establish with confidence that the expression is below a certain value, but also to know that the expression is above zero and evaluate the plausible extent to which it exceeds zero.

Suppose the analyst has access to the expression and predicted expression of $n = 200$ random promoter sequences. These examples will serve as the calibration sample.

We consider the score function $S_y(x) = |\mu(x) - y|$, where $\mu(x)$ is the predicted expression (more sophisticated score functions, such as the locally weighted or quantile based functions discussed in § 4, cannot be used since we only have access to $\mu(x)$ from the prediction machine). We define an example as informative if its prediction interval is entirely below the threshold $a = 9$. Therefore, the $p$-value for testing the (random) null hypothesis that the example is not informative is

$$q_i = \max_{y \ge a} \bar{p}_i^{(y)} = \bar{p}_i^{(a)} \times \mathbf{1}\{\mu(X_{n+i}) < a\} + \mathbf{1}\{\mu(X_{n+i}) \ge a\},$$

see Corollary 4.1 with $b = +\infty$.

In order to assess our methods using the 61150 labeled native yeast promoter sequences, we randomly sample labeled and unlabeled data sets as follows. For each of $B = 10000$ trials, we randomly sample $n+m$ promoters, out of which $n$ points serve as the labeled calibration data set and $m$ points as the unlabeled test data set. The results are summarized in Table 1. Due to selection of informative examples, the classic conformal procedure that constructs each prediction interval at level $1 - \alpha = 0.9$ fails to control the FCR at the $\alpha = 0.1$ nominal level. InfoSP provides the required control, but at the cost of some non-negligible power loss. InfoSCOP applies an initial selection step in which examples from the test sample and the second half the calibration sample are forwarded to InfoSP, only if their conformal $p$-value, computed using the first half of the calibration sample, for testing the (random) null hypothesis that the example is not informative, is at most $\alpha$. InfoSCOP controls the FCR at level $\alpha$ and has only slightly reduced power compared to classic conformal.

Figure 9 shows the prediction intervals for a single data generation. InfoSCOP selects almost the same number of informative examples as classic conformal, but all prediction intervals constructed with InfoSCOP cover the true value. Moreover, the lower bounds are all above zero. In practice, following such an analysis, the analyst can use the list of prediction intervals from
Classical conformal

FCR = 0.205

Res.Adj.Pow = 31

<figure><img src="image_21.png" /></figure>

FCR = 0.00664

Res.Adj.Pow = 19

<figure><img src="image_22.png" /></figure>

FCR = 0.128

Res.Adj.Pow = 36.5

<figure><img src="image_23.png" /></figure>

FCR = 0.00495

Res.Adj.Pow = 71.8

<figure><img src="image_24.png" /></figure>

InfoSP

FCR = 0.0687

Res.Adj.Pow = 17.5

<figure><img src="image_25.png" /></figure>

FCR = 1e-04

Res.Adj.Pow = 5.41

<figure><img src="image_26.png" /></figure>

FCR = 0.0742

Res.Adj.Pow = 25.5

<figure><img src="image_27.png" /></figure>

FCR = 0

Res.Adj.Pow = 45.7

<figure><img src="image_28.png" /></figure>

InfoSCOP

FCR = 0.081

Res.Adj.Pow = 19.4

<figure><img src="image_29.png" /></figure>

FCR = 0.0948

Res.Adj.Pow = 20

<figure><img src="image_30.png" /></figure>

FCR = 0.0856

Res.Adj.Pow = 27.5

<figure><img src="image_31.png" /></figure>

FCR = 0.0894

Res.Adj.Pow = 143

<figure><img src="image_32.png" /><figcaption>Fig. 7: Informative selection by excluding the null value $y_0$ in the (homoscedastic or heteroscedastic) regression case, see § G.1.</figcaption></figure>
InfoSCOP to the next stage: knowing with confidence the prediction intervals for informative promoters for this set can help the future design of *n* + *m* examples with even lower expressions (see Vaishnav et al. (2022) for uses for the design of promoters for low gene expression). There will be a lab cost of culturing the yeast of *n* examples to actually measure the expression level for *n* promoters to serve as the calibration data, but *m* can be quite large.

<figure><img src="image_12.png" /><figcaption>Fig 9: The observed versus predicted values for yeast expression, as well as the classic conformal prediction intervals upper and lower bounds (red lines), and the InfoSCOP prediction intervals (purple vertical lines), for a single data generation (see text in § 1.3 for details). The false coverage proportion for classic conformal is 0.176 and 0 for InfoSCOP. Classic conformal reports 17 informative examples, but three examples are not covered by their prediction interval. InfoSCOP reports 16 informative examples (a subset of the 17 reported by classic conformal, including the three examples with non-covering classic conformal prediction intervals).</figcaption></figure>

**Remark I.1.** The selection in the above analysis fits within the Jin and Candes (2023) selection framework outlined in the previous section, but with a different score function than the scores suggested Jin and Candes (2023), since our score function is selected to provide two-sided prediction intervals.

## I.4. Comparison with existing selective prediction sets

Figure 10 is useful to visualize the difference between the selection proposed by Bao et al. (2024) and our informative selection. We display selective prediction intervals for two FCR controlling selections proposed in Bao et al. (2024): SCOPa uses a thresholding rule $S = \{i \in [m] : \mu(X_{n+i}) \ge y_0\}$, while SCOPb selects the largest $\mu(X_{n+i})$'s. Each time, a suitable conditional conformal prediction set is built in Bao et al. (2024) and reported in Figure 10 in green and purple. As one can see, either the selection is too conservative, or the prediction interval could include the nominal $y_0$. This is not the case of InfoSCOP which always exclude $y_0$ by essence.
Classical conformal

FCR = 0.163

<figure><img src="image_33.png" /></figure>

FCR = 0.0632

<figure><img src="image_34.png" /></figure>

InfoSP

FCR = 0.0775

<figure><img src="image_35.png" /></figure>

FCR = 0.0212

<figure><img src="image_36.png" /></figure>

InfoSCOP

FCR = 0.0877

<figure><img src="image_37.png" /></figure>

FCR = 0.0941

<figure><img src="image_38.png" /><figcaption>Fig. 8: Same as Figure 7 with Jin and Candes (2023) type selection and one-sided prediction intervals, see § G.2.</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>Fig. 9: Comparison to previous conformal selective inferences in the regression case: new informative prediction intervals (InfoSCOP) versus non-informative prediction intervals (SCOPab) (left: homoscedastic Gaussian regression with perfect variance prediction and errors in the mean prediction; right: heteroscedastic Gaussian regression with perfect mean prediction and errors in the variance prediction). Informative means here prediction intervals that do not contain $y_0$ (dashed line), see § G.3.</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Fig 10: Comparison to previous conformal selective inferences in the regression case: new informative prediction intervals (InfoSCOP) versus non-informative prediction intervals (SCOPab) (left: homoscedastic Gaussian regression with perfect variance prediction and errors in the mean prediction; right: heteroscedastic Gaussian regression with perfect mean prediction and errors in the variance prediction). Informative means here prediction intervals that do not contain $y_0$ (dashed line), see § 1.4.</figcaption></figure>

# Appendix J: Additional illustrations in the classification case

## J.1. Three classes, each bivariate normal

For the iid model, as described in § 5.2, we demonstrate the initial selection step in infoSCOP for excluding a null class in one realization of the data generation in Figure 11.

For non-trivial classification, as described in § 5.3, we provide numerical results in Figure 12.

## J.2. Three classes, each a mixture of bivariate normals

For the iid model, as in § 5.2, we consider the balanced and unbalanced cases of 3 classes. However, here we consider that each class comes from a mixture of two bivariate normals, where one component of the mixture is identical in all three classes, and given the class there is probability half of coming from that common mixture component. Thus the overlap between the classes is much greater than in the settings considered in § 5.2.

In Figure 13 we show that the qualitative conclusions with regard to the respective procedures are as in § 5.2. Interestingly, in the hardest setting at the bottom row we see that the power of InfoSCOP is even larger than classic conformal. This is because the initial selection step not only prevents paying too much for selection, but it can also improve the calibration set for computing the conformal p-values after selection. In this hard setting, there is a large improvement of the tail probability that matters for inference. We demonstrate this in one realization of the data generation in Figure 14, where we can see that the 0.95 quantile of the original calibration set is much larger than the 0.95 quantile of examples that remain in the calibration set after the initial selection step. This is because the data generation is such that examples in the central cloud tend
<figure><img src="image_40.png" /><figcaption>Fig. 10: The test sample in a single data generation for selecting informative prediction sets that exclude a null class. The setting is that of unbalanced classes, and the SNR is 3. The data points from the test sample of each of the three classes, where the null group is in blue: left panel for the entire test sample, right panel the remaining examples from the test set after the initial selection step.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Fig 11: The test sample in a single data generation for selecting informative prediction sets that exclude a null class. The setting is that of unbalanced classes, and the SNR is 3. The data points from the test sample of each of the three classes, where the null group is in blue: left panel for the entire test sample, right panel the remaining examples from the test set after the initial selection step.</figcaption></figure>

to have larger nonconformity scores $S_{Y_i}(X_i)$ than the examples that are not in the central cloud, and most of the examples from the central cloud were eliminated from the calibration sample with the initial selection step. So the remaining nonconformity scores after the initial selection step tend to be smaller. Since the nonconformity scores of the examples from the test sample are unchanged, the improved calibration set after the initial selection step results in *p*-values that tend to be smaller when testing the null group, and in particular there are many more *p*-values that are below the 0.05 level.

### J.3. Three classes of animals

For the class-conditional model with label shift described in § 6, Figure 15 provides the resulting FCR and power measures.
<figure><img src="image_41.png" /><figcaption>Fig. 11: Selecting non-trivial prediction sets in the classification iid model, in the case of balanced classes (top row) and unbalanced classes (bottom row), for the Gaussian mixture model with *K* = 3 classes described in § 5.2. FCR (first column), resolution-adjusted power (second column) and the expected fraction of covering prediction sets (third column) versus SNR. The number of data generations was 2000, 1000 data points were used for training, and *n* = *m* = 500.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>Fig 12: Selecting non-trivial prediction sets in the classification iid model, in the case of balanced classes (top row) and unbalanced classes (bottom row), for the Gaussian mixture model with K = 3 classes described in § 5.2. FCR (first column), resolution-adjusted power (second column) and the expected fraction of covering prediction sets (third column) versus SNR. The number of data generations was 2000, 1000 data points were used for training, and *n* = *m* = 500.</figcaption></figure>
<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /><figcaption>Fig. 12: Selecting informative prediction sets in the iid setting with a large overlap between three classes. FCR (left column), resolution-adjusted power (middle column) and the expected fraction of covering prediction sets (right column) versus SNR for: balanced classes for minimally informative prediction sets (first row) and for prediction sets excluding a null class (second row); unbalanced classes for minimally informative prediction sets (third row) and for prediction sets excluding the largest class (fourth row). Each class is a mixture of a common component and a unique component, see § H.2 for details. Based on 2000 data generations, 1000 data points were used for training, and $n = m = 500$.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Fig 13: Selecting informative prediction sets in the iid setting with a large overlap between three classes. FCR (left column), resolution-adjusted power (middle column) and the expected fraction of covering prediction sets (right column) versus SNR for: balanced classes for minimally informative prediction sets (first row) and for prediction sets excluding a null class (second row); unbalanced classes for minimally informative prediction sets (third row) and for prediction sets excluding the largest class (fourth row). Each class is a mixture of a common component and a unique component, see § J.2 for details. Based on 2000 data generations, 1000 data points were used for training, and *n* = *m* = 500.</figcaption></figure>
<figure><img src="image_44.png" /><figcaption>Fig. 13: The setting is that of unbalanced classes, with each class having probability half of being in the same mixture component, and the SNR is 8. In the first row, the data points from each of the three classes, where the null group is in blue: top left panel for the entire test sample, top right panel the remaining examples after the initial selection step. In the second row, left panel, their *p*-values using the entire calibration set (black circles) and using the examples from the calibration set remaining after the initial selection step (blue triangles). In the second row, right panel, the empirical CDF of the nonconformity scores for true classes in the entire calibration set (gray) and using the examples from the calibration set remaining after the initial selection step for the of the (blue).</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Fig 14: The setting is that of unbalanced classes, with each class having probability half of being in the same mixture component, and the SNR is 8. In the first row, the data points from each of the three classes, where the null group is in blue: top left panel for the entire test sample, top right panel the remaining examples after the initial selection step. In the second row, left panel, their *p*-values using the entire calibration set (black circles) and using the examples from the calibration set remaining after the initial selection step (blue triangles). In the second row, right panel, the empirical CDF of the nonconformity scores for true classes in the entire calibration set (gray) and using the examples from the calibration set remaining after the initial selection step for the of the (blue).</figcaption></figure>
<figure><img src="image_45.png" /><figcaption>d) Non-null classification with label shift</figcaption></figure>

<figure><img src="image_46.png" /></figure>

Fig. 14: For the class-conditional model, the FCR, average size of the selected, SR, and resolution-adjusted power for the methods, for $\alpha = 0.1$.
<figure><img src="image_18.png" /><figcaption>Fig 15: For the class-conditional model, the FCR, average size of the selected, SR, and resolution-adjusted power for the methods, for $\alpha = 0.1$.</figcaption></figure>