Conditional Testing based on Localized Conformal $p$-values
Authors: Xiaoyang Wu
Lin Lu
Zhaojun Wang
Changliang Zou
Venue: ICLR 2025 Poster (2025)
Keywords: Conditional testing; Conformal inference; False discovery rate; Family-wise error rate; U-statistic.
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose localized conformal $p$-values within a conformal inference framework to address several conditional testing problems.

Abstract: In this paper, we address conditional testing problems through the conformal inference framework. We define the localized conformal $p$-values by inverting prediction intervals and prove their theoretical properties. These defined $p$-values are then applied to several conditional testing problems to illustrate their practicality. Firstly, we propose a conditional outlier detection procedure to test for outliers in the conditional distribution with finite-sample false discovery rate (FDR) control. We also introduce a novel conditional label screening problem with the goal of screening multivariate response variables and propose a screening procedure to control the family-wise error rate (FWER). Finally, we consider the two-sample conditional distribution test and define a weighted U-statistic through the aggregation of localized $p$-values. Numerical simulations and real-data examples validate the superior performance of our proposed strategies.

Published as a conference paper at ICLR 2025
CONDITIONAL TESTING BASED ON LOCALIZED CON- FORMAL V-LUUS
Xiaoyang Wu", Lin Lu", Zhaojun Wang, Changliang Zou't NITFID, School of Statistics and Data Science, LPMC and KLMDASR and LEBPS, Nankai University, Tianjin, China xywuêma i 1. nankai. edu. en, dinlullon gma il com, z jwangnkk 126. com, zoue edu. en
ABSTRACT
In this paper, we address conditional testing problems through the conformal in- ference framework. We define the localized conformal p-values by inverting pre- diction intervals and prove their theoretical properties. These defined p-values are then applied to several conditional testing problems to illustrate their practicality. Firstly, we propose a conditional outlier detection procedure to test for outliers in the conditional distribution with finite-sample false discovery rate (FDR) control. We also introduce novel conditional label screening problem with the goal of screening multivariate response variables and propose a screening procedure control the family-wise error rate (FWER). Finally, we consider the two-sample conditional distribution test and define weighted U-statistic through the aggrega- tion of localized p-values. Numerical simulations and real-data examples validate the superior performance our proposed strategies.
INTRODUCTION
Nowadays, conformal inference has become an increasingly popular framework for quantifying uncertainty of machine learning models. Suppose we have i.i.d. training data D, = {(X )= and test data D2 {(22j. /- where test responses {Y2 are unobserved. The goal is to by
construct prediction intervals P1(X2;) for X2j D2 with marginal coverage guarantee
Pr(Y2/ PI(X2;)) 1-
(1.1)
By sample splitting or cross-fitting, conformal prediction methods (Vovk et al., 2005) can be coupled with any machine learning algorithm to construct distribution-free prediction intervals with finite- sample coverage guarantee. However, the marginal coverage (1.1) alone is not sufficient for an efficient prediction interval. A marginally valid prediction interval could have miscoverage rate much higher than a some subgroups of the data. Therefore, the conditional coverage is also an important aspect:
Pr(Y2, E PI(X2;) X2j = 21
(1.2)
Although appealing, achieving (1.2) in finite-sample and distribution-fre context is impossible (Vovk, 2013). Recent works have proposed many methods to construct prediction intervals with approximate asymptotic conditional coverage guarantee by either modifying the calibration step (Lei & Wasserman, 2013; Guan, 2023) or using different score functions (Romano et al., 2019; Chernozhukov al., 2021).
In addition to prediction intervals, the conformal inference framework is also valuable for other in- ference problems. With similar calibration procedure, the conformal p-value is defined address various testing problems. This includes classical two-sample tests (Hu & Lei, 2023) and multiple testing problems such as outlier detection (Bates et al., 2023; Zhang et al., 2022) and data selec- tion/sampling (Jin & Candès, 2023b; Wu et al., 2023). Therefore, as a parallel development, our
"Equal contribution. Corresponnence Changliang Zou, Email: zoucl vannai..cd..cn.

---

Published as a conference paper at ICLR 2025
paper moves beyond conditional prediction intervals and defines conformal p-values tailored for conditional testing problems.
We define localized conformal p-values by leveraging recent works Guan (2023) and Hore & Barber (2024) which constructed prediction intervals to adapt to the conditional distribution of the response. We present some fundamental properties of our defined p-values and demonstrate why they ef- fectively resolve these problems. More importantly, we also consider several non-trivial applications of localized conformal p-values, encompassing various testing problems with different errоr criteria (e.g.. FDR, FWER or type error). We show that the proposed testing rule can ensure valid corre- sponding error rate control. Here we provide a brief overview of those problems and defer rigorous formulations to later sections.
Conditional outlier detection: Outlyingness in data sets may come from various forms of heterogeneity. One of the important cases is that a few individuals in the test data do not share the same regression functions with majority (Peng al., 2023). This amounts to detecting outliers in the functional relationship between the response and covariates that can be represented by the conditional distribution of Y X. This is common in many spatial temporal contexts where the scale or variance of the response variable varies with location or time (Catterson al., 2010).
Conditional label screening: Consider scenario where the response Y is multivariate. Our goal is to determine whether each component of the response satisfies a pre-specified rule for unlabeled data point. For example, in the LLM factuality problem (Mohri & Hashimoto, 2024; Cherian et al., 2024) we have a vector of claims output by a LLM. It is desirable to screen false claims ensure reliability. In this example, the conditional performance of the screening procedure is important, which needs to be addressed properly to avoid extreme or imbalanced results.
Tampl conditional distribution test: The goal is to test for equality of conditional distributions Y between two samples (Hu Lei, 2023). This is also known as problem of comparison of regression curves (Dette & Neumeyer, 2003). Some contempo- rary applications include assessing casuality by testing whether the conditional distribution of the response given the covariates remains the same between two samples (Bühlmann, 2020) and testing whether the pre-trained model can still perform well on a new test sam- ple (Farahani et al., 2021).
1.1 OUR CONTRIBUTIONS
In this article, building on Hore & Barber (2024)'s randomly localized conformal prediction interval, construct localized conformal p-values, study their theoretical properties, and apply them to several conditional testing problems. Our contributions to the applications are as follows:
For the conditional outlier detection problem, we propose a procedure by utilizing the Benjamini-Hochherr (BH) procedure (Benjamini Hochberg, 1995) and conditional cali- bration technique (Fithian & Lei, 2022). Our procedure controls the FDR in finite-sample.
We elaborate a novel conditional label screening problem with rigorous formulation. We use the localized conformal p-value to construct screening procedure with knite-sample marginal FWER control. We also present a conditional FWER inflation bound of our pro- cedure demonstrate its robustness against conditionality.
We propose U-statistic for the two-sample conditional distribution testing problem by aggregating the localized conformal p-values. The asymptotic normality of the statistic under null and local alternatives is established.
We also validate methods through simulations and real-data experiments. With commonly used prediction algorithms, our proposed methods exhibit superiority terms of both validity and power compared to existing approaches.
1.2 RELATED WORKS
Conformal inference was originally designed to construct prediction intervals and enjoy valid and distribution-free properties with the only assumption that the data exchangeable. There have
2

---

Published as a conference paper at ICLR 2025
been several extensions to this framework. For example, Tibshirani et al. (2019) proposed weighted conformal prediction for covariate shift settings, and Romano et al. (2019) introduced conformalized quantile regression to account for heteroscedasticity. Other applications of the conformal inference framework include causal inference (Lei & Candès, 2021), selective inference (Bao et al., 2024), and survival analysis (Candès et al., 2023), among others. Besides conformal inference, also briefly review recent literature about our application problems in Appendix B.
1.3 NOTATIONS
The 1{-} the indicator function and || 1122-1 |l Iloo are the L2- and Les -orm. The [n]] denotes the set {1, (1.2....,7) n.. The 6₂ denotes the point mass at value z. The notation Q(a; P) denotes the a-th quantile of distribution The subscripts of E and Pr indicate distributions of random variables in the expectations and probabilities.
LOCALIZED CONFORMAL V-VALS
In this section, we first revisit the definition of the split conformal prediction and its localized ex- tensions by Guan (2023) and Hore & Barber (2024) in Section 2.1. Then we propose the localized conformal p-values by inverting the localized prediction intervals Section 2.2. At last we discuss basic properties of the defined p-vuuues in Section 2.3.
2.1 RECAP: CONFORMAL PREDICTION AND LOCALIZATION
We consider data pair E X V. Suppose we have D1 = (X1; Y.) ~ P1= PI Pl,y and D2 ((2g, Y2j
-- Y with each data point (X2j. Y2j) ~ P2.j = P2, x P2 In different problems, the responses Y25 of the second sample can be observed or unobserved.
In the classical split conformal prediction, D1 is divided into the training and calibration sets Di DD U De with index sets T U C (1, ... The training set Df is used to train prediction function (i() for the response Y and construct non-conformity function V(X, Y) which measures the similarity between the prediction and the true response. We then apply the tile threshold
score function to the calibration set De compute calibration scores (VI and obtain the quan-
-1(1-- 1-
1
Σóv ov. +
C + 800
1
(2.1)
cI+1
iEC
The split conformal prediction interval defined as PI(X2;) = ty V(X2,, 9) S ). Dj {(X2j. } exchangeable, the prediction interval PI(X2)) is samiee valid in the sense that Pr(Y2; E PI((2))) 2 1-a.
However, calibrating marginally with (2.1) does not account for the local information of the test point. Guan (2023) proposed the localized conformal prediction by calibrating with the quantile of weighted empirical distribution
Ta =
0 (1-10
H HX X2j)ōv, + (X2j, X23)000
SEC
where Hx(xx') Etec H (2)++ the similarity between its two arguments. Here â is the adjusted level to guarantee finite-sample coverage. Despite its efficiency, the LCP method needs compute the adjusted level o, which complex and computationally inefficient. As an improvement, Hore & Barber (2024) proposed randomization technique to circumvent level adjustment. Their method first samples Xaj from the distribution H H(X2j; +). takes the threshold as
H(xx')
for some kernel function H(,,)) characterizing
L = Q (1 (1-ax) Σ i''(, X2j)ōvs: + fr (X2j, X21)600
SEC
and define prediction interval as (X,))))) {v. V(X2j, v) 7n,1,
(2.2)
3

---

Published as a conference paper at ICLR 2025
where (x, Enec H(X.a. X2, x2,)+ H (2,)) If ((X2,, Y2j)) U D exchangeable, can
x(x.x')
compute the density ratio of X2j and X11 conditional on X2j as
=c (x, X2j) x X2)); dP 2 (x) fxxx(x) H( X2j) (2)
(x) X2j)
(2.3)
dPixx
which matches the weights in the empirical distribution up constant By the weighted exchangeabilityy in Tibshirani et al. (2019), the randomly localized confor- mal prediction interval is finite-sample valid in the sense that Pr(Y2j E (l1.))))))
Pr (X(2),, Y24)= S To. X2j) 2 a.
2.2 FROM PREDICTION INTERVALS TO D-VALUSS
Motivated by the capability of (randomly) localized conformal prediction capture local informa- tion, we invert these prediction intervals to construct the localized conformal p-value. Due to its simplicity, follow Hore& & Barber (2024) and take the randomization technique in (2.2).
xxx ( be a bi-variate kernel function with bandwidth h, where K(-) is
Let H(x x') = Nor
kernel density function and is the dimension of the feature vector x under consideration. Here (-) be taken the Gaussian kernel, the box kernel or any other nonparametric kernel function as long as is a symmetric density function. Define the localized conformal p- p-value for (X2j, Y25) E D2
PL.j Exec H(X X2) < Vi.) + +, H(X2) 2/ X2j)
(2.4)
Eiec H X2,) H (X2), X2,)
where Ej U|O, 1] is an independent random variable and X2j is randomly sampled from density H (X2 The non-conformity score V may depend on both X and Y or only on X, contingent on the specific problem. The PL.j can be viewed as a localized counterpart of the conformal p- value investigated by Bates et al. (2023) by using weighted empirical distributions. To simplify terminology, we will use CP and LCP to refer to the unweighted conformal p-value and our localized conformal p- value, respectively the rest of the article. Since we do not discuss prediction intervals, this should not lead to confusion.
2.3
BASIC PROPERTIES
In this section we state some basic properties of the localized conformal p-value. The first property its finite-sample validity.
Lemma (Finite-sample validity). For all 0 S a under the condition P1 P2.j, the localized conformal p-value satisfies Pr(pr.j I a:) S a. Furthermore, if the score VXX,V Y) has a continuous distribution, then Pr(pL. a) a.
This lemma is a direct corollary of the weighted exchangeability of ((22j. 2j Y2;) UC given Xoj. By leveraging this property, the LCP can be used for multiple testing to guarantee finite-sample FDR or FWER control.
By the nature of the local-weighting scheme, the LCP can adapt to potential covariate shifts. The following lemma is an analog of the robustness result in Hore & Barber (2024).
Lemma (Robustness against covariate shift). Under the condition Py,ri = P2.3 denote the dP2
covariate density ratio as g(x) al',x
(z). The LCP satisfies
Pr(pL:: a) + 1 Ex- flg(X + hU) -
where the distribution PH.x has density function fH.x(x) Ex-P, (X)x
This lemma gives deviation bound of the distribution of LCP from uniform distribution under covariate shift. The excess term will be small with a small h when the density ratio function satisfies some regularity conditions. For example, if g is Lipchitz continuous, the excess term will
4

---

Published as a conference paper at ICLR 2025
be of order O(h) vanish as h + O. For another example, if we want to perform conditional testing on some fixed region C. X, we can take g(z) = 2f E / Pr(X E B) with the shifted covariate distribution being Pi,x conditional on B. The excess term is then dominated by Pr(d(X,, aB) s h), which is also an (hh) term. This indicates the ICP is approximately valid conditional on any fixed region B.
As an indirect power characterization, the next lemma studies the point-wise limit of the LCP func- tion which is defined
(L y) Eiec H(XIs, X)IlU S Via) + H(x, X)
Eiec H(X X) + H()
where = (x, y) and is sampled from (x, With a score function V chosen properly based on the specific problem, a larger score value will indicate stronger evidence against the pre- specified null hypothesis. Our defined p-value therefore reflects evidence against the null contained a single data point. For the p-value be powerful, the value of pL (X, should be small (X, is sampled under the alternative. Therefore, for a fixed score value under the alternative, we can regard p- p-value function be asymptotically more powerful than another if its limit function takes smaller value at
We need some regularity conditions which are commonly used in nonparametric estimations. Assumption 1. The following conditions hold for (X,Y) P1:
Y) has a continuous distribution with bounded density;
The conditional distribution of the score V = =V(X,Y) satisfies
[[Evix=x(v) Fvix=x (V)) L ||x x
for some constant 0,0 8 1. That the conditional distribution function Fvix varies smoothly with x.
The density function fi,((x) is continuous, and the conditional density function fi x) is continuous in x.
Lemma 3 (Asymptotic behavior). Assume Assumption I holds and the split ratio [C]/[Di| 7 for some constant ? 0, then the LCP function converges in probability
[x,, u) (1 (1-Fvvv -V))x=x(x)) Op (0,
reled
for any fixed (x, v), as h nid - co.
By the weak law of large number, the unweighted CP function satisfies
PCP (x, v) Evec Hu S Vi. +
& Ev(u).
|C| +1
we take h such that h 0, nh the LCP function converges to 1-Fvvx==x (v) in probability. Comparing the power then amounts to comparing the value of Fvix - (v) and 1 Fvv.) For a score value v under the alternative, our conditional testing problem can generally ensure uniformly small value of 1 - Ffvx=x(00 (v) since the signal lies in the deviation of the conditional distribution. In contrast, the value of - Fy (v) could be quite large for (x, y) pairs with small conditional scale V X even if these pairs sampled under the alternative. Although not uniformly more powerful, the LCP can identify signals in these regions of X' with small conditional scale V X x, which tends to be missed by the CP. This property further motivates us to apply the LCP on conditional testing problems.
where (v(.) is the marginal distribution function of score
3 APPLICATIONS ON CONDITIONAL TESTING
In this section, we apply the LCP on several conditional testing problems. We first provide a straight- forward application sample selection problem in Section 3.1. Then in Section 3.2 study the
5

---

Published as a conference paper at ICLR 2025
conditional outlier detection problem and adopt the conditional calibration technique to achieve finite-saplle FDR control. In Section 3.3 we introduce a novel conditional label screening problem and leverage the LCP to design screening procedure with FWER control. Our application the two-sample conditional distribution testing problem is deferred to Appendix due to space limits.
3.1 WARM-UP: BALANCED DATA SELECTION
Consider the sample selection problem which was investigated by Jin & Candès (2023b) and Wu et al. (2023). The test sample D2 (X2;) و=1 is unlabeled. The goal is to select test samples satisfying a pre-specified rule Y E AA. Therefore, whether to select the jth sample amounts to
conducting the following hypothesis test:
Huj- Y25 & A. versus H1j Y23 E A.
In this case, the CP can be accordingly defined
EiEC. T(V23 S V12 + E Pj
KiE Y1 & A}} +
and if pj a we select X2j. Such methods directly controls the per selection error rate (PSER), say
Pr(Xgj selected Y2, A) a.
Analogously, we can apply our proposed localized conformal p-value instead to achieve certain improvement terms of conditional performance. The LCP is defined
PrL.j Eiec,rugd H(,,;, X2,); TVV,, S V1. & H(X2,, X2y)
ExecYynEA (X,,,, X2y) + H(2 X2j)
where a non-conformity score function depending on A. We use 5, = 1.0 to indicate selecting X2j not, where 6j [(Pr... I a). As corollary of Lemmas 1-2, the following theorem provides the marginal and conditional properties of this simple selection rule.
Theorem (Finite-sample PSER control and conditional PSER bound). Suppose {(Xis 1i and {(X 1i, Yui) ) are exchangeable, the selection rule ò3 = 1(PLL, S a) can ensure finite- sample marginal PSER control Pr(ό, inflation bound is given by
Y2j A) S a. Moreover, the conditional PSER
PIX-PIX U (OU KO 2 2dx(x 0B))
Pr(6, = 1 Y2j $ 2 E a a 2
Pr(X2; EB Y22 $ AA
where fi.X.A is the conditional density of X1; Yn & A. PHx.AA has a density function fH. Ex-11xx.A (H(,,))) and 0B is the boundary set of B.
The second result is obtained by taking the function g(x) 1[x E BJ/ Pr(X2; B) in Lemma 2 and simplifying the deviation term. For general sets B of regular form (e.g., balls or hypercubes), the excess term will be small with a small h. This indicates that using the LCP for data selection can lead to a more balanced selection result since the PSER inflation for different sub-groups is bounded. For instance, by choosing selection probability (PSER) will be more evenly distributed among different genders and races via ICP. A related issue is addressed by Rava al. (2021), which focuses controlling group-wine wise error rates to mitigate imbalances, aligning closely with our objectives.
appropriate B, can expect that the burden of incorrect
3.2 CONDITIONAL OUTLIER DETECTION
In the conditional outlier detection problem, the available data consists of clean data Dj and test data D2 with potential outliers. Both samples are labeled with observed responses. The inliers in Dr have the same conditional distribution P2,,Y = Pl,ri with D, while the outliers may have different conditional distributions from each other. Detecting conditional outliers can be formulated as the following multiple testing problem:
Hlo P2, Pi,yix=x for almost all x, versus Hiy otherwise,
(3.1)

---

Published as a conference paper at ICLR 2025
where (X2j. Y23) E D2 is an inlier if Huy holds and outlier if 1H,, holds. Our goal is determine the detection set (or the rejection set equivalently) R S f1.2...., m based on the observed data D and D2. Denote I, O S f1....... m)
as the index sets of inliers and outliers. The rejection set R should contain as many indices in as possible while guaranteeing finite-sample false discovery rate (FDR) control
INR R V1
FDR = B(FDP) = E
Sa
Bates et al. (2023) utilized the conformal p-value to test for marginal outliers by applying the BH procedure on conformal p-values computed on the test data. However, this is generally not effective when testing for conditional outliers. As discussed in the previous section, the classical CP targets on the joint distribution of (X, Y) rather than Y X, and thus cannot identify all information in the conditional distribution of score variables. In light of the capability the LCP to capture deviations in conditional distributions, we can take it as refinement to detect conditional outliers.
As proved by Bates et al. (2023), the unweighted conformal p-values based on the same calibration set is positive regression dependent on a subset (i.e., PRDS), under which the BH procedure can still guarantee finite-sample FDR control. This property, however, does not hold for the weighted con- formal p-vauues. In order to achieve the finite-sample property, we adopt the conditional calibration technique (Fithian & Lei, 2022) to prune the rejection set output by the BH procedure.
To perform multiple testing with the LCP, first train the non-conformity score function V(x, y) on Dy and then compute scores (Vislec and {V25 1m on De and D2, respectively. After sampling
1m are computed in Eq. (2.4). Define the auxiliary p-values X 2j for each X2j. the LCP's {pr.i)
as
Eiec HX1,, X2 11V2, V1.) + 2X2, X2 V2
PL.
(1)
(3.2)
Exec H H X2,) H(2)) 2/. X2,)
for 1 E {1, .1.,,...,7... m3 {i}. Let R8-0 be the rejection set of the BH procedure applied on (PL.1- (PL.1>--.. .......-Pi..y- 11 O, Pi.j+t>.... ...........m.. and Rinit {j Pr.jj alog-m be the initial
())
(1)
pL,)
jection set. We determine the final rejection set by generating independent CI...., Sm ~ U[0, 1] and pruning Rini into
R| GIR S srr srrj..
R j:
Pt, S
(3.3)
m
where max rr: En pnns S aaRR-ol/m, C R S 2 2r}
The outlier detection procedure is summarized in Algorithm in Appendix D.
The following theorem shows that our detection procedure can guarantee finite-sample FDR control if the distribution of the covariates does not change.
Theorem 2 sample FDR control). Under the condition that PIx = P2.x. the final output R given by Algorithm 1 ensures FDR a.
Remark 1. Since the condition Pı,x = P2,x together with the null hypothesis PLYII = P2. lead to the equality of the joint distribution of (X,Y), Y), our method is not an exact testing procedure for (3.1). Instead, it can be viewed extended marginal outlier detection approach with remark- ably boosted power on conditional outliers. This aligns with our motivation, as the classical CP often fails detect conditional outliers effectively.
3.3 CONDITIONAL LABEL SCREENING
In the conditional label screening problem, the response variable is multivariate with Y = (Yi, Y2,.... . Ys) for some random variable S. The first sample D1 is the labeled data while the second sample D2 is unlabeled. The response vectors are denoted as Y& for 2,2. Our goal is to determine whether each component of the unobserved response Y2j = (Y2j.1,.... Y22..,2)) satisfies a pre-specified rule for each Xaj E D2. For each entry we define the screening rule as Y2j,a E A, with pre-chosen sets As- This can be formulated as multiple testing problem for each test sample
(Ya....... Yu.s,.
Hajs: : Y2j, & A.: versus Hijs: : Yei, E As; 15 S S2j.
(3.4)
7

---

Published as a conference paper at ICLR 2025
For example, Mohri & Hashimoto (2024); Cherian al. (2024) considered using conformal predic- tion techniques to improve the output factuality of large language models (LLM). Specifically, they transform the output of the LLM into set of claims and aim to construct filtered claim set that contains no false claims with high probability. Such application can be framed into our conditional label screening problem stated above. The covariate X2j includes the input prompt Paj. the out- put Raj of the LLM model and a claim vector C2 of length S2j summarized by another language model. The response vector is 0-1 vector with Y2g, claim is correct or not. Here we can take A,- {1} for every S S S2y to screen out false claims.
0 indicating whether the corresponding
To ensure reliability of the screening procedure, we seek to control the probability of failing to screen out any component of Y2 that does not meet the rule. Let 6y.- 1 or 0 indicate whether Y2,,s retained after screening. This can be rigorously formulated
S2s
Pr
Σ
HY22, & Az, = 1 0 aa
(3.5)
say, controlling the FWER for (3.4). However, the marginal FWER might not be sufficient in real problems. We can use the localization technique to improved conditional validity while still guaran- teeing marginal FWER control.
Since the test data D2 is unlabeled in the current problem, the non-conformity score function V should depend only on the covariate X. We can use the training data Dy to estimate the probability Pr(Y A..). This can be achieved by training classification models for each component of Y if the length S' is a fixed constant or fitting a joint classification model that outputs probability vector. In both scenarios, we can define the score vector
v(X2,) = (V25,1, Ve5,2,..., )
where each V2j,. approximates Pr(Y2j.. E A.). By this definition, we should reject Hoj, if V2j is large, and controlling the FWER amounts to examining the distribution of max [V2j. Y2j.s & A.). This motivates to define the localized p-value
Exec H(X1s, X2,) S v1i) H(22j. X2j)
PL.j.
(3.6)
Eiec H(X1, X2j) H(22,, X2j)
where Vi MaxsVVi,> Viso & As}. We can show that the PL.jas 's satisfy the group super- uniform property, i.e.,
Pr U E2s.
UpLie
(PLAs S sa} Sa,
and thus we can screen out components of Y2j with PL.ja S a. We summarize our label screening procedure in Algorithm in Appendix D.
We have the following result.
Theorem (Finite-sample FWER control). Suppose ((X 16: Y S1.) (X22), Y2j, S2 Im are exchangeable, then the label screening procedure given by Algorithm 2 ensures
FWER control
FWER Pr
Σ 1YY3 a & Aa,PL.js S a} >0
Sa.
Regarding the conditional property, ation bound for our procedure.
also establish the finite-sample conditional FWER devi-
Theorem 4 (Conditional FWER bound). Suppose the ussumptions in Theorem 3 hold. For any fixed set B C. X with Pr(X2/ e B) > 0, the conditional FWER has the following bound
Szj
Pr
Σ
1222 $ A..PLass Sa >
X25 B
Pix-PEXU-RO 2 h- d(X, 0B))
Sa + 2| fi,l
Pr(X2, E B)
where 0B is the boundary of set B.
8

---

Published as a conference paper at ICLR 2025
In this theorem, the inflation bound is similar to that in the second result of Theorem 1. By a similar rationale, this demonstrate the advantage of our method to mitigate conditional error rate inflation.
4 EXPERIMENTS AND EVALUATION
We provide synthetic and real-data experiment results to show the validity and efficiency of our proposed methods for two main applications, respectively in Sections 4.1-4.2. More comprehensive experiments for all our applications on conditional testing are detailed in Appendix
4.1 SYNTHETIC-DATA RESULTS FOR CONDITIONAL OUTLIER DETECTION
Data Description. For the conditional outlier detection problem, we consider heterogeneous linear regression model with label Y which the data are generated as follows:
Rd' with d' 10 and an additional time feature t E R. The model is Y = XB + (3 sin(2x t)) E, with U[, ~ U and E ~ [N, 1) independently. The coefficient vector is B = (0.5, 0.5, 0.5, 0,5, 0, 0). The test data contains 10% outliers following the model where r(t) = 3- (3+1 1.5+1..imm22 1)) and Pr(g = +1) 1/2. We also consider scenario without label Y in Appendix E. (Scenario B1).
Scenario AI: the covariate vector consists of X = (XI....,Xa:-1))
X1 ,
Y
We abbreviate our method as LCP-od (outlier detection) and compare it with a bench- mark method abbreviated CP. The CP method is implemented by computing the unweighted conformal p-values as defined in Bates et al. (2023) and then applying the BH procedure. For the LCP-od and CP methods, we use two score functions: CQR scores and absolute residual scores, and apply various regression or classification algorithms, which are detailed in Appendix E.
Benchmarks.
Results. The average FDP above nominal level and power across 500 replications for Scenario Al are shown in Figure 1. It can be seen that both the LCP-od method and the CP method accurately control the FDR around the nominal level, which validates their fiitte-saple FDR control property. In terms of power, the LCP-od method demonstrates higher power than the CP benchmark with both two score functions when the sample size n is relatively large. Compared with the absolute residual score, the CQR score leads to relatively higher power due to its adaptivity to the conditional distribution of X. However, the power gain is still significant after applying the LCP-od method, illustrating the advantage of local weighting. Moreover, the power of the LCP-od method grows significantly with the sample size n, while the power of the CP method only rises slightly. As discussed in the Section 2.3, the unweighted CP cannot identify those outliers with small conditional variance of the score. This explains why the power of the CP method remains "stuck" around the same value. In contrast, the power of the LCP-od method approaches demonstrating the full identifiability of the LCP-od method in testing for conditional outliers.
the sample size increases,
0.4
0.6-
02:
4000
2000
udto 2200
Natted i1 LCP-od C2
Figure I: FDP above the nominal level and power under different clean sample sizes n for Scenario AI. The test sample size is fixed at 2000 and the nominal level is fixed at 0.15.

---

Published as a conference paper at ICLR 2025
4.2 CONDITIONAL LABEL SCREENING ON HEALTH DATA
Dataset. We utilize the health indicators dataset from Kaggle (Kaggle, 2021) to demonstrate the performance of our conditional label screening method. The dataset comprises 70, 692 samples and 22 variables, including demographic attributes (e.g., sex, age, BMJ), lifestyle-related features, as well as several binary health indicators. These indicators capture whether an individual has con- ditions such as diabetes, coronary heart disease, or has experienced myocardial infarction, among others. Accordingly, we define the response variable Y as these disease-related binary labels, with the goal of predicting the risk of these conditions in test data. More specifically, the response vector (Yi, 22 Ya) {0, 1 is set as follows:
=
Y1: Indicates whether an individual has diabetes.
12: Indicates whether an individual has coronary heart disease or myocardial infarction.
. Yg: Indicates whether an individual has experienced stroke.
Consequently, the label screening targets are Y, E A, {1} for 2, We fix the size of the labeled data D, at n = 500 and the test data D2 at m = 2, 000, randomly sampling them from the original dataset without replacement each replication. We fix the nominal level at a 0.05, and apply two different algorithms to compute the score vectors: linear logistic regression (LL) and random forest (RF).
Benchmarks and performance measures. We compare our conditional label screening method via the LCP (abbreviated as LCP-Is) with the thresholding procedure without weighting (abbreviated as THR). The THR method is performed by only replacing the LCP by the classical unweighted CP. We calculate the empirical marginal FWER and conditional FWER (conditional on B for specific set B) for the screening procedure via LCP-Is and the simple thresholding rule, respectively, where (X sex female and BMI > 30). The precise definitions of two measures are detailed in Section 3.3.
Results. We show the average of measures for all m test samples across 500 repetitions. The results reported in Table 1. We observe that both benchmarks can ensure valid marginal FWER control, while only the label screening procedure via the LCP-Is is capable of controlling both conditional and marginal FWER simultaneously.
Table 1: Empirical conditional FWER (cFWER) and marginal FWER (mFWER) under nominal level a =
0.05 for health indicator dataset across 500 replications by using LL and RF algorithms, respectively. The bracket contains the standard error.
Method LCP-Is
cFWER
cFWERRF
mFWER
mFWER
0.0329 (0.018)
0.0372 (0.018)
0.0487 (0.009)
0.0507 (0.009)
THR 0.0740 (0.028)
0,0887 (0.029)
0.0500 (0.011) 0,0538 (0.012)
5 CONCLUDING REMARKS
We conclude the paper with two remarks. Firstly, the localized conformal prediction is efficient in capturing local or conditional information, but this comes at the cost of a much smaller effective sample size. In our simulation experiments, the power of localized methods grows significantly as the sample size increases in many scenarios. However, when the sample size is small, our proposed methods often exhibit lower power compared to other methods. Therefore, it would be beneficial to increase the effective sample size by utilizing additional data or modifying the definition of the LCP.
Secondly, we define the LCP with random sampling step for each test point to achieve finite-sample validity. However, the random sampling step introduces external randomness, which degrades the stability of the related methods. Additionally, the LCP is defined by computing the similarity be- tween covariates of the calibration data and the sampled covariate X instead of the test point X itself. Although these two issues are equivalent asymptotically, they generally differ in unite-samples. This makes the LCP not effective enough in characterizing local information. A potential solution is to use a different definition for the localized conformal valuee to ensure finite sample validity without randomization, which warrants further consideration.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Code for implementing our methods and reproducing the experiments and figures in our paper is available at https::/ thu.. com/ ments, please see the pseudo-codes for our all proposed algorithms in Appendix D and the imple- mentation details Appendix Proofs of all theoretical results are included in Appendix F.
ing, For details of our imple-
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We thank anonymous area chair and reviewers for their helpful comments. This research was supported by the National Key R&D Program of China (Grant Nos. 2022YFA 1003703, 2022YFA1003800, and the National Natural Science Foundation of China (Grant Nos. 123B2010, 11925106, 12231011, 12326325).
REFERENCES
Yajie Bao, Yuyang Huo, Haojie Ren, and Changliang Zou, Selective conformal inference with false coverage-statement rate control. Biometrika, pp. asac010, 02 2024. ISSN 1464-3510.
Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149-178 2023.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and power- ful approach multiple testing. Journal of the Royal Statistical Society: Series (Statistical Methodology), 57(1):289-300 1995.
Thomas Brooks, D Pope, and Michael Marcolini. Airfoil Self-Noise. UCI Machine Learn- ing Repository bttps://archhs ics. ccs.uci......................... www.wucw.eeeeee..... aedu/daaasee/291111
noise, Rei 2014.
Peter Bühlmann. Invariance, causality and robustness. Statistical Science, 35(3):404 2020.
Emmanuel Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. Journal of the Royal Statistical Society: Series B, 85(1):24-45, 2023.
Victoria M. Catterson, Stephen D. McArthur, and Graham Moss. Online conditional anomaly detection in multivariate data for transformer monitoring. IEEE Transactions Power Delivery, 25(4):2556225 22666,2564777 2010.
Andrea Cerioli. Multivariate outlier detection with high-breakdown estimators. Journal of the American Statistical Association, 105(489):147-15, 2010.
Yuchen Chen and Jing Lei. De-biased two-sample u-statistics with application to conditional distri- bution testing. arXiv preprint a0Xiv:2402.00/64, 2024.
John J Cherian, Isaac Gibbs, and Emmanuel Candès. Large language model validity via enhanced conformal prediction methods. arXiv preprint arXiv:2406.09771, 2024.
Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. Pro- ceedings of the National Academy of Sciences, 11844))=22007994118 2021.
Holger Dette and Natalie Neumeyer. Nonparametric comparison of regression curves: an empirical process approach. The Annals of Statistics, 31(3):880-8 2003.
Sarah M. Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Pattern Recognition, 58:121-134, 2016.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. A brief review of domain adaptation. In Advances in Data Science and Information Engineering, pp. 877-894, 2021.
11

---

Published as a conference paper at ICLR 2025
William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under depen- dence. The Annals of Statistics, 50(6):3091 - 3118, 2022.
Jiti Gao and Irene Gijbels. Bandwidth selection in nonparametric kernel testing, Journal of the American Statistical Association, 103(484):1584-1355 2008.
Leying Guan. Localized conformal prediction: generalized inference framework for conformal prediction. Biometrika, 110(1):33-50, 2023.
Peter Hall and Joel D. Hart. Bootstrap test for difference between means in nonparametric regres- sion. Journal of the American Statistical Association, 85(412):1039-1049 1990.
Charmgil Hong and Milos Hauskrecht. Mcode: Multivariate conditional outlier detection. arXiv
preprint 1r05
2015.
Rohan Hore and Rina Foygel Barber. Conformal prediction with local weights: randomization en- ables robust guarantees. Journal the Royal Statistical Society Series B: Statistical Methodology, pp. gkae103, 2024.
Xiaoyu Hu and Jing Lei. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statisticul Association, 0(0): 109) 2023.
Yuyang Huo, Lin Lu, Haojie Ren, and Changliang Zou. Real-time selection under general con- straints via predictive inference. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Ying Jin and Emmanuel J Candès. Model-free selective inference under covariate shift via weighted conformal p-values. arXiv preprint MiXi::2307709291, 2023a.
Ying Jin and Emmanuel J. Candès. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244):1-41, 2023b.
Kaggle. House sales in king county, usa. https://www https:/www.kaglee.coom htt.k//gggle..cmmaaaaaaattt
2016.
Kaggle. Diabetes health indicators dataset. https: / /www kagg htt.c//c///aaaaa.aaeaae
a alexteboul/ /
httttebbul/ddabbteseeeee abittseeeseeeee in wawtiiictossdaataseeeee 2021.
Jing Lei and Larry Wasserman. Distribution-fre free Prediction Bands for Non- parametric Regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71-999 07 2013.
ISSN 1369-7412.
Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal the Royal Statistical Society Series B: Statistical Methodology, 93(5):911-938, 2021.
Ziyi Liang, Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for -istriiiiitt testing with labelled outliers. Journal of the Royal Statistical Suciety Series B: Statistical Method- ology, pp. qkad138, 2024.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou, Isolation forest. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, pp. 413-422. IEEE Computer Society, 2008
Ariane Marandon, Lihua Lei, David Mary, and Etienne Roquain. Adaptive novelty detection with false discovery rate guarantee. The Annals of Statistics, 52(1):157 - 183, 2024.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guaran- tees. arXiv preprint arXiv:2402. 2024.
Liuhua Peng, Guanghui Wang, and Changliang Zou. Measuring, testing, and identifying hetero- geneity of large parallel datasets. Statistica Sinica, 33:1-22, 2023.
James L Powell, James H Stock, and Thomas M Stoker. Semiparametric estimation of index coeffi- cients. Econometrica, 57(6))1033-44 1989.
12

---

Published as a conference paper at ICLR 2025
Bradley Rava, Wenguang Sun, Gareth M James, and Xin Tong. A burden shared is a burden halved: A faimess-adjusted approach to classification. arXiv preprint arxtv:2110.05720, 2021.
Marco Riani, Anthony C. Atkinson, and Andrea Cerioli. Finding an unknown number of multivariate outliers. Journal of the Royal Stafistical Society: Series B (Statistical Methodology), 71(2):447- 466, 2009.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, pp. 911 911-938, 2019.
Xiuyao Song. Mingxi Wu, Christopher Jermaine, and Sanjay Ranka. Conditional anomaly detection. IEEE Transactions Knowledge and Data Engineering, 19(5):331664, 2007.
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal pre- diction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, 2019.
Vladimir Vovk. Conditional validity of inductive conformal predictors. Mach. Learn., 92(2-3): 349-376, sep 2013. ISSN 0885-6125.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. New York: Springer, 2005.
Xiaoyang Wu, Yuyang Huo, Haojie Ren, and Changliang Zou. Optimal subsampling via predictive inference. Journal of the American Statistical Association, 0(j)::1-2,, 2023.
Yifan Zhang, Haiyan Jiang, Haojie Ren, Changliang Zou, and Dejing Dou. Automs: Automatic
model selection for novelty detection with error rate control. In Advances in Neural Information Processing Systems, volume 35, pp. 19917-19929, 2022.
John Xu Zheng. A consistent test functional form via nonparametric estimation techniques. Journal of Econometrics, 75(2):263-2 1996.
13

---

Published as a conference paper at ICLR 2025
Supplementary Material for "Conditional Testing based on Localized Conformal p-values"
This supplementary material contains:
Preliminary terms for self-containment (Appendix A).
Additional related works (Appendix B).
An application on the two-sample conditional distribution test (Appendix C).
Additional details of our algorithms (Appendix D).
Additional experiments and implementation details (Appendix E).
Proofs of all the theoretical results (Appendix F).
A PRELIMINARY TERMS FOR SELF-CONTAINMENT
Here, we list the preliminary terms we use in the paper for the sake of clarity and self-containment.
FDR (Benjamini & Hochberg, 1995), false discovery rate, a widely-adopted error rate notion the field of multiple testing, is defined the expected proportion of incorrectly rejected null hypotheses as follows:
Ho n R
FDR
VI
where Ho is the unknown set of true null hypotheses, represents the set of rejected null hypotheses and then Ho n R is the set of false discoveries.
FWER, family-wise error rate, defined as the probability of falsely rejecting more or equal to hypothesis testing:
FWER Pr(V > 0),
where V is the number of false discoverics (or rejections).
PCER (PSER), per comparison (selection) error rate, is defined as follows:
PCER (E())nn,
where n is the total number of hypotheses.
B ADDITIONAL RELATED WORKS
Outlier detection. Classical outlier detection methods include multivariate outlier detection meth- ods that model assumptions identify outliers (Riani al., 2009; Cerioli, 2010), and ma- chine learning algorithms (Liu et al., 2008; Erfani et al., 2016). Most existing works on conditional outlier detection fall into these two categories (Song et al., 2007: Catterson et al., 2010; Hong & Hauskrecht, 2015). Recent studies have used the conformal inference framework to test for outliers with advanced machine learning models (Bates et al., 2023; Marandon al., 2024; Liang et al., 2024). Although these methods ensure finite-sample FDR control, they focus primarily on marginal cases and do not address outlier detection in conditional distributions.
Data selection and subsampling. Our introduced label screening problem is similar to label-based data selection (Jin & Candès, 2023a;b; Rava et al., 2021; Huo et al., 2024) and subsampling problems (Wu et al., 2023). These works consider semi-supervised setting where the selection sampling rule based on the value of the response variable. The key difference in our problem that we focus screening within the multivariate response vector for each data point, rather than selecting different data points from the unlabeled dataset. Additionally, existing works do not account for the conditional properties of their methods, which is the main concern of paper.
Two-sample conditional distribution test. Most existing works focus on testing the equality of two conditional moments (Hall & Hart, 1990; Dette & Neumeyer, 2003), which is less stringent than
14

---

Published as a conference paper at ICLR 2025
testing for equality of distributions. A notable contribution is Hu & Lei (2023), which proposed U-statistic based on the conformal inference framework by aggregating conformal p-values. Chen & Lei (2024) improved this method by de- de-biasing the functions estimated using machine learning algorithms. Our porposed test statistic, a kernel-weighted U-statistic, is closely related to Hu & Lei (2023) by aggregating localized conformal p-values instead.
C APPLICATION ON TWO-SAMPLE CONDITIONAL DISTRIBUTION TEST
In the two-sample conditional testing problem, the second sample D2 is labeled and i.i.d. following a potentially different distribution (X2). Y2j) ~ P2,x x P2,1|x from the first sample D1. The conditional distribution test can be formulated as
Ho P Xx= for almost all x, versus H. otherwise.
(C.1)
Hu Lei (2023) proposed wwopsa conditional distribution test based on the conformal infer- ence framework. To be specific, they first split both samples as D1 = Dr, LDe, and D2 Dr UD, with [Dc, = [Dcg n1, [PT, [Dral n2- For notation convenience, we perform an equal- sized sample splitting with score function and density ratio estimator
and let C1 = C2 = (1.....n1) ..) in this section. Thereafter, the
V(x, v) = fi x
O(x) = J2,x(x)
f2 x))
f1,((x)
trained DT UD₁, by fitting classification models distinguish DT. and Dy.. After comput- ing scores (Viehie,, and [Vijbeeca the weighted conformal p-values are obtained as
Dj 111 Liec, (i n1 Eiec I(X):)
E2.
The test statistic F 1 Exec, P3 is constructed by averaging these p-values, where Di [(22; < Ví-} + E Vid} and 22 is the variance estimator. Under the null hypothesis, they proved that /niT is asymptotically normally distributed and the rejection region is given by < (1
We extend the above strategy by aggregating the localized conformal p-values computed on the second sample. Since p-values are longer necessary to be finite-sample valid in this case, here we use a simplified variant of the localized conformal p-value without randomization
Eiec H(X (V2) S Vi. + & H(22j. X2,)
PSL.j
Eiec H(X 12, X2j) H(22j, X2))
(C.2)
and our proposed test statistic is defined as
Σ j=l H(X X2;) Dis
(C.3)
m
where D.j 1/2-1112 ITV22 < Vid C,I[V Vnd and 0 is the variance estimator: This test statistic is constructed by averaging unnormalized LCP's in Eq. (C.2) and performing standardiza- tion. From different perspective, is also related to the classical U-statistic for model checking problems (Zheng, 1996; Gao & Gijbels, 2008) in which the Diy's 's are replaced by the residuals. In- herited from the nice property of conformal techniques, our method enjoys model-agnostic features and allows us to employ of-te-of-the-rr art algorithms to construct efficient score function V that is able to better measure the discrepancy between two conditional distributions.
We need the following technical assumptions to establish the asymptotic normality of T.p. We also make the same regularity assumptions to those in Hu Lei (2023) without further declaration in theorems, which guarantee identifiability of the problem.
Assumption 2. Let V* v) = f2(y)))) (ylx) be the true conditional density ratio. Denote the oracle statistics as Vi = V* Y Y1a), V2z yx(2,,.Y2) and Dij 1/2 I[
V2J) EV(VV V2 Suppose that
15

---

Published as a conference paper at ICLR 2025
Voj has continuous distribution;
( (D) Piy)) op(1/ (n)))
Recall that we use classification construct the score function V(x, y) approximate the true conditional density ratio (x) in this problem. This assumption requires the approximation error is sufficiently small after local-weighting.
similar assumption has been made Hu & Lei (2023) with H li, X2,) replaced by (()):) These two assumptions are introduced both to ensure a vanishing asymptotic bias and achieve asymptotic normality in the presence covariate shift.
Theorem 5 (Asymptotic normality). Under the null hypothesis. suppose Assumption holds for Pı, P2 and h -> 0, nith 2 - xc. Further assume Pi.x = P2.x Assumplion holds. Then the test stutistic (C.3) is asymptotically normully distributed
Tw DE E
H(X
X
X2)
Dij
4 Z N(0,1),
Fn
where the variance estimator is given by
1
H X2;)
H
Dis
11
71
i=1
R1
f1
2
11 2
H(x1; X2,) 2
H H 1s X2y)
j=1
Note that the convergence of the score function only needed under covariate shift settings. Given the asympiotic normality property, we can construct our testing procedure summarized in Algo- rithm
Theorem (Behavior under local alternatives). Suppose Assumption I holds for P1, Pa, Assumption 2 holds, and ni 4 o > 0. Then we have
Vniōw
T
(1+ o1+))(1)) + 2 op()),
40
where 6. = =Exx-xxxXY.Y-PP UI Y) - V*(X, YDI and 2~ N(O, 1).
This thcorem is similar to Proposition in Hu & Lei (2023). The main difference lies in the form of "signal strength" 6w. which is 5 xx. >")~ (IV* V* Y')}} in their article. In comparison, the difference term in 8, is based on the same covariate value and therefore better captures the deviation in conditional distribution. This reflects the superiority of our proposed test statistic in the two-sample conditional testing problem.
D ADDITIONAL DETAILS OF OUR ALGORITTMS
Figure displays workflow. We also provide the detailed pseudo-codes for three algorithms in Algorithm I-Algorithm 3 below for the problems of conditional outlier detection in Section 3.2, conditional label screening in Section 3.3, and two-sample conditional distribution test in Appendix C, respectively.
E ADDITIONAL EXPERIMENTS RESULTS AND IMPLEMENTATION DETAILS
In this section, provide implementation details and additional experiments results for our three conditional testing applications, including: (1) more synthetic and real-data results for conditional outlier detection in Appendix E. and E.4; (2) synthetic data results for conditional label screening in Appendix E.2; and (3) synthetic and real-data results for two-sample conditional distribution test in Appendix E.3 and E.5.
16

---

Published as a conference paper at ICLR 2025
Applications
Theoretical gurranies
Balanced
PSEIt control
New tools
Conformity score
Localized conformal p-values
oullier detection
Conditional
Fizie-sampl
ML model
Conditional label screening
Fistessngle Conditinnl 1'antt hond
property
Two--ample conditional distribution lesl
Asyripttki connelly Bebestor under lool ahenotives
Figure 2: flow chart of our work.
Algorithm Conditional Outliers Detection via the LCP
Input: Clean data D {(X1,, Yı,)] and test data D2 {(X2j, {2,)) FDR target level a E (0, 1); Kernel function (---))
1: Randomly split D1 into DD U De, train the non-conformity score function V(x, y) on Dy and compute score values {Vixiec and {V2,)
mm on De and D2
Outlier detection
2: for = ,,...,m m do
3: Sample X2j from density H(22j, :) sample E U(0, 1] and construct the LCP PL.j in (2.4)
4: Compute auxiliary p-values (8)
PL. as in (3.2)
5: (BH procedure) Compute r = max r: 1+ E HPLL S ar/m) 2r}
6: Compute R, (1) (e # j: PL. S /r3/m}}
(0)
7: end for
8: Compute the first-step rejection set Rinit = {j PL.j I Rg-oo//m
9: Prune the initial set Riuit and obtain the final rejection set R as in (3.3)
Output: Detected conditional outlier set
Algorithm 2 Conditional Label Screening via the LCP
Input: Labeled data D, {(X,, Y and test data D2 (X2j) FWER target level a E (0, 1); Kernel function H(:-))
1: Randomly split Dj into D7 U De and train the non-conformity score function, compute score vectors (V(Xi)))ee (V(X2,) ) on De, D2 and summary scores (Vi))ee
SScreening
2: jor m do
3: Sample X2 from density H (X2,, and E, ~ U[
4: for s 1,... S2j do
5: Construct the LCP pr.j.s as in (3.6)
6: Determine the screening decision oj.- T(pr,jss S a)
7: end for
8: end for
Output: Screening decision vectors 6, = (6,,1,...,65)) for each S S m
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Two-sample conditional distribution test via aggregation of simplified LCP's
Input: Two samples D1 and D2; density ratio estimation subroutines A1, A2, kernel density ((),)).
the nominal type I error level E (0, 1)
1: Randomly split the two samples as DD = Dr, U Dc, and D2 = Drg u De.
2: Ø(-) = (1 E T1, _ T2)
3: o(:..) = Az (X Yu.), E T1, 2x Y2j). T2)
4: for jorj E C2 do
5: Sample Ey ~ 00, 1]. independently
6: Calculate Dij 1/2 I[V1, V2j V V25}
7: Calculate the kernel weights H (X X2j),
8: end for
9: Calculate the variance estimator 2h and obtain the weighted statistic Tu as in (C.3)
10: Reject Hg if (Tww) 2
Output: The decision for two-sample conditional distribution test
E. ADDITIONAL SYNTHETIC-DATE RESULTS FOR CONDITIONAL OUTLIER DETECTION
Implementation details. Since our goal to test for outliers conditional on or s in Scenarios A1 and B1, we only use these variables when computing the weights. For both scenarios, the kernel function of the LCP-od method is taken as the Gaussian kernel H x') = (2xh²)-442 exp( exp(-r [-||-- x' (211/(22²)) } with bandwidth & = (t/2)-1/(d+2 for d = 1, in Scenarios Al and B1. This corresponds the optimal convergence rate for 3 of score functions: the CQR score and the absolute residual of different regression algorithms. Similar Romano et (2019), for the CQR score, we consider using quantile neural networks (CQR-QNNY and quantile random forests (CQR QRF). For the absolute residual score, we take two regression algorithms: random forest (Res-RF), and support vector machine (Res-SVM). For Scenario B1, we use one-class classifiers for both the I.CPod and CP methods. We take three kinds of one-class classification algorithms: the isolation forest (IOF), 4-Nearest-Neighher (k-NN) with k 5 and one-class support vector machine (one-class SVM).
For Scenario AI, we consider two kinds
We consider either fixing a = 0.1 or 0.15, m 2000 and varying n {800, 1600, 2400, 3200, 4000} or fixing the sample size m 2000 and varying (0.05, 0.1, 0.15, 0.2).
Results. The FDP above the nominal level and power under different nominal levels for Scenario Al are shown in Figure 3. The pattern is quite similar to Figure in the main text. The exception here is that the CP method also has its power increasing with the nominal level a. This is reasonable since all methods will reject more hypotheses when the nominal error level increases.
We also consider scenario without label Y.
Scenario B1 (without label Y): Another example which does not include the label Y. To
fit our problem, we consider a spatial setting with X = (s, X*) where s is the spatial vari- able and X* contains the remaining features. We consider X E Rdd with (81,82) E R and X* E RRd -2 with d' = 50:
x ~ r()) Ta--2), r(s) = 0.2 2 0.9
where 81, 82 ~ U[-1, 1] independently. The test data contains 10% outliers with the same distribution for and different conditional distribution
x 8 (0, 4- r(s) 18--2).
Results. The results for Scenario B1 are shown in Figure 4-5. In Scenario B1, the power of the LCP- od method grows significantly with the sample size n and the nominal level a, while the power of the CP method only rises slightly. Our discussion for Scenario A1 also applies to these results. The only exception that here the CP method has a relatively higher fixed power than those in Scenario A1.
18

---

Published as a conference paper at ICLR 2025
COR QNN
DOR QRF
Ros RF
Res SUM
0.8
0.6
LANAE 201
0.2
0.0
0.6
0.2
0.06
0.1
0.15
0.2
0.05
0.1
0.15
0.2
0.05
0.1
0.15
0.2
0.05
0.1
0.15
0.2
0t
Method LCP-od
Figure 3: FDP above the nominal level and power under different nominal levels a for Scenario A1. The sample sizes fixed m 2000.
IDA
KVV
Ioo 5V
1.00
0.75
0.00
1.00
0.75
0.50
0.25
800
1600
2400
3200
4000
800 1800
2400
3000 4000
1000 2400 3200 4000
Method E LCP-od dB CP
Figure 4: FDP above the nominal level and power under different sample sizes n for Scenario B1. The test sample size fixed m 2000 and the nominal level fixed at a 0.1.
E.2 SYNTHETIC-DATA RESULTS CONDITIONAL LABEL SCREENING
For the conditional label screening problem, we consider nonlinear regression scenario:
Scenario A2 (nonlinear regression): We take constant S 2 and the response Y (Y1, Y2) with Y, = -2X, + +-3 exp((xXe X3) + X 1] and (0, 1). The screening target is Y, A. [a,, +00) where a, the 70% quantile Y, for s = 2, respectively.
Yg = -6X,
Implementation details. We fix the sample sizes n = 500, m = 2000 and vary a (0.05, 0.1, 0.15, 0.2). We apply three different algorithms to train probability prediction func- tion for each component of Y: linear logistic regression (LL), neural network (NN) and random forest (RF).
Benchmarks. We compare our conditional label screening method via the LCP (abbreviated as LCP-Is) summarised in Algorithm 2 with the thresholding procedure without weighting (abbrevi-
19

---

Published as a conference paper at ICLR 2025
1.00
0.75
0.30
0.75
0.50
0.25
0.05
0.3
0.15
0.00
0.15
0.2
0.00
0.35
0.2
Method 1el E.PPod
CP
Figure 5: FDP above the nominal level and power under different nominal levels a for Scenario B The sample sizes are fixed at 2000.
ated as THR). The THR method is performed by only replacing the LCP by the classical unweighted CP.
Results. We calculate the empirical marginal FWER and four kinds of conditional FWER's (condi- tional on X B for four different sets B). The evaluation measures are defined follows:
(Marginal FWER): mF mFWER; Pr (2 1(Y2j $ As, = 13
(Conditional FWER): cFWER, Pr E (Y2j, & Ac, = 1 0| E B.) for
E {1,2, 3, 4} and j E {l...., .m) where Bi = {X E Rdd X1 and X3 < 0),
B2 [X E Rdd X1 0 and X3 0), B3 X E Rd X1 < and X3 0}. and
B4 = [X E Rd X1 > 0 and X3 > 0).
ePWRR abore mmnnd
dPW ER2 blore ZEmUna
CWER abore cominal
ER above semunl
mPW ER above Dumital
0.2
0.1
0.3
0.0
0.2
0.1
0.0
....
0.05
0.1
0.15
0.2
0.06
01
0.15
02
0.05
0.1
0.15
0.2
0.05
0.1
0.15
02
0.05
0.1
0.15
02
Method da LCP-Is da THR
Figure 6: Conditional FWER (cFWER) and marginal FWER (mFWER) above a under different
nominal levels a for Scenario A2. The sample size is fixed at n 500, m 2000.
20

---

Published as a conference paper at ICLR 2025
The results for Scenario A2 are reported in Figure 6. As theoretically ensured, both methods control the marginal FWER at nominal levels. However, the LCP-Is is much more robust against condition- ality and exhibit much smaller conditional FWER inflation than the THR method on subsets B3 and B4. Accordingly, the conditional FWER of the THR method on B2 approaches 0 as the nominal level increases, while the conditional error rate of the LCPPIs remains much closer to the nominal level. These results suggest that the LCP-Is method offers a more balanced screening result.
E.3 SYNTHETIC-DATT RESULTS FOR TWO-SAMPIEE CONDITIONAL DISTRIBUTION TEST
We consider three different scenarios for the problem of two-sample conditional distribution test, which are analogous to those in Hu & Lei (2023). Since the whole covariate vector X needs to be conditioned on in this problem, we have d" d here.
Scenario A3: Let Y1, 01 + x + i = 1,..., and Y2j 02 + +X + = 1....., m,
where X1; id Ia), 22j idd N(p, Ia) with μ (1,1,-1, -1, and Eii E2j idd N() independently. We set 01 = 022 = 0 under the null and (1 = 0,02 = 0.5 under the alternative.
Scenario B3: Let Yú; 01 33 X + + + 1 1 83 + i 1,...,?
and Y23- = 02g + X22,1 + 82 8,X2 + 24x 2,.4 8g + E2.. .....,,,, m, where
iid ~ 0.5. (0, la) + 0.5N (0, 1.51a) and Gi6, B2j ind 1(5) independently. We set ai = 02 under the null and ai = 02 0.8 (1 0.1 Xx (2) under the alternative.
X1 sid ~ 0.5, Ia) + 0,5,V Ia) with
(0.5, 0.5, 0.5, 005, X2x
Scenario C3: Let Y1.- (())) + +Ec, i = I,..., and Y21- = 0 (x²) + E2j.
= j=:.:j=11 L.... m. where ((X)= = BByy| 2) is an additive function of B-splines, and X ~ 0.5.N (0, 41) + 0.55N (M,, Ia) with iid
iid
μ = (0.5, 0.5, 00.5, -0.5, O)T, X 0.5 (0, 4a) +0.55 (0, 1.5Lg) and we set lid NN,0 4/+++ (1)) under the null and 1 NO, 4/(1 (1)). E2I id (0, 1.5/(1 + (1)) under the alternative.
id
Implementation details. Under each scenario, we fix d = and consider different sample sizes m E {200, 400, 600, 800, 1000} with equal splitting. The coefficient 3 in model A and B are taken as B = -1,1, -I, For our localized conformal test (LCT) method, we take the Gaussian kernel function and choose the same bandwidth h (n/2) -1/(d+2), For all three scenarios, we use three different probabilistic classification models to estimate density ratios: linear logistic (LL), random forest (RF) and neural network (NN). The type I error (size) under the null and the power under the alternative are calculated for each scenario across 500 replications with nominal type I error level a = 0.05.
Benchmarks. We compare our localized conformal test (LCT) as summarised in Algorithm 3 with the following two tests:
CT: Hu & Lei (2023)'s conformal test as described in Appendix C.
DCT: Chen & Lei (2024)'s de-biased conformal test, where they formulate the covariate shift problem within the nonparametric framework and utilize the doubly-robust theory to correct the bias of the test statistic.
Results. The results for all three scenarios are summarised in Figure 7. Regarding type I error, the CT method is only valid when the training model is correctly specified and exhibits a severely inflated type I error rate under mis-specified models. The DCT method is more robust than the CT method and controls the type error rate under the nominal level for most scenarios and training models. This corresponds to doubly-robust property, which relaxes the requirement on accuracy of the estimated density ratios. In contrast, the LCT method controls the type I error rate accu- rately around the nominal level for any combination scenario and training model. This shows the robustness and wide applicability of our proposed test statistic.
In terms of power, the CT method is not valid with misspecified training models, so we only discuss the DCT and LCT methods. In Scenarios A3 and B3, the LCT method exhibits significantly higher power than the DCT method especially when the sample size is large. In Scenario C3, the LCT more powerful than the DCT under the NN algorithm and the two methods perform similarly
21

---

Published as a conference paper at ICLR 2025
when using the LL RF algorithm. In general, the power improvement is more significant when the classification algorithm can distinguish the two populations better. Except for the numerical performance, the DCT method involves not only data splitting but also cross-fitting step. The LCT test does not require the latter and is therefore much easier to implement and computationally more efficient.
L1
NN
RF
1.00
0.75
15g00 DETCE
0.50
Sccrrari A3
0.25
0.00
1.00
0.75
0.50
0.25
0.00
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
1
1.00
0.75
Typel
0.50
Scenario 133
0.25
0.00
1.00
0.75
0.50
0.25
0.00
200
400
600
1000
200
400
600
1000
200
400
600
800
LL
NN
HF
1.00
0.75
Typ00 CODE
0.50
Scenario C3
0.25
0.00
1.004
0.75
0.50
0.25
0.00
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
n
Method
LCT
CT
DCT
Figure 7: Results for two-sample conditional distribution test: Type I error (size) under H, and power under HiI across different sample sizes n = m E {200, 400, 600, 800, 1000). The red dashed lines denote the nominal level 0.05. Shading represents emror bars of one standard error above and below.
22

---

Published as a conference paper at ICLR 2025
E.4 CONDITIONAL OUTLIER DETECTION ON SPATIAL DATA
Dataset. We consider using the House Sales in the King County, USA dataset (Kaggle, 2016) to demonstrate the performance of our method on the spatial scenario with label Y, which has = 21613 individuals with 21 attributes including two spatial features: longitude (s1) and latitude (s2). The response Y the price of houses.
Implementation details. We randomly sample three parts of the data from the whole dataset: Ahy 000 training data, Real 2, 000 calibration data and The 3, 000 test data. Since the original dataset contains no conditional outliers, we create synthetic outliers and apply different methods for detection. We randomly sample 10% of the testing data to be outliers. For the outliers, we add random noise e to the original response Y, We consider two kinds of synthetic outliers.
Conditional Maantile-bssed outlier:
Y: = Yi + Ei, E; = 0.75 Quantilee. |22) E,
where Quantile,, 0.9 sY =1) 1/2.
$2) is the 90% conditional quantile of Y given and Pr(f =
Conditional variance- based outlier:
Y, Y; Ex, 6 = 2 v Var(Y; 82) E, where Var(Y $2) is the conditional variance of Y given and Pr(f = 2) = =Pr(f =
-1) 1/2.
All results based on 500 replications.
We compare the LCP-od and the CP methods with the CQR score constructed by two dif- ferent algorithms: quantile random forest (QRF) and quantile neural network (QNN). The empirical FDR and power for conditional outlier detection are reported in Table 2-3. The result is similar to the simulation part, where all methods control the FDR below the nominal. While the advantage is not as significant as before, the LCP-od method still enjoys the highest power in all cases.
Results.
Table 2: Empirical FDR and power for the House Sales dataset with quantile- based outliers. Bold numbers represent the best results. The brackets contain the standard errors.
Method
LCP-od
CP
Score
CQR-QNN
CQR-QRF
CQR-QNN
CQR-QRF
FDR
0.139 (0.087)
0.131 (0.030)
0.128 (0.047)
0.145 (0.036)
0.15
Power
0.657 (0.096)
0.871 (0.048)
0.593 (0.037)
0.858 (0.070)
FDR
0.177 (0.033)
0.175 (0.030)
0.174 (0.049)
0.175 (0.037)
0.2
Power
0.882 (0.131)
0.913 (0.021)
0.824 (0.078)
0.907 (0.022)
Table 3: Empirical FDR and power for the House Sales dataset with variance-based outliers. Bold numbers represent the best results. The brackets contain the standard errors.
Method
LCP-od
CP
αt
Score
CQR-QNN
CQR-QRF
CQR-ONN
CQR-QRF
FDR
0.141 (0.051)
0.127 (0.029)
0.142 (0.043)
0.128 (0.035)
0.15
Power
0.520 (68)
0.628 (0.092)
0.480 (0.085)
0.581 (0.119)
FDR
0.181 (0.036)
0.176 (0.038)
0.196 (0.034)
0.186 (0.030)
0.2
Power
0.630 (0.084)
0.722 (0.056)
0.597 (0.099)
0.692 (0.106)
E.5 TWO-SAMPLE CONDITIONAL DISTRIBUTION TEST ON AIRFOIL DATA
Dataset. Refer to previous related works (Hu & Lei, 2023; Chen & Lei, 2024), consider using the airfoil dataset (Brooks et al., 2014) from the UCI Machine Learning Repository to demonstrate the effectiveness of our proposed LCT on real data. This dataset investigates the sound pressure
23

---

Published as a conference paper at ICLR 2025
of various airfoils with 1503 observations of the response Y: the scaled sound pressure and the covariates with = 5 dimensions: log frequency, angle of attack, chord length, free-stream velocity, and suction side log displacement thickness.
Implementation details. We use part of the rules in Hu & Lei (2023) to split the observations into two samples follows:
ii) Random partition. Randomly partition the dataset into two groups with sizes [Pil 751 and [Da] 752.
(ii) Exponential tilting. First randomly partition the data into D1, D2. Then construct D2 by sampling 25% of the points from D2 with replacement, with probabilities proportional to (xx) exp(x* a),, where 0,,,0,
(iii) Partition along the response. Split the data according to the value of response, where the first group contains the samples with smaller response values.
In the first two partitions, the two samples satisfy the null hypothesis. The first partition has samples from the same distribution, while the second shows non-trivial covariate shift. In the last partition, the covariate shift assumption is not satisfied and the alternative hypothesis holds. For all three cases, we use the linear logistic (LL) and neural network (NN) to estimate the density ratios.
Results. Similar to Hu & Lei (2023), we use -ut-oe-sample marginal classification error as a proxy of the accuracy of marginal density ratio estimation for all cases (i-iii). The percentage of rejections for cases (i-ii) across 500 repetitions are summarised in Table 4. We can see the rejection proportions of both LCT and DCT are close the desired level when using LL or NN algorithms, while the CT test fails to control type error when using NN algorithm due to its low accuracy. For case (iii), we only have a single deterministic generation of the training and testing data. The median p-values across 500 random splits for case (iii) are reported in Table While all three tests correctly reject the null in this case, the LCT test gives the smallest p-value and is therefore most powerful.
Table 4: Percentage of rejections (PR) and average error of estimating (Err) in Airfoil dataset for cases (i-ii) using algorithms I.I. and NN over 500 repetitions, with nominal level a 0.05 for L.T,, CT and DCT. The bracket contains the standard error.
Cases
Method
PRL
PRAN
EmTu
ErTsI
LCT
0.054 (0.0325)
0.053 (0.0374)
Case (i)
CT
0.058 (0.0289)
0.441 (0.0610)
0.429
0.608
DCT
0.053 (0.0299)
0.066 (0.0268)
LCT
0.067 (0.0323)
0.066 (0.0390)
Case (ii)
CT
0.056 (0.0320)
0.135 (0.0405)
0.213
0.367
DCT
0.056 (0.0327)
0.035 (0.0250)
Table 5: Median p-values (Pval) and average enrrr of estimating y (Err) in Airfoil dataset for (iii) using algorithms LL and NN over 500 random splits, with nominal level a 0.05 for LCT, CT and DCT.
Cases
Method
Pvahr.
Pvalsx
Ema.
ErroN
LCT
0.000
0.000
Case (iii)
CT
0.000
0.022
0.279
0.429
DCT
0.000
0.018
F TECHNICAL DETAILS
F.1 PROOF OF THE DENSITY RATIO EQUATION (2.3)
Proof. We calculate the density ratio of X2, and X₁ conditional on X2 here for the sake of complement. This proof refers to the proof of Proposition in Hore & Barber (2024).
24

---

Published as a conference paper at ICLR 2025
Note that by the sampling step, have X2y I (Xzj, Y2s) (X2)..)). :). The joint distribution of (X2j. Y25, X23) is as follows:
X2j P2,x,
Y2j X2j Py|x( X2j),
X2j (X2j; Y22) H )X
direct calculation then yields (X2j: Y22) X2j (P2.x o Hi: X2y)) x Pa by Bayesian's theorem, where P2.x o H( X22) = P2.x which is the distribution P2, reweighted by (-,X2,) X2j).
La ((,))g)) x for all A X.
That is, (A) (P2,x H x2))) Jx H H(x),))) (e)
This is an instance of a covariate shift problem: the distribution of X2j و2ح + P2.x. but the conditional distribution distribution of Y2, remains the same as for the training data. i.c., P
Hence, by the definition of covariate shift, the density ratio X2j and X1: conditional on X2j is
dP, 2 Raj (x) Sx H., X2j) d2) Jx H X2j) dp dP2
Hx, Xzy)
H(x, X2))
Hx, X25),
dPix
where c c= 1/ 1/
Sx H., X21) d22.x (22))(() is constant, which completes the proof.
F.2 PROOF OF LEMMA 2
Proof. Denote Pixx: P2.x as the marginal distribution of X if we fist sample X from Pixx or P2. and then sample from -X..). The distributions of the calibration and test points are PI,XXY PIx and X2 xxxxxxax X22x where ) and Pa )) denote the correspond- ing conditional distributions of calibration and test data points.
Notice that the super-uniform property still holds if the conditional distribution (Ixx =
PR,CKYN)I Therefore, the super-uniform bound holds for an independent tuple (X", v,,"*) X*) PLYXX, x P2.x That
pi Eisc H(1,, X.)(( Via+ & ((X,X')
Eiec H(1,; x H(X* X*)
Pr(pi. < a) S a.
The probability for (Xan+j, Yentj, Xzn+s) is then upper bounded by
Pr(pt.j S aa) Sa X 2 ddrv P + E Exxx, {drv (PIxII:PPa) Pax1x)}
For fixed X we have
dP 2,x (X)
(X)
drv (Pixxx:
Ex~P,
dP dPi,x x (X)
9(X)
Ex=P
Ex Ex/mPixxixx0 (X)}
E Ex Ex.x'~P, x (9(( g(X'))>
2E p xix (g(
Ex Pi PIx (1 X)}
-Ex,x~~, x1R (la(X) - -(X')l3
21Ex- Pr.x (l x))
25

---

Published as a conference paper at ICLR 2025
Taking expectation with respect to X we have
Ex ddTV (2xxxxxP2xxxx)
Ex- Pi (X, x)} dx
< E x) BXX (g((X) g(X'Il)
2Ex~P. xX
/ EXXXIPP,XX [l((X) xX
/
Ex.x (lx() -1(X'))XXXXXX- dP
where Pи.x has a density function [x) = Ex (H(X,x)) x)).
By integration
Ex.x' P.x x-x (19(X) - 9(X'I}
/
x' xx=x) K
x" (x xx) fi (xx(xx)/xxxxx")))l)xx g(x")(xx)))))))))))) (dx"))xxxxx"
K
12dd
h
= f K (u) K ((v))l(x + hu) g(x + hv) [ixx(xxxxxxxxxxxxx hu) hho)duude
ho)ddudo
/ Bu (la(x + +U) g(x)}}-
Combining the results above we conclude
Pr(pr.j aa Sax + Ex-r, {dTV (PixIXx P2x1x)
Sa |I 11xp///ll//Xxx-xxxx.... A19(X - g(x)]}.
F.3 DISCUSSION ON LEMMA
The deviation bound in Lemma 3 has two terms corresponding to the bias and variance of p-value functions. This different from Lemma in which we only need h -> to eliminate bias and ensure validity. Fixing the sample size and taking h - 0, the LCP will degenerate the independent uniform random variable 5, which remains valid but does not contain any information of the data. Otherwise if we take h - co it will degenerate the unweighted CP. Therefore, the bandwidth h controls the trade-off between the conditional validity of the LCP and the effective sample size.
F.4 PROOF OF LEMMA 3
Proof. Remember the definition of the RBF and box kernel
x
x11
1
HRBF (x, x') Va exp
Hbox (x, x')
Va [||x x'112 S V2h)
2h2
where VARBE, Va,box are normalizing constants.
The LCP function is
Eiec H(X X)T(U S Vii) + X)
PL v)
(F.1)
Eiee H(X X) H(x,X) X)
For the difference we have
p y) (1 fv)x=x(v))1
Eiec H H(X1,, X.([FU S Vn.} {1 - Fvix (v))
- (1))) H x)
Eiec H (X x) + H(, X)
Eiec HX1,, X)L & Vid - {1 F Fv]x-x(1)H]
Hl (x, x') Eiec H HXi,,
Eiec H (X11),)
26

---

Published as a conference paper at ICLR 2025
For the denominator 7n = Eiec H HX1 X), first notice for the RBF kernel
1
X11
X
TRBF
Va.RBE exp
2h/2
Va, box
2 exp(-
(-1}
X S v2h} X1
VI.RBF Va ha
Va, box
exp( (11
Va.BBF
Tbbox
Define Thox Eiec X11 - X112 S V2hJ. for constants 0 < A,E < we have
Pr
(((nhox) Rbox (n( X
Pr
(xp (B(nbox) mbox 2 exp (AE((bbox)) x)
<B
exp (E(box) bbox |X exp (XXB(nnox))
=E exp T(X's B (X)) Pr(X's E B BE (x)) x |C1
(nXT(n
exp
[Pi(X /2z2 (X)) exp (rr(X & B B22h (x)) Pr(X & B B2n (X)) exp { -Pr(X B2,x (x))(
eepp (-xL((box))
(22a (c)
<
Pr(x B B22
+
exp (xlE--EE(niso)
(x))]]1 101
exp (-ICIPPP(X' e BBa (xa
(e ( (X)) +
<exp { { [C er E (X)) +e| (22)
and
Cah
2Cah 2Cxl 2 Call o(1) Pr(X (x) 2 Cah o(1)
2
for sufficiently large n, where C. only depends on d. Taking X = 0.9 we have
- - nh Caf(() + 3.6ml ca +1
Pr Thhox
21 exp
20V
Therefore with probability tending to 1 we have nn 2 C. (c]((x) for some fixed constant C' > 0. Then
(pr.(x, 3) - (1 -V-xxxxx(V
1 C]())
Eiec H(1,, Xi[lfu < Vnd - (1- Fv (x)]]
1
(F2)
+
[C]
nhdaVa..
For the first term
EEEC XXX & Vid} (1- (v)))=
E
|c12
[((ii x) & Vu+) - 11- -
C
C
- 1
+
E
( X)H x| Via) - (1- (ixx=(1)]]-- < Viz} - (1 - Fv x= x=x(x))] (v))i)
C
1
E
+ (,(,,(X,,,X)/11XXX xXI PrIx=xllax|IFIIxxx,, Fvix=xllx)
C
27

---

Published as a conference paper at ICLR 2025
By integration
H( 1i, x)²}
E
x [x=x]
=
heak2 K2
fxx((x)ddxe
h2d
h
E { { (w) f1,((X ( haaddu
hd
1i hd
K2
E
H( (X))
()) FvIl - F - Fvix=xllo0
x
x" -x
=B
K
| - F Fv Fi - f (i fixx (x" d dxe"
112d
h
h
$11f1xx
E
K (2) K (w') ||E Fvix=x F X+hw FVIx=xlloa
x11,11133 E
K ((u) K (u') | x hul + hu'll
1 1
=11f1x
K(u) - hull hu'
dudu'dx dx
hd
h
=h22 11x / (K(w)((()) (u') K (w") llee +
aon+""lll1 u'l duddu du"
+ $ ((u))llul dm
<31 13/.91151.x13 Iso
As all integrations in the final expressions exist, we have
Eiec HX1,, X)[T(U s Vn.) {1 (x))))
E
|CP2
Ckallfixxlle
+ Cx.ellfi.x11
nhe
Together with inequality (F2) we complete the proof.
F.5 PROOF OF THEOREM
Proof. By the construction of the refined LCP's, they are valid conditional on the event Y2, & A in finite sample
Pr(prt.) S a)) S a,
and marginal PSER control result is proved. For the conditional PSER inflation bound, the proof exactly the same Theorem 4 with the only difference that the distribution of X is replaced by the conditional distribution of X given the event Y & A. We therefore delay the computation of the deviation term to the proof of Theorem
F.6 PROOF OF THEOREM
Proof. Denote the index set of inliers in D2 as Hj, We firstly need a lemma commonly used in literature relating the conditional calibration technique.
Lemma 4. The pruned rejection set R satisfies
E Hij E R.j H3}
[P < a|R m
.j E HGG)
FDR = E
E
R V
28

---

Published as a conference paper at ICLR 2025
The proof of Lemma 4 can be found in Jin & Candès (2023a) and is therefore omitted.
Denote Zy and Zez as tuples Zi = (X1s: Y1e), Zzj = =X2,, Y25) and the index set of calibra- tion data C = {å1. 12, .......,,... .nn) where [C] Ne. Let 2 -Zii...... Zisn. Z25l be the unordered value set of Ziey...., Ziine: Zzj. Let X = (X21,.... ....., be the set of all sampled covariates. Note that the unordered value set ( are fully determined by X, Z and (Zahtt More- over, the p-value PL. is fully determined by X2j. 2 and Zzj. As (Z2re is independent of {Zis...... Zisoo [nj} conditional on X, we know Pt.j' is independent (PL, conditional on X and 2. By the fact that |Ry-ol is fully determined by 1p,,1l435 we further have
PL. 11 | R2
By the independence we have
HPL. S alR
alR2
j E Ho.
[PPu
E
m
2 z
I E
m
X2 2
Ro
|R
Pr pT. < a|R
Ri-ool
Pr Pins S
a R
2,
Ri
where
Eiec HX (11V/ < Vii + 11121 V1.t) (X2),
Pi.j
Eiec 11 (X ;, X2j) 11 (X2j X2,)
By the weighted exchangeability, Pi.j X ~ U 1]. Taking the expectation we have
m
IPL. S a|R
m
Pr
PL.
a R2
vo
X, Z
FDR S
E
E
X
5
E
m
Ri-ool
R
Ri
j=1
a
m78
which completes the proof.
F.7 PROOF OF THEOREM
Define the uncomputable oracle value as
Pï. = Eiec H(XI,, (2) S Vis) + E, H(X2)) X21) Eiec H(X,,, X2j) + H (X2j,, X2))
By Lemma ((X YI,, St))lec ((X2j, Y2j: S2n)) are exchangeable, this oracle p- value will be super-uniform
Pr(pi.j S a) a
Note that Vi2- = max V2s,a Y25. & A.). This indicates
PL. min Y2j. As).
29

---

Published as a conference paper at ICLR 2025
By direct tranformation have
S25
FWER = Pr
Σ
HY2j $ Ap:PL saf20)
Pr
U
(prnii Sa)
Pr(pi.;
aa) S a,
which completes the proof.
F.8
PROOF OF THEOREM
Proof. By our screening procedure, the conditional FWER can be transformed
Sas
Pr
1(22i,s & As, = 1} > 0 X2j E B = Pr
U
{PL.S Sa} X2, e B
12s
Pr(pī., Sa X2j E B).
Since we assume exchangeability, this equivalent the covariate shift setting with g(x) 1fx // Pr(X2j E B). Therefore, we only need to compute the excess term in Lemma 2 with this specific g. By transformation
EU~K() fla(x + hU) 9(x]}}
/ k(w) hu B, B) + M2 + ha $ B,x E B)
Pr(X2j E B)
/ K(u)I([|ull d(x, 0B) ddu
Pr(X2) 2j E B)
-2 ([o])(((((UU]]) 2 2 0B)) Pr(X2) B)
Following the same proof Lemma 2 we conclude
Sas
sa+2l15ixxllo PIX-PAx.U~KK((UU 2 d(X, AB)) Pr(X2; E B
Pr
HY23, A و3,0 1} > 0 X2j EB
F.9 PROOF OF THEOREM
Proof. Remember that assume n1 n2 in the main text. We first introduce lemma about defined U-statistic.
Lemma 5. For some kernel function k(z, z') depending on ni, define the two-sample U-statistic and its projection as
1
n n1
Unj
k kZ Zzj)
n
and
Ün,
1
EfK(Zi. Z21) Zid +
1
Z2, E(k(Zii Zei) E(K(Z Z21)
12
71
j=1
Then if (k 2) - o(11), Un, and On, satisfy
(ni(Ung Ūn,))= =
30

---

Denote the unnormalized version of 7, and its projection as T. and To.p our defined statistic Published as a conference paper at ICLR 2025
This lemma is extension of Lemma 3.1 in Powell et al. (1989) which constructs similar result for one sample U-statistics of degree 2. The proof follows exactly the same and therefore omitted.
k(Z 22j) H(1,, X2,) D.y) We first check the condition in Lemma By integration and Dig <
[ik(Z Z2j)²} (X X2;)
[ 12dd
1 (x' (x=xx) fxx((x) (x') ddxdx (xx'xxxxx
1
/ (xxx + hu)dxdu
hd
hd /
1 k(u)ddu.
Together with the assumption h + oc we have B[k(Z1i. Z2j)²} = o(n1). Define the centralized projection statisties vi (Zn)) Efk:ZZii Ze1) 12.1 (22))
E(k(Z11, Z2z)
Z2j } and
11
Blh(Zic, Z21) Zn. [[ Zzj)].
1
m2
2
Efk(Z,1 Z2j)
22j} E(k(Zzi Z2j)).
11
The projection can be decomposed as
(Zz,)
=EfRi Zii
Z21)
Zni}
1 ha
K
X1i
[1
I[V, < v(x,
- / [[[ V fax((x))2(1 x)dxdy
2
K(u)
[
T[Vi; < V(Xi; + ha, hu, TTV VVVi; + hes, tew,
fe.x (X1:
14
+ f2 X1: + hu)dudy
/ k(u)
1
1[V1 V (X w)}
1 2
LVV ('(x..))) fex(Xx)) Xx)dudy
+ 11,m (Z1i).
= 1 (Z))) + +1i,,... (Zn)).
By Assumption and + O we know 11, (Zu,) = (p(). Similar results also hold for 4/2,m (Z2j) and 22 (Z2z)-
Note that under the null hypothesis v(x, y) = 1. So we have
E[H((I,, E [(III X2s) (5-1)
If Assumption holds then
(T (x. ( (x) -Di)) = op()
If PIx = Pax then under the null hypothesis we have Pi = Pz and therefore B(H((1;;X2,)) by symmetric.
31

---

Published as a conference paper at ICLR 2025
Now the statistic can be decomposed as
1
1
m1
(T
w (Zie) - (2()((11)))) +
1/2 (Z2)) - (B(22((22)) (22))
10.0)
n1
71
1=1
1
11
1
M1.. (Z1i)=
BB (Zn:)]+
122 (Z2z)
B 2 m. (Z2))H
11
18
E
E (11 18 X2,) Diy}
bb (Zn:) -
Σ
(2(
j=1
As 1/ (Z1.) and (2 are independent random variables not depending on Rj, we have
T
(0.1).
(F3)
om
where
Var (Zi:) +
Var(t's(Za)))
711
11
Hereafter we only need to prove the consistency of the variance estimator. Denote o Var(T the non-asymptotic variance can be decomposed as
[X X 2
2D
n1
E
{ (X
Xoj) (x) Diy Des)
mg
1
Ti
11
E 1:: X2) (X X²x Dis
2n1 -11 E {
H X D.s
)Ds)) }
nf
By similar integration computation, the cross-term expectation converges to Var [b> [Z,)} and Var (Z2))} in probability. Removing the vanishing term we have o o op 0p 11(1]) and
T /on1 3 1.
The variance estimator a takes the form
2
R2
n1 X
H(X
Di
(X
Des)
2
T11
j=1
j=1
2
11
(X 18 X2y) D
2
1
H
X 15 X2y)
n1
ni
1=1 j=l
i=1
j=l
1
H( 15 Xoj
1
H(X X2,) X2)) X28 ) Doe
n1
1=1 o
+ 1 n1
2 Th1
D1
H(I,,, 1
Dus
H Des
It is easy to see that 02, consists of term-wise approximations of the expectations in ox2 By further computing the second moments of the estimators and Markov's inequality we have 0% (1/1)) Together with (F.3) we conclude
-0-2 +2
0
4 N 1).
32

---

Published as a conference paper at ICLR 2025
F.10 PROOF OF THEOREM 6
Proof. By the same proof with Theorem 5 we know
4, N0, (2; Dzy) To (X
(F:4)
By Assumption 2 we have ( (Di) - Dij)) - 0p(1/ /mi. Combining with the con-
sistency of te we know vn1ō ww - and
T IX X2, D Z BIH(X Dis) op)))(1))++ +0)
Tw
Tw/ hn/
Dig }. By integration The rest is to compute the bias term [X; (2,)
B(H(X
X2;)Das)
K
X1
X2
[1
< V (x2. (1xx(x))faxxxx))55( x1)52(122 xx)dxx dy2
hd
h
/ K (a)
TIV* (x1, <V hu, (1 h
hw) xx) x1 hu)
2
dx, dxddddydy
=
x1)dx(dyıdys ) (1 + op(1))
IIV < (x1, fxx(x))a2xxx)) /x))
(1) |x
2
(
1 I[V (x1,01) < V* 82)1 ("tss,)))iixx)))/x2xx))) |x) xyddx ) (1 op(1))
=Ex~Prx f1x(X) [ ("t. <xx),,))) } (1 op(1))
And by transformation
By x (X, (V(X Y2))
P (V < (X, K2) + 2 2 By V' (( vX(X)))
By,
2
1 Yo (IV (X, -" (x (*X
2
1
1
2
The bias term is then simplified as
[[H(X,, X2) = "xtp://////.a//////////////////"
Combining all the results above we conclude
E(H(X Dig) {1 + op(1)) + +1p(1=
VTide
Tu.
(1 (1)) + Z + (1)).
Tw
Ow/ /ni
40w
33


Reviews:
[{"review_id": "d0R0h8Rb8n", "reviewer": "Reviewer_24ms", "summary": "The paper introduces localized conformal $p$-values and applies them to several conditional testing problems, including conditional outlier detection, conditional label screening, and two-sample conditional distribution testing.", "questions": "1. Could the authors provide a rigorous proof of the density ratio of $X_{2j}$ and $X_{1i}$ conditional on $\\tilde{X}_{2j}$.\n\n2. Where is $d$ defined? \n\n3. In different applications of conditional testing, what are the score functions $V$?\n\n4. Please clarify the statement \"This indicates that using the LCP for data selection can lead to a more balanced selection result since the PSER inflation for different sub-groups is bounded\", and the statement related to fairness.\n\n5. Avoid using abbreviations before introducing them, such as PRDS.\n\n6. In conditional outlier detection, what are the theoretical and empirical performances if $P_{1,X} \\neq P_{2,X}$?\n\n7. From my understanding, the theoretical results do not account for the estimation errors. Moreover, kernel functions can lead to the curse of dimensionality, which hinders application in high-dimensional settings. \n\n8. How do the tuning parameters in kernel functions affect the performance, and how should these parameters be chosen in practice?\n\n9. Why does the method in two-sample conditional distribution testing not require randomization? Does the proposed method in this context require the marginal density ratio? Moreover, why does the type I error increase with sample size in Figure 7?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents the concept of localized conformal $p$-values and demonstrates their application to some interesting testing problems. The authors conduct appropriate experiments to illustrate the performance of their method.", "weaknesses": "Please refer to the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dsHYhPSaS6", "reviewer": "Reviewer_sXfi", "summary": "Authors introduce a novel (conformal) p-value by converting an existing conformal prediction method that is locally valid, asymptotically conditionally valid, and computationally advantageous. They further discuss the applicability of this approach in various multiple conditional testing problems. From an application perspective, their contributions are four folds: balanced data selection, conditional outlier detection, conditional label screening, and a two-sample conditional distribution test. Each application is rigorously justified through theoretical analysis, and the utility of the proposed methods is empirically demonstrated via extensive numerical experiments.", "questions": "1. In the application of conditional outlier detection, the goal was to control the FDR using the BH procedure, whereas in the conditional label screening problem, LCP guarantees FWER control. Is there any reason behind controlling different quantities in each case? What motivated the decision to adhere to FWER control instead of FDR control in conditional label screening despite this conservativeness?\n  \n  2. In Section 3.1, within the balanced data selection problem, it is assumed that the response variable is not observed. In this case, how is $V_{2j}$ specifically obtained? My understanding is that both $X_{2j}$ and $Y_{2j}$ are required for this, but $Y_{2j}$  is not observed. In this case, is an arbitrary value of $Y_{2j}$ (i.e., any y which is not in $\\mathcal{A}$) satistfing $\\mathbb{H}_{0j}$ used?\n  \n  3. Could you provide more detailed explanation or intuition for how the specific choice of g(x) makes the covariate shift problem equivalent to conditional validity?\n\n  4. In Section 3.3, the context of large language models’ factuality is introduced to frame the conditional label screening problem. However, the numerical study presents results only on more structured synthetic and real datasets. I was wondering if any numerical study reflecting the natural language processing context described in the paper has been attempted. If so, what were the outcomes of those experiments? If not, what additional research would be necessary to conduct such studies?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The most notable strength of this work lies in the variety of applications presented. The proposed applications of LCP are all promising, largely due to the LCP’s ability to effectively address conditional testing. In particular, the authors contextualize each application; for instance, in the case of the conditional label screening problem, they suggest that this application could be used to ensure the factuality of large language models, thereby justifying its utility.\n\nFurthermore, the authors thoroughly discuss the theoretical justification of the proposed applications. They rigorously prove how each problem controls specific quantities (e.g., FSER, FDR, and FWER) by synthesizing a considerable amount of literature to support their claims. This strength, the ability to creatively adapt existing methodologies and introduce multiple new applications, demonstrating an original contribution to the field.​", "weaknesses": "While they have successfully introduced locally conformal p-values and thoroughly demonstrated their various applications and theoretical properties, their work seems limited in terms of novelty. Specifically, the locally conformal p-values appear to be derived from directly converting the prediction intervals proposed by Hore & Barber (2023), and their application seems to involve integrating these p-values into several already established methods.\n\nMeanwhile, one of the main contributions of their work lies in achieving the asymptotically conditional super-uniform property of the new conformal p-values. Specifically, Theorem 4 and Theorem 7 provide theoretical justification for this, though the authors merely mention that a specific choice of $g(x)$ renders the covariate shift problem equivalent to conditional validity, which is not clear. While this approach poses no issue in terms of soundness, as previously discussed by Hore & Barber (2023), it may lack readability in this regard.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "01dVqiwjT0", "reviewer": "Reviewer_mWDU", "summary": "This paper introduces localized conformal p-values for conditional testing problems.\nSpecifically, the authors inverting the prediction intervals which are constructed by\n existing works to adapt to the conditional distribution of the response. They establish theoretical results on the fundamental properties of their defined p-values, as well as applications of these p-values to conditional testing. The empirical results demonstrate the effectiveness of their proposed methods.", "questions": "1.What circumstances the \"average FDP above nominal level\" shown in Figure 1 would become negative?\n\n2.Based on the experimental results in the paper, it is observed that in many cases, LCP-od exhibits greater fluctuations compared to CP. How can this phenomenon be explained?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a clear problem statement, solid theoretical results, and the experimental results effectively reflect the efficiency of their proposed method.", "weaknesses": "The structure of paper could be better. The authors could give concise and appropriate names to the important theorems to summarize their meanings, and mark the others as lemmas. There are many theorems in the main text of the paper, so presenting the theorems this way would make them more intuitive and help readers better understand the meaning of the theorems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZbnWb2Pgi0", "reviewer": "Reviewer_hT7S", "summary": "This work investigates the possible application of localized conformal p-values, which can be naturally constructed from recent localized conformal prediction methods.", "questions": "1. Could the authors provide clarifications on the goal of Theorem 3? It feels somewhat distracting from the main focus of this work, as it essentially presents an 'asymptotic property under additional assumptions,' while the main point of the conformal prediction framework is the finite-sample theory without distributional assumptions.\n\n2. Regarding Section 3.2, 'Conditional Outlier Detection,' I'm not sure if the hypothesis (3.1) is what is actually being tested. Specifically, isn't the p-value super-uniform under a weaker condition than $H_{0,j}$ in (3.2), since it is marginally valid? \n\nMy understanding is that the localized conformal prediction method from Hore and Barber basically achieves the marginal coverage guarantee, while the conditional miscoverage rates are \"likely to be small\" if we frame everything in a distribution-free sense.\n\nThus, I feel that the outlier detection counterpart of the localized conformal prediction framework controls the FDR for testing marginal outliers, but conditional outliers are more likely to be detected—rather than directly detecting conditional outliers with a distribution-free guarantee. Am I missing something?\n\n3. (continued from point 2) In Theorem 5, is the goal of the condition $P_{1,X}=P_{2,X}$ to claim that any detected distribution shift is a label shift rather than a covariate shift, i.e., that any outlier is a conditional outlier? My confusion is that under the condition $P_{1,X}=P_{2,X}$, the null hypothesis in (3.1) is essentially equivalent to stating that the marginal distribution $P_{(X,Y)}$ is the same for the test point and the calibration set, since it is the combination of \"$P_X$ is the same\" and \"$P_{Y|X}$ is the same\". \n\nSo, to me, Theorem 5 sounds like \"Algorithm 1 controls FDR for testing marginal outliers. But if we additionally assume that $P_X$ is the same, then an outlier means $P_{Y|X}$ is different, implying that it is a conditional outlier\", which is not very interesting compared to directly detecting the conditional outliers.\n\nTherefore, as mentioned in point 2, for a more useful result, shouldn't the statement be something like: \"The procedure detects marginal outliers, but conditional outliers are likely to be better detected if we use the localized conformal p-values\"?\n\n4. Could the authors provide explanations for the terms in Theorem 4 and 7, since it's difficult to see how small they are?\n\n5. minor comment: \"independency\" -> \"independence\" in page 28", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work extends the localized conformal prediction methods to a broader range of inferential targets.", "weaknesses": "I believe the writing and presentation should be improved, and more effort needs to be made to demonstrate the usefulness of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)