Title: Conformal Inference under High-Dimensional Covariate Shifts via Likelihood-Ratio Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: conformal prediction

covariate shift

uncertainty quantification

distribution shift

distribution-free prediction

prediction sets

high-dimensional data
Abstract: We consider the problem of conformal prediction under covariate shift. Given labeled data from a source domain and unlabeled data from a covariate shifted target domain, we seek to construct prediction sets with valid marginal coverage in the target domain. Most existing methods require estimating the unknown likelihood ratio function, which can be prohibitive for high-dimensional data such as images. To address this challenge, we introduce the likelihood ratio regularized quantile regression (LR-QR) algorithm, which combines the pinball loss with a novel choice of regularization in order to construct a threshold function without directly estimating the unknown likelihood ratio. We show that the LR-QR method has coverage at the desired level in the target domain, up to a small error term that we can control. Our proofs draw on a novel analysis of coverage via stability bounds from learning theory. Our experiments demonstrate that the LR-QR algorithm outperforms existing methods on high-dimensional prediction tasks, including a regression task for the Communities and Crime dataset, an image classification task from the WILDS repository, and an LLM question-answering task on the MMLU benchmark.

Conformal Inference under High-Dimensional Covariate Shifts via Likelihood-Ratio Regularization

Abstract

We consider the problem of conformal prediction under covariate shift. Given labeled data from a source domain and unlabeled data from a covariate shifted target domain, we seek to construct prediction sets with valid marginal coverage in the target domain. Most existing methods require estimating the unknown likelihood ratio function, which can be prohibitive for high-dimensional data such as images. To address this challenge, we introduce the likelihood ratio regularized quantile regression (LR-QR) algorithm, which combines the pinball loss with a novel choice of regularization in order to construct a threshold function without directly estimating the unknown likelihood ratio. We show that the LR-QR method has coverage at the desired level in the target domain, up to a small error term that we can control. Our proofs draw on a novel analysis of coverage via stability bounds from learning theory. Our experiments demonstrate that the LR-QR algorithm outperforms existing methods on high-dimensional prediction tasks, including a regression task for the Communities and Crime dataset, an image classification task from the WILDS repository, and an LLM question-answering task on the MMLU benchmark.

1 Introduction

Conformal prediction is a framework to construct distribution-free prediction sets for black-box predictive models [e.g., 45, 60, 61, etc]. Given a pretrained prediction model f : X → Y mapping features x ∈ X to labels y ∈ Y , and n 1 calibration datapoints ( X i , Y i ) : i ∈ [ n 1 ] sampled i.i.d. from a calibration distribution P 1 , we seek to construct a prediction set C ( X test ) ⊆ Y for test features X test sampled from a marginal test distribution P 2 ,X . We aim to cover the true label Y test with probability at least 1 -α for some α ∈ (0 , 1) : that is, P ( Y test ∈ C ( X test )) ⩾ 1 -α . The left-hand side of this inequality is the marginal coverage of the prediction set C , averaged over the randomness of both the calibration datapoints and the test datapoint ( X test , Y test ) ∼ P 2 . In the case that the calibration and test distributions coincide ( P 1 = P 2 ), there are numerous conformal prediction algorithms that

construct distribution-free prediction sets with valid marginal coverage; e.g., split and full conformal prediction [e.g., 29, 36].

However, in practice, it is often the case that test data is sampled from a different distribution than calibration data. This general phenomenon is known as distribution shift [e.g., 41, 52]. One particularly common type of distribution shift is covariate shift [50], where the conditional distribution of Y | X stays fixed, but the marginal distribution of features changes from calibration to test time. For instance, in the setting of image classification for autonomous vehicles, the calibration and test data might have been collected under different weather conditions [25, 65]. Under covariate shift, ordinary conformal prediction algorithms may lose coverage.

Recently, a number of methods have been proposed to adapt conformal prediction to covariate shift, e.g., in [15, 19, 37, 38, 40, 55, 64]. Most existing approaches attempt to estimate the likelihood ratio function r : X → R , defined as r ( x ) = ( d P 2 ,X /d P 1 ,X )( x ) for all x ∈ X , where P i,X denotes the marginal distribution of P i over the features. One can construct an estimate ˆ r of the likelihood ratio if one has access to additional unlabeled datapoints sampled i.i.d from the test distribution P 2 . Methods for likelihood ratio estimation include using Bayes' rule to express it as a ratio of classifiers [12, 40] and domain adaptation [13, 37]. However, such estimates may be inaccurate for high-dimensional data. This error propagates to the coverage of the resulting conformal predictor, and the prediction sets may no longer attain the nominal coverage level. Thus, it is natural to ask the following question: Can one design a conformal prediction algorithm that attains valid coverage in the target domain, without estimating the entire function r ?

In this paper, we present a method that answers this question in the affirmative. We construct our prediction sets by introducing and solving a regularized quantile regression problem, which combines the pinball loss with a novel data-dependent regularization term that can be computed from onedimensional projections of the likelihood ratio r . Crucially, the objective function can be estimated at the parametric rate, with only a mild dependence on the dimension of the feature space. This regularization is specifically chosen to ensure that the first order conditions of the pinball loss lead to coverage at test-time. Geometrically, the regularization aligns the selected threshold function with the true likelihood ratio r . The resulting method, which we call likelihood ratio regularized quantile regression (LR-QR), outperforms existing methods on high-dimensional datasets with covariate shift.

Our contributions include the following:

We propose the LR-QR algorithm, which constructs a conformal predictor that adapts to covariate shift without directly estimating the likelihood ratio.

We show that the minimizers of the population LR-QR objective have coverage in the test distribution. We also show that the minimizers of the empirical LR-QR objective lead to coverage up to a small error term that we can control. Our theoretical results draw on a novel analysis of coverage via stability bounds from learning theory.

We demonstrate the effectiveness of the LR-QR algorithm on high-dimensional datasets under covariate shift, including the Communities and Crime dataset, the RxRx1 dataset from the WILDS repository, and the MMLU benchmark. Here, we crucially leverage our theory by choosing the regularization parameter proportional to the theoretically optimal value.

1.1 Related work

Here we only list prior work most closely related to our method; we provide more references in Appendix C. The early ideas of conformal prediction were developed in Saunders et al. [45], Vovk et al. [61]. With the rise of machine learning, conformal prediction has emerged as a widely used framework for constructing prediction sets [e.g., 36, 58, 59]. Classical conformal prediction guarantees validity when the calibration and test data are drawn from the same distribution. In contrast, when there is distribution shift between the calibration and test data [e.g., 5, 41, 50, 52, 54], coverage may not hold. Covariate shift is a type of dataset shift that arises in many settings, e.g., when predicting disease risk for individuals whose features may evolve over time, while the outcome distribution conditioned on the features remains stable [41].

Numerous works have addressed conformal prediction under distribution shift [37, 38, 40, 51, 55]. For example, Tibshirani et al. [55] investigated conformal prediction under covariate shift, assuming the likelihood ratio between source and target covariates is known. Lei and Candès [32] allowed

the likelihood ratio to be estimated. Park et al. [37] developed prediction sets with a calibration-set conditional (PAC) property under covariate shift. Qiu et al. [40], Yang et al. [64] developed prediction sets with asymptotic coverage that are doubly robust in the sense that their coverage error is bounded by the product of the estimation errors of the quantile function of the score and the likelihood ratio. Cauchois et al. [7] construct prediction sets based on distributionally robust optimization. In contrast, our method entirely avoids estimating the likelihood ratio function.

To achieve coverage under a predefined set of covariate shifts, Gibbs et al. [15] develop an approach based on minimizing the quantile loss over a linear hypothesis class. We build on their quantile regression framework, but develop a novel regularization scheme that allows us to effectively optimize over a data-driven class, adaptive to the unknown shift r . A similar regularization is used in [66], which performs supervised learning under covariate shift by minimizing an upper bound of the test risk. However, when one sets the loss function to equal the pinball loss, minimizing the objective in [66] is not guaranteed to provide coverage at test-time, whereas our construction has asymptotically valid coverage.

2 Problem Formulation

Preliminaries and notations. For α ∈ (0 , 1) , the quantile loss ℓ α is defined for all c, s ∈ R as

Let the source or calibration distribution be denoted P 1 = P 1 ,X × P Y | X , where P 1 ,X is the marginal distribution of P 1 over features. Let the target or test distribution be denoted P 2 = P 2 ,X × P Y | X , where P 2 ,X is the marginal distribution of P 2 over features. Since the conditional distribution of labels given features P Y | X is common to P 1 and P 2 , the test distribution is a covariate shifted version of the calibration distribution. Let E i denote the expectation over P i , i = 1 , 2 . Let x ↦→ r ( x ) = ( d P 2 ,X /d P 1 ,X )( x ) denote the unknown likelihood ratio function.

We consider both discrete and continuous label spaces Y . When Y = R , prediction sets correspond to prediction intervals. Recall that a prediction set C : X → 2 Y has marginal (1 -α ) -coverage in the test domain if P 2 [ Y ∈ C ( X )] ⩾ 1 -α . Let S : ( x, y ) ↦→ S ( x, y ) denote the nonconformity score associated to a pair ( x, y ) ∈ X × Y . Given a threshold function q : X → R , we consider the corresponding prediction set C : X → 2 Y given by C ( x ) = { y ∈ Y : S ( x, y ) ⩽ q ( x ) } for all x ∈ X . Thus a threshold function q yields a conformal predictor with marginal (1 -α ) -coverage in the test domain if P 2 [ S ( X,Y ) ⩽ q ( X )] ⩾ 1 -α . Note that the use of an adaptive threshold function is common in the conformal prediction literature, going back to [55]. We assume that α ⩽ 0 . 5 . For our theory, we consider [0 , 1] -valued scores; however, in Appendix O, we comment on conditions under which unbounded scores can be handled.

In this paper, a linear hypothesis class refers to a linear subspace of functions from X → R that are square-integrable with respect to P 1 ,X . An example is the space of functions representable by a pretrained model with a scalar read-out layer. If Φ : X → R d denotes the last hidden-layer feature map of the pretrained model, where Φ = ( ϕ 1 , . . . , ϕ d ) for ϕ i : X → R for all i ∈ [ d ] , then the linear class of functions representable by the network is given by {⟨ γ, Φ ⟩ : γ ∈ R d } , where ⟨· , ·⟩ is the ℓ 2 inner product on R d .

Problem statement. We observe n 1 labeled calibration (or, source) datapoints { ( X i , Y i ) : i ∈ [ n 1 ] } drawn i.i.d. from the source distribution P 1 , and an additional n 3 unlabeled calibration datapoints S 3 . We also have n 2 unlabeled (target) datapoints S 2 drawn i.i.d. from the target distribution P 2 . Given α ∈ (0 , 1) , our goal is to construct a threshold function q : X → R that achieves marginal (1 -α ) -coverage in the test domain: P 2 [ S ( X,Y ) ⩽ q ( X )] ⩾ 1 -α.

3 Algorithmic Principles

Here we present the intuition behind our approach. Our goal is to construct a prediction set of the form C ( x ) = { y ∈ Y : S ( x, y ) ⩽ q ( x ) } , where q should be close to a conditional quantile of S given X = x . The quantile loss ℓ α is designed such that for any random variable Z , the minimizers of the objective κ ↦→ E ℓ α ( κ, Z ) are the (1 -α ) th quantiles of Z . This has motivated prior work

[15, 22], where the authors minimize the objective h ↦→ E ℓ α ( h ( X ) , S ( X,Y )) for h in some linear hypothesis class H . At a minimizer h ∗ , the derivatives in all directions g ∈ H should be zero. Since the derivative of the pinball loss with respect to its first argument is given by

the chain rule implies that the directional derivative of h ↦→ E ℓ α ( h ( X ) , S ( X,Y )) in the direction g equals

where in the first step we interchanged derivative and expectation, applied the chain rule, and evaluated at ε = 0 , and in the second step we used the formula for the derivative of the pinball loss. Setting this equal to zero, if g takes the form g ( x ) = d Q X /d P 1 ,X ( x ) for some distribution Q X , then 2 this equality reads E Q [ 1 [ S ( X,Y ) ⩽ h ∗ ( X )] -(1 -α )] = 0 , which can be viewed as exact coverage under the covariate shift induced by g for the prediction set x ↦→ { y ∈ Y : S ( x, y ) ⩽ h ∗ ( x ) } . In other words, if the test distribution is Q = Q X × P Y | X , then we have the exact coverage result

Therefore, if the hypothesis class H is large enough to include the true likelihood ratio r = d P 2 ,X /d P 1 ,X , then the threshold function h ∗ attains valid coverage in the test domain P 2 , as desired.

3.1 Our approach

An adaptive choice of the hypothesis class. The above approach requires special assumptions on the hypothesis class H . The choice of the hypothesis class poses a challenge in practice: if H is too small, then coverage may fail, while if H is too large, then finite-sample performance may suffer due to large estimation errors.

To address this challenge, our idea is to choose H adaptively. We start by considering the class of hypotheses h that are close to the true likelihood ratio r , as measured by E 1 [( h ( X ) -r ( X )) 2 ] being small. By our remarks above, if we minimize E 1 [ ℓ α ( h ( X ) , S ( X,Y ))] for h restricted to this set, we obtain a threshold function with valid coverage under the covariate shift r .

Removing the explicit dependence on the likelihood ratio. The quantity E 1 [( h ( X ) -r ( X )) 2 ] depends on the unknown r . However, we can expand this to obtain

The term E 1 [ r ( X ) 2 ] does not depend on the optimization variable h , so it is enough to consider the first two terms. Due to the change-of-measure identity E 1 [ r ( X ) h ( X )] = E 2 [ h ( X )] , the sum of these terms equals

A key observation is that neither of the terms E 1 [ h ( X ) 2 ] or E 2 [ -2 h ( X )] explicitly involve r , and thus they can be estimated by sample averages over the source and target data, respectively. Thus, we can minimize E 1 [ ℓ α ( h ( X ) , S ( X,Y ))] over h ∈ H while keeping E 1 [ h ( X ) 2 ] + E 2 [ -2 h ( X )] bounded. The threshold h ∗ will have valid coverage under the covariate shift r .

Introducing a normalizing scalar. We also need to make sure that h is a valid likelihood ratio under d P 1 ,X , of the form g ( x ) = d Q X /d P 1 ,X ( x ) for some distribution Q X . This imposes the constraint ∫ h ( x ) d P 1 ,X ( x ) = 1 , which can be equivalently achieved for any non-negative h by scaling it with an appropriate scalar β . In our analysis, it turns out to be convenient to use the optimization variable βh and consider the class of functions h such that E 1 [( βh ( X ) -r ( X )) 2 ] is bounded for some scalar β ∈ R . By the above discussion, the term E 1 [ r ( X ) 2 ] is immaterial and it is sufficient to impose the constraint that min β ∈ R ( E 1 [ β 2 h ( X ) 2 ] + E 2 [ -2 βh ( X )]) is bounded.

Algorithm 1 Likelihood-ratio regularized quantile regression

Input: n 1 labeled source datapoints, n 2 unlabeled target datapoints, n 3 unlabeled source datapoints

1: Compute scores S i = S ( x i , y i ) for all i ∈ [ n 1 ]

2: Solve ( ˆ h, ˆ β ) ∈ arg min h ∈H ,β ∈ R ˆ E 1 [ ℓ α ( h ( X ) , S ( X,Y ))] + λ ˆ E 3 [ β 2 h ( X ) 2 ] + λ ˆ E 2 [ -2 βh ( X )] , where ˆ E 1 , ˆ E 2 , ˆ E 3 denote expectations over the source, unlabeled target, and unlabeled source data;

Return: Prediction set ˆ C ( x ) ←{ y ∈ Y : S ( x, y ) ⩽ ˆ h ( x ) } with asymptotic 1 -α coverage in the target distribution

Replacing the constraint with a regularization. Instead of imposing a constraint on min β ∈ R ( E 1 [ β 2 h ( X ) 2 ]+ E 2 [ -2 βh ( X )]) , we can use this term as a regularizer. Given a regularization strength λ ⩾ 0 , we can solve

Since the first term does not depend on β , this is equivalent to the joint optimization problem

3.2 Algorithm: likelihood ratio regularized quantile regression

Wesolve an empirical version of this objective. We use our labeled source data { ( X i , Y i ) : i ∈ [ n 1 ] } to estimate E 1 [ ℓ α ( h ( X ) , S ( X,Y ))] , our additional unlabeled source data S 3 to estimate E 1 [ β 2 h ( X ) 2 ] , and our unlabeled target data S 2 to estimate λ E 2 [ -2 βh ( X )] . Letting ˆ E 1 , ˆ E 2 , and ˆ E 3 denote empirical expectations over { ( X i , Y i ) : i ∈ [ n 1 ] } , S 2 , and S 3 , respectively, we then solve the following empirical likelihood ratio regularized quantile regression problem, for λ ⩾ 0 :

Our proposed threshold is q = ˆ h . See Algorithm 1. In the following section, we justify this algorithm through a novel theoretical analysis of the test-time coverage.

4 Theoretical Results

4.1 Infinite sample setting

We first consider the infinite sample or 'population" setting, characterizing the solutions of the LR-QR problem from (LR-QR) in an idealized scenario where the exact values of the expectations E 1 , E 2 can be calculated. In this case, we will show that if the hypothesis class H is linear and contains the true likelihood ratio r , then the optimizer achieves valid coverage in the test domain. Let r H be the projection of r onto H in the Hilbert space induced by the inner product ⟨ f, g ⟩ = E 1 [ fg ] . 3 The key step is the result below, which characterizes coverage weighted by r H .

Proposition 4.1. Let H be a linear hypothesis class consisting of square-integrable functions with respect to P 1 ,X . Then under regularity conditions specified in Appendix E (the conditions of Lemma L.3), if ( h ∗ , β ∗ ) = ( h ∗ λ , β ∗ λ ) is a minimizer of the objective in Equation (LR-QR) with regularization strength λ > 0 , then we have E 1 [ r H ( X ) 1 [ S ( X,Y ) ⩽ h ∗ ( X )]] ⩾ 1 -α.

The proof is given in Appendix I. As a consequence of Proposition 4.1, if H contains the true likelihood ratio r , so that r H = r , then in the infinite sample setting, the LR-QR threshold function h ∗ attains valid coverage at test-time:

However, in practice, we can only optimize over finite-dimensional hypothesis classes, and as a result we must control the effect of mis-specifying H . If r is not in H , we can derive a lower bound on the coverage as follows. First, write

By Proposition 4.1, the first term on the right-hand side is at least 1 -α . Since the random variable 1 [ S ( X,Y ) ⩽ h ∗ ( X )] is { 0 , 1 } -valued, the second term on the right-hand side is at least -E 1 [( r ( X ) -r H ( X )) + ] , where ( x ) + = max { 0 , x } for x ∈ R . We set our threshold function q to equal h ∗ , so that our conformal prediction sets equal C ∗ ( x ) = { y ∈ Y : S ( x, y ) ⩽ h ∗ ( x ) } for all x ∈ X . Thus, we have the lower bound

Geometrically, this coverage gap is the result of restricting to H . This error decreases if H is made larger, but in the finite sample setting, this comes at the risk of overfitting.

4.2 Finite sample setting

From the analysis of the infinite sample regime, it is clear that if the hypothesis class H is made larger, the test-time coverage of the population level LR-QR threshold function h ∗ moves closer to the nominal value. However, in the finite sample setting, optimizing over a larger hypothesis class also presents the risk of overfitting. By tuning the regularization parameter λ , we are trading off the estimation error incurred for the first term of Equation (LR-QR), namely ( ˆ E 1 -E 1 )[ ℓ α ( h ( X ) , S ( X,Y ))] , and the error incurred for the second and third terms of Equation (LRQR), namely λ ( ˆ E 3 -E 3 )[ β 2 h ( X ) 2 ] + λ ( ˆ E 2 -E 2 )[ -2 βh ( X )] . Heuristically, for a fixed h , the former should be proportional to 1 / √ n 1 , and the latter should be proportional to λ (1 / √ n 3 +1 / √ n 2 ) . Thus, if we pick λ to make these two errors of equal order, it will be proportional to √ ( n 2 + n 3 ) /n 1 .

Put differently, in order to ensure that the Empirical LR-QR threshold ˆ h from Equation (EmpiricalLR-QR) has valid test coverage, one must choose the regularization λ based on the relative amount of labeled and unlabeled data. The unlabeled datapoints carry information about the covariate shift r , because r depends only on the distribution of the features. The labeled datapoints provide information about the conditional (1 -α ) -quantile function q 1 -α , which depends only on the conditional distribution of S | X . When λ is large, our optimization problem places more weight on approximating r (the minimizer of E 1 [( βh ( X ) -r ( X )) 2 ] in βh ), and if λ is small, we instead aim to approximate q 1 -α (the minimizer of E 1 [ ℓ α ( h ( X ) , S ( X,Y ))] in h ). Therefore, if the number of unlabeled datapoints ( n 2 + n 3 ) is large compared to the number of labeled datapoints ( n 1 ), our data contains much more information about the covariate shift r , and we should set λ to be large. If instead n 1 is very large, the quantile function q 1 -α can be well-approximated from the labeled calibration datapoints, and we set λ to be close to zero. In the theoretical results, we make this intuition precise.

In order to facilitate our theoretical analysis in the finite sample setting, we consider constrained versions of Equation (LR-QR) and Equation (Empirical-LR-QR). Fix a collection Φ = ( ϕ 1 , . . . , ϕ d ) ⊤ of d basis functions, where ϕ i : X → R for i ∈ [ d ] . Let I = [ β min , β max ] ⊂ R be an interval with β min > 0 . Let H B = {⟨ γ, Φ ⟩ : ∥ γ ∥ 2 ⩽ B < ∞} be the B -ball centered at the origin in the linear hypothesis class spanned by { ϕ 1 , . . . , ϕ d } . We equip H B with the norm ∥ h ∥ = ∥ γ ∥ 2 for h = ⟨ γ, Φ ⟩ .

At the population level, consider the following constrained LR-QR problem: ( h ∗ , β ∗ ) ∈ arg min h ∈H B ,β ∈I L λ ( h, β ) . Also consider the following empirical constrained LR-QR problem 4 :

We begin by bounding the generalization error of an ERM ( ˆ h, ˆ β ) computed via Equation (2).

Theorem 4.2 (Suboptimality gap of ERM for likelihood ratio regularized quantile regression) . Under the regularity conditions specified in Appendix E, and for appropriate choices of the optimization hyperparameters 5 , for sufficiently large n 1 , n 2 , n 3 , with probability at least 1 -δ , any optimizer

( ˆ h, ˆ β ) of the empirical constrained LR-QR objective from (2) with regularization strength λ > 0 has suboptimality gap L λ ( ˆ h, ˆ β ) -L λ ( h ∗ , β ∗ ) with respect to the population risk (LR-QR) bounded by

and c, c ′ , c ′′ are positive scalars that do not depend on λ .

The proof is in Appendix J. The generalization error E gen is minimized for an optimal regularization on the order of

which yields an optimized upper bound of order E ∗ gen = O ( n -1 / 3 1 (1 /n 2 +1 /n 3 ) 1 / 6 +1 / √ n 1 ) . As can be seen from Appendix F, c, c ′ , c ′′ depend only polynomially on the radius B .

As a corollary of Theorem 4.2, we have the following lower bound on the excess marginal coverage of our ERM threshold ˆ h in the covariate shifted domain. Let r B denote the projection of r onto the closed convex set H B in the Hilbert space induced by the inner product ⟨ f, g ⟩ = E 1 [ fg ] .

Theorem 4.3 (Main result: Coverage under covariate shift) . Under the same conditions as Theorem 4.2, consider the LR-QR optimizers ˆ h and ˆ β from (2) with regularization strength λ > 0 . Given any δ > 0 , for sufficiently large n 1 , n 2 , n 3 , we have with probability at least 1 -δ that 6

where E cov := A (1 /n 2 +1 /n 3 ) 1 / 4 λ + A ′ ( λn 1 ) -1 / 4 + λ 1 / 2 /n 1 / 4 1 , r B denotes the projection of r onto H B , and A,A ′ are positive scalars that do not depend on λ .

The proof is in Appendix K. This result states that our LR-QR method has nearly valid coverage at level 1 -α under covariate shift, up to small error terms that we can control. The quantity E cov vanishes as we collect more data. The term E 1 [ | r ( X ) -r B ( X ) | ] captures the level of mis-specification by not including the true likelihood ratio function r in our hypothesis class H B . This can be decreased by making the hypothesis class H B larger. Of course, this will also increase the size of the terms A,A ′ in our coverage error, but in our theory we show that the dependence is mild. Indeed, the terms depend only on a few geometric properties of H B : they depend polynomially on the radius B , on the eigenvalues of the sample covariance matrix of the basis Φ( X ) under the source distribution, and on a quantitative measure of linear dependence of the features; but not explicitly on the dimension of the basis. We also note that the dimension of the feature space dim ( X ) does not appear in our results; only dim ( H ) affects our bounds.

We highlight the term 2 ˆ βλ E 1 [( r B ( X ) -ˆ β ˆ h ( X )) 2 ] , which is an error term relating the projected likelihood ratio r B to the LR-QR solution ˆ β ˆ h . Crucially, this term is a non-negative quantity multiplied by λ , and so for appropriate λ it may counteract in part the coverage error loss. Consistent with the above observations, we find empirically that choosing small nonzero regularization parameters improves coverage. Moreover, we find that choosing the regularization parameter to be on the order of the optimal value for E cov is suitable choice across a range of experiments.

Our proofs are quite involved and require a number of delicate arguments. Crucially, they draw on a novel analysis of coverage via stability bounds from learning theory. Existing stability results cannot directly be applied, due to our use of a data-dependent regularizer. For instance, in classical settings, the optimal regularization tends to zero as the sample size goes to infinity, but this is not the case here. To overcome this challenge, we combine stability bounds [48, 49] with a novel conditioning argument, and we show that the values of L at the minimizers of ˆ L and L are close by introducing intermediate losses that sequentially swap out empirical expectations ˆ E 1 , ˆ E 2 , ˆ E 3 with their population counterparts. We then leverage the smoothness of L , to derive that the gradient of L at ( ˆ β, ˆ h ) is small. Finally, we show that a small gradient implies the desired small coverage gap.

Black

White

Asian

Hispanic

Features

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Coverage

Black

White

Asian

Hispanic

Features

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Size

Nominal Coverage (0.8)

LR-QR

Weighted-CP

Split conformal

2R-CP

DRO-CP

DR-iso

Robust-CP

5 Experiments

We compare our method with the following baselines: (1) Split/inductive conformal prediction [31, 36]; (2) Weighted-CP: Weighted conformal prediction [55]; (3) 2R-CP: The doubly robust method from Yang et al. [64]; (4) DRO-CP: Distributionally robust optimization [7]; (5) DR-iso: Isotonic distributionally robust optimization [19]; (6) Robust-CP: Robust weighted conformal prediction [1].

5.1 Choosing the Regularization Parameter

Equation (3) suggests an optimal choice of the regularization parameter λ in the LR-QR algorithm. Guided by this, we form a uniform grid of size ten from λ ∗ / 10 to λ ∗ . We then perform three-fold cross-validation over the combined calibration and unlabeled target datasets (without using any labeled test data) as follows: we train the LR-QR threshold for each λ , and compute as a validation measure the ℓ 2 -norm of the gradient of the LR-QR objective on the held-out fold. We pick λ with the smallest average validation measure across all folds.

This validation measure is motivated by our algorithmic development: the first-order conditions of the LR-QR objective play a fundamental role in ensuring valid coverage in the test domain. While the model is trained to satisfy these conditions on the observed data, we seek to ensure this property generalizes well to unseen data. Thus, our selection criterion is based on two key observations: (1) a small gradient of the LR-QR objective implies reliable coverage, and (2) the regularization parameter λ balances the generalization error of the two terms in LR-QR. By minimizing this measure, we select a λ that optimally trades off these competing factors.

Finally, we re-train the LR-QR threshold on the entire calibration and unlabeled target datasets using this best λ , and report coverage and interval size on the held-out labeled test set. This ensures that no test labels are used during hyperparameter tuning. Additionally, in Appendix B, we provide deeper insights on different regimes of regularization in practice through an ablation study.

5.2 Communities and Crime

We evaluate our methods on the Communities and Crime dataset [42], which contains 1994 datapoints corresponding to communities in the United States, with socio-economic and demographic statistics. The task is to predict the (real-valued) per-capita violent crime rate from a 127-dimensional input.

We first randomly select half of the data as a training set, and use it to fit a ridge regression model ˆ f as our predictor. We tune the ridge regularization with five-fold cross-validation. We use the remaining half to design four covariate shift scenarios, determined by the frequency of a specific racial subgroup (Black, White, Hispanic, and Asian). For each of these features, we find the median value m over the remaining dataset. Datapoints with feature value at most m form our source set, and the rest form our target set. In other words, in each scenario, the source set consists of data points with below-median frequency of the specified racial subgroup, while the target set contains those with above-median frequency. This creates a covariate shift between calibration and test, as the split procedure only observes the covariates and is independent of labels. We then further split the target set into roughly equal unlabeled and labeled subsets. The unlabeled subset and the calibration data (without the labels) is used to estimate r , while the labeled test subset is held out only for final evaluation. The same procedure is applied to each of the four racial subgroups, creating four distinct partitions.

8

10

28

29

30

31

34

50

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Coverage

8

10

28

29

30

31

34

50

0

200

400

600

800

1000

Weighted-CP

Standard

2R-CP

DRO-CP

LR-QR

Robust-CP

DR-iso

Nominal Coverage (0.9)

Experimental details. The nonconformity score is s ( x, y ) = | y -ˆ f ( x ) | . Several baselines require an estimate of the likelihood ratio r , which we obtain by training a logistic regression model ˆ p to distinguish unlabeled source and target data. We then set ˆ r = ˆ p 1 -ˆ p , where ˆ p ( x ) is the predicted probability that x came from the target distribution. The hypothesis class H consists of all linear maps from the feature space to R . All experimental results are averaged over 1000 random splits.

Results. Figure 1 displays the results. Notably, split conformal undercovers in two setups and overcovers in the other two. Methods that estimate r and DRO fail to track the nominal coverage, particularly in the first setup on the left. However, the LR-QR method is closer to the nominal level of coverage, showing a stronger adaptivity to the covariate shift.

5.3 RxRx1 data - WILDS

Our next experiment uses the RxRx1 dataset [53] from the WILDS repository [25], which is designed to evaluate model robustness under distribution shifts. The RxRx1 task involves classifying cell images based on 1339 laboratory genetic treatments. These images, captured using fluorescent microscopy, originate from 51 independent experiments. Variations in execution and environmental conditions lead to systematic differences across experiments, affecting the distribution of input features (e.g., lighting, cell morphology) while the relationship between inputs and labels remains unchanged. This situation creates covariate shift where the marginal distribution of inputs shifts across domains, but the conditional distribution P Y | X remains the same.

We use a ResNet50 model [20] trained by the WILDS authors on 37 of the 51 experiments. Using the other experiments, we construct 14 distinct evaluations, where each experiment is selected as the target dataset, and its data is evenly split into an unlabeled target set and a labeled test set. The labeled data from the other 13 experiments serves as the source dataset.

Experimental details. The nonconformity score is s ( x, y ) = -log f x ( y ) , where f x ( y ) is the probability assigned the image-label pair ( x, y ) . To estimate r , we train a logistic regression model ˆ p on top of the representation layer of the pretrained model to distinguish unlabeled source and target data, and we set ˆ r = ˆ p 1 -ˆ p . We set the hypothesis class H to be a linear head on top of the representation layer of the pretrained model. Experimental results are averaged over 50 random splits.

Results. Figure 2 presents the coverage and average prediction set size for all methods. To enhance visual interpretability, we display results for eight randomly selected settings out of the 14, with the full plot provided in Figure 3 in the Appendix. The x-axis shows the indices of the test condition. LR-QR adheres more closely to the nominal coverage value of 0 . 9 compared to other methods.

Notably, split conformal prediction, which assumes exchangeability between calibration and test data, shows under- and overcoverage due to the covariate shift. The coverage of weighted CP and 2R-CP is also far from the nominal level, showing that directly estimating the likelihood ratio and conditional quantile is insufficient to correct the coverage violations in the case of high-dimensional image data. Further, the superior coverage of LR-QR is not due to inflated prediction sets.

5.4 Multiple choice questions - MMLU

Finally, we evaluate all methods using the MMLU benchmark, which covers 57 subjects spanning a wide range of difficulties. To induce a covariate shift, we partition the dataset by subject difficulty: prompts from subjects labeled as elementary or high school are used for calibration, while those from college and professional subjects form the test set.

Motivated by the design from [26], we follow a prompt-based scoring scheme adapted for LLMs: we append the string 'The answer is the option:' to the end of each MMLU question and feed the resulting prompt into the Llama 13B model without generating any output. We then extract the next-token logits corresponding to the first decoding position (i.e., immediately after the prompt) and consider the logits associated with the characters A , B , C , and D . These four logits are normalized using the softmax function to produce a probability vector over the answer options.

Experimental details. The nonconformity score is s ( x, y ) = 1 -f ( x ) y , where f ( x ) y is the probability assigned to the correct answer. For ˆ r and H , we compute prompt embeddings as follows. We extract the final hidden layer outputs from GPT-2 Small to obtain 768-dimensional embeddings. We then apply average pooling across all token embeddings in a prompt to obtain a single fixed-length vector representation for each input. We fit a probabilistic classifier ˆ p using logistic regression on the unlabeled pooled embeddings from the source and target data, and we set ˆ r = ˆ p 1 -ˆ p . We set H to be a linear head on top of the representation layer of the pretrained model.

Results. As shown in Table 1, our LR-QR method achieves near-nominal coverage and has the smallest average prediction set size among methods that achieve approximately 90% or higher coverage, demonstrating both validity and efficiency under covariate shift.

6 Discussion

We proposed the LR-QR method to construct prediction sets under covariate shift. While we have provided strong guarantees on the coverage of our method, it would be desirable to have results that control of the slack in coverage in specific scenarios depending on the structure of the likelihood ratio and the hypothesis space. Our work concerns uncertainty quantification and may have positive social impact for reliable decision-making. We do not envision any negative social impact of our work.

7 Acknowledgments

ED and SJ were supported by NSF, ARO, ONR, AFOSR, and the Sloan Foundation. The work of HH, SK, and GP was supported by the NSF Institute for CORE Emerging Methods in Data Science (EnCORE) and the ASSET (AI-Enabled Systems: Safe, Explainable and Trustworthy) Center.

References

Jiahao Ai and Zhimei Ren. Not all distributional shifts are equal: Fine-grained robust conformal inference. arXiv preprint arXiv:2402.13042 , 2024.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics , 51(2):816-845, 2023.

Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics , 51(1):149-178, 2023.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79: 151-175, 2010.

Aabesh Bhattacharyya and Rina Foygel Barber. Group-weighted conformal prediction. arXiv preprint arXiv:2401.17452 , 2024.

Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association , pages 1-66, 2024.

Victor Chernozhukov, Kaspar Wuthrich, and Yinchu Zhu. Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data. In Proceedings of the 31st Conference On Learning Theory, PMLR , volume 75, pages 732-749. PMLR, 2018. URL http://arxiv.org/abs/1802.06300 .

Edgar Dobriban and Mengxin Yu. Symmpi: Predictive inference for data with group symmetries. arXiv preprint arXiv:2312.16160 , 2023.

Robin Dunn, Larry Wasserman, and Aaditya Ramdas. Distribution-free prediction sets for two-layer hierarchical models. Journal of the American Statistical Association , pages 1-12, 2022.

Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertaintyaware classifiers with conformalized deep learning. Advances in Neural Information Processing Systems , 2022.

Jerome H Friedman. On multivariate goodness-of-fit and two-sample testing. Statistical Problems in Particle Physics, Astrophysics, and Cosmology , 1:311-313, 2003.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In 32nd International Conference on Machine Learning, ICML 2015 , volume 2, pages 1180-1189. PMLR, 2015. ISBN 9781510810587.

Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology , page qkaf008, 2025.

Leying Guan. A conformal test of linear models via permutation-augmented regressions. arXiv preprint arXiv:2309.05482 , 2023.

Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika , 110(1):33-50, 2023.

Leying Guan and Robert Tibshirani. Prediction and outlier detection in classification problems. Journal of the Royal Statistical Society: Series B , 84(2):524-546, 2022.

Yu Gui, Rina Foygel Barber, and Cong Ma. Distributionally robust risk evaluation with an isotonic constraint. arXiv preprint arXiv:2407.06867 , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Batch multivalid conformal prediction. In International Conference on Learning Representations (ICLR) , 2023.

Kevin Kasa, Zhiyu Zhang, Heng Yang, and Graham W Taylor. Adapting conformal prediction to distribution shifts without labels. arXiv preprint arXiv:2406.01416 , 2024.

Ramneet Kaur, Susmit Jha, Anirban Roy, Sangdon Park, Edgar Dobriban, Oleg Sokolsky, and Insup Lee. idecode: In-distribution equivariance for conformal out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: Abenchmark of in-the-wild distribution shifts. In International conference on machine learning , pages 5637-5664. PMLR, 2021.

Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404 , 2023.

Yonghoon Lee, Eric Tchetgen Tchetgen, and Edgar Dobriban. Batch predictive inference. arXiv preprint arXiv:2409.13990 , 2024.

Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76(1):71-96, 2014.

Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets. Journal of the American Statistical Association , 108(501):278-287, 2013.

Jing Lei, Alessandro Rinaldo, and Larry Wasserman. A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence , 74(1):29-43, 2015.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distributionfree predictive inference for regression. Journal of the American Statistical Association , 113 (523):1094-1111, 2018.

Lihua Lei and Emmanuel J. Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society. Series B: Statistical Methodology , 83(5):911-938, 2021. ISSN 14679868. doi: 10.1111/rssb.12445. URL http://arxiv.org/ abs/2006.06138 .

Shuo Li, Xiayan Ji, Edgar Dobriban, Oleg Sokolsky, and Insup Lee. Pac-wrap: Semi-supervised pac anomaly detection. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022.

Ziyi Liang, Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers. arXiv preprint arXiv:2208.11111 , 2022.

Ziyi Liang, Yanfei Zhou, and Matteo Sesia. Conformal inference is (almost) free for neural networks trained with early stopping. In International Conference on Machine Learning , 2023.

Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning , pages 345-356. Springer, 2002.

Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. PAC prediction sets under covariate shift. In International Conference on Learning Representations , 2022.

Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. PAC prediction sets for meta-learning. In Advances in Neural Information Processing Systems , 2022.

Jing Qin, Yukun Liu, Moming Li, and Chiung-Yu Huang. Distribution-free prediction intervals under covariate shift, with an application to causal inference. Journal of the American Statistical Association , 0(0):1-26, 2024. doi: 10.1080/01621459.2024.2356886. URL https://doi. org/10.1080/01621459.2024.2356886 .

Hongxiang Qiu, Edgar Dobriban, and Eric Tchetgen Tchetgen. Prediction sets adaptive to unknown covariate shift. Journal of the Royal Statistical Society Series B: Statistical Methodology , 85(5):1680-1705, 2023.

Joaquin Quiñonero-Candela, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset shift in machine learning . Mit Press, 2009.

Michael Redmond. Communities and Crime. UCI Machine Learning Repository, 2002. DOI: https://doi.org/10.24432/C53W3X.

Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 2020.

Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least Ambiguous Set-Valued Classifiers With Bounded Error Levels. Journal of the American Statistical Association , 114(525):223-234, 2019. ISSN 1537274X. doi: 10.1080/01621459.2017.1395341.

Craig Saunders, Alexander Gammerman, and Volodya Vovk. Transduction with confidence and credibility. In IJCAI , 1999.

Henry Scheffe and John W Tukey. Non-parametric estimation. i. validation of order statistics. The Annals of Mathematical Statistics , 16(2):187-192, 1945.

Matteo Sesia, Stefano Favaro, and Edgar Dobriban. Conformal frequency estimation using discrete sketched data with coverage for distinct queries. Journal of Machine Learning Research , 24(348):1-80, 2023.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research , 11:2635-2670, 2010.

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 90(2):227-244, 2000.

Wenwen Si, Sangdon Park, Insup Lee, Edgar Dobriban, and Osbert Bastani. PAC prediction sets under label shift. International Conference on Learning Representations , 2024.

Masashi Sugiyama and Motoaki. Kawanabe. Machine learning in non-stationary environments : introduction to covariate shift adaptation . MIT Press, 2012. ISBN 9780262017091.

Maciej Sypetkowski, Morteza Rezanejad, Saber Saberian, Oren Kraus, John Urbanik, James Taylor, Ben Mabey, Mason Victors, Jason Yosinski, Alborz Rezazadeh Sereshkeh, et al. Rxrx1: Adataset for evaluating experimental batch correction methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4285-4294, 2023.

Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems , 33:18583-18599, 2020.

Ryan J Tibshirani, Rina Foygel Barber, Emmanuel J Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems , 32, 2019.

John W Tukey. Non-parametric estimation ii. statistically equivalent blocks and tolerance regions-the continuous case. The Annals of Mathematical Statistics , pages 529-539, 1947.

John W Tukey. Nonparametric estimation, iii. statistically equivalent blocks and multivariate tolerance regions-the discontinuous case. The Annals of Mathematical Statistics , pages 30-39, 1948.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning , volume 25, pages 475-490. PMLR, 2013. doi: 10.1007/s10994-013-5355-6.

Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer Science & Business Media, 2005.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World . Springer Nature, 2022.

Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In International Conference on Machine Learning , 1999.

Abraham Wald. An Extension of Wilks' Method for Setting Tolerance Limits. The Annals of Mathematical Statistics , 14(1):45-55, 1943. ISSN 0003-4851. doi: 10.1214/aoms/1177731491.

S. S. Wilks. Determination of Sample Sizes for Setting Tolerance Limits. The Annals of Mathematical Statistics , 12(1):91-96, 1941. ISSN 0003-4851. doi: 10.1214/aoms/1177731788.

Yachong Yang, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Doubly robust calibration of prediction sets under covariate shift. Journal of the Royal Statistical Society Series B: Statistical Methodology , page qkae009, 2024.

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2636-2645, 2020.

Tianyi Zhang, Ikko Yamane, Nan Lu, and Masashi Sugiyama. A one-step approach to covariate shift adaptation. In Asian Conference on Machine Learning , pages 65-80. PMLR, 2020.

A Additional figures

B Ablation studies

Here we provide an ablation study for λ , the regularization strength that appears in the LR-QR objective. In the same regression setup as Section 5.2, instead of selecting λ via cross-validation, here we sweep the value of λ from 0 to 1 , and we plot the coverage of the LR-QR algorithm on the test data. Here, note that the split ratios between train, calibration, and test (both labeled and unlabeled data) are fixed and similar to the setup in Section 5.2. We report the averaged plots over 100 independent splits.

Figure 4 displays the effect of different regimes of λ . At one extreme, when λ is close to zero, the LRQR algorithm reduces to ordinary quantile regression. In this regime, the LR-QR algorithm behaves similarly to the algorithm from [15], without the test covariate imputation. In other words, when we set λ = 0 , we try to provide coverage with respect to all the covariate shifts in the linear hypothesis class that we optimize over. As we can see in Figure 4, this can lead to overfitting and undercoverage of the test labels. As we increase λ , as a direct effect of the regularization, the coverage gap decreases. This is primarily due to the fact that larger λ restricts the space of quantile regression optimization in such a way that it does not hurt the test time coverage, since the regularization is designed to shrink the optimization space towards the true likelihood-ratio. Thus, the regularization improves the generalization of the selected threshold, as the effective complexity of the hypothesis class is getting smaller. That being said, this phenomenon is only applicable if λ lies within a certain range; once λ grows too large, due to the data-dependent nature of our regularization, the generalization error of the regularization term itself becomes non-negligible and hinders the precise test-time coverage of the LR-QR threshold. As is highlighted in Figure 4, our theoretical results suggest an optimal regime for λ which can best exploit the geometric properties of the LR-QR threshold.

C Related work

The basic concept of prediction sets dates back to foundational works such as Wilks [63], Wald [62], Scheffe and Tukey [46], and Tukey [56, 57]. The early ideas of conformal prediction were developed in Saunders et al. [45], V ovk et al. [61]. With the rise of machine learning, conformal prediction has emerged as a widely used framework for constructing prediction sets [e.g., 2, 4, 8-11, 16-18, 2731, 34-36, 43, 58, 59]. A wide range of predictive inference methods have been developed [e.g., 14, 24, 33, 37, 38, 40, 44, 47, 51].

Coverage Comparison

8

9

10

28

29

30

31

32

33

34

43

44

45

50

0.70

0.75

0.80

0.85

0.90

0.95

1.00

1.05

Coverage

Weighted-CP

Standard

2R-CP

DRO-CP

LR-QR

Robust-CP

DR-iso

Nominal Coverage (0.9)

8

9

10

28

29

30

31

32

33

34

43

44

45

50

0

200

400

600

800

1000

Weighted-CP

Standard

2R-CP

DRO-CP

LR-QR

Robust-CP

DR-iso

the regime that our theory suggests

no regularization

large regularization

Numerous works have addressed conformal prediction under various types of distribution shift [37, 38, 40, 51, 55]. For example, Tibshirani et al. [55] investigated conformal prediction under covariate shift, assuming the likelihood ratio between source and target covariates is known. Lei and Candès [32] allowed the likelihood ratio to be estimated, rather than assuming it is known. Park et al. [37] developed prediction sets with a calibration-set conditional (PAC) property under covariate shift. [3] present the nonexchangeable conformal prediction algorithm for arbitrary distribution shifts, assuming that the optimal weights for their method are known. Qiu et al. [40], Yang et al. [64] developed prediction sets with asymptotic coverage that are doubly robust in the sense that their coverage error is bounded by the product of the estimation errors of the quantile function of the score and the likelihood ratio. Cauchois et al. [7] construct prediction sets based on a distributionally robust optimization approach. Gui et al. [19] develop methods based on an isotonic regression estimate of the likelihood ratio. They provide theoretical guarantees for the difference between the population-level distributionally robust risk and its empirical counterpart. However, their results do not directly lead to coverage guarantees under distribution shift in our setting, as that would further require characterizing the effect of estimating the likelihood ratio.

Qin et al. [39] combine a parametric working model with a resampling approach to construct prediction sets under covariate shift. Bhattacharyya and Barber [6] analyze weighted conformal prediction in the special case of covariate shifts defined by a finite number of groups. Ai and Ren [1] reweight samples to adapt to covariate shift, while simultaneously using distributionally robust optimization to protect against worst-case joint distribution shifts. Kasa et al. [23] construct prediction sets by using unlabeled test data to modify the score function used for conformal prediction.

Our algorthm works by constructing a novel regularized regression objective, whose stationary conditions ensure coverage in the test domain. We can minimize the objective by estimating certain expectations of the data distribution-which implicitly involve estimating only certain functionals of the likelihood ratio. We further show that the coverage is retained in finite samples via a novel analysis of coverage leveraging stability bounds [48, 49]. We illustrate that our algorithms behave better in high-dimensional datasets than existing methods.

D Notation and conventions

Constants are allowed to depend on dimension only through properties of the population and sample covariance matrices of the features, and the amount of linear independence of the features; see the quantities λ min (Σ) , λ max, c min, c max, and c indep defined in Appendix E. In the Landau notation ( o , O , Θ ), we hide constants. We say that a sequence of events holds with high probability if the probability of the events tends to unity. We define S 1 as the features of the labeled calibration dataset. All functions that we minimize can readily be verified to be continuous, and thus attain a minimum over the compact domains over which we minimize them; thus all our minimizers will be well-defined. We may not mention this further. We denote by 1 [ A ] the indicator of an event A . Recall that H denotes the linear hypothesis class H = {⟨ γ, Φ ⟩ : γ ∈ R d } . This defines a one-to-one correspondence between R d and H . This enables us to view functions defined on R d equivalently as defined on H . In our analysis, we will use such steps without further discussion. Unless stated otherwise, H is equipped with the norm ∥ h ∥ := ∥ γ ∥ 2 for h = ⟨ γ, Φ ⟩ . Given a differentiable function φ : H → R , its directional derivative at f = ⟨ γ, Φ ⟩ ∈ H in the direction defined by the function g ∈ H is defined as d dε ∣ ∣ ε =0 φ ( f + εg ) . Note that if we write g = ⟨ ˜ γ, Φ ⟩ for some ˜ γ ∈ R d , then the directional derivative of φ at f equals ⟨ ˜ γ, ∇ γ φ ( γ ) ⟩ , where ∇ γ φ ( γ ) denotes the gradient of φ : R d → R evaluated at γ ∈ R d . When it is clear from context, we drop the subscript λ from the risks L λ and ˆ L λ .

E Conditions

Condition 1. Suppose C Φ = sup x ∈X ∥ Φ( x ) ∥ 2 is finite.

Condition 2. For the population covariance matrix Σ = E 1 [ΦΦ ⊤ ] , we have λ min (Σ) > 0 and λ max (Σ) is of constant order, not depending on the sample size, or any other problem parameter.

Condition 3. For the sample covariance matrix ˆ Σ = 1 n 3 ∑ n 3 k =1 Φ( x k )Φ( x k ) ⊤ , we have both λ min ( ˆ Σ) ⩾ c min > 0 and λ max ( ˆ Σ) ⩽ c max of constant order with probability 1 -o ( n -1 3 ) .

Condition 4. Defining C 1 as in (7) in Appendix H, assume there exists an upper bound C 1 , upper on E [ C 1 ] of constant order.

Condition 5. The conditional density f S | X = x exists for all x ∈ X , and C f = sup x ∈X ∥ f S | X = x ( s ) ∥ ∞ is a finite constant.

The following can be interpreted as an independence assumption on the basis functions.

Condition 6. Suppose inf v ∈ S d -1 E 1 [ |⟨ v, Φ ⟩| ] ⩾ c indep > 0 for some constant c indep .

Condition 7. Suppose E 1 [ rh ∗ 0 ] E 1 [ | h ∗ 0 | 2 ] 1 / 2 ⩾ c align > 0 for some minimizer h ∗ 0 of the objective in Equation (18) with regularization λ = 0 .

Condition 8. Suppose E 1 [ r 2 ] is finite.

Condition 9. The constant function h : X → R given by h ( x ) = 1 for all x ∈ X is in H .

The following ensures that the zero function 0 ∈ H is not a minimizer of the objective in Equation (LRQR).

Condition 10. For each λ ⩾ 0 , there exists h ∈ H and β ∈ R such that

F Constants

The following are the constants that appear in Theorem 4.2:

Further,

The following are the constants that appear in Theorem 4.3:

G Generalization bound for regularized loss

The following is a generalization of Shalev-Shwartz and Ben-David [48, Corollary 13.6].

LemmaG.1 (Generalization bound for regularized loss; extension of [48]) . Fix a compact and convex hypothesis class ˜ H equipped with a norm ∥ · ∥ ˜ H , a compact interval I ⊆ R , and a sample space Z . Consider the objective function f : ˜ H×I×Z → R given by ( h, β, z ) ↦→ f ( h, β, z ) := J ( h, β, z ) + R ( h, β ) , where R : ˜ H×I → R is a regularization function, and J : ˜ H×I × Z → R can be decomposed as J ( h, β, z ) := J 1 ( h, β, z 1 )+ J 2 ( h, β, z 2 ) for two functions J 1 , J 2 : ˜ H×I×Z → R .

Given distributions D 1 , D 2 on Z , let L : ˜ H×I → R be given for all h, β by

denote the population risk, averaging over independent datapoints Z 1 ∼ D 1 and Z 2 ∼ D 2 . Suppose that for both Z ∼ D 1 and Z ∼ D 2 , |J 1 ( h, β, Z ) | and |J 2 ( h, β, Z ) | are almost surely bounded by a quantity not depending on h ∈ ˜ H and β ∈ I .

Let ˆ L : ˜ H×I → R denote the empirical risk computed over Z i, 1 i.i.d. ∼ D 1 , i ∈ [ m 1 ] and Z j, 2 i.i.d. ∼ D 2 , j ∈ [ m 2 ] , given by

Assume that for each fixed β ∈ I and z ∈ Z ,

h ↦→J 1 ( h, β, z ) is convex and ρ -Lipschitz with respect to the norm ∥ · ∥ ˜ H ,

h ↦→J 2 ( h, β, z ) is convex and ρ -Lipschitz with respect to the norm ∥ · ∥ ˜ H , and

h ↦→ ˆ L ( h, β ) is µ -strongly convex with respect to the norm ∥ · ∥ ˜ H with probability 1 -o ( m -1 1 + m -1 2 ) ,

where the deterministic values µ = µ ( β ) and ρ = ρ ( β ) may depend on β .

Let ( ˆ h, ˆ β ) denote an ERM, i.e., a minimizer of ˆ L ( h, β ) over ˜ H×I . Let ˆ h β denote a minimizer of the empirical risk in h for fixed β .

Suppose the stochastic process β ↦→ W β given by W β = L ( ˆ h β , β ) -ˆ L ( ˆ h β , β ) for β ∈ I obeys | W β -W β ′ | ⩽ K | β -β ′ | for all β, β ′ ∈ I for some random variable K , and suppose that the probability of K m 1 ,m 2 ⩽ K max converges to unity as m 1 , m 2 → ∞ , for some constant K max . Suppose that there exists a constant C > 0 such that for all β ∈ I ,

Then for sufficiently large m 1 , m 2 , with probability at least 1 -δ ,

Remark G.2 . A special case is when we do not have any data from D 2 , and instead all m 1 datapoints are sampled i.i.d. from D 1 . In this case, defining with a slight abuse of notation J := J 1 , the statement simplifies to the analysis of the empirical risk

If for each fixed β ∈ I , we have that h ↦→J ( h, β, z ) is convex and ρ -Lipschitz with respect to the norm ∥ · ∥ ˜ H , and if |J ( h, β, Z ) | is almost surely bounded by a quantity not depending on h ∈ ˜ H and β ∈ I for Z ∼ D 1 = D 2 , then under the remaining assumptions, we obtain the slightly stronger bound

We omit the proof, because it is exactly as below.

Remark G.3 . Werelax the strong convexity assumption on the regularizer R from Shalev-Shwartz and Ben-David [48, Corollary 13.6], substituting it with the less restrictive condition of strong convexity of the empirical loss ˆ L . In order to use assumptions that merely hold with high probability, we impose a boundedness condition on J .

Proof. Fix β and let E denote the event that h ↦→ ˆ L ( h, β ) is µ -strongly convex in h . By assumption, E occurs with probability 1 -o ( m -1 1 + m -1 2 ) .

We modify the proof of Shalev-Shwartz and Ben-David [48, Corollary 13.6] as follows. Let Z ′ 1 ∼ D 1 and Z ′ 2 ∼ D 2 be drawn independently from all other randomness. For a fixed i ∈ [ m 1 ] , let h ↦→ ˆ L i, 1 ( h, β ) denote the empirical risk computed from the sample ( Z 1 , 1 , . . . , Z i -1 , 1 , Z ′ 1 , Z i +1 , 1 , . . . , Z m 1 , 1 ) ∪ ( Z 1 , 2 , . . . , Z m 2 , 2 ) , and let ˆ h ( i ) β denote an ERM for this

sample. Let I be drawn from [ m 1 ] uniformly at random. The variables J, ˆ L J, 2 ( h, β ) , ˆ h ( J ) β are defined similarly but for the sample from D 2 .

Note that for fixed β , similarly to the argument in Shalev-Shwartz and Ben-David [48, Theorem 13.2], we have

and

Therefore

Further, splitting the expectations over E and its complement E c , this further equals

On the event E , h ↦→ ˆ L ( h, β ) is µ -strongly convex. Now, consider the setting of Shalev-Shwartz and Ben-David [48, Corollary 13.6]. We claim that the arguments in their proof hold if we replace the regularizer h ↦→ λ ∥ h ∥ 2 by h ↦→ R ( h, β ) , as they only leverage the strong convexity of the overall empirical loss ˆ L . Indeed, working on the event E , since ˆ L is µ -strongly convex, we have that ˆ L ( h ) -ˆ L ( ˆ h β ) ⩾ 1 2 µ ∥ h -ˆ h β ∥ 2 for all h ∈ ˜ H . Next, for any h 1 , h 2 ∈ ˜ H , we have

Setting h 2 = ˆ h ( I ) β and h 1 = ˆ h , since ˆ h ( I ) β minimizes h ↦→ ˆ L I, 1 ( h, β ) , and using our lower bound on ˆ L ( h ) -ˆ L ( ˆ h β ) , we deduce

Since by assumption, h ↦→ J 1 ( h, β, z ) is ρ -Lipschitz, we have the bounds |J 1 ( ˆ h ( I ) β , β, Z I, 1 ) -J 1 ( ˆ h β , β, Z I, 1 ) | ⩽ ρ | ˆ h ( I ) β -ˆ h β | and |J 1 ( ˆ h ( I ) β , β, Z ′ 1 ) -J 1 ( ˆ h β , β, Z ′ 1 ) | ⩽ ρ | ˆ h ( I ) β -ˆ h β | . Plugging these into Equation (6), we obtain 1 2 µ ∥ ˆ h ( I ) β -ˆ h β ∥ 2 ⩽ 2 ρ m 1 ∥ ˆ h ( I ) β -ˆ h β ∥ , so that ∥ ˆ h ( I ) β -ˆ h β ∥ ⩽ 4 ρ ( β ) µ ( β ) m 1 . Using once again that h ↦→J 1 ( h, β, z ) is ρ -Lipschitz, we find |J 1 ( ˆ h ( I ) β , β, Z I, 1 ) -J 1 ( ˆ h β , β, Z I, 1 ) | ⩽ 4 ρ ( β ) 2 µ ( β ) m 1 .

Similarly, on the event E , we have the bound |J 2 ( ˆ h ( J ) β , β, Z J, 2 ) -J 2 ( ˆ h β , β, Z J, 2 ) | ⩽ 4 ρ ( β ) 2 µ ( β ) m 2 . Thus the first and third terms are bounded in magnitude by 4 ρ ( β ) 2 µ ( β ) m 1 and 4 ρ ( β ) 2 µ ( β ) m 2 , respectively. Due to (4), their sum is at most C ( m -1 1 + m -1 2 ) .

By our assumption that |J 1 ( h, β, Z ) | and |J 2 ( h, β, Z ) | are almost surely bounded by a constant for both Z ∼ D 1 and Z ∼ D 2 , and our assumption that P [ E c ] = o ( m -1 1 + m -1 2 ) , the second term and fourth terms from (5) sum to o ( m -1 1 + m -1 2 ) . Thus for for each β , for sufficiently large m 1 , m 2 , we have E [ | W β | ] ⩽ 2 C ( m -1 1 + m -1 2 ) . By Markov's inequality, for any fixed t > 0 , | W β | > t with

probability at most 2 C t ( m -1 1 + m -1 2 ) . We now use chaining. Let N be an ε -net for I . Then using the fact that by assumption, the process W is K m 1 ,m 2 -Lipschitz, and by a union bound,

Pick N with | N | = 1 /ε , and set t = 4 C δ ( m -1 1 + m -1 2 ) 1 ε . We deduce that

with probability at most δ 2 . Set ε = √ 4 C K m 1 ,m 2 δ ( m -1 1 + m -1 2 ) . We deduce that

with probability at most δ 2 . Since the probability of K m 1 ,m 2 ⩽ K max converges to unity, for sufficiently large m 1 , m 2 ,

holds with probability at most δ . Since | W ˆ β | ⩽ sup β ∈I | W β | , we may conclude.

H Lipschitz process

Lemma H.1 (Lipschitzness of minimizer of perturbed strongly convex objective) . Let C ⊆ R d be a closed convex set. Suppose ψ : C → R is µ -strongly convex and g : C → R is L -smooth. Suppose also that ψ + g is convex. Let x ψ denote the minimizer of ψ in C , and let x ψ + g denote the minimizer of ψ + g in C . Then for any x ∈ C ,

Proof. Since ψ is µ -strongly convex and since x ψ + g , x ψ are minimizers of ψ + g, ψ respectively,

so that by L -smoothness of g ,

which implies the result.

Lemma H.2 (Lipschitzness of minimizer of perturbed ERM) . Under Condition 1, with ˆ Σ from Condition 3, and with the notations of Lemma H.4, we have with respect to the norm ∥ · ∥ on H B that β ↦→ ˆ h β is C 1 -Lipschitz on I , and β ↦→ β ˆ h β is C 2 -Lipschitz on I , where

Proof. First, consider ˆ h β . Fix β > β ′ in I . Recalling the definition of ˆ L from (Empirical-LR-QR), the difference between the objectives ˆ L ( h, β ) and ˆ L ( h, β ′ ) is the quadratic

We claim that g is 2 λ ( β 2 -( β ′ ) 2 ) λ min ( ˆ Σ) -strongly convex and 2 λ ( β 2 -( β ′ ) 2 ) λ max ( ˆ Σ) -smooth in h . To see this, write h = ⟨ γ, Φ ⟩ for γ ∈ R d , and note that g can be rewritten as

a quadratic whose Hessian equals 2 λ ( β 2 -( β ′ ) 2 ) ˆ Σ , which implies the claim.

Similarly, we claim that the function ψ ( h ) := ˆ L ( h, β ′ ) is 2 λ ( β ′ ) 2 λ min ( ˆ Σ) -strongly convex in h . To see this, again write h = ⟨ γ, Φ ⟩ for γ ∈ R d , and note that ψ can be rewritten as

By Lemma N.3, the second term is convex, and since the third term is linear, it too is convex. The Hessian of the quadratic first term is 2 λ ( β ′ ) 2 ˆ Σ , from which it follows that ψ is 2 λ ( β ′ ) 2 λ min ( ˆ Σ) -strongly convex.

Thus ψ and g satisfy the conditions of Lemma H.1, which implies the bound

where ˆ h g = ˆ h g,β,β ′ denotes the minimizer of g in H B . Since

and by | β | , | β ′ | ⩽ β max, ∥ γ ∥ ⩽ B , and Condition 1, we have

for β, β ′ ∈ I and h ∈ H B . Plugging this into the bound (8) on ∥ ˆ h β -ˆ h β ′ ∥ and using the fact that β ′ ⩾ β min and ∥ ˆ h β ∥ , ∥ ˆ h g ∥ ⩽ B ,

Thus we may take

For the map β ↦→ β ˆ h β , fix β > β ′ in I , and write ∥ β ˆ h β -β ′ ˆ h β ′ ∥ ⩽ | β -β ′ |∥ ˆ h β ∥ + | β ′ |∥ ˆ h β -ˆ h β ′ ∥ . For the first term, note that since ˆ h β ∈ H B implies ∥ ˆ h β ∥ ⩽ B , the first term is bounded by B | β -β ′ | . For the second term, note that since | β ′ | ⩽ β max and since β ↦→ ˆ h β is C 1 -Lipschitz on I , the second term is bounded by β max C 1 | β -β ′ | . Summing, we deduce that β ↦→ β ˆ h β is C 2 -Lipschitz on I , where C 2 = B + β max C 1 .

Lemma H.3 (Lipschitzness of minimizer of perturbed auxiliary ERM) . Under Condition 1, we have that β ↦→ ˜ h β is C 1 -Lipschitz on I , and β ↦→ β ˜ h β is C 2 -Lipschitz on I .

Proof. The proof is almost identical to Lemma H.2.

Recalling c min and c max from Condition 3, define

so that by Condition 3, C 1 ⩽ C 1 , max and C 2 ⩽ C 2 , max with probability tending to unity over the randomness in S 3 .

We now compute the Lipschitz constants of the processes used in the proof of Theorem 4.2.

Recall ¯ L from (13), ˆ L from (Empirical-LR-QR), ˜ L from (12), and H B = {⟨ γ, Φ ⟩ : ∥ γ ∥ 2 ⩽ B < ∞} from Section 4. For any fixed β ∈ I , define ˆ h β as the minimizer of h ↦→ ˆ L ( h, β ) over H B , which exists under the conditions of Theorem 4.2 due to our argument checking the convexity of h ↦→ ˆ L ( h, β ) in Term (I) in the proof of Theorem 4.2.

Lemma H.4. Assume the conditions of Theorem 4.2. Define the stochastic processes ¯ W β and ˜ W β on I given by β ↦→ ( ¯ L -ˆ L )( ˆ h β , β ) and β ↦→ ( ˜ L -ˆ L )( ˆ h β , β ) , respectively. Then ¯ W β is K 1 ,λ -Lipschitz on I with probability tending to unity as n 1 , n 2 , n 3 → ∞ , and ˜ W β is K 2 ,λ -Lipschitz on I with probability tending to unity as n 1 , n 2 , n 3 →∞ , where

with C 1 , max and C 2 , max are defined in (9) and where C 1 , upper satisfies Condition 4 and C 2 , upper := BC Φ + β max C Φ C 1 , upper . In fact, ¯ W is K 1 ,λ -Lipschitz on I with probability tending to unity conditional on S 1 , and ˜ W is K 2 ,λ -Lipschitz on I deterministically, when conditioning on S 2 , S 3 , when the event C 1 ⩽ C 1 , max holds.

Proof. We start with the process ˜ W . Consider β, β ′ ∈ I . Note that for any ( h, β ) , using the definition of ˜ L from (12), we have the identity

Thus we may write

so that

Note that we have the uniform bound

where in the first step we applied Lemma N.2, in the second step we used Condition 1 to apply Lemma N.4, and in the third step we used Lemma H.2. Thus the first term in Equation (10) is bounded by (1 -α ) C Φ E 1 [ C 1 ] | β -β ′ | , and the second term in Equation (10) is bounded by (1 -α ) C Φ ˆ E 1 [ C 1 ] | β -β ′ | . Summing, we deduce that

so that the process ˜ W is K 2 -Lipschitz with K 2 := (1 -α ) C Φ ( E 1 [ C 1 ] + ˆ E 1 [ C 1 ]) .

We now condition on S 2 , S 3 . Observe that C 1 , C 2 are S 3 -measurable (as ˆ Σ from Condition 3 is S 3 -measurable). Since E 1 [ C 1 ] ⩽ C 1 , upper, on the event that C 1 ⩽ C 1 , max, we have K 2 ⩽ K 2 ,λ , where K 2 ,λ = (1 -α ) C Φ ( C 1 , upper + C 1 , max ) , as claimed.

We now continue with the process ¯ W . Consider β, β ′ ∈ I . Note that for any ( h, β ) , using the definition of ¯ L from Equation (13), we have the identity

Thus we may write

so that

The integrands of the first and third terms of Equation (11) can be uniformly bounded as

where in the first step we used difference of squares, in the second step we used Condition 1 to apply Lemma N.4, in the third step we applied Lemma H.2 to bound the first factor and the triangle inequality and the bounds β ⩽ β max for β ∈ I and ∥ h ∥ ⩽ B for h ∈ H B to bound the second factor. The integrand of the second and fourth term in (11) can be bounded as | β ˆ h β -β ′ ˆ h β ′ | ⩽ C Φ ∥ β ˆ h β -β ′ ˆ h β ′ ∥ ⩽ C Φ C 2 | β -β ′ | , where in the first step we used Condition 1 to apply Lemma N.4, and in the second step we applied Lemma H.2.

Plugging these into our bound in Equation (11), we deduce

so that the process ¯ W is K 1 -Lipschitz with

Wenowwork conditional on S 1 . On the event that C 1 ⩽ C 1 , max and C 2 ⩽ C 2 , max, and by Condition 4, we have K 1 ⩽ K 1 , max, where

Since C 1 ⩽ C 1 , max and C 2 ⩽ C 2 , max with probability tending to one due to Condition 3, K 1 ⩽ K 1 ,λ and K 2 ⩽ K 2 ,λ both hold with probability tending to one if we uncondition on S 1 , and we are done.

I Proof of Proposition 4.1

Fix λ ⩾ 0 . Under the assumptions of Lemma L.3, there exists a global minimizer ( h ∗ , β ∗ ) of L ( h, β ) . The first order condition with respect to β reads 2 λ E 1 [ h ∗ ( X )( β ∗ h ∗ ( X ) -r ( X ))] = 0 . By Lemma N.5, the first order condition with respect to h reads

for all h ∈ H . Setting h = r H in the second equation, and subtracting ( β ∗ ) 2 times the first equation from the second, we deduce that

Therefore,

which implies the result.

J Proof of Theorem 4.2

Recall that S 1 are the features of the labeled calibration dataset. We also recall the notation E j and ˆ E j for j = 1 , 2 , 3 from Section 2. Given the unlabeled test data S 2 and the unlabeled calibration data S 3 , define the auxiliary risks for h ∈ H B , β ∈ I ,

and

Let

For convenience, we leave implicit the dependence of ˜ L and ( ˜ h, ˜ β ) on S 2 , S 3 and the dependence of ¯ L on S 1 .

In order to study the generalization error, we write

Since ( ˆ h, ˆ β ) is a minimizer of the risk ˆ L , we have ˆ L ( ˆ h, ˆ β ) -ˆ L ( ˜ h, ˜ β ) ⩽ 0 , and since ( ˜ h, ˜ β ) is a minimizer of the risk ˜ L , we have ˜ L ( ˜ h, ˜ β ) -˜ L ( h ∗ , β ∗ ) ⩽ 0 . Thus our generalization error is bounded by the remaining four terms:

We study the generalization error by conditioning on the unlabeled calibration or test data. Then our regularization becomes data-independent. Conditional on S 1 , Term (I) can be handled with Lemma G.1 above. Conditional on S 2 , S 3 , Term (II) can be handled with Lemma G.1 above. Terms (III) and (IV) are empirical processes at fixed functions, conditional on S 2 , S 3 .

Term (I): We work conditional on S 1 . First, note that due to the definition of ˆ L from (Empirical-LR-QR), we can write for any ( h, β ) ,

Since ¯ L ( h, β ) -ˆ L ( h, β ) can be viewed as a difference of a population risk λ E 3 [ β 2 h 2 ] + λ E 2 [ -2 βh ] and an empirical risk λ ˆ E 3 [ β 2 h 2 ] + λ ˆ E 2 [ -2 βh ] with 'regularizer" ˆ E 1 [ ℓ α ( h, S )] , this expression enables us to apply Lemma G.1 to bound ¯ L ( ˆ h, ˆ β ) -ˆ L ( ˆ h, ˆ β ) .

Explicitly, we can write

Hence, fixing β , we can apply Lemma G.1, choosing m 1 = n 3 and m 2 = n 2 . Further, we choose ˜ H := H B = {⟨ γ, Φ ⟩ : ∥ γ ∥ 2 ⩽ B < ∞} with the norm ⟨ γ, Φ ⟩ = ∥ γ ∥ 2 . Moreover, letting z = ( x ′′ , x ′ ) for x ′′ , x ′ ∈ X , and ξ = 1 /λ , we use the objective function given by ( h, z ) ↦→ f 1 ( h, z ) = J ( h, β, z ) + R ( h, β ) , where J ( h, β, z ) = J 1 ( h, β, z ) + J 2 ( h, β, z ) , and where

We now check the conditions of Lemma G.1.

Boundedness: Note that |J 1 ( h, β, z ) | = | β | 2 | h ( x ′′ ) | 2 ⩽ β 2 max ( BC Φ ) 2 , where in the second step we used | β | ⩽ β max for β ∈ I , and we used h ∈ H B and Condition 1 to apply Lemma N.4. Similarly, note that |J 2 ( h, β, z ) | = 2 | β || h ( x ′ ) | ⩽ 2 β max BC Φ , where in the second step we used | β | ⩽ β max for β ∈ I , and we used h ∈ H B and Condition 1 to apply Lemma N.4. Thus |J 1 ( h, β, z ) | and |J 2 ( h, β, z ) | are both bounded by the sum β 2 max ( BC Φ ) 2 +2 β max BC Φ .

Convexity: Write h = ⟨ γ, Φ ⟩ for γ ∈ R d . The map h ↦→ J 1 ( h, β, z ) can equivalently be written as γ ↦→ β 2 γ ⊤ Φ( x ′′ )Φ( x ′′ ) ⊤ γ , a quadratic whose Hessian equals the positive semidefinite matrix 2 β 2 Φ( x ′′ )Φ( x ′′ ) ⊤ . Thus h ↦→J 1 ( h, β, z ) is convex. The map h ↦→J 2 ( h, β, z ) can equivalently be written as γ ↦→-2 βγ ⊤ Φ( x ′ ) , which is linear, hence convex.

Lipschitzness: Write h = ⟨ γ, Φ ⟩ for γ ∈ R d . The map h ↦→J 1 ( h, β, z ) can equivalently be written as γ ↦→ β 2 γ ⊤ Φ( x ′′ )Φ( x ′′ ) ⊤ γ . The gradient of this quadratic is given by γ ↦→ 2 β 2 Φ( x ′′ )Φ( x ′′ ) ⊤ γ . The norm of this gradient can be bounded by

where in the first step we applied the Cauchy-Schwarz inequality, in the second step we used | β | ⩽ β max for β ∈ I , ∥ γ ∥ 2 ⩽ B , and Condition 1. Next, the map h ↦→J 2 ( h, β, z ) can equivalently

be written as γ ↦→ -2 βγ ⊤ Φ( x ′ ) . The gradient of this linear map is given by γ ↦→ -2 β Φ( x ′ ) . The norm of this gradient can be bounded by 2 | β |∥ Φ( x ′ ) ∥ ⩽ 2 β max C Φ , where we used | β | ⩽ β max for β ∈ I and Condition 1. Thus the norm of each of these gradients is bounded by the sum ρ 1 := 2 β 2 max BC 2 Φ + 2 β max C Φ , and the maps h ↦→ J 1 ( h, β, z ) and h ↦→ J 2 ( h, β, z ) are both ρ 1 -Lipschitz.

Strong convexity: Since h ↦→ ℓ α ( h, s ) is convex for all s ∈ R by Lemma N.3 and since h ↦→ ˆ E 2 [ βh ] is linear, the map h ↦→ ξ ˆ E 1 [ ℓ α ( h, S )] -2 ˆ E 2 [ βh ] is convex. Consider the map h ↦→ ˆ E 3 [ β 2 h 2 ] . Writing h = ⟨ γ, Φ ⟩ for γ ∈ R d , this can be rewritten as γ ↦→ β 2 γ ⊤ ˆ Σ γ , a quadratic whose Hessian equals 2 β 2 ˆ Σ . By β ⩾ β min for β ∈ I and Condition 3, it follows that with probability 1 -o ( n -1 3 ) = 1 -o ( n -1 2 + n -1 3 ) , the map h ↦→ ˆ E 2 , 3 [ f 1 ( h, Z )] is µ 1 -strongly convex, where Z = ( X ′′ , X ′ ) with X ′ is uniform over X 2 and X ′′ is uniform over X 3 , and where µ 1 := 2 β 2 min c min. In particular, h ↦→ 1 λ ˆ L ( h, β ) is convex.

Let ˜ C 1 = 4 ρ 2 1 µ 1 . Let K 1 denote the Lipschitz constant of the process ¯ W β , where K 1 ⩽ K 1 ,λ with probability tending to unity conditional on S 1 by Condition 4 and Lemma H.4. From Lemma G.1 applied with ξ = 1 /λ , L = 1 λ ¯ L , and ˆ L = 1 λ ˆ L , and W = ( ¯ L -ˆ L ) /λ , we obtain that conditional on S 1 , for sufficiently large n 2 , n 3 , with probability at least 1 -δ 4 , we have for Term (I) from (15),

Thus

where we define A 1 = √ 64 ˜ C 1 a 1 δ . Since the right-hand side does not depend on S 1 , the same bound holds when we uncondition on S 1 .

Term (II): We work conditional on S 2 , S 3 . The risks ˆ L and ˜ L share the same data-independent regularization λ ˆ E 3 [ β 2 h 2 ] + λ ˆ E 2 [ -2 βh ] . Write z = ( x, s ) for x ∈ X and s ∈ [0 , 1] . Fixing β , we apply Lemma G.1 with the objective function ( h, z ) ↦→ f ( h, z ) = J ( h, β, z ) + R ( h, β ) , where

Since the empirical risk ˆ L is computed over the i.i.d. sample Z i = ( X i , S i ) for i ∈ [ n 1 ] , we use the modified version of Lemma G.1 given in Remark G.2. In particular, we check boundedness, convexity, and Lipschitzness of J without writing it as a sum J 1 + J 2 .

Boundedness: we have the uniform bound, for all h, β, z

where in the first step we used Lemma N.1, in the second step we used the triangle inequality and s ∈ [0 , 1] , and in the third step we used h ∈ H B and Condition 1 to apply Lemma N.4.

Convexity: By Lemma N.3, h ↦→J ( h, β, z ) is convex.

Lipschitzness: Fix h = ⟨ γ, Φ ⟩ and h ′ = ⟨ γ ′ , Φ ⟩ in H B , where γ, γ ′ ∈ R d . Note that

where in the second step we used Lemma N.2, and in the third step we used Condition 1 to apply Lemma N.4. Thus h ↦→J ( h, β, z ) is ρ 2 -Lipschitz, where ρ 2 := (1 -α ) C Φ .

Strong convexity: To analyze R , first observe that since h ↦→ λ ˆ E 2 [ -2 βh ] is linear, it is convex. Writing h = ⟨ γ, Φ ⟩ for γ ∈ R d , the term h ↦→ λ ˆ E 3 [ β 2 h 2 ] in R can be rewritten as γ ↦→ λβ 2 γ ⊤ ˆ Σ γ , a quadratic whose Hessian equals 2 λβ 2 ˆ Σ . By β ⩾ β min for β ∈ I and Condition 3, it follows

that with probability 1 -o ( n -1 3 ) over S 2 , S 3 , the map h ↦→R ( h, β ) is µ 2 -strongly convex, where µ 2 ( λ ) := 2 λβ 2 min c min.

Let ˜ C 2 ( λ ) = 4 ρ 2 2 µ 2 ( λ ) . Let K 2 denote the Lipschitz constant of the process W β ; recall that conditional on S 2 , S 3 , K 2 ⩽ K 2 ,λ deterministically on the event C 1 ⩽ C 1 , max by Lemma H.4. By the version of Lemma G.1 given in Remark G.2, conditional on S 2 , S 3 , if h ↦→R ( h, β ) is µ 2 ( λ ) -strongly convex, and if C 1 ⩽ C 1 , max, then for sufficiently large n 1 , with probability at least 1 -δ 8 , we have

where we define A 2 = √ 128 ̂ C 2 a 2 δ and ̂ C 2 = 4 ρ 2 2 2 β 2 min c min . Unconditioning on S 2 , S 3 , since R ( h, β ) is µ 2 ( λ ) -strongly convex with probability tending to unity by the above analysis, and since by Condition 3 we have C 1 ⩽ C 1 , max with probability tending to unity, we deduce that for sufficiently large n 1 , n 2 , n 3 , with probability at least 1 -δ 4 , (17) still holds.

Term (III): We work conditional on S 2 , S 3 . Since ˜ h from (14) lies in H B , we may use the bound in Equation (16) to obtain sup x ∈X | ℓ α ( ˜ h, S ) | ⩽ (1 -α )( BC Φ +1) . Thus by Hoeffding's inequality [21], with probability at least 1 -δ 4 we have

Thus we have Term (III) ⩽ A 3 √ n 1 , where we define A 3 = (1 -α )( BC Φ +1) √ 1 2 log 8 δ .

Term (IV): Note that we may write

Since ∥ h ∗ ∥ ⩽ B by h ∗ ∈ H B and since Condition 1 holds, we may apply Lemma N.4 to deduce that sup x ∈X | h ∗ ( x ) | ⩽ BC Φ . Consequently, for β ∈ I , we have the uniform bound sup x ∈X | βh ∗ ( x ) | ⩽ β max BC Φ . By Hoeffding's inequality [21], with probability at least 1 -δ 8 , we have

By another application of Hoeffding's inequality, with probability at least 1 -δ 8 , we have

Summing, with probability at least 1 -δ we have the bound

Using the inequality a + b ⩽ √ 2 √ a 2 + b 2 for all a, b ∈ R , we deduce Term (IV) ⩽ A 4 λ √ 1 n 2 + 1 n 3 , where we define

Returning to the analysis of (15), and summing all four terms while defining A 5 = A 1 + A 4 , with probability at least 1 -δ we obtain a generalization error bound of

The result follows by taking c = A 5 , c ′ = A 3 , and c ′′ = A 2 .

K Proof of Theorem 4.3

We use the following result to convert the generalization error bound in Theorem 4.2 to a coverage lower bound.

Lemma K.1 (Bounded suboptimality implies bounded gradient for smooth functions) . Let f : R d ′ → R , for some positive d ′ . Suppose x ∗ is a global minimizer of f . Suppose x ′ is such that f ( x ′ ) ⩽ f ( x ∗ ) + ε . Suppose h ∈ R d is such that the map g : R → R given by t ↦→ f ( x ′ + th ) is L -smooth, i.e. | g ′′ ( h ) | is uniformly bounded by L . Then

Proof. Assume there exists h and δ > 0 with f ′ ( x ′ ; h ) > δ ∥ h ∥ . Setting y = x ′ -th ,

Set t = δ/ ( L ∥ h ∥ ) to obtain

Since f ( x ′ ) ⩽ f ( x ∗ ) + ε , we have f ( x ′ -th ) ⩽ f ( x ∗ ) + ε -δ 2 2 L . If δ > √ 2 Lε , then f ( x ′ -th ) < f ( x ∗ ) , a contradiction.

A similar argument with f ′ ( x ′ ; h ) < -δ ∥ h ∥ and y = x ′ + th yields the same contradiction. Hence -√ 2 Lε ∥ h ∥ ⩽ f ′ ( x ′ ; h ) ⩽ √ 2 Lε ∥ h ∥ .

By Condition 1 and Condition 5, we may apply Lemma N.5 to deduce that the Hessian of our population risk L from (LR-QR) in the basis { ϕ 1 , . . . , ϕ d } is the block matrix

Thus by β ⩽ β max, ∥ h ∥ ⩽ B for h ∈ H B , Condition 5, and Jensen's inequality, we have the uniform bounds

and

By Lemma L.3 and Lemma L.4, a global minimizer of the objective in Equation (LR-QR) exists, and since β min ⩽ β lower , β max ⩾ β upper, and B ⩾ B upper, any such minimizer lies in the interior of H B ×I . Thus we may apply Lemma K.1 to the objective function L . We utilize two directional derivatives in the space H× R . The first is in the direction 0 H × 1 , the unit vector in the β coordinate. Since ( ˆ h, ˆ β ) ∈ H B ×I , the magnitude of the second derivative of L along this direction is bounded by ν 1 .

The second is in the direction of the vector r B × 0 , where r B the projection of r onto the closed convex set H B in the Hilbert space induced by the inner product ⟨ f, g ⟩ = E 1 [ fg ] . Since ( ˆ h, ˆ β ) ∈ H B ×I , the magnitude of the second derivative of L along this direction is bounded by ν 2 .

Given ˆ h , let ̂ Cover ( X ) := P [ S ⩽ ˆ h ( X ) | X ] -(1 -α ) . Now, on the event E that L ( ˆ h, ˆ β ) -L ( h ∗ , β ∗ ) ⩽ E gen, we apply Lemma K.1 with f being ( γ, β ) ↦→ L ( h γ , β ) , x ∗ being ( h ∗ , β ∗ ) , x ′ being ( ˆ h, ˆ β ) , ε = E gen, and the directions specified above, with their respective smoothness parameters derived above. Using the formulas for ∇ L from Lemma N.5 and the bound ∥ r B ∥ ⩽ B , we obtain that on the event E ,

where E 1 = √ 2 ν 1 E gen , E 2 = √ 2 B 2 ν 2 E gen .

For any h and β , we may write

Evaluating at ( ˆ h, ˆ β ) , the first term is at most E 2 in magnitude, the second term is at most ˆ β 2 E 1 in magnitude, and the third term equals 2 ˆ βλ E 1 [( r B -ˆ β ˆ h ) 2 ] . We deduce

Since ̂ Cover ∈ [ -(1 -α ) , α ] ,

We deduce that

We now bound the quantity ˆ β 2 E 1 + E 2 . First, since √ a + b ⩽ √ a + √ b for all a, b ⩾ 0 , Theorem 4.2 implies that

We may write E 1 = √ 2 ν 1 E gen = √ 4 B 2 λ max (Σ) · λ 1 / 2 √ E gen, so that for ˆ β ∈ I we have

Using the inequality √ a + b ⩽ √ a + √ b for all a, b ⩾ 0 , we may bound

Thus

Plugging in our bound on √ E gen and grouping terms according to the power of λ , we deduce that ˆ β 2 E 1 + E 2 ⩽ E cov, where E cov equals

and where A 10 , . . . , A 14 are the positive constants given in Appendix F. It follows that on the event E ,

By Theorem 4.2, E occurs with probability 1 -δ for sufficiently large n 1 , n 2 , n 3 , and we may conclude.

L Unconstrained existence and boundedness

In this section, we prove apriori existence and boundedness of unconstrained global minimizers of the population objective Equation (LR-QR). We write ( h ∗ λ , β ∗ λ ) for a minimizer of the unconstrained objective in Equation (LR-QR) with regularization strength λ ⩾ 0 .

In Lemma L.1, we show that under Condition 10, we may eliminate β from Equation (LR-QR), so that Equation (LR-QR) is equivalent to solving the following unconstrained optimization problem over h :

Lemma L.1. Under Condition 10, for λ ⩾ 0 , given any minimizer ( h ∗ λ , β ∗ λ ) of the objective in Equation (LR-QR) with regularization λ , h ∗ λ is a minimizer of the objective in Equation (18) with regularization λ . Conversely, if h is a minimizer of the objective in Equation (18) with regularization λ , then there exists a minimizer ( h ∗ λ , β ∗ λ ) of the objective in Equation (LR-QR) with regularization λ such that h ∗ λ = h .

̸

Proof. By Condition 10, the minimization in Equation (LR-QR) with regularization λ can be taken over H\{ 0 } . Further, since the projection of r onto span { h } := { ch : c ∈ R } , for h = 0 is given by E 1 [ rh ] E 1 [ h 2 ] h , we may explicitly minimize the objective in Equation (LR-QR) over β via

where in the second step we applied the Pythagorean theorem. Since the term λ E 1 [ r 2 ] does not depend on the optimization variable h , we may drop it from the objective, which yields the objective in Equation (18). It follows that h is a minimizer of the objective in Equation (18) iff h = h ∗ λ for some minimizer ( h ∗ λ , β ∗ λ ) of the objective of Equation (LR-QR).

Lemma L.2. Let r H denote the projection of r onto H in the Hilbert space induced by the inner product ⟨ f, g ⟩ = E 1 [ fg ] . Then under Condition 5 and Condition 9, there exists θ ∗ > 0 such that E 1 [ S ] -α -1 E 1 [ ℓ α ( θ ∗ r H , S )] > 0 .

Proof. Define g : R → R by g ( θ ) = E 1 [ S ] -α -1 E 1 [ ℓ α ( θ ∗ r H , S )] . Clearly g (0) = 0 . Note that by Condition 5, P S | X [ S = 0] = 0 , so that

By Condition 9, E 1 [ r H ] = E 1 [ r H · 1] = E 1 [ r · 1] = E 1 [ r ] = 1 , so g ′ (0) > 0 . Thus there exists θ ∗ > 0 such that g ( θ ∗ ) > g (0) = 0 , as claimed.

Lemma L.3 (Existence of unconstrained minimizers) . Under Condition 2, Condition 5, Condition 6, Condition 7, Condition 8, Condition 9, and Condition 10, for each λ ⩾ 0 , there exists a global minimizer ( h ∗ λ , β ∗ λ ) of the objective in Equation (LR-QR) .

Proof. Fix λ ⩾ 0 . By Condition 10 and Lemma L.1, it suffices to show that there exists a global minimizer of the objective in Equation (18). Let G ( h ) denote the objective of Equation (18). Define the function ˜ h = θ ∗ r H ∈ H \ { 0 } , where θ ∗ is chosen to satisfy Lemma L.2. With c indep from Condition 6, define ˜ B ( λ ) := 2 c -1 indep (1 + α -1 E 1 [ ℓ α ( ˜ h, S )]) > 0 and

We show that if ∥ h ∥ ⩾ ˜ B ( λ ) or ∥ h ∥ ⩽ ˜ b ( λ ) , then G ( h ) > G ( ˜ h ) . Consequently, the minimization in Equation (18) can be taken over the compact set {⟨ γ, Φ ⟩ : ˜ b ( λ ) ⩽ ∥ γ ∥ 2 ⩽ ˜ B ( λ ) } ⊆ H , so that by continuity of G on H\{ 0 } , a global minimizer h ∗ λ exists.

To see this, first suppose ∥ h ∥ ⩾ ˜ B ( λ ) . Then writing h = ⟨ γ, Φ ⟩ for γ ∈ R d and applying Lemma N.1, the triangle inequality, and S ∈ [0 , 1] ,

By Condition 6 and our assumption that ∥ h ∥ ⩾ ˜ B ( λ ) , this implies that E 1 [ ℓ α ( h, S )] ⩾ α ( ˜ B ( λ ) c indep -1) . Further, by the Cauchy-Schwarz inequality,

Thus by Lemma N.1 and Condition 8, G ( h ) ⩾ α ( ˜ B ( λ ) c indep -1) -λ E 1 [ r 2 H ] . To prove the inequality G ( h ) > G ( ˜ h ) , it suffices to show that

Indeed, since ˜ h is a scalar multiple of r H , we have E 1 [ r 2 H ] = E 1 [ r ˜ h ] 2 E 1 [ ˜ h 2 ] , so the inequality reduces to α ( ˜ B ( λ ) c indep -1) > E 1 [ ℓ α ( ˜ h, S )] . This holds by our choice of ˜ B ( λ ) , which finishes the argument in this case.

Next, suppose ∥ h ∥ ⩽ ˜ b ( λ ) . By Lemma N.1, the triangle inequality, and S ∈ [0 , 1] ,

As above, the Cauchy-Schwarz inequality implies the bound E 1 [ rh ] 2 E 1 [ h 2 ] ⩽ E 1 [ r 2 H ] . We deduce that

Writing h = ⟨ γ, Φ ⟩ for γ ∈ R d , our assumption that ∥ h ∥ ⩽ ˜ b ( λ ) implies that

which when plugged into our lower bound on G ( h ) yields

To prove the inequality G ( h ) > G ( ˜ h ) , it suffices to show that

As above, since ˜ h is a scalar multiple of r H , we have E 1 [ r 2 H ] = E 1 [ r ˜ h ] 2 E 1 [ ˜ h 2 ] , so the inequality reduces to

This holds for our choice of ˜ b ( λ ) , finishing the proof.

Lemma L.4 (Bounds on unconstrained minimizers) . Under the conditions used in Lemma L.3, for all λ > 0 , for any minimizer ( h ∗ λ , β ∗ λ ) of the objective in Equation (LR-QR) , we have that ∥ h ∗ λ ∥ ∈ ( B lower , B upper ) and β ∗ λ ∈ ( β lower , β upper ) , where

and where θ ∗ > 0 is as in Lemma L.2 and r H denotes the projection of r onto H in the Hilbert space induced by the inner product ⟨ f, g ⟩ = E 1 [ fg ] .

Proof. In order to derive our bounds, we consider the reparametrized optimization problem

for ξ ⩾ 0 . We claim that for ξ > 0 , any minimizer of the objective in Equation (22) is of the form h ∗ 1 /ξ . To see this, note that for ξ > 0 , the objective of Equation (18) with regularization λ = 1 /ξ can be obtained by scaling the objective of Equation (22) by the positive factor 1 /ξ . Next, by Condition 10, we may apply Lemma L.1 to deduce that h ∈ H \ { 0 } is a minimizer of the objective in Equation (18) with regularization λ = 1 /ξ iff h = h ∗ 1 /ξ .

In particular, by Lemma L.3, for all ξ > 0 , there exists a global minimizer of Equation (22) with regularization ξ . In the case that ξ = 0 , it is clear that any minimizer h ∗ ∞ of the objective in Equation (22) with regularization ξ = 0 has the form h ∗ ∞ = θr H for some scalar θ > 0 .

Since there exists a minimizer of the objective in Equation (22) for all regularizations ξ in the interval [0 , ∞ ) , we may apply Lemma M.1 to deduce that for all ξ > 0 we have E 1 [ ℓ α ( h ∗ 1 /ξ , S )] ⩽ E 1 [ ℓ α ( h ∗ ∞ , S )] .

We prove lower and upper bounds on ∥ h ∗ 1 /ξ ∥ for all ξ > 0 . We begin with the lower bound.

Lower bound: By (20), we have E 1 [ ℓ α ( h ∗ 1 /ξ , S )] ⩾ α ( E 1 [ S ] -E 1 [ | h ∗ 1 /ξ | ]) . Rearranging, we obtain the lower bound

By Lemma L.2, there exists θ ∗ > 0 such that E 1 [ S ] -α -1 E 1 [ ℓ α ( θ ∗ r H , S )] > 0 . Setting h ∗ ∞ = θ ∗ r H and plugging in the expression for B lower given in (21), our lower bound becomes E 1 [ | h ∗ 1 /ξ | ] > λ max (Σ) 1 / 2 B lower . We now convert this L 1 norm bound to an L 2 norm bound as follows. Write h ∗ 1 /ξ = ⟨ γ ∗ 1 /ξ , Φ ⟩ for γ ∗ 1 /ξ ∈ R d . By the Cauchy-Schwarz inequality, we obtain the upper bound

Combining this with the lower bound E 1 [ | h ∗ 1 /ξ | ] > λ max (Σ) 1 / 2 B lower, we deduce that ∥ h ∗ 1 /ξ ∥ = ∥ γ ∗ 1 /ξ ∥ 2 > B lower, as claimed.

Upper bound: We prove the upper bound in a similar manner. By the first two steps in (19), and using S ∈ [0 , 1] , we have

̸

Rearranging, we obtain the upper bound E 1 [ | h ∗ 1 /ξ | ] ⩽ α -1 E 1 [ ℓ α ( h ∗ ∞ , S )] + 1 . Write h ∗ 1 /ξ = ⟨ γ ∗ 1 /ξ , Φ ⟩ for γ ∗ 1 /ξ ∈ R d . Since we have already established that ∥ h ∗ 1 /ξ ∥ > B lower > 0 , we know that γ ∗ 1 /ξ = 0 . Thus we may write

By Condition 6, this is at least ∥ γ ∗ 1 /ξ ∥ 2 c indep. Combining these upper and lower bounds on E 1 [ | h ∗ 1 /ξ | ] , we obtain ∥ γ ∗ 1 /ξ ∥ 2 c indep ⩽ α -1 E 1 [ ℓ α ( h ∗ ∞ , S )] + 1 . Isolating ∥ γ ∗ 1 /ξ ∥ 2 , we have

as claimed.

Having established 0 < B lower < inf λ> 0 ∥ h ∗ λ ∥ ⩽ sup λ> 0 ∥ h ∗ λ ∥ < B upper < ∞ , we turn to upper and lower bounds on β ∗ λ . As shown in the proof of Lemma L.1, if ( h ∗ λ , β ∗ λ ) is a minimizer of the objective in Equation (LR-QR) with regularization λ , then β ∗ λ = E 1 [ rh ∗ λ ] E 1 [ | h ∗ | 2 ] . By Condition 7,

E 1 [ rh ∗ 0 ] E 1 [ | h ∗ 0 | 2 ] 1 / 2 ⩾ c align > 0 for some minimizer ( h ∗ 0 , β ∗ 0 ) of the objective in Equation (LR-QR) with regularization 0 . By Condition 10 and Lemma L.1, h is a minimizer of the objective in Equation (18) with regularization λ ⩾ 0 iff h = h ∗ λ for some minimizer ( h ∗ λ , β ∗ λ ) of the objective in Equation (LRQR). Thus by Lemma L.3, for all λ ⩾ 0 , there exists a global minimizer of Equation (18), and we may apply Lemma M.1 to Equation (18) to deduce that for any λ ⩾ 0 we have E 1 [ rh ∗ λ ] E 1 [ | h ∗ λ | 2 ] 1 / 2 ⩾ c align > 0 . Consequently, by our bounds on h ∗ λ , Condition 8, and the Cauchy-Schwarz inequality, if we write h ∗ λ = ⟨ γ ∗ λ , Φ ⟩ for γ ∗ λ ∈ R d , then we have

λ

and

completing the proof.

M Monotonicity

Lemma M.1. For some set X and f, g : X → R , let x ( c ) = arg min x ∈X ( f ( x ) + cg ( x )) , where f, g are such that for some interval I ⊂ R , the minimum is attained for all c ∈ I . Then G : I → R , G : c ↦→ g ( x ( c )) is non-increasing in c .

Proof. Let c 1 , c 2 ∈ I , c 1 < c 2 . At c = c 1 , the minimizer x ( c 1 ) satisfies:

At c = c 2 , the minimizer x ( c 2 ) satisfies:

Adding the two inequalities, we find

Subtracting the common terms f ( x ( c 1 )) + f ( x ( c 2 )) leads to

Rearranging, and factoring out c 1 and c 2 , we find

Thus, ( c 1 -c 2 ) [ g ( x ( c 1 )) -g ( x ( c 2 )) ] ≤ 0 . Since c 2 -c 1 > 0 , the inequality implies g ( x ( c 1 )) ≥ g ( x ( c 2 )) , as desired.

N Helper lemmas

Lemma N.1. If α ⩽ 0 . 5 , then α | c -s | ⩽ ℓ α ( c, s ) ⩽ (1 -α ) | c -s | for all c, s ∈ R .

Proof. If s ⩾ c , then ℓ α ( c, s ) = (1 -α )( s -c ) . Since s -c ⩾ 0 and α ⩽ 1 -α , we have α ( s -c ) ⩽ ℓ α ( c, s ) ⩽ (1 -α )( s -c ) , which implies α | c -s | ⩽ ℓ α ( c, s ) ⩽ (1 -α ) | c -s | . If s < c , then ℓ α ( c, s ) = α ( c -s ) . Since c -s > 0 and α ⩽ 1 -α , we have α ( c -s ) ⩽ ℓ α ( c, s ) ⩽ (1 -α )( c -s ) , which implies α | c -s | ⩽ ℓ α ( c, s ) ⩽ (1 -α ) | c -s | .

Lemma N.2. If α ⩽ 0 . 5 , then the map R → R given by c ↦→ ℓ α ( c, s ) is (1 -α ) -Lipschitz.

Proof. If s ⩽ c 1 ⩽ c 2 , we have 0 ⩽ ℓ α ( c 2 , s ) -ℓ α ( c 1 , s ) = α ( c 2 -c 1 ) , which by α ⩽ 0 . 5 is at most (1 -α )( c 2 -c 1 ) . Hence | ℓ α ( c 2 , s ) -ℓ α ( c 1 , s ) | ⩽ (1 -α ) | c 2 -c 1 | . If c 1 ⩽ s ⩽ c 2 and ℓ α ( c 2 , s ) ⩾ ℓ α ( c 1 , s ) , then we have

which by α ⩽ 0 . 5 implies | ℓ α ( c 2 , s ) -ℓ α ( c 1 , s ) | ⩽ (1 -α ) | c 2 -c 1 | . If c 1 ⩽ s ⩽ c 2 and ℓ α ( c 2 , s ) ⩽ ℓ α ( c 1 , s ) , then

hence | ℓ α ( c 2 , s ) -ℓ α ( c 1 , s ) | ⩽ (1 -α ) | c 2 -c 1 | . Finally, if c 1 ⩽ c 2 ⩽ s , we have 0 ⩽ ℓ α ( c 1 , s ) -ℓ α ( c 2 , s ) = (1 -α )( c 2 -c 1 ) , hence | ℓ α ( c 2 , s ) -ℓ α ( c 1 , s ) | ⩽ (1 -α ) | c 2 -c 1 | .

Lemma N.3. The map H → R given by h ↦→ ℓ α ( h ( x ) , s ) is convex for all x ∈ X and s ∈ R .

Proof. Write h ( x ) = ⟨ γ, Φ ⟩ for γ ∈ R d . It suffices to show that the mapping R d → R given by γ ↦→ ℓ α ( γ ⊤ Φ( x ) , s ) is convex. But this map is the composition of the linear function R d → R given by γ ↦→ γ ⊤ Φ( x ) and the convex function R → R given by c ↦→ ℓ α ( c, s ) , hence it is convex.

Lemma N.4. Under Condition 1, if h ∈ H , then sup x ∈X | h ( x ) | ⩽ C Φ ∥ h ∥ , where we use the norm given by ∥ h ∥ = ∥ γ ∥ 2 for h = ⟨ γ, Φ ⟩ . In particular, if h ∈ H B , then sup x ∈X | h ( x ) | ⩽ BC Φ .

Proof. Writing h = ⟨ γ, Φ ⟩ for γ ∈ R d , we have sup x ∈X | h ( x ) | = sup x ∈X |⟨ γ, Φ( x ) ⟩| ⩽ sup x ∈X ∥ γ ∥ 2 ∥ Φ( x ) ∥ 2 ⩽ C Φ ∥ h ∥ , where in the second step we applied the Cauchy-Schwarz inequality.

Lemma N.5. Consider the function φ : R d → R given by φ ( γ ) = E 1 [ ℓ α ( h γ ( X ) , S )] , where h := h γ : X → R is given by h ( x ) = ⟨ γ, Φ( x ) ⟩ for all x ∈ X . Then under Condition 1 and Condition 5, φ is twice-differentiable, with gradient and Hessian given by

Consequently, given ˜ γ ∈ R d , defining g : X → R as g ( x ) = ⟨ ˜ γ, Φ( x ) ⟩ for all x ∈ X , the directional derivative of φ : H → R in the direction g is given by ⟨ ˜ γ, ∇ γ φ ( γ ) ⟩ = E 1 [( P S | X [ h ( X ) > S ] -(1 -α )) g ( X )] .

Proof. For each x ∈ X , define the function η ( · ; x ) : R → R given, for all u , by η ( u ; s ) = E S | X = x [ ℓ α ( u, S )] . For each s ∈ R , define the function χ ( · ; s ) : R → R , where for all u , χ ( u ; s ) = α 1 [ u > s ] -(1 -α ) 1 [ u ⩽ s ] .

By the definition of the pinball loss ℓ α ( · , · ) , and since by Condition 5 the conditional density f S | X = x ( · ) of S | X = x exists for all x ∈ X , the derivative of ℓ α ( u, S ) with respect to u agrees with the random variable χ ( u ; S ) almost surely with respect to the distribution S | X = x . Also, note that for fixed u ∈ R , | χ ( u ; S ) | is bounded by the constant (1 -α ) . By the dominated convergence theorem, it follows that u ↦→ η ( u ; x ) is differentiable, and that its derivative equals ∂ ∂u η ( u ; x ) = E S | X = x [ χ ( u ; S )] , which, by the formula for χ ( u ; S ) , can be written as α P S | X = x [ u > S ] -(1 -α ) P S | X = x [ u ⩽ S ] . Thus for all u ∈ R and x ∈ X , we may write ∂ ∂u η ( u ; x ) = P S | X = x [ u > S ] -(1 -α ) . Since by Condition 5 the conditional density f S | X = x of the distribution S | X = x exists for all x ∈ X , it follows that the cdf u ↦→ P S | X = x [ u > S ] is differentiable for all u ∈ R and all x ∈ X with derivative given by u ↦→ f S | X = x ( u ) . Thus the map u ↦→ ∂ ∂u η ( u ; x ) is differentiable for all x ∈ X with derivative given by u ↦→ f S | X = x ( u ) . In particular, η ( · ; x ) is twice-differentiable with second derivative given by f S | X = x ( · ) .

Next, for each x ∈ X , define the function ψ ( · ; x ) : R d → R given by ψ ( γ ; x ) = E S | X = x [ ℓ α ( h γ ( x ) , S )] , where h = h γ = ⟨ γ, Φ ⟩ . For each x ∈ X , let ev ( · ; x ) : R d ↦→ R be given by ev ( γ ; x ) = h γ ( x ) , where h = h γ = ⟨ γ, Φ ⟩ . Then ψ ( · ; x ) is given by the composition η ( · ; x ) ◦ ev ( · ; x ) . Since ev ( γ ; x ) = ⟨ γ, Φ( x ) ⟩ , ev ( · ; x ) is linear, it is smooth. Its gradient is given by ∇ γ ev ( γ ; x ) = Φ( x ) for all γ ∈ R d , and its Hessian is zero. It follows that ψ ( · ; x ) is twicedifferentiable. By the chain rule, the gradient of ψ ( · ; x ) is given by

Since the map γ ↦→ P S | X = x [ h ( x ) > S ] -(1 -α ) is given by the composition ∂ ∂u η ( · ; x ) ◦ ev ( · ; x ) , we may again apply the chain rule to deduce that the Hessian of ψ ( · ; x ) is given by

Returning to our original function φ , note that by the tower property, φ ( γ ) = E 1 [ ψ ( γ ; X )] . Note that ∥∇ γ ψ ( γ ; x ) ∥ 2 is at most

where in the first step we used the triangle inequality, and in the second step we used the fact that P S | X = x [ h ( x ) > S ] ⩽ 1 and Condition 1. Similarly, we may bound the Frobenius norm ∥ · ∥ F of ∇ 2 γ ψ ( γ ; x ) by

where in the first step we used Condition 1, the identity ∥ vv ⊤ ∥ F = ∥ v ∥ 2 2 , and in the second step we used Condition 5. Since the entries of ∇ γ ψ ( · ; x ) and ∇ 2 γ ψ ( · ; x ) are bounded by constants, we may apply the dominated convergence theorem to deduce that φ is twice-differentiable, with gradient given by ∇ γ φ ( γ ) = E 1 [ ∇ γ ψ ( γ ; X )] and Hessian given by ∇ 2 γ φ ( γ ) = E 1 [ ∇ 2 γ ψ ( γ ; X )] .

Finally, since the directional derivative of φ in the direction g is defined as ⟨ ˜ γ, ∇ γ φ ( γ ) ⟩ , we may plug in our expression for the gradient to deduce

The result follows.

O Unbounded scores

In this section, we comment on our assumption that S ∈ [0 , 1] a.s. Given an arbitrary a.s. finite score S , and given the sigmoid function g : R → R given by g ( x ) = 1 1+ e -x , the composition g ( S ) is a.s. [0 , 1] -valued. As the proof of Theorem 4.2 only requires boundedness, this allows one to obtain the generalization bound for arbitrary score functions.

However, the proof of Theorem 4.3 places a boundedness assumption on the conditional density of S | X = x in Condition 5, which precludes us from directly applying the transformation trick given above. We claim that Condition 5 can be replaced with the following alternate condition:

Condition 11. (1) The conditional density f S | X = x exists for all x ∈ X ; (2) there exists a constant C f > 0 and a real k > 0 such that for all s ∈ (0 , 1) , we have

uniformly in x ∈ X ; (3) the basis Φ obeys

and (4) the quantity B upper defined in Lemma L.4 obeys B upper < C -1 Φ , where C Φ is defined in Condition 1.

In other words, we can allow the conditional density of S | X = x to diverge at a polynomial rate near s = 0 and s = 1 , so long as Φ obeys a certain moment condition, and so long as apriori, the population LR-QR objective can be restricted to a sufficiently small ball.

One can consider point (3) of Condition 11 as a slight strengthening of Condition 6, a quantitative independence condition on the basis functions. Regarding point (4) of Condition 11, note that by inspecting the definition of B upper, we see that an upper bound on B upper imposes (a) a lower bound on c align in Condition 7, as well as (b) an upper bound on E 1 [ ℓ α ( θ ⋆ r H , S )] , which states that optimally scaling the projection r H can yield a threshold function with low pinball loss.

Now, we sketch how utilizing Condition 11 implies Theorem 4.3 for unbounded scores. In the original proof, Condition 5 is used in order to control expressions of the form | E 1 [Φ( X )Φ( X ) T f S | X ( h ( X ))] | , uniformly for γ ∈ R d with 0 < B lower ⩽ ∥ γ ∥ 2 ⩽ B upper, where h ( X ) = ⟨ γ, Φ( X ) ⟩ . By Condition 1 and Jensen's inequality, this is bounded by E 1 [ | f S | X ( h ( X )) | ] , up to constants. By point (2) of Condition 11, this in turn is bounded by E 1 [ | h ( X ) | -k + | 1 -h ( X ) | -k ] , up to constants. By point (3) of Condition 11 and the bounds 0 < B lower ⩽ ∥ γ ∥ 2 ⩽ B upper, the first term E 1 [ | h ( X ) | -k ] is uniformly bounded. Next, by the triangle inequality, the Cauchy-Schwarz inequality, Condition 1, and point (4) of Condition 11, we have

so we may uniformly bound the second term by

Putting these bounds together, we see that Condition 11 provides the desired uniform control.

Finally, if we utilize Condition 11 instead of Condition 5, then the sigmoid transformation allows us to generalize Theorem 4.2 beyond bounded scores. Note that for g , the conditional density f g ( S ) | X = x of the transformed score g ( S ) obeys

for all t ∈ (0 , 1) . Consequently, if the original density f S | X = x ( s ) is supported on R with polynomially-decaying tails as s → ±∞ , then the transformed density diverges like ∼ ( t (1 -t )) -(1+ o (1)) as t → 0 , 1 , which satisfies Condition 11 with k = 1 + ε for any ε > 0 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction are supported by the theoretical results in Section 4 and the experiments in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in the Discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems are always stated with their required assumptions, and full proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental details are provided in Section 5, including dataset information. An open-source GitHub repository will be released upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The software package implementing our method and reproducing the experiments will be released as an open-source GitHub repository upon publication. The datasets used are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are described in Section 5. We evaluate pretrained models without training.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our plots include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We perform small-scale experiments and provide sufficient detail on the set-up.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper follows the NeurIPS Code of Ethics in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential social impact of our work in the Discussion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve the release of any pretrained models, image generators, or datasets that pose a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and sources used in this paper are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core methodology in this research does not involve LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of covariate shift in conformal prediction, where existing methods often rely on estimating a likelihood ratio function that may be hard to compute in high dimensions. The authors propose a new method, LRQR, which avoids estimating the likelihood ratio directly but still achieves valid marginal coverage. This is done under a mild assumption formalized through a hypothesis class $\\mathcal{H}$. The method is supported by strong theoretical results and is benchmarked against several state-of-the-art approaches on real datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. I find the paper clearly written and easy to follow.\n2. The paper addresses a case where existing techniques may not apply due to data-dependent regularizers, and introduces a novel conditioning argument to overcome this.\n3. The theoretical development is comprehensive and appears well-grounded. Although I did not verify all proofs in detail, the results are built upon solid theoretical foundations.\n4. The proposed method is compared against strong baselines and evaluated on realistic benchmark datasets.\n\nWeaknesses\n1. I would appreciate a more formal and precise definition of the hypothesis class $\\mathcal{H}$, as mentioned around lines 105–110. It is unclear what exactly constitutes a “linear hypothesis class” in the authors’ framework.\n2. It is not clear how to verify whether a given dataset satisfies the assumptions required by $\\mathcal{H}$. I would like to see a simulation that intentionally violates this assumption and examines how the method behaves in such cases.\n\nQUESTIONS:\n1. It seems that the method does not aim to achieve conditional conformal coverage. If I am wrong, I would appreciate clarification. Otherwise, could the authors explain why they perform conditional evaluations across subgroups (e.g., by race or category) in data analysis, if the method is only designed for marginal validity?\n\n2. On line 104, the authors state that their theory requires scores in the $[0,1]$ range. But many common scores (e.g., residuals or negative log-likelihoods) are not bounded this way, as seen in the later experiments. Could the authors clarify the necessity of this restriction and how it aligns with practice?\n\n3. I would appreciate a high-level explanation of why the linear hypothesis class is needed and how it is used in the proof.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of covariate shift in Conformal Prediction. The standard approach to establishing CP validity in that scenario requires estimating a likelihood ratio. The paper describes a computationally advantageous approach based on regularising the pinball objective used in quantile regression.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe observation linking the optimum of the pin-ball objective and coverage under distribution shift is interesting. Is this the first time this has been observed and used to build an estimation approach?\n\n**Weaknesses**\n\nIt is unclear why the likelihood ratio *may be inaccurate for high-dimensional data*. As this is the motivation for their work, the authors can spend a few more words explaining the reasons for the claim. One of the simplest likelihood ratio estimators is based on fitting a logistic regression model, whose performance intuitively increases with the input dimensionality. The authors could have also run an ablation experiment to show that the likelihood ratio estimation becomes inaccurate in high dimensions.\n\nThe approach seems to require estimating a model for the likelihood ratio, exploiting the change-of-measure identity. The authors should clarify whether the idea has appeared elsewhere, why it can not be used to direct likelihood ratio estimation, and how it removes the high-dimensional inaccuracy.\n\nQUESTIONS:\n- Does the \"linear hypothesis class\" contain non-linear regression models?\n- Setting $g=dP/dQ$ implicitly implies that the likelihood ratio is in the linear hypothesis class. How restrictive is this assumption?\n- The validity in Equation 2 is different from the finite-sample validity of Conformal Prediction. In [1], a threshold correction is estimated on a calibration set to conformalize the obtained intervals. How does that approach relate to the finite-sample setup in Section 4.2?\n- Does the correction in Theorem 4.3 account for the non-optimality of the likelihood ratio estimator, h? Can a similar bound be obtained for non-linear $h$?\n\n[1]\nRomano, Yaniv, Evan Patterson, and Emmanuel Candes. *Conformalized quantile regression.* Advances in neural information processing systems 32 (2019).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of conformal prediction under covariate shift, where the distribution of features differs between a source (calibration) and a target (test) domain. The authors propose a new method called Likelihood Ratio Regularized Quantile Regression (LR-QR) to construct prediction sets that achieve valid coverage in the target domain without explicitly estimating the density ratio $r(x)=dP_{2,X}/dP_{1,X}$ (which can be prohibitive at high dimensions). The key idea is to incorporate a regularization term into a quantile regression target that implicitly encourages the learned threshold function $h(x)$ to adapt to the unknown likelihood ratio.\n\nThe paper addresses the problem of constructing conformal prediction sets under covariate shift—when calibration data comes from a source distribution $P_1$ and test data from a different target distribution $P_2$ sharing the same $Y\\mid X$ but with shifted marginals (pp. 1–2). Existing methods often estimate the density-ratio\n\n$$\nr(x)\\;=\\;\\frac{dP_{2,X}}{dP_{1,X}}(x),\n$$\n\nwhich is challenging in high dimensions (p. 2).\n\n---\n\n## Method (LR-QR)\n\nThe authors propose **Likelihood-Ratio Regularized Quantile Regression** (LR-QR), which replaces direct ratio estimation with a novel regularizer in a quantile-regression objective (pp. 3–4). Concretely, they solve\n\n$$\n\\min_{h,\\;\\beta}\\;\\mathbb{E}_{P_1}\\bigl[\\ell_\\alpha\\bigl(h(X),S(X,Y)\\bigr)\\bigr]\n\\;+\\;\\lambda\\;\\mathbb{E}_{P_1}\\bigl[(\\beta\\,h(X)-r(X))^2\\bigr],\n$$\n\ninvolving only \\(\\mathbb{E}_{P_1}[h^2]\\) and \\(\\mathbb{E}_{P_2}[h]\\), both easily estimated from unlabeled samples (pp. 4–5; Algorithm 1).\n\n---\n\n## Theory\n\n* **Population regime:**\n  If the hypothesis class contains the true ratio $r$, LR-QR attains exact $(1-\\alpha)$-coverage under $P_2$ (Prop. 4.1, p. 5).\n\n* **Finite-sample regime:**\n  They prove a generalization bound (Theorem 4.2, p. 6) showing the empirical minimizer’s risk is within\n\n  $$\n    O\\Bigl(n_1^{-\\tfrac13}\\,(n_2+n_3)^{-\\tfrac13}\\Bigr)\n  $$\n\n  of optimal, and derive a coverage guarantee (Theorem 4.3, p. 7) that the test-domain coverage is at least $(1-\\alpha)$ minus controllable error terms.  The optimal regularization scales as\n\n  $$\n    \\lambda \\;\\propto\\; n_1^{-\\tfrac13}\\,(n_2+n_3)^{-\\tfrac13}\n    \\quad\\text{(Eq. 4, p. 6).}\n  $$\n\n---\n\n## Empirical Results\n\nOn three high-dimensional benchmarks, LR-QR consistently tracks nominal coverage and yields competitive (often smaller) set sizes:\n\n1. **Communities & Crime** regression under four race-based splits (Fig. 1).\n2. **RxRx1** cell-image classification across 14 biological batches (Fig. 2).\n3. **MMLU** multiple-choice with an LLM (Table 1).\n\nAn ablation over $\\lambda$ confirms that moderate regularization (theory-predicted regime) best balances coverage and efficiency (Fig. 4, p. 14).\n\n---\n\n**Overall**, LR-QR offers a scalable, theoretically grounded approach to conformal prediction under high-dimensional covariate shift **without** explicit ratio estimation, backed by strong empirical performance.\n\nSTRENGTHS AND WEAKNESSES:\nThe technical quality of the paper is high. The methodology is derived from sound principles of quantile regression and importance weighting, and the authors provide a rigorous theoretical analysis to support their claims. In particular, Proposition 4.1 and its corollaries prove that the LR-QR solution achieves valid target coverage in the population setting if the hypothesis class contains the true density ratio (and even if it does not, one can quantify the coverage gap). This is a non-trivial theoretical guarantee, and the proof (which has been moved to the appendix) seems to use tools from learning theory (stability bounds, etc.) to deal with finite sample deviations. The paper also relies on a novel analysis of coverage by stability – this suggests that the theoretical results are quite advanced and not just a routine application of existing theorems. The derivations appear to be mathematically sound, given the stated regularity conditions (e.g. linear hypothesis space, integrability and convex assumptions in Appendix E). I found no errors in the reasoning, and important steps  are adequately justified (footnote 1). The authors even derive a guideline for the choice of the regularization parameter $\\lambda$ in practice based on the trade-off of terms of order $1/\\sqrt{n_1}$ and $1/\\sqrt{n_2}$ (number of labeled vs. unlabeled points), which is a valuable insight to avoid ad hoc tuning.\n\nExperimental validation is another strong point: the paper evaluates LR-QR in three different domains (tables, vision, NLP) and compares it to six baselines, showing superior or comparable performance. It is shown that LR-QR achieves coverage very close to the nominal 90% in the target domain, whereas some baselines fall short (e.g., the default split conformal method under Shift drops to ~78% coverage) or overshoot (some robust methods exceed 95% coverage because they are too conservative). At the same time, LR-QR maintains smaller prediction sets than the overly conservative methods, indicating a favorable trade-off between validity and efficiency. The authors also perform  cross-validation to select $\\lambda$ (theory-guided) without using test labels, and an ablation study of regularization (Appendix B) to provide deeper insight.\n\nQUESTIONS:\n- A minor problem in terms of technical quality is the fact that the theory is based on certain assumptions. For example, the main coverage guarantee assumes a linear hypothesis class (and even then conditions such as strong convexity or an independence condition for features according to Appendix E/F are required). In practice, authors restrict $H$ to linear functions (often a linear layer over a pre-trained representation), which is consistent with theory, but this can limit the flexibility of $H$ to approximate $r(x)$ when $r(x)$ is very complex. It is understandable that generalizing the theory to nonlinear or very large function classes is challenging (due to overfitting concerns), and the work mitigates this problem by using representation learning (pre-trained models) to make linear $H$ more expressive. Nonetheless, the overfitting guarantee is asymptotic in finite samples and up to an error term, and it depends on conditions that are not completely under the control of the experimenter (e.g., how well $r(x)$ lies within the range of the chosen features). This is not a critical flaw, – just a recognition that, as with any theory-heavy work, practical performance depends on how well reality matches assumptions. - The experimental results suggest that the method is robust, but the paper could be strengthened by a brief discussion of how sensitive the coverage is to misspecification of $H$ or violation of the assumptions (the appendix addresses this with the distance term $||r - r_H||$, but it might be worth highlighting this in the main text). Another minor quality issue is that the method introduces an additional hyperparameter $\\lambda$; while theory dictates the order, in practice the exact choice is tuned – however, the paper handles this well via cross-validation, so I don’t see it as a major drawback.\n- One area where clarity could be improved is the explanation of the “Algorithmic Principles” (section 3) – in particular lines 117–129 of the draft. The derivation presented there is somewhat difficult to follow and seems a little sloppy in its current form. In this section, the authors use the first-order optimality conditions of quantile loss minimization to argue that if the hypothesis class $H$ contains a function proportional to the ratio of the true density, then the learned predictor achieves exact coverage under this covariate shift. While the idea is sound, the explanation is rather poor. Essentially, it imposes the condition that at the optimum $h^(x)$, the expected gradient in each direction $g(x)$ in $H$ is zero. By inserting a special choice $g(x) = \\frac{dQ_X}{dP_{1,X}}(x)$ (the Radon-Nikodym derivative of a shifted distribution $Q$ after that of the source distribution), they deduce that $\\mathbb{E}_{P_1}[g(X)(\\mathbf{1}{S \\le h^(X)} - (1-\\alpha))] = 0$ implies that $\\mathbb{E}_{Q}[,\\mathbf{1}{S(X,Y) \\le h^(X)},] = 1-\\alpha$. In other words, the prediction set defined by $h^$ has an exact $(1-\\alpha)$ coverage under any covariate shift $Q$ whose density ratio is in $H$. The paper’s presentation quickly skips through these steps. I find the argument in Cherian and Gibbs clearer to follow.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a conformal prediction method for the covariate shift setting (i.e. when the distribution of the $X$ in the test set is different from the distribution of the $X$ in the calibration set). Whereas existing methods require the estimation of a particular likelihood ratio function, the authors here present an algorithm, called likelihood ratio regularized quantile regression (LR-QR), which outputs a threshold function without directly estimating the likelihood ratio. To do this, they assume that unlabeled data from both distributions are available and minimize a particular objective function (Eq. in line 163). They show theoretically that when there is no estimation error (i.e. expectations are known) and under several assumptions, the function produced by the optimizer achieves valid coverage in the test domain. In the finite sample setting, the validity becomes asymptotic if the class of functions on which we optimize is sufficiently large. Finally, they show the effectiveness of their method on several data sets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1) The paper is well-written and the problematic is well motivated.\n\n2) The main idea is interesting and well-presented.\n\n3) This line of research is of particular interest for the CP community.\n\nWeaknesses:\n\n1) The supplementary material does not include the code necessary to run the algorithm. Providing this code would improve reproducibility.\n\n2)  The experimental section lacks clarity compared to the theoretical section (reason why I put a borderline accept but I am willing to increase it because overall this is a good paper).\n    \n     2.1) Section 5.2 \"Communities and Crime\": This section is unclear (at least for me), particularly regarding how the covariate shift is simulated.\n\n    2.2) The results on the size for LR-QR appear to be missing.\n\n    2.3) It is unclear where the results for the coverage of LR-QR are presented in Figure 1.\n\n    2.4) There is no error bar in Table 1.\n\n3) The conclusion only contains one limitation (and this limitation is not clear to me) --> Line 356 : \"to have results that control of the slack in coverage\" ??\n\nQUESTIONS:\n1\\ Is Proposition 4.1 true for any $\\lambda$? If yes, I do not understand where is the impact of this hyper-parameter.\n\n2\\ Furthermore, if there is no impact, is the intuition being that in the infinite sample size setting, if the true $r$ is in $H$ then we do not need to regularized the objective as the optimizer is going to return $r$ anyway?\n\n3\\ Line 121: Why $h$ need to be in a linear hypothesis class H ?\n\n4\\ Do you think it could be possible to use the $X$ from $P_{1, X}$ of the calibration instead of assuming that we have another unlabeled set?\n\n5\\ In Eq (4), you provide an optimal value for $\\lambda$. Is this value consistent with the one obtained across all the experiments? (not only Figure 4).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript considers conformal prediction under high-dimensional covariate shift. The\nauthors introduce the Likelihood Ratio-Regularized Quantile Regression (LR-QR) algorithm. By\ncombining the pinball loss with a suitable choice of regularization, the authors construct a threshold\nfunction by getting around direct estimation of the unknown likelihood ratio, which can be difficult\nin high-dimensional setting. The authors provide theoretical properties and experiments to back up\ntheir proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Conformal prediction under covariate shift is an interesting problem that is of practical\nimportance. This manuscript proposes a promising method to handle such a problem.\n2, The proposed approach offers a natural and effective way to incorporate regularization techniques\nin situations where the likelihood ratio function is unknown. The underlying idea is both novel\nand well-justified.\n3. Theoretical results are provided to support the proposed methodology, and they are nontrivial\nto establish.\n4. The manuscript is generally well written, and the presentation follows a logical progression.\n\nWeaknesses:\n1. While the title and main text emphasize the consideration of ’high-dimensional covariate\n(shift),’ the manuscript lacks a clear discussion of this feature. How high is the dimension of\nthe covariate X? Is it treated fixed or diverging with the data size? Relative to the sizes $n_1$, $n_2$,\nand $n_3$ for labeled or unlabeled source data points and target data points, how large is the\ndimension of the covariate? How do different dimensionality settings influence the theoretical\ndevelopments, including the established bounds, regularity conditions, and proofs?\n2. The presentation can be improved with careful revision. Several notations are used inconsistently\nor unclearly, and further clarification is needed in several places (see specific comments below)\n\nQUESTIONS:\n\\textbf{Questions:}\n\n\\begin{itemize}\n\\item Key symbols $\\mathbb{P}_1$ and $\\mathbb{P}_2$ are not consistently defined or used, Their meanings and usage vary between Sections 1 and 2.\n  \n    1. Lines 25–26: The statement ``In the case that the calibration and test distributions coincide ($\\mathbb{P}_1 = \\mathbb{P}_2$), ...\"\n    does not align with the definition of $\\mathbb{P}_1$ and $\\mathbb{P}_2$:  $\\mathbb{P}_1$ is defined over both inputs and outputs, \nwhile $\\mathbb{P}_2$ is defined only over inputs (Lines 21–22).\n\n    2. Lines 36–37: In the definition of the likelihood ratio function $r(x) = d\\mathbb{P}_{2,X}/d\\mathbb{P}_{1,X}$, the \ndistributions  $P_{2,X}$ and $P_{1,X}$ are not clearly defined in Section 1,\n though they appear more explicitly (yet still implicitly) in Section 2.\n  \n\\item Line 100: The prediction set $C: \\mathcal{X} \\rightarrow 2^{\\mathcal{Y}}$ suggests that the output space $\\mathcal{Y}$ is discrete (e.g., for classification). If the method is intended for regression,\n the terminology \"prediction set” could be adjusted to `\"prediction interval/region,” \nor the authors should clarify that the method is focused on classification.\n\n\\item Lines 105–106: It states, ``In this paper, a linear hypothesis class refers to a linear subspace of functions \nfrom $\\mathcal{X} \\rightarrow \\mathbb{R}$ that are square-integrable with respect to $\\mathbb{P}_{1,X}$.” \nWhy is this class particularly considered? Provide motivation or justification.\n\n\\item Lines 111–113: Calibration data are assumed to be i.i.d., which may be unnecessarily restrictive. One strength of conformal prediction is its reliance on weaker assumptions such as exchangeability. Can the method be extended to exchangeable data? If not, explain why the i.i.d. assumption is essential.\n\n\\item Notation inconsistency: The function $C(x)$ is defined differently in Lines 100 and 118 using different threshold symbols $q(x)$ and $\\tilde{q}(x)$. This variation seems unnecessary. Define $C(x)$ once and refer to it consistently.\n\n\\item Line 170: The statement that valid coverage is guaranteed if ${\\cal H}$ is linear and contains the true likelihood ratio $r$ implicitly requires $r$ to be linear. How plausible is this in real-world settings? Provide discussion or examples to help evaluate the adequacy of restricting ${\\cal H}$ to linear functions.\n    How likely would\n this happen for distributions often used in applications? \n    Provide discussion or examples to help evaluate the adequacy of restricting ${\\cal H}$ to linear functions. While it is understood that focusing on\n the linear hypothesis class ${\\cal H}$ can be critical in establishing meaningful theoretical results, a natural concern is its feasibility or adequacy in handling various practical distributions.\n\n\\item Lines 181–183: The lower bound involving $-E_1[(r(X) - r_{\\cal H}(X))^2]^{1/2}$ is derived via the Cauchy–Schwarz inequality. Given the indicator function’s $\\{0,1\\}$ values, could a tighter bound, such as $\\max(0, E_1[r(X) - r_{\\cal H}(X)])$, be used to describe the lower bound in Line 185?\n\n\\item Lines 212–213: Why is a fixed collection $\\Phi = (\\phi_1, \\ldots, \\phi_d)^\\top$ of $d$ basis functions assumed? What is $d$? Is it related to $\\dim(X)$?\n    (The dimension of $X$ does not appear to be explicitly represented using a dedicated notation.)\n    How are these basis functions chosen?\n\n\\item Lines 212–214: The introduction of interval ${\\cal I}$ appears to restrict the domain of the minimization in (LR-QR) from $\\beta \\in \\mathbb{R}$ to a compact subinterval of positive values. Justify this restriction and explain its impact on the solution.\n\n\\item Line 214: The $B$-ball ${\\cal H}_B$ defined in Line 214 depends on the choice of $B$, which in turn may influence the solution $(\\hat h, \\hat \\beta)$ in (3). However, the  results in Theorems 4.2 and 4.3 do not explicitly depend on $B$. Is this discrepancy  meaningful? The lack of $B$ in the bounds  makes to statements in Lines 239-240 less meaningful, though one may intuitively understand the impact of varying the value of $B$.\n\n \\item Line 233: What is $r_B(X)$ in Theorem 4.3? It is used without prior definition.\n\n   \\end{itemize}\n\n\n\n\n\\textbf{Suggestions:}\n\n\\begin{itemize}\n\n\\item The example in Lines 106–110 does not seem to be helpful. It can either be removed or further explained in terms of its relevance to the subsequent development.\nThe description of the “linear hypothesis class” in Lines 105–106 could be moved to the location around Line 122, where the linear hypothesis class ${\\cal H}$ first appears.\n\n\\item Clearly describe the key notations  $E_1$ \nand $E_2$ in relation to\nthe distributions $P_{1,X}$ \nand $P_{2,X}$  \n(or even $P_1$ \nand $P_2$). This clarification will help align the use \nof $E_{\\mathbb{Q}}$ in Line 126 with the other notations.\nBy the way, $P$ and $Q$ in footnote 1 on page 3 should be written as $\\mathbb{P}$ and\n$\\mathbb{Q}$, respectively.\n\n\\item The equation labels “(LR-QR)” in Line 157 and “(Empirical-LR-QR)” in Line 163\nshould be changed to “(1)” and “(2)” to be consistent with numerical labels such as\n(3) in Line 217 and (4) in Line 226 (though authors might intend to use lettered names\nto emphasize the differences in those minimization problems).\n\n\\item In Lines 171–172, it would be helpful to also present the mathematical expression\nof the projection of $r$ onto ${\\cal H}$ in the Hilbert space.\n\n\\item In Line 119, use a symbol different from $S$ to represent a random variable\nto avoid confusion with the use of $S(\\cdot, \\cdot)$ as the nonconformity score defined in Line 98.\n\n\\end{itemize}\n\n(NOTE: For unknown reasons, some LaTeX symbols cannot be displayed properly  in the OpenReview display system, even though they appear properly in my locally compiled PDF).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}