Title: Localized Adaptive Risk Control
Venue: NeurIPS 2024
Accepted: yes
Keywords: online conformal risk control

uncertainty quantification

conformal predictions
Abstract: Adaptive Risk Control (ARC) is an online calibration strategy based on set prediction that offers worst-case deterministic long-term risk control, as well as statistical marginal coverage guarantees. ARC adjusts the size of the prediction set by varying a single scalar threshold based on feedback from past decisions. In this work, we introduce Localized Adaptive Risk Control (L-ARC), an online calibration scheme that targets statistical localized risk guarantees ranging from conditional risk to marginal risk, while preserving the worst-case performance of ARC. L-ARC updates a threshold function within a reproducing kernel Hilbert space (RKHS), with the kernel determining the level of localization of the statistical risk guarantee. The theoretical results highlight a trade-off between localization of the statistical risk and convergence speed to the long-term risk target. Thanks to localization, L-ARC is demonstrated via experiments to produce prediction sets with risk guarantees across different data subpopulations, significantly improving the fairness of the calibrated model for tasks such as image segmentation and beam selection in wireless networks.

Localized Adaptive Risk Control

Abstract

Adaptive Risk Control (ARC) is an online calibration strategy based on set prediction that offers worst-case deterministic long-term risk control, as well as statistical marginal coverage guarantees. ARC adjusts the size of the prediction set by varying a single scalar threshold based on feedback from past decisions. In this work, we introduce Localized Adaptive Risk Control (L-ARC), an online calibration scheme that targets statistical localized risk guarantees ranging from conditional risk to marginal risk, while preserving the worst-case performance of ARC. L-ARC updates a threshold function within a reproducing kernel Hilbert space (RKHS), with the kernel determining the level of localization of the statistical risk guarantee. The theoretical results highlight a trade-off between localization of the statistical risk and convergence speed to the long-term risk target. Thanks to localization, L-ARC is demonstrated via experiments to produce prediction sets with risk guarantees across different data subpopulations, significantly improving the fairness of the calibrated model for tasks such as image segmentation and beam selection in wireless networks.

1 Introduction

Adaptive risk control (ARC), also known as online risk control, is a powerful tool for reliable decision-making in online settings where feedback is obtained after each decision [Gibbs and Candes, 2021, Feldman et al., 2022]. ARC finds applications in domains, such as finance, robotics, and health, in which it is important to ensure reliability in forecasting, optimization, or control of complex systems [Wisniewski et al., 2020, Lekeufack et al., 2023, Zhang et al., 2023, Zecchin et al., 2024]. While providing worst-case deterministic guarantees of reliability, ARC may distribute such guarantees unevenly in the input space, favoring a subpopulation of inputs at the detriment of another subpopulation.

As an example, consider the tumor segmentation task illustrated in Figure 1. In this setting, the objective is to calibrate a pre-trained segmentation model to generate masks that accurately identify tumor areas according to a user-defined reliability level [Yu et al., 2016]. The calibration process typically involves combining data from various datasets, such as those collected from different hospitals. For an online setting, as visualized in the figure, ARC achieves the desired long-term reliability in terms of false negative ratio. However, it does so by prioritizing certain datasets, resulting in unsatisfactory performance on other data sources. Such behavior is particularly dangerous, as it may result in some subpopulations being poorly diagnosed. This paper addresses this shortcoming of ARC by proposing a novel localized variant of ARC.

Long-termFNR

SegmentationModel

Pre-trained

Calibration

Online

Long-term FNR

0.10

ARC

0.05

ARC

L-ARC

0.00

0

1000

2000

Time-step (t)

LaribDataset

Dataset-conditioned FNR

0.6

Marginal

0.4

Kvasir

L-ARC

Larib

0.2

FeatureInformation

0.0

ARC

L-ARC

1.1 Adaptive Risk Control

To elaborate, consider an online decision-making scenario in which inputs are provided sequentially to a pre-trained model. At each time step t ≥ 1 , the model observes a feature vector X t , and based on a bounded non-conformity scoring function s : X ×Y → [0 , S max ] and a threshold λ t ∈ R , it outputs a prediction set

where Y is the domain of the target variable Y . After each time step t , the model receives feedback in the form of a loss function

that is assumed to be non-negative, upper bounded by B < ∞ and non-increasing in the predicted set size | C t | . A notable example is the miscoverage loss

Accordingly, for an input-output sequence { ( X t , Y t ) } T t =1 the performance of the set predictions { C t } T t =1 in (1) can be gauged via the cumulative risk

For a user-specified loss level α and a learning rate sequence { η t } T t =1 , ARC updates the threshold λ t in (1) as [Feldman et al., 2022]

where L t -α measures the discrepancy between the current loss (2) and the target α . For step size decreasing as η t = η 1 t -1 / 2 for a ∈ (0 , 1) and an arbitrary η 1 > 0 , the results in [Angelopoulos et al., 2024b] imply that the update rule (5) guarantees that the cumulative risk (4) for the miscoverage loss (3) converges to target level α for any data sequence { ( X t , Y t ) } t ≥ 1 as thus offering a worst-case deterministic long-term guarantee. Furthermore when data are generated i.i.d. as ( X t , Y t ) ∼ P XY for all t ≥ 1 , in the special case of the miscoverage loss (3), the set predictor produced by (5) enjoys the asymptotic marginal coverage guarantee

where the probability is computed with respect to the test sample ( X,Y ) ∼ P XY , which is independent of the sequence of samples { ( X t , Y t ) } T t =1 , and the convergence is in probability with respect to the sequence { ( X t , Y t ) } t ≥ 1 . Note that in [Angelopoulos et al., 2024b], a stronger version of (7) is provided, in which the limit holds almost surely.

1.2 Conditional and Localized Risk

The convergence guarantee (7) for ARC is marginalized over the covariate X . Therefore, there is no guarantee that the conditional miscoverage Pr[ Y / ∈ C T | X = x ] is smaller than the target α . This problem is particularly relevant for high-stakes applications in which it is important to ensure a homogeneous level of reliability across different regions of the input space, such as across subpopulations. That said, even when the set predictor C ( X |D cal ) is obtained based on an offline calibration data set D cal with i.i.d. data ( X,Y ) ∼ P XY , it is generally impossible to control the conditional miscoverage probability as

without making further assumptions about the distribution P XY or producing uninformative prediction sets [Vovk, 2012, Foygel Barber et al., 2021].

A relaxed marginal-to-conditional guarantee was considered by Gibbs et al. [2023], which relaxed the marginal miscoverage requirement (8) as

where W is a set of non-negative reweighting functions, and the expectation is taken over the joint distribution of the calibration data D cal and the test pair ( X,Y ) . Note that with a singleton set W encompassing a single constant function, e.g., w ( x ) = 1 , the criterion (9) reduces to marginal coverage. Furthermore, as illustrated in Figure 2, depending on the degree of localization of the functions in set W , the criterion (9) interpolates between marginal and conditional guarantees.

At the one extreme, a marginal guarantee like (7) is recovered when the reweighting functions are constant. Conversely, at the other extreme, conditional guarantees as in (8) emerge when the reweighting functions are maximally localized, i.e., when W = { w ( x ) = δ ( x -µ ) : µ ∈ X} , where δ ( x ) denotes the Dirac delta function. In between these two extremes, one obtains an intermediate degree of localization. For example, this can be done by considering reweighting functions such as

where l ≥ 0 is a fixed length scale, κ ≥ 0 is a fixed scaling parameter, and ∥·∥ denotes the Euclidean norm. Furthermore, function w ( x ) may also depend on the output of the pre-trained model, supporting calibration requirements via constraints of the form (9) [Zhang et al., 2024].

In Gibbs et al. [2023], the authors demonstrated that it is possible to design offline set predictors C ( X |D cal ) that approximately control risk (9), with an approximation gap that depends on the degree of localization of the family W of weighting functions.

1.3 Localized Risk Control

Motivated by the importance of conditional risk guarantees, we propose Localized ARC (L-ARC), a novel online calibration algorithm that produces prediction sets with localized statistical risk control guarantees as in (9), while also retaining the worst-case deterministic long-term guarantees (6) of ARC. Unlike Gibbs et al. [2023], our work focuses on online settings in which calibration is carried out sequentially based on feedback received on past decisions.

The key technical innovation of L-ARC lies in the way set predictions are constructed. As detailed in Section 2, L-ARC prediction sets replace the single threshold in (1) with a threshold function g ( · ) mapping covariate X to a localized threshold value g ( X ) . The threshold function is adapted in an online fashion within a reproducing kernel Hilbert space (RKHS) family G based on an input data stream and loss feedback. The choice of the RKHS family determines the family W of weighting functions in the statistical guarantee of the form (9), thus dictating the desired level of localization.

The main technical results, presented in Section 2.3, are as follows.

In the case of i.i.d. sequences, ( X t , Y t ) ∼ P XY for all t ≥ 1 , L-ARC provides localized statistical risk guarantees where the reweighting class W corresponds to all non-negative functions w ∈ G with a positive mean under distribution P XY . More precisely, given a target loss value α , the time-averaged threshold function

ensures that for any function w ∈ W , the limit

holds, where convergence is in probability with respect to the sequence { ( X t , Y t ) } t ≥ 1 and the average is over the test pair ( X,Y ) . The gap A ( G , w ) depends on both the RKHS G and function w ; it increases with the level of localization of the functions in the RKHS G ; and it equals zero in the case of constant threshold functions, recovering (7) for the special case of the miscoverage loss.

Furthermore, for an arbitrary sequence { ( X t , Y t ) } t ≥ 1 L-ARC has a cumulative loss that converges to a neighborhood of the nominal reliability level α as

where B ( G ) and C ( G ) are terms that increase with the level of localization of the function in the RKHS G . The quantity C ( G ) equals zero in the case of constant threshold functions, recovering the guarantee (6) of ARC.

In Section 3 we showcase the superior conditional risk control properties of L-ARC as compared to ARC for the task of electricity demand forecasting, tumor segmentation, and beam selection in wireless networks.

2 Localized Adaptive Risk Control

2.1 Setting

Unlike the ARC prediction set (1), L-ARC adopts prediction sets that are defined based on a threshold function g t : X → R . Specifically, at each time t ≥ 1 the L-ARC prediction set is obtained based on a non-conformity scoring function s : X × Y → R as

By (14), the threshold g t ( X t ) is localized, i.e., it is selected as a function of the current input X t . In this paper, we consider threshold functions of the form

where c t ∈ R is a constant and function f t ( · ) belongs to a reproducing kernel Hilbert space (RKHS) H associated to a kernel k ( · , · ) : X × X → R with inner product ⟨· , ·⟩ H and norm ∥·∥ H . Note that the threshold function g t ( · ) belongs to the RKHS G determined by the kernel k ′ ( · , · ) = k ( · , · ) + 1 .

We focus on the online learning setting, in which at every time set t ≥ 1 , the model observes an input feature X t , produces a set C t , and receives as feedback the loss L t = L ( C t , Y t ) . Note that label Y t may not be directly observed, and only the loss L ( C t , Y t ) may be recorded. Based on the observed sequence of features X t and feedback L t , we are interested in producing prediction sets as in (14) that satisfy the reliability guarantees (12) and (13), with a reweighting function set W encompassing all non-negative functions w ( · ) ∈ G with a positive mean E X [ w ( X )] under distribution P X , i.e.,

Importantly, as detailed below, the level of localization in guarantee (12) depends on the choice of the kernel k ( · , · ) .

2.2 L-ARC

Given a regularization parameter λ > 0 and a learning rate η t ≤ 1 /λ , L-ARC updates the threshold function g t ( · ) = f t ( · ) + c t in (14) based on the recursive formulas

with f 1 ( · ) = 0 and c 1 = 0 . In order to implement the update (17)-(18), it is useful to rewrite the function g t +1 ( · ) as

where the coefficients { a i t +1 } t i =1 are recursively defined as

Accordingly, if the loss L t is larger than the long-term target α , the update rule (20)-(21) increases the function g t +1 ( · ) around the current input X t , while decreasing it around the previous inputs X 1 , . . . , X t -1 . Intuitively, this change enhances the reliability for inputs in the neighborhood of X t .

It is important to note that, at any time t , computing the threshold function (19) requires storing the coefficients { a i t } t -1 i =1 and c t , as well as the input data { X t } t i =1 . Consequently, L-ARC has a linear memory requirement in t , which is a known limitation of non-parametric learning in online settings [Koppel et al., 2020]. Previous research has explored methods that trade memory efficiency for accuracy [Kivinen et al., 2004]. In Appendix C.3, we build on these approaches to present a memory-efficient variant of L-ARC that allows for a trade-off between localized risk control and memory requirements.

2.3 Theoretical Guarantees

In this section, we formalize the theoretical guarantees of L-ARC, which were informally stated in Section 1.3 as (12) and (13).

Assumption 1 (Stationary and bounded kernel) . The kernel function is stationary, i.e., k ( x, x ′ ) = ˜ k ( ∥ x -x ′ ∥ ) , for some non-negative function ˜ k ( · ) , which is ρ -Lipschitz for some ρ > 0 , upper bounded by κ < ∞ , and coercive, i.e., lim z →∞ ˜ k ( z ) = 0 .

Many well-known stationary kernels, such as the radial basis function (RBF), Cauchy, and triangular kernels, satisfy Assumption 1. The smoothness parameter ρ and the maximum value of the kernel function κ determine the localization of the threshold function g t ( · ) ∈ G . For example, the set of functions W defined in (10) corresponds to the function class (16) associated with the RKHS defined by the raised RBF kernel k ( x, x ′ ) = κ exp( -∥ x -x ′ ∥ 2 /l ) + 1 , with length scale l = 2 e ( κ/ρ ) 2 . As illustrated in Figure 2, by increasing κ and ρ , we obtain functions with an increasing level of localization, ranging from constant functions to maximally localized functions.

Assumption 2 (Bounded non-conformity scores) . The non-conformity scoring function is nonnegative and bounded, i.e., s ( x, y ) ≤ S max < ∞ for any pair ( x, y ) ∈ X × Y .

Assumption 3 (Bounded and monotone loss) . The loss function is non-negative; bounded, i.e., L ( C, Y ) ≤ B < ∞ for any C ⊆ Y and Y ∈ Y ; and monotonic, in the sense that for prediction sets C ′ and C such that C ′ ⊆ C , the inequality L ( C, Y ) ≤ L ( C ′ , Y ) holds for any Y ∈ Y .

2.3.1 Statistical Localized Risk Control

To prove the localized statistical guarantee (12) we will make the following assumption.

Assumption 4 (Strictly decreasing loss) . For any fixed threshold function g ( · ) ∈ G , the loss E Y [ L ( C ( X,g ) , Y ) | X = x ] is strictly decreasing in the threshold g ( x ) for any x ∈ X .

Assumption 5 (Left-continuous loss) . For any fixed threshold function g ( · ) ∈ G , the loss L ( C ( x, g + h ) , y ) is left-continuous in h ∈ R for any ( x, y ) ∈ X × Y .

Theorem 1. Fix a user-defined target reliability α . For any regularization parameter λ > 0 and any learning rate sequence η t = η 1 t -1 / 2 < 1 /λ , for some η 1 > 0 , given a sequence { ( X t , Y t ) } T t =1 of i.i.d. samples from P XY , the time-averaged threshold function (11) satisfies the limit

for any weighting function w ( · ) = f w ( · ) + c w ∈ W where the expectation is with respect to the test sample ( X,Y ) .

By (22), the average localized loss converges in probability to a quantity that can be bounded by the target α with a gap A ( G , w ) that increases with the level of localization κ .

2.3.2 Worst-Case Deterministic Long-Term Risk Control

Theorem 2. Fix a user-defined target reliability α . For any regularization parameter λ > 0 and any learning rate sequence η t = η 1 t -1 / 2 < 1 /λ with η 1 > 0 , given any sequence { ( X t , Y t ) } T t =1 with bounded input ∥ X t ∥ ≤ D < ∞ , L-ARC produces a sequence of threshold functions { g t ( · ) } T t =1 in (19) that satisfy the inequality

Formalizing the upper bound in (13), Theorem 2 states that the difference between the long-term cumulative risk and the target reliability level α decreases with a rate B ( G ) T -1 / 2 to a value C ( G ) = κB that is increasing with the maximum value of the kernel κ . In the special case, κ = 0 which corresponds to no localization, the right-hand side of (23) vanishes in T , recovering ARC long-term guarantee (6).

3 Experiments

In this section, we explore the worst-case long-term and statistical localized risk control performance of L-ARC as compared to ARC. Firstly, we address the task of electricity demand forecasting, utilizing data from the Elec2 dataset [Harries et al., 1999]. Next, we present an experiment focusing on tumor segmentation, where the data comprises i.i.d. samples drawn from various image datasets [Jha et al., 2020, Bernal et al., 2015, 2012, Silva et al., 2014, Vázquez et al., 2017]. Finally, we study a problem in the domain of communication engineering by focusing on beam selection, a key task in wireless systems [Ali et al., 2017]. A further example concerning applications with calibration constraints can be found in Appendix C.2. Unless stated otherwise, we instantiate L-ARC with the RBF kernel

0

10000

20000

30000

40000

Time-step (

t

)

0

.

89

0

.

90

0

.

91

0

.

92

Long-term Coverage

ARC

L-ARC

l

=2.0

L-ARC

l

=1.0

L-ARC

l

=0.5

ARC

l

=2

l

=1

l

=0.5

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Average Miscoverage Error

α

Marginal

Mon-Fri

Sat-Sun

0

500

1000

1500

2000

Time-step (

t

)

0

.

050

0

.

075

0

.

100

0

.

125

Long-term FNR

ARC

L-ARC

l

=2.0

L-ARC

l

=1.0

L-ARC

l

=0.5

ARC

l

=2

l

=1

l

=0.5

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Average FNR Error

α

Marginal

Kvasir

CVC-300

CVC-ClinicDB

CVC-ColonDB

ETIS-LaribPolypDB

ARC

l

=2

l

=1

l

=0.5

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Average Mask Size

Marginal

Kvasir

CVC-300

CVC-ClinicDB

CVC-ColonDB

ETIS-LaribPolypDB

k ( x, x ′ ) = κ exp( -∥ x -x ′ ∥ 2 /l ) with κ = 1 , length scale l = 1 and regularization parameter λ = 10 -4 . With a smaller length scale l , we obtain increasingly localized weighting functions. All the experiments are conducted on a consumer-grade Mac Mini with an M1 chip. The simulation code is available at https://github.com/kclip/localized-adaptive-risk-control.git .

3.1 Electricity Demand

The Elec2 dataset comprises T = 45312 hourly recordings of electricity demands in New South Wales, Australia. The data sequence { Y t } T t =1 is subject to distribution shifts due to fluctuations in demand over time, such as between day and night or between weekdays and weekends. We adopt a setup akin to that of Angelopoulos et al. [2024b], wherein the even-time data samples are used for online calibration while odd-time data samples are used to evaluate coverage after calibration. At time t , the observed covariate X t corresponds to the past time series Y 1: t -1 , and the forecasted electricity demand ˆ Y t is obtained based on a moving average computed from demand data collected within the preceding 24 to 48 hours. We produce prediction sets C t based on the non-conformity score s ( X t , Y t ) = | ˆ Y t -Y t | and we target a miscoverage rate α = 0 . 1 using the miscoverage loss (3). Both ARC and L-ARC use the learning rate η t = t -1 / 2 . L-ARC is instantiated with the RBF kernel k ( x, x ′ ) = κ exp( -∥ ϕ ( x ) -ϕ ( x ′ ) ∥ 2 /l ) , where ϕ ( x ) is a 7-dimensional feature vector corresponding to the daily average electricity demand during the past 7 days.

In the left panel of Figure 3, we report the cumulative miscoverage error of ARC and L-ARC for different values of the localization parameter l . All algorithms converge to the desired coverage level of 0.9 in the long-term. The right panel of Figure 3, displays the average miscoverage error on the hold-out dataset at convergence. We specifically evaluate both the marginalized miscoverage rate and the conditional miscoverage rate separately over weekdays and weekends. L-ARC is shown to reduce the weekend coverage error rate as compared to ARC providing balanced coverage as the length scale l decreases.

3.2 Tumor Image Segmentation

In this section, we focus on the task of calibrating a predictive model for tumor segmentation. Here, the feature vector X t represents a d H × d W image, while the label Y t ⊆ P identifies a subset of the image pixels P = { (1 , 1) , . . . , ( d H , d W ) } that encompasses the tumor region. As in Angelopoulos et al. [2022], the dataset is a compilation of samples from several open-source online repositories: Kvasir, CVC-300, CVC-ColonDB, CVC-ClinicDB, and ETIS-LaribDB. We reserve 50 samples from each repository for testing the performance post-calibration, while the remaining T = 2098 samples are used for online calibration. Predicted sets are obtained by applying a threshold g ( X t ) to the pixel-wise logits f ( p H , p W ) generated by the PraNet segmentation model [Fan et al., 2020], with the objective of controlling the false negative ratio (FNR) L ( C t , Y t ) = 1 -| C t ∩ Y t | / | Y t | . Both ARC and L-ARC are run using the same decaying learning rate η t = 0 . 1 t -1 / 2 . L-ARC is instantiated with the RBF kernel k ( x, x ′ ) = κ exp( -∥ ϕ ( x ) -ϕ ( x ′ ) ∥ 2 /l ) , where ϕ ( x ) is a 5-dimensional feature vector obtained via the principal component analysis (PCA) from the last hidden layer of the ResNet model used in PraNet.

In the leftmost panel of Figure 4, we report the long-term FNR for varying values of the localization parameter l , targeting an FNR level α = 0 . 1 . All methods converge rapidly to the desired FNR level, ensuring long-term risk control. The calibrated models are then tested on the hold-out data, and the FNR and average predicted set size are separately evaluated across different repositories. In the middle and right panels of Figure 4, we report the average FNR and average prediction set size averaged over 10 trials.

The model calibrated via ARC has a marginalized FNR error larger than the target value α . Moreover, the FNR error is unevenly distributed across the different data repositories, ranging from FNR = 0 . 08 for CVC-300 to FNR = 0 . 32 for ETIS-LaribPolypDB. In contrast, L-ARC can equalize performance across repositories, while also achieving a test FNR closer to the target level. In particular, as illustrated in the rightmost panel, L-ARC improves the FNR for the most challenging subpopulation in the data by increasing the associated prediction set size, while maintaining a similar size for subpopulations that already have satisfactory performance.

3.3 Beam Selection

0

5000

10000

15000

20000

Time step (

t

)

0

.

090

0

.

095

0

.

100

Long-term Risk

ARC

L-ARC

0

.

1

0

.

2

0

.

3

0

.

4

2

.

5

5

.

0

Avg. Beam Set Size

-

50

0

50

p

x

-

20

0

20

40

60

p

y

Transmitter

ARC

-

50

0

50

p

x

Transmitter

Mondrian ARC

-

50

0

50

p

x

Transmitter

L-ARC

-

45

-

40

-

35

-

30

-

25

SNR

Target Risk (

L

∗

)

Motivated by the importance of reliable uncertainty quantification in engineering applications, we address the task of selecting location-specific beams for the initial access procedure in sixth-generation wireless networks [Ali et al., 2017]. Further details regarding the engineering aspects of the problem and the simulation scenario are provided in Appendix C.1.1. In the beam selection task, at each time t , the observed covariate corresponds to the location X t = [ p x , p y ] of a receiver within the network deployment, where p x and p y represent the geographical coordinates. Based on the observed covariate, the transmitter chooses a set, denoted as C t ⊆ [1 , · · · , B max ] , consisting of a subset of the B max available communication beams.

Each communication beam i is associated with a wireless link characterized by a signal-to-noise ratio Y t,i , which follows an unknown distribution depending on the user's location X t . We represent the vector of signal-to-noise ratios as Y t = [ Y t, 1 , . . . , Y t,B max ] ∈ R B max . For a set C t , the transmitter sweeps over the beam set C t , and the performance is measured by the ratio between the SNR obtained

on the best beam in set C t and the best SNR on all the beams, i.e.,

Given an SNR predictor ˆ Y t = f SNR ( X t ) for all beams at location X t , we consider sets that include only beams with a predicted SNR exceeding a threshold g t ( X t ) as

In this setting, localization refers to the fair provision of service across the entire deployment area. As a benchmark, we thus also consider an additional calibration strategy that divides the deployment area into two regions: one encompassing all locations near the transmitter, which are more likely to experience high SNR levels, and the other including locations far from the transmitter. For each of these regions, we run two separate instances of ARC algorithms. Inspired by the method introduced in Boström et al. [2021] for offline settings, we refer to this baseline approach as Mondrian ARC.

In the left panels of Figure 5, we compare the performance of ARC, Mondrian ARC, and L-ARC with an RBF kernel with l = 10 , using a calibration data sequence of length T = 25000 . All methods achieve the target long-term SNR regret, but L-ARC achieves this result while selecting sets with smaller sizes, thus requiring less time for beam sweeping. Additionally, as illustrated on the right panel, thanks to the localization of the threshold function, L-ARC ensures a satisfactory communication SNR level across the entire deployment area. In contrast, both ARC and Mondrian ARC produce beam-sweep sets with uneven guarantees over the network deployment area.

4 Related Work

Our work contributes to the field of adaptive conformal prediction (CP), originally introduced by Gibbs and Candes [2021]. Adaptive CP extends traditional CP [Vovk et al., 2005] to online settings, where data is non-exchangeable and may be affected by distribution shifts. This extension has found applications in reliable time-series forecasting [Xu and Xie, 2021, Zaffran et al., 2022], control [Lekeufack et al., 2023, Angelopoulos et al., 2024a], and optimization [Zhang et al., 2023, Deshpande et al., 2024]. Adaptive CP ensures that prediction sets generated by the algorithm contain the response variable with a user-defined coverage level on average across the entire time horizon. Recently, Bhatnagar et al. [2023] proposed a variant of adaptive CP based on strongly adaptive online learning, providing coverage guarantees for any subsequence of the data stream. While their approach offers localized guarantees in time, L-ARC provides localized guarantees in the covariate space. More similar to our work is [Bastani et al., 2022], which studies group-conditional coverage. Our work extends beyond coverage guarantees to a more general risk definition, akin to Feldman et al. [2022]. Angelopoulos et al. [2024a] studied the asymptotic coverage properties of adaptive conformal predictions in the i.i.d. setting; and our work extends these results to encompass covariate shifts. Finally, the guarantee provided by L-ARC is similar to that of Gibbs et al. [2023], albeit for an offline conformal prediction setting.

5 Conclusion and Limitations

We have presented and analyzed L-ARC, a variant of adaptive risk control that produces prediction sets based on a threshold function mapping covariate information to localized threshold values. L-ARC can guarantee both worst-case deterministic long-term risk control and statistical localized risk control. Empirical analysis demonstrates L-ARC's ability to effectively control risk for different tasks while providing prediction sets that exhibit consistent performance across various data subpopulations. The effectiveness of L-ARC is contingent upon selecting an appropriate kernel function. Furthermore, L-ARC has memory requirements that grow with time due to the need to store the input data { X t } t> 1 and coefficients (20)-(21). These limitations of L-ARC motivate future work aimed at optimizing online the kernel function based on hold-out data [Kiyani et al., 2024] or in an online manner [Angelopoulos et al., 2024a], and at studying the statistical guarantees of memory-efficient variants of L-ARC [Kivinen et al., 2004].

6 Acknowledgments

This work was supported by the European Union's Horizon Europe project CENTRIC (101096379). The work of Osvaldo Simeone was also supported by the Open Fellowships of the EPSRC (EP/W024101/1) by the EPSRC project (EP/X011852/1), and by Project REASON, a UK Government funded project under the Future Open Networks Research Challenge (FONRC) sponsored by the Department of Science Innovation and Technology (DSIT). We would also like to express our gratitude to Anastasios Angelopoulos for valuable insights on the technical content of the paper.

References

Anum Ali, Nuria González-Prelcic, and Robert W Heath. Millimeter wave beam-selection using outof-band spatial information. IEEE Transactions on Wireless Communications , 17(2):1038-1052, 2017.

Anastasios Angelopoulos, Emmanuel Candes, and Ryan J Tibshirani. Conformal PID control for time series prediction. Advances in Neural Information Processing Systems , 36, 2024a.

Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814 , 2022.

Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Online conformal prediction with decaying step sizes. arXiv preprint arXiv:2402.01139 , 2024b.

Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Practical adversarial multivalid conformal prediction. Advances in Neural Information Processing Systems , 35:29362-29373, 2022.

Jorge Bernal, Javier Sánchez, and Fernando Vilarino. Towards automatic polyp detection with a polyp appearance model. Pattern Recognition , 45(9):3166-3182, 2012.

Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics , 43:99-111, 2015.

Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference on Machine Learning , pages 2337-2363. PMLR, 2023.

Henrik Boström, Ulf Johansson, and Tuwe Löfström. Mondrian conformal predictive distributions. In Conformal and Probabilistic Prediction and Applications , pages 24-38. PMLR, 2021.

Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory , 50(9):2050-2057, 2004.

Shachi Deshpande, Charles Marx, and Volodymyr Kuleshov. Online calibrated and conformal prediction improves bayesian optimization. In International Conference on Artificial Intelligence and Statistics , pages 1450-1458. PMLR, 2024.

Deng-Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse attention network for polyp segmentation. In International conference on medical image computing and computer-assisted intervention , pages 263-273. Springer, 2020.

Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving risk control in online learning settings. arXiv preprint arXiv:2205.09095 , 2022.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA , 10(2):455-482, 2021.

Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616 , 2023.

Andrea Goldsmith. Wireless communications . Cambridge university press, 2005.

Michael Harries, New South Wales, et al. Splice-2 comparative evaluation: Electricity pricing. 1999.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jakob Hoydis, Fayçal Aït Aoudia, Sebastian Cammerer, Merlin Nimier-David, Nikolaus Binder, Guillermo Marcus, and Alexander Keller. Sionna RT: Differentiable ray tracing for radio propagation modeling. arXiv preprint arXiv:2303.11103 , 2023.

Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas De Lange, Dag Johansen, and Håvard D Johansen. Kvasir-seg: A segmented polyp dataset. In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, Proceedings, Part II 26 , pages 451-462. Springer, 2020.

Jyrki Kivinen, Alexander J Smola, and Robert C Williamson. Online learning with kernels. IEEE transactions on signal processing , 52(8):2165-2176, 2004.

Shayan Kiyani, George Pappas, and Hamed Hassani. Conformal prediction with learned features. arXiv preprint arXiv:2404.17487 , 2024.

Alec Koppel, Amrit Singh Bedi, Ketan Rajawat, and Brian M Sadler. Optimally compressed nonparametric online learning: Tradeoffs between memory and consistency. IEEE Signal Processing Magazine , 37(3):61-70, 2020.

Jordan Lekeufack, Anastasios A Angelopoulos, Andrea Bajcsy, Michael I Jordan, and Jitendra Malik. Conformal decision theory: Safe autonomous decisions from imperfect predictions. arXiv preprint arXiv:2310.05921 , 2023.

Horea Muresan and Mihai Oltean. Fruit recognition from images using deep learning. Acta Universitatis Sapientiae, Informatica , 10(1):26-42, 2018.

Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer. International journal of computer assisted radiology and surgery , 9:283-293, 2014.

David Vázquez, Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Antonio M López, Adriana Romero, Michal Drozdzal, and Aaron Courville. A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of healthcare engineering , 2017, 2017.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning , pages 475-490. PMLR, 2012.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world , volume 29. Springer, 2005.

Wojciech Wisniewski, David Lindsay, and Sian Lindsay. Application of conformal prediction interval estimations to market makers' net positions. In Conformal and probabilistic prediction and applications , pages 285-301. PMLR, 2020.

Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning , pages 11559-11569. PMLR, 2021.

Lequan Yu, Hao Chen, Qi Dou, Jing Qin, and Pheng Ann Heng. Integrating online and offline three-dimensional deep learning for automated polyp detection in colonoscopy videos. IEEE journal of biomedical and health informatics , 21(1):65-75, 2016.

Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning , pages 25834-25866. PMLR, 2022.

Matteo Zecchin, Sangwoo Park, and Osvaldo Simeone. Forking uncertainties: Reliable prediction and model predictive control with sequence models via conformal risk control. IEEE Journal on Selected Areas in Information Theory , 2024.

Lujing Zhang, Aaron Roth, and Linjun Zhang. Fair risk control: A generalized framework for calibrating multi-group fairness risks. arXiv preprint arXiv:2405.02225 , 2024.

Yunchuan Zhang, Sangwoo Park, and Osvaldo Simeone. Bayesian optimization with formal safety guarantees via online conformal prediction. arXiv preprint arXiv:2306.17815 , 2023.

A Proof of Theorem 1

We are interested in bounding the localized risk in (22) of the threshold function (11) for all weighting functions in set W defined in (16). To study the limit in (22) we first note that L-ARC update rule (18) corresponds to an online gradient descent step for a loss function ℓ ( g, x, y ) , with respect to function f ( · ) and constant c in function g ( · ) = f ( · ) + c as in (15). In particular, interpreting the update rule (17)-(18) as a gradient descent step, we obtain that the partial derivatives of the loss function ℓ ( g, x, y ) evaluated at g ( · ) = f ( · ) + c are

so that the first order approximation of the loss ℓ ( g, x, y ) around g ( · ) is given by

In order to study the convexity of the loss ℓ ( g, x, y ) in g ( · ) , we compute the the derivatives of (26)(27) with respect to f ( · ) and c . The derivative of (26) with respect to f is the operator A : H → H satisfying

It follows that

Similarly, the derivative of (26) with respect to c is the operator B : R →H is given by

The derivative of (27) with respect to c is given by

and the derivative with respect to to f is the operator C : H → R given by

which satisfies so that

From Assumption 4, the inequality L ′ = -E Y [ ∂ L ( C ( x,g ) ,Y ) ∂g ( x ) | X = x ] ≥ γ > 0 holds. Thus, the second-order term of the approximation of E Y [ ℓ ( g, X, Y ) | X = x ] around g ( · ) = 0 satisfies

̸

We then conclude that the loss function E Y [ ℓ ( g, X, Y ) | X = x ] is strongly convex in g ( · ) , and that the population loss minimizer

is unique. For any covariate shift w ( · ) ∈ W denote its components f w ( · ) ∈ H and c w ∈ R such that w ( · ) = f w ( · ) + c w . From the first order optimality conditions, it holds that the directional derivatives with respect to f w ( · ) and c w must satisfy

which implies that for the optimal solution g ∗ ( · )

Equality (41) amounts to a localized risk control guarantee for the threshold g ∗ ( · ) for covariate shift in w ( · ) ∈ W . The following lemma states that the time-average L-ARC threshold function ¯ g T ( · ) defined in (11) converges to the population risk minimizer g ∗ ( · ) .

Lemma 1. For any regularization parameter λ > 0 and any learning rate sequence η t = η 1 t -1 / 2 < 1 /λ , for some η 1 > 0 , given a sequence { ( X t , Y t ) } T t =1 of i.i.d. samples from P XY , the time-averaged threshold function (11) satisfies for any ϵ > 0

Proof. To prove convergence in probability, we need to show that the loss function ℓ ( g, X, Y ) is bounded. To this end, we first show that ℓ ( g, X, Y ) is Lipschitz in g ( · ) by studying the norm of the derivatives (26)-(27). For g t ( · ) = f t ( · ) + c t returned by the update rule (18), the gradient with respect to f t ( · ) satisfies

where the first inequality follows from the boundedness on the kernel (Assumption 1) and the boundedness on the loss (Assumption 3), while the last follows from Proposition 1. The gradient with respect to c can be similarly bounded as

From the mean value theorem it follows that for g ( · ) = f ( · ) + c and g ′ ( · ) = f ′ ( · ) + c ′

Since L-ARC returns functions f t ( · ) with bounded RKHS norm and infinity norm (Proposition 1), and thresholds function g t ( · ) with bounded in infinity norm (Proposition 3), we conclude that there exists a finite ℓ max < ∞ such that | ℓ ( g, X, Y ) | ≤ ℓ max . Given that the loss is bounded we can apply

[Kivinen et al., 2004, Theorem 4] and obtain that for the threshold { g t ( · ) } t ≥ 1 returned by L-ARC and the population loss minimizer g ∗ ( · ) it holds

By Hoeffding's inequality the empirical average on the right-hand side of (46) converges to its expected value. Formally, we have that with probability at least 1 -δ with respect to the sequence { ( X t , Y t ) } T t =1

Similarly, by [Cesa-Bianchi et al., 2004, Theorem 2] the empirical risk on the left-hand side of (46) converges to the population risk of the time-averaged solution (11). With probability at least 1 -δ with respect to the sequence of samples { ( X t , Y t ) } T t =1 , it holds

Combining the two inequalities, with probability at least 1 -2 δ with respect to { ( X t , Y t ) } T t =1 ,

Since the E X,Y [ ℓ ( g, X, Y )] is strongly convex there exists a value γ > 0 such that the second order approximation of E X,Y [ ℓ ( g, X, Y )] at g ∗ ( · ) satisfies

Combining (50) and (49), and leveraging ∥ f ∥ ∞ ≤ √ κ ∥ f ∥ H , which follows from the Assumption 1, we conclude that with probability 1 -2 δ

Choosing δ = 1 T , for any ϵ > 0 , it holds

By itself, the convergence of the threshold function ¯ g T ( · ) to the population risk minimizer g ∗ ( · ) is not sufficient to provide localized risk control guarantees for L-ARC time-averaged solution. However, under the additional loss regularity assumption in Assumption 5, we can show that set predictor C ( X, ¯ g T ) enjoys conditional risk control for T →∞ .

Having assumed that the loss L ( C ( x, g ) , y ) is left-continuous and decreasing for larger prediction sets (Assumption 3 and 5), for any δ ′ > 0 there exists ϵ > 0 such that for g ( · ) such that ∥ g ∗ -g ∥ ∞ ≤ ϵ it holds

For such g ( · ) the following inequality holds

As stated in Lemma 1, we can always find T large enough, such that ∥ g ∗ -¯ g T ∥ ∞ ≤ ϵ with arbitrary large probability. This implies, that for any δ ′ > 0 and w ∈ W ,

where the inequality (55) follows from (54), the inequality (56) follows from (41) and the inequality (57) from Proposition 1.

B Proof of Theorem 2

We are interested in bounding the absolute difference between the cumulative loss value incurred by the set predictors { C ( g t , X t ) } T t =1 produced by L-ARC (18) and the target reliability level α , i.e.,

From Assumption 1 and having assumed ∥ X t ∥ ≤ D for t ≥ 1 , it follows that

A bound on the cumulative risk can then be obtained by bounding

where for a function f : X → R , the infinity norm ∥ f ∥ ∞ is defined as max x ∈X | f ( x ) | . In fact, from (60) we directly obtain a bound on the cumulative risk

To this end, we first note that functions { f t ( · ) } t ∈ N generated by (18) have bounded RKHS norm and are smooth.

Proposition 1. For every t ≥ 1 , we have the inequalities ∥ f t ∥ H ≤ B √ κ λ and ∥ f t ∥ ∞ ≤ κB λ .

Proof. The proof is by induction, with the base case ∥ f 1 ( · ) ∥ H ≤ B √ κ/λ being satisfied as f 1 ( · ) = 0 . The induction step is given as

where the equality (62) follows from the update rule (18); the inequality (63) from the properties of the norm, the inequality (64) from Assumption 1 and 3, and the inequality (65) from the induction hypothesis ∥ f t ∥ H ≤ B √ κ λ .

Proposition 2. For t ≥ 1 and any ( x, x ′ ) ∈ X × X we have

Proof. Denote the evaluation function at x as K x = k ( x, · ) . From Proposition 1 and the Lipschitz continuity assumed in Assumption 1, it follows that

where the first inequality follows from Cauchy-Schwarz inequality, the second from the Lipschitz continuity of the kernel, and the last one from Proposition 1 together with ∥ x ∥ ≤ D for x ∈ X .

Leveraging the above characterization of the function f t ( · ) returned by L-ARC, we now show that the threshold function g t ( · ) has maximum and minimum values that are uniformly bounded.

Proposition 3. For every t ≥ 1 and x ∈ X we have g t ( x ) ∈ [ G min , G max ] with and

Proof. We now prove the upper bound (68). The proof is by contradiction and it start by assuming that there exists a t > 1 and x ∈ X such that g t ( x ) ≥ G max while g t ′ ( · ) < G max for all t ′ < t . From the update rule (18) we have that

From Proposition 2 we also have

where the last inequality follows from G max being defined as (68). From Assumption 2, for all x ∈ X ,

which contradicts with the original assumption that there exists x such that g t ( x ) ≥ G max.

The proof of the lower bound (69) follows similarly. Assume there exists t > 1 and x ∈ X such that g t ( x ) ≤ G min while g t ′ ( · ) > G min for t ′ < t . From the update rule (18) we have that

From Proposition 2 we also have

where the last inequality follows from G min being defined as (69). From Assumption 2, for all x ∈ X ,

which contradicts the assumption that there exists min x ∈X g t ( x ) ≤ G min.

Having established an upper and lower bound on the maximum value of the function g t ( · ) generated by (18) we can now bound (60). Define ∆ t = η -1 t -η -1 t -1 and ∆ 1 = η -1 1 and note that

The first term can be bounded based on Proposition (3) as

and similarly, for the second term, we have

Fix a decreasing learning rate η t = η 1 t -ω and a regularization parameter λ = λ 0 T -ξ , then the E 1 becomes

For any ω < 1 , it follows

C Additional Experiments

C.1 Beam Selection

C.1.1 Simulation Details

-

50

0

50

p

x

-

20

0

20

40

60

p

y

Transmitter

For the beam selection experiment, we consider the network deployment depicted in Figure 6, in which a transmitter (green circle) communicates with users in an urban environment with multiple buildings (grey rectangles). We assume that communication occurs at a frequency f c = 2 . 14 GHz and that the transmitter is equipped with N t = 8 transmitting antennas while receiving users have singleantenna equipment. The transmitter adopts a discrete Fourier transform beamforming codebook of size B max = 11 , with each beam b i given by

where j = √ -1 . The wireless channel response h R ∈ C N t between the transmitter and a receiver located at X t = [ p x , p y ] ∈ R 2 , is modeled using Sionna ray-tracer [Hoydis et al., 2023], and we account for small scale fading using a Rayleigh noise model [Goldsmith, 2005]. The resulting channel vector is distributed as

where h R ( X t ) is the ray tracer output and Rayleigh ( σ ) is a Rayleigh distributed random variable with parameter σ = 10 -4 . Assuming unit power transmit symbols and receiver noise, for a channel vector h t the communication signal-to-noise ratio (SNR) obtained using the beamformer b i is given by

Beam sets are obtained calibrating an SNR predictor ˆ Y t = f SNR ( X t ) realized using a 3-layer fully connected neural network that is trained on 2500 samples with the user location generated uniformly at random within the deployment area.

C.1.2 Effect of the Length Scale

In Figure 7, we study the effect of the length scale l of the kernel function on the time-averaged threshold function ¯ g T ( X ) returned by L-ARC. We report the value of the L-ARC time-averaged threshold, ¯ g T ( X ) , in (11), for the same experimental set-up as in Section 3.3, and for increasing localization of the kernel function. As the length scale parameter l decreases, corresponding to a more localized kernel, the value of the threshold is allowed to vary more across the deployment area. In particular, the threshold function reduces its value around areas where the beam selection problem becomes more challenging, such as building clusters, in order to create larger beam selection sets.

-

50

-

25

0

25

50

p

x

-

20

0

20

40

60

p

y

Transmitter

L-ARC

l

= 50

-

50

-

25

0

25

50

p

x

Transmitter

L-ARC

l

= 25

-

50

-

25

0

25

50

p

x

Transmitter

L-ARC

l

= 10

-

30

-

25

-

20

-

15

-

10

-

5

¯

gT

(

p

x

, p

x

) [dBm]

0

2000

4000

6000

8000

Time-step (

t

)

0

.

72

0

.

74

0

.

76

Long-term Coverage

ARC

κ

=0

L-ARC

l

=1

L-ARC

l

=0.5

L-ARC

l

=0.25

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Model's Confidence (Conf(

X

))

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Coverage Rate

Target Coverage Level

ARC

L-ARC

l

=1

L-ARC

l

=0.5

L-ARC

l

=0.25

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Model's Confidence (Conf(

X

))

0

5

10

15

Set Size

ARC

L-ARC

l

=1

L-ARC

l

=0.5

L-ARC

l

=0.25

C.2 Image Classification with Calibration Requirements

In this section, we consider an image classification task under calibration requirements based on the fruit-360 dataset [Muresan and Oltean, 2018]. For this problem, the feature vector X t is an image of size 100 × 100 , and the corresponding label Y t ∈ Y = { 1 , . . . , 130 } is one of 130 types of fruit, vegetable, or nut in image X t . We study the online calibration of a pre-trained ResNet18 model [He et al., 2016]. For an input image X t , the prediction set is obtained from the model's predictive distribution ˆ p ( y | X t ) as

and we target the miscoverage loss (3) with a target miscoverage rate α = 0 . 25 . In order to capture calibration requirements, we impose coverage constraints that are localized in the model's confidence. The model's confidence indicator is given by the maximum value of the model's predictive distribution ˆ p ( y | X t ) , i.e.,

Accordingly, we run ARC and L-ARC calibration with a sequence of T = 8000 samples and we instantiate L-ARC using the exponential kernel k ( x, x ′ ) = κ exp( -∥ ϕ ( x ) -ϕ ( x ′ ) ∥ 2 /l ) , where the feature vector is given by the model's uncertainty, i.e., ϕ ( x ) = Conf( x ) .

In the left-most panel of Figure 8 we report the long-term coverage of ARC and L-ARC for an increasing level of localization obtained by decreasing the length scale l . All methods guarantee long-term coverage. In the middle panel, we use hold-out data to evaluate the coverage of the calibrated model conditioned on the model's confidence level. For small length scale l , L-ARC yields prediction sets that satisfy the coverage requirement across different levels of the model's confidence. In contrast, ARC, due to its inability to adapt the threshold function, has a large miscoverage rate for small model confidence levels. As illustrated in the right panel, this is achieved by producing a larger set size when the model's confidence is low.

0.5

Marginal

CVC-ClinicDB

Kvasir

CVC-ColonDB

Error

0.4

CVC-300

ETIS-LaribPolypDB

Decreasingmemory budget

AverageFNRE

0.3

0.2

0.1

0.0

ARC

Mmax=500

Mmax=1000Mmax=1500

L-ARC

L-ARC with memory budget Mmax

Decreasingmemorybudget

ARC

L-ARC Mmax =500

L-ARC Mmax=1000

L-ARC

60

-25

40

30

20

-35

R

0

40

20

Transmitter.

Transmitter

Transmitter

Transmitter

45

—50

0

50

-50

0

50

50

0

50

50

0

50

C.3 On the memory efficiency of L-ARC

In a manner similar to [Kivinen et al., 2004], it is possible to obtain a memory-efficient version of L-ARC that adopts a truncated version of L-ARC threshold (19) given by

Unlike the threshold (19), which has a linear memory requirement, the truncated version (86) requires a constant memory and computational load that are proportional to the number of coefficients M max. It is known that in online non-parametric learning, there exists a trade-off between memory efficiency and performance. In the following, we empirically study the trade-off between the localized risk control of L-ARC and its memory requirements by varying the parameteR M max.

C.3.1 Tumor Segmentation

Using the setup described in Section 3.2, we now consider calibrating the image segmentation model using L-ARC with a truncated threshold (86). In Figure 9, we report the average FNR conditioned on different data sources for M max ∈ { 500 , 1000 , 1500 } . As a benchmark, we also compare against ARC and L-ARC without truncation. By adjusting the value of M max, it is possible to trade off localized risk control for memory efficiency. In fact, the effect of truncation on L-ARC's performance is minimal when the number of coefficients in the truncation is large ( M max = 1500) . However, for greater memory savings ( M max = 500) , L-ARC's performance becomes similar to that of ARC. In all cases, L-ARC provides better localized risk control than ARC.

C.3.2 Beam Selection

We consider the beam selection problem discussed in Section 3.3. In Figure 10, we report the SNR levels across the deployment attained by ARC, L-ARC, and L-ARC with a truncated threshold with a maximum number of coefficients M max ∈ { 500 , 1000 } . As the number of coefficients M max and the memory requirement reduce, the localized risk control performance of L-ARC also decreases. Nonetheless, even for small M max, L-ARC delivers a more consistent SNR level across the deployment compared to ARC.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The theoretical claims about the proposed calibration algorithm are supported by Theorem 1 and Theorem 2, while experimental results in Section 3 demonstrate its capability to control long-term risk and to improve fairness across data subpopulations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 5, we highlight two primary limitations of L-ARC: its memory requirements and the necessity of specifying a suitable kernel. Additionally, we suggest potential directions for future research to address these issues. A memory-efficient version of L-ARC is given in the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Both theorems are preceded by a necessary set of assumptions. In the proofs, provided in the appendix, we reference these assumptions when using them.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the main text, we provide details of all the experiments, including factors influencing the proposed solution, such as datasets and algorithm parameters like the choice of kernel, localization parameters, and learning rate.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In the supplementary material, we include the source code of the experiments along with a concise guide containing all the necessary information to replicate the results. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the information, such as the type of datasets, the learning rate, the type of kernel function, and localization parameters, is reported in the main text. In the appendix, we provide additional details about the data generation for the beam selection experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Where possible we report 95% confidence intervals in the form of error bars or shaded areas.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We indicate that the experiments are run on a Mac Mini.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We carefully read and comply with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper provides a new calibration scheme that offers long-term risk control and statistical localized risk guarantees. We do not see any negative societal impacts associated with the proposed algorithm.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and models used in the experiments are properly credited by citing the corresponding papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not use crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel localized adaptive risk control algorithm that provides not only average case risk guarantees but also worst-case guarantees. Simulations in several different applications are provided, demonstrating the improved performance when compared with adaptive risk control.\n\nSTRENGTHS:\nThe paper is very well written. The illustration figures are very helpful in explaining the localization effects of difference choices of $w(x)$.\n\nThe problem is also well motivated: the worst case risk guarantee is indeed very important in applications such as medical imaging.\n\nThe proposed algorithm is novel.\n\nWEAKNESSES:\n1. My major concern is on the simulation results. Though I appreciate the improved performance compared with ARC in different settings, the paper does not provide any comparisons with the state-of-the-art approaches in these applications. For example, for electricity demand prediction, the SOTA method is based on quantile regression and there are many papers published trying to provide better predictions on demand. How does the proposed LARC compare with these SOTA methods? If LARC does not outperform the existing methods, what are the benefits of using LARC? Similarly, tumor image segmentation is also a standard task in medical imaging, so it will be interesting to compare LARC with the SOTA methods in that field too.\n\n2. It seems that the worst-case guarantee is only provided for i.i.d. data, is that correct? If so, what's the challenge of obtaining a worst-case guarantee for arbitrary data sequence? \n\nIn addition, the paper mentioned that the major improvement of this paper compared with [Angelopoulos et al., 2024b] is the online setting. However, usually, i.i.d. data is obtained by sampling uniformly from an offline data set. So what is the significance of the theoretical results if the worst-case guarantee can only be provided for i.i.d. cases?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the design and analysis of the localized version of adaptive risk control (L-ARC).  In the first section, the problem of classical ARC is nicely introduced, showing the threshold updating mechanism and the convergence analysis of the resulting loss.  Then, the problem setting, design, and analysis of ARC are naturally generalized to those of L-ARC.  Finally, L-ARC is applied to three practical problems of electricity demand forecasting, tumor segmentation, and beam selection.  The result of the experiments supports the usefulness of L-ARC.\n\nSTRENGTHS:\nThe application scope of L-ARC is broad, and it can be utilized in various application examples.  The reviewer believes L-ARC has a high potential for broad impacts.  The update model (17) and (18) developed for L-ARC in this paper is novel.  The theoretical guarantee, in particular, the convergence analysis of the localized loss is also given under standard assumptions.\n\nWEAKNESSES:\nBoth Theorems 1 and 2 state that the localized loss is bounded by \\kappa * B, meaning the maximum value of the kernel and an upper bound of the loss.  In the reviewer's understanding, the theorems are technically correct, but he/she cannot find their value.  They seem to be just deriving an obvious upper bound from the assumptions.\n\nQUESTIONS:\nRelated to Theorems 1 and 2, could the author comment on their value of the upper bound of the localized loss?\n\nL-ARC updates the threshold function, denoted by $g_t()$.  The setting seems to be restrictive.  Could the authors extend the setting to the case of multiple prediction sets?  In other words, multiple scoring functions s_1, s_2, ... are considered and their thresholds are adaptively updated in a similar manner to (17) and (18).\n\nLIMITATIONS:\nThe limitation of L-ARC is stated in Section 5, saying that it requires a memory function of storing the input data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper introduces Localized Adaptive Risk Control (L-ARC), an enhancement of Adaptive Risk Control (ARC). L-ARC updates a threshold function within a Reproducing Kernel Hilbert Space (RKHS) to provide localized risk guarantees while maintaining ARC's worst-case performance. Experiments show that L-ARC improves fairness across different data subpopulations in tasks like image segmentation.

STRENGTHS:
The introduction of Localized Adaptive Risk Control (L-ARC) is a significant advancement over traditional ARC. By focusing on localized risk guarantees, L-ARC addresses the critical issue of uneven risk distribution across different data subpopulations, which is a well-known limitation of ARC.

WEAKNESSES:
1. The introduction of a threshold function within an RKHS and the associated online adaptation process may complicate the implementation. Practitioners might find it challenging to understand and apply the method without a significant background in RKHS and online learning algorithms.
2. The paper primarily compares L-ARC with traditional ARC. Including comparisons with other state-of-the-art risk control or calibration methods would provide a more comprehensive evaluation of L-ARC's strengths and weaknesses.

QUESTIONS:
Refer to Weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an online calibration method to enhance the Adaptive Risk Control (ARC) framework. ARC traditionally adjusts prediction sets based on a scalar threshold to ensure long-term risk control and marginal coverage guarantees. However, as mentioned in the paper, it may unevenly distribute risk guarantees across different subpopulations. L-ARC addresses this by updating a threshold function within a reproducing kernel Hilbert space (RKHS) to provide localized statistical risk guarantees, ranging from conditional to marginal risk, while preserving the worst-case performance. My additional comments are as follows.\n\nSTRENGTHS:\n- The paper is easy to read. It could have been better structured, but it is okay. \n\n- The authors have proposed a new technique for calibration and provided theoretical and experimental results.\n\nWEAKNESSES:\n- The concept of input space is not clear in the introduction. \n\n- Is the pre-trained model trained on all the data sources that will be used during the calibration process if they are different? \n\n- Is the loss function convex in (2) convex?  And what the intuitive explanation that (6) should hold is. Does it somehow follow from the regret analysis of the online gradient descent algorithm, which converges at the same rate? \n\n- What is the crucial reason for using the RKHS function instead of a scalar? How is this useful? \n\n- Ideally, the introduction of RKHS should have helped with the rates, but it looks like the rates in (12) and (13) are worse than the rates in (9) and (6), so then in what terms localized ARS is better?\n\n- On the other hand, keeping track of a function g_t in RKHS for each t is way more expensive than just keeping track of a scalar in practice. Why the proposed method makes sense is not clear. \n\n- What is the reason to consider the functional form in (15)\n\n- How are the updates in (17)-(21) derived?  Equation (19) is fine, which is the standard representation of a function in RKHS\n\n- As mentioned before, the additional challenge with RKHS is that we need to now store a^i_t from i =1 to t because we need to update them, and for large T, this could be problematic in practice. And if you make the dictionary size of storing them constantly, then it would incur additional regret in the final terms. \n\n- The experimental results are weak. They are not descriptive enough to make sense of the advantages of the proposed techniques over the advantages of existing methods. The long-term coverage plots look almost similar, so why is the benefit coming in other figures? Also, authors should report the downside of the proposed RKHS-based approach, which would require more memory access to implement the proposed method, which is not required for ARS. So, comparisons are also not fair.\n\nQUESTIONS:\nPlease check the weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Localized Adaptive Risk Control (L-ARC) scheme for learning to perform conformal prediction from online data. In the setting under consideration, the data is potentially non-i.i.d. and the scalar threshold parameter typically used by existing methods in the construction of the prediction set is replaced by a function learned online from the data stream via functional stochastic gradient descent over a reproducing kernel Hilbert space (RKHS). It is shown that the selection of the RKHS -- i.e., the properties of the associated kernel, such as the scale parameter of a radial basis function (RBF) kernel -- yields \"localization\" of the resulting conformal predictor to the data set in the sense that prediction performance (miscoverage error) remains consistent across distinct subpopulations within the data. This is demonstrated through experiments on electricity forecasting, medical image segmentation, and beam selection problems that illustrate the localization property, while theoretical results characterize the price paid for this localization in terms of looser bounds on suboptimality and convergence rate.\n\nThe key algorithmic contribution of the paper is the use of *online* kernel learning, which is well-studied (compare eqs. (19)-(21) in the present submission with (10)-(12) in [Kivinen et al., 2004]), to learn a *function* to replace the scalar threshold typically used in conformal prediction. Existing methods for online conformal prediction [Gibbs and Cand&#232;s, 2021; Bhatnagar et al., 2023; Feldman et al., 2023; Angelopoulos et al., 2024] consider either a fixed scalar threshold in the construction of the prediction set, while recent work on conformal prediction with a threshold function [Gibbs et al., 2023] does not directly apply to the online setting. The primary theoretical contributions are the establishment of upper bounds characterizing the localization effects of the choice of RKHS kernel in terms of a certain notion of suboptimality and the rate of convergence to a neighborhood of optimality, where the size of the neighborhood is shown to depend on the choice of kernel. The experimental results illustrate that L-ARC outperforms the existing, non-localized ARC method [Gibbs and Cand&#232;s, 2021; Feldman et al., 2022] at achieving the desired level of miscoverage error across distinct subpopulations.\n\nSTRENGTHS:\nWhile the existence of an online method for conformal prediction over non-i.i.d. data that uses threshold functions appears to be an open problem in the recent conformal prediction literature, the significance and potential utility of using a threshold *function* taken from a RKHS instead of a scalar or another class of function are not immediately obvious. However, the theoretical and experimental results of this paper indicate that a useful notion of \"localization\", where the choice RKHS kernel allows reliable conformal prediction on data with distinct subpopulations, results from considering this class of threshold functions -- this is a very interesting and original insight that is likely to draw attention in the conformal prediction community, and is a major strength of the paper. The theoretical results also provide some useful insight into the effect of kernel choice in L-ARC, and the main steps in the analysis appear to be sound (I read but did not thoroughly check all details in the appendix). Finally, the experimental results provide strong support to the utility of the localization effect of the choice of kernel, without which the theoretical results would lose much of their force and the significance and potential utility of L-ARC would remain unclear.\n\nWEAKNESSES:\nThe primary weaknesses of this work arise from lack of context with previous work and lack of motivation and discussion of the technical results. These issues make it difficult to accurately judge its significance and contribution. Specifically:\n1. Important context with previous work is missing. First, the L-ARC method proposed in Sec. 2.2 is essentially an adaptation of online kernel learning (see [Kivinen et al., 2004] and its many citers) to the conformal prediction setting (as mentioned in the summary above, eqs. (19)-(21) in the present submission are very similar to (10)-(12) in [Kivinen et al., 2004]), yet this connection is not mentioned. This information is important for clarifying the limits of the present paper's contribution.\n2. The relationship between the technical results presented and previous analyses is unclear. In particular, it is unclear from the text (including the proofs in the appendix) what parts of the analysis draw on previous analyses of conformal prediction methods -- if the results are entirely independent and original, this can be highlighted -- and what key technical innovations were required in the theoretical analysis. \n3. Motivation of the technical results and discussion and clarification of their meaning is generally lacking. As a result, the technical meaning and effect of \"localization\" characterized in Sec. 2.3, especially in the Assumptions and Theorem 1, remain unclear. Specific questions regarding these issues are included in **Questions** 3, 4, and 5 below.\n\nQUESTIONS:\n1. What are the main technical innovations in the proofs of the main results?\n2. What parts of the analysis draw on previous work?\n3. How reasonable are the assumptions presented in Sec. 2.3 and when do they hold (especially Assumptions 4 and 5)?\n4. What is significance of the weighting inside the expectation on the LHS of eq. (22) and why is this relaxed, reweighted expectation meaningful?\n5. What is the role and importance of the weighting function $w(\\cdot)$ and the corresponding terms containing $f_w(\\cdot)$ in eq. (22)? The presence of $w(\\cdot)$ seems like an artifact arising from considering the covariate shift $w(\\cdot)$ in lines 418-419 in the appendix, which might be expected to go away when $max_w$ is taken in the proof of Lemma 1 in Sec. A; why do $w(\\cdot)$ and $f_w(\\cdot)$ persist in the statement of Theorem 1 and how do we interpret them?\n6. The present paper uses the term Adaptive Risk Control (ARC) to refer to the methods proposed in [Gibbs and Cand&#232;s, 2021] and [Feldman et al., 2022] on lines 18-19, but these works call their methods Adaptive Conformal Inference (ACI) and Rolling Risk Control (Rolling RC), respectively; which of these does ARC refer to, and which is implemented in the experiments?\n\nLIMITATIONS:\nAside from the issues raised above, the limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-31"}