Title: Conformal PID Control for Time Series Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: conformal prediction

time series

uncertainty quantification

distribution shift
Abstract: We study the problem of uncertainty quantification for time series prediction, with the goal of providing easy-to-use  algorithms with formal guarantees. The algorithms we present build upon ideas from conformal prediction and control theory, are able to prospectively model conformal scores in an online setting, and adapt to the presence of systematic errors due to seasonality, trends, and general distribution shifts. Our theory both simplifies and strengthens existing analyses in online conformal prediction. Experiments on 4-week-ahead forecasting of statewide COVID-19 death counts in the U.S. show an improvement in coverage over the ensemble forecaster used in
official CDC communications. We also run experiments on predicting electricity demand, market returns, and temperature using autoregressive, Theta, Prophet, and Transformer models. We provide an extendable codebase for testing our methods and for the integration of new algorithms, data sets, and forecasting rules at [this link](http://github.com/aangelopoulos/conformal-time-series).

Conformal PID Control for Time Series Prediction

Abstract

We study the problem of uncertainty quantification for time series prediction, with the goal of providing easy-to-use algorithms with formal guarantees. The algorithms we present build upon ideas from conformal prediction and control theory, are able to prospectively model conformal scores in an online setting, and adapt to the presence of systematic errors due to seasonality, trends, and general distribution shifts. Our theory both simplifies and strengthens existing analyses in online conformal prediction. Experiments on 4-week-ahead forecasting of statewide COVID-19 death counts in the U.S. show an improvement in coverage over the ensemble forecaster used in official CDC communications. We also run experiments on predicting electricity demand, market returns, and temperature using autoregressive, Theta, Prophet, and Transformer models. We provide an extendable codebase for testing our methods and for the integration of new algorithms, data sets, and forecasting rules. 1

1 Introduction

Machine learning models run in production systems regularly encounter data distributions that change over time. This can be due to factors such as seasonality and time-of-day, continual updating and retraining of upstream machine learning models, changing user behaviors, and so on. These distribution shifts can degrade a model's predictive performance. They also invalidate standard techniques for uncertainty quantification, such as conformal prediction [36, 35].

To address the problem of shifting distributions, we consider the task of prediction in an adversarial online setting, as in [16]. In this problem setting, we observe a (potentially) adversarial time series of deterministic covariates x t ∈ X and responses y t ∈ Y , for t ∈ N = { 1 , 2 , 3 , . . . } . As in standard conformal prediction, we are free to define any conformal score function s t : X × Y → R , which we can view as measuring the accuracy of our forecast at time t . We will assume with a loss of generality that s t is negatively oriented (lower values mean greater forecast accuracy). For example, we may use the absolute error s t ( x, y ) = | y -f t ( x ) | , where f t is a forecaster trained on data up to but not including data at time t .

The challenge in the sequential setting is as follows. We seek to invert the score function to construct a conformal prediction set ,

where q t is an estimated 1 -α quantile for the distribution of the score s t ( x t , y t ) at time t . In standard conformal prediction, we would take q t to be a level 1 -α sample quantile (up to a finite-sample correction) of s t ( x i , y i ) , i < t ; if the data sequence ( x i , y i ) , i ∈ N were i.i.d. or exchangeable, then this would yield 1 -α coverage [35] at each time t . However, in the sequential setting, which does not assume exchangeability (or any probabilistic model for the data for that matter), choosing q t in (1) to yield coverage is a formidable task. In fact, if we are not willing to make any assumptions about the data sequence, then a coverage guarantee at time t would only be possible with trivial methods, which construct prediction intervals of infinite sizes.

Therefore, our goal is to achieve long-run coverage in time. That is, letting err t = ✶ { y t / ∈ C t } , we would like to achieve, for large integers T ,

under few or no assumptions, where o (1) denotes a quantity that tends to zero as T →∞ . We note that (2) is not probabilistic at all, and every theoretical statement we will make in this paper holds deterministically. Furthermore, going beyond (2), we also seek to design flexible strategies to produce the sharpest prediction sets possible, which not only adapt to, but also anticipate distribution shifts.

We call our proposed solution conformal PID control . It treats the system for producing prediction sets as a proportional-integral-derivative (PID) controller. In the language of control, the prediction sets take a control variable , q t , and then produce a process variable , err t . We seek to anchor err t to a set point , α . To do so, we apply corrections to q t based on the error of the output, g t = err t -α . By reframing the problem in this language, we are able to build algorithms that have more stable coverage while also prospectively adapting to changes in the score sequence, much in the same style as a control system. See the diagram in Figure 1.

scorecaster

integrator

1.1 Peek at results: methods

Three design principles underlie our methods:

Quantile tracking (P control). Running online gradient descent on the quantile loss (summed over all past scores) gives rise to a method that we call quantile tracking , which achieves long-run coverage (2) under no assumptions except boundedness of the scores. This bound can be unknown. Unlike adaptive conformal inference (ACI) [16], quantile tracking does not return infinite sets after a sequence of miscoverage events. This can be seen as equivalent to proportional (P) control.

Error integration (I control). By incorporating the running sum ∑ t i =1 (err i -α ) of the coverage errors into the online quantile updates, we can further stabilize the coverage. This error integration scheme achieves long-run coverage (2) under no assumptions whatsoever on the scores (they can be unbounded). This can be seen as equivalent to integral (I) control.

Scorecasting (D control). To account for systematic trends in the scores-this may be due to aspects of the data distribution, fixed or changing, which are not captured by the initial forecaster-we train a second model, a scorecaster , to predict the quantile of the next score. While quantile tracking and error integration are merely reactive, scorecasting is forward-looking. It can residualize out systematic trends in the errors and lead to practical advantages in terms of coverage and efficiency (set sizes). This can be seen as equivalent to derivative (D) control. Traditional control theory would suggest using a linear approximation g ′ t = g t -g t -1 , but in our problem, we will typically choose more advanced scorecasting algorithms that go well beyond simple difference schemes.

These three modules combine to make our final iteration, the conformal PID controller :

In traditional PID control, one would take r t ( x ) to be a linear function of x . Here, we allow for nonlinearity and take r t to be a saturation function obeying

for constants b, c > 0 , and a sublinear, nonnegative, nondecreasing function h -we call a function h satisfying these conditions admissible . An example is the tangent integrator r t ( x ) = K I tan( x log( t ) / ( tC sat )) , where we set tan( x ) = sign( x ) · ∞ for x / ∈ [ -π/ 2 , π/ 2] , and C sat , K I > 0 are constants. The choice of integrator r t is a design decision for the user, as is the choice of scorecaster g ′ t .

We find it convenient to reparametrize (3), to produce a sequence of quantile estimates q t , t ∈ N used in the prediction sets (1), as follows:

Taking ˆ q t +1 = ηg t + g ′ t recovers (3), but we find it generally useful to instead consider the formulation in (5), which will be our main focus in the exposition henceforth. Now we view ˆ q t +1 as the scorecaster, which directly predicts q t +1 using past data. A main result of this paper, whose proof is given in Appendix A, is that the conformal PID controller (5) yields long-run coverage for any choice of integrator r t that satisfies the appropriate saturation condition, and any scorecaster ˆ q t +1 .

Theorem 1. Let { ˆ q t } t ∈ N be any sequence of numbers in [ -b/ 2 , b/ 2] and let { s t } t ∈ N be any sequence of score functions with outputs in [ -b/ 2 , b/ 2] . Here b > 0 , and may be infinite. Assume that r t satisfies (4) , for an admissible function h . Then the iterations in (5) achieve long-run coverage (2) .

To emphasize, this result holds deterministically, with no probabilistic model on the data ( x t , y t ) , t ∈ N . (Thus in the case that the sequence is random, the result holds for all realizations of the random variables.) As we will soon see, this theorem can be seen as a generalization of existing results in the online conformal literature.

1.2 Peek at results: experiments

COVID-19 death forecasting. To demonstrate conformal PID in practice, we consider 4-week-ahead forecasting of COVID-19 deaths in California, from late 2020 through late 2022. The base forecaster f t that we use is the ensemble model from the COVID-19 Forecast Hub, which is the model used for official CDC communications on COVID-19 forecasting [10, 29]. In this forecasting problem, at each time t we actually seek to predict the observed death count y t +4 at time t +4 .

Figure 2 shows the central 80% prediction sets from the Forecast Hub ensemble model on the left panel, and those from our conformal PID method on the right. We use a quantile conformal score function, as in conformalized quantile regression [30], applied asymmetrically (i.e., separately) to the lower and upper quantile levels). We use the tan integrator, with constants chosen heuristically (as described in Appendix C), and an ℓ 1 -regularized quantile regression as the scorecaster-in particular, the scorecasting model at time t predicts the quantile of the score at time t +4 based on all previous forecasts, cases, and deaths, from all 50 US states . The main takeaway is that conformal PID control is able to correct for consistent underprediction of deaths in the winter wave of late 2020/early 2021. We can see from the figure that the original ensemble fails to cover 8 times in a stretch of 10 weeks, resulting in a coverage of 20%; meanwhile, conformal PID only fails to cover 3 times during this stretch, restoring the coverage to 70% (recall the nominal level is 80%).

How is this possible? The ensemble is mainly comprised of constituent forecasters that ignore geographic dependencies between states [11] for the sake of simplicity or computational tractability. But COVID infections and deaths exhibit strong spatiotemporal dependencies, and most US states experienced the winter wave of late 2020/early 2021 at similar points in time. The scorecaster is thus able to learn from the mistakes made on other US states in order to prospectively adjust the ensemble's forecasts for the state of California. Similar improvements can be seen for other states, and we include experiments for New York and Texas as examples in Appendix F, which also gives more details on the scorecaster and the results.

Electricity demand forecasting. Next we consider a data set on electricity demand forecasting in New South Wales [18], which includes half-hourly data from May 7, 1996 to December 5, 1998.

1.0

50%

75%

100%

Coverage

Base Forecaster

Conformal PID Control

2020-10

2021-01

2021-04

2021-07

2021-10

2022-01

2022-04

2022-07

0

2000

4000

6000

Sets

2020-10

2021-01

2021-04

2021-07

2021-10

2022-01

2022-04

2022-07

0.0

0.2

0.4

0.6

0.8

Time

0.0

0.2

0.4

0.6

0.8

1.0

For the base forecaster we use a Transformer model [34] as implemented in darts [19]. This is only re-trained daily, to predict the entire day's demand in one batch; this is a standard approach with Transformer models due to their high computational cost. For the conformal score, we use the asymmetric (signed) residual score. We use the tan integrator as before, and we use a lightweight Theta model [2], re-trained at every time point (half-hour), as the scorecaster.

1.0

50%

75%

100%

Coverage

ACI

Conformal PID Control

1996-05-13

1996-05-17

1996-05-21

1996-05-25

1996-05-29

1996-06-01

1996-06-05

1996-06-09

1996-06-13

1996-06-17

0.0

0.5

1.0

Sets

1996-05-13

1996-05-17

1996-05-21

1996-05-25

1996-05-29

1996-06-01

1996-06-05

1996-06-09

1996-06-13

1996-06-17

0.0

0.2

0.4

0.6

0.8

Time

0.0

0.2

0.4

0.6

0.8

1.0

The results are shown in the right panel of Figure 3, where adaptive conformal inference (ACI) [16] is also compared in the left panel. In short, conformal PID control is able to anticipate intraday variations in the scores, and produces sets that 'hug' the ground truth sequence tightly; it achieves tight coverage without generating excessively large or infinite sets. The main reason why this is improved is that the scorecaster has a seasonality component built into its prediction model; in general, large improvements such as the one exhibited in Figure 3 should only be expected when the base forecaster is imperfect, as is the case here.

1.3 Related work

The adversarial online view of conformal prediction was pioneered by [16] in the same paper that first introduced ACI. Since then, there has been significant work towards improving ACI, primarily by setting the learning rate adaptively [17, 40, 7], and incorporating ideas from multicalibration to improve conditional coverage [5]. It is worth noting that [7] also makes the observation that the ACI iteration can be generalized to track the quantile of the score sequence, although their focus is on adaptive regret guarantees. Because the topic of adaptive learning rates for ACI and related algorithms has already been investigated heavily, we do not consider it in the current paper. Any such method, such as those of [17, 7] should work well in conjunction with our proposed algorithms.

A related but distinct line of work surrounds online calibration in the adversarial sequence model, which dates back to [14, 15], and connects in interesting ways to both game theory and online learning. We will not attempt to provide a comprehensive review of this rich and sizeable literature, but simply highlight [25, 24, 23] as a few interesting examples of recent work.

Lastly, outside the online setting, we note that several researchers have been interested in generalizing conformal prediction beyond the i.i.d. (or exchangeable) data setting: this includes [33, 28, 26, 12, 8], and for time series prediction, in particular, [9, 31, 38, 39, 3]. The focus of all of these papers is quite different, and they all rely on probabilistic assumptions on the data sequence to achieve validity; we include further discussion in Appendix B.

2 Methods

We describe the main components of our proposal one at a time, beginning with the quantile tracker.

2.1 Quantile tracking

The starting point for quantile tracking is to consider the following optimization problem:

for large T , where we abbreviate s t = s t ( x t , y t ) for the score of the test point, and ρ 1 -α denotes the quantile loss at the level 1 -α , i.e., ρ τ ( z ) = τ | z | for z > 0 and (1 -τ ) | z | for z ≤ 0 . The latter is the standard loss used in quantile regression [22, 21]. Problem (6) is thus a simple convex (linear) program that tracks the 1 -α quantile of the score sequence s t , t ∈ N . To see this, recall that for a continuously distributed random variable Z , the expected loss E [ ρ 1 -α ( Z -q )] is uniquely minimized (over q ∈ R ) at the level 1 -α quantile of the distribution of Z .

In the sequential setting, where we receive one score s t at a time, a natural and simple approach is to apply online gradient descent to (6), with a constant learning rate η > 0 . This results in the update: 2

where the second line follows as ∇ ρ 1 -α ( s t -q t ) = 1 -α if s t > q t ⇐⇒ err t = 1 , and ∇ ρ 1 -α ( s t -q t ) = -α if s t ≤ q t ⇐⇒ err t = 0 . Note that the update in (7) is highly intuitive: if we miscovered (committed an error) at the last iteration then we increase the quantile, whereas if we covered (did not commit an error) then we decrease the quantile.

Even though it is extremely simple, the quantile tracking iteration (7) can achieve long-run coverage own its own, provided the scores are bounded.

Proposition 1. Let { s t } t ∈ N be any sequence of numbers in [ -b, b ] , for 0 < b < ∞ . Then the quantile tracking iteration (7) satisfies

for any learning rate η > 0 and T ≥ 1 . In particular, (7) yields long-run coverage as in (2) .

A few remarks are in order. First, although Proposition 1 assumes boundedness of the scores, we do not need to know this bound in order to run (7) and obtain long-run coverage. As long as the scores lie in [ -b, b ] for any finite b , the guarantee goes through-clearly, the quantile tracker proceeds agnostically and performs the same updates in any case. Notably, the adaptive conformal inference algorithm can be expressed as a special case of the quantile tracker; see Appendix B.1 for details.

Second, for the learning rate, in practice we typically set η heuristically, as some fraction of the highest score over a trailing window ˆ B t = max { s t -∆+1 , . . . , s t } . On this scale, setting η = 0 . 1 ˆ B t usually gives good results, and we use it in all experiments unless specified otherwise (we also set the window length ∆ to be the same as the length of the burn-in period for training the initial base forecaster and scorecaster). 3 Extremely high learning rates result in volatile sets, while very low ones may fail to keep up with rapid changes in the score distribution.

2.2 Error integration

Error integration is a generalization of quantile tracking that follows the iteration:

where r t is a saturation function that satisfies (4) for an admissible function h ; recall that we use admissible to mean nonnegative, nondecreasing, and sublinear. As we saw in (13), the quantile tracker uses a constant threshold function h , whereas h is now permitted to grow, as long as it grows sublinearly, i.e., h ( t ) /t → 0 as t →∞ . Anon-constant threshold function h can be desirable because it means that r t will 'saturate' (will hit the conditions on the right-hand sides in (4)) less often, so corrections for coverage error will occur less often, and in this sense, a greater degree of coverage error can be tolerated along the sequence.

The next proposition, in particular its proof, makes the role of h precise. Importantly, Proposition 2 suffices to prove Theorem 1.

Proposition 2. Let { s t } t ∈ N be any sequence of numbers in [ -b, b ] , where b > 0 , and may be infinite. Assume that r t satisfies (4) , for an admissible function h . Then the error integration iteration (8) satisfies

for any T ≥ 1 , where c is the constant in (4) . In particular, this means (8) yields long-run coverage (2) .

The choice of saturation function essentially corresponds to a choice of adaptive learning rate; see Appendix D for details.

2.3 Scorecasting

The final piece to discuss is scorecasting. A scorecaster attempts to forecast q t +1 directly, taking advantage of any leftover signal that is not captured by the base forecaster. This is the role played by ˆ q t +1 in (5). Scorecasting may be particularly useful when it is difficult to modify or re-train the base forecaster. This can occur when the base forecaster is computationally costly to train (e.g., as in a Transformer model); or it can occur in complex operational prediction pipelines where frequently updating a forecasting implementation is infeasible. Another scenario where scorecasting may be useful is one in which the forecaster and scorecaster have access to different levels of data. For example, if a public health agency collects epidemic forecasts from external groups, and forms an ensemble forecast, then the agency may have access to finer-grained data that it can use to recalibrate the ensemble's prediction sets (compared to the level of data granularity granted to the forecasters originally).

This motivates the need for scorecasting as a modular layer that 'sits on top' of the base forecaster and residualizes out systematic errors in the score distribution. This intuition is made more precise

by recalling, as described above (following Proposition 2), that scorecasting combined with error integration as in (5) is just a reparameterization of error integration (8), where q ′ t = q t -ˆ q t and s ′ t = s t -ˆ q t are the new quantile and new score, respectively. A well-executed scorecaster could reduce the variability in the scores and make them more exchangeable, resulting in more stable coverage and tighter prediction sets, as seen in Figure 3. On the other hand, an aggressive scorecaster with little or no signal can actually hurt by adding variance to the new score sequence s ′ t , which could result in more volatile coverage and larger sets.

There is no limit to what we can choose for the scorecasting model. We might like to use a model that can simultaneously incorporate seasonality, trends, and exogenous covariates. Common choices would be SARIMA (seasonal autoregressive integrated moving average) and ETS (error-trendseasonality) models, but there are many other available methods, such as the Theta model [2], Prophet model [32], and neural network forecasters; see [20] for a review.

2.4 Putting it all together

Briefly, we revisit the PID perspective, to recap how quantile tracking, error integration, and scorecasting fit in and work in combination. It helps to return to (3), which we copy again here:

̸

Quantile tracking is precisely given by taking g ′ t = q t and r t = 0 . This can be seen as equivalent to P control: subtract q t from both sides in (10) and treat the increment u t +1 = q t +1 -q t as the process variable; then in this modified system, quantile tracking is exactly P control. For this reason, we use 'conformal P control' to refer to the quantile tracker in the experiments that follow. Similarly, we use 'conformal PI control' to refer to the choice g ′ t = q t , and r t = 0 as a generic integrator (for us, tan is the default). Lastly, 'conformal PID control' refers to letting g ′ t be a generic scorecaster, and r t = 0 be a generic integrator.

3 Experiments

In addition to the statewide COVID-19 death forecasting experiment described in the introduction, we run experiments on all combinations of the following data sets and forecasters.

Data sets:

Electricity demand [18]

Return (log price) of Amazon, Google, and Microsoft stock [27]

Temperature in Delhi [37]

Forecasters (all via darts [19]) :

Autoregressive (AR) model with 3 lags

Theta model with θ = 2 [2]

Prophet model [32]

Transformer model [34]

In all cases except for the COVID-19 forecasting data set, we: re-train the base forecaster at each time point; construct prediction sets using the asymmetric (signed) residual score; and use a Theta model for the scorecaster. For the COVID-19 forecasting setting, we: use the given ensemble model as the base forecaster (no training at all); construct prediction sets using the asymmetric quantile score; and use an ℓ 1 -penalized quantile regression as the scorecaster, fit on features derived from previous forecasts, cases, and deaths, as described in the introduction. And lastly, in all cases, we use a tan function for the integrator with constants chosen heuristically, as described in Appendx C.

The results that we choose to show in the subsections below are meant to illustrate key conceptual points (differences between the methods). Additional results are presented in Appendix G. Our GitHub repository, https://github.com/aangelopoulos/conformal-time-series , provides the full suite of evaluations.

3.1 ACI versus quantile tracking

We forecast the daily Amazon (AMZN) opening stock price from 2006-2014. We do this in log-space (hence predicting the return of the stock). Figure 4 compares ACI and the quantile tracker, each

̸

1.0

50%

75%

100%

Coverage

ACI

Conformal P Control

2008

2010

2012

2014

2016

2018

0

500

1000

Sets

2008

2010

2012

2014

2016

2018

0.0

0.2

0.4

0.6

0.8

Time

0.0

0.2

0.4

0.6

0.8

1.0

1.0

50%

75%

100%

Coverage

ACI

Conformal P Control

2008

2010

2012

2014

2016

2018

0

500

1000

Sets

2008

2010

2012

2014

2016

2018

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

Time

with its default learning rate: η = 0 . 005 for ACI, and η = 0 . 1 ˆ B t for quantile tracking. We see that the coverage from each method is decent, but oscillates nontrivially around the nominal level of 1 -α = 0 . 9 (with ACI generally having larger oscillations). Figure 5 thus increases the learning rate for each method: η = 0 . 1 for ACI, and η = 0 . 5 ˆ B t for the quantile tracker. We now see that both deliver very tight coverage. However, ACI does so by frequently returning infinite sets; meanwhile, the corrections to the sets made by the quantile tracker are nowhere near as aggressive.

As a final comparison, in Appendix E, we modify ACI to clip the sets in a way that disallows them from ever being infinite. This heuristic may be used by practitioners that want to guard against infinite sets, but it no longer has a validity guarantee for bounded or unbounded scores. The results in the appendix indicate that the quantile tracker has similar coverage to this procedure, and usually with smaller sets.

3.2 The effect of integration

Next we forecast the daily Google (GOOGL) opening stock price from 2006-2014 (again done in log-space). Figure 6 compares the quantile tracker without and without an additional integrator component (P control versus PI control). We purposely choose a very small learning rate, η = 0 . 01 ˆ B t , in order to show how the integrator can stabilize coverage, which it does nicely for most of the time series. The coverage of PI control begins to oscillate more towards the end of the sequence, which we attribute at least in part to the fact that the integrator measures coverage errors accumulated over all time -and by the end of a long sequence, the marginal coverage can still be close to 1 -α even if the local coverage deviates more wildly. This can be addressed by using a local version of the integrator, an idea we return to in the discussion.

50%

75%

100%

Coverage

Conformal P control

Conformal PI Control

2008

2010

2012

2014

2016

2018

500

1000

Sets

2008

2010

2012

2014

2016

2018

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Time

3.3 The effect of scorecasting

Figures 2 and 3 already showcase examples in which scorecasting offers significant improvement in coverage and set sizes. Recall that these were settings in which the base forecaster produces errors (scores) that have predictable trends. Further examples in the COVID-19 forecasting setting, which display similar benefits to scorecasting, are given in Appendix F.

We emphasize that it is not always the case that scorecasting will help. In some settings, scorecasting may introduce enough variance into the new score sequence that the coverage or sets will degrade in stability. (For example, this will happen if we run a highly complex scorecaster on a sequence of i.i.d. scores, where there are no trends whatsoever.) In practice, scorecasters should be designed with care, just as one would design a base forecaster; it is unlikely that using 'out of the box' techniques for scorecasting will be robust enough, especially in high-stakes problems. Appendix G provides examples in which scorecasting, run across all settings using a generic Theta model, can hurt (for example, it adds noticeable variance to the coverage and sets in some instances within the Amazon data setting).

4 Discussion and extensions

Discussion. Our work presents a framework for constructing prediction sets in time series that is analogous (and indeed formally equivalent) to PID control, consisting of quantile tracking (P control), which is simply online gradient descent applied to the quantile loss; error integration (I control) to stabilize coverage; and scorecasting (D control) to remove systematic trends in the scores (errors made by the base forecaster).

We found that the combination of quantile tracking and integration consistently yields robust and favorable performance in our experiments. Scorecasting provides additional benefits if there are trends left in scores that are predictable (and the scorecaster is well-designed), as is the case in some of our examples. Otherwise, scorecasting may add variability and make the coverage and prediction sets more volatile. Overall, designing the scorecaster (which includes the choice to even use one at all) is an important modeling step, just like the design of the base forecaster.

It is worth emphasizing that, with the exception of the COVID-19 forecasting example, our experiments are intended to be illustrative, and we did not look to use state-of-the-art forecasters, or include any and all possibly relevant features for prediction. Further, while we found that using heuristics to set constants (such as the learning rate η , and constants C sat , K I for the tan integrator) worked decently well, we believe that more rigorous techniques, along the lines of [17, 7], can be used to tune these adaptively in an online fashion.

Extensions. We now present an extension of our analysis to conformal risk control [1, 6, 13]. In this setting, we are given a sequence of loss functions L t : 2 Y ×Y → [0 , 1] satisfying L t ( Y , y ) = 0 for all y , and L t ( ∅ , y ) = 1 for all y . The goal is to bound the deviation of the average risk 1 T ∑ T t =1 L t ( C t , y t ) from α . We state a result for the integrator below, and give its proof in Appendix A.

Proposition 3. Consider the iteration q t +1 = r t ( ∑ t i =1 ( L i ( C i , y i ) -α )) , with L t as above. Assume that r t satisfies (4) , for an admissible function h . Also assume that C t ( C t , y t ) = ∅ if q t ≤ -b and Y if q t ≥ b , where b > 0 , and may be infinite. Then for all T ≥ 1 ,

for any T ≥ 1 , where c is the constant in (4) .

We briefly conclude by mentioning that we believe many other extensions are possible, especially with respect to the integrator. Broadly, we can choose to integrate in a kernel-weighted fashion,

As a special case, the kernel could simply assign weight 1 if t -i ≤ w , and weight 0 otherwise, which would result in an integrator that aggregates coverage over a trailing window of length w . This can help consistently sustain better local coverage, for long sequences. As another special case, the kernel could assign a weight based on whether x i and x t lie in the same bin in some pre-defined binning of X space, which may be useful for problems with group structure (where we want group-wise coverage). Various other choices and forms of kernels are possible, and it would be interesting to consider adding together a number of such choices (12) in combination, in a multi-resolution flavor, for the ultimate quantile update.

Acknowledgments

We would like to thank Tijana Zrni´ c, Amit Kohli, and Jordan Lekeufack for their valuable feedback. This work was supported by the National Science Foundation (NSF) Graduate Research Fellowship Program under grant no. 2146752, and the Office of Naval Research (ONR) Multi-University Research Initiative (MURI) Program under grant no. N00014-20-1-2787.

References

Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814 , 2022.

Vassilis Assimakopoulos and Konstantinos Nikolopoulos. The theta model: A decomposition approach to forecasting. International Journal of Forecasting , 16(4):521-530, 2000.

Andreas Auer, Martin Gauch, Daniel Klotz, and Sepp Hochreiter. Conformal prediction for time series with modern Hopfield networks. arXiv preprint arXiv:2303.12783 , 2023.

Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415 , 2022.

Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Practical adversarial multivalid conformal prediction. In Advances in Neural Information Processing Systems , 2022.

Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM , 68(6):1-34, 2021.

Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. arXiv preprint arXiv:2302.07869 , 2023.

Emmanuel J. Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. Journal of the Royal Statistical Society: Series B , 85(1):24-45, 2023.

Victor Chernozhukov, Kaspar Wuthrich, and Zhu Yinchu. Exact and robust conformal inference methods for predictive machine learning with dependent data. In Proceedings of the Annual Conference on Learning Theory , 2018.

Estee Y. Cramer, Yuxin Huang, Yijin Wang, Evan L. Ray, Matthew Cornell, Johannes Bracher, Andrea Brennen, Alvaro J. Castro Rivadeneira, Aaron Gerding, Katie House, Dasuni Jayawardena, Abdul Hannan Kanji, Ayush Khandelwal, Khoa Le, Vidhi Mody, Vrushti Mody, Jarad Niemi, Ariane Stark, Apurv Shah, Nutcha Wattanchit, Martha W. Zorn, Nicholas G. Reich, and US COVID-19 Forecast Hub Consortium. The United States COVID-19 Forecast Hub dataset. Scientific Data , 9, 2022.

Estee Y. Cramer, Evan L. Ray, Velma K. Lopez, Johannes Bracher, Andrea Brennen, Alvaro J. Castro Rivadeneira, Aaron Gerding, Tilmann Gneiting, Katie H. House, Yuxin Huang, Dasuni Jayawardena, Abdul H. Kanji, Ayush Khandelwal, Khoa Le, Anja Mühlemann, Jarad Niemi, Apurv Shah, Ariane Stark, Yijin Wang, Nutcha Wattanachit, Martha W. Zorn, Youyang Gu, Sansiddh Jain, Nayana Bannur, Ayush Deva, Mihir Kulkarni, Srujana Merugu, Alpan Raval, Siddhant Shingi, Avtansh Tiwari, Jerome White, Spencer Woody, Maytal Dahan, Spencer Fox, Kelly Gaither, Michael Lachmann, Lauren Ancel Meyers, James G. Scott, Mauricio Tec, Ajitesh Srivastava, et al. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proceedings of the National Academy of Sciences , 119(15):e2113561119, 2022.

Clara Fannjiang, Stephen Bates, Anastasios N. Angelopoulos, Jennifer Listgarten, and Michael I. Jordan. Conformal prediction under feedback covariate shift for biomolecular design. Proceedings of the National Academy of Sciences , 119(43):e2204569119, 2022.

Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving risk control in online learning settings. arXiv preprint arXiv:2205.09095 , 2022.

Dean P Foster. A proof of calibration via Blackwell's approachability theorem. Games and Economic Behavior , 29(1-2):73-78, 1999.

Dean P. Foster and Rakesh V. Vohra. Asymptotic calibration. Biometrika , 85(2):379-390, 1998.

Isaac Gibbs and Emmanuel J. Candès. Adaptive conformal inference under distribution shift. In Advances in Neural Information Processing Systems , 2021.

Isaac Gibbs and Emmanuel J. Candès. Conformal inference for online prediction with arbitrary distribution shifts. arXiv preprint arXiv:2208.08401 , 2022.

Michael Harries. Splice-2 comparative evaluation: Electricity pricing. Technical report, University of New South Wales, 1999.

Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, Maxime Dumonal, Jan Ko´ scisz, Dennis Bader, Frédérick Gusset, Mounir Benheddi, Camila Williamson, Michal Kosinski, Matej Petrik, and Gaël Grosch. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research , 23(1):5442-5447, 2022.

Rob J. Hyndman and George Athanasopoulos. Forecasting: Principles and Practice . OTexts, 2018.

Roger Koenker. Quantile Regression . Cambridge University Press, 2005.

Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica , 46(1):33-50, 1978.

Volodymyr Kuleshov and Shachi Deshpande. Online calibrated regression for adversarially robust forecasting. arXiv preprint arXiv:2302.12196 , 2023.

Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online against an adversary. In Association for the Advancement of Artificial Intelligence , 2017.

Volodymyr Kuleshov and Percy Liang. Calibrated structure prediction. In Advances in Neural Information Processing Systems , 2015.

Lihua Lei and Emmanuel J. Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society: Series B , 83(5):911-938, 2021.

Cam Nguyen. S&P 500 stock data. https://www.kaggle.com/datasets/camnugent/ sandp500 , 2018.

Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence , 2021.

Evan L. Ray, Logan C. Brooks, Jacob Bien, Matthew Biggerstaff, Nikos I. Bosse, Johannes Bracher, Estee Y. Cramer, Sebastian Funk, Aaron Gerding, Michael A. Johansson, Aaron Rumack, Yijin Wang, Martha Zorn, Ryan J. Tibshirani, and Nicholas G. Reich. Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States. International Journal of Forecasting , 39(3):1366-1383, 2023.

Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems , 2019.

Kamile Stankeviciute, Ahmed M. Alaa, and Mihaela van der Schaar. Conformal time-series forecasting. Advances in Neural Information Processing Systems , 2021.

Sean J. Taylor and Benjamin Letham. Forecasting at scale. The American Statistician , 72(1):3745, 2018.

Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World . Springer, 2005.

Vladimir Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In International Conference on Machine Learning , 1999.

Sumanth Vrao. Daily climate time series data. https://www.kaggle.com/datasets/ sumanthvrao/daily-climate-time-series-data , 2017.

Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning , 2021.

Chen Xu and Yao Xie. Sequential predictive conformal inference for time series. In International Conference on Machine Learning , 2023.

Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning , 2022.

A Proofs

We begin with the proof of Proposition 1. The proof is very simple, and we derive it as a corollary of Proposition 2, given next, because the proof reveals something perhaps unforeseen about the quantile tracker: it acts as an error integrator, despite only adjusting the quantile based on the most recent time step.

Proof of Proposition 1. Without a loss of generality, set q 1 = 0 . Unraveling the iteration (7) yields

For r t ( x ) = ηx , h ( t ) = b , we see (4) holds with c = 1 /η . Proposition 2 now applies.

Proof of Proposition 2. Abbreviate E T = ∑ T t =1 (err t -α ) . We will prove one side of the absolute inequality in (9), namely, E T ≤ c · h ( T ) + 1 , and the other side follows similarly. We use induction. The base case, for T = 1 , holds trivially. Now assume the result is true up to T -1 . We divide the argument into two cases: either c · h ( T -1) < E T -1 ≤ c · h ( T -1) + 1 or E T -1 ≤ c · h ( T -1) . In the first case, note that that (4) implies q T = r t ( E T -1 ) ≥ b and therefore s T ≤ q T and err t = 0 . This means that

as h is nondecreasing, which is the desired result at T . In the second case, we just use err T ≤ 1 , so

This again gives the desired result at T , and completes the proof.

Proof of Theorem 1. We can transform (5) by setting q ′ t +1 = q t +1 -ˆ q t +1 , and this becomes an update of the form (8) with respect to q ′ t +1 . Further, the score sequence in this new parameterization is s ′ t = s t -ˆ q t , which is in [ -b, b ] because both s t and ˆ q t are in [ -b/ 2 , b/ 2] . Applying Proposition 2 gives the result.

Proof of Proposition 3. The proof is similar to that of Proposition 2-as in that proof, we only prove one side of the absolute inequality (11), and use induction. Abbreviate E T = ∑ T t =1 ( L i ( C i , y i ) -α )) . The base case holds trivially. For the inductive step, either c · h ( T -1) < E T -1 ≤ c · h ( T -1) + 1 or E T -1 ≤ c · h ( T -1) . In the first case, we have saturated, so L t ( C t , y t ) = 0 , and

as h is nondecreasing, which is the desired result at T . In the second case, we just use the boundedness of the loss L t ( C t , y t ) ≤ 1 , so

This again gives the desired result at T

, and completes the proof.

B Comparisons to existing methods

This section contains descriptions of the relationship of our method to pre-existing methods for conformal prediction under distribution shift.

B.1 ACI as a special case

Though it may not be immediately obvious, adaptive conformal inference (ACI) is actually a special case of the quantile tracker. ACI follows the iteration:

which is equivalent to

for β t = inf { β : s t ≤ Quantile β ( { s 1 , . . . , s t -1 } ) . This shows that ACI is a particular instance of the quantile tracker that uses a secondary score s ′ t = β t and quantile q ′ t = 1 -α t . Thus, because quantile tracking (7) is the same as a linear coverage integrator (13), so is ACI.

We can see here that ACI transforms unbounded score sequences into bounded ones, which then implies long-run coverage for any score sequence. This may, however, come at a cost: ACI can sometimes output infinite or null prediction sets (when α t drifts below 0 or above 1, respectively). Direct quantile tracking on the scale of the original score sequence does not have this behavior.

B.2 Weighted conformal prediction

A line of work by [33] and [4] deals with variants of weighted conformal prediction. Specifically, the latter variant allows the incorporation of sample weights for each data point. When the data come from a probabilistic model, the sample weights allow for probabilistic bounds on the coverage of a new test point.

These methods, and others (e.g., [9]) in the probabilistic setting, differ greatly from the approach herein. When there is no probabilistic model, for example, all such approaches give vacuous coverage guarantees. The reason is that coverage on a new test point is impossible without a probabilistic model. However, in our setting we target long-run coverage, a completely different (and achievable) goal. Thus, neither the methods nor the proof techniques and theorem statements share any technical similarity to speak of.

For good measure, we include below in Appendix G evaluations against a version of [4] with a trailing window the size of the burn-in period. This approach generally performs worse than both our approach and ACI, and does not have guarantees in our setting, so we do not include it in main text.

B.3 Sequential predictive conformal inference (SPCI)

A recently proposed line of work exemplified by Ensemble Batch Prediction Intervals (EnbPI) [38] and its improved version, SPCI [39], seeks to provide coverage guarantees for black-box time-series predictors. However, the methods therein are not distribution-free, and SPCI rely on the following assumptions for validity:

The use of the quantile random forest (QRF) algorithm.

Model assumptions, such as consistency.

Distributional assumptions, such as smoothness.

Thus, the validity of SPCI is significantly weaker than the methods herein, perhaps along the lines of a standard nonparametric analysis. In contrast, our method has finite-sample validity averaged over time for any input sequence. However, it is possible to use the QRF/SPCI algorithm as the scorecaster, which would endow it with a distribution-free guarantee. We do not do this because the QRF is not a particularly effective time series forecasting method, and was outperformed by quantile LASSO (which we use for the COVID experiments) and the Theta model. The performance of the QRF tends to be 'blocky' and high-variance; for an example, see plots on the SPCI GitHub page.

However, we note that the EnbPI and SPCI algorithms can be seen as specific choices of scorecasters, and one consequence of our paper is that these algorithms can be endowed distribution-free guarantees when used as scorecasters in (5).

C Heuristics for setting constants

Consider the tan integrator r t ( x ) = K I tan( x log( t ) / ( tC sat )) , where we set tan( x ) = sign( x ) · ∞ whenever x / ∈ [ -π/ 2 , π/ 2] , and C sat , K I > 0 are constants. The constant C sat is primarily in charge of guaranteeing that by time T , we want to have an absolute guarantee of at least 1 -α -δ coverage.

Then we can set

to ensure the tan function has an asymptote at the correct point. The purpose of the constant K I is to place the integrator on the same scale as the scores. So if B ′ is a hypothesized bound on the magnitude of the scores, then one can set K I = B ′ . In practice, these heuristics can be taken as a starting place, and then the numbers can be fine-tuned during a burn-in period by hand or algorithmically. As alluded to previously, we believe there is room for work in the style of [17, 7] to rigorously tune these parameters online, but it is not the focus of our paper.

D Coverage integration as adaptive learning rate

As already mentioned in the introduction, in all our experiments we use a nonlinear saturation function r t ( x ) = K I tan( x log( t ) / ( tC sat )) , where we set tan( x ) = sign( x ) · ∞ for x / ∈ [ -π/ 2 , π/ 2] , and C sat , K I > 0 are constants that we choose heuristically (described in Appendix C). In a sense, this tan integrator is akin to a quantile tracker whose learning rate adapts to the current coverage gap. To see this, we can use a first-order Taylor approximation, which shows (ignoring constants):

Above, sec( x ) = 1 / cos( x ) is the secant function, which has a U-shape; thus we can see from the above that the effective learning rate is higher for larger errors. Similar analyses for different integrators will give different adaptive learning rates; see below for another example with a decaying learning rate.

Consider r t ( x ) = ηx/ √ t . (This will give long-run coverage only for bounded scores, because condition (4) is only met for finite and not infinite b .) Then (8) becomes q t +1 = η √ t ∑ t i =1 (err i -α ) , which can be rewritten as

This is approximately the quantile tracker (7) with a decaying learning rate, on the order of 1 / √ t .

E Comparison to clipped ACI

Figures 7 and 8 compare the quantile tracker to a clipped version of ACI which disallows infinite-sized sets by clipping the sets to the largest score seen so far.

F More details on COVID-19 forecasting

In this experiment, the scorecaster receives as input the three most recent scores (i.e., quantile errors) of the ensemble forecaster, as well as the three most recent case and death counts, from all 50 states . The scorecaster is an ℓ 1 -penalized quantile regression as implemented by sklearn.linear_model.QuantileRegressor . We fixed tuning parameter for the ℓ 1 penalty at 10; in our experience, the performance of the scorecaster was fairly robust to this choice. Automatic selection (e.g., using cross-validation) could be the topic of future study. Figures 9 and 10 shows the analogous experiments but for forecasting in New York and Texas.

G Further experiments

We give a more comprehensive view of our results, examing all data sets, and a range of tuning parameters for each method. We restrict our attention to AR as the base forecaster; for the rest of the base forecasters, we refer to the GitHub repository: https://github.com/aangelopoulos/ conformal-time-series .

1.0

50%

75%

100%

Coverage

ACI (clipped)

Conformal P Control

2008

2010

2012

2014

2016

2018

0

500

1000

Sets

2008

2010

2012

2014

2016

2018

0.0

0.2

0.4

0.6

0.8

Time

0.0

0.2

0.4

0.6

0.8

1.0

50%

75%

100%

Coverage

ACI (clipped)

Conformal P Control

2008

2010

2012

2014

2016

2018

0

500

1000

Sets

2008

2010

2012

2014

2016

2018

0.0

0.2

0.4

0.6

0.8

1.0

Time

0.0

0.2

0.4

0.6

0.8

1.0

Time

50%

75%

100%

Coverage

Base Forecaster

Conformal PID Control

2020-10

2021-01

2021-04

2021-07

2021-10

2022-01

2022-04

2022-07

0

1000

2000

Sets

2020-10

2021-01

2021-04

2021-07

2021-10

2022-01

2022-04

2022-07

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

1.0

50%

75%

100%

Coverage

Base Forecaster

Conformal PID Control

2020-10

2021-01

2021-04

2021-07

2021-10

2022-01

2022-04

2022-07

0

2000

4000

Sets

2020-10

2021-01

2021-04

2021-07

2021-10

2022-01

2022-04

2022-07

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

Time

1.0

Table 2: Summary statistics for electricity forecasting, as in Figure 3. Results for the Prophet and Theta models are not available because darts does not support intermittent retraining for these algorithms.

For each experiment, we describe the data set in a new subsection, and two plots are included: one for the coverage, and one for the prediction sets. Each column in the plots represents a different method, and each row is a different learning rate. For the quantile tracker, the learning rate is to be interpreted as the multiplier in front of ˆ B t . Each method is given a different color, which stays consistent throughout the plots. We use a tan integrator and a Theta scorecaster throughout, just as in the main text experiments. There is an additional method, 'Trail', corresponding to the proposal of Barber et al. [4] with a trailing window.

G.1 Amazon/Google

These data sets are part of a multivariate time series consisting of thirty blue-chip stock prices, including those of Amazon (AMZN) and Google (GOOGL), from January 1, 2006 to December 31, 2014. We attempt to forecast the daily opening price of each of Amazon and Google stock, on a log scale. Available to the scorecaster are the previous open prices of all 30 stocks .

G.2 Microsoft

This data set is a univariate time series consisting of a single stock open price, that of Microsoft (MSFT), from April 1, 2015 to May 31, 2021.

G.3 Daily temperature in Delhi

This data set contains the daily temperature (averaged over 8 measurements in 3 hour periods), humidity, wind speed, and atmospheric temperature in the city of Delhi from January 1, 2003 to April 24, 2017, scraped using the Weather Underground API.

G.4 Electricity demand forecasting

This data set measures electricity demand in New South Wales collected at half-hour increments from May 7th, 1996 to December 5th, 1998 (we zoom in on the first 2000 time points). There are also

several other variables collected, such as the demand and price in Victoria, the amount of energy transfer between New South Wales and Victoria, and so on. These are given as covariates to the scorecaster. The demand value is normalized by default to lie in [0 , 1] .

G.5 Synthetic data sets

We perform some experiments on two synthetic score sequences which include change points and other behaviors difficult to produce using real data. In this setting, there is no ground truth y t sequence, so we do not plot the sets. Instead, we plot the scores themselves in one column, and the quantiles q t produced by each algorithm in a different column (when q t ≥ s t , we cover). The general goal is for q t to track the 1 -α quantile of s t , and if it is too far off, that corresponds to the 'set being too large or too small' in a situation where we would be constructing sets out of these scores.

We consider an i.i.d. sequence of scores, a noisy increasing sequence of scores, and a mix of change points and trends. Our codebase describes the score generation procedure in more detail.

0.0

0.5

1.0

Trail

cvg=88.1%

ACI

Conformal P

lr=1, cvg=89.9%

Conformal PI

lr=1, cvg=89.9%

Conformal PID

lr=1, cvg=89.8%

0.0

0.5

1.0

lr=0.05, cvg=90.0%

lr=0.5, cvg=89.8%

lr=0.5, cvg=89.7%

lr=0.5, cvg=89.4%

0.0

0.5

1.0

lr=0.01, cvg=89.8%

0.0

0.5

1.0

lr=0.005, cvg=89.5%

lr=0.05, cvg=89.2%

lr=0.05, cvg=89.2%

lr=0.05, cvg=86.6%

1000

2000

3000

0.0

0.5

1.0

1000

2000

3000

lr=0.0001, cvg=80.1%

1000

2000

3000

lr=0, cvg=0.0%

1000

2000

3000

lr=0, cvg=87.9%

1000

2000

3000

lr=0, cvg=77.9%

Time

Coverage

500

1000

y

ground truth

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

500

1000

forecast

lr=0.05

lr=0.5

lr=0.5

lr=0.5

500

1000

lr=0.01

500

1000

lr=0.005

lr=0.05

lr=0.05

lr=0.05

0

1000

2000

3000

500

1000

0

1000

2000

3000

0

1000

2000

3000

lr=0.0001

0

1000

2000

3000

lr=0

0

1000

2000

3000

lr=0

0

1000

2000

3000

lr=0

Time

t

0.0

0.5

1.0

Trail

cvg=87.7%

ACI

Conformal P

lr=0.5, cvg=90.0%

Conformal PI

lr=0.5, cvg=89.9%

Conformal PID

lr=0.5, cvg=89.8%

0.0

0.5

1.0

lr=0.05, cvg=90.0%

0.0

0.5

1.0

lr=0.01, cvg=89.8%

lr=0.05, cvg=89.7%

lr=0.05, cvg=89.6%

lr=0.05, cvg=89.6%

0.0

0.5

1.0

lr=0.005, cvg=89.6%

lr=0.01, cvg=89.5%

lr=0.01, cvg=89.6%

lr=0.01, cvg=89.6%

1000

2000

3000

0.0

0.5

1.0

1000

2000

3000

lr=0.0001, cvg=87.1%

1000

2000

3000

lr=0, cvg=0.0%

1000

2000

3000

lr=0, cvg=89.6%

1000

2000

3000

lr=0, cvg=89.6%

Time

Coverage

500

1000

y

ground truth

Trail

ACI

Conformal P

lr=0.5

Conformal PI

lr=0.5

Conformal PID

lr=0.5

500

1000

forecast

lr=0.05

500

1000

lr=0.01

lr=0.05

lr=0.05

lr=0.05

500

1000

lr=0.005

lr=0.01

lr=0.01

lr=0.01

0

1000

2000

3000

500

1000

0

1000

2000

3000

0

1000

2000

3000

lr=0.0001

0

1000

2000

3000

lr=0

0

1000

2000

3000

lr=0

0

1000

2000

3000

lr=0

Time

t

0.0

0.5

1.0

Trail

cvg=87.5%

ACI

Conformal P

lr=1, cvg=90.0%

Conformal PI

lr=1, cvg=89.9%

Conformal PID

lr=1, cvg=89.9%

0.0

0.5

1.0

lr=0.05, cvg=89.9%

lr=0.5, cvg=89.9%

lr=0.5, cvg=89.6%

lr=0.5, cvg=89.6%

0.0

0.5

1.0

lr=0.01, cvg=89.8%

0.0

0.5

1.0

lr=0.005, cvg=89.7%

lr=0.05, cvg=89.8%

lr=0.05, cvg=89.8%

lr=0.05, cvg=89.7%

1000

2000

3000

0.0

0.5

1.0

1000

2000

3000

lr=0.0001, cvg=88.7%

1000

2000

3000

lr=0, cvg=0.0%

1000

2000

3000

lr=0, cvg=89.8%

1000

2000

3000

lr=0, cvg=89.8%

Time

Coverage

25

50

75

100

y

ground truth

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

25

50

75

100

forecast

lr=0.05

lr=0.5

lr=0.5

lr=0.5

25

50

75

100

lr=0.01

25

50

75

100

lr=0.005

lr=0.05

lr=0.05

lr=0.05

0

1000

2000

3000

25

50

75

100

0

1000

2000

3000

0

1000

2000

3000

lr=0.0001

0

1000

2000

3000

lr=0

0

1000

2000

3000

lr=0

0

1000

2000

3000

lr=0

Time

t

0.0

0.5

1.0

Trail

cvg=88.3%

ACI

Conformal P

lr=1, cvg=90.0%

Conformal PI

lr=1, cvg=90.0%

Conformal PID

lr=1, cvg=90.0%

0.0

0.5

1.0

lr=0.05, cvg=90.1%

lr=0.5, cvg=90.1%

lr=0.5, cvg=90.1%

lr=0.5, cvg=90.1%

0.0

0.5

1.0

lr=0.01, cvg=90.0%

0.0

0.5

1.0

lr=0.005, cvg=90.3%

lr=0.05, cvg=89.9%

lr=0.05, cvg=89.9%

lr=0.05, cvg=90.0%

1000

1200

1400

1600

0.0

0.5

1.0

1000

1200

1400

1600

lr=0.0001, cvg=92.5%

1000

1200

1400

1600

lr=0, cvg=0.0%

1000

1200

1400

1600

lr=0, cvg=70.8%

1000

1200

1400

1600

lr=0, cvg=76.5%

Time

Coverage

10

20

30

40

y

ground truth

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

10

20

30

40

forecast

lr=0.05

lr=0.5

lr=0.5

lr=0.5

10

20

30

40

lr=0.01

10

20

30

40

lr=0.005

lr=0.05

lr=0.05

lr=0.05

0

500

1000

10

20

30

40

0

500

1000

0

500

1000

lr=0.0001

0

500

1000

lr=0

0

500

1000

lr=0

0

500

1000

lr=0

Time

t

0.0

0.5

1.0

Trail

cvg=90.5%

ACI

Conformal P

lr=1, cvg=90.4%

Conformal PI

lr=1, cvg=90.0%

Conformal PID

lr=1, cvg=90.1%

0.0

0.5

1.0

lr=0.05, cvg=90.1%

lr=0.5, cvg=90.1%

lr=0.5, cvg=90.1%

lr=0.5, cvg=90.2%

0.0

0.5

1.0

lr=0.01, cvg=90.1%

0.0

0.5

1.0

lr=0.005, cvg=90.1%

lr=0.05, cvg=89.8%

lr=0.05, cvg=89.5%

lr=0.05, cvg=90.0%

1000

1500

2000

0.0

0.5

1.0

1000

1500

2000

lr=0.0001, cvg=93.8%

1000

1500

2000

lr=0, cvg=0.0%

1000

1500

2000

lr=0, cvg=86.5%

1000

1500

2000

lr=0, cvg=89.2%

Time

Coverage

0.25

0.50

0.75

1.00

y

ground truth

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

0.25

0.50

0.75

1.00

forecast

lr=0.05

lr=0.5

lr=0.5

lr=0.5

0.25

0.50

0.75

1.00

lr=0.01

0.25

0.50

0.75

1.00

lr=0.005

lr=0.05

lr=0.05

lr=0.05

0

500

1000

1500

0.25

0.50

0.75

1.00

0

500

1000

1500

0

500

1000

1500

lr=0.0001

0

500

1000

1500

lr=0

0

500

1000

1500

lr=0

0

500

1000

1500

lr=0

Time

t

0.0

0.5

1.0

Trail

cvg=90.0%

ACI

Conformal P

lr=1, cvg=89.9%

Conformal PI

lr=1, cvg=90.0%

Conformal PID

lr=1, cvg=89.2%

0.0

0.5

1.0

lr=0.05, cvg=90.4%

lr=0.5, cvg=90.1%

lr=0.5, cvg=91.2%

lr=0.5, cvg=91.0%

0.0

0.5

1.0

lr=0.01, cvg=90.0%

0.0

0.5

1.0

lr=0.005, cvg=89.9%

lr=0.05, cvg=90.5%

lr=0.05, cvg=90.8%

lr=0.05, cvg=91.2%

250

500

750

1000

0.0

0.5

1.0

250

500

750

1000

lr=0.0001, cvg=89.3%

250

500

750

1000

lr=0, cvg=0.0%

250

500

750

1000

lr=0, cvg=86.9%

250

500

750

1000

lr=0, cvg=93.2%

Time

Coverage

0.4

0.6

Scores

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

0.4

0.6

lr=0.05

lr=0.5

lr=0.5

lr=0.5

0.4

0.6

lr=0.01

0.4

0.6

lr=0.005

lr=0.05

lr=0.05

lr=0.05

960

980

1000

0.4

0.6

960

980

1000

960

980

1000

lr=0.0001

960

980

1000

lr=0

960

980

1000

lr=0

960

980

1000

lr=0

Time

qt

0.0

0.5

Trail

cvg=0.9%

ACI

Conformal P

lr=1, cvg=89.2%

Conformal PI

lr=1, cvg=89.4%

Conformal PID

lr=1, cvg=90.3%

0.0

0.5

lr=0.05, cvg=89.8%

lr=0.5, cvg=88.2%

lr=0.5, cvg=88.9%

lr=0.5, cvg=90.4%

0.0

0.5

lr=0.01, cvg=88.9%

0.0

0.5

lr=0.005, cvg=87.9%

lr=0.05, cvg=73.5%

lr=0.05, cvg=84.6%

lr=0.05, cvg=90.1%

250

500

750

1000

0.0

0.5

250

500

750

1000

lr=0.0001, cvg=1.0%

250

500

750

1000

lr=0, cvg=0.0%

250

500

750

1000

lr=0, cvg=82.2%

250

500

750

1000

lr=0, cvg=90.1%

Time

Coverage

0.9

1.0

1.1

Scores

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

0.9

1.0

1.1

lr=0.05

lr=0.5

lr=0.5

lr=0.5

0.9

1.0

1.1

lr=0.01

0.9

1.0

1.1

lr=0.005

lr=0.05

lr=0.05

lr=0.05

960

980

1000

0.9

1.0

1.1

960

980

1000

960

980

1000

lr=0.0001

960

980

1000

lr=0

960

980

1000

lr=0

960

980

1000

lr=0

Time

qt

0.0

0.5

1.0

Trail

cvg=91.0%

ACI

Conformal P

lr=1, cvg=91.8%

Conformal PI

lr=1, cvg=91.4%

Conformal PID

lr=1, cvg=91.7%

0.0

0.5

1.0

lr=0.05, cvg=92.0%

lr=0.5, cvg=92.4%

lr=0.5, cvg=92.1%

lr=0.5, cvg=92.3%

0.0

0.5

1.0

lr=0.01, cvg=93.1%

0.0

0.5

1.0

lr=0.005, cvg=92.7%

lr=0.05, cvg=92.2%

lr=0.05, cvg=92.7%

lr=0.05, cvg=92.2%

2000

3000

4000

0.0

0.5

1.0

2000

3000

4000

lr=0.0001, cvg=87.6%

2000

3000

4000

lr=0, cvg=20.9%

2000

3000

4000

lr=0, cvg=86.5%

2000

3000

4000

lr=0, cvg=90.3%

Time

Coverage

2

0

2

Scores

Trail

ACI

Conformal P

lr=1

Conformal PI

lr=1

Conformal PID

lr=1

2

0

2

lr=0.05

lr=0.5

lr=0.5

lr=0.5

2

0

2

lr=0.01

2

0

2

lr=0.005

lr=0.05

lr=0.05

lr=0.05

3960

3980

4000

2

0

2

3960

3980

4000

3960

3980

4000

lr=0.0001

3960

3980

4000

lr=0

3960

3980

4000

lr=0

3960

3980

4000

lr=0

Time

qt

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an adaptation to existing adaptive conformal prediction in two novel ways: (i) by tracking the quantile via online regression over the running sum of the errors; and (ii) by incorporating a second model to anticipate the quantile in the next instant (\"scorecasting\"). The authors include theoretical guarantees for the resulting method, as well as comparisons against a benchmark in the field (ACI, which is shown to be a a special case of the proposed algorithm) over many different datasets.\n\nSTRENGTHS:\n- Clear: the paper is well-written and very clear regarding its contributions.\n- Limitations: the discussions on the method's limitations are very helpful.\n- Theoretical results: the mathematical proofs are easy to follow and the notation is clear.\n- Empirical results: the empirical studies are extensive and cover several different datasets. The experiments in the supplementary material are also very thorough.\n- Code: the code is well-organized and readable (but please include the `environments.yml` file, all the necessary datasets, and instructions on how to reproduce figures in the paper! See below).\n\nWEAKNESSES:\n- Theoretical guarantees: the guarantees of the proposed method are only asymptotic. Unfortunately, no finite-sample results are included. None of the theoretical developments in the paper showcase advantages that the proposed method has over leading alternatives. \n- Baselines: there are many recent proposals in the field (even in your related works section), but the only baseline is ACI. SPCI (Xu and Xie, 2022) also obtained asymptotic longitudinal coverage of time series without creating infinitely-wide intervals; can you underline in what ways your proposal is better? \n- Experimental results: Can you include more details on how the hyperparameters were chosen for the examples in the main paper (for example, $r_t$)? Having too many hyper parameters can be overwhelming to the end-user (I appreciate the discussion in appendices C and D!). It would be great to include at least one example of adaptive learning in the proposed algorithm. \n- Scalability: all the time series considered, both in the main text and in the supplementary material (SM), are small, with the largest ones comprising around 3000 data points. In scenarios where new data arrives every second, for example, it is not clear if the method would work as-is or if any modifications would be needed. A discussion about this and an experiment with a very long time series would be great.\n- References: the authors point out the field of online calibration, and mention connections to game theory and online learning, but never explicitly discuss the connection to their work. It would be nice to better understand what tools from the field you are using or building upon.\n- Relationship to control theory: seems slightly out-of-place. Either explain the connection better (perhaps in the SM) or develop it in another paper? As it is, it's hard for the CP audience understand what the PID is, or why this connection is relevant.\n- Reproducibility: adding the proposed method's pseudocode would help a lot, either to the main text or to the SM if space is lacking. Trying to reproduce the results following the README.md, I was instructed to install the dependencies listed on `environment.yml`, but no such file was provided and I had to track them down one by one. There also seems to be a dataset missing (`datasets/preds_cases.csv`) and not all plots from the paper were generated.\n- Minor writing suggestion: on one hand, the relation to control theory seems currently very incipient and more of a curiosity, but it is presented prominently in a section of its own. On the other hand, the interesting Proposition 3 is cramped in the final section. You might consider adding an independent section for extensions and incorporate the control theory analogy into the final discussion section.\n- Other minor suggestions: I don't see the point of including the zoom in the figures; it oftentimes hides parts of the figure and the underlying patterns are clear from the whole graph. Besides the many plots, adding a table with a summary of the results among all datasets could help a reader to draw conclusions faster. The meaning of sublinear function is given in line 164, but the term is used before in lines 55 and 67, so we suggest bringing the definition up. Display (5) has a floating \\cdot in the last inequality and inconsistent usage (c \\cdot h(t) vs -ch(t)). Figure 1 caption says prediction sets are in purple, but it is actually in orange.\n- Typos: line 115: $(1-\\alpha)$ should be $\\tau$; line 139: the em-dash is confusing.; line 151: \"an particular\" should be \"a particular; proof of proposition 2: $C$ should be $c$; display under line 192: the second $\\text{err}_i$ should be $\\text{err}_t$; line 201: \"can made\" should be \"can be made\"; line 295: \"saturation function as in 2\" should be \"saturation function that satisfies (5)\" or \"saturation function as in Proposition 2\"line 301: missing \\mathrm for err_i.; the bibliography is not printed in the SM and the references do not match the main text, so it ca be confusing.\n\nQUESTIONS:\n- Why is asymptotic coverage the best one can hope for in the adversarial setting? Is this a formal result? It would be helpful to develop this idea a bit more in the text.\n- What are advantages of using the scorecaster in the conformal prediction setup vs adapting the model instead?\n- Any guidance on how should one pick $r_t$ and the learning rate? How was this decided for the figures in the main paper? The paper seems to imply trial and error. The impact on figures in the appendix is extremely large!\n- When do you expect ACI to be better than Quantile+Integrator+Scorecaster?\n- What are advantages and disadvantages of your proposal vs SPCI, given both achieve coverage of time series without creating infinitely-wide intervals? How does the empirical performance compares? In what ways is your proposal better?\n- Can you explain what you mean by \"Our algorithms may not be universally useful on time series different from those we used for evaluation\" (line 283)? When should it be useful?\n- There is a method called \"Trail\" in the appendix plots, but no explanation. What is it?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of conformal UQ under distribution shifts. This is a very realistic setting where the upstream prediction model cannot be frequently updated, and we still want to certify some level of safety. The paper cleverly fames the problem as a PID control problem, and provides a practical algorithm that they show is effective on synthetic and real datasets.\n\nSTRENGTHS:\n- The setting addressed is impactful and practical. A major hindrance of using CP in ML systems deployment is that the exchangeability assumption is often violated. This work provides a practical adaptation while still maintaining some guarantees of the method. \n\n- The PID framing of the problem & solution is clever and insightful. (very Big Brain, if i may use gen-z lingo.) The two Gibbs & Candes paper from 2021 and 2022 for CP under distribution shifts was able to circumvent the exchangeability assumption and achieve some guarantees by transforming the distribution shift into a online optimization problem for a single parameter $\\alpha^*$, which they prove is similar to a P controller.  This work takes the idea a step further, and introduces the integrater (I) and forecasting (D) algorithms, completing the picture (fig 2 <- such a nice figure). \n\n- The method can utilize time-series modeling approaches to adapt to seasonality. \n\n- The experiment section is well-presented. It walks you through part-by-part of the PID controller and shows the impact of each. The examples chosen are clear and convincing.\n\nWEAKNESSES:\n1. The paper did not define some concepts it used in its writing. For example: \n- line 29, exchangeability \n- line 25, sharpest\n- line 83, burn-in period\n- line 137 (not really a definition just... what are you trying to say here?) \"the quantile tracker proceeds agnostically and performs the same updates in any case.\"\n\nI think explaining them, even just in the appendix, will help with clarity. \n\n2. The paper only compared to ACI & ablations of its own method. A common choice of algorithm for the scenario introduced by the authors would be Enbpi and SPCI from Xu & Xie (references 29 and 30 in the paper).\n\nQUESTIONS:\n- Can you elaborate on the reason why Enbpi and SPCI from Xu & Xie are not good baselines for comparing your model? Many have used previous timesteps as covariate X, similar to your scorecasting approach.\n\nLIMITATIONS:\nThe authors provided a good address of limitations in their discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the issue of how to parameterize conformal prediction in the time series setting. It argues that standard conformal inference methods would not provide valid inferences in the sequential setting, which lacks exchangeability. To resolve this, the paper suggests using online quantile tracking, integrating errors for stabilizing coverage, and utilizing scorecasting in the presence of systematic trends. The authors establish the corresponding theoretical properties, such as asymptotic coverage and conduct several experiments to justify their approach.\n\nSTRENGTHS:\n1. A new method for conformal prediction in the in the time series setting with a clear presentation of the details and background.\n2. The authors show the proposed method has a major practical benefit, that unlike ACI (Gibbs and Candes, 2021), no infinite sets are produced.\n3. The authors demonstrate the practical utility of their methods with an extensive series of experiments on a wide range of real-time series data sets.\n\nWEAKNESSES:\n1. Although this paper present a general theorem that their method achieves asymptotic coverage, there is insufficient discussion on the coverage guarantees such as the lower bounds and upper bounds on coverage.\n2. The authors also explicitly avoid addressing the choice of automatic algorithms for the\ntwo parameter $C_{sat}$ and $K_I$ in the nonlinear saturation function.\n3. In the sequential setting (7), the authors didn’t show how to select the learning rate\n$\\eta$, what factors might affect the selection of the learning rate $\\eta$, and what impact this\nselection have on the performance of the method.\n\nQUESTIONS:\n1. How the specific selection of the function $r_t$ and $\\hat{q}_t$ influence the\noverall performance of the proposed method in conformal prediction for time series?\n2. In the context of time series conformal prediction, how does this new method compare\nwith other popular methods in terms of robustness, and computational efficiency?\n3. How does a non-constant threshold function $h$ help with tolerating a greater degree of\ncoverage error throughout the sequence?\n\nLIMITATIONS:\n1. Despite demonstrating strong results across various experiments, the authors acknowledge\npotential limitations, including the challenge of proving asymptotic coverage, potential\ninapplicability to different time series, and limitations in scorecasting.\n2. The methods they develop are specific to cases where response $y_t$ is revealed at each time\npoint. However, there are many settings in which we receive the response in a delayed\nfashion or in large batches.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to provide asymptotically valid conformal prediction (CP) regions for the time series prediction problem. Similar to the adaptive conformal inference (ACI) framework [11], the main idea is to choose the appropriate quantile of the non-conformity scores (or equivalently tuning the miscoverage rate $\\alpha$) at each time step to fulfill the required error rate. The framework performs an online quantile update while taking into account the sum of coverage errors overtime to stabilize the coverage better. They provided theoretical analysis that guarantees the asymptotic coverage of the predictions made by their framework. Numerical experiments on real datasets show that their method brings more stable and efficient prediction sets than ACI.\n\nSTRENGTHS:\nIn general, I liked the idea of designing a PID-like framework. Thanks to this structure, the prediction regions turn out to be more stable and efficient. Moreover, it ensures that their framework never outputs infinite size prediction sets, which is a big advantage over ACI. Also, they provided many numerical experiments in the paper and the Appendix to illustrate the power of their proposed framework highlighting the effect of each block and different choices of hyperparameters.\n\nWEAKNESSES:\n1. The related work section is not well-written in general. Also, it does not represent the works in the intersection of conformal prediction and time series prediction in a fair manner. Indeed many good papers are missing, such as \n\t- Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415, 2022,\n\t- Sun, S. and Yu, R. Copula conformal prediction for multi-step time series forecasting. ArXiv, abs/2212.03281, 2022,\nto mention a few. \n\n2. The comparisons are limited to ACI. Though it is a natural baseline to compare with, many other CP frameworks are designed for the time series prediction problem. It is interesting to see how they behave compared to the methods proposed in this paper. \n\n3. The scorecasting block seems unnecessary and, to some extent, redundant. Accounting for a wrong choice of learning model (e.g., a model that does not account for the seasonality effect) can not be a part of the design of a CP framework. This point is also mentioned in the paper in lines 286-288. However, it seems that the authors tried to put this block to force the similarity of their framework and the (well-known) PID controller.  \n\n4. There are numerous design choices (such as the saturation function) and hyperparameters in the framework, and it's worth noting that several of them may not be immediately intuitive for people in the Machine Learning or Conformal Prediction community when it comes to making appropriate choices.\n\n5. Generally speaking, I am still unsure to what extent it is worth exploring frameworks with an asymptotic coverage property, while the popularity of conformal prediction lies in its ability to provide a finite-sample guarantee.\n\n6. Minor Comments:\n\t- There are some typos in Equation 5. \n\n\t- Line115: There is a typo in the definition of the pinball loss (\\tau should be replaced with \\alpha). \n\n\t- Line 178: The equation reference should be (10) instead of (11).  \n\n\t- In the proof of Proposition 2., the capital C has to be replaced with the small c. \n\n\t- Bringing the extensions of the proposed method (e.g., conformal risk control and Proposition 3. or conditional coverage) into the discussion section is unappealing.\n\t- In order to facilitate the comparisons for the readers,  it is a good idea to report the Avg. size of the prediction sets and Avg. coverage in the figures like what you have in the Appendix figures.\n\nQUESTIONS:\nIsn't it a good idea to provide a concrete explanation of why the proposed framework never returns infinite sets while this can happen in ACI?\n\nLIMITATIONS:\nThe authors fairly mention the limitations of the proposed framework in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}