SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Emmanuel J. Candès
Stanford University candes@stanford.edu
We study the problem of uncertainty quantification for time series prediction, with the goal of providing easy-to-use algorithms with formal guarantees. The algorithms we present build upon ideas from conformal prediction and control theory, are able to prospectively model conformal scores in an online setting, and adapt to the presence of systematic errors due to seasonality, trends, and general distribution shifts. Our theory both simplifies and strengthens existing analyses in online conformal prediction. Experiments on 4-week-ahead forecasting of statewide COVID-19 death counts in the U.S. show an improvement in coverage over the ensemble forecaster used in official CDC communications. We also run experiments on predicting electricity demand, market returns, and temperature using autoregressive, Theta, Prophet, and Transformer models. We provide an extendable codebase for testing our methods and for the integration of new algorithms, data sets, and forecasting rules. $^{1}$
Machine learning models run in production systems regularly encounter data distributions that change over time. This can be due to factors such as seasonality and time-of-day, continual updating and retraining of upstream machine learning models, changing user behaviors, and so on. These distribution shifts can degrade a model's predictive performance. They also invalidate standard techniques for uncertainty quantification, such as conformal prediction [36, 35].
To address the problem of shifting distributions, we consider the task of prediction in an adversarial online setting, as in [16]. In this problem setting, we observe a (potentially) adversarial time series of deterministic covariates  $x_{t} \in \mathcal{X}$  and responses  $y_{t} \in \mathcal{Y}$ , for  $t \in \mathbb{N} = \{1,2,3,\ldots\}$ . As in standard conformal prediction, we are free to define any conformal score function  $s_t: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ , which we can view as measuring the accuracy of our forecast at time  $t$ . We will assume with a loss of generality that  $s_t$  is negatively oriented (lower values mean greater forecast accuracy). For example, we may use the absolute error  $s_t(x,y) = |y - f_t(x)|$ , where  $f_{t}$  is a forecaster trained on data up to but not including data at time  $t$ .
The challenge in the sequential setting is as follows. We seek to invert the score function to construct a conformal prediction set,
$$
\mathcal {C} _ {t} = \{y \in \mathcal {Y}: s _ {t} \left(x _ {t}, y\right) \leq q _ {t} \}, \tag {1}
$$
where  $q_{t}$  is an estimated  $1 - \alpha$  quantile for the distribution of the score  $s_{t}(x_{t},y_{t})$  at time  $t$ . In standard conformal prediction, we would take  $q_{t}$  to be a level  $1 - \alpha$  sample quantile (up to a finite-sample correction) of  $s_{t}(x_{i},y_{i}), i < t$ ; if the data sequence  $(x_{i},y_{i}), i\in \mathbb{N}$  were i.i.d. or exchangeable, then this would yield  $1 - \alpha$  coverage [35] at each time  $t$ . However, in the sequential setting, which does not assume exchangeability (or any probabilistic model for the data for that matter), choosing  $q_{t}$  in (1) to yield coverage is a formidable task. In fact, if we are not willing to make any assumptions about the data sequence, then a coverage guarantee at time  $t$  would only be possible with trivial methods, which construct prediction intervals of infinite sizes.
<sup>1</sup>http://github.com/aangelopoulos/conformal-time-series
Therefore, our goal is to achieve long-run coverage in time. That is, letting  $\mathrm{err}_t = \mathbb{1}\{y_t \notin \mathcal{C}_t\}$ , we would like to achieve, for large integers  $T$ ,
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {e r r} _ {t} = \alpha + o (1) \tag {2}
$$
under few or no assumptions, where  $o(1)$  denotes a quantity that tends to zero as  $T \to \infty$ . We note that (2) is not probabilistic at all, and every theoretical statement we will make in this paper holds deterministically. Furthermore, going beyond (2), we also seek to design flexible strategies to produce the sharpest prediction sets possible, which not only adapt to, but also anticipate distribution shifts.
We call our proposed solution conformal PID control. It treats the system for producing prediction sets as a proportional-integral-derivative (PID) controller. In the language of control, the prediction sets take a control variable,  $q_{t}$ , and then produce a process variable,  $\mathrm{err}_t$ . We seek to anchor  $\mathrm{err}_t$  to a set point,  $\alpha$ . To do so, we apply corrections to  $q_{t}$  based on the error of the output,  $g_{t} = \mathrm{err}_{t} - \alpha$ . By reframing the problem in this language, we are able to build algorithms that have more stable coverage while also prospectively adapting to changes in the score sequence, much in the same style as a control system. See the diagram in Figure 1.
Figure 1: Conformal PID Control, expressed as a block diagram.
1.1 Peek at results: methods
Three design principles underlie our methods:
1. Quantile tracking (P control). Running online gradient descent on the quantile loss (summed over all past scores) gives rise to a method that we call quantile tracking, which achieves long-run coverage (2) under no assumptions except boundedness of the scores. This bound can be unknown. Unlike adaptive conformal inference (ACI) [16], quantile tracking does not return infinite sets after a sequence of miscoverage events. This can be seen as equivalent to proportional (P) control.
2. Error integration (I control). By incorporating the running sum  $\sum_{i=1}^{t} (\mathrm{err}_i - \alpha)$  of the coverage errors into the online quantile updates, we can further stabilize the coverage. This error integration scheme achieves long-run coverage (2) under no assumptions whatsoever on the scores (they can be unbounded). This can be seen as equivalent to integral (I) control.
3. Scorecasting (D control). To account for systematic trends in the scores—this may be due to aspects of the data distribution, fixed or changing, which are not captured by the initial forecaster—we train a second model, a scorer, to predict the quantile of the next score. While quantile tracking and error integration are merely reactive, scoring is forward-looking. It can residualize out systematic trends in the errors and lead to practical advantages in terms of coverage and efficiency (set sizes). This can be seen as equivalent to derivative (D) control. Traditional control theory would suggest using a linear approximation  $g_{t}^{\prime} = g_{t} - g_{t-1}$ , but in our problem, we will typically choose more advanced scoring algorithms that go well beyond simple difference schemes.
These three modules combine to make our final iteration, the conformal PID controller:
$$
q _ {t + 1} = \underbrace {\eta g _ {t}} _ {\mathrm {P}} + \underbrace {r _ {t} \left(\sum_ {i = 1} ^ {t} g _ {t}\right)} _ {\mathrm {I}} + \underbrace {g _ {t} ^ {\prime}} _ {\mathrm {D}}. \tag {3}
$$
In traditional PID control, one would take  $r_t(x)$  to be a linear function of  $x$ . Here, we allow for nonlinearity and take  $r_t$  to be a saturation function obeying
$$
x \geq c \cdot h (t) \Rightarrow r _ {t} (x) \geq b, \quad \text {a n d} \quad x \leq - c \cdot h (t) \Longrightarrow r _ {t} (x) \leq - b, \tag {4}
$$
for constants  $b, c > 0$ , and a sublinear, nonnegative, nondecreasing function  $h$  we call a function  $h$  satisfying these conditions admissible. An example is the tangent integrator  $r_t(x) = K_{\mathrm{I}} \tan (x \log(t) / (t C_{\mathrm{sat}}))$ , where we set  $\tan(x) = \operatorname{sign}(x) \cdot \infty$  for  $x \notin [-\pi/2, \pi/2]$ , and  $C_{\mathrm{sat}}, K_{\mathrm{I}} > 0$  are constants. The choice of integrator  $r_t$  is a design decision for the user, as is the choice of scorer  $g_t'$ .
We find it convenient to reparametrize (3), to produce a sequence of quantile estimates  $q_{t}$ ,  $t \in \mathbb{N}$  used in the prediction sets (1), as follows:
let  $\hat{q}_{t + 1}$  be any function of the past:  $x_{i},y_{i},q_{i}$  , for  $i\leq t$
$$
\text {t h e n} q _ {t + 1} = \hat {q} _ {t + 1} + r _ {t} \left(\sum_ {i = 1} ^ {t} \left(\operatorname {e r r} _ {i} - \alpha\right)\right). \tag {5}
$$
Taking  $\hat{q}_{t + 1} = \eta g_t + g_t'$  recovers (3), but we find it generally useful to instead consider the formulation in (5), which will be our main focus in the exposition henceforth. Now we view  $\hat{q}_{t + 1}$  as the scorer, which directly predicts  $q_{t + 1}$  using past data. A main result of this paper, whose proof is given in Appendix A, is that the conformal PID controller (5) yields long-run coverage for any choice of integrator  $r_t$  that satisfies the appropriate saturation condition, and any scorer  $\hat{q}_{t + 1}$ .
Theorem 1. Let  $\{\hat{q}_t\}_{t\in \mathbb{N}}$  be any sequence of numbers in  $[-b / 2,b / 2]$  and let  $\{s_t\}_{t\in \mathbb{N}}$  be any sequence of score functions with outputs in  $[-b / 2,b / 2]$ . Here  $b > 0$ , and may be infinite. Assume that  $r_t$  satisfies (4), for an admissible function  $h$ . Then the iterations in (5) achieve long-run coverage (2).
To emphasize, this result holds deterministically, with no probabilistic model on the data  $(x_{t},y_{t})$ ,  $t\in \mathbb{N}$ . (Thus in the case that the sequence is random, the result holds for all realizations of the random variables.) As we will soon see, this theorem can be seen as a generalization of existing results in the online conformal literature.
1.2 Peek at results: experiments
COVID-19 death forecasting. To demonstrate conformal PID in practice, we consider 4-week-ahead forecasting of COVID-19 deaths in California, from late 2020 through late 2022. The base forecaster  $f_{t}$  that we use is the ensemble model from the COVID-19 Forecast Hub, which is the model used for official CDC communications on COVID-19 forecasting [10, 29]. In this forecasting problem, at each time  $t$  we actually seek to predict the observed death count  $y_{t+4}$  at time  $t + 4$ .
Figure 2 shows the central  $80\%$  prediction sets from the Forecast Hub ensemble model on the left panel, and those from our conformal PID method on the right. We use a quantile conformal score function, as in conformalized quantile regression [30], applied asymmetrically (i.e., separately) to the lower and upper quantile levels). We use the tan integrator, with constants chosen heuristically (as described in Appendix C), and an  $\ell_1$ -regularized quantile regression as the scorer—in particular, the scorecasting model at time  $t$  predicts the quantile of the score at time  $t + 4$  based on all previous forecasts, cases, and deaths, from all 50 US states. The main takeaway is that conformal PID control is able to correct for consistent underprediction of deaths in the winter wave of late 2020/early 2021. We can see from the figure that the original ensemble fails to cover 8 times in a stretch of 10 weeks, resulting in a coverage of  $20\%$ ; meanwhile, conformal PID only fails to cover 3 times during this stretch, restoring the coverage to  $70\%$  (recall the nominal level is  $80\%$ ).
How is this possible? The ensemble is mainly comprised of constituent forecasters that ignore geographic dependencies between states [11] for the sake of simplicity or computational tractability. But COVID infections and deaths exhibit strong spatiotemporal dependencies, and most US states experienced the winter wave of late 2020/early 2021 at similar points in time. The scorer is thus able to learn from the mistakes made on other US states in order to prospectively adjust the ensemble's forecasts for the state of California. Similar improvements can be seen for other states, and we include experiments for New York and Texas as examples in Appendix F, which also gives more details on the scorer and the results.
Electricity demand forecasting. Next we consider a data set on electricity demand forecasting in New South Wales [18], which includes half-hourly data from May 7, 1996 to December 5, 1998.
Figure 2: Results for 4-week ahead COVID-19 death forecasting in California. The left column shows the COVID-19 Forecast Hub ensemble model, and the right column shows conformal PID control using the tan integrator, and a scorer given by  $\ell_1$ -penalized quantile regression on all past forecasts, cases, and deaths from all 50 states. The top row plots coverage, averaged over a trailing window of 10 weeks. The nominal coverage level is  $1 - \alpha = 0.8$  and marked by a gray dotted line. The bottom row plots the prediction sets in gold along with the ground-truth times series (death counts). Miscoverage events are indicated by red dots. Summary statistics such as the coverage and average set size are available in Table 1.
For the base forecaster we use a Transformer model [34] as implemented in darts [19]. This is only re-trained daily, to predict the entire day's demand in one batch; this is a standard approach with Transformer models due to their high computational cost. For the conformal score, we use the asymmetric (signed) residual score. We use the tan integrator as before, and we use a lightweight Theta model [2], re-trained at every time point (half-hour), as the scorer.
Figure 3: Results for electricity demand forecasting. The left column shows adaptive conformal inference (ACI), and the right column shows conformal PID control. The base forecaster is a Transformer model, and we use a tan integrator and a Theta scorer. The format of the figure follows that of Figure 2, except the nominal coverage is now  $1 - \alpha = 0.9$ , and the coverage is averaged over a trailing window of 50 points (we also omit the red dots which mark miscoverage events). Summary statistics are available in Table 2.
The results are shown in the right panel of Figure 3, where adaptive conformal inference (ACI) [16] is also compared in the left panel. In short, conformal PID control is able to anticipate intraday variations in the scores, and produces sets that "hug" the ground truth sequence tightly; it achieves tight coverage without generating excessively large or infinite sets. The main reason why this is improved is that the scorer has a seasonality component built into its prediction model; in general, large improvements such as the one exhibited in Figure 3 should only be expected when the base forecaster is imperfect, as is the case here.
1.3 Related work
The adversarial online view of conformal prediction was pioneered by [16] in the same paper that first introduced ACI. Since then, there has been significant work towards improving ACI, primarily by setting the learning rate adaptively [17, 40, 7], and incorporating ideas from multicalibration to improve conditional coverage [5]. It is worth noting that [7] also makes the observation that the ACI iteration can be generalized to track the quantile of the score sequence, although their focus is on adaptive regret guarantees. Because the topic of adaptive learning rates for ACI and related algorithms has already been investigated heavily, we do not consider it in the current paper. Any such method, such as those of [17, 7] should work well in conjunction with our proposed algorithms.
A related but distinct line of work surrounds online calibration in the adversarial sequence model, which dates back to [14, 15], and connects in interesting ways to both game theory and online learning. We will not attempt to provide a comprehensive review of this rich and sizeable literature, but simply highlight [25, 24, 23] as a few interesting examples of recent work.
Lastly, outside the online setting, we note that several researchers have been interested in generalizing conformal prediction beyond the i.i.d. (or exchangeable) data setting: this includes [33, 28, 26, 12, 8], and for time series prediction, in particular, [9, 31, 38, 39, 3]. The focus of all of these papers is quite different, and they all rely on probabilistic assumptions on the data sequence to achieve validity; we include further discussion in Appendix B.
We describe the main components of our proposal one at a time, beginning with the quantile tracker.
2.1 Quantile tracking
The starting point for quantile tracking is to consider the following optimization problem:
$$
\underset {q \in \mathbb {R}} {\text {m i n i m i z e}} \sum_ {t = 1} ^ {T} \rho_ {1 - \alpha} \left(s _ {t} - q\right), \tag {6}
$$
for large  $T$ , where we abbreviate  $s_t = s_t(x_t, y_t)$  for the score of the test point, and  $\rho_{1 - \alpha}$  denotes the quantile loss at the level  $1 - \alpha$ , i.e.,  $\rho_{\tau}(z) = \tau |z|$  for  $z > 0$  and  $(1 - \tau)|z|$  for  $z \leq 0$ . The latter is the standard loss used in quantile regression [22, 21]. Problem (6) is thus a simple convex (linear) program that tracks the  $1 - \alpha$  quantile of the score sequence  $s_t$ ,  $t \in N$ . To see this, recall that for a continuously distributed random variable  $Z$ , the expected loss  $\mathbb{E}[\rho_{1 - \alpha}(Z - q)]$  is uniquely minimized (over  $q \in \mathbb{R}$ ) at the level  $1 - \alpha$  quantile of the distribution of  $Z$ .
In the sequential setting, where we receive one score  $s_t$  at a time, a natural and simple approach is to apply online gradient descent to (6), with a constant learning rate  $\eta > 0$ . This results in the update:
$$
\begin{array}{l} q _ {t + 1} = q _ {t} + \eta \nabla \rho_ {1 - \alpha} \left(s _ {t} - q _ {t}\right) \\ = q _ {t} + \eta \left(\operatorname {e r r} _ {t} - \alpha\right), \tag {7} \\ \end{array}
$$
where the second line follows as  $\nabla \rho_{1 - \alpha}(s_t - q_t) = 1 - \alpha$  if  $s_t > q_t \iff \mathrm{err}_t = 1$ , and  $\nabla \rho_{1 - \alpha}(s_t - q_t) = -\alpha$  if  $s_t \leq q_t \iff \mathrm{err}_t = 0$ . Note that the update in (7) is highly intuitive: if we miscovered (committed an error) at the last iteration then we increase the quantile, whereas if we covered (did not commit an error) then we decrease the quantile.
Even though it is extremely simple, the quantile tracking iteration (7) can achieve long-run coverage own its own, provided the scores are bounded.
Proposition 1. Let  $\{s_t\}_{t\in \mathbb{N}}$  be any sequence of numbers in  $[-b,b]$ , for  $0 < b < \infty$ . Then the quantile tracking iteration (7) satisfies
$$
\left| \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\operatorname {e r r} _ {t} - \alpha\right) \right| \leq \frac {b + \eta}{\eta T},
$$
for any learning rate  $\eta > 0$  and  $T \geq 1$ . In particular, (7) yields long-run coverage as in (2).
${}^{2}$  Technically, this is the online subgradient method; in a slight abuse of notation, we write  $\nabla {\rho }_{1 - \alpha }\left( 0\right)$  to denote a subgradient of  ${\rho }_{1 - \alpha }$  at 0,which can take on any value in  $\left\lbrack  {-\alpha ,1 - \alpha }\right\rbrack$  .
A few remarks are in order. First, although Proposition 1 assumes boundedness of the scores, we do not need to know this bound in order to run (7) and obtain long-run coverage. As long as the scores lie in  $[-b, b]$  for any finite  $b$ , the guarantee goes through—clearly, the quantile tracker proceeds agnostically and performs the same updates in any case. Notably, the adaptive conformal inference algorithm can be expressed as a special case of the quantile tracker; see Appendix B.1 for details.
Second, for the learning rate, in practice we typically set  $\eta$  heuristically, as some fraction of the highest score over a trailing window  $\hat{B}_t = \max \{s_{t - \Delta +1},\ldots ,s_t\}$ . On this scale, setting  $\eta = 0.1\hat{B}_t$  usually gives good results, and we use it in all experiments unless specified otherwise (we also set the window length  $\Delta$  to be the same as the length of the burn-in period for training the initial base forecaster and scorer).<sup>3</sup> Extremely high learning rates result in volatile sets, while very low ones may fail to keep up with rapid changes in the score distribution.
2.2 Error integration
Error integration is a generalization of quantile tracking that follows the iteration:
$$
q _ {t + 1} = r _ {t} \left(\sum_ {i = 1} ^ {t} \left(\operatorname {e r r} _ {i} - \alpha\right)\right), \tag {8}
$$
where  $r_t$  is a saturation function that satisfies (4) for an admissible function  $h$ ; recall that we use admissible to mean nonnegative, nondecreasing, and sublinear. As we saw in (13), the quantile tracker uses a constant threshold function  $h$ , whereas  $h$  is now permitted to grow, as long as it grows sublinearly, i.e.,  $h(t) / t \to 0$  as  $t \to \infty$ . A non-constant threshold function  $h$  can be desirable because it means that  $r_t$  will "saturate" (will hit the conditions on the right-hand sides in (4)) less often, so corrections for coverage error will occur less often, and in this sense, a greater degree of coverage error can be tolerated along the sequence.
The next proposition, in particular its proof, makes the role of  $h$  precise. Importantly, Proposition 2 suffices to prove Theorem 1.
Proposition 2. Let  $\{s_t\}_{t\in \mathbb{N}}$  be any sequence of numbers in  $[-b,b]$ , where  $b > 0$ , and may be infinite. Assume that  $r_t$  satisfies (4), for an admissible function  $h$ . Then the error integration iteration (8) satisfies
$$
\left| \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\operatorname {e r r} _ {t} - \alpha\right) \right| \leq \frac {c \cdot h (T) + 1}{T}, \tag {9}
$$
for any  $T \geq 1$ , where  $c$  is the constant in (4). In particular, this means (8) yields long-run coverage (2).
The choice of saturation function essentially corresponds to a choice of adaptive learning rate; see Appendix D for details.
2.3 Scoring
The final piece to discuss is scorecasting. A scorer attempts to forecast  $q_{t+1}$  directly, taking advantage of any leftover signal that is not captured by the base forecaster. This is the role played by  $\hat{q}_{t+1}$  in (5). Scorecasting may be particularly useful when it is difficult to modify or re-train the base forecaster. This can occur when the base forecaster is computationally costly to train (e.g., as in a Transformer model); or it can occur in complex operational prediction pipelines where frequently updating a forecasting implementation is infeasible. Another scenario where scorecasting may be useful is one in which the forecaster and scorer have access to different levels of data. For example, if a public health agency collects epidemic forecasts from external groups, and forms an ensemble forecast, then the agency may have access to finer-grained data that it can use to recalibrate the ensemble's prediction sets (compared to the level of data granularity granted to the forecasters originally).
This motivates the need for scorecasting as a modular layer that "sits on top" of the base forecaster and residualizes out systematic errors in the score distribution. This intuition is made more precise
3Technically, this learning rate is not fixed, so Proposition 1 does not directly apply. However, we can view it as a special case of error integration and an application of Proposition 2 thus provides the relevant coverage guarantee.
by recalling, as described above (following Proposition 2), that scorecasting combined with error integration as in (5) is just a reparameterization of error integration (8), where  $q_{t}^{\prime} = q_{t} - \hat{q}_{t}$  and  $s_t^\prime = s_t - \hat{q}_t$  are the new quantile and new score, respectively. A well-executed scorer could reduce the variability in the scores and make them more exchangeable, resulting in more stable coverage and tighter prediction sets, as seen in Figure 3. On the other hand, an aggressive scorer with little or no signal can actually hurt by adding variance to the new score sequence  $s_t^\prime$ , which could result in more volatile coverage and larger sets.