SlowGuess/NeurIPS_2023_OCR huggingface dataset:
There is no limit to what we can choose for the scorecasting model. We might like to use a model that can simultaneously incorporate seasonality, trends, and exogenous covariates. Common choices would be SARIMA (seasonal autoregressive integrated moving average) and ETS (error-trend-seasonality) models, but there are many other available methods, such as the Theta model [2], Prophet model [32], and neural network forecasters; see [20] for a review.
Briefly, we revisit the PID perspective, to recap how quantile tracking, error integration, and score-casting fit in and work in combination. It helps to return to (3), which we copy again here:
$$
q _ {t + 1} = g _ {t} ^ {\prime} + \eta \left(\operatorname {e r r} _ {t} - \alpha\right) + r _ {t} \left(\sum_ {i = 1} ^ {t} \left(\operatorname {e r r} _ {i} - \alpha\right)\right). \tag {10}
$$
Quantile tracking is precisely given by taking  $g_{t}^{\prime} = q_{t}$  and  $r_t = 0$ . This can be seen as equivalent to P control: subtract  $q_{t}$  from both sides in (10) and treat the increment  $u_{t + 1} = q_{t + 1} - q_{t}$  as the process variable; then in this modified system, quantile tracking is exactly P control. For this reason, we use "conformal P control" to refer to the quantile tracker in the experiments that follow. Similarly, we use "conformal PI control" to refer to the choice  $g_{t}^{\prime} = q_{t}$ , and  $r_t\neq 0$  as a generic integrator (for us, tan is the default). Lastly, "conformal PID control" refers to letting  $g_{t}^{\prime}$  be a generic scorecaster, and  $r_t\neq 0$  be a generic integrator.
In addition to the statewide COVID-19 death forecasting experiment described in the introduction, we run experiments on all combinations of the following data sets and forecasters.
Data sets:
Forecasters (all via darts [19]):
Electricity demand [18]
- Autoregressive (AR) model with 3 lags
- Return (log price) of Amazon, Google, and Microsoft stock [27]
- Theta model with  $\theta = 2$  [2]
- Temperature in Delhi [37]
- Prophet model [32]
- Transformer model [34]
In all cases except for the COVID-19 forecasting data set, we: re-train the base forecaster at each time point; construct prediction sets using the asymmetric (signed) residual score; and use a Theta model for the scorer. For the COVID-19 forecasting setting, we: use the given ensemble model as the base forecaster (no training at all); construct prediction sets using the asymmetric quantile score; and use an  $\ell_1$ -penalized quantile regression as the scorer, fit on features derived from previous forecasts, cases, and deaths, as described in the introduction. And lastly, in all cases, we use a tan function for the integrator with constants chosen heuristically, as described in Appendix C.
The results that we choose to show in the subsections below are meant to illustrate key conceptual points (differences between the methods). Additional results are presented in Appendix G. Our GitHub repository, https://github.com/aangelopoulos/conformal-time-series, provides the full suite of evaluations.
3.1 ACI versus quantile tracking
We forecast the daily Amazon (AMZN) opening stock price from 2006-2014. We do this in log-space (hence predicting the return of the stock). Figure 4 compares ACI and the quantile tracker, each
Time
AR	Prophet	Theta	Transformer
ACI	P Ctrl	ACI	P Ctrl	ACI	P Ctrl	ACI	P Ctrl
Marginal coverage	0.894	0.894	0.904	0.888	0.895	0.894	0.906	0.887
Longest err sequence	6	3	13	6	5	5	21	9
Average set size	∞	17.6	∞	51.7	∞	17.8	∞	70.4
Median set size	13.4	13.3	50.7	37.3	12.8	13.1	61.9	44.3
75% quantile set size	28.4	22.3	117	72.1	20.9	22.6	179	98.4
90% quantile set size	44.1	37.7	236	114	38.6	38.4	302	153
95% quantile set size	49.9	46.2	∞	140	48.2	46.9	∞	196
Figure 4: Results for forecasting Amazon stock return, comparing ACI and quantile tracking (P control). The plots show AR as the base forecaster; the table summarizes the results of all four base forecasters. We use the default learning rates for both ACI and quantile tracking:  $\eta = 0.005$  and  $\eta = 0.1\hat{B}_t$ , respectively.
Figure 5: As in Figure 4, but with larger learning rates for ACI and quantile tracking:  $\eta = 0.1$  and  $\eta = 0.5\hat{B}_t$ , respectively.
Time
AR	Prophet	Theta	Transformer
ACI	P Ctrl	ACI	P Ctrl	ACI	P Ctrl	ACI	P Ctrl
Marginal coverage	0.9	0.898	0.901	0.897	0.9	0.896	0.902	0.897
Longest err sequence	2	2	8	3	2	2	7	3
Average set size	∞	28.4	∞	42.6	∞	27.8	∞	54.2
Median set size	∞	18.6	60.6	31.9	∞	18.7	46	35.5
75% quantile set size	∞	37.4	∞	56.5	∞	37.8	∞	69.7
90% quantile set size	∞	66.3	∞	93.4	∞	63.4	∞	123
95% quantile set size	∞	81.8	∞	116	∞	78.5	∞	164
with its default learning rate:  $\eta = 0.005$  for ACI, and  $\eta = 0.1\hat{B}_t$  for quantile tracking. We see that the coverage from each method is decent, but oscillates nontrivially around the nominal level of  $1 - \alpha = 0.9$  (with ACI generally having larger oscillations). Figure 5 thus increases the learning rate for each method:  $\eta = 0.1$  for ACI, and  $\eta = 0.5\hat{B}_t$  for the quantile tracker. We now see that both deliver very tight coverage. However, ACI does so by frequently returning infinite sets; meanwhile, the corrections to the sets made by the quantile tracker are nowhere near as aggressive.
As a final comparison, in Appendix E, we modify ACI to clip the sets in a way that disallows them from ever being infinite. This heuristic may be used by practitioners that want to guard against infinite sets, but it no longer has a validity guarantee for bounded or unbounded scores. The results in the appendix indicate that the quantile tracker has similar coverage to this procedure, and usually with smaller sets.
3.2 The effect of integration
Next we forecast the daily Google (GOOGL) opening stock price from 2006-2014 (again done in log-space). Figure 6 compares the quantile tracker without and without an additional integrator component (P control versus PI control). We purposely choose a very small learning rate,  $\eta = 0.01\hat{B}_t$ , in order to show how the integrator can stabilize coverage, which it does nicely for most of the time series. The coverage of PI control begins to oscillate more towards the end of the sequence, which we attribute at least in part to the fact that the integrator measures coverage errors accumulated over all time—and by the end of a long sequence, the marginal coverage can still be close to  $1 - \alpha$  even if the local coverage deviates more wildly. This can be addressed by using a local version of the integrator, an idea we return to in the discussion.
Time
AR	Prophet	Theta	Transformer
P Ctrl	PI Ctrl	P Ctrl	PI Ctrl	P Ctrl	PI Ctrl	P Ctrl	PI Ctrl
Marginal coverage	0.895	0.896	0.882	0.888	0.891	0.896	0.817	0.897
Longest err sequence	6	3	31	13	58	6	106	10
Average set size	19.2	24.7	95.6	68.2	42.9	33.5	281	108
Median set size	16.4	22.8	91.5	60.7	44.3	32.8	219	73.8
75% quantile set size	21.3	30.1	120	88	60	42.6	386	121
90% quantile set size	30.7	38.4	150	118	66.2	51.3	482	244
95% quantile set size	32.8	44.7	156	139	69.1	57.5	524	341
Figure 6: Results for forecasting Google stock return, comparing quantile tracking with and without the integrator (P control versus PI control). The plots show Prophet as the base forecaster; the table summarizes the results of all four base forecasters. We purposely use a very small learning rate,  $\eta = 0.01\hat{B}_t$ , in order to show how the integrator can stabilize coverage.
3.3 The effect of scorecasting
Figures 2 and 3 already showcase examples in which scorecasting offers significant improvement in coverage and set sizes. Recall that these were settings in which the base forecaster produces errors (scores) that have predictable trends. Further examples in the COVID-19 forecasting setting, which display similar benefits to scorecasting, are given in Appendix F.
We emphasize that it is not always the case that scorecasting will help. In some settings, scorecasting may introduce enough variance into the new score sequence that the coverage or sets will degrade in stability. (For example, this will happen if we run a highly complex scorer on a sequence of i.i.d. scores, where there are no trends whatsoever.) In practice, scorecasters should be designed with care, just as one would design a base forecaster; it is unlikely that using "out of the box" techniques for scorecasting will be robust enough, especially in high-stakes problems. Appendix G provides examples in which scorecasting, run across all settings using a generic Theta model, can hurt (for example, it adds noticeable variance to the coverage and sets in some instances within the Amazon data setting).
4 Discussion and extensions
Discussion. Our work presents a framework for constructing prediction sets in time series that is analogous (and indeed formally equivalent) to PID control, consisting of quantile tracking (P control), which is simply online gradient descent applied to the quantile loss; error integration (I control) to stabilize coverage; and scorecasting (D control) to remove systematic trends in the scores (errors made by the base forecaster).
We found that the combination of quantile tracking and integration consistently yields robust and favorable performance in our experiments. Scorecasting provides additional benefits if there are trends left in scores that are predictable (and the scorer is well-designed), as is the case in some of our examples. Otherwise, scorecasting may add variability and make the coverage and prediction sets more volatile. Overall, designing the scorer (which includes the choice to even use one at all) is an important modeling step, just like the design of the base scorer.
It is worth emphasizing that, with the exception of the COVID-19 forecasting example, our experiments are intended to be illustrative, and we did not look to use state-of-the-art forecasters, or include any and all possibly relevant features for prediction. Further, while we found that using heuristics to set constants (such as the learning rate  $\eta$ , and constants  $C_{\mathrm{sat}}$ ,  $K_{\mathrm{I}}$  for the tan integrator) worked decently well, we believe that more rigorous techniques, along the lines of [17, 7], can be used to tune these adaptively in an online fashion.
Extensions. We now present an extension of our analysis to conformal risk control [1, 6, 13]. In this setting, we are given a sequence of loss functions  $L_{t}:2^{\mathcal{Y}}\times \mathcal{Y}\to [0,1]$  satisfying  $L_{t}(\mathcal{V},y) = 0$  for all  $y$ , and  $L_{t}(\emptyset ,y) = 1$  for all  $y$ . The goal is to bound the deviation of the average risk  $\frac{1}{T}\sum_{t = 1}^{T}L_{t}(\mathcal{C}_{t},y_{t})$  from  $\alpha$ . We state a result for the integrator below, and give its proof in Appendix A.
Proposition 3. Consider the iteration  $q_{t + 1} = r_t(\sum_{i = 1}^t (L_i(\mathcal{C}_i,y_i) - \alpha))$ , with  $L_{t}$  as above. Assume that  $r_t$  satisfies (4), for an admissible function  $h$ . Also assume that  $C_t(\mathcal{C}_t,y_t) = \emptyset$  if  $q_{t}\leq -b$  and  $\mathcal{V}$  if  $q_{t}\geq b$ , where  $b > 0$ , and may be infinite. Then for all  $T\geq 1$
$$
\left| \frac {1}{T} \sum_ {t = 1} ^ {T} \left(L _ {t} \left(\mathcal {C} _ {t}, y _ {t}\right) - \alpha\right) \right| \leq \frac {c \cdot h (T) + 1}{T}. \tag {11}
$$
for any  $T\geq 1$  , where  $c$  is the constant in (4).
We briefly conclude by mentioning that we believe many other extensions are possible, especially with respect to the integrator. Broadly, we can choose to integrate in a kernel-weighted fashion,
$$
r _ {t} \left(\sum_ {i = 1} ^ {t} \left(\operatorname {e r r} _ {i} - \alpha\right) \cdot K \left(\left(i, x _ {i}, y _ {i}\right), \left(t, x _ {t}, y _ {t}\right)\right)\right). \tag {12}
$$
As a special case, the kernel could simply assign weight 1 if  $t - i \leq w$ , and weight 0 otherwise, which would result in an integrator that aggregates coverage over a trailing window of length  $w$ . This can help consistently sustain better local coverage, for long sequences. As another special case, the kernel could assign a weight based on whether  $x_{i}$  and  $x_{t}$  lie in the same bin in some pre-defined binning of  $\mathcal{X}$  space, which may be useful for problems with group structure (where we want group-wise coverage). Various other choices and forms of kernels are possible, and it would be interesting to consider adding together a number of such choices (12) in combination, in a multi-resolution flavor, for the ultimate quantile update.
We would like to thank Tijana Zrnic, Amit Kohli, and Jordan Lekeufack for their valuable feedback. This work was supported by the National Science Foundation (NSF) Graduate Research Fellowship Program under grant no. 2146752, and the Office of Naval Research (ONR) Multi-University Research Initiative (MURI) Program under grant no. N00014-20-1-2787.
[39] Chen Xu and Yao Xie. Sequential predictive conformal inference for time series. In International Conference on Machine Learning, 2023.
[40] Margaux Zaffran, Olivier Feron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, 2022.
We begin with the proof of Proposition 1. The proof is very simple, and we derive it as a corollary of Proposition 2, given next, because the proof reveals something perhaps unforeseen about the quantile tracker: it acts as an error integrator, despite only adjusting the quantile based on the most recent time step.
Proof of Proposition 1. Without a loss of generality, set  $q_{1} = 0$ . Unraveling the iteration (7) yields
$$
q _ {t + 1} = \eta \sum_ {i = 1} ^ {t} \left(\operatorname {e r r} _ {i} - \alpha\right). \tag {13}
$$
For  $r_t(x) = \eta x$ ,  $h(t) = b$ , we see (4) holds with  $c = 1 / \eta$ . Proposition 2 now applies.
Proof of Proposition 2. Abbreviate  $E_{T} = \sum_{t=1}^{T} (\mathrm{err}_{t} - \alpha)$ . We will prove one side of the absolute inequality in (9), namely,  $E_{T} \leq c \cdot h(T) + 1$ , and the other side follows similarly. We use induction. The base case, for  $T = 1$ , holds trivially. Now assume the result is true up to  $T - 1$ . We divide the argument into two cases: either  $c \cdot h(T - 1) < E_{T - 1} \leq c \cdot h(T - 1) + 1$  or  $E_{T - 1} \leq c \cdot h(T - 1)$ . In the first case, note that (4) implies  $q_{T} = r_{t}(E_{T - 1}) \geq b$  and therefore  $s_{T} \leq q_{T}$  and  $\mathrm{err}_{t} = 0$ . This means that
$$
E _ {T} = E _ {T - 1} - \alpha \leq c \cdot h (T - 1) + 1 - \alpha \leq c \cdot h (T) + 1,
$$
as  $h$  is nondecreasing, which is the desired result at  $T$ . In the second case, we just use  $\mathrm{err}_T \leq 1$ , so
$$
E _ {T} \leq E _ {T - 1} + 1 - \alpha \leq c \cdot h (T - 1) + 1 - \alpha \leq c \cdot h (T) + 1.
$$
This again gives the desired result at  $T$ , and completes the proof.
Proof of Theorem 1. We can transform (5) by setting  $q_{t+1}' = q_{t+1} - \hat{q}_{t+1}$ , and this becomes an update of the form (8) with respect to  $q_{t+1}'$ . Further, the score sequence in this new parameterization is  $s_t' = s_t - \hat{q}_t$ , which is in  $[-b, b]$  because both  $s_t$  and  $\hat{q}_t$  are in  $[-b/2, b/2]$ . Applying Proposition 2 gives the result.
Proof of Proposition 3. The proof is similar to that of Proposition 2—as in that proof, we only prove one side of the absolute inequality (11), and use induction. Abbreviate  $E_{T} = \sum_{t=1}^{T}(L_{i}(\mathcal{C}_{i},y_{i}) - \alpha)$ . The base case holds trivially. For the inductive step, either  $c \cdot h(T-1) < E_{T-1} \leq c \cdot h(T-1) + 1$  or  $E_{T-1} \leq c \cdot h(T-1)$ . In the first case, we have saturated, so  $L_{t}(\mathcal{C}_{t},y_{t}) = 0$ , and
as  $h$  is nondecreasing, which is the desired result at  $T$ . In the second case, we just use the boundedness of the loss  $L_{t}(\mathcal{C}_{t},y_{t})\leq 1$ , so
B Comparisons to existing methods
This section contains descriptions of the relationship of our method to pre-existing methods for conformal prediction under distribution shift.
B.1 ACI as a special case
Though it may not be immediately obvious, adaptive conformal inference (ACI) is actually a special case of the quantile tracker. ACI follows the iteration:
$$
\alpha_ {t + 1} = \alpha_ {t} - \eta (\operatorname {e r r} _ {t} - \alpha),
$$
$$
\begin{array}{l} 1 - \alpha_ {t + 1} = 1 - \alpha_ {t} + \eta (\operatorname {e r r} _ {t} - \alpha) \\ = 1 - \alpha_ {t} + \eta \nabla \rho_ {1 - \alpha} \left(\beta_ {t} - \left(1 - \alpha_ {t}\right)\right), \\ \end{array}
$$
for  $\beta_{t} = \inf \{\beta : s_{t} \leq \mathrm{Quantile}_{\beta}(\{s_{1}, \ldots, s_{t - 1}\})\}$ . This shows that ACI is a particular instance of the quantile tracker that uses a secondary score  $s_{t}' = \beta_{t}$  and quantile  $q_{t}' = 1 - \alpha_{t}$ . Thus, because quantile tracking (7) is the same as a linear coverage integrator (13), so is ACI.
We can see here that ACI transforms unbounded score sequences into bounded ones, which then implies long-run coverage for any score sequence. This may, however, come at a cost: ACI can sometimes output infinite or null prediction sets (when  $\alpha_{t}$  drifts below 0 or above 1, respectively). Direct quantile tracking on the scale of the original score sequence does not have this behavior.
B.2 Weighted conformal prediction
A line of work by [33] and [4] deals with variants of weighted conformal prediction. Specifically, the latter variant allows the incorporation of sample weights for each data point. When the data come from a probabilistic model, the sample weights allow for probabilistic bounds on the coverage of a new test point.
These methods, and others (e.g., [9]) in the probabilistic setting, differ greatly from the approach herein. When there is no probabilistic model, for example, all such approaches give vacuous coverage guarantees. The reason is that coverage on a new test point is impossible without a probabilistic model. However, in our setting we target long-run coverage, a completely different (and achievable) goal. Thus, neither the methods nor the proof techniques and theorem statements share any technical similarity to speak of.
For good measure, we include below in Appendix G evaluations against a version of [4] with a trailing window the size of the burn-in period. This approach generally performs worse than both our approach and ACI, and does not have guarantees in our setting, so we do not include it in main text.
B.3 Sequential predictive conformal inference (SPCI)
A recently proposed line of work exemplified by Ensemble Batch Prediction Intervals (EnbPI) [38] and its improved version, SPCI [39], seeks to provide coverage guarantees for black-box time-series predictors. However, the methods therein are not distribution-free, and SPCI rely on the following assumptions for validity:
1. The use of the quantile random forest (QRF) algorithm.
2. Model assumptions, such as consistency.
3. Distributional assumptions, such as smoothness.
Thus, the validity of SPCI is significantly weaker than the methods herein, perhaps along the lines of a standard nonparametric analysis. In contrast, our method has finite-sample validity averaged over time for any input sequence. However, it is possible to use the QRF/SPCI algorithm as the scorer, which would endow it with a distribution-free guarantee. We do not do this because the QRF is not a particularly effective time series forecasting method, and was outperformed by quantile LASSO (which we use for the COVID experiments) and the Theta model. The performance of the QRF tends to be "blocky" and high-variance; for an example, see plots on the SPCI GitHub page.
However, we note that the EnbPI and SPCI algorithms can be seen as specific choices of scorecasters, and one consequence of our paper is that these algorithms can be endowed distribution-free guarantees when used as scorecasters in (5).
C Heuristics for setting constants
Consider the tan integrator  $r_t(x) = K_{\mathrm{I}} \tan (x \log(t) / (t C_{\mathrm{sat}}))$ , where we set  $\tan(x) = \mathrm{sign}(x) \cdot \infty$  whenever  $x \notin [-\pi/2, \pi/2]$ , and  $C_{\mathrm{sat}}, K_{\mathrm{I}} > 0$  are constants. The constant  $C_{\mathrm{sat}}$  is primarily in charge of guaranteeing that by time  $T$ , we want to have an absolute guarantee of at least  $1 - \alpha - \delta$  coverage.
Then we can set
$$
C _ {\mathrm {s a t}} = \frac {2}{\pi} \left(\lceil \log (T) \delta \rceil - 1 / \log (T)\right)
$$
to ensure the tan function has an asymptote at the correct point. The purpose of the constant  $K_{\mathrm{I}}$  is to place the integrator on the same scale as the scores. So if  $B'$  is a hypothesized bound on the magnitude of the scores, then one can set  $K_{\mathrm{I}} = B'$ . In practice, these heuristics can be taken as a starting place, and then the numbers can be fine-tuned during a burn-in period by hand or algorithmically. As alluded to previously, we believe there is room for work in the style of [17, 7] to rigorously tune these parameters online, but it is not the focus of our paper.
D Coverage integration as adaptive learning rate
As already mentioned in the introduction, in all our experiments we use a nonlinear saturation function  $r_t(x) = K_{\mathrm{I}} \tan (x \log(t) / (t C_{\mathrm{sat}}))$ , where we set  $\tan(x) = \operatorname{sign}(x) \cdot \infty$  for  $x \notin [-\pi/2, \pi/2]$ , and  $C_{\mathrm{sat}}, K_{\mathrm{I}} > 0$  are constants that we choose heuristically (described in Appendix C). In a sense, this tan integrator is akin to a quantile tracker whose learning rate adapts to the current coverage gap. To see this, we can use a first-order Taylor approximation, which shows (ignoring constants):
$$
q _ {t + 1} = \tan \left(\frac {\log (t)}{t} \sum_ {i = 1} ^ {t} (\mathrm {e r r} _ {i} - \alpha)\right) \approx q _ {t} + \underbrace {\frac {\log (t)}{t} \sec^ {2} \left(\frac {\log (t - 1)}{t - 1} \sum_ {i = 1} ^ {t - 1} (\mathrm {e r r} _ {i} - \alpha)\right)} _ {\mathrm {e f f e c t i v e l e a n i n g r a t e}} \nabla \rho_ {1 - \alpha} (s _ {t} - q _ {t}).
$$
Above,  $\sec (x) = 1 / \cos (x)$  is the secant function, which has a U-shape; thus we can see from the above that the effective learning rate is higher for larger errors. Similar analyses for different integrators will give different adaptive learning rates; see below for another example with a decaying learning rate.
Consider  $r_t(x) = \eta x / \sqrt{t}$ . (This will give long-run coverage only for bounded scores, because condition (4) is only met for finite and not infinite  $b$ .) Then (8) becomes  $q_{t + 1} = \frac{\eta}{\sqrt{t}}\sum_{i = 1}^{t}(\mathrm{err}_i - \alpha)$ , which can be rewritten as
$$
q _ {t + 1} = \frac {\sqrt {t - 1}}{\sqrt {t}} \frac {\eta}{\sqrt {t - 1}} \sum_ {i = 1} ^ {t - 1} (\mathrm {e r r} _ {i} - \alpha) + \frac {\eta}{\sqrt {t}} (\mathrm {e r r} _ {i} - \alpha) \approx q _ {t} + \frac {\eta}{\sqrt {t}} (\mathrm {e r r} _ {t} - \alpha).
$$
This is approximately the quantile tracker (7) with a decaying learning rate, on the order of  $1 / \sqrt{t}$ .
E Comparison to clipped ACI
Figures 7 and 8 compare the quantile tracker to a clipped version of ACI which disallows infinite-sized sets by clipping the sets to the largest score seen so far.
F More details on COVID-19 forecasting
In this experiment, the scorer receives as input the three most recent scores (i.e., quantile errors) of the ensemble forecaster, as well as the three most recent case and death counts, from all 50 states. The scorer is an  $\ell_1$ -penalized quantile regression as implemented by sklearn.linear_model.QuantileRegressor. We fixed tuning parameter for the  $\ell_1$  penalty at 10; in our experience, the performance of the scorer was fairly robust to this choice. Automatic selection (e.g., using cross-validation) could be the topic of future study. Figures 9 and 10 show the analogous experiments but for forecasting in New York and Texas.
G Further experiments
We give a more comprehensive view of our results, examining all data sets, and a range of tuning parameters for each method. We restrict our attention to AR as the base forecaster; for the rest of the base forecasters, we refer to the GitHub repository: https://github.com/aangelopoulos/conformal-time-series.