SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AR	Prophet	Theta	Transformer
clip ACI	P Ctrl	clip ACI	P Ctrl	clip ACI	P Ctrl	clip ACI	P Ctrl
Marginal coverage	0.898	0.898	0.884	0.897	0.898	0.896	0.884	0.897
Longest err sequence	2	2	8	3	2	2	7	3
Average set size	44.9	28.4	52.6	42.6	43.3	27.8	60.5	54.2
Median set size	41.8	18.6	38.8	31.9	27.3	18.7	36.6	35.5
75% quantile set size	58.9	37.4	66.9	56.5	59.5	37.8	85.5	69.7
90% quantile set size	93.9	66.3	137	93.4	94.7	63.4	148	123
95% quantile set size	136	81.8	166	116	136	78.5	182	164
Figure 7: As in Figure 4, but with clipped ACI.
AR	Prophet	Theta	Transformer
clip ACI	P Ctrl	clip ACI	P Ctrl	clip ACI	P Ctrl	clip ACI	P Ctrl
Marginal coverage	0.894	0.894	0.896	0.888	0.895	0.894	0.89	0.887
Longest err sequence	6	3	13	6	5	5	21	9
Average set size	19.5	17.6	69.5	51.7	17.9	17.8	115	70.4
Median set size	13.4	13.3	48.7	37.3	12.8	13.1	61.7	44.3
75% quantile set size	27.9	22.3	91.1	72.1	20.7	22.6	165	98.4
90% quantile set size	44	37.7	168	114	38.5	38.4	248	153
95% quantile set size	48.7	46.2	195	140	47.2	46.9	304	196
Figure 8: As in Figure 5, but with clipped ACI.
Time
Base Forecaster	Conformal PID Ctrl
Marginal coverage	0.78	0.85
Longest err sequence	8	3
Average set size	310	472
Median set size	249	440
75% quantile set size	388	608
90% quantile set size	667	821
95% quantile set size	754	1.03e+03
Figure 9: Results for 4-week ahead COVID-19 death forecasting in New York.
Figure 10: Results for 4-week ahead COVID-19 death forecasting in Texas.
Time
Base Forecaster	Conformal PID Ctrl
Marginal coverage	0.75	0.85
Longest err sequence	7	2
Average set size	556	827
Median set size	482	706
75% quantile set size	759	1.03e+03
90% quantile set size	993	1.36e+03
95% quantile set size	1.11e+03	1.57e+03
Marginal coverage	0.82	0.86
Longest err sequence	6	2
Average set size	625	858
Median set size	512	688
75% quantile set size	754	1.01e+03
90% quantile set size	1.12e+03	1.47e+03
95% quantile set size	1.45e+03	1.8e+03
Table 1: Summary statistics for COVID-19 death forecasting in California, as in Figure 2.
AR	Transformer
ACI	Conformal PID Control	ACI	Conformal PID Ctrl
Marginal coverage	0.899	0.9	0.899	0.901
Longest err sequence	3	2	3	2
Average set size	∞	0.177	∞	0.174
Median set size	0.406	0.178	0.426	0.175
75% quantile set size	0.484	0.21	0.574	0.206
90% quantile set size	0.672	0.236	∞	0.233
95% quantile set size	∞	0.252	∞	0.249
Table 2: Summary statistics for electricity forecasting, as in Figure 3. Results for the Prophet and Theta models are not available because darts does not support intermittent retraining for these algorithms.
For each experiment, we describe the data set in a new subsection, and two plots are included: one for the coverage, and one for the prediction sets. Each column in the plots represents a different method, and each row is a different learning rate. For the quantile tracker, the learning rate is to be interpreted as the multiplier in front of  $\hat{B}_t$ . Each method is given a different color, which stays consistent throughout the plots. We use a tan integrator and a Theta scorer throughout, just as in the main text experiments. There is an additional method, 'Trail', corresponding to the proposal of Barber et al. [4] with a trailing window.
G.1 Amazon/Google
These data sets are part of a multivariate time series consisting of thirty blue-chip stock prices, including those of Amazon (AMZN) and Google (GOOGL), from January 1, 2006 to December 31, 2014. We attempt to forecast the daily opening price of each of Amazon and Google stock, on a log scale. Available to the scorer are the previous open prices of all 30 stocks.
G.2 Microsoft
This data set is a univariate time series consisting of a single stock open price, that of Microsoft (MSFT), from April 1, 2015 to May 31, 2021.
G.3 Daily temperature in Delhi
This data set contains the daily temperature (averaged over 8 measurements in 3 hour periods), humidity, wind speed, and atmospheric temperature in the city of Delhi from January 1, 2003 to April 24, 2017, scraped using the Weather Underground API.
G.4 Electricity demand forecasting
This data set measures electricity demand in New South Wales collected at half-hour increments from May 7th, 1996 to December 5th, 1998 (we zoom in on the first 2000 time points). There are also
several other variables collected, such as the demand and price in Victoria, the amount of energy transfer between New South Wales and Victoria, and so on. These are given as covariates to the scorer. The demand value is normalized by default to lie in [0, 1].
G.5 Synthetic data sets
We perform some experiments on two synthetic score sequences which include change points and other behaviors difficult to produce using real data. In this setting, there is no ground truth  $y_{t}$  sequence, so we do not plot the sets. Instead, we plot the scores themselves in one column, and the quantiles  $q_{t}$  produced by each algorithm in a different column (when  $q_{t} \geq s_{t}$ , we cover). The general goal is for  $q_{t}$  to track the  $1 - \alpha$  quantile of  $s_{t}$ , and if it is too far off, that corresponds to the "set being too large or too small" in a situation where we would be constructing sets out of these scores.
We consider an i.i.d. sequence of scores, a noisy increasing sequence of scores, and a mix of change points and trends. Our codebase describes the score generation procedure in more detail.
or the Amazon data set.
for the Google data set.
or the Microsoft data set.
Figure 14: Results for the Delhi temperature data set.
Figure 15: Results for the electricity demand forecasting data set.
Figure 16: Results for an i.i.d. score sequence.
Figure 17: Results for an increasing score sequence.
Figure 18: Results for a score sequence that is a mix of change points and trends.