Conformal Prediction with Temporal Quantile Adjustments
Authors: Zhen Lin
Shubhendu Trivedi
Jimeng Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: conformal prediction
time series forecasting
cross-sectional data
time series regression
prediction interval
uncertainty quantification

Abstract: We develop Temporal Quantile Adjustment (TQA), a general method to construct efficient and valid prediction intervals (PIs) for regression on cross-sectional time series data. Such data is common in many domains, including econometrics and healthcare. A canonical example in healthcare is predicting patient outcomes using physiological time-series data, where a population of patients composes a cross-section. Reliable PI estimators in this setting must address two distinct notions of coverage: cross-sectional coverage across a cross-sectional slice, and longitudinal coverage along the temporal dimension for each time series. Recent works have explored adapting Conformal Prediction (CP) to obtain PIs in the time series context. However, none handles both notions of coverage simultaneously. CP methods typically query a pre-specified quantile from the distribution of nonconformity scores on a calibration set. TQA adjusts the quantile to query in CP at each time $t$, accounting for both cross-sectional and longitudinal coverage in a theoretically-grounded manner. The post-hoc nature of TQA facilitates its use as a general wrapper around any time series regression model. We validate TQA's performance through extensive experimentation: TQA generally obtains efficient PIs and improves longitudinal coverage while preserving cross-sectional coverage.

Conformal Prediction with Temporal Quantile Adjustments
Zhen Lin' Shubhendu Trivedi Jimeng Sun 1.3
Department of Computer Science, University of Illinois at Urbana-Champaiga
2 Massachusetts Institute of Technology
3 Carle Illinois College Medicine, University of Illinois at Urbana- Champaann {zhenlin4, jiiiiimmmmllinnneeee innii.iie.uuu edu shubhendu@csail mit. edu
Abstract
We develop Temporal Quantile Adjustment (TQA), general method to construct efficient and valid prediction intervals (PIs) for regression cross-sectional time series data. Such data common in many domains, including econometrics and healthcare. A canonical example in healthcare is predicting patient outcomes using physiological time-series data, where a population of patients composes cross-section. Reliable PI estimators in this setting must address two distinct notions of coverage: cross- pposs-sectiona coverage across a cross-sectional slice, and longitudinal coverage along the temporal dimension for each time series. Recent works have explored adapting Conformal Prediction (CP) to obtain PIs the time series context. However, none handles both notions of coverage simultaneously. CP methods typically query a pre-specified quantile from the distribution of non- conformity scores on calibration set. TQA adjusts the quantile query CP at each time i, accounting for both cross-sectional and longitudinal coverage in a theoretically-groumee
manner. The post-hoc nature of TQA facilitates its use as a general wrapper around any time series regression model. We validate TQA's 's performance through extensive experimentation: TQA generally obtains efficient PIs and improves longitudinal coverage while preserving cross-sectional coverage. Our code is available at https: ////ttttt com/z1in7/TQA
Introduction
The impressive predictive performance of modern "black-box" machine learning methods has started to make them critical ingredients various high-stakes ingly important quantify the predictive uncertainty of such models reliably and efficiently, which remains fundamental challenge. Conformal Prediction (CP), pioncered by Vovk et al. [53], is a powerful framework for quantifying uncertainty under mild assumptions. The model-agnostic and distribution-frr nature of CP makes particularly suitable for large neural network models, and has started attract the attention of the deep learning community [2, 3, 6, 9, 12, 13, 33, 59]. The primary assumption most current CP methods that data exchangeability. For instance, only construct
pipelines. It is thus increas-
assuming exchangeability of the calibration and test data, intervals by simply querying the corresponding quantile of nonconformity scores on the calibration set. Recent works have started exploring the adaptation of CP in settings that go beyond the usual exchangeability assumption [4, 17, 21, 38, 42, 43, 44, 50, 57], and to more complex data such as time series [17, 48, 56. 58].
valid prediction
We study the adaptation of CP to the cross-sectional time series regression setting. More formally, suppose data comprises of N time series, denoted (S, 1 1 with each S; sampled from
During the initiation and pursuance of this research, the author's primary affiliation was MIT.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(e-g. days hospital)
(A): Cross-actionaally Invalid, Longitidiallly Invalid
coverage
patient
(B): Cros-scctionlly Valid, Longitudizally Invalid
Patient
preditedd
missed
Data
covered
Cottolectioollly Valid. Longitudizally Valid
prediction interval
Figure 1: PI estimators with different cross-sectional longitudinal properties/validitie. x denotes the ground-truth that is outside of the PI. (A) features Pls that are not valid in either sense: Y is never covered. (B) is cross- sectionlcy valid: for any t, 90% of the Y., are covered. It is longitudinally invalid, as 10% of the TS receive no coverage at all. (C) exhibits both cross-sectional and longitudinal validity, which is ideal.
arbitrary distribution Ps. Further, each time series S. is a sequence of temporally-depent with Zi,t (X..,, Y2,)) consisting of covariates X., E IRd and [- f==1/ and XN+1,141 for new time series
random
variables [Z. ...., .i....... Zerr).
the response Y., E R. Given data until time (fZN+1.) SN+1, the time series regression problem entails predicting the response +1+11 at (an unknown) time I 1. We are interested quantifying the uncertainty of each prediction by constructing valid prediction intervals (PI). That is, given confidence level 1 a, we are interested in constructing prediction interval, Ca,N+1.1+1 that will cover YN+1.411 with probability of at least 1
crucial requirement in cross-sectional time series regression is distinguish two notions of validity, longitudinal and cross-sectional to ensure reliable performance. Longitudinal validity is concerned with validity along the temporal axis for each time series. On the other hand, pross-sectional validity is concerned with validity across the populational cross-section of the time series data. Figure illustrates both notions of validity and a standard real-world occurrence of such a problem setting. Recently, several research groups have explored adapting CP to the time series setting. Some of such works [17, 58, 56] focus only on longitudinal validity, which is extremely difficult without strong distributional assumptions [4]. Furthermore, such methods cannot leverage rich information inherent in the cross-section. The only work which addresses cross-sectional validity is to due to Stankovičiüit al. [48], but it ignores the temporal dependence. Accounting for both notions of coverage is critical to obtain reliable performance.
To remedy the above situation, we propose Temporal Quantile Adjustment (TQA) for CP in the cross-seas sessisectional time series regression setting. TQA the first method that can account for both cross- sectional and longitudinal validity simultaneously. Although TQA can be used as wrapper around any time series regression model, we focus on neural networks, as our main inspiration comes from complicated time series regression problems in healthcare?. Neural networks are particularly suited for such tasks, which can involve modeling the evolution of heterogeneous entities such as diagnostic and drug codes, patient and physician embeddings and regressing over target of interest. Taking inspiration from [17]. TQA adjusts the quantile to query at each time step in a eesorrtically-grounne manner. Based the nature of quantile adjustment,
also propose two variants of TQA, which further shed light on the generality of our method. The ability of TQA to handle both cross secstonectionl and longitudinal validity is borne out in extensive experimentation, where it significantly outperforms competing methods.
2 Related Work
Our work falls squarely within the Conformal Prediction (CP) framework. The original formulation of CP was in a purely transductive setting [46, 47. 53], and was computationally inefficient. More
We include results for other models in the Appendix.

---

efficient variants dubbed as Inductive Conformal Prediction (ICP) [39, 40, 52] were proposed soon after and were more broadly popularized by followup works in Statisties [29, 31, 32]. A similar idea, often referred to as Split Conformal [30], is now more or less used interchangeably with ICP, and is fundamental to our paper. The model-agnostic and distribution-fres nature of split conformal makes it suitable for large black-box models, and thus it has seen adoption in deep learning-baad bbaiin pipelines (e.g. [2, 11, 26, 33, 36, 13, 48)).
One of the mainstays of the CP framework is the assumption of exchangeability between calibration and test data. Extensions of CP "beyond exchangeability" have attracted relatively little attention until recently. A key publication in the area is [50] which used the notion of weighted exchangeability to handle covariate shift. Several notable works that address various aspects of the covariate shift problem include [21, 42, 57, 44]. More recent work [4, 17] handles gradual distribution drifts. [4] additionally proposes an extension of CP when the data-points cannot be treated in a symmetric manner. Extensions of the split conformal method to broad class of dependent processes such as sta- tionary 3-mixing processes was proposed by [38]. These works provide some general methodological insights to model temporal dependence in case, but are otherwise not directly related. In particular, the online adaptive method [17], served as a major source of inspiration for the development of a variant of TQA (dubbed TQA-E). The most related works have focus generating valid intervals in time series regression [17, 58, 4, 56]. These works end up ignoring cross-sectional aspects, which is understandable given the tasks they study. On the other extreme is [48], which focuses only cross-sectionss coverage, ignoring the temporal dimension in constructing Pls.
Various other techniques, outside the ambit of CP, have also been extended to quantify uncertainty in time series forecasting as well. For instance, approximate Bayesian methods [8. 54, 37, 35, 25, 15, 28] are quite popular for uncertainty quantification, and have been extended to RNNs [14, 7]. Finally, one may also use the idea of directly predicting the quantiles (as opposed to the point estimate) in regression tasks [49, 27], and applying it to time series forecasting [55, 16]. However, such methods usually require changing the base model and typically do not come with coverage guarantees.
Preliminaries
This section builds foundation for exposition of TQA. We begin by expounding further on longitudinal and cross-sectional validity, followed by presenting the exchangeability assumption. Finally, we discuss the use of split conformal prediction to construct (cross sectioalccio valid Pls.
3.1 Cross-sectional Validity vs Longitudinal Validity
Cross-sectional validity is the more common type of validity encountered in CP, being the only type of validity in non-time -imeeseriessssseeeeeeee settings. More formally:
Definition 1. Prediction interval C. is 1 a cross-sectionally valid if. for any t,
PS (Yn+1, E ČN41,) 2 21-a0 a.
PSx.. means the probability is taken over the randomness of SN-1- If ČN+1, i, is random (c.g. depends on { then the probability is taken over the randomness of Č 1, as well. Note that we consider the case where every time series only consists of one step (T = 1), then we recover the usual definition of marginal validity. Cross-sectional validity translates to high-probability in coverage for a randomly drawn time series. As we will see later, cross-sectional validity is casier to achieve, since we can assume inter-time-serics exchangeability.
Longitudinal validity, on the other hand, concerned with coverage along the temporal axis for particular TS. We use the following definition:
Definition 2. Prediction interval Č.,, is longitudinally valid if for almost every time-series SN+1 ~ Pss there exists a To such that:
t To = PYs (YN41, E CN4LE) 21 a
(2)
Here, the qualifier "almost every" means that the set of time series' for which such coverage may fail is of measure zero (under Ps). The threshold To allows some "time" for Ó' to potentially adapt the
3

---

temporal information in a particular TS. It should be clear that longitudinal validity is harder to attain, because we can no longer marginalize the probability over randomly drawn time series.
3.2 Conformal Prediction for Cross-sectionally Valid PI
In this section we explain how to use conformal prediction to construct cross-sectionally valid Pls. We first introduce exchangeability assumption, which is the central assumption in conformal prediction, and slightly weaker than the standard i.i.d assumption. More formally:
Definition (Exchangeabillty (53]) A sequence of random variables, Z1, Zz..... Zn E Z are exchangeable for any permutation π (1.2....,n1) + (1.2.....,1) and every measurable set E _ Z", we have
P[(Z1, Z2.... .n) E} = P(((7(1): ... (x(x)) E E}
(3)
Definition& 3 can naturally be extended to sequence of randomly drawn time series:
Definition 4. (Exchangeable Time Series) Given time series S1, S2.S2...., Sn where S; [Z,,..... Zi,',... ...b, denote Zi)(,, the concatenated random variable of (Zid...... Zietm )- Time series S1, ......, Sn are exchangeable if. for any finitely many t1 <... tm, the random are exchangeable.
variables 11, 1" (n,
As concrete example, suppose randomly pick 100 patients from hospital's 's EHR database for predicting readmission risk. It is fairly reasonable to assume that these time series are exchangeable, despite the obvious strong temporal dependence within each time series. Throughoul this paper, will assume S .,.... Sx1 are exchangeable time series.
We now explain how to construct cross- sessisnccti valid PIs. To construct PI for Yú,, we first split our data | into a proper training set and a calibration set [41]. The training set is used train models for the nonconformity score function, and the calibration set used collect such nonconformity scores (denoted as V-))). For example, one may train a mean estimator f (e.g. an RNN) on the training set, and use the absolute residual U 13t, Hi.tl. where Úi. A(X,,r; St.(-1). as the nonconformity score. Here S denotes [Z....... Zu.-1)) The idea behind the split conformal method that the scoring function (e.g. j only fit on the proper training
set, implying that nonconformity scores on the calibration set and #N+11+ are also exchangeable. Here on, for the sake of simplicity of notation, we will use { to denote the calibration set only, assuming all necessary models have already been trained.
Given nonconformity score V--) (possibly using some trained model (i) and set of exchangeable time series [++ - the split conformal method can be used to generate the following 1 a cross-sectionally valid prediction interval:
Coplit C++1++1 = V VN+L.+1(0.0) s @((1 a; (o u (Vxiieen(0011))
(4)
Here, Q(3; A) means the 3-quantile for the set A. As concrete example, if we let ug,: Mi. D.rl and employ the standard trick that replaces 1,+1 with do avoid plugging in (uncountably) many values for y [4], the prediction interval for N441441 becomes:
Ceplit N+1 = j0 0.9 o] where 0 o(1 (1aae4 trett 1 U {ox)))
(5)
easily show the cross-sectional validity of Csplitt Theorem 3.1. ([4. 53]) Esplit is a sess-sectionally valid (Def. 1).
Assuming exchangeability,
The intuition behind the proof is that the exchangeability of the time series translates to the exchange- follows
ability of the nonconformity scores, which means the rank of UN+11+11 among {1t,411 N+1 J a uniform distribution. The coverage guarantee in Theorem 3.1 then follows. Note that with a finite calibration set, the 1 - a-quantile could be ambiguously defined, and in practice one would use
N+1 [(1-a)(N+]] get slightly more conservative PI, "flip (biased) coin" to choose between [[1 [[1-)][++]] and [[=a))[+1]] for precise 1 coverage (e.g. the "smoothed" ICP in [53] the N+1
tie-breaking trick in [2]). In Section 4, we assume the precise coverage PI for the case of discussion.

---

While sosssseectonnlly valid, Õaplit ignores the temporal dependence in the nonconformity scores completely. We will explain Section 4 how to adapt to the temporal dependence by "quantile adjustment", thus improving longitudinal coverage as well.
4 Temporal Quantile Adjustment (TQA)
In Section 3 we discussed classical conformal prediction method and also highlighted its inherent limitations the time series setting. In this section we will formally introduce Temporal Quantile Adjustment (TQA), which queries quantiles differently than in the aforementioned split conformal procedures. We first explicate the goals and motivations of TQA in Section 4.1. Then, in Section 4.2 and 4.3 propose two principled adjustment methods along with theoretical analyses.
4.1 Improving Longitudinal Coverage
Although it is tempting to directly pursue distribution- free finite-sample PI estimator that achieves longitudinal coveruge guarantee (Def. 2). is likely too optimistic due to the fact that [Z,,,...., Zi.rl are not exchangeable and we cannot characterize them meaningfully without imposing (strong) distributional assumptions. For example, in [4], the bound for coverage gap-which captures the loss in coverage compared what is achievable under data becomes lime-dependent. In fact, concurrent work [34] proves that longitudinal validity as specified in Def 2 is impossible to achieve unless trivially, following similar proof techniques for the impossibility of distribution- -fre sinie-ssmple conditional validity [5, 32].
e becomes 1 as the
Nevertheless, longitudinal coverage may still be empirically improved if we can adapt to the temporal dependence. From our EHR example, suppose á has been giving high-error predictions for a patient for 8 out of the past 10 days, we might suspect high error going forward for this patient well. If this is indeed the case, then naively applying Cspplit can only attain low coverage for this patient, matter how long a history we observe. To address this, we propose to query different quantiles based on the partially observed time series.
From now on, we use Ca..,i.t to denote a PI for Y,, with pre-specified target coverage of 1 - ai,t+ in order to emphasize dependence on the "quantile to query". We let a;,t 8,,, where ò,, is the quantile adjustment. Classical split conformal PIs (e.g. [48]) entail a special case: Vi, Vl, di,, = O. We refer this method as Temporal Quantile Adjustment (TQA).
Now, denote the random variable Ri, as the rank/quantile of Vi, among {V W
N++1
N+1 = Hj : 0gae < v.tl
Fh.e == Q-'@@'''v (vis (
(6)
11
For example, if ug.r is the smallest among { N+1 then ra.t = O. Intuitively, we would like to use a more conservative (smaller) +N+ when we believe S as a whole is less "conformal" and RN+ is likely high. erucial observation is that there is no actual temporal dependence between the nonconformity scores (i.e. we are just adjusting a, based on some "noise"), then we do not lose any coverage as long as the expected adjustment is zero.
Theorem 4.1. If the nonconformity score's rank (RN41,t) is independent of the quantile adjustment 2 - a + B[88+11)).
(6x+1,t), then Psx (YN+1,t E
All proofs are deferred to the Appendix.
The assumption in Theorem 4.1 is not that S 1 itself is not temporally dependent, nor is the slightly weaker assumption, that of the temporal independence of either { or V The assumption in Theorem 4.1 only suggests that there is no temporal pattern in the prediction errors that 8 can capture. This could happen, for example, when the RNN captures the underlying data generating process fully, but only misses the random noise (aleatoric uncertainty). This could also happen our quantile adjustments (0) are pure noise.
Remark:
Unfortunately, although it can be tempting to conclude that E( = implies finite-sample cross- sectional validity, this conclusion would be incorrect, because it ignores the dependence between
5

---

Coverage: 80.0%
80.0%
Coverage: 84.2% 80.0%
Coverage: 80.2% 80.0%
0.8
0.8
0.8
1 d,
Realbbed
0.6
missed rank ft covered rank gained
02
0.2
0.2
sacrificed
0.2
0.4
0.6
1.0
0.2
0.4
0.6
0.8 0.0
0.4
0.8
10
Predicted f. Quantile Adjustment
Predicted Ft
Predicted P
TQA-B w/ Good i
TQA-B w/ Noise
Figure 2: Coverage profiles with hypothetical realized rank r condition on prediction f, with a = 0.2 for readability. (Y,, Cτ,1 + t S o,,,.) As f follows a uniform distribution, the proportion of dots below the red line represents the cross-scctional coverage probability. TQA-B generally improves coverage if f is correlated with the realized r (middle), and does not lose coverage otherwise (right). "Budgeting" refers to the constraint that sacrificed and gained have equal areas.
11y++1,1 E Cox-L.. -1.. and ON+1,I- However, we will next discuss how to perform quantile adjustment, and why it typically improves coverage.
4.2 Quantile Budgeting (TQA-B)
Theorem 4.1 provides an interesting constraint that we should consider while designing ò. That is, we should let B[Ø] O so as to keep the same coverage when we cannot predict the quantiles, but hopefully improving coverage when can. This suggests design of that refer to type of "budgeting". Although it is possible directly predict good 81,411, , adopt more principled two-step approach:
We predict quantile fr,e+1 which estimates ri,++1. We pre-defined mapping g to define the quantile adjustment 81,411 ((f,141;))
This also permits research to improve each component independently. We introduce one alternative for each step in the Appendix.
(i) Quantile Prediction: The quantile prediction fr,41 is estimated by a function of the form f(S..t) ( Since fi, is supposed to predict ri the rank of the nonconformity score, we impose the constraint that fr,4+1 should follow a uniform distribution over { 1N focus on a simple rank prediction method stated below:
Jj=o We will
Iyve Di,er g(t-1)
2 == Q (cc,e) ( where c,
(7)
Here, The is the exponentially decayed mean residual of time series up to time f, and we use 3 = 0.8. Note that taking the rank with Q-! achieves the uniformity requirement, and ε. could be replaced by any scoring function that takes into account the temporal information.
(ii) Budgeting: Given prediction Pr. propose the following adjustment 8,, 9 8( (f)
(r:a) = C(( (0fr-(1-00 (1-a)) (r < -a-aa a) 2r 1 - aa)
(2aN [aNN)((NN]] + 1)
where C' [(1-a) 2a) + 1 [aN])"
(8)
(1--(1-a))
TQA-BB Denote UN-11. the quantile to query by TQA-B. Our particular coefficient design ensures that: TQA-B TQA-B a. (Recall @x+1, = aa- (000a-g))) (B)))
Theorem 4.2. Using pmm and g Vt, Es
[a [+
Thus, by Theorem 4.1, TQA-B will not lose coverage if and independent. The following theorem provides a worst-case cross-sectional coverage guarantee, regardless of how "bad" is:
(a+ 2N 2 (1
Theorem 4.3. PS P
(YN+LHHL E TQQ-B C 2 a (1-att NN
worst-case loss

---

The worst-case loss term is typically small: about 0.012 for = 0.1 and N = 100, although it can also be also be high for a large a like 0.5. In practice, it is unlikely that f is worse than a random guess; the coverage is typically greater than 1 a, as we will see in the experiments and illustrated in Figure In the Appendix, we present a more aggressive quantile adjustment function g that provides weaker guarantee than Theorem 4.3, but empirically performs better.
Implementation Details To avoid creating infinitely-wide PIs, we could also let å = >g(f; a) so ag is bounded away from 0 (In our experiments we choose such that a, 2 0.01). Moreover, the specific form of C' presented in this section depends on the concrete distribution of f. For example, C' would take a different form if f is defined to be uniform over W 1++1 rather than { 1
Practically, we can simply let C a ax- regardless of N. The additional loss in Theorem 4.3 will become a²/(( /(1 - a), and the change in Theorem 4.2 is negligible for a reasonable value N.
4.3 Error-based adjustment (TQA-E)
Another simple quantile adjustment approach is using a heuristic that depends the past "errors": De- fine err, (Y, & C.,, ), and increase 6.41 (conservative) if we too many errors in (erre)esert compared with a, and vice versa. Since this approach does not depend on the cross-section, we drop the subscript 'N+1 for simplicity, We use the following update rule (with 80 = 0) inspired by [17]*:
66 ((rr, o) (8, a (otherwise)
8
( 71-
(9)
Note that we do not explicitly impose the restriction that a - 8, at E [0, 11, which means the PI could have infinite width. However, infinite-wide PI means error, so 8,41 will decrease and resume to a finite PI gradually.
As & depends on the entire error history, it is not immediately clear whether TQA-E is still valid with the assumption in Theorem 4.1. Below we state "no-worse" type theorem for TQA-E:
Theorem 4.4. If we assume the nonconformity score's 's rank has no temporal dependence, then
vt, ESEEES NN11 5 $
(RAAEE1
(10)
Thus, ÖTQA-E is finite-sample cross-sectional 1 a valid following Theorem 4.1.
Finally, an asymptotic longitudinal validity result can also be shown for long time series":
Theorem 4.5. (Asymptotic Longitudinal Coverage) For any time series S, limm ET erre < a.
Remarks: Although Theorem 4.5 seems to suggest some sort of longitudinal validity. does not contradict the hardness claim in Section 4.1, because TQA-E achieves this via infinitely-we -wide PIs. We also refer interested readers to [58] as an example of an different heuristic based on errors a single time series. However, it will require further modifications for finite-sample cross-sectional validity.
Experiments
In this section, our goal is to verify the following empirically:
1. TQA maintains sectionectional coverage and achieves competitive PI efficiency. Ignoring temporal dependence in naive split conformal prediction leads to low longitudinal coverage for some TS. 'QA improves longitudinal coverage.
'Despite the similarity the surface, [17] has notion of cross-section.
"The update rule as shown in Eq. 9 creates an asymmetry to accommodate for the asymptotic guarantee in Theorem 4.5, which is why the expectation in Theorem 4,4 is not equality but an inequality. If asymptoiic coverage is not a concern (because is small), one could use (1 - 7)6, when a - = a, < as well, in which case Theorem 4.4 becomes an equality, as we discuss in the proof for Theorem 4.4 in the Appendix. In practice (as observe in experiments), the difference in behavior is negligible.
7

---

Baselines: We use the following bate-of-thre te--hhean art baselines for PI construction: Conformal forecasting
RNN (CFRNN (Split)) [48]), a direct application of split-conformal prediction [53]"; Quantile RNN (QRNN) [55], which directly predicts the two endpoints (represented by two quantiles) of the Pl; RNN with Monte-Carlo Dropout (DP-RNN) [15]; Conformalized Quantile Regression with QRNN (CQRNN) [45], which, as the name suggests is a conformalized version of quantile regression;
Locally adaptive split conformal prediction (LASplit) [29], which uses a normalized absolute error as the nonconformity score (we follow the implementation in [45]).
Table I: Number of TSs in each dataset along with the length.
Properties
MINIC
CLAIM
COVID
EEG
CEFCom/GEFCom-1
# train/cal/test
192/100/100
2393/500/500
200/100/80
300/100/200
1198/200/700
''' (length)
30
30
30
63
24
CLAIM
Datasets We test our methods and baselines on the following datasets: Electronic health records data for white blood cell counts (WBCC) prediction (MIMIC [23, 18, 22]). COVID-19 cases prediction (COVID [10],, Electroenceephallgrrpyy trajectory pre- diction after visual stimuli (EEG [51]), energy load fore- casting (GEFCOM [20]), and healthcare claim amount prediction (CLAIM) using data from a large American healthcare data provider. Among these, we mostly fol- low [48] in preparing MIMIC, COVID and EEG. Note that GEFCom is originally single time series (hourly obser- vations for years). Therefore, we treat each day as a single TS, and perform a strict temporal splitting (test data is preceded by calibration data, which is preceded by the training data), which means exchangeability is broken. We also include GEFCom-R R (andom) version that preserves the exchangeability by ignoring the tem- poral order in data splitting. Table 1 provides brief summary of the data. Due to space constraints, the details for each dataset are relegated to the Appendix.
TOMB
TRAE
0.4
CFRNS
00HNN
Lt
0.2
DPRNN
0.0
0.00
0.02
0.04 0.06 0.08 0.10
Figure 3: Coverage rate (Y-axis) vs the percentile among all test TS (X-axis, with zero meaning the least-covered TS) for the 10% least-covered TS in CLAIM. The bands denote the center 80% realizations. TQA-E has an natural advantage by using infinitely Pls. However, even TQA-B still significantly improves the longitudinal coverage rate over all baselines.
8
GEFCom-R
normalizdd [yrر
Evaluation Metries and Experiment Setup We use RNN as the base point estimator due to its flexibility and for comparison with [48]. We use a 0.1, and LSTM ([19)) similar to that in [48] (full implemen- tation details in the Appendix). For TQA-E, we use 7 0.005 following [17]. For each dataset, we repeat the prediction task 50 times, and report the mean and standard deviation of the average coverage rate, tail coverage rate, and inverse coverage efficiency for the last 20 time-steps. Here, tail coverage rate means the average coverage rate of the least-covered 10% of the time series. A high tail coverage rate thus implies bet- ter longitudinal coverage. Inverse coverage efficiency is measured by the average PI width divided by the marginal coverage rate (the smaller the better). Since TQA-E could create infinite Pls, replace with 2x the widest finite PI. We also include in the Appendix results on full time series, and with Linear Regression instead of LSTM to show the model-agnostie nature of TQA.
6
4
EEG
CLAIM
2
GEFCom
0
0.0
0.2
0.4
0.6
0.8
1.0
percentile
Figure 4: Sorted absolute residuals (ly
@) for t = T 10. Each dataset is normal- ized so the mean of the residuals is 1. To cover extreme values, even if allowed to "sacrifice" some less extreme values, the PI on average is expected to get much wider. It is thus surprising that TQA-B could im- prove both efficiency and the tail coverage.
Note that although we use equal-length time series in these experiments, for data with variable lengths (such CLAIM or MIMIC), one could filter the calibration set before querying the quantile. As long as we assume exchangeability conditioning on length, all theoretical analysis still holds.
'[48] suggests performing Bonferroni correction to jointly cover the entire horizon (all 2 steps). We explain in the Appendix why this problematic.
8

---

Results The results are presented in Tables 2, 3, and 4. In Table 2, we verify that conformal prediction methods- CFRNN (Split), CQRNN, LASplit and TQA- are empirically soct-nnccy valid. The non-conformal methods (QRNN and DPRNN) have unreliable coverage. In Table 3, we show that TQA can greatly improve the (longitudinal) average coverage rate for the worst TS. (TQA-E is consistently better than TQA-B due to the presence of infinitely-wide Pls.) This is also visualized in Figure 3. Note that although CQRNN and LASplit do not perform quantile adjustment, they model uncertainty directly, which also helps improve the longitudinal coverage but is less robust. In Table 4, we verify that TQA did not achieve better coverage simply by using very wide PIs (which is however the case for DPRNN GEFCom). This is somewhat surprising because from Figure 4, the marginal gain in coverage decreases fast as ag decreases. The PIs for TQA-B should be wider due to the slight over-coverage, and the convexity (with any quantile adjustment). This suggests that TQA-B performs the budgeting very efficiently to cancel out both effects. The efficiency of TQA-E seems low due to the infinitely-wide PIs (replaced by 2x maximum finite width in this computation), but will see that it generates mostly finite Pls, and the median width is still competitive.
Finally, we would like to also emphasize that any nonconformity scores could theoretically be combined with TQA. In this paper, and in our experiments, we mostly tried to combine the simplest nonconformity scores used in CFRNN (Split) with TQA. The question of how to combine TQA with other nonconformity scores (such as those used in CQRNN or LASplit) is left for future research.
Table 2:
Average coverage rate. Empirically valid methods are in bold (at p 0.01). As expected, conformal baselines are valid, while others (QRNN and DPRNN) are not. Note that GEFCom does not satisfy the exchangeability assumption, causing invalid coverage for most conformal methods However, TQA still outperforms all conformal baselines, with TQA-E still valid.
Coverage
TQA-B
TQAE
CFRNN (Split)
CQRNN
LASphi
QRNN
DPRNN
NIMIC
91.21-1332
91.19:10.48
90.06 1.73
10.15::1.24
90.33-1154
86.9011
46.30
CLAIN
91.19=0.49
91.56=20.35
90.21=0.56
90.15-t0.68
9..20=0.64
85,90=+0,78
24.79=t0.85
COVID
90.79:11.45
91.73-10.85
90.25=1.69
90.08:t1.62
90.18=1.466
89.19:t1.554
67.51#3.766
BBG
9.273:1..11
90.63:10.75
89.92
19.99.11.76
89.80
87.96.10.88
3..24:1:1.30
GBPCOn
89.58:10.25
90.94:10.14
88.61 0.
89, 19.16:10.17
88 98.96.10.11 18
80.40.1 1.36
89.
09.5001:0.73
GEFCon-R
90.56:20.64
90.72:10.45
89.92=0.78
90.07-1:0.63
89.95:00.2
85,49
91.03:100.6
Table 3: The tail coverage rate (mean longitudinal coverage for the least-covered 10% TS), the higher the better. The best method is in bold, and the best method without using any infinitely-wide PI is underscored. Both versions of TQA consistently outperform all baselines.
Tail Coverage Rate f
TQA-B
TQA-E
CFRNN (Split)
CQRNN
LASplit
QRNN
DPRNN
MINIC
4..59
1..6
62.22:7..09
68.6013..84
65.05 12
61.80:13911
17.24 38
7..16=11.22
81.53:10.77
65.95=1.88
66.45+3.119
08.04:2..44
53.89+3.559
1.65+0.54
CDVID
4:4.5
82.39
64.41
66.41
67.38-4463
65. 15
36.65-5..6 63
EBC
2.199=2.18
1..03++1.22
64.14=3.42
61.95+4.71
67.13+2..32
57.82-42.78
12.99=
GRFCon
68.96-1 1.70
81.77-10-36
58.49 1.38
1.56
60.46:1.66
47.56-12.27
1.64
GEFCon-R
75.28=11.28
81.80-10.69
68. 28.7622.18
1..95::1.66
10.
64.99:11.92
71.88 1.78
Table 4: Inverse Efficiency, measured by the mean PI width divided by the coverage rate. Since
TQA-E can create infinite PI, the width is computed by replacing oo with 2x the maximum finite PI width. The most efficient (and valid) method is in bold (p- (p-valu=00.01). As can see, TQA-B is highly competitive in efficiency.
Inverse Efficiency 4
TQA-B
TQA-E
CFRNN (Split)
CQRNN
LASplit
QRNN
DPRNN
MINIC
1.990-10.165
2.382-00265
1.964:201770
1.738:10.145
2.072-:0.23
16623+0.116
1.258:001132
CLAIM
3 3.000=
3.279-0.074
3.003:0.052
2.902-t0.044
3.009+ 0..4
2.00
2.401
COVID
0.831
L167:10.337
0.826::0.04
0.908:100091
0.826:10.037
0.098
0.744=0.050
BBG
1.449 0.025
1.74910.125
1.445 1.445:0.003 0.031
1.586.10.022
1.442.0.005
1497.10.044
1.061 1.001
GEFCon
0.238-50.005
0.280-100113
0.205
0.242-1..005
0.238-L0.005
0..015
0.636-10.009
GEFCon-R
0.200-100004
0.222+0.010
0.198=0.004
0.207=000004
0.201=000004
0.193:+0.004
0.590=000009
6 Conclusions
In this paper, we proposed Temporal Quantile Adjustment, or TQA, to quantify uncertainty (create prediction intervals) in time series forecasting with a cross-section. TQA belongs to the framework of conformal prediction, and the main idea is to adjust the quantile to query using temporal information collected so far. This allows TQA to work with any model and any nonconformity score design.

---

TQA theoretically is "no-worse" in cross- rros-secctinal coverage than vanilla split conformal as long as the expected value of adjustment is zero, and empirically improves the coverage. We also proposed two variants, TQA-B and TQA-E, both of which significantly outperform baselines in improving temporal/oongindinal coverage across many real world datasets. We hope that this work will serve as a foundation for the future design of PIs with both high cross-sectional and temporal coverage.
7
Acknowledgment
This work was supported by NSF award SCH-2205289, SCH-2014438, IIS- 1838822,,, NIH award R01 IROINS107291-01.
References
[1] Ahmed Alaa and Mihaela Van Der Schaar. Frequentist uncertainty in recurrent neural networks via blockwise influence functions. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 175-190. PMLR, 13-18 Jul 2020.
[2] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncer- tainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021.
[3] Anastasios Nikolas Angelopoulos, Amit Kohli, Stephen Bates, Michael I. Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution free uncertainty quantification and applications imaging. ArXiv, abs/2202.05265. 2022.
[4] Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability, 2022.
[5] Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan Tibshirani. The limits of distribution-free conditional predictive inference. arXiv, abs//1903.r 046644, 2020.
[6] Stephen Bates, A. Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution- free, risk-controlling prediction sets. J. ACM, 68:43:1 13::34, 2021.
[7] Jose Caceres, Danilo Gonzalez, Taotao Zhou, and Enrique Lopez Droguett. A probabilistic bayesian recurrent neural network for remaining useful life prognostics considering epistemic and aleatory uncertainties. Structural Control and Health Monitoring, 28(10);e2811, 2021.
[8] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32:2 of Proceedings of Machine Learning Research, pages 1693, Bejing, China, 22-24 Jun 2014. PMLR.
[9] Isidro Crirés-CCriano and Andreas Bender. Concepts and applications of conformal prediction in computational drug discovery. ArXiv, Abs/1198.03350 2019.
(10) Coronavirus 19) in the uk. https://tttps://orrravviii dattrraaiiiuusdaaaaaaaaaa.aaa dat..gov. Accessed: 2022-04-14.
gov. gov.uk/, 2022.
[11] Clara Fannjiang, Stephen Bates, Anastasios Nikolas Angelopoulos, Jennifer Listgarten, and Michael Jordan. Conformal prediction for the design problem. ArXiv, abs/2202.03611,, 2022.
[12] Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Efficient conformal prediction via cascaded inference with expanded admission. In ICLR, 2021.
[13] Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Few-shot conformal pre- diction with auxiliary tasks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3339-3339. PMLR, 18-24 Jul 2021.
10

---

[14] Meire Fortunato. Charles Blundell, and Oriol Vinyals. Bayesian recurrent neural networks. CoRR, abbs//704,02798, 2017.
[15] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In 33rd International Conference on Machine Learning, ICML 2016, 2016.
[16] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function mns. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1901-1910. PMLR, 16-18 Apr 2019.
[17] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[18] A. L. Goldberger, L. A. Amaral, Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 2000.
L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E.
[19] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(3):1735-178,, nov 1997.
[20] Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, and Rob J. Hyndman. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond. International Journal of Forecasting, 32(3):896 913, 2016.
[21] Xiaoyu Hu and Jing Lei. A distribution-free test of covariate shift using conformal prediction. arXiv: Methodology, 2020.
[22] A. Johnson, T. Pollard, and R. Mark. Mimic-iii clinical database demo (version 1.4). https: 2019.
//archive. ics. uci.
[23] Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC- III, a freely accessible critical care database. Scientific Data, 3(1):160035, 2016.
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[25] Diederik Kingma and Max Welling. Auto-encoding variational bayes. Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014.
[26] Danijel Kivaranovic, Kory D. Johnson, and Hannes Leeb. Adaptive, distribution- prediction intervals for deep networks. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 4346-4356. PMLR, 26-28 Aug 2020.
[27] Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica, 46(1):33-50, 1978.
[28] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
[29] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association, 2018
[30] Jing Lei, Alessandro Rinaldo, and Larry Wasserman. conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence, 74(1):29-4) 43, 2015.
11

---

[31] Jing Lei, James M. Robins, and Larry A. Wasserman. Distribution-frue free prediction sets, Journal of the American Statistical Association, 108:278 287, 2013.
[32] Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Sociery: Series B (Statistical Methodology), 76(1):71-96, 2014.
[33] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Locally valid and discriminative prediction intervals for deep learning models. In M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8378-8391. Curran Associates, Inc., 2021.
[34] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Conformal prediction intervals with temporal dependence. Transactions of Machine Learning Research, 2022.
[35] Christos Louizos and Max Welling. Multiplicatiive normalizing flows for variational Bayesian neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th Inter- national Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2218-2227. PMLR, 06-11 Aug 2017.
[36] Sergio Matiz and Kenneth E. Barner. Inductive conformal predictor for convolutional neural networks: Applications to active learning for image classification. Pattern Recognition, 90:172- 182, 2019.
[37] Radford Neal. Bayesian learning via stochastic dynamics. In S. Hanson, J. Cowan, and C. Giles, editors, Advances in Neural Information Processing Systems, volume Morgan-Kann Kaufmann, 1992.
[38] Roberto I. Oliveira, Paulo Orenstein, Thiago Ramos, and João Vitor Romano. Split conformal prediction for dependent data, 2022.
[39] Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural net- works. In Paula Fritzsche, editor, Tools in Artificial Intelligence, chapter 18. IntechOpen, Rijeka, 2008.
[40] Harris Papadopoulos, Kostas Proedrou, Vladimir Vovk, and Alexander Gammerman. Inductive confidence machines for regression. In ECML, 2002.
[41] Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Tapio Elomaa, Heikki Mannila, and Hannu Toivonen, editors, Machine Learning: ECML 2002, pages 345-356, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
[42] Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. Pac prediction sets under co- variate shift. In Proceedings of the 10th International Conference on Learning Representations. PMLR, 21-25 April 2022.
[43] Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In UAI, 2021.
[44] Hongxiang Qiu, Edgar Dobriban, and Eric Tchetgen Tchetgen. Distribution-free prediction sets adaptive unknown covariate shift, 2022.
[45] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[46] Craig Saunders, Alexander Gammerman, and Vladimir Vovk. Transduction with confidence and credibility. In IJCAI, 1999.
[47] Craig Saunders, Alexander Gammerman, and Vladimir Vovk. Computationally efficient trans- ductive machines. In ALT, 2000.
[48] Kamilè Stankevičiüté, Ahmed Alaa, and Mihaela van der Schaar. Conformal time-series forecasting. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
12

---

[49] Ingo Steinwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 17(1):211 225, 2011.
[50] Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In NeurIPS, 2019.
[51] Eeg database. https://archive www.:iis.........o.oo uci.cii... www.uuc..c..m/eeeeee htt.p//.t/..mm.aaaaaaae/////aaaa cessed: 2022-04-23.
1999. Ac-
[52] Vladimir Vovk. Conditional validity of inductive conformal predictors. Machine Learning, 92:349-376, 2013.
[53] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer US, 2005.
[54] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. Proceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011.
[55] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi- horizon quantile recurrent forecaster, 2017.
[56] Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11559-11569, PMLR, 18-24 Jul 2021.
[57] Yachong Yang, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Doubly robust calibra- tion of prediction sets under covariate shift, 2022.
[58] Margaux Zaffran, Aymeric Dieuleveut, Olivier Féron, Yannig Goude, and Julie Josse. Adaptive conformal predictions for time series, 2022.
[59] Jin Zhang, Ulf Norinder, and Fredrik Svensson. Deep learning- based conformal prediction of toxicity. Journal of chemical information and modeling, 2021.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section...
Did you include the license to the code and datasets? [No] The code and the data are proprietary. Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] TQA could theoretically lead to worse (although empirically better) coverage, as discussed in Section 4.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not foresee any potential negative societal impact of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Apart from exchange- ability of time series (which we assume throughout this paper), any assumption is stated right before the relevant theorem.
(b) Did you include complete proofs of all theoretical results? [Yes] In the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] In the supplemental material and will be published if accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the Appendix.
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] We repeat all experiments 50 times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] Most datasets are publicly available, except for CLAIM which is proprictary and provided to us by a company.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] They have been de-identified if applicable.
If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "r1P-y9zi3fn", "reviewer": "Reviewer_2Rxg", "summary": "Temporal Quantile Adjustment (TQA) is proposed for time-series data with a focus on constructing efficient and valid prediction intervals (PIs) for regressing the models with two distinct notions of coverage: cross-sectional coverage and longitudinal/temporal coverage. TQA adjusts the quantile to query in Conformal Prediction at each time t, accounting for both cross-sectional and longitudinal coverage in a theoretically-grounded manner. The theory part supports the validity of coverages. Two variants are also provided for generality of the TQA, following the nature of quantile adjustment.", "questions": "It is probably out of the scope of this paper, but would there be a method to deliver a target coverage when possible?", "limitations": " ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Intends to resolve a critical challenge in conformal prediction, that is to construct efficient valid predictions from both cross-sectional and longitudinal perspectives.\n- Clear writing and contribution to the Conformal Prediction problem\n- Solid experiment results supporting the methodology in Appendix, Fig 8,9\n\nWeakness:\n- lack of guarantee or operational guidance for improving the coverages", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YSpXXPQI_Bh", "reviewer": "Reviewer_QQnG", "summary": "The paper develops methods to construct predictive intervals on regression for cross-sectional time series data that come with valid coverage guarantees. The method developed in the paper focus on two types of validity: cross-sectional validity  where the randomness Is the randomness over time series, and longitudinal validity where the randomness is time series specific and is typically temporal dependent. While cross-sectional validity can be achieved following the standard conformal inference approach, achieving the longitudinal validity is the harder one. The paper suggests temporal quantile adjustment to achieve longitudinal validity via quantile budgeting (TQA-B) and error based adjustment (TQA-E). The authors give error bounds for coverage and also support their claim empirically.", "questions": "I do not have additional questions.", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The underlying problem studied in the paper is extremely relevant to the machine learning community.\n2) The paper is supported both theoretically and experimentally showing efficacy of the methods developed.\n3) Overall the paper is well written.\n4) The ideas developed in the paper are novel.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RArIxbXL0v1", "reviewer": "Reviewer_me7X", "summary": "The paper considers the task of predictive inference with panel data consisting of a time dimension and sample dimension. Two types of coverage guarantees are considered here--- longitudinal coverage and cross-section coverage. In contrast to previous works that focus on either of the coverage guarantees, this paper aims at achieving both (at least empirically). The authors leverage the conformal inference technique, adapting it to the temporal structure. Under certain conditions, the proposed method reserves the cross-section coverage ensured by the conformal inference method; by incorporating the temporal structure, the method(s) achieves better longitudinal coverage empirically.\n", "questions": "1. Line 35: is there a comma missing between $X_{i,t}$ and $Y_{i,t}$\n\n2. Line 36: I assume that the prediction interval also depends on \n$X_{N+1, t+1}$ right? Should it be \"Given data $[Z_{N+1,j}]^t_{j=1}$ and $X_{N+1, t+1}$\"? \n\n3. Line 124:  should the \"...\" after $Z_{i,T}$ be removed?\n\n4. Line 132: using both the terms \"training data\" and \"training set\"\ncould be confusing.\n\n5. Line 208: is there any reason why $\\beta$ is taken to be $0.8$?\n\n6. Line 214: the notation $f(S_{i,:t};[S_{j,:t}]^{N+1}_{j=1})$\nis confusing.\n\n7. Line 222: is the quantity $\\alpha^{\\rm TQA-B}_{N+1,t}$ formally defined?\n\n8. line 293: what does it mean to repeat the prediction 50 times? Is the randomness from sample splitting? If so, how are the time series selected?\n\n9. Fig3: what is the coverage in the y-axis?\n\n10. Appendix A1: $\\hat{r}_{N+1,t} = \\lceil (1- \\alpha + \\hat{\\delta}_{N+1,t})(N+1) \\rceil$?\n\n\n\n\n", "limitations": "The authors have thoroughly addressed the limitations of their work and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is very well-written: there is a clear motivation, and a thorough introduction to the related works and their limitations; the proposed method is presented with explanations and intuitions. I enjoyed reading this paper.\n\n2. The research question raised is interesting and can be relevant to practitioners.\n\n3. The proposed method(s) achieves satisfactory empirical performance compared with competitors.\n\nWeaknesses/room for improvement\n\n1. The theoretical guarantee of the proposed method is a bit weak. Even for cross-section coverage, the guarantee (Theorem 4.1) holds under a fairly strong condition (as in the examples given in the remark after Theorem 4.1), although this is mitigated by the lower bound given in Theorem 4.3. There is no guarantee for longitudinal coverage (admittedly, this is impossible w/o further assumptions, but it might be worth understanding it under mild conditions).\n\n2. It might be helpful to present simulation results. With synthetic data, it is possible to directly evaluate the validity of the proposed method and compare it with other competitors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PaIDnePErLu", "reviewer": "Reviewer_HtoR", "summary": "This paper proposes a new approach called temporal quantile adjustments (TQA) that can improve longitudinal coverage and preserve cross-sectional coverage for the prediction interval built for regression on cross-sectional time series data.\n\nThe definitions of cross-sectional validity and longitudinal validity of a prediction interval are first presented in Eq (1) and (2). The existing split conformal method is reviewed and it can generate a cross-sectionally valid prediction interval. However, it cannot be directly applied to achieve  longitudinal validity as the exchangeability assumption is not satisfied.\n\nThe proposed TQA technique is introduced in section 4. Theorem 4.1 proved the coverage probability is lowered bounded by 1 - \\alpha + E[\\hat{\\delta}_{N+1,t}]. Motivated by Theorem 4.1, TQA-B (quantile budgeting) is introduced in section 4.2 to make the E[\\hat{\\delta}_{N+1,t}] close to zero. TQA-B consists of 1) quantile prediction, and 2) budgeting. The error-based adjustment (TQA-E) is introduced to increase \\delta_{t+1} if there are many errors in the past. \n\nExperimental results show that while existing conformal baseline approaches can achieve cross-sectional validity. TQA-E can achieve longitudinal validity besides cross-sectional validity.", "questions": "1) In the definition of longitudinal validity (Eq. 2), the distribution on the RHS is Y_{N+1, t} | S_{N+1,:t-1}, shouldn't it be S_{N+1, t} | S_{N+1,:t-1} instead? Is S_{N+1, t} a random variable in this case?\n\n2) can the author explain why the two-step approach in TQA-B can make E[\\hat{\\delta}] close to 0?\n", "limitations": "I didn't see major potential negative societal impact of this work.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\n1) the major contribution of this work, i.e., improving longitudinal coverage and preserve cross-sectional coverage for the prediction interval built for regression on cross-sectional time series data, seems novel.\n2) the proposed approach has solid theoretical guarantees.\n3) experiment results support the major claim in the paper.\n\nWeaknesses\n---------------\nI cannot find major weakness", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept