Conformal Prediction Intervals with Temporal Dependence
Authors: Zhen Lin
Shubhendu Trivedi
Jimeng Sun
Venue: Accepted by TMLR ()

Abstract: Cross-sectional prediction is common in many domains such as healthcare, including forecasting tasks using electronic health records, where different patients form a cross-section. We focus on the task of constructing valid prediction intervals (PIs) in time series regression with a cross-section. A prediction interval is considered valid if it covers the true response with (a pre-specified) high probability. We first distinguish between two notions of validity in such a setting: cross-sectional and longitudinal. Cross-sectional validity is concerned with validity across the cross-section of the time series data, while longitudinal validity accounts for the temporal dimension. Coverage guarantees along both these dimensions are ideally desirable; however, we show that distribution-free longitudinal validity is theoretically impossible. Despite this limitation, we propose Conformal Prediction with Temporal Dependence (CPTD), a procedure that is able to maintain strict cross-sectional validity while improving longitudinal coverage. CPTD is post-hoc and light-weight, and can easily be used in conjunction with any prediction model as long as a calibration set is available. We focus on neural networks due to their ability to model complicated data such as diagnosis codes for time series regression, and perform extensive experimental validation to verify the efficacy of our approach. We find that CPTD outperforms baselines on a variety of datasets by improving longitudinal coverage and often providing more efficient (narrower) PIs.

Published in Transactions on Machine Learning Research (09/2022)
Conformal Prediction Intervals with Temporal Dependence
Zhen Lin
University of Blinois at Urbono- Champpign Urbana, IL 61801
ahenling@ill min.gelllnoosaa edu
Shubhendu Trivedi*
shubhendu@csari L. mit. edu
Jimeng Sun
University of Illinois at Urbana- Champpign Urbana, IL 61801 i.meng@illémois. edu
Reviewed on OpenReview: http://////////o///////////////a
Abstract
Cross-sectiee sessssectiioal prediction common in many domains such as healthcare, including fore- casting tasks using electronic health records, where different patients form cross-section. We focus the task of constructing valid prediction intervals (PIs) in time series regression with cross-section. A prediction interval is considered valid if it covers the true response with (a pre-specified) high probability. We first distinguish between two not of validily in such a setting: pross-sectional and longitudinal. Cross-sectional validity is concerned with validity across the cross-section the time scries data, while longitudinal validity accounts for the temporal dimension. Coverage guaranlees along both these dimensions are ideally desirable; however, we show that distribution -feee longitudinal validity theoretically impossible. Despite this limitation, we propose Conformal Prediction with Temporal De- pendence (CPTD), procedure that is able to maintain strict cross-scctional validity while improving longitudinal coverage. CPTD is post-hoc and light- and can easily be used in conjunction with any prediction model as long as a calibration set is available. We focus on neural networks due to their ability to model complicated data such as diagnosis codes for time series regression, and perform extensive experimental validation to verify the efficacy of our approach. We find that CPTD outperforms basclines on a variety of datasets by improving longitudinal coverage and often providing more efficient (narrower) Pls. Our code is available at https: // gighttt........ wtw.thtu.ooooooeeeeeooommoooottt 2/i////PII
1 Introduction
Suppose we are given N independent and identically distributed (i.id) or exchangeable time series (TS), denoted [ Assume that each S, is sampled from arbitrary distribution Ps, and consists of Each Zí, is pair (X,,, V.,)) comprising of covariates X₁, E I and the response Y,, E IR. Given data {2N+1. until time t for new time scries SN+1, the time series regression problem amounts to predicting the response YN+1,41 al an unknown time + An illustrative example is predicting the white blood cell count (WBCC) a patient after she is
temporally-dependent observations S, = [?,, 7.tt.... 7.
*During the initiation and pursuance of this research, author's primary affiliation was MIT.

---

Published in Transactions on Machine Learning Research (09/2022)
Long itudinally ly Invalia
Longitudinally Invalid
Longitudinally Valid
missed y covered y
Cross Pross-sectional invalid
Crus-sectonally Valid
Cruss-sectinally Valli
Figure 1: The figure illustrates cross-scctional validity vs. longitudinal validity, which can be seen as inter- and intra- time series coverage guarantees. The black curves are predictions by the model, and the shaded blue bands denote the Pls. Red crosses are the ground-truth not covered by Pls, while blue dots are the ground-truth y which are covered. Idcally, we want a small number of red crosses (i.e., misses) that are randomly distributed across samples sertionlcttonnlly valid) and along the time dimension within each TS (longitudinally valid). The leftmost illustration (A) features Pls that are not valid in either sense i.e. Y is never covered, neither across time series nor across time within a single time series. (B) shows scenario with cross-sectional validity: for any t, the majority of TS are covered. It is however longitudinally invalid, because the PI of some TS has zero coverage. C (right) shows both cross-scctional and longitudinal validity.
administered an antibiotic. In such case, X could include covariates such as the weight or blood pressure of the i-th patient / days after the antiobiotic is given, and Yú, the WBCC of this patient.
While obtaining accurate point forecasts is often interest, our chief concern is in quantifying the uncertainty of each prediction by constructing valid prediction intervals (PI). More precisely, we want to obtain an interval estimate Ci,t C R, that covers Y2,, with a pre-selected high probability (1 a). Such Ct,t is generated by an interval estimator C. utilizing available training data, We focus on scenarios with both cross-sectional and time series aspects such as electronic health record data (such as in Stankevičiuté al. (2021)), where different patients together form cross-section. In such setting there are two distinct notions of validity: cross-scctional validity and longitudinal validity. These notions are illustrated in Figure 1. Cross-scctional validily is a lype of inter-lime series coverage requirement, whereas longitudinal validity focuses coverage along the temporal dimension in an individual time series. An effective uncertainty quantification method should idcally incorporate both notions satisfactorily.
In general, conformal prediction, owing to ils distribution- and model-agnostic nature, has gradually seen wider adoption for complicated models such as neural networks (Fisch et al., 2021; Angelopoulos al., 2021; Bates et al., 2021; Lin et al., 2021; Zhang et al., 2021; Cortés-Ciriant & Bender, 2019; Angelopoulos et al., 2022). In the time series context, recent research effort, including Gibbs & Candes (2021); Zaffran et al. (2022); Xu & Xie (2021), has focused on obtaining PIs using variants of conformal prediction. However, these works invariably only consider the target TS, ignoring cross-sectional information along with the attendant notion of coverage. Moreover, such methods typically provide no longitudinal validity without strong distributional assumptions. The work of Stankevīčiūt et al. (2021), which also uses conformal prediction, is the only mothod that operates in the cross-scctional setting. However, Stankevičiūté et al. (2021) ends up ignoring the temporal information while constructing PIs at different steps. On a different tack, popular (approxinately) Bayesian methods such Chen et al. (2014); Welling & Teh (2011); Neal (1992); Louizos & Welling (2017); Kingma & Welling (2014); Gal & Ghahramani (2016); Lakshminarayanan et al. (2017); Wilson & Izmailov (2020) could also be adapted to time series contexts (Fortunato et al., 2017; Caceres et al., 2021). However, such methods require changing the underlying regression model and typically provide no coverage guarantees.
A method to construct valid PIs that can handle both aforementioned notions of validity simultancously, while preferably also being light- and post-hoe, is missing from the literature. In this paper, we fill this gap by resorting to the framework of conformal prediction. Our contributions are summarized as follows:
2

---

Published in Transactions on Machine Learning Research (09/2022)
We first dissect coverage guarantees in the cross-sectional time series setting to shed light on both cross- sectional and longitudinal validity. We show that longitudinal coverage is impossible to achieve in a distribution-free manner.
Despite the impossibility of distribution-free longitudinal validity, we propose a general and effective procedure (dubbed Conformal Prediction with Temporal Dependence or CPTD for short) to incorporate temporal information conformal prediction for time scries, with a focus to improve longitudinal coverage. We theoretically establish the cross-sectional validity of the prediction intervals obtained by our procedure. Through extensive experimentation, we show that CPTD is able to maintain cross-sectional validity while improving longitudinal coverage.
2
Related Works
Work most related to ours falls along a few closely related axes. We summarize some such work below to contextualize our contributions.
Bayesian Uncertainty Quantification a popular line of research in uncertainty quantification for neural networks. While the posterior computation is almost always intractable, various approximations have been proposed, including variants of Bayesian learning based on Markov Chain Monte Carlo Chen al. (2014); Welling & Teh (2011); Neal (1992), variational inference methods Louizos & Welling (2017); Kingma & Welling (2014) and Monte-Carke Dropout Gal & Ghahramani (2016). Another popular uncertainty quantification method sometimes considered approximately Bayesian is Deep Ensemble Lakshminarayanan et al. (2017). Bayesian methods have also been extended to RNNs Fortunato et al. (2017); Caceres et al. (2021). The credible intervals provided by approximate Bayesian methods, however, do not provide frequentist coverage guarantocs. Morcover, modifications to the network structure (such as the introduction of many Dropout layers), which could be considered additional constraints, could hurt model performance. In contrast such methods, CPTD comes with provable coverage guarantees. More specifically, it is cross-sectionally valid, and improves longitudinal coverage. CPTD is also post-hoc, and does not interfere with the base neural network.
Quantile Prediction methods directly generate a prediction interval for each data point, instead of providing a point estimate. Such methods typically predict two scalars, representing the upper and lower bound for the Pls, with a pre-specified coverage level - a. The loss for point estimation (such as MSE) is thus replaced with the "pinball" /quantile loss Steinwart & Christmann (2011); Koenker & Bassett (1978), which takes a as a parameter. Recent works applied quantile prediction time series forecasting settings via direct prediction by an RNN Wen et al. (2017) or by combining RNN and linear splines to predict quantiles in nonparametric manner Gasthaus et al. (2019). Such methods still do not provide provable coverage guarantees, and can suffer from the issue of quantile crossing as in the case of Wen et al. (2017).
Conformal Prediction (CP): Pionccred by Vork ct al. (2005), conformal prediction (CP) provides mothods to construct prediction intervals or regions that are guaranteed to cover the true response with a probability 2 11aa, a, under the exchangeability assumption. Recently, CP has seen wider attention and has been heavily explored in deep learning (Lin et al. (2021); Angelopoulos et al. (2021); Stankevičiuté et al. (2021)) due to its distribution-free nature, which makes it suitable for constructing valid PIs for complicated models like deep neural networks. It is worth noting that although most CP methods apply to point-estimators, methods like conformalized quantile regression Romano et al. (2019) can also be applied to quantile estimators like Wen et al. (2017). CPTD is a conformal prediction mothod, but in the cross-sectional time series sctting.
Exchangeable Time Series and Cross-Sectional Validity: The work that is most relevant to ours is Stankevičiuté al. (2021), which directly applies (split) conformal prediction (Vovk et al. (2005)) assuming cross-sectional exchangeability of the time series". It however studies only the multi-horizon prediction setting, completely ignoring longitudinal validity. Although pcoos--ectionallyy valid, Stankevičhtt et al. (2021) leads to unbalanced coverage (i.e. some TS receives poor coverage longitudinally while others high) and inefficient Pls. To the best of our knowledge, no other works explore the cross-sectional exchangeability in the context of time series forecasting.
The authors of Stankevičinté et al. (2021) did not the term "cross-setional", but this exactly what they mean.
3

---

Published in Transactions on Machine Learning Research (09/2022)
Long and Single Time Series and Longitudinal Validity: Longitudinal validity is the type of validity that most works studying PI generation for time series focus on. Such works, including Gibbs & Candes (2021); Zaffran et al. (2022); Barber et al. (2022); Xu & Xie (2021), focus on the task of creating a PI at each step in a very long time series (often with over thousands of steps). For example, Gibbs & Candes (2021) propose a distribution-Iree conformal prediction method called ACI, which uses the realized residuals as conformal scores, and adapts the a each time basing on the average coverage rate of recent Pls. To achieve distribution-free marginal validity, ACI has (often) create non-informative infinitely-wide Pls, which is reasonable given the intrinsic difficulty stemming from the lack of exchangeability, Such methods also do not apply to our setting, because they typically require very long window to estimate the error distribution for a particular time series as a burn-in period. Furthermore, they do not provide a way to leverage the rich information from the cross-section.
We now proceed to first discuss some preliminaries that will be required to describe CPTD detail.
3 Preliminaries
Given a target coverage level 1-a a-a, we want to construct PIs that will cover the true response for a specific time series, and at a specific time step, with probability at least 1- a. However, we have not specified what kind of probability (and thus validity²) we are referring to. In this section, we will formally define cross-sectional and longitudinal validity, both important in our setting (See Figure 1 for an illustration). However, before doing so, we will first state the basic exchangeability assumption, a staple of the conformal prediction literature.
Definition 1. (The Exchangeabilityy Assumption Vork et al. (2005)) sequence of random variables, Z1, Z,.Ze....,,Z are enchangeable if the joint probability density distribution does not change under any permulalion upplied to the subscripl. That is, for any permulation a E Sn, and every measurable sel E C 2":
P[(Z1,
E E) = P[((Z(()) Zm(2):-- Zx(0)) E E}
(1)
where each Z, E Z (the corresponding measurable space for the random variable Z.).
Note that exchangeability is a weaker assumption than the "independent and identically distributed" (i.i.d) assumption. We extend the definition to a sequence of random time series:
Sn Definition 2. (The Exchangeable Time Series Assumption) Given time series S1, , where we denole Z,, as the random variable comprised of the tuple (74,,,... Ze,t ) Time series S,, S2,...... Sn are exchangeable if, for any fonitely many .... < bm, the random variables Z 1,(7)%" ................................ Zn, (t, are exchangeable.
S. =
17......,, 7........ 17
It should be clear that the exchangeability is "inter"-time series. Such an assumption could be reasonable in many settings of interest. For instance, collecting electronic health data time series for different patients from hospital. Notice that Def. 2 reduces to Def. 1 when we have only one specific value of t. Throughout this paper, we will assume Si.... SN+1 are exchangeable time scrios.
3.1 Cross-sectional Validity
The first type of validity of Pls is what we refer to as the cross-sectional validity. This validity is widely discussed in the non-time series regression settings, often referred to as just "validity" or "coverage guaranteee" (e.g. in Barber et al. (2020)), but is rarely discussed in the context of time series regression. Cross-sectional validity refers to the type of coverage guarantee when the probability of coverage is taken over the cross-section i.e. across different points. The formal definition as follows:
Definition 3. Prediction interval estimator Č. is (1 - a) cross-sectionally valid if. for any t 1,
PSN-IPPS (YN+LIH1 (N+1.H1 2 1 1-aa
(2)
Thhooughou the paper, "validity" and "coverage guarantee" are used interchangeably i.e. "valdd" PI synonymous with with "coverage guarantee".
4

---

Published in Transactions on Machine Learning Research (09/2022)
We will sometimes use additional subscript a for C (i.e., Ca) to emphasize the target coverage level. As reminder, C.,,, the estimator, denotes the model used to generate specific PI (a subset of IR) for each i and t. Using an example similar to one used carlier: suppose we want to predict the WBCC of a patient after the observation toms. In the first visit, there is really no time series information that can be used. Thus, the only type of coverage guarantee cross-sectional. In simple terms, we could con- struct a cross-scctionally valid PI and say if we keep sampling new patients and construct the PI using the same procedure, about 2 1 - a of the patients' initial WBCC will fall in the corresponding PI.
Symbol
Meaning
some symp-
8, 1Z.. Zi.rl
Time series
S,.
the first I obeervations of S;
Zi. (X.,,, Ys,s)
Observation for the i-th time series time Distribution of S
Ps
Coverage target
only be
Č.a
Prediction interval for Y,, Nonconformity score function
V() Vi,(()
Nouconformity score associated with Vt, quantile of
((8.))
Normalizer used in CPTD nonconformity scores A permutation invariant function (for CPTD-R)
Table 1: Notations used in this paper
It might be worth small digression here to note that the validity in Def. 3 is marginal. That is, the PI will cover an "average patient" with probability 2 1--a a. If we only consider patients from a minority group, the probability of coverage could be much lower, even if Ô is (cross-sectinnally) valid. We direct interested readers to Barber et al. (2020) for a more thoroughgoing discussion.
3.2 Longitudinal Validity
Following the above example, in later visits of particular patient, we would idcally like to construct valid PIs that also consider information from previous visits. That is, we would like to use information already revealed to us for improved coverage, regardless of the patient. As might be apparent, this already moves beyond the purview of cross-sectional validity and leads to the notion of longitudinal validity:
Definition 4. Prediction interval č. is 1 - longitudinally valid if for almost every time series S ~ Ps there ezists To such that:
t To Py (YN+1. CN+1.3 2
(3)
We impose threshold To because it should be clear that there is no temporal information that use for small t such as YN-1,t=- Here, the event A being true for "almost every" SN+1 means that the probability of occurrence of A is one under Ps. Note that the crucial difference between cross-sectional validity and longitudinal validity is that the latter is similar to a "conditional validity", indicating a coverage guarantee conditional on a specific time series. Although highly desirable, it should be clear that this is a much stronger type of coverage. In fact, we can show that distribution-free longitudinal validity impossible to achieve without using (many) infinitelywwide PIs that contain little information. We do so by adapting results on conditional validity, such those in Lei & Wasserman (2014); Barber el al. (2020). We formally state our impossibility claim in the following theorem:
Theorem 3.1. (Impossibility of distribution-free finite-sample longitudinal validity) For any Ps with atom?, suppose C. is 1 1-a longitudinally valid estimator defined in Def. 4. Then, for almost all SN+1 that we fix,
B[A(Ca (X N-1..-1: 00,
(4)
where X(-) denotes the Lebesgue measure. The expectation over the randomness of the calibration set.
At a high level, we will construct a distribution very close to Ps except for in a small region with low probability mass. We will however require the distribution of in this new distribution to spread out on R. Therefore, a distribution-fres O₂ is required to be (arbitrarily) wide as we take the limit. The actual proof is deferred to the Appendix.
A point is atom of Ps there exists such that Pslfd' d(s'8) 8) 8)}) > for any d(..)) denates the
Fuclidean distance.
5

---

Published in Transactions on Machine Learning Research (09/2022)
Remarks: Theorem 3.1 suggests that for continuous distributions, any longitudinally valid PI estimator can only give infinitely-wide (trivial) PIs all the time. This impossibilityy is due to the lack of exchangeability on the time dimension. In the case of cross-sectional validity, we condition on one particular time-step, but still have the room to leverage the fact that we have exchangeable patient records to construct the PI (using conformal prediction. See Section 4). the case of longitudinal validity, we condition on a particular patient. However, we cannot make any exchangeability assumption along the time dimension. Indeed, such an assumption would defeat the purpose of time series modeling; beside the fact that we cannot see the future before making prediction for the past.
We should also note that Theorem 3.1 does not preclude the use of temporal information in a mcaningful way. In fact, the main contribution of this paper is to incorporate temporal information to improve longitudinal coverage while maintaining pross-scctinnal validity.
4
Conformal Prediction with Temporal Dependence (CPTD)
4.1
Conformal Prediction
For the task of generating valid prediction intervals, conformal prediction (CP) is a basket of powerful tools with minimal assumptions on the underlying distribution. In this paper we will focus on the case of inductive conformal prediction (Papadopoulos et al., 2002: Lei al., 2015) (now often referred as "split conformal"), which is relatively light, thus more suitable and widely used for tasks that require training deep neural networks (Lin et al., 2021; Kivaranovic et al., 2020; Matiz & Barner, 2019). For this section only, suppose are only interested in PIs for Y, We denote Z; = (X.... Y,,) and drop the t subscript in X., and Y,,, In split conformal prediction, if we want to construct PI for particular Y,, we would first split our training data { into a proper training set and a calibration set (Papadopoulos et al., 2002). The proper training set is used to fit a (nonconformity) score function V. We could begin with one of the simplest such scoring functions: V(z)= = ly ø where ÿ is predicted by a function pa-) fitted on the proper training set.
For case of exposition and to keep notation simpler, we will assume any estimator like á has already been learned, and use (2 to denote the calibration set only. The crucial assumption for conformal prediction is that | N+1 are exchangeable. We could construct the PI for YN+1 by having
(a,N4I(XN+1) [[((N+1) w, ((XN-I) + +w]
(5)
where w = Q [(1 aN +1] {lu 0. 00 (
(6)
N +1
Vg=V((a))
where Q(3,-) denotes the B-quantile of The [-] operation ensures validity with a finite N with discrete quantiles. To simplify our discussion, we will also assume that there is no tie amongst the [ur) N41 with probability 1, ensuring that there is no ambiguity for Q This is reasonable assumption for regression tasks (e.g, Lei et al. (2018)).
If the exchangeability assumption holds, then we have the following coverage guarantee (Vovk et al. (2005);
Barber et al. (2022)):
PzNZi PZN+I(YN+L (YN+1 & Ca.N+1(XN+I)) <a
(7)
Because YN+1 is unknown, we typically replace V(ZN+1) in Eq. 6 with oo, which can only lead to a larger w and thus a conservative estimate that still preserves validity. The output of V--) is called the nonconformity score. The absolute residual used above is one of the most popular nonconformity scores, e.g. used in Stankevičtnté et al. (2021); Lin et al. (2021); Xu & Xie (2021); Barber et al. (2021).
4.2 Temporally-informed Nonconformity Scores (CPTD-M)
In this section we will describe first attempt to improve longitudinal coverage, with a focus on the underlying intuition of the more general idea. Directly applying the split conformal method from above

---

Published in Transactions on Machine Learning Research (09/2022)
(like in Stankevičiüté et al. (2021)) ensures cross-scctional validity, but comes with an important limitation. In sense, when test point is queried on calibration set, the nonconformity scores are supposed to be uniform in ranking. It is implied that the point estimates camnot be improved, for instance, when we use the absolute residual as the nonconformity score. In our task, suppose the prediction errors for a patient always rank amongst the top 5% using the calibration set up to time L. Even if we started assuming that this is an "average" patient, we might revise our belief and issue wider PIs going forward, or our model may suffer consistent under-coverage for this potient. These considerations motivate the need of temporally-informeed nonconformity scores. We hope to improve the nonconformity score used at time + 1 by incorporating temporal information thus far, making it more uniformly distributed (in ranking), so that whether Y is covered at different is less dependent on previous cases.
We propose to compute a normalizer mN+1.441 for each t, and use the following nonconformity score:
10
VN+L,++100 y; SN+1.))
mN+10++1
where S.,, denotes the first I observations of S.. The idea is that if we expect the average magnitude of prediction errors for a patient to be high, we could divide it by a large m to bring the nonconformity scores of all patients back to similar distribution. This is heavily inspired by popular nonconformity score in the non-times-series scttings -the "normalized" residual (Lei et al., 2018; Bellotti, 2020; Papadoponlos al., 2002), where (z) = | and can be any function fit on the proper training sel. We use simple mean absolute difference normalization strategy, or MAD-normalization in short, for m.
m 1 == t Wir Dipel-
(9)
The superscript M stands for MAD. One could potentially replace this simple average with an exponentially weighted moving average.
Note that one crucial difference between our i and the error prediction normalizer lies in the source of information used. This source for is mostly the proper training set, which means it faces the issue of over-fitting This especially problematic if there is distributional shift. For example, when hospital deploys model trained on a larger cohort of patients from a different database, but continues to its own patients small calibration set (which is more similar to any patient it might admit in the future). In our setting, however, conditioning on the point estimator, r does not depend on the proper training set at all. As we will see in the experiments (Section 5), in is more robust than an error predictor trained on the proper training set. We refer to this method as CPTD-M.
Once we have 1N+11111 the PI is constructed in the following way:
ACPTD-M M N41.+1
L0 - -- + G. MN+11411
(10)
i [(--)) aN +1]
[ - Destal
8
(11)
N +1
m4,41
im1
4.3
Nonconformity Scores (CPTD-R)
In the previous section, we gave an example of incorporating temporal information into the nonconformity score. However, we still have not fully leveraged the cross-sectional data in the calibration set. In fact, even in the non-time scrics setting, the nonconformity score v; is not constrained to depend only on Z,. All we need for the conformal PI to be valid is that the nonconformily scores (V.) N+1 themselves (as random variables) are eschangeable when { N+1 are exchangeable. This means that the nonconformity score can be much more complicated and take a form such as e (Z,; (2) depending on the un-ordered set of all
While obvious, more discussion on this can be found in Guan (2021).
7

---

Published in Transactions on Machine Learning Research (09/2022)
It might be hard to imagine why and how could adopt complicated version of the nonconformity score for the non-time series case, but it is natural when also have the longitudinal dimension. Suppose are to construct a Pl for YN+1++1 using conformal prediction, the nonconformity scores can depend on both S and the unordered data S.y == (S........ SN+1.17) for any t' = <t+1 1. To be precise, the nonconformity score VN+1.++1 could take the following general form:
VN+1.4+1(00)) 3) = f(0 SN SN41.441))
(12)
where g satisfies the following property:
permutation n, g(S+()). ) = -000=g(1n g(Sl.ttt)......
SN+L,,4)
(13)
Here, propose Ratio-to-Median-Resiie off, notice that while MAD-normalization can adapt to the scale of errors, it is less robust when there is heteroskedasticity along the longitudinal dimension; á will be influenced by the noisiest step t' < f++ 1. To cope with this issue, we could base mi,++1 on the ranks, which are often more robust to outliers. Specifically, at + 1, we first compute the (cross-sectional) median absolute errors the past:
(ÉCPTD-R)
simple example. First
Vs S L, m, == m where Ti,s = Vi.s - Di.s-
(14)
Then, for each i E [N + 1), and each i, we compule the expanding mean of the melian--orrmalize
ria
(15)
ms
Way can be viewed as an estimate of the relative non-conformity of S up time l. Thus, if we have a guess of the rank for |ra4t)), denoted as d,t41, we could look up the corresponding quantile as:
m R 1 1.1+1 = Q(1,,+11 { (
(16)
Following the notation in Eq. 12, the output of g(Si,t+), SN+1 is simply Q(s (
To obtain h,t4,, we can use the following rule (expanding mean with a prior):
0.5A + E. Fa(Irs,a))
(17)
t+
where F, is the empirical CDF over (|ri, N+ For example, F. (maxxKIra.))) = 1. Here use À = 1, which means our "prior" rank-percentile of 0.5 has the same weight as any actual observation. The full algorithm to compute m is presented in 1lg.1.
With all these muts and bolts in place, we can construct the PI ČCPTD-R as usual by using Eq. 10 and Eq. 11. The dependence on the (L + 1)-th observation simply dropped avoid plugging in hypothetical values for IN+1,4+1 5. CCPTD-R is somewhat complicated partially because we hope to exemplify how to let g depend on the cross-section, but it also tends to produce more efficient Pls empirically (see Section 5).
We dub our general method as CPTD (Conformal Prediction with Temporal Dependence), which includes both CPTD-M and CPTD-R.
4.4 Theoretical Guarantees
To formally state that CPTD provides with cross-sectional validity, we first need basic lemma:
Lemma 4.1. If S1,...,SN+F
ezchangeable time series, then Vt, [VCPTD-M ...., N+1,441 1,H41 ..... CCTD--G are both exchangeable sequences of random variables.
DCPTD-M)
and
CPTTD-
It could still be incorporated by performing "full" transductive conformal prediction, which typically much more expensive.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm Ratio-to-mmddin--esiihhll Normalization (CPTD-R)
Input:
se Response on the calibration set and the test TS up to L. (M Predictions on the calibration set and the test TS up to t+
Output:
{mu,41): Normalization factors for the nonconformity scores at 1.
Procedures:
E [N + 11, E [4], compute Ti.a < lit. - Di,sls and m, + (nedians(rrra)) Vi e [NV + 1], estimate the overall rank Q..+1 using Eq. 17.
Compute the empirical distribution of the median-normalized residuals (nres):** using Eq. 15.
Vi E [ + 1], look-up the normalizer 1,+1 using Eq. 16.
The validity for both our variants follows as a direct consequence: Theorem 4.2. CCPTD-M and ÖCPTD-R are both (1 a) cross- sectionaal valid.
N+1,1+1
All proofs are deferred to the Appendix.
Additional Remarks: Since the methods proposed are only cross-sectionally valid, they might raise the following natural question for some readers: What do we gain from using split-conformal, by going through the above troubles? First, expect that the average coverage rate for the least-covered time series will be higher. Imagine the scenario where the absolute errors are highly temporally dependent. In such case, CPTD-M and CPTD-R will try to capture the average scale of the errors, so that an extreme TS will not always fall out of the Pls (as long as such extremeness is somewhat predictable). This should be viewed as improved longitudinal coverage (despite the lack of guarantec). Secondly, we might observe improved efficiency the PIs might be narrower on average.
Finally, one could replace á in Section 4.2 and Section 4.3, making use of different function g that could potentially be more suitable for larget dataset. As long as the new g satisfies Eq. 13, the cross-sectional validity will still hold. We want to emphasize that CPTD should be viewed as general proposal to leverage both longitudinal and cross-soctional information to adjust the nonconformity score, for improved longitudinal coverage. Our focus is not on optimizing for ń.. However, in our experiments, we found that both CPTD-M and CPTD-R already perform well despite the simple choice of m.
5 Experiments
Through a set of experiments, we will first verify the validity of both CPTD-M and CPTD-R, as well as the efficiency (average width of the Pls). Then, more importantly, we will verify our assumption that ignoring the temporal dependence will lead to some TS being consistently under/over-coveree,, and that CPTD-M and CPTD-R improve the longitudinal coverage by appropriately adjusting the nonconformity scores with additional information.
Baselines: We use the following state-of-the-art baselines for PI construction in time series forecasting: Conformal forecasting RNN (CFRNN) Stankevičiūté et al. (2021)), a direct application of split-conformal prediction"; Quantile RNN (QRNN) Wen et al. (2017); RNN with Monte-Carlo Dropout (DP-RNN) Gal & Ghahramani (2016); Conformalized Quantile Regression with QRNN (CQRNN) Romano et al. (2019); Locally adaptive split conformal prediction (LASplitt Lei et al. (2018), which uses normalized absolute error as the nonconformity score (we follow the implementation in Romano al. (2019)). Among the baselines, CQRNN and LASplit are existing conformal prediction methods extended to cross-sectional time series forecasting by us, and QRNN and DP-RNN are not conformal methods (and not valid).
RCPTD-M could also be viewed as having constant g that ignores the input.
TTee authors suggest performing Bouferroni correction to jointly cover the entire horizon (all T steps). This however means if T (Hl in Stankevičiūté al. (2021)) is greater than a(N 1), all Pls infinitely wide Barber al. (2022). The authors performed an incorrect split-conformal experiment, which is why the COVID19 dataset still has finite width in Stankevidlnté al. (2021).
9

---

Published in Transactions on Machine Learning Research (09/2022)
Datasets We test our methods and baselines variety of datasets, including:
MIMIC: Electronic health records data for White Blood Cell Count (WBCC) prediction (Johnson et al. (2016); Goldberger al. (2000); Johnson et al. (2019)). The cross-section is across different patients. Insurance: Health insurance claim amount prediction using data from a. healthcare data analytic company in North America. The cross-section is across different patients.
COVID19: COVID-19 case prediction in the United Kingdom (UK) (COVID). The cross-section is along different regions in UK.
EEG: Eleetroencophalography trajectory prediction after visual stimuli (UCI EEG). The cross-section comprises of different trials and different subjects.
Load: Utility (electricity) load forecasting (Hong et al. (2016)). The original data consists of one TS of hourly data for 9 years. We split the data by the date, with different days treated as the cross-scction.
MIMIC, COVID19 and EEG are used in Stankevičiuté et al. (2021) and we follow the setup closely. Note that for Load, we perform a strict temporal splitting (Lest data is preceded by calibration data, which is preceded by the training data), which means the exchangeability is broken, We also include a Load-R (random) version that preserves the exchangcability by ignoring the temporal order in data splitting. A summary of cach dataset is in Table 2.
Evaluation Metrics and Experiment Setup We follow Stankevičiuté et al. (2021) and use LSTM Hochre- iter & Schmidhuber (1997) as the base time scries regression model (mean cstimator) for all mothods. We use ADAM (Kingma & Ba (2015)) as the optimizer with learning rale of 10-3, and MSE loss. The LSTM has one layer and a hidden size of 32, and is trained with 200, 1000, 100, 500 and 1000 epochs on MIMIC, COVID19, EEG, Insurance and Load, respectively. For QRNN, we replace the MSE loss with quantile loss. Except for QRNN, CQRNN and DPRNN, all methods share the same base LSTM point estimator. For the residual predictor for LASplit, we follow Romano et al. (2019) and change the target from y to ly G--
We repeat each experiment 20 times, and report the mean and standard deviation of:
Average coverage rate: EM
M I,, where T, T.= f 11ff Ca.as).
Tail coverage rate: EjiT,e EL 11 C, where L, (C= < ((,,,, (C,) In other words, we look the average coverage rate of the least-covered time series. We wish it as high as possible.
Average PI width: N/. E. Σ M(Ca...) where pa-) is the width/length of..
In the above, M denotes the size of the test set. All metrics here consider the last 20 steps. The target a = 0.1 (corresponding to 90% Pls). We use the same LSTM architecture as Stankevičiuté et al. (2021) with minor changes in the number of epochs or learning rate, except for the Insurance dataset where we introduce additional embedding training modules to encode hundreds of discrete diagnoses and procedures codes. In the Appendix, we include results for Linear Regression instead of LSTM.
Results We first report the mean coverage rates in Table 3. We see that in terms of average coverage rate, all conformal methods are valid (90% coverage) for the exchangeable datasets. In the case of Load, since we did not enforce exchangeability during the sample splitting, there is clear (minor) under-coverage for all conformal methods. However, CPTD is still slightly better than baselines, potentially because it can leverage information from the calibration set better. The benefits of CPTD, however, are best illustrated in Tables and 5. In terms of efficiency (width), CPTD-R generally provides the most efficient valid Pls. The improvement is generally not large, but still significant. We note that for MIMIC, directly predicting quantiles (QRNN and CQRNN) provides more efficient Pls, which might be due to an asymmetric distribution of the prediction errors by the point-estinator. Designing temporally adjusted nonconformity scores for quantile regression (potentially basod on Romano et al. (2019)) will be an interesting direction for future rescarch. Finally, if we examine the lail coverage in Table 5, we see that both CPTD-R and CPTD-M consistently outperform baselines (with the mean width rescaled to the same). Table 5 suggests improved longitudinal coverage, which is the major focus of our paper. This can also be observed in Figure 2. The results suggest CPTD significantly improves longitudinal coverage with the semporally-adjusted nonconformity scores.
10

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2: Size of each dataset, and the length of the time series. Note that the Insurance dataset has up to 14 diagnoses codes, up to 17 CPT codes, and 3 other features. If we use onc-hot encoding for the discrete codes, Insurance has 14*20++771001+ features instead. All results presented in this paper measures the last 20 steps, while the full results are in the Appendix.
Properties
NIMIC
Insurance
COVID19
EBC
Load/Load-R
train/cal/test
192/100/100
2393/500/300
200/100/80 30
300/100/200 63
1198/200/700
T (length)
30
30
24
# features
34
26
EEG
Load
0.8
0.8
0.7
0.6
mean coverage
0.5
CPTD-R
0.4
CPTD-R
0.5
CPTD-M
CPTD-M
CFRNN
CFRNN
0.4
CORNN
0.2
CORNN
LASplit
LASplit
0.3
ORNN
0.1
ORNN
DPRNN
0.0
DPRNN
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
0.10
percentile of mean coverage rate
percentile of mean coverage rate
Figure 2: We show the coverage rate for the bottom 10% of the time series for EEG (left) and Load (right). All methods are re-scaled to the same mean PI width for a fair comparison. The Y-axis the average coverage rate. The X-axis denotes the percentile among all test time series, with 0.00 mcaning the loast-covered time scrics. The band is an empirical 80% confidence band. CPTD significantly improves the lddiat coverage rate, especially for the least-covered time series.
Table 3: Average coverage rate for each time series. Empirically valid methods are in bold (with p-value = 0.05). We verify that conformal prediction methods are valid, while non-conformal methods could under-cover. Note that Load does not satisfy the exchangeability assumption, which why conformal methods look invalid (slightly below the larget of 90%).
Coverage (2 90%)
CPTD-R
CPTD-M
Split (CFRNN)
CQRNN
LASplit
QRNN
DPRNN
NIMIC
90.22:1.72
90.17:1.50
90.32
89.93#1.30
90.46=1.92
86. 1.33
46.2244..1 15
Ineurance
30.01+0.63
90.10::0.16
90.05=#0.4
90.06=0.76
90.05=0.72
85.81+0.76
24.56+0.77
COVID19
90.1311.55
90.27:11.07
90.09:11.75
90.08=1.61
90.15=1.51
89.18=1.8 1.52
68.3743.99
EEG
89.9011.75
80.0811..4
89.90.1..7 1.79
2.226
89.56
87.9410.99
38.84.1..35
Load Load-R
88.7310.14
89.23-0.15
18.6420.17
10.211:0.1
88.071:.200
60. 10:1 1.38
89.67-10.64
90.05=0..56
90.17=20.7
90.03:+0.00
90.23:0.62
90.11=0.53
85.35+1.06
90.9710.70
6 Conclusions
This paper introduces CPTD, simple algorithm for constructing prediction intervals for the task of time series forecasting with a cross-section. CPTD is the first algorithm that can improve longitudinal coverage while maintaining strict cross-scctional coverage guarantec. Being conformal prediction method, the cross-sectional validity comes from the empirical distribution of nonconfornity scores on the calibration set. To construct prediction intervals for YN+1.441, we propose CPTD-M, which leverages only the temporal information for the time series of interest (Sx+1), and CPTD-R, which exemplifics how to use the entire calibration set to improve temporal coverage. Our experiments confirm that both CPTD-M and CPTD-R significantly outperform satee-ofthhe-att baselines by a wide margin. Moreover, CPTD could easily be applied
11

---

Published in Transactions on Machine Learning Research (09/2022)
Table 4: Mean of Pl width. The most efficient (and valid) method is bold, including methods
significantly worst than the best onc. For Load, we show the most efficient conformal method. CPTD-R generally provides the most efficient Pls.
Width
CPTD-R
CPTD-M
Split (CFRNN)
CQRNN
LASplit
QRNN
DPRNN
1.69640.
1876:40.200
1.759440.160
1.56040.14
1.87240
1.407=0.1
0.58440.022
Inaurance
2.594=0.061
2.723=0,051
2.090=0.057
2,613+0.050
2,694+0,007
2.314=0.034
0.585=0.014
COVID19
0.713=0.027
0.824=0.102
0.737=0.033
0.82740.082
0.73740.038
0.803=0.082
0.315=20.048
EEC
1.275.1..048
1.30110.049
1.301 10.056
1..6610.075
1.294.10.033
1.31910.042
0.414.10.020
Load
0.200:10.004
0.230:1.005
0.209:10:004
0.216:10.005
0.213:10.00
0.168-000005
0.569-10.008
0.178:0.003
0.2001100007
0.17810.004
0.187:10.004
0.18110.005
0.16420.005
0.334:10.012
Table 5: The tail coverage rate (mean coverage rate for the least-covered 10% time series). For a fair comparison, we re-scaled all methods to have the same mean PI width (as CFRNN). Unlike average coverage rate, we want the tail coverage rate to be as high as possible. The best method in bold, with the sccond-best underscored. Generally, both CPTD methods significantly outperform the baselines, providing better longitudinal coverage.
Tail Coverage
CPTD-R
CPTD-M
Split (CFRNN)
CQRNN
LASplit
QRNN
DPRNN
MINIC
69.2044.18
69.10:43.00 3.95
64. 52.322
73.55-13.49
62.93-6.47
78.2543348
65.60=5.22
Insurance
11.13+1992
72.19=1..22
26.15
68.28#2.91
08.22=2.29
61..2=2.52
47.82=2.94
COVIDIS
5..22:5.05
70.47:2..47
63.78-66.74
09.7526.30
67.3424.41
39.81
52.56:6.60
1.33
71.09=3.73
61.35=11.23
57.02=6.01
66.88=2.03
37.07=3.11
51.06+2.99
Load Load-R
70.58=0.98
68.85+00.94
1.480=1144
59.65#1.62
59.62=1.33
59.87=2.00
29.56#1.91
78.0341.46
71.36: 1.36
1.96
69.61#1.33
09.83=2.03
69.42-1.86
31.92:2.19
to any model and data distribution. We hope CPTD will inspire future research in uncertainty quantification in time series forecasting with a cross-section.
Acknowledgments
This work was supported by NSF award SCH-2014438, ITS-1838042 and NTH award RO1 1R01 NS1072177....... ST was partially supported by the NSF under grant No.
12

---

Published in Transactions on Machine Learning Research (09/2022)
References
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations,
2021. URL https://oparerviev.nu wttp:/rwww.....toooooo//daaaaaaa
Anastasios Nikolas Angelopoulos, Amit Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-fre uncertainty quantification and applications in imaging. ArXiv, abs/2202.05265, 2022.
Rina Foygel Barber, Emmanucl J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution- free conditional predictive inference. arviv, abs/19000 1903.0404,,,, 2020. URL https://arxiv www.eorggggoaeooo prgg/ba//a aaaaabaa199 1903. 04684.
Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife-+. The Annals of Statisties, 49(1):486-507, 2021. doi: 2..114//0-AOS1965 URL
https://doi.. org/10. arg/10.1100 10910-9--
Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability, 2022. URL httpp:://axx: org/abs/2200
Stephen Bates, A. Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets. J. ACM, 68:43:1-43:34, 2021.
Anthony Bellotti. Constructing normalized nonconformity measures based on maximizing predictive efficiency. In Alexander Gammerman, Vladimir Vovk, Zhiyuan Luo, Evgueni Smirnov, and Giovanni Cherubin (eds.), Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, volume 128 of Proceedings of Machine Learning Research, pp. 41 54. PMLR, 09 11 Sep 2020. URL http://proceediing. mlr wwwpppeesaaaeeeee
hateee//vv///eelle htt////ellotttt/e 20m.mml...
Jose Caceres, Danilo Gonzalez, Taotao Zhou, and Enrique Lopez Droguett. A probabilistic bayesian recurrent neural network for remaining useful life prognostics considering epistemic and aleatory uncertainties. Structural Control and Health Monitoring, 28(10):22811, 2021. doi:
URL
httpps//oonineelibee
wiley. www/doo/abs///ttcttoooc 10. 1002/stc. 2811.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Eric P. Xing and Tony Jebara ), Proceedings of the 31st International Conference on Machine Learning, volume 32:2 of Proceedings of Machine Learning Research, pp. 1683 1691, Bejing, China, 22 24 Jun 2014. PMLR. html.
URL https: http::s//procceeing .mlr.. pptps//e/e/ceeeeeeeeeeeee
Isidro Cortés-Ciriano and Andreas Bender. Concepts and applications of conformal prediction in computational drug discovery, ArXie, abs/ 1t1/1190.00000 2019.
COVID. Coronavirus (covid-19) in the uk. http:://tttps//coroeriiivs.. www..gov....oo..o.ooooooooooooo gov.uk/, 2022. Accessed: 2022-04-14.
Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Efficient conformal prediction via cascaded inference with expanded admission. In ICLR, 2021.
Meire Fortunato, Charles Blundell, and Oriol Vinyals. Bayesian recurrent neural networks CoRR, hls//1704.02988, 2017. URL http:// /tapivv//axxv...// arg/abs//ee abs//704. 111///07444.......
Yarin Gal and Zoubin Ghahramani. Dropout a Bayesian approxination: Representing model uncertainty in deep learning. In 33rd International Conference on Machine Learning, ICML 2016, 2016. ISBN
9781510829008.
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Jamuschowski. Probabilistic forecasting with spline quantile function rnns. In Kamalika Chaudhuri and Masashi Sugiyama (ods.), Proceedings of the Twenty-Secome International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1901-1910. PMIRR 16-16 18 Apr 2019. URL /tttps:/prroceeeiee ings mlr www.mml.........ee.mmmmmmmme prw.s//89/ 1at/99/gasaaus///oo htmm.....
13

---

Published in Transactions on Machine Learning Research (09/2022)
Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. In A. Beygelzimer, Y. Dauphin, P. Tiang, and J. Wortman Vaughan (eds.), Aduances in Newral Informalion Processing Systems, 2021. URL
A. L. Goldberger, L.. A. Amaral, T.. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. F. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of new research resource for complex physiologic signals. Circulation, 2000. ISSN 15244539. doi: 10.1161/01.crr.01..22
Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction, 2021. URL httpps://aax.v.oorr//aa/////1/ http://ba2211...0855
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compul., 9(8):1735 1780, nov 1997. ISSN 0899-7667. doi: 8.1735.
URL https:/
1997.9.
Tao Hong, Pierre Pinson, Shu Fan, Hamidroza Zarcipour, Alberto Troccoli, and Rob J. Hyndman. Probabilistic energy forecasting: Global energy forecasting competition 2014 and beyond. International Journal of Forecasting, 32(3) 2016. ISSN 0169-2070. doi:
URL https: www sww.ncedireccct.c....o.. http://wwwwe..eeeeeeee/aaaaaaa/a
A. Johnson, T. Pollard, and R. Mark. Mimic-iii clinical database demo (version 1.4). https: betps://arcen https://archive..ee
uci www.edd///daaaetaaeeeeeeeee EEEe//EaDDDaaaaeeeeeaeaeaaaaaeee ocd.ecccc//l//eeaeeeeee////eeeer
2019.
Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, freely accessible critical care database. Scienlific Dala, 3(1):100035, 2016. ISSN 2052-4463. doi: 10. 10.1038/sdala 2016... URL https: //doi. o0g/10. /03t/sdata.aao.....ooaaaaaaa/a sdata. odata.2016.33. 2016.35.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (ods.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. URL http: http:///rxiv/ org/ http//11...999.
Dioderík P. Kingma and Max Welling. Auto-cncoding variational baycs. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URI. http::/arxiv.cr org abs/ hatp://1.....1
Danijel Kivaranovic, Kory D. Johnson, and Hannes Leeb. Adaptive, distribution free prediction intervals for deep networks. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 4346-4350 PMLR, 26 28 Aug 2020. URL https://proceediii: mlr. prrss// #108/kivaranovicc22 httm........
Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrion, 46(1):33 -0, 1978. ISSN 00129682, 14680262. URL http:://ww. jstor org/stable//91366)
Balaji Lakshminarayaaan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty cstimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non- non-paramctric regression. Journal of
the Royal Slatislical Sociely; Series B (Stalistical Methodologyj, 76(1):71 96, 2014. doi: httss://oo.org//// 1111/rssb. 111/rssb.12021. URL Utt wtwpa//wwwweee...///////aaaaaaaa wwt./oi/aa////e/eco///o/ooa /10/aaa.//////////////////////// http:///.......1.......aa/. 1111/rssb. 12021.
Jing Lei, Alessandro Rinaldo, and Larry Wasserman. A conformal prediction approach explore functional data. Arnals of Mathematics and Artificial Intelligence, 74(1):29-433 2015. ISSN 1573-7470. doi: 10.1007/
s10472-013-9366- URL https://doi.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-Fri Predictive Inference for Regression. Journal of the American Statistical Association, 2018. ISSN 1537274X.
doi: 10. 10.1080000 01 621 459.2011.1307116,
14

---

Published in Transactions on Machine Learning Research (09/2022)
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Locally valid and discriminative prediction intervals for deep learning models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advences in Neural Information Processing Systems, volume 34, pp. 8378 8391. Curran Associates, Inc., 2021. URL
wwtp:/iww.....p.prrreeeeeee heaarppp.cc/ppper//////////
96776500777777588888888776-66 Pdf.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2218-2227. PMLR, 06-11 Aug 2017. URL https::/ppoceeedi/e aereiiinnnmmm.......mmm.meme.e.m prrsss/770/oouizos///7 otw.eeddngg........rrse//////oso
html.
Sergio Matiz and Kenneth E. Barner. Inductive conformal predictor for convolutional neural networks: Applications active learning for image classification. Pattern Recognition, 90:172- 182, 2019. ISSN 0031-3203. doi: http::///bb.orr//1.0000/////i/n
sccecce/aaticce/p/////0033233330 201.000000.00.............. 2019.01.035. URL http:://wwww.cdeeeeereee...oo
Radford Neal. Bayesian learning via stochastic dynamics. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufman, 1992. URT, https:
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Tapio Elomaa, Heikki Mannila, and Hanmu Toivonen (eds.), Machine Learning: ECML 2002, pp. 345-356, 356, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In H. Wallach, H. Larochelle, A. Beygelzime, F. d' A'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceeddin pdf.
neurips.cc/paperrs https/sccppppppe/e/eeeeeeeep http://0000001ee.000003383333330
Kamilè Stankevičiuté, Ahmed Alaa, and Mihaela van der Schaar. Conformal time-series forecasting, In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL attps://ogg /tppr//////eeeeee.///m/mm
Ingo Steinwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 17(1):211 - 225, 2011. doi: 10.3150/10-BEJJ26; URL httpp::///di..orrg 00///...1///111111
UCI EEG. Eeg database. httpp://archive.iic.. ucw.ii.......iieoooooo...e...eee 2022-04-23.
http:///zrchiee.........eeeaaeas htt.:///w......e////a/attttttttt 1999.
Accessed:
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmicc learning in random world. Springer US, 2005. ISBN 0387001522. doi: 10.1007/b106715
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011. ISBN 9781450306195,
Ruofeng Wen, kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster, 2017. URL
11053.
Andrew Gordon Wilson and Pavel Ixmailov. Bayesian deep learning and probabilistic perspective of generalization. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advonces in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 2020, virtual, 2020. URL https: //proceedings. / http:////////////aaaooaaaaaaaaaa Abstttta html.
Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In Marina Meila and Tong Zhang (ods.), Proceedings of the 98th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11559-11569. PMLR, 18-26 24 Jul 2021. URL https: https://proceeding: mlr.
wtw.sgg..ww.......eaeeeee wwtpss/eie/eulleeeeeee html.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Margaux Zaffran, Aymeric Diculeveut, Olivier Féron, Yannig Goude, and Julie Josse. Adaptive conformal predictions for time series, 2022. URL http:///aaxxivvooaaa////a/e
Jin Zhang, Ulf Norinder, and Fredrik Svensson. Deep learning- -ased conformal prediction of toxicity. Journal of chemical information and modeling, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2022)
A
Proofs
A.1
Proof for Theorem 3.1
A.1.1 Lemmas
We first present established result the impossibility of (non-degenerate) finite-sample distribution-free conditional coverage guarantee from Lei & Wasserman (2014):
Lemma A.1. Let P be the joint distribution of two random variables (X,Y). Suppose ÖN is conditionally valid, as defined by the following:
P[YN+1 Cx(e)|XN+1 = x} 2 11-0 for all P and almost all x.
(18)
Then, for any P and any Do that not an atom of P::
Plim ess L(CN(z)) oo} =
(19)
8-+0 - 26
The subscript N means ČN depends on a calibration set of size N, and L(-) is the Lebesgue measure.
A slightly stronger statement of Lemma A.1 is given by:
Lemma A.2. Let P be the joint distribution of two random variables (X,Y). Let Pe be the distribution of on additional random variable U that C could use. Demole the joint distribulion of (X, Y, Y,U) as p== P x Pu, and X+ = (X, Suppose ON is conditionally valid with respect to the original P, as defined by the following:
P[YN+1 E CN(X = x121-0 = a for all p+ and almost all x.
(20)
Then, for anay P- and any xo that is not atom of
Plim ess sss [(CN(I,UNH1)) oo) =. 6-40 ||005
(21)
Below a proof mostly following Lei & Wasserman (2014). First, we define 6n and TV like in proof for lemma 1 in Lei & Wasserman (2014). For any pair of distributions and Q, define the total variation distance between them as:
TV(P. Q) (P() Q(A))
(22)
For any > 0, define ex = 2((1 -)))); and we will have (Tei & Wasserman (2014))
TV 2 S EN TV(PN, ,N) Se. 2
(23)
B Fix c <>> 0. Let 2n be a non-atom and choose 5 such that Px (B(x, 8)} <CN- Fix B > 0 and let Bo 2(1-a)
Given pt, define another distribution Q+ by
2 (A() - (AnS=)
(24)
where S ((.,
B(xg, 8)), and U has total mass under P+(() and is uniform in {(I, y. u) E Bxxo, 8), ly| < Bo,ue u B(0, C)). (We will see that the only thing that matters is that Y is uniform in this small region). Note that TVPP+. Q+) I eN, which means (V(++N,
Sc.
For all æ E B(xo, 8) and all u E B(0, C), Sc(r.M) o (ykr)dy 2 1 - a implies leb(C(x, u) 2 2(1 - a) = B. Therefore, Q + (csS sup leb(C(x, U)) 2 B} =. Therefore,
p+N sus leb(C(x U)) 2 B} 2 2 ess sss sup leb(C(z,U)) 2 B} - =EB(...6)
(25)
=EB()..5)
Lemma A.2 follows and B arbitrary.
17

---

Published in Transactions on Machine Learning Research (09/2022)
A.1.2 Main Proofs
Proof. Now, for any i, we could view all previous observations (including Y) as the new "X", and Xi as the
"U". That
X; Zi,,, Zi,1.... ................................ Z1
(26)
U: Xi,
(27)
x = [x Us] pt
(28)
Then, the question is whether we could use the new X N and {(X), Yno to create a prediction interval Ô such that
Py Fiii Õ 21-a.
(29)
Lemma A.2 tells if Ò satisfies such coverage guarantee (note the conditioning XN41), have, for all
40.
P(lim ess leb(E(x, UN+1)) oo) =
(30)
8-0 |
Ex xil
[leb(C(X N+1)) oc.
(31)
A.2 Proof for Lemma 4.1
Proof. We will show the general case that any nonconformity scores Vit...,, generated Eq. 12 and Eq. 13 are exchangeable SNn4 are, because VCPTD-M and VCPTD-R are special cases of nonconformity scores generated this way.
First of all, if denote V., vector, it is clearly row-pormitatioonqquiiaaant function (denoted as G) the matrix S.,r, where the i-th row (out of N + 1) is [Zi,,...., Zie). Formally, for any permulation of N + elements, Vx, = G(S For each measurable subset E of VN+1, define Es c 2 N-1 as
Es = {s.. G(s.a) E E}
(32)
Because S,..... SN+1 are exchangeable, for any permutation T, we have
P(V,, E E} = P(S..t e E Es} = P(Sn,a E Es} P(V E E)
(33)
A.3 Proof for Theorem 4.2
Proof. Follows from Lemma 4.1 immediately. We provide a brief sketch here. With Lemma 4.1, we know that the following random variable ON+1
{i [N] U/ S +w +1
ON+1
(34)
N +1
follows uniform distribution { means there is always strict ordering.) Since
N+1 (Again we assume the probability of having tie is zero, which [==
ON+1 < [11 (N + 1]]
[11 - a) + 1)]
UN41441 S
(v..41) = YN41441 E CN414H1
N+1
N +1
(35)
have
[(1 (1-aa [(1-a) + +1]
P(YN41:41 Cx411417 2 P(oN+1 <
221-a
(36)
N +1
7
18

---

Published in Transactions on Machine Learning Research (09/2022)
B Why does normalization help?
In this section, we will consider some simple scenarios and show why CPTD-M and CPTD-R can improve longitudinal coverage. While these scenarios are over-iimllifictions of the reality, we use them mainly to illustrate the main ideas behind CPTD more rigorously.
Suppose the error both time-dependent and heteroskedastic in the cross-section. Formally, suppose the error RA, Y,,- - p(X St,+-1) is a random variable that factors out into a product R = S,E, where the marginal distribution of Ex is PE.- We impose mild assumption that P(F, = 0) 0 for simplicity of discussion. That is, there is an intrinsic "error scale" for cach time scries. Note this assumption is not overly simplistic either: While the assumption in many related works (e.g. Barber et al. (2022); Xu & Xie (2021)) is that of mild distributional shift, we allow arbitrary distribution for PE.
The case for CPTD-M: Denote the percentile of 1 m. examine the following probability:
among { ( N+1 i=1
We will
16
P|YN+IT E CCPTD-M 1Ps(SN 2 33
N,N+1
(37)
-P [(1 aN + +1] [Fs(SN+1) 2 3}
(38)
N +1
for large & such as 0.99.
This can be thought of a worst case coverage rate. For T > 1. we define new (which is defined with probability one). is clear that R.T ~ PEy for all i. m.,
random variable Fy--
Er
E.
As result, P(N
M [1-aXN++1] [F((S++)) 2 3} [V-+))+++]] for any B. N+1
N+1
The case for CPTD-R: Now, suppose also have heteroskedasticcty along the longitudinal dimension. For simplicity of discussion, we assume E, = C,e.V( where C. is a non-random scalar for each 1. Denote the percentile of the nonconformity scores for CPTD-R as d.T and that for the basic split conformal as h.T. While the previous discussion still holds, consider a slightly different quantity than the above:
[(1-a)(N) + 1)]
P(IN+I. S
iN+L1 < <1 <1-3)
(39)
++
What would happen if, say, C >> (supr, 5, by the randomness of Ej. Therefore, m. will be too small as long @N-1,1 is very small, even if the target normalization constant SN+1 is large. This is however not issue for CPTD-R, because C. is always cancelled out.
8y
Ei= C₂? Essentially, if we use CPTD-M, mM is dominated
C
Additional Experimental Details
C.1
Full TS
In Table 6 we show the same metrics in the main text, but on the entire time series (instead of the last 20 steps).
C.2 Tail coverage rate over time
We plot the average longitudinal coverage rate for the bottom 10% TS up t, for each & <= T,, in Figure 3. For clarity, only include CFRNN and CPTD (including CPTD-M and CPTD-R). Ideal is the ideal scenario where the event of coverage is temporally independent for each time series. That is, each 1fYN+1c Ò} follows Bernoulli distribution of = 1 - a (independently). We did not perform re-scaling in this plot because Ideal corresponds to an average coverage rate of 1 - which is why CPTD-M (which typically generates wider Pls) shows better coverage than CPTD-R. We could see that there are still gaps between CPTD and Ideal, which a room of improvement for future works. It also interesting that most of the gain in coverage seems to happen at the beginning- that is, CPTD-R and CPTD-M adapt the "extreme" TSs in few steps, and maintain the gain.
19

---

Published in Transactions on Machine Learning Research (09/2022)
Table Mean coverage, mean width, and tail coverage (re-scaled to same mean PI width) using the full time series. Valid mean coverage and the best of tail coverage and mean PI width are in bold. The conclusion is the same as in the main text CPTD greatly improves the longitudinal coverage for the least-covered TSs, and maintains very efficient PIs (width).
Coverage
CPTD-R
CPTD-M
Split (CFRNN)
CQRNN
LASplit
QRNN
DPRNN
MIMIC
00.23111.00
90.12. 0.96
90.23.1 1.28
89.9311.11
90.24 1.42
94.79.11.28
44.9613.99
Insurance
00.97
90.02:0.32
00.0
90.05:1..22
89.99-10.49
86.3241..33
25.1540.69
CUVID19
19.97
90.22=0.94
90.02=1.08
90.11=1.36
90.10=1.300
89. 19.10+1.35
65.98
89.63=0.92
90.11=0.77
89.53=1.14
89.83+1.21
89.48=00993
86.61+0.70
35.1941..17
Load
88 88.68100.12
89.23.10.14
88.52:10.18
89.0910.11
88.8410.19
80.29 1.344
90.6110.61
Load-R
90.01=0.54
90.12=0.69
89.98=0.61
90.27=0.60
90.03=t0.48
85.69+1.10
91.9110.62
Mean Width
NIMIC
1.767:10.17
2.136=0.185
1.80810.18
1.640:1..137
1.913:10.1 147
1.416:10.128
0.575-10.022
Insurance
2.64140.048
3.055=0.075
2.71140.044
2.647#0.041
2.712+0.050
2.369=40.028
0.659:
COVID19
0.713=0.028
0.918=0.131
0.731
0.003
0.796+0.066
0.731#0.033
0.771
0.486=0.010
E2G
1.207:0.022
1.343=0.051
1.220+0.003
1.345:0.00
1.220::0.028
1.228:0.0033
0.34910.014
Load
0.186.10.003
0.004
0.1044
0.200.10.000
0.197:10.005
0.156110.005
0.559110.008
Load-R
0.165::0.008
0910020.007
0.16510.003
0.172:00004
0.167:10.004
0.151 000004
0.539:1.011
Thil Coverage Rate f
MIMIC
73.83=22.0
70.15:22.88
68.95-42.88
35.7813.37
68 3.
75.02=3.99
69.82:44.36
Insurance
74.31 1.199
72.99
68.97
71.22
71.30 1.55
68.35 1.007
55.9912.11
COVID19
70.65=5.72
68.54+2.07
6.36
53.67
68.75-44.66
63.6916.16
37.7344.51
E2G
14.6
76.58 1.33 1.39
69.91 1.2.5
65.0612280
71.01 1.011:1.65
64.65 12.01
60.75:12.26
70.7411.774
69.42:10.79
58.534
59.6711.56
59.68
60.26+1.71
32.9841.088
Load-R
73.62#1.39
71.21#1.26
68.88+1.90
1..87
70.58+1.95
70.72+1.69
34.3242.62
C.3 Cross-sectional coverage over time
In Figure 4, we plot the (cross-sectional) mcan coverage rate at different t. We can see that all conformal methods exhibit cross-sectional validity as expected, whereas the coverage rate for non-conformal methods vary greatly through time.
C.4 Normalization Quality
In this section we compare the rank (Spearman) correlation between different quantities and the realized residuals. The quantitics consider are M. for CPTD-M, for LASplit, and the width of PI prodicted by QRNN (CQRNN). The correlation could be considered measure of the normalization quality. That is, if the correlation is high, the distribution of the rank of the normalized residual/nonconffnn will be closer to uniform distribution, which will mitigate the under-coverage of "outlier patients", For each t, corr, == CaaarnanCorreltion.. mentioned above. The (pooled) mean and standard deviation across all and seed pairs are reported in Table 7.
score
Mi, δr,l)) where 4ar should be interpreted as m, é or PI width
As we can see, both QRNN and LASplit typically have lower correlations at test time. This is simply because training errors are typically lower than test error (not over-fitting.. LASplit is especially fragile to this because this effect is two-fold, for both the base point estimator and the error predictor. A similar argument can be found in Romano et al. (2019) explaining why CQR is beller than LASplit in terms of efficiency. On the contrary, the correlation for CPTD-M is typically higher on the test set, benefiting from the same effect: With a simple expanding mean as m the point estimator's 's error on the test set is easier to predict than on the training set. Note also that QRNN does not actually issue a point estimate, so Φi,t is replaced with the middle of the PI. This means the level of correlation itself probably not comparable, but the change from training testing still informative.
C.5 Linear Regression the place of RNN
We perform additional experiments replacing the base LSTM with linear regression model. This model consists of T sub-models, one for each (using data up to as input). The results are presented in Table 8.
20

---

Published in Transactions on Machine Learning Research (09/2022)
MIMIC
CLAIM
COVID
1.0
1.0
1.0
0.8
0.8
0.8
Tail Coverage
0.6
0.6
0.6
0.4
0.4
0.4
CPTD-R
CPTD-R
CPTD-R
CPTD-M
CPTD-M
CPTD-M
0.2
0.2
0.2
CFRNN
CFRNN
CFRNN
Ideal
Ideal
Ideal
0.0
0.0
EEG
GEFCom
GEFCom-R
1.0
1.0
0.8
0.8
Tail Coverage
CPTD-R
CPTD-R
CPTD-R
CPTD-M
CPTD-M
CFRNN
CFRNN
CFRNN
Ideal
Ideal
0.0
0.0
20
40
60
0
5
10
15
20
10
15
20
Figure 3: Tail Coverage Rate function of time. We plot the mean of 20 experiments. Ideal refers to simulated coverage events that have no temporal dependence. The X-axis is t, and the Y-axis is the longitudinal mean tail coverage rate to time t. CPTD-M and CPTD-R typically adapt to the overall nonconformity the TS in a few steps and maintain the advantage afterwards. There is, however, still gap between CPTD and Ideal.
Table 7: Normalization quality, as measured by the (cross-scctioanl) rank correlation between the realized residual and Mf for CPTD-M, PI width for QRNN, or é for LASplit.
Train QRNN
Tesst QRNN
Rank Correlation
CPTD-M
LASplit
CPTD-M
LASplit
MINIC
18.87466.20 20
35.21
42.28-16.85
24.80-9.21
27.60-1.886
14.54 10.49
Inaurance
28.3243.90
30.10=4.00
40.6443.119
25.52-5.21
20.93::4.80
14.83:44.91
COVID19
23.18.19.05
23.74.11..48
20.66.19.16
24.79.12.53
24.23.11..29
19.40
EEG
23.58-6..75
1.57
9.28-7.91
22.41=8.27
5.9347.70
6.84=7.92
9.1145548
26.63=7.50
21.67+9.43
19.29=8.78
20.30
13.38
13.38449.02
Load-R
10.52:16.00
2.990=7.99
26.2918.11
12.56=6.97
23.46::7.57
12.7745..98
Overall
20.28+9.112
22.08+133.1
21.36=14. 14.87
22. 02.03=9.60
18.03+12.89
12.51+9.95
21

---

Published in Transactions on Machine Learning Research (09/2022)
MIMIC
CLAIM
COVID
1.0
1.0
1.0
0.8
0.8
0.8
Coverage
0.6
0.6
CPTD-R
CPTD-R
CPTD-R
CPTD-M
CPTD-M
CPTD-M
0.4
CFRNN
0.4
CERNN
0.4
CFRNN
CQRNN
CORNN
CORNN
LASplit
LASplit
LASplit
QRNN
QRNN
QRNN
DPRNN
DPRNN
DPRNN
10
20
30
10
20
30
EEG
GEFCom
GEFCom-R
1.0
0.8
0.8
Coverage
0.6
0.6
CPTD-R
CPTD-R
CPTD-M
CPTD-M
CPTD-M
0.4
0.4
CFRNN
CORNN
CORNN
CQRNN
LASplit
LASplit
LASplit
0.2
0.2
0.2
QRNN
QRNN
QRNN
DPRNN
DPRNN
DPRNN
0.0
0.0
0.0
20
40
60
10
20
10
15
20
Figure Mcan coverage rate different Conformal methods exhibit cross-soctional validity as expected. On the other hand, coverage rate for non-conformal methods vary greatly through time.
Table Mean coverage, mean width, and tail coverage (re-scaled to same mean width) with linear regression point estimator. Valid mean coverage and the best of tail coverage and mean PI width are in bold.
Coverage
CPTD-R
CPTD-M
Split (CFRNN)
CQRNN
LASplit
89.72-11.58
89.98-10.79
89.78
89.95111.45
89.60-11.65
CDVID19
90.10+t1.99
90.25=1.24
90.07=1.82
90.01
90.08+11.22
EBC
80.01+1.73
90.13=1.41
89.90=1.77
90.40+2.17
90.29+1.22
Load
89.48:10.14
89.8610.11
89. 25.114
86.93110.26
89.2420.18
Load-R
90.16.10.65
90.2410068
90.07
89.98
10.13.10.4
Mean Width 4
MIMIC
2.812400.14
3.115#00.33
2 0.89902222
2.608::0.22
2.977=0022
CDVID19
0.780:10.019
0.876:5.0073
0. 0.808-00.331
0.829:10.047
0.807:10.022
1.632+0.082
1.663#0.005
1.653=0.091
1..98
1.810 070
Load
0.274110.003
0.309:1.000 004
0.277500.00.
0.332
0.298:10.000
Load-R
0.266.0.009
0.295.10.009
0.321100.099
0.28410.007
Coverage Rate (Scaled)
MIMIC
64.1224.97
69.05=2.24
60.5545.00 94
63.80+44..39
62, 02.55+5..2 22
CDVID19
70.16:15.80
71.59=2.69
61.16:66.86
66.62:11.99
69.0615.52
BBC
63.46:14.09
16.26.12.66
61.44.1..65
52.04
62.7912.55
73.22-10.86
60.9310.73
60.07:41.35
Load-R
74.36.11.26
7..56:1.26
70.95:11.09
63.21111.64
70.161:1.23
Tail Coverage Rate (Unscaled)
MIMIC
63.4345..0
72.70=22.0
60.55445.94
59.2744.14
63.75+5.30
COVID19
68.28=5.52
75.12=2.71
61.16+66.4
67.8845.05
69.03=5.55
BBC
63.02214.04
66.57
61.44.1..66
35.2411..25
67.8612.38
Load
73.15.10.85
70.55
89.18.11.10
69.75-11.52
71.39
Load-R
1..1521.25
16.0
70.95: 1.09
72.484 1.39
73.21#1.66
22


Reviews:
[{"review_id": "Uk9FF6e46x", "reviewer": "Reviewer_mkp5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, rigorous and applies conformal prediction to a rather under-explored domain (time-series forecasting). I have read the paper carefully and I believe all the technical claims to be correct.\n\n*The key weaknesses of the paper are:*\n\n**1) [Writing-related]** I think that the notions of \"cross-sectional\" and \"longitudinal\" validity are rather unnecessary and can mislead the readers into thinking that these are conceptually different notions of validity than what is already being used in the CP literature. Essentially, cross-sectional validity is marginal validity, and longitudinal validity is conditional validity where conditioning is on filtrations of a stochastic process rather than a feature vector. Moreover, this terminology can be confusing since cross-sectional validity is required to hold for all $t$, which implies some form of longitudinal validity as well. Overall, I think validity is a probabilistic concept that should be grounded in probability measures (marginal/conditional) rather than feature dimensions (cross-sectional/longitudinal).\n\n**2) [Theory-related]** Theorem 2.1. is identical to the results from Lei and Wasserman and its reiteration by R. Barber et al. Building on my previous comment, if we take longitudinal validity to be just a special case of the well-known conditional validity then there is no reason for this work to dedicate so much space for the same impossibility result. Moreover, I expected the paper to propose a procedure that would achieved approximate longitudinal coverage (i.e., asymptotically or up to some gap in finite samples). The fact that the following sections did not propose a method that achieve longitudinal validity in any sense makes me wonder whether the theoretical discussion of longitudinal validity was needed in the first place.\n\n**3) [Methodological]** The proposed score is inspired by some of the earlier work on adaptive CP by normalizing residuals e.g. using a fitted location-scale model as in Lei and Wasserman, 2018. I do believe the reason this kind of construction is useful is that it enables proving approximate conditional coverage, etc, but it is not necessarily the approach that would perform the best in terms of conditional coverage and efficient. Since the authors did not show any provable benefit from the proposed score (better efficiency or provable approximate longitudinal coverage), the particular choice of the proposed score is not sufficiently justifiable. Why not just start with a model that estimates the conditional CDF given $S_{1:t}$ and then conformalize interval estimates? This would probably perform better as it conditions on the entire time-series context and not just aggregate information on residuals, which will be beneficial in heteroskedastic noise setups within a time-series (which is precisely the setup of interest for this paper). Methods that conformalize interval estimates (similar to the CQRNN that the authors use as baseline) are SOTA in non-time-series settings and they seem like a more sensible modeling choice than the proposed score.\n\n**4) [Experiments]** While the experiments demonstrated the benefit of the proposed method, it did not fully explain why the proposed method outperforms others. Particularly, I am very surprised that the proposed score outperforms CQRNN since both methods should be incorporating the same temporal information in their scores. It is also not clear how the RNN model was conformalized and how where the hyper-parameters selected? Wouldn't some selections of hyper-parameters favor one method over the other? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper applies the conformal prediction approach to time-series data with the goal of providing provably valid predictive forecasts of sequential regression targets. The paper defines two notions of validity that pertain to time series data; *cross-sectional validity* which refers to the satisfaction of frequentist coverage across realizations of time series, and \"longitudinal validity\" which refers to coverage guarantees that hold conditioned on a realization. The authors show that a distribution-free version of the latter is impossible to satisfy.\n\nThe main contribution of the paper is a conformity score that allows for incorporation of the observed model errors in previous time steps when constructing the predictive intervals for future time steps. Unlike previous methods which ignore temporal dependencies, the two proposed methods incorporate temporal information by normalizing the conformity scores (residuals) based on aggregate information on observed residuals within the same time-series. The first variant of the proposed method (CPTD-M) applies a temporally-informed MAD normalization to each residual, and the second variant (CPTD-R) incorporates cross-sectional information as well. In both cases, the exchangeability condition is respected and hence the marginal validity guarantees of CP applies. \n\nThe experiments section demonstrates the effectiveness of the proposed method in terms of efficiency (interval width) and achieved coverage on a variety of real world data sets. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, I believe the paper is adequate for publication in TMLR conditioned on a major revision. Particularly, I encourage the authors to consider the following:\n\n1) Providing clearer explanation on how longitudinal validity differs from conditional validity and perhaps consider modifying the terminology.\n\n2) Adding a discussion on whether approximate longitudinal validity is achievable with the proposed methods. Ideally, a further theoretical analysis of temporally normalized residuals and whether it achieves asymptotic longitudinal validity will make this work significantly more complete.\n\n3) Providing more discussion justifying the choice of residual normalization over other conditional approaches (such as conditional histograms, conformalized QR, localized CP, etc) for the time-series setup.\n\n4) Expanding the set of experiments to provide other simulations (perhaps ablation experiments) that highlight the sources of performance gain from the proposed method compared to CQRNN to enable the readers to understand what information does your conformity score incorporates that CQRNN does not.", "broader_impact_concerns": "I do not believe there are ethical implications for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PFmsveVmmA", "reviewer": "Reviewer_7oD3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Caveat: I am not well acquainted with the conformal prediction literature and the underlying theory, as such I cannot report on the coverage of related work or critique the theoretical results.\n\nThe paper is well-written and generally easy to understand. The figures and presentation are clean.\n\nThe breadth of empirical validation seems adequate, and the proposed methods work well.\n\nOverall, my answers for the questions we are supposed to ask are (under above caveats):\n\n1. Are the claims made in the submission supported by accurate, convincing and clear evidence? Yes.\n\n2. Would at least some individuals in TMLR's audience be interested in knowing the findings of this paper? Yes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper first distinguishes between cross-sectional and longitudinal validity (coverage guarantees) for time-series predictions, in a conformal prediction setting. (longitudinal means validity across the temporal access for a single time series).\n\nIt shows that meaningful longitudinal coverage is impossible to achieve in general.\n\nDespite this, it introduces Conformal Prediction with Temporal Dependence in two variants (CPTD-M/R) and evaluates the performance compared to baselines, examining the theoretical properties (cross-sectional validity).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The claims in regards to outperforming SotA baselines in the conclusion ought to be clarified slightly more: the method is better in regards to the mean of PI width and tail coverage rate.  A reader might look at table 3 and wonder whether CPTD-* are outperforming the baselines or not. Conformal methods obviously ought to have an average coverage rate according to the target $\\alpha$, but this might not clear on a first glance.\n\n## Minor requests:\n\nWBCC in the introduction should be highlighted more or again explained later when the abbreviation is used. Took me a bit to figure out where it came from.\n\nIn eq. 6. $V(Z_{N+1})$ has already been replaced with $\\infty$ (text below eq (7)) and the underbrace should only be on $|y_i - \\hat y_i|$ in eq (6).\n\n\"As a proof of concept, we use...\" <-- why only PoC? Does this mean CPTD-M should not be used in practice? CPTD-R which outperforms CPTD-M is also only introduced as \"a simple example\". Same question: if these are only PoC and simple examples, should we not use them and what should we use instead? The wording might have to be tweaked here.\n\nIn eq. (11).: why the fraction with $\\infty$ and how should it be resolved?\n\nFigure 2. Remove EEG label and add sub captions.\n", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UMx8VLJWzj", "reviewer": "Reviewer_oJ9d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths:\n\n1. One contribution of this paper is to distinguish the two types of coverages (cross-sectional and longitudinal) and show the limitations in achieving the longitudinal one. Figure 1 illustrates the difference clearly.\n\n2. The problem of predictive inference in time series is important. The paper provides good motivation for predicting white blood cell count (WBCC) as a running example for the whole paper. \n\n3. The simulations are conducted on a wide range of data sets and are compared with several baselines.\n\n### Weakness:\n\n1. The discussion of two types of coverages is an interesting contribution. That being said, I do have some questions about Theorem 1. The probability of Theorem 1 is written as $P_{Y_{N+1} | S_{N+1, 1:t-1}}$, but the complete notation should be $P_{ (X_{N+1}, Y_{N+1}) | S_{N+1, 1:t-1}}$ where $X_{N+1}$ are the covariates for the N+1 unit. In that sense, the coverage is still marginal rather than conditional at the time point level. Can the Barber et al. (2020) proof technique for the negative results on the conditional coverage be directly applied in Theorem 1? I think there is a third type of coverage here with the probability $P_{Y_{N+1} | S_{N+1, 1:t-1}, X_{N+1}}$, which corresponds to conditional coverage in the non-temporal data. \n\n2. The empirical results of this paper are not very strong. Any valid conformal prediction method can achieve the target coverage, and an important metric for evaluation is sharpness. From Table 4, the sharpness of CPTD is on par with the existing conformal methods that can deal with time series as well. This makes the contribution less significant. One advantage might be the empirical improvement of the longitudinal coverage. But it is unclear why the algorithm design helps the longitudinal coverage, and this advantage is only shown on 2 out of 6 data sets in Figure 2. \n\n3. The clarity of the paper needs improvement by revision. See the comments in the requested changes. \n \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThis paper proposes a conformal prediction method that can provide valid cross-sectional coverage for the time series data. The paper mathematically distinguishes cross-sectional coverage and longitudinal coverage. CPTD is proposed with a new nonconformity score design. Empirical studies show that CPTD has valid marginal coverage and improves longitudinal coverage. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\n1. Add a proof sketch of Theorem 1 to the paper. Highlight the failure mode for the longitudinal coverage. Maybe discuss the third type of coverage as mentioned in Weakness (1). \n\n2. Have a systematic discussion on the sharpness. For example, it can be new theoretical results on the coverage upper bound and the discussion (and simulation study) on why CPTD can improve longitudinal coverage;  see Weakness (2).\n\n3. Improve clarity:\n\n(a) Adding implementation details in Section 4:  Are all the methods using the same neural network and training methods? If so,  $\\hat{y}$ are the same for all the methods, which is the center of the predictive interval. Does this mean all methods with similar coverage would have similar interval lengths? Are the QRNN and DPRNN in Table 3 conformal methods? Add title and axis label in Figure 2. \n\n(b) Current paper does not situate itself well in literature. The Related Works should be much earlier than just before the conclusion. Ideally, the Related Works should not just introduce existing works but describe what current methods can do, what they cannot, and what existing CPTD can solve infeasible problems. Currently, it is unclear what contributions CPTD brings to the existing literature. Some methods mentioned in \"Long and Single Time-Series and Longitudinal Validity,\" such as Xu & Xie (2021), seem to be proper baselines. \n\n(c) Several minor places may cause confusion: (i) Figure 1 caption \"red crosses that are randomly distributed across samples\". It seems the red crosses are always at the bottom sample; (ii) superscript in Eq. (9); (iii) The paragraph above Eq. (10). ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates conformal prediction for time series data. The authors define two notions of validity, namely cross-sectional validity and longitudinal validity. The authors propose "conformal prediction with temporal dependence" (CPTD), a method to maintain cross-sectional validity while improving longitudinal coverage. The reviewers found the paper well-written and interesting. Experimental results demonstrate the effectiveness of CPTD on a variety of real world datasets. The authors satisfactorily addressed the reviewer comments and all the reviewers lean towards acceptance.

I recommend accept. Congratulations on the nice work! 
Decision: Accept (poster)