Copula Conformal prediction for multi-step time series prediction
Authors: Sophia Huiwen Sun
Rose Yu
Venue: ICLR 2024 poster (2024)
Keywords: Conformal Prediction
time series
uncertainty quantification
calibration
RNN
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: significantly improve efficiency/sharpness of conformal prediction confidence intervals, for multi-step time series forecasting, by modeling dependence of time steps using copulas

Abstract: Accurate uncertainty measurement is a key step in building robust and reliable machine learning systems. Conformal prediction is a distribution-free uncertainty quantification framework popular for its ease of implementation, finite-sample coverage guarantees, and generality for underlying prediction algorithms. However, existing conformal prediction approaches for time series are limited to single-step prediction without considering the temporal dependency. In this paper, we propose the Copula Conformal Prediction algorithm for multivariate, multi-step Time Series forecasting, CopulaCPTS. We prove that CopulaCPTS has finite-sample validity guarantee. On four synthetic and real-world multivariate time series datasets, we show that CopulaCPTS produces more calibrated and efficient confidence intervals for multi-step prediction tasks than existing techniques. Our code is open-sourced at https://github.com/Rose-STL-Lab/CopulaCPTS.

Published as a conference paper at ICLR 2024
COPULA CONFORMAL PREDICTION FOR MULTI-STEP TIME SERIES FORECASTING
Sophia Sun
Rose Yu
University of California, San Diego shs066eucsd. edu
University of California, San Diego
roseyu@ucad. edu
ABSTRACT
Accurate uncertainty measurement is a key step in building robust and reliable machine learning systems. Conformal prediction is distribution-free uncertainty quantification framework popular for its case of implementation, finite-sample coverage guarantees, and generality for underlying prediction algorithms. However, existing conformal prediction approaches for time series are limited to single-step prediction without considering the temporal dependency. In this paper, we propose the Copula Conformal Prediction algorithm for multivariate, multi-step Time Series forecasting, CopulaCPTS. We prove that CopulaCPTS has finite-sample lidity guarantee. On four synthetic and real-world multivariate time series datasets, we show that CopulaCPT' produces more calibrated and efficient confidence intervals for multi-step prediction tasks than existing techniques. Our code is open-sourced /tttpp:://gthaae com/ Rose STL. DabpoppacCCp
INTRODUCTION
Deep learning models are becoming widely used in high-risk settings such as healthcare and trans- portation. In these settings, is important that model produces calibrated uncertainty to reflect its own confidence. Confidence regions are a common approach to quantify prediction uncertainty (Khosravi al. 2011). A (1 a) -confidence region rl-a for random variable y is valid if contains y true value with high probability: PPy E [ 2 - a. Note that one can make the confidence region infinitely large to satisfy validity. But for the confidence region to be useful, also want to minimize its area while remaining valid; this is known as the efficiency of the region.
Conformal prediction (CP) is a powerful framework to produce confidence regions with finite-sampl guarantees of validity (Vovk et al.) 20055] Lei et al., 2018). Furthermore, it makes no assumptions about the underlying prediction model or the data distribution. CP's generality, simplicity, and statistical guarantees have made it popular for many real-world applications including time series prediction [u & Xie 202I), drug discovery (Eklund et al. 2015) and safe robotics (Luo et al. 2021).
This paper considers the setting of multi-step time series forecasting from set of independent sequences. Consider the problem vehicle trajectory prediction, illustrated in Figure Given
Time steps: temporal dependence (non-i.i.d.)
Dataset D: independent observations (l..d.)
Xy
. Yk
Figure 1: Ilustrationn the multi-step time series forecasting setting. (Left) The timesteps within time series are temporally dependent, and (Right) the observations in the dataset independent.

---

Published as a conference paper at ICLR 2024
dataset of trajectories, the task is to predict future trajectory for k steps given its past trajectory of t time steps. We assume that the trajectories are independent from each other. For each trajectory, these time steps temporally dependent.
There are many real-world tasks that present the same challenges as the example above, such as EEG forecasting (each patient is independent), short-term weather forecasting (local meteorology history is independent), etc. They require predicting multiple time steps into the future, so is desired to have "cone of uncertainty" that covers the entire course of the forecasts. Existing CP methods for time series data either only provide coverage guarantee for individual time steps (Gibbs & Candes) 2021 Xu & Xie (02I) or produce confidence regions that are often too inefficient be useful, especially in long horizons or multivariate settings (Stankevičiüte et al., (2021).
In this paper, we present a practical and effective conformal prediction algorithm for multi-step time series forecasting. We introduce CopulaCPTS, Copula- based Conformal Prediction algorithm for multi-step Time Series forecasting. copula is multivariate cumulative distribution function that models the dependence between multiple random variables. By using copulas to model the uncertainty jointly over future time steps, we can shrink the confidence regions significantly while maintaining validity. Copulas have been used for conformal prediction (Messoudi et al. (02T), but they focus on multiple target prediction in non-temnnn non-temporal settings and did not provide a validity proof.
In summary, contributions are:
We introduce CopUlaCPTS, a general uncertainty quantification algorithm that can be applied to any multivariate multi-step forecaster.
We prove that CopulaCPTS produces valid confidence regions for the full forecast horizon.
CopulaCPTS produces significantly sharper and more calibrated uncertainty estimates than state-of-the-ang baselines on two synthetic and two real-world benchmark datasets.
We extend CopulaCPTS obtain valid confidence intervals for time series forecasts of varying lengths.
2 RELATED WORK
Deep Uncertainty Quantification for Time-Series Forecasting. The two major paradigms of Uncertainty Quantification (UQ) methods for deep neural networks are Bayesian and Frequentist. Bayesian approaches estimate a distribution over the model parameters given data, and then marginal- ize these parameters to form output distributions via Markov Chain Monte Carlo (MCMC) sampling
(Welling& & Teh 2011: Neal, 2012, Chen et al.) 2014) or variational inference (VI) (Graves 201
Kingma et al. 2015 Blundell et al.] [015]] Louizos & Welling 2017). Waang et al. (2019): ; Wu
ctt al. (2021) propose Bayesian Neural Networks (BNN) for UQ of spatiotemporal forecasts. In
practice, Bayesian UQ can be computationally expensive and difficult to optimize, especially for
larger networks (Lakshminarayanan et al., 2017 Zadrozny & Elkan 2001). Furthermore, Bayesian methods do not provide any finite sample coverage guarantees. Therefore, UQ for deep neural network time series forecasts often adopts approximate Bayesian inference such as MC-dropout (Gal
Ghahramani 2016b
Gal al. 2017).
Frequentist UQ methods emphasize robustness against variations in the data. These approaches either rely on resampling the data or learning an interval bound encompass the dataset. For time series forecasting UQ. approaches include ensemble methods such as bootstrap (Efron & Hastie 2016 Alaa
Van Der Schaar 2020) and jackknife methods (Kim et al. 2020 Alaa & Van Der Schaar (2020);
interval prediction methods include interval regression through proper scoring rules ( (Kivaranovic
jtt al., 2020 Wu et al. (02T)) and quantile regression (Takeuchi et al.] (0066) with many recent advances for time series UQ (Tagasovska & Lopez-Paz [019 Gasthaus et al. [019 Park et al. 2022 Kan et al. 2022). Many of the frequentist methods produce asymptotically valid confidence regions and can be categorized as iistribution-fres UQ techniques as they (1) agnostic to the underlying model and (2) agnostic to data distribution.
Conformal Prediction. Conformal prediction (CP) is an important member of distribution-fre freee UQ methods; we refer readers to AAngllopoulos & Bates (202T) for a comprehensive introduction and survey of CP. CP has become popular because of its simplicity, generality, theoretical soundness, and
2

---

Published as a conference paper at ICLR 2024
low computational cost. A key feature of CP is that under the exchangeability assumption, conformal methods guarantee validity in finite samples (Vokk et al. 2005).
Most relevant to our work is the recent endeavor in generalizing CP time-series forecasting. According to Stankevičiūte et al. (202T) there are two settings: data generated from (1) one single
time series (2) multiple independent time series. For the first setting, ACI (Gibbs & Candes 2021) and EnbPI (Xu & Xie 2021) developed CP algorithms that relax the exchangeability assumption while maintaining asymptotic validity via online learning (former) and ensembling (later): Zaffran et al. 2022) further improves online adaptation. Sousa et al. (2022) combines EnbPI with conformal quantile regression (Romano al. 2019) to model heteroscedastic time series. However, because these algorithms operate on one single time series, the validity guarantees do not cover the full horizon, posing issues for application in high-risk settings.
We focus on the setting where data consists of many independent time series. Stankevičiūte et al. (202T) shares the same setting as ours provides only univariate time series solution. We show that their method of applying Bonferroni correction produces inefficient confidence regions, especially for multidimensional data or long prediction horizons. [Messoudi et al. (2021) uses a copula function for multi-target CP for non-temporal data, creating box-like regions account for the correlations between the labels. However, they do not provide theoretical proof and empirical results have shown that are often invalid. This paper builds upon these works and presents novel two-step algorithm with guaranteed multivariate multi-step coverage and efficient confidence regions.
BACKGROUND
3.1 INDUCTIVE CONFORMAL PREDICTION (ICP)
Let D = {2- (''-"")) }/ be a dataset with input x' E X and output y² E y such that each data point 2' = x D is drawn i.i.d. from unknown distribution Z.
We will briefly present the algorithm and theoretical results for conformal prediction, and refer readers to Angelopoulos & Bates (021) for thorough introduction. The goal of conformal prediction is to produce a valid confidence region (Def. 3.1 for any underlying prediction model.
Definition 3.1 (Validity). Given a new data point (xr, y) and a desired confidence 1 1 E (0, 1), the confidence region I [l-0(x) is subset of y containing probable outputs ÿ ) given x. The region rl-a is valid if
Ply E - (l-a(x)) (x)) 2 a
Conformal prediction splits the dataset into a proper training set Dtrain and a calibration set Deal- A prediction model f X' + y is trained on D Derain- We use nonconformity score 2 g(Poriill x Z + R to quantify how well data sample from calibration conforms to the training dataset. Typically, we choose a metric of disagreement between the prediction and the true label as the non-conformity score, such as the Euclidean distance:
A(D (2,8)) cR d(y, f(x)) o By
f(x)
For conciseness, we write A(Drrain (x", y²)) A(=²) in rest of the paper.
Let S = (A<='))=<ED... denote the set of nonconformity scores of all samples the calibration set Deat- We can define a quantile function for the nonconformity scores S as:
- Ql inf(s"
2
S'ES
Conformal prediction is guaranteed to produce valid confidence regions (Vok et al. 2005). under the exchangeablility assumption defined follows,
Definition 3.2 (Exchangeability). In a dataset {21) of size n, any of its n! permutations are equally probable.
The procedure introduced above is known as inductive conformal prediction. as it splits the dataset into training and calibration sets to reduce computation load (Vovk et al.] (0055] Lei & Wasserman 2012). Our method is based inductive CP, but can also be easily adapted for other CP variants.
3

---

Published as a conference paper at ICLR 2024
3.2 COPULA AND ITS PROPERTIES
Copula is concept from statistics that describes the dependency structure in a multivariate distri- bution. has also been used in generative models for multivariate time series (Salinas al. 2019 Drouin et al,, 2022). We can use copulas to capture the joint distribution for multiple future time steps. We briefly introduce its notations and concepts.
Definition 3.3 (Copula). Given random vector (X1.... Xx), define the marginal cumulative density function (CDF) for each variable Xn. h E {l..... kk as
(x( x S x]
The copula of (X1,".. X&) the joint CDF of (X1); ... , ;Fxx)))) written as
C(u1.- ux) = P $ " F((R)) S ux]
In other words, the copula function captures the dependency structure between the variable Xs; we view an & dimensional copula C:: [0, 1] [0, 1] CDF with uniform marginals, illustrated in Figure 2 A fundamental result in the theory of copula is Sklar's theorem.
Theorem 3.4 (Sklar's theorem). Given joint CDF as F(X1,". _xx) and the marginals Fi(x)...., (T)). there exists a copula such that
F(I1,'. II) C(F(TI), (,F(((()))))))
for all z; E 00), E (ENj)E((,.... k}.
Sklar's theorem states that for all multivariate distribution functions, there exists copula function such that the distribution can be expressed using the copula and multiple univariate marginal dis- tributions. When all the Xgs are independent, the copula function known as the product copula:
C(u 4x) = I 1 44-
Sample distribution
Copula Cltp.
CDF of 1
680
CDT 25 55
21000.2
455
Sumple denittttion CD
Figure 2: An example copula, where express multivariate Gaussian with correlation p 0.8 with two univariate distributions and a copula function C(u1, W2).
COPULA CONFORMAL PREDICTION FOR TIME SERIES (COPULACPTS)
UQ methods are evaluated on two properties: validity and efficiency. A model valid when the predicted confidence is greater than equal to the probability of events falling into the predicted range (Definition T. The term calibration describes the case of equality Efficiency, the other hand, refers to the size of the confidence region. In practice, we want the measure of the confidence region (c.g. its area or length) to be as small as possible, given that the validity condition holds. CopulaCPTS improves the efficiency of confidence regions by modeling the dependency of the time steps using a copula function.
the validity condition.
Denote the time series dataset of size D {2' (Fi..> vi.e) where T1. R is input steps, and Visk k is k prediction steps, both with dimension d at each step. Each data point z' is sampled i.i.d. from an unknown distribution Z. In the multi-step forecasting setting, given a confidence level 1 a, the algorithm produces & confidence regions for test input x denoted as [[]- ..... pl-o]. We say the confidence regions valid if all time steps in the forecast covered:
P[Vj (1, (1...., k), Yi E -
4

---

Published as a conference paper at ICLR 2024
In the following sections, we introduce CopulaCPTS, conformal prediction algorithm that is both valid and efficient for multivariate multi-step time series forecasts.
4.1 ALGORITHM DETAILS
The key insight of our algorithm is that we can model the joint probability of uncertainty for multiple predicted time steps with copula, hence better capturing the confidence regions. We divide the calibration set Dent into two subsets: Dont-1, which we use estimate a Cumulative Distribution Function (CDF) the nonconformity score of cach time step, and Deal-2, to calibrate the copula.
The two calibration sets allow us to prove validity for both components of our algorithm. At the cost of using subset of the data to calibrate copula, our approach produces provably more efficient confidence regions compared to worst-case corrections such as union bounding in Stankevičiüte et al. (2021) which is lower bound for copulas (Appendix B.II. and more valid regions than Messoudi et al. 2021) (table T).
Nonconformity of Multivariate Forecasts. If the time series is multivariate, we have each target time step yj Rdd. Given z (x, Z, let the nonconformity score be the L-2 distance llv} f(x))] for each timestep j 1,.... k. where f(x)) is a forecasting model
8) A(=") c trained on Derain- The confidence region ['=0(z) therefore is a d- d-dimensional ball. We chose this metric for simplicity, but one can choose other metrics such as Mahalanobis (Johnstone & Cox 2021) 1.-1 (Messoudi et al. 202T) distance based domain needs, and our algorithm will remain valid.
For brevity, we will use $1 = {s*) =' to denote the set of nonconformity scores of data in Deat-1 and S2 = { beeeDen- the set of nonconformity scores of data in Deat-2 Subscript will be used to index the set of specific time steps of the scores: S1, {s) xie.aie S2, =E
Calibrating CDF Deat-1- We Deat-1 build conformal predictive distributions for each time step's anomaly scores, which provides desirable validity properties (Vovk et al. (017). The conformal cumulative distribution function constructed follows.
(5();)
T+ Σ
Lsfesy): where (0,1), for j E (I,...,k) s'ESI,
(5)
S1 +1
Copula Calibration on Deat Next, for every data point in Deat-2 we evaluate the cumulative probability of its anomaly scores with the estimated conformal predictive distributions:
u = Hu' u = (u.....,uuy utt) (f.(s)..... Pa(s)))
(6)
We adopt the empirical copula (Ruschendori 1976) for modeling and proof in this work. The empirical copula is non-parametric method of estimating marginals directly from observation. and hence does not introduce any bias. For the joint distribution of k time steps, we construct Cempirical
1] [0, 1] Eqn
Cempiricat(()
Σ
II
DDeat
uj <uy
iED..t 20((0)) f=1
Here boldface oo is k-dimensional vector with each oo, = oo for j l....,k
To fulfill the full-horizon validity condition of Eqn 4 we only need to find appropriate u² such that
Cempirical(1t (u*) 2 a.
arg min u
u
s.t. Compirical (u') 2
(8)
'Because of the random component, equation inconsequential when the calibration set is large. See |Vokk et al. (207)) for theoretical justifications. In implementation, for simplicity and better coverage.
"thick" CDF of thickness 1 which becomes
5

---

Published as a conference paper at ICLR 2024
Note that the u" is not and does not have to be unique; any solution that satisfies the constraint in Eq. [ will guarantee multi-steg validity (Appendix A). The minimization helps with efficiency, i.e. the sharpness of the confidence regions. We use gradient descent algorithm for the optimization in implementation (see Appendix [B.2 for details, and Appendix CC5 for a study of its effectiveness).
Lastly, We obtain (si...., , by F (uj) and construct the confidence region for each time step jE (1....,k) } the set of all y, Rd such that the nonconformity score is less than s.. Algorithm 1 summarizes the CoupulaCPTS procedure.
The full proof of CopulaCPTS's validity (theorem 4.1 can be found in Appendix A Intuitively, CopulaCPTS performs conformal prediction twice: first calibrating the nonconformity scores of each time step with 1 and then calibrating the copula with Doal-
Theorem 4.1 (Validity of CopulaCPTS). CopulaCPTS (algorithm ? produces valid confidence regions for the entire forecast horizon. i.e.
Pi E (L...., k}, Mo E 1 1-a
Algorithm I: Copula Conformal Time Series Prediction (CopulaCP'SS)
Input: Dataset DD test inputs Drest, target significant level a. for each test input.
Output: 1 aa ................................ 1 pl-a
// Training Randomly split dataset D into Drrain and Deal-
Train k-step forecasting model f on training set D Dtrain-
// Calibration bibbiationn
Randomly split Deat into Dest-1 and Deat
for
(ri virt) D DDell2 2 do
Vi + (x(
8) < llu} ا5 for 1....,
10 end for
Fi...., PE Eq. on Deal-
12 Cempinical
Eq. on Deal-2
u Eq.
14 8 Fr- (u)) for j .....
15 Prediction
16 for 2i E Diest do
Fia + f( {y: |1 - 1 < <83] for j 1....,K
17
r
18
yield r rl-a
19
20 end for
EXPERIMENTS
In this section, show that CopulaCPTS produces calibrated and efficient confidence regions compared to existing methods on two synthetic datasets and two real-world datasets. We demonstrate that CopulaCPTS's advantage is more evident over longer prediction horizons in Section 5.2 We also show its effectiveness in the autoregressive prediction setting in Section 5.2
All experiments in this paper split the calibration Although the split does not significantly impact the result when calibration data is ample, performance deteriorates when there are not enough data either one of the subsets.
half into equal- D P and D Pcal-2-
Baselines.
We compare our model with three representative works in different paradigms of deep uncertainty quantification: the Bayesian motivated
Sayesian-motivvate Monte Carlo dropout RNN (MC-dropout) by Gal Ghahramani (2016a), the frequentist blockwise jackknife RNN (BJRNN) by Alaa & Van
Der Schaar (2020), conformal forecasting RNN (CF-RNN) by Stankevičiüte et al. (002 and

---

Published as a conference paper at ICLR 2024
Table 1: Performance in synthetic and real-world datasets with target confidence 0.9. Methods that are invalid (coverage below 90%) are greyed out. CopulaCPTS achieves high level of calibration (coverage close to 90%) while producing more efficient confidence regions.
MC-dropout
B.RNN
CF-RNN
Copula
CopulaCPTS
Partiele Sim (o = .01)
Cov
91.5 2.0
98.9 002
97.3 +
86.9 1
91.3 = 1.5
Arca
2.22 0.05
2.24 0.59
1.97 #04
0.63 0007
1.08
0034
Particle Sim (a = .05)
Cov
46.1
= 3.7
100,0
= 0.0
94.5
1.5
88.6 =
90.6
= 0.6
Area
2.16 0.08
12.13
+ 0.39
5.80
+ 0.52
4.67 # 0.16
5.27
= 1.02
Drone Sim
Cov
84.5
= 10.8
90.8
2.8
91.6
9.2
89.2
+ L.
90.0
= 0.8
(a = .02)
Vol
9.64
2.13
49.57
+ 3.77
32.18
+ 13.66
16.92 8.9
17.12
= 6.93
COVID-19
Cov
19.1
5.1
79.2
d: 30.8
95.4
+: 1.9
90.8
1.4
90.5
+: 1.6
Daily Cases
Area
34.14
4: 0.84
823.3
529.7
610.2
+: 96.0
414,42
5.08
408.6
: 65.8
Argoverse
Cov
27.9
+ 3.1
92.6
+ 9.2
98.8
1.9
89.7
0.9
90.2
+ 0.1
Trajectory
Area
127.6
:: 20.9
880.8
+ 156.2
396.9
+: 18.67
107.2
+ 9.56
126.8
12.22
the multi-target copula algorithm that does not have the two step calibration (Copula) by Messoudi ct al. (202I). We use the same underlying prediction model for post-hoc uncertainty quantification methods BJRNN, CF-RNN, and CopulaCPTS. The MC-dropout RNN is of the same architecture but is trained separately, as it requires an extra dropout step during training and inference.
Metrics. We evaluate calibration and efficiency for each method. For calibration, we report the empirical coverage on the test set. Coverage should be as close to the desired confidence level
as possible. Coverage is calculated as:
Coverage E [')](=)]
Drese Exteg'EDenn 1 (T'-(('))..
For efficiency, we report the average area (2D) volume (3D) of the confidence regions. The measure should be as small as possible while being valid (coverage maintains above-specified confidence level). The area or volume is calculated as:
Areaj = Ex (a)l-((())) R Decat Exte| || (
5.1 SYNTHETIC DATASETS
We first test the effectiveness of our models on two synthetic spatiotemporal datasets interacting particle systems (Kipf et al.) 2018), and drone trajectory following simulated with RobhonRobotics (akai et al. 2018). For particle simulation, we predict Y++1:++h where t = 35, = 25 and ye E IR?; for drone simulation
60, h 10, and ye E R To add randomness to the tasks, added Gaussian noise of a = .01 and .05 to the dynamics of particle simulation and o = -02 to drone dynamics. We generate 5000 samples for each dataset, and split the data by 45/45/10 for train, calibration, and test, respectively. For baselines that does not require calibration, the calibration split used for training the model. Please see Appendix C.I for forecaster model details.
We visualize the calibration and efficiency of the methods in Figure 3 for confidence levels 1-a = 0.5 0.95. We can see that Copula-RNN, the red lines, are more calibrated and efficient compared to other baseline methods, especially when the confidence level is high (90% and 95%). We can see that for harder tasks (particle a 0.05, and drone trajectory prediction), MC-Dropout is overconfident, whereas BJ-RNN and CF-RNN produce very large (hence inefficient) confidence regions. This behavior CF-RNN is expected because they apply Bonferroni correction to account for joint prediction for multiple time steps, which is an upper bound of copula functions. Numerical results for confidence level 90% are presented in Table [ A qualitative comparison of confidence regions for drone simulation be found in Figure in Appendix C.4
5.2 REAL WORLD DATASETS
COVID-19. We replicate the experiment setting of Stankevičiūte et al. (2021) and predict new daily cases of COVID-19 19 in regions of the UK. The models take 100 days of data as input and forecast 50 days into the future. We used 200 time series for training, 100 for calibration, and 80 for testing.
7

---

Published as a conference paper at ICLR 2024
Figure 3: Calibration (upper row) and efficiency (lower row) comparison on different a levels for synthetic data sets. Shaded regions are + 2 standard deviations over 3 runs. For calibration, the goal is to stay above the green dotted (validity) and coincide as closely as possible (calibration). CopulaCPTS is more calibrated across different significance levels. For efficiency, want the metric to small. CopulaCPT'S outperforms the baselines consistently. (MC-dropout for the right two experiments produces invalid regions, so we don't consider its efficiency.)
Coverage (particle == 01)
1.0 Coverage (particle a =. 05)
Coverage (drone o= 02)
1.0
1.0
0.8
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
1-
1
Area (particle =001)
Area (particle .05)
Volume (drone 02)
15
60
40
20
0.9
1
1
1
MC-Dropout
BJRNN
CF-RNN
CopulaCPTS
Calibration Line
Vehicle trajectory prediction. The Argoverse autonomous vehicle motion forecasting dataset (Chang al. 2019) a widely used vehicle trajectory prediction benchmark. The task predict 3 second trajectories based on all vehicle motion in the past 2 seconds sampled at 10Hz. Because trajectory prediction a challenging task, we utilize state-of-the -fe-tt---rrtttt prediction algorithm LaneGCN (Liang et al. 2020) as the underlying model for both CF-RNN and Copula-RNN (details in Appendix C.I). Flexibility of underlying forecasting model is an advantage of post-hoc UQ methods such as conformal prediction. For model-dependent baselines MC-dropout and BJRNN, we have to train an
RNN forecasting model from scratch for each method, which induces additional computational cost.
CopulaCPTS is both more calibrated and efficient compared to baseline models for real-world datasets [.. The Covid-19 dataset demonstrates a potential failure case for our model when calibration data scarce. Because there only 100 calibration data, CDF and copula estimations stochastic depending on the dataset split, resulting in 1 case of invalidity among 3 experiment trials. Even so, CopulaCPTS shows strong performance on average by remaining valid and reducing the confidence width by 33%. For the trajectory prediction task, learning the copula results in a 40% sharper confidence region while still remaining valid for the 90% confidence interval. We visualize two samples from each dataset in Figure 3. The importance of efficiency in these scenarios is clear the confidence regions need be narrow enough for them to be useful for decision making. Given the same underlying prediction model, we can see that CopulaCPTS produces much more efficient region while still remaining valid.
more
Comparison of models at different horizon lengths. CopulaCPTS is algorithm designed to produce calibrated and efficient confidence regions for multi-step time series. When the prediction horizon long, CopulaCPTS's advantage is more pronounced. Figure 5 shows the performance comparison over increasing time horizons on the particle dataset. See Table 3 of Appendix C for numerical results. CopulaCPTS achieves 30% decrease in area 20 time steps compared to CF-RNN, the best performing baseline; the decrease is above 50% at 25 time steps. This experiment shows significant improvement of using copula to model the joint distribution of future time steps.
8

---

Published as a conference paper at ICLR 2024
Figure Illustrations of 90% confidence regions given by CF-RNN (blue) and CopulaCPTS (orange) on two real-world datasets, COVID-19 forecast (left 2) and Argoverse (right 2 at time steps 1, 10, 20, and 30). For the Argoverse data, The red dotted lines (ego agent) and blue dotted lines (other agents) are input to the underlying prediction model and the red solid lines are the prediction output. Note that the confidence region produced CF-RNN is uninformatively large, as it covers the lanes: these examples illustrate the importance of efficiency. Overall, CopulaCPTS is able to produce much more efficient confidence regions while maintaining valid coverage.
13M
Serret
Bulton
MAB Purccan
Cop PKM
ground truth
3200
CopulaCPTS for Autoregressive prediction. The autoregressive extension of CopulaCPTS is illustrated in detail in Appendix [B.3 To provide preliminary evidence of effectiveness, we present test results on the COVID-19 dataset. We train an RNN model with forecast the next 14 steps. Table 2 compares the performance of re-estimating the copula for each 7-step forecasts versus using fixed copula calibrated using the first 7 steps. We also compare the model 14-step joint forecaster using CopulaCPTS. is evident that daily cases of the pandemic is a non-stationary time series, where re-estimating the copula is necessary for validity.
and use it to autoregressively
Coverage (Partitle .000
Area (Particle
1.00
Method
Coverage
Area
0.95
2.0
0.00
1.5
AR re-estimate
90.1
89.4
AR fixed
88.2
75.9
0.85
10
Joint
90.7
102.3
0.80
0.5
0.75
13
20
10
15
Table Performance of autoregres- sive (AR) CopulaCPTS. Re-estimating copula gives valid confidence re- gion over time and is more efficient than joint CopulaCPTS forecast.
preciction on harizon
prediction norizon
MC-Droxcut
BRWN
CF-NNN
Cooula-AhiN
Colitretion Line
Figure 5: CopulaCPTS remains more calibrated and efficient than baselines over increasing forecast horizons.
6 CONCLUSION AND DISCUSSION
In this paper, we present CopulaCPTS, a conformal prediction algorithm for multi-step time series prediction. CopulaCPTS significantly improves calibration and efficiency of multi-step conformal confidence intervals by incorporating copulas to model the joint distribution of the uncertainty at each time step. We prove that CopulaCPTS has a finite sample validity guarantee horizon. Our experiments show that CopulaCPTS produces confidence regions that are (1) valid, and (2) more efficient than state-of thethee-rt UQ methods on all 4 benchmark datasets, and over varying prediction horizons. The improvement is especially pronounced when the data dimension is high or the prediction horizon is long, cases when other methods are prone be highly inefficient. Hence, argue that method is practical and effective way produce useful uncertainty quantification for machine learning forecasting models.
the entire prediction
The limitations of our algorithm suitable only when there are abundant data for calibration. The validity proof relies on using the empirical copula, it does not apply to other learnable copula classes. Future work includes (1) improving the autoregressive extension CopulaCPTS, to achieve coverage over the whole horizon, and (2) developing online settings of CopulaCPTS for decision making.
follows. As CopulaCPTS requires two calibration steps, it

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported part by Army-BCASE, award W911NN-23-1-0231, the U.S. Department Of Energy, Office of Science under #DE-SC0022255, IARPA HAYSTAC Program, CDC-RFA-FT- 23-0069, NSF Grants #2205093, #2146343, and #2134274.
We would like to thank Bo Zhao for her helpful comments on the paper.
REFERENCES
Ahmed Alaa and Mihaela Van Der Schaar. Frequentist uncertainty in recurrent neural networks via blockwise influence functions. In International Conference Machine Learning, pp. 175-190. PMLR, 2020.
Anastasios Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.05111 20221.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Ming-Fang Chang, John W Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Alexandre Drouin, Étienne Marcotte, and Nicolas Chapados. Tactis: Transformer-attentionn copulas for time series. arXiv preprint arXiv:2202.03525 2022.
Bradley Efron and Trevor Hastie. Computer Age Statistical Inference, volume 5. Cambridge University Press, 2016.
Martin Eklund, UIf Norinder, Scott Boyer, and Lars Carlsson. The application of conformal prediction to the drug discovery process. Annals of Mathematics and Artificial Intelligence, 74(1):117-132, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016a.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29, 2016b.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In NIPS, pp. 3581-3590, 2017.
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function RNNs. In AISTATS 22, pp. 1901-1910, 2019.
Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34, 2021.
Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
Chancellor Johnstone and Bruce Cox. Conformal uncertainty sets for robust optimization. In
Conformal and Probabilistie Prediction and Applications, pp. 72-90. PMLR, 2021.
Kelvin Kan, François François-Xavier Xavier Aubet, Tim Januschowski, Youngsuk Park, Konstantinos Benidis, Lars Ruthotto, and Jan Gasthaus. Multivariate quantile function forecaster. In International Conference Artificial Intelligence and Statistics, pp. 10603-10621. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2024
Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F Atiya. Comprehensive review of neural network- prediction intervals and new advances. IEEE Transactions on neural networks, 22(9):1341-1356 2011.
Byol Kim, Chen Xu, and Rina Barber. Predictive inference free with the jackkmifct-affer-boottee In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4138-4149. Curran Associates, Inc., 2020.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizatioo trick. Advances in neural information processing systems, 28, 2015.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint arXiv:1802.04633 2018
Danijel Kivaranovic. Kory D Johnson, and Hannes Leeb. Adaptive, distribution- prediction intervals for deep networks. In AISTATS, pp. 4346-4356, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive
uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Jing Lei and Larry Wasserman. Distribution free prediction bands. arXiv preprint arXiv: /203.5422, 2012.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribation-free predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094-1111, 2018.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In European Conference Computer Vision, pp. 541-556. Springer, 2020.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pp. 2218-2227. PMLR, 2017.
Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and Marco Pavone. Sample-efficient safety assurances using conformal prediction. arXiv preprini arXiv:2109. 14082, 2021.
Soundouss Messoudi, Sébastien Destercke, and Sylvain Rousseau. Copula-based conformal predic- tion for multi-target regression. Pattern Recognition, 120:108101, 2021.
Soundouss Messoudi, Sébastien Destercke, and Sylvain Rousseau. Ellipsoidal conformal inference for multi-target regression. In Conformal and Probabilistic Prediction with Applications, pp- 294-306. PMLR, 2022.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Youngsuk Park, Danielle Maddix, Lrançois-Xavien Aubet, Kelvin Kan, Jan Gasthaus, and Yuyang Wang- Learning quantile functions without quantile crossing for distribution-free time series forecasting. International Conference on Artificial Intelligence and Statistics, pp. 8127-8150. PMLR, 2022.
Yaniy Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
Ludger Ruschendorf. Asymptotic distributions of multivariate rank order statistics. The Annals of Statistics, pp. 912-923, 1976.
Atsushi Sakai, Daniel Ingram, Joseph Dinius, Karan Chawla, Antonin Raffin, and Alexis Paques. Pythonrobotics: python code collection of robotics algorithms. CoRR, 0ts/1808.1000 2018.
11

---

Published as a conference paper at ICLR 2024
David Salinas, Michael dimensional multivariate forecasting with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
Laurent Callot, Roberto Medico, and Jan Gasthaus. High-
Martim Sousa, Ana Maria Tomé, and José Moreira. A general framework for multi-step ahead adaptive conformal heteroscedastic time series forecasting. arXiv preprini arXiv:2207.142) 14219, 2022.
Kamilé Stankevičiüté, Ahmed Alaa, and Mihaela van der Schaar. Conformal time-series forecasting. In Advances in Neural Information Processing Systems, 2021.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In NewrIPS, pp. 6417-6428, 2019.
Ichiro Takeuchi, Quoc Le, Timothy Sears, Alexander Smola, et al. Nonparametric quantile estimation. MIT Press, 2006.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning random world. Springer Science & Business Media, 2005.
Vladimir Vovk, Jieli Shen, Valery Manokhin, and Min-ge Xie. Nonparametrie predictive distributions based on conformal prediction. In Conformal and probabilistic prediction and applications, pp- 82-102. PMLR. 2017.
Bin Wang, Jie Lu, Zheng Yan, Huaishao Luo. Tianrui Li, Yu Zheng, and Guangquan Zhang. Deep uncertainty quantification: A machine learning approach for weather forecasting. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 2087-2095, 2019.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML, pp. 681-688, 2011.
Dongxia Wu, Liyao Gao, Matteo Chinazzi, Xinyue Xiong, Alessandro Vespignani, Yi-An Ma, and Rose Yu. Quantifying uncertainty in deep spatiotemporal forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1841-1851, 2021.
Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning. PMLR, 2021.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616. Citeseer, 2001.
Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pp. 35834-25866 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM 4.1
Theorem A.1 (Validity of CopulaCPTS), The confidence region provided by CopulaCPTS (algorithm 1 is valid. iee. P[Vj (I,...,k), k), yttj E 1 2
Proof. Define notations to be the same as in Section 4. Let D = {2 (x', be dataset with imput x' E Rexd, time series with length t, and output y¹ E -R a time series with length Each data sample (of entire time series, not time step) 2' = (x",y") is drawn i.idd from an unknown distribution Z. This means that any other sample drawn 2 exchangeable with D. from Dataset DD is divided into training set Drrain and two calibration sets ceul-1 and Deal-2.
We have nonconformity score function A with prediction model f trained on Dirain For each data point z = (x', y²) E Deint we calculate the nonconformity score for each time step j, concatenating them into vector s² of dimension k:
A(=); c 1ly)- f(x')(())))) j=
Let S1 {st (E.. be the set of nonconformity scores of data in Deat-1 and $2 = ( (eE)eeDDaa-- the set of nonconformity scores of data in Deat-2- Subseript will be used to index the set of specific time steps of the scores: S1, = ( S2, (i)
CDF Estimation on Deal-1. We use Deal-1 to build conformal predictive distributions (CPD) (Vovk al.] (017)) for each time step's 's anomaly scores. The cumulative distribution function constructed as:
(g)) |111+11
E
1 Lajee, for j {1....,k) k]
(10)
8'ES1,U(00)
Lemma A.2 (Validity of CPD. Theorem 11 Vovk et al. (017)) Given nonconformity score
function A 2 + R and a data sample ~ 2, calculate the nonconformity score as 8 = A(z).
Then, the distribution (-()) is valid in the sense that Pz (s)(s)) s 1 - a] = for any 0 < <
Copula Calibration on Deal-2- u {u JiEDea
Next, for every data point Deat we calculate
u' = (wi...... utt) (81), Pa(ok))
Each u' can be seen a multivariate nonconformity score for data sample z², We will now illustrate that an empirical copula on N is a rank statistic, and hence we can apply the proof conformal prediction prove finite sample validity guarantee.
Definition A.3 (Vector partial order). Define a partial order for k-dimensional vectors 3.
u A v
iff.. Vj (l......., uy SV,
(11)
i.e.
1 10,sk;
(12)
Next, define an empirical multivariate guantile function for a of k-dimensional vectors, based on the partial order defined in Eqn
Q a, arg min Σn; uy
Σ Insur) >
(13)
U
The empirical copula formula in CopulaCPTS (Fqn in section 4.1) is the same as the expression inside the inf function of Q(1 1f U (oo)). Therefore, finding si s 5% by Equation 8
implies:
Q U
There exist other definitions of multivariate quantiles, but they cannot be used in place of our definition in this proof. We have chosen this form because connects directly the empirical copula.
13

---

Published as a conference paper at ICLR 2024
The rest of the proof follows that of Inductive Conformal Prediction (ICP) in Vovk et al. (2005).
Given a test data sample 2nt1 = (a Viik V ~ 2, we want to prove that the confidence regions
r T|-a , Dl-a output by CopulaCPTS satisfies:
Plu, E 121-
Vje (1.....
We first calculate
u F)((t=))))) for {l....,kk
Let = -01-aaap a }). [0, 1]s.. An important observation for the conformal prediction proof is that if 11" 3 un+1 then
Q - a1-a, = au
the quantile remains unchanged. This fact be re-written
u n+ < Q - (0[1-auuuuaaae = n+1 3 QQ or, (u*+1})
The above describes the condition where u is among the [(1 an + 1)] smallest of U. By exchangability, the probability of u un+l's rank among U is uniform. Therefore.
[11 (1-4) + +1] (141) + +1)
Plu 3 u (oo)))
2 1
Hence we have
P(u QQ11 a,u 21-a
(14)
Note again that:
uu QQ - a;(1-a, ((()))).
u=+1 = (f ((.,),,(....)))) ()(+t1))
The uncertain regions are constructed as (Algorithm 1, line 17):
r + (y: Ily - 85-11 < $5}
(15)
By definition of 3, we have
u n+1 = u
(16)
Vj E (1....,
k}, u 1, S Sn
(17)
Lemma A
Vj e (1....,
k}, 59 Ss
(18)
B
Aj (1...,,,,
y; E
(19)
Combining Eqn 14 and Eqn 19
we have
P k). y,
- -
(20)
14

---

Published as a conference paper at ICLR 2024
B ADDITIONAL ALGORITHM DETAILS
B.1 UPPER AND LOWER BOUNDS FOR COPULAS
To provide a better understanding of the properties of Copulas, consider the Frechet- Frechet-Hoffdding Bounds (Theorem B.Ip. In fact, the Frechet-Hoeffding upper- and lower- bounds are both copulas. The lower bound is precisely the Bonferroni correction used in Stankevičtüte et al. (202T) therefore by estimating the copula more precisely instead of using lower bound, we have a guaranteed efficiency improvement for the confidence region.
Theorem B.1 (The Trechet-Hoeffding Bounds). Consider a copula C(ui...., uk). Then
max (1 - k + U4, 0} S C(ui,.... 4+) S min (ui,.... ,
B.2 NUMERICAL OPTIMIZATION WITH SGD FOR SEARCH
We continue use the notation defined in Appendix A The inverse of the predictive distributions (Equation TOp can be written as follows, similar to the empirical quantile function (Equation 3
= inf (=)
E
Lofess) p}
(21)
S1
S'ES1,U(o))
We find the optimal 8) in Equation 8 and Algorithm | by minimizing the following loss:
k
L(s
[ (1 -x) aa) Deett iEDeal-2J=1
Σ
The indicator function implemented using a sigmoid function whose input is multiplied by constant for differentiability. A small amount of L2 regularization is added to the loss function to ensure the searched scores are as low as possible. We use the Adam optimizer and perform gradient descent for 500 steps to get the final result. The optimization process to find s* typically takes a few seconds on CPU. For each run of our experiments, the calibration and prediction steps of CopulaCPTS combined took less than I minute to run Apple MI CPU. Please refer to the CP class in the reference code for implementation details.
B.3 COPULACPTS IN RETOOREGRESSIVE FORECASTING
Auto-regressive forecasting a common framework in time series forecasting. So far, we have been looking at forecasts for predetermined number of time steps k. One can use fixed-length model to forecast variable horizons k' autoregressively. taking the model output as part of the input. In the conformal prediction setting, we want not only to autoregressively use the point forecasts, but also to propagate the uncertainty measurement.
If the time series and uncertainty are stationary (for example additive Gaussian noise), the copula remains the same for any sliding window steps, i.e. C(u), u&) C(u2. ( There- fore, after finding (ui,... ug)) such that C(ui...., ut)) > I a, can simply search for w such that C(u,, ...,,.... Wi: ")+1) 2 a. The guarantee proven in Theorem 4. 4till new estimate. this way, we can achieve the coverage guarantee over the entire autoregressive forecasting horizon.
still holds for the
On the other hand, the time series non-stationary, we need to fit copulas C (u ux). C2 1)+1), ... Cr-AA (w .,. uer), one for each autoregressive prediction, which re- quires a calibration set with 2 k time steps. The k step autoregressive problem then reduced to R' - k multi-step forecasting problems that can be solved by CopulaCPTS. It follows that each of the autoregressive predictions valid. Appendix B.4 provides an example scenario where re-estimating the copula is necessary for validity.
15

---

Published as a conference paper at ICLR 2024
B.4 AUTOREGRESSIVE PREDICTION
In the context of this paper to forecast autoregressively is given input Xist and & step forecasting model f. perform prediction
841+++ f(x1:))
8442044+1 = fxx.111
until all k time steps predicted.
We now provide a toy scenario to illustrate when re-estimating the copula is necessary and improves validity. Consider time series of three time steps for 16.11:22. 12- The two scenarios in Figure 6. In both scenarios, the mean and variance of all time steps are 0 and 1 respectively. In scenario (a), fo i and hence their covariance is The copula estimated on to and is F(tu) F(11). This copula will significantly underestimate the confidence region of 22 where its covariance with 1 is 11. In fact the coverage of Co.1 (Fi (((t)) 0.74. On the other hand, (b) illustrates scenario where the copula for any 2 consecutive time series remains same Co- Ci. In this case, applying Co directly to forecast C₁ achieves precisely 90% coverage.
illustrated
Co.1 (Fto), (tt))
84
110
24
(a) Time steps 0 and positively corre- lated while and negatively correlated
(b) Stationary time series where each time steps are uncorrelated
Figure 6: Two scenarios to illustrate the autoregressive case
C EXPERIMENT DETAILS AND ADDITIONAL RESULTS
C.I UNDERLYING FORECASTING MODELS
Particle Dataset. The underlying forecasting model for the particle experiments is an 1-layer L.STM network with embedding size 24. The hidden state then passed through linear network to forecast the timesteps concurrently (output has dimension & dyy. We train the model for 150 epochs with batch size 128. Hyperparameters of the network are selected through a model search by performance 5-fold cross-validation split of the dataset. The architecture and hyperparameters are shared for baselines and CopulaCPTS in Table
Drone. For the drone trajectory forecasting task, we use the same LSTM forecasting network as the particle dataset, but with its hidden size increased to 128. We train the model for 500 epochs with batch size 128. The same architecture and hyperparameters are shared for all baselines and CopulaCPTS reported in Table 1.
Covid-19. The COVID-19 dataset is downloaded directly from the official UK government website https: //tp://v/oooviiooo// data gov. d 1 /dwwn load by selecting region for area
16

---

Published as a conference paper at ICLR 2024
type and newCarreBByPuliishDiee for metric. There are in total 380 regions and over 500 days of data, depending on when it is downloaded. We selected 150-day time series from the collection to construct our dataset.
The base forecasting model for Covid- 19 dataset is the same as the model for synthetic datasets, with hidden size = 128, and were trained for 150 epochs with batch size 128. The same architecture and hyperparameters are shared for all baselines and CopulaCPTS reported in Table 1.
Argoverse. As highlighted in the main text, we utilize state-of-the-ate prediction algo- rithm LaneGCN (Liang et al., 2020) as the underlying forecaster model for CF-RNN and Copula-RNN.
refer readers to their paper and code base model details. The ar- chitecture of the RNN network used for MC-Dropout and BJRNN is an Encoder- net- work. Both the encode and decoder contain a LSTM layer with encoding size 8 and hid- den size 16. We chose this architecture because the is part of the official Argoverse base- lines (https: : /tt:t//gtbttt com/ jagjeet-singteee aeriinggrrgoveeeefrrrreeee strates competitive performance.
ing) and demon-
C.2 CALIBRATION AND EFFICIENCY CHART FOR COVID-19
Figure shows comparison of calibration and efficiency for the daily new COVID 19 cases forecasting.
Coverage (50 days)
Width (daily average)
0.8
1250
1000
0.6
750
500
0.2
250
0.0
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
1
1
1-E
mcdp
cfrnn
bjrnn
copula-mn
Figure 7: Calibration and efficiency comparison on different level for COVID-19 Daily Forecasts. The copula methods (orange and red lines) are more calibrated (coinciding with the green dotted line) and sharp (low width) compared baselines.
To see if the daily fluctuation due to testing behavior disrupts other methods, also ran the same experiment on weekly aggregated new cases forecast. We take 14 weeks data as input and output forecasts for the next weeks. The results are illustrated in Figure 8. The weekly forecasting scenario gives us similar insights as the daily forecasts.
C.3 ARGOVERSE
The Argoverse autonomous vehicle dataset contains 205,94 25,9922 samples, consisting of diverse driving scenarios from Miami and Pittsburgh. The data can be downloaded from the official Argoverse dataset website. We split 90/10 into training set and validation set of size 185,348 and 20,594 respectively. The official validation set size 39,472 is used for testing and reporting performance. We preprocess the scenes to filter out incomplete trajectories and cap the number of vehicles modeled to 60. If there are less than 60 cars in the scenario, we insert dummy cars into them to achieve consistent car numbers. For map information, only include center lanes with lane directions as features. Similar to vehicles, we introduce dummy lane nodes into each scene to make lane numbers consistently equal to 650.
17

---

Published as a conference paper at ICLR 2024
Coverage (over 6 weeks)
Width (weekly average)
1.0
3000
2000
1500
1000
500
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
mcdp
dmnn
bjmn
copula-mn
copula-EncDec
Figure 8: Covid Weekly Forecasts
C.4 ADDITIONAL EXPERIMENT RESULTS
We present in Figures 8 and 9 some qualitative results for uncertainty estimation.
To test how the effects of copulaCPTS compare with baseline other base forecasters, also include an encoder-decoder architecture with the same embedding size as the RNN models introduced in Appendix C.I for each dataset. The results are presented in Table We omit these results in the main text because we found that they did not bring significant improvement to time series forecasting UQ
Table 4 compares model performance compared across different prediction horizons. We show that the advantage of our method is more pronounced for longer horizon forecasts.
41
(a) Copela-EncDDec
(b) MC Dropout
(c) CF-RNN
Figure 9: 99% Confidence region produced by three methods for the drone dataset. Copula methods (a) produce consistent, expanding cone of uncertainty compared to MC-Dropout (b) sharper one compared to CF-RNN (c).
C.5 STUDY ON SEARCH
Figure shows the ay values for each a, 15 of ATgorithm We present aj values searched using two methods of searching, with dichotomy search for constant value for the horizon as Messoudi et al. (202T). and by stochastic gradient descent outlined in section 4.2.
F used in Copula CPTS as outlined in line
The aj values are an indicator how interrelated the uncertainty between each time step are: Bonferroni Correction used in Staanevvičitt et al. (2021) (grey dotted line in Figure I1) assumes that the time steps are independent, with CopulaCPT'S we have lower a, levels while having valid coverage (blue and orange lines in Figure IT.. This shows that the uncertainty of the time steps is not independent, and we are able to utilize this dependency to shrink the confidence region while still maintaining the coverage guarantee.
18

---

Published as a conference paper at ICLR 2024
Particle Simulation (a .01)
Coverage (90%)
Area (90%)
Coverage (99%)
Area (99%
MC-dropout
691.5 20
2.22 0.06
95.2 + 1.4
3.16 1.08
BJRNN
98.9
2.24 0.59
99.6 0.3
2.75 1.71
CF-RNN
97.1 + 0.8
1.2 + 0.21
99.3 106
3.16 + 0.86
CF-EncDec
97.3
1.97 =
98.9 0.6
2.75 0.42
Copula-vanilla
86.9 + 1.9
0.63 = 0.07
91.9 + 1.8
0.76 0.12
Copula-RNN
91.3 1
1.08 1 0.14
99.4 1.0
2.23
1 0.19
Depula-EncDec
90.8 2.5
1.19 1 0.08
99.3 0.5
2.16
1 0.23
Particle Simulation (a .05)
Coverage (90%)
Area (90%)
Coverage (99%)
Area (99%)
MC-dropout
16.1 4.3
0.79
+ 0.02
33.9 5.1
2.12
+: 0.03
BJRNN
100.0 +.00
12.13
0.39
100.0 0.0
15.43 0.85
CF-RNN
94.5 + 1.5
5.79
0.51
99.8 2.2
19.21 1 8.19
Copula-vanilla
88.5
4.37
= 0.16
91.7
4.8 = 0.18
Copula-RNN
90.3 + 0.7
4.50
+ 0.07
99.1 0.8
12.82 + 3.98
Copula-EncDec
91.4
4.40
= 0.15
98.7 0.1
9.31 + 1.97
Drone Simulation .02)
Coverage (90%)
Area (90%)
Coverage (99%)
Area (99%)
MC-dropout
84.5 + 10.8
9.64
= 2.13
90.0 778
16.02
+ 3.62
BJRNN
90.8 22
49.57
100.0 4.0
65.77
4.56
CF-RNN
91.6 192
32.18
+ 13.66
100.0
= 0.0
36.79
+ 14.03
CF-EncDec
100.0 0.0
21.83
+:2 26.29
100.0
25.03
+
Copula-vanilla
89.5 + 1.3
54.67
= 28.9
94.5 0.5
68.9
+ 33.42
Copula-RNN
90.0
16.52
: 15.08
98.5 +
21.48 8.91
COVID-19 Daily Cases Dataset
Coverage (90%)
Area (90%)
Coverage (99%)
Area (99%)
MC-dropout
19.1 =5,1
34.14 = 0.84
100,0 0.0
1106.57
+ 25.41
BJRNN
79.2 =
823.3 + 529.7
85.7 + 27.5
149187.
51044.
CF-RNN
95,4 + 1.9
610.2 96.0
100.0 = 0.0
121435.
+: 26495.
CF-EncDec
91.7 114
570.3 = 22.1
100.0 = 0.0
108130.
10889.
Copula-vanilla
90.8 144
414.42
5.08
91.2 1.3
41346.
59.0
Copula-RNN
92.1 + 1.0
429.0
+ 15.1
100.0 = 0.0
88962.
+ 9643.
Copula-EncDeec
90.8 1.0.
429.4
27.9
100.0 - 0.0
60852. 12263.
Argoverse Trajectory Prediction Dataset
Coverage (90%)
Area (90%)
Coverage (99%)
Area (99%)
MC-dropout
27.9 13311
127.6 1 20.9
31.5 1.39
242.1 54.0
BJRNN
92.6 992
880.8 + 156.2
100.0 0.0
3402.8 268.
CF-LaneGGN
98.8 + 1.9
396.9 + 18.67
100. 0.2
607.2
+ 8.67
Copula-vanilla
89.7
107.2
96.5 2.3
289.0
38.1
Copula-LaneGCNN
90.4 0.3
126.8 + 12.22
99.1 0.44
324.1 + 42.22
Table 3: Additional results. Copula methods achieve high level of calibration while producing sharper prediction regions. The sharpness gain is even more pronounced at higher confidence levels (99%), where the prediction region to be useful while remaining valid.
19

---

Published as a conference paper at ICLR 2024
Coja ACPF
Figure 10: Illustrations for confidence regions given by CF-RNN (blue) and CopulaCPTS (orange) at time steps 0, 10, 20, and 30. Note that in order achieve 90% coverage, the regions are larger than needed, especially in straight-lane cases like the middle two. Using copulas to couple together time steps results in much smaller region while achieving similarly good coverage.
Step
Steps
15 Steps
Method
Coverage
Area
Coverage
Area
Coverage
Area
MC-Dropout
97.8 2.0
0,4 + 0.04
88.0
7.0
0.69
+ 0.25
52.3 = 1.4
0.94
0.2
BJRNN
45.3 i 39.4
0.27
+: 0.18
97.7
2.1
2.69
i: 1.79
95.5 28
19.99
1 4883
CF-RNN
100.0 1.00
0.01
1 0.01
77.8
19.2
0.8
1 0.64
66.7 1.00
18.82
3.73
CF-EncDec
89.9 = 19.2
0.01
0.01
100.0
0.0
0.75
+ 0.99
88.9 = 19.2
13.07
= 16.1
Copula-RNN
90.1 0.2
0.01 + 0.01
89.8 0.6
0.54
0.45
90.1 = 1.2
8.25
# 3.44
Copula-Enche Enc
90.0 0.3 0.01 1: 0.0
90.3
1 0.6
0.67 +: 1.01
90.5 + 0.5
7.13 1 9.5
Table 4: Performance comparison across different horizons 90% confidence level the drone simulation dataset. The improvement on efficiency is more pronounced when the horizon is longer.
Table that there are no significant differences between coverage and area performance for the two search methods within the scope datasets we study in this paper. However, we want to highlight that SGD search is (n) complexity to optimization steps, regardless of the prediction horizon. SGD also allows for varying a, which might be useful in some settings, for example capturing uncertainty in the COVID-19 dataset of Figure II Dichotomy search, spikes for some time steps complexity to the search space depends on granularity, and will be the other hand, is (n)og(n))
O (knloggkn) if we want search for varying azj-
Dataset
Coverage (90%)
Area
Fixed aj
Varying aj
Fixed ag
Varying ay
Particle (a .01)
91.7 + 1.9
91.5 2.1
1.13
+ 0.45
1.06
+ 0.36
Particle (a = .05)
92.1 +: 1.3
90.3 + 0.7
4.89
+ 0.05
4.50
+ 0.07
Drone
90.3 0.5
90.0 + 1.5
15.92
+ 1.98
16.52
+ 7.08
Covid-19
92.9 1.0.1
92.1
+ 1.0
498.44
6.36
429.0
+ 15.1
Argoverse
90.2
90.4
= 0.3
117.1
7.3
126.8
+
Table 5: Coverage and area comparison between stochastic search for fixed aj and SGD for Varying Ojj We do not see significant difference between the performance of the two.
C.6 COMPARISON TO ADDITIONAL BASELINES
We include a comparison to two additional simple UQ baselines on the particle simulation dataset.
L2-Conformal. L2- 12-Conformal uses the same underlying RNN forecaster CF-RNN and Copula RNN. We use the nonconformity score of the vector norm of all timesteps concatenated together l 1 Ne+1atel to perform ICP. As there are analytic way represent A x dy dimennssll
20

---

Published as a conference paper at ICLR 2024
Partides
Particlos
Simulation
53.4
E:00
6136
$3.00
117%
8570
0373
Prodicton Horzon
coon Honzon
Predition Herpon
Argeverse
0019
11823
Monferoni Comection
Prode3ion Honzzn
PreSictoo MORZOR
Verying
Figure 11: Comparison between dichotomy search for fixed oj values (blue) and stochastic gradient search for varying ay (blue) through timesteps. Shaded regions are the standard deviation of the values over runs.
uncertainty region on 2-D space, calculate the area and plot the region for L2 Conformal bascline with the maximum deviation at each timestep such that the vector norm still stays within range.
Direct Gaussian. Direct Gaussian uses the same model architecture and training hyperparameters. with the addition of a linear layer that outputs the variance for each timestep, and is optimized using negative log loss, proper scoring rule for probabilistic forecasting. We obtain the area by analytically calculating the 90% confidence interval for each variable.
Results in Table 6 show that L2-conformal produces inefficient confidence area, and directly out- putting variance under-covers test data. These results align with previous findings and motivate our method, which both more calibrated and sharper compared these baselines. We show visualization in Figure 12 to illustrate the different properties of the methods qualitatively.
Particle (a .01)
Particle (a .05)
Method
Coverage (90%)
Area
Coverage (90%)
Area 4
L2-Conformal
88.5 0.4
7.21 +: 0.35
89.7 + 0.6
7.21 0.35
Direct Gaussian
11.9 = 0.09
0.07 = 0.31
0.0 0.0
0.08 = 0.02
CF-RNN
97.0 2.3
3.13 = 3.24
97.0
5.79 = 0.51
CopulaCPTS
91.3 + 2.1
1.08 + 0.36
90.3 + 0.7
4.50 + 0.07
Table 6: Comparison with two additional baselines on the particle dataset.
Ellipsoidal conformal inference for Multi- Multi-Targel Target Regression We also compare CopulaCPTS to newer work, Ellipsoidal CP (Messoudi et al. 2022). The result is presented in Table 7 This method models the uncertainty region of multi-target outputs as high-dimensional ellipsoid, by estimating covariance matrix calibration data. We apply EllipsoidalCP the time and space dimensions, so the particle simulation, for example, is treated as a multi-target prediction of dim 50 = 25 (time steps) x 2 (dims). We that the results comparable in experiment. When the correlation is more pronounced such as in the covid experiment, EllipsoidalCP capture the correlation better than CopulaCPTS resulting in improved efficiency. On the other hand, the flexibility of our method allows us to achieve better efficiency than that of EllipsoidalCP. A notable concern for using EllipsoidalCP is that for higher output dimensions, the determinant of the covariance matrix can be extremely large (up to 1050 in our experiments) and can result in numerical instabilities.
data by flattening
21

---

Published as a conference paper at ICLR 2024
05
62-
-1
(a) 1.2 Conformal
(h) Direct RNN Gaussian
04.
02
81
94
22
-18
-0
+0
-04
-2
+2
(c) CF-RNN
(d) Copula-RNN
Figure 12: Visualization of on a sample from the Particle dataset's test set.
Table 7: Performance comparison with EllipsoidalCP in synthetic and real-world datasets with target confidence 1 a = 0.9.
EllipsoidalCP
CopulaCPTS
Particle Sim (a .01)
COV
90.1
= 0.9
91.3
1.5
area
0.84
1.08 =
Particle Sim (a = .05)
Cov
90.8 0.4
90.6
area
8.76 0.41
5.27
= 1.02
Drone Sim (o = .02)
COv
90.5 0.2
90.0
area
28.3 = 3.1
17.12
+ 6.93
COVID-19
Cov
93.3
- 1.5
90.5 1.6
Daily Cases
area
231.5
22.4
408.6
65.8
Argoverse Trajectory
90.3
0.1
90.2
area
144.8 4 8.1
126.8
4 12.2
22


Reviews:
[{"review_id": "hizE82GAHu", "reviewer": "Reviewer_3Gfg", "summary": "The authors have proposed an extension of classical conformal prediction for multi-step, multivariate time series forecasting. Their work builds upon the foundations laid by Messoudi et al. without implying a significant improvement.", "questions": "Q1: What distinguishes the proposed method from Messoudi's work, apart from the introduction of multivariate entries?\n\nQ2: In comparison to another work[1] focusing on the multi-output version of conformal prediction that uses quantiles and can be extended to the multivariate case with the same score function, what distinguishes and contributes to this paper? I also recommend adding [1] as another baseline in the experimental part to prove the efficiency improvement.\n\nQ3: In Equation 8, when aiming to enhance efficiency by minimizing the L1 norm of u, what is the rationale for choosing this particular norm?\n\nQ4: Is there any typo in the loss function in the B.2 part? The loss function appears to be incorrect. Additionally, in the appendix, both equation (16) and equation (19) contain typos. The proof details regarding the validity (appendix A) are crucial; it would be beneficial to double-check this part.\n\n[1] Feldman, S., Bates, S., & Romano, Y. (2023). Calibrated multiple-output quantile regression with representation learning. Journal of Machine Learning Research, 24(24), 1-48.\n\nFollowing a thorough review of the authors' response and considering the feedback from other reviewers, I have decided to adjust the score to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The authors employ copulas to capture the relationships between different time steps in their time series analysis.\n\n(2) Based on the experimental results, their method demonstrates a narrower predicted region compared to other approaches, suggesting enhanced efficiency.", "weaknesses": "(1) The primary weakness of this paper lies in its novelty. The use of copulas to capture uncertainty for multi-output scenarios is not a new concept. Notably, Messoudi et al. have previously proposed a similar approach, even employing the same empirical copula. While the authors assert that their contribution is the multivariate nature of every entry, it's important to note that the nonconformity score remains scalar. Therefore, the process of conformal prediction remains largely the same as in the univariate case. \n\n(2) Another notable issue pertains to data type. While this paper primarily focuses on time series, upon closer examination, it becomes apparent that the temporal aspect of the data may not be as critical as initially assumed. The dataset comprises numerous time series, and each data point within these series appears to be exchangeable with one another. This exchangeability is what enables the application of conformal prediction. Furthermore, it's crucial to differentiate this work from other studies that delve into conformal prediction methods that extend beyond exchangeability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxXg3jqBzG", "reviewer": "Reviewer_XEiW", "summary": "This paper introduces a method called CopulaCPTS which allows to perform conformal prediction to any multivariate multi-step forecaster, with statistical guarantees of validity. Based on the notion of copula to model the dependency between forecasted time steps, the authors prove that CopulaCPTS has finite sample validity guarantee. On both synthetic and real multivariate time series, they show that CopulaCPTS produces more efficient confidence intervals than existing techniques.", "questions": "-- The paper uses a particular definition for the multivariate empirical quantile function (Eq. 13). Is this the only possible definition? If not, why use this one and not another?\n\n-- In Lemma A.2, how is it possible to have an equality? (Do scores not need to be continuous?).\n\n-- Isn't the definition of \"exchangeability\" in the paper rather a consequence of the \"true\" definition?\n\n-- Is the \"$\\forall j$\" in the probabilty or outside ? In the proof, Eq. (20), this is inside but in Theorem 4.1, this is outside.\n\n-- In the experiments, the score is chosen to be an L2 norm (see, for example, step 9 of Algorithm 1). What are the implications of this choice on the results? For example, are the results very different if we use another norm?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- The analysis of time series is a very interesting problem in conformal prediction.\n\n-- The experience shows that the method performs quite well.\n\n-- The paper is clear and easy to follow.", "weaknesses": "-- It seems that the algorithm needs a large amount of data as the calibration dataset is split a second time.\n\n-- In the experiment, computational times are not given.\n\n-- The standard deviations of the experiments are made with only 3 runs of the algorithm. Maybe the algorithm is very time-consuming (this should be more discuss).\n\n\nMinor:\n\n-- 'the the green'\n\n-- Lemma ?? (eq 18)\n\n-- In the right-hand side of equation 12, there should be a $v$ and not a $u^*$ (?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbvGtkBIlt", "reviewer": "Reviewer_bjQ8", "summary": "The paper studies a special setting of multi-step time series forecasting with the focus on predicting confidence intervals when a traning dataset of similar time series is available.  The authors combine conformal prediction with copula modelling in a two-step algorithm that uses a part of the training dataset for calibration of confidence intervals and can be based on any other time series forecasting model. They prove the validity of the introduced algorithm and evaluate its performance on two synthetic and two real-world datasets.", "questions": "* Equation (4) presents validity in terms of marginal distributions, while the prior work, e.g. Stankeviciute et al. (2021) uses a joint distribution. The proof of your theorem also uses the definition based on the joint distribution. Which one is correct? \n* In experiments, did you use the same training dataset for all methods or did you adapt it depending on whether the method need a calibration set? I assume you used the same one for all baselines, but it seems fair to give methods that don't rely on calibration more data.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality \nThe paper presents a novel application of copula based conformal prediction explored in multi-target regression to time series setting\n## Quality \n* The presented method is both theoretically grounded and practically applicable\n* Authors discuss the limitations of the algorithm\n* Authors showcase the performance of their algorithm from different angles including exemplar visualizations\n## Clarity \nThe paper is well-written and the presentation is very clear\n## Significance\nThe paper is a continuation of an existing line of work on copula based conformal prediction into time series domain. Due to method limitations, in particular the requirements for vast calibration set size for non-parametric modelling, I would expect the paper to be of limited significance.", "weaknesses": "The paper doesn't have major weaknesses. The main downside of the presented method is its relience on large calibration dataset, which is a luxury in practical problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8j2TMRd2g", "reviewer": "Reviewer_ruzn", "summary": "This work considers conformal prediction for multi-step time-series prediction using copula, demonstrating improved performance than existing baselines.", "questions": "No additional questions are raised.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The idea of multi-step time-series forecasting using copula CP is novel.\n* Strong empirical performance on various tasks.", "weaknesses": "1. Theoretical analyses: the i.i.d. assumption is imposed on $z^i=(x^i_{1:t}, y^i_{1:t})$ for $i=1,...,n$. However, this eliminates the dependency of data over time, which is typical and expected for time-series. Hence, this seems very restrictive for the purpose of theoretical analyses, and it is important to discuss what this assumption actually imposes on the data collection process.\n\n2. The simulation performance of CopulaCPTS seems very similar to Copula [Messoudi et al. 2021], which notably was not developed for this setting. I think this goes back to the assumption on $z^i$ being i.i.d., where the use of CopulaCPTS is not essential under the absence of temporal dependency.\n\n3. Performance of Copula on real-data examples is not reported.\n\n4. Given that Copula does not seem to under-perform much, I think additional comparison is needed. For example, the locally ellipsoid CP in [Ellipsoidal conformal inference for Multi-Target Regression](https://copa-conference.com/papers/COPA2022_paper_7.pdf) by [Messoudi et al. 2022]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)