Title: Conformal Prediction for Uncertainty-Aware Planning with Diffusion Dynamics Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uncertainty

Conformal Prediction

Dynamics Model
Abstract: Robotic applications often involve working in environments that are uncertain, dynamic, and partially observable. Recently, diffusion models have been proposed for learning trajectory prediction models trained from expert demonstrations, which can be used for planning in robot tasks. Such models have demonstrated a strong ability to overcome challenges such as multi-modal action distributions, high-dimensional output spaces, and training instability. It is crucial to quantify the uncertainty of these dynamics models when using them for planning. In this paper, we quantify the uncertainty of diffusion dynamics models using Conformal Prediction (CP). Given a finite number of exchangeable expert trajectory examples (called the “calibration set”), we use CP to obtain a set in the trajectory space (called the “coverage region”) that is guaranteed to contain the output of the diffusion model with a user-defined probability (called the “coverage level”). In PlanCP, inspired by concepts from conformal prediction, we modify the loss function for training the diffusion model to include a quantile term to encourage more robust performance across the variety of training examples. At test time, we then calibrate PlanCP with a conformal prediction process to obtain coverage sets for the trajectory prediction with guaranteed coverage level. We evaluate our algorithm on various planning tasks and model-based offline reinforcement learning tasks and show that it reduces the uncertainty of the learned trajectory prediction model. As a by-product, our algorithm PlanCP outperforms prior algorithms on existing offline RL benchmarks and challenging continuous planning tasks. Our method can be combined with most model-based planning approaches to produce uncertainty estimates of the closed-loop system.

Conformal Prediction for Uncertainty-Aware Planning with Diffusion Dynamics Model

Abstract

Robotic applications often involve working in environments that are uncertain, dynamic, and partially observable. Recently, diffusion models have been proposed for learning trajectory prediction models trained from expert demonstrations, which can be used for planning in robot tasks. Such models have demonstrated a strong ability to overcome challenges such as multi-modal action distributions, highdimensional output spaces, and training instability. It is crucial to quantify the uncertainty of these dynamics models when using them for planning. In this paper, we quantify the uncertainty of diffusion dynamics models using Conformal Prediction (CP). Given a finite number of exchangeable expert trajectory examples (called the 'calibration set'), we use CP to obtain a set in the trajectory space (called the 'coverage region') that is guaranteed to contain the output of the diffusion model with a user-defined probability (called the 'coverage level'). In PlanCP, inspired by concepts from conformal prediction, we modify the loss function for training the diffusion model to include a quantile term to encourage more robust performance across the variety of training examples. At test time, we then calibrate PlanCP with a conformal prediction process to obtain coverage sets for the trajectory prediction with guaranteed coverage level. We evaluate our algorithm on various planning tasks and model-based offline reinforcement learning tasks and show that it reduces the uncertainty of the learned trajectory prediction model. As a by-product, our algorithm PlanCP outperforms prior algorithms on existing offline RL benchmarks and challenging continuous planning tasks. Our method can be combined with most model-based planning approaches to produce uncertainty estimates of the closed-loop system.

1 Introduction

Uncertainty is a major challenge in robotics, arising from a variety of sources, including errors in modeling, sensing, planning, and control [1, 2, 3]. Understanding the effect of uncertainty is important for developing successful motion strategies, as it allows robots to anticipate and plan for unexpected events. Therefore, planning with uncertainty is a crucial challenge in robotics and accurately modeling this uncertainty is important for ensuring the safety and reliability of robotic systems, particularly in safety-critical applications. A range of techniques, such as probabilistic modeling [4] and robust optimization [5, 6], can be used to account for the uncertainty of dynamics models in robotics.

The dynamics model is especially critical for model-based planning [7] and reinforcement learning (MBRL) [8]. Due to the advantages such as expressing multi-modal action distributions, highdimensional output space, and stable training, there has been a growing trend in learning-based planning approaches to utilizing diffusion models as dynamics models [9, 10]. Diffusion models can also address the challenge of multi-modal distribution of actions in planning, which arises when multiple actions may be appropriate for a given state. This is achieved by enabling policies to express a distribution over multiple actions rather than a deterministic choice of a single action [11]. Furthermore, diffusion models hold the potential to be applied to Offline Reinforcement Learning, which aims to learn effective strategies using offline datasets without exploration [12]. However, existing model-based algorithms struggle when encountering out-of-distribution (OOD) actions or states [9]. A crucial element absent from these methods is the adequate management of uncertainty in the offline scenario [13, 14]. Uncertainty-aware dynamics models that convey algorithmic confidence are valuable if they possess accurate calibration and sharpness [15]. Calibration entails aligning predicted probabilities with actual outcomes, whereas sharpness refers to the concentration of predicted probabilities around the true probability [16]. Nevertheless, many machine learning algorithms generate poorly calibrated probability estimates, which can result in overconfident or underconfident predictions and lead to suboptimal decision-making [17, 18]. Therefore, to ensure accuracy and optimal decision-making, it is crucial to utilize uncertainty-aware models that are well-calibrated and sharp.

Conformal Prediction (CP) [19] is a potent approach to assessing the uncertainty associated with predictions generated by black-box models. CP offers several advantages over traditional methods for measuring uncertainty. For example, CP is model-agnostic [20], meaning it can be applied to any type of model without requiring knowledge of the model's inner mechanisms. This is particularly advantageous in cases where the underlying model is complex and not well understood. Additionally, CP allows for customization of the desired confidence level, providing the ability to control the trade-off between predictive accuracy and the level of uncertainty [21]. This feature is especially important in safety-critical applications where a higher level of confidence is required. Furthermore, CP can be used for online learning [22], meaning that it can adapt to new data points as they become available. This makes it a particularly useful approach for planning in dynamic environments. Lastly, CP is distribution-free [23], providing statistical guarantees [24] and reliable uncertainty estimates from finite samples [25]. Given these advantages, we choose to use CP as our method of uncertainty quantification for diffusion dynamics models in planning. The flexibility and reliability of CP provide a rigorous and mathematically proven framework for uncertainty quantification, which is critical for ensuring the safety and reliability of robotic systems in a variety of applications.

In this paper, we connect conformal prediction to planning and propose PlanCP , adopting a practical and effective conformal prediction uncertainty estimation method for planning algorithms. Empirically, we observe that PlanCP substantially improves model stability. In addition, PlanCP reduces uncertainty without sacrificing performance on datasets with demonstrations collected from experts. PlanCP can be applied to reinforcement learning or imitation learning from demonstrations [26, 27] as well. However, since we build on the Diffuser [9], which calls itself a planning method, we will generally stick with the planning terminology for consistency with the literature.

In summary, our main contributions are as follows:

Weintroduce PlanCP , a framework that enables the use of a conformal prediction to estimate uncertainty in planning tasks, without sacrificing the accuracy of dynamics models.

Our approach is compatible with diffusion dynamics models. We demonstrate that by employing a diffusion model as the foundational dynamics model, we can construct a conformal predictor that guarantees trustworthy and accurate uncertainty estimates.

We extensively evaluate our approach, and the experimental results demonstrate that it exhibits lower uncertainty without compromising performance across multiple benchmarks, including Offline Reinforcement Learning.

2 Related Work

2.1 Conformal Prediction

Conformal Prediction (CP) [19, 28, 29, 30, 21, 24, 31, 32, 33] is effective for measuring the uncertainty associated with predictions produced by black-box models. As a variant of conformal

prediction, Transductive Conformal Predictor (TCP) can be applied on top of existing machine learning methods to establish valid prediction regions, [19]. TCP takes into account the structure of the data being predicted and uses the entire dataset during the prediction phase to construct prediction regions around each data point [34]. To improve the computational efficiency of the initial transductive approach, a more computationally efficient framework Inductive Conformal Predictor (ICP) was proposed, which involves dividing the training data into a proper training set and a calibration set [35]. However, ICP can be informationally inefficient as some examples are used solely for modeling while others are used only for calibration. To address this, ensembles of conformal predictors have been proposed [36, 37] and [38]. Examples include Cross Conformal Predictor (CCP) [39], Bootstrap Conformal Predictor (BCP) [19] and the generalized Aggregated Conformal Predictor (ACP) [40]. In a recent study [41], conformal prediction was used to construct prediction regions for recurrent neural networks (RNNs). Strawn et al. [42] propose conformal predictive safety filters that use statistical techniques to provide uncertainty intervals around predictions and learn an additional safety filter that closely follows the RL controller but avoids uncertainty intervals. Tonkens et al. [43] expand upon prior work in planning with probabilistic safety guarantees using conformal prediction. Conceptually closest to our work are [44, 45, 42, 43]. However, in these works, only quantification of uncertainty is provided, and reduction of the uncertainty during training is absent. In comparison, we are able to reduce uncertainty in training and extend conformal prediction to diffusion models.

2.2 Diffusion Models

In recent years, diffusion models have gained significant attention as a powerful class of generative models that formulates the data-generating process as a sequence of denoising steps [46, 47, 48, 11]. Inspired by score matching [49] and energy-based models (EBMs) [50, 51, 52, 53], diffusion models rely on a denoising procedure, which can be viewed as a way of parameterizing the gradients of the data distribution [54]. From the model-based trajectory-planning perspective, Diffuser [9] applies a diffusion model as a trajectory generator. From the model-free policy-optimization perspective, Wang et al. [11] utilize a diffusion model to represent the policy, applying it to the action space and conditioning it on the states to form a conditional diffusion model. Diffusion Policy [10] utilizes a conditional diffusion-based process to model policies for real robots, exhibiting robust handling of multi-modal action distributions. However, the aforementioned approaches lack the capability to quantify the associated uncertainty, which is precisely the focus of our paper.

2.3 Planning, Predictive Dynamics Model, and Model-Based Reinforcement Learning

Model-based reinforcement learning (MBRL) has emerged as a promising solution for real-world sequential decision-making problems with distinct advantages such as analytic gradient computation [55]. A significant body of literature focuses on learning a dynamics model and utilizing the learned model for policy learning through model-based planning [56, 57, 58, 59]. PILCO [60] utilizes a Gaussian process to model system dynamics, while World Models [61] and Dreamer [62] learn latent dynamics through two-stage and online planning, respectively. MGAIL [63] introduces a forward model in GAIL, but its discriminators output a uniform representation that is less transferable. PETS [64] combines probabilistic neural networks and deterministic neural networks to create uncertainty-aware models. To capture long-term dependencies, FNNs, RNNs, and Transformers have been used to develop internal models [65, 66, 67]. Our study distinguishes itself from prior research by investigating techniques to measure and minimize uncertainty using conformal prediction for diffusion-based dynamics models.

3 Method

In this section, we describe our framework PlanCP (as Fig. 1 shows) to solve the problem of uncertainty-aware planning. We begin by outlining our overall methodology, which involves three key steps. First, we introduce the diffusion-based dynamics model to predict future trajectories. Second, we apply the conformal prediction technique on the dynamics model to quantify and reduce the uncertainty. Finally, we summarize the whole procedure of our algorithm. In the following subsections, we provide more details on each of these steps and explain the specific techniques we used to implement them. As for notation, we use superscript n to denote diffusion timestep, i or k to denote different trajectories, and t to denote trajectory timestep.

Ƹ

Ƹ

Dynamics Model

(Diffusion Model)

Uncertainty Quantification

𝑓

𝑟 𝑖

∣ 𝜏

𝑖

𝑟 𝑖

ෝ 𝒒

Prediction Set

Uncertainty-aware Optimization

Training

Calibration

Testing

𝑟 𝑖

lies within?

Conformal

Quantile

መ

𝐶

Validating

𝐷𝑐𝑎𝑙

𝐷𝑡𝑒𝑠𝑡

∇𝜃ℒ𝑟𝑒𝑤𝑎𝑟𝑑

∇𝜃ℒ𝑝𝑙𝑎𝑛

3.1 Background

We consider a Markov Decision Process (MDP) represented by the tuple ( S , A , T , r, ρ 0 , γ ) with state space S , action-space A , dynamics T ( s ′ | a , s ) , reward function r ( s , a ) , initial state distribution ρ 0 ( s ) , and discount factor γ ∈ (0 , 1) . We assume a stochastic policy π that maps a state to a distribution over actions. We define τ as trajectories, which consist of sequences of state-action pairs ( s 0 , a 0 , · · · , s T , a T ) generated by a policy π . τ E are expert trajectories generated by an expert policy π E . T represents the time horizon. We use a two-dimensional array to represent the trajectory τ :

with each column corresponding to a timestep in the planning horizon.

3.1.1 Planning with Conditional Diffusion Probabilistic Models

In Bayesian inference, it is intractable to compute the exact posterior distribution due to the high dimensionality and the complexity of the model. One type of latent variable model approximates posterior by the forward or diffusion process q ( τ n | τ n -1 ) , which is modeled as a fixed Markov chain that progressively adds Gaussian noise to the data. Diffusion Probabilistic Models, as described in [46, 47], characterize the process of data generation as an iterative denoising procedure, formalized as the probability distribution p θ ( τ n -1 | τ n ) . The denoising operation (as Eq. 2 shows) is performed in reverse of a forward process q ( τ n | τ n -1 ) , which progressively introduces noise and alters the underlying structure of the data.

where p ( τ N ) = N ( τ N ; 0 , I ) is a standard Gaussian prior and τ 0 ∼ q ( τ 0 ) is noiseless problem data. The reverse process is parameterized as Gaussian (Eq. 3):

in which µ θ and Σ are the mean and covariance of the Gaussian distribution respectively. Janner et al. [9] proposed a way to turn diffusion models into planners that can incorporate different conditions. To achieve this, a learned/expert-designed condition function w ( · ) is added to the diffusion model, which yields a planner P θ described by Eq. 4:

where p θ ( τ ) is the diffusion model's probability distribution, and w ( · ) is a conditional function that can include information such as observations history, desired goals, or rewards. Practically, this can be implemented by sampling from the unperturbed reverse process and replacing the sampled values with conditioning values w ( · ) at all diffusion timesteps. We can use a learned reward function as a condition, as introduced in Sec. 3.2.

3.2 Learned Reward Function

Learning a reward model is important because it allows an intelligent agent to estimate the expected rewards of a given action and state, which is useful for the agent to make informed decisions in an environment. Additionally, using a learned reward model can enable us to adapt to changes in the environment and adjust our predictions accordingly. However, learning an accurate reward model from data is challenging. Thus, we introduce a transformer-based reward model as Eq. 5, which can perform accurate long-term temporal predictions.

where R ( · ) is an MLP layer to map the multi-head attention feature to a predicted reward. d f is the dimension of the attention feature. For more details of transformer architecture, please refer to [68]. The transformer-based reward model is used as a condition for diffusion models to generate reasonable trajectories.

3.3 Quantifying Uncertainty with Conformal Prediction

The inherent uncertainty in the diffusion-based dynamics model P θ can lead to inaccurate predictions. To address these challenges, we employ the conformal prediction technique to obtain prediction uncertainty. Unlike other approaches [4], conformal prediction does not rely on assumptions about the underlying distribution or the predictive model, making it a flexible and robust method for uncertainty quantification in machine learning [24]. This allows us to quantify the uncertainty associated with the predictions and provides more robust estimates of the model's performance.

We have split the dataset D into three parts D train , D cal , and D test -for training, calibration, and testing, respectively. The planner P θ and reward function R ( · ) are learned from the D train dataset.

In our problem setting, we use reconstruction nonconformity scores during training and reward conformity scores during calibration and testing. Reconstruction nonconformity scores is defined as

Diffuser

PlanCP

(a) Maze2D U-Maze

Diffuser (Failed) PlanCP (b) Maze2D Medium

Diffuser

PlanCP

(c) Maze2D Large

23

r=

the mean squared error of the learned trajectories from the expert trajectories. Reward conformity scores are exactly the reward returned by the environment for a given trajectory.

Training To be able to compute and reduce the uncertainty during training, we need to perform conformal prediction on the training set D train . First, we compute the reconstruction nonconformity score as Eq. 6 for each trajectory τ i of the training set D train . A lower reconstruction nonconformity score corresponds to better predictions.

where τ i is the predicted i th trajectory while τ E i is the ground-truth i th trajectory from the expert demonstration of D train .

Calibration and Testing During calibration and testing, we use reward conformity score to quantify the uncertainty. Let r 0 , r 1 , · · · , r k be k + 1 rewards corresponding to τ 0 , τ 1 , · · · , τ k trajectories predicted by our planner P θ . These rewards are defined and returned by the environments, which can be regarded as independent and identically distributed random variables. For planning, the reward conformity score can be seen as a metric that reflects the planner's ability to predict the quality of plans. A higher reward conformity score corresponds to better predictions, indicating that the planner is more accurate in its estimations. Our objective is to quantify the uncertainty associated with the reward r i based on a set of reference rewards { r 0 , · · · , r k } \ r i . Specifically, we aim to construct a prediction interval C i that contains r i with probability at least 1 -α ∈ (0 , 1) . Formally, we seek to find a valid prediction interval C i that satisfies the following inequality:

where C i ( { r 0 , · · · , r k } \ r i ) denotes the prediction interval constructed using the reference rewards { r 0 , · · · , r k } \ r i . The prediction interval C i should provide a measure of the uncertainty associated with the reward r i , and it should be constructed in such a way as to guarantee that with a probability of at least 1 -α , it contains r i .

Lemma 3.1. Given k +1 exchangable random trajectories ( τ E 0 , τ E 1 , · · · , τ E k ) ∼ D train , predicted trajectory ( τ 0 , τ 1 , · · · , τ k ) , corresponding (non)conformity scores d 0 , d 1 , . . . , d k , and a failure probability α ∈ (0 , 1) , then

where the prediction interval C i are defined by

During training, we use reconstruction nonconformity score as d i , while during calibration and testing, we use the reward conformity score as d i . Note that if α is less than 1 /k , our confidence intervals become infinitely wide as we have insufficient data to make predictions with that accuracy.

3.4 Trajectory Optimization on Uncertainty-aware Planning

Trajectory optimization for a discrete-time dynamics system defined by s t +1 = f ( s t , a t ) involves identifying a sequence of actions a 0: T that maximizes an objective function J , which is the sum of per time-step rewards or costs c ( s t , a t ) . The optimization problem can be represented as follows:

Here, T represents the planning horizon.

While training, we utilize the uncertainty calculated from { ˜ r i } as a cost function to optimize the parameters of the planner model. After computing the reconstruction nonconformity scores on our dataset and setting ˆ C = Quantile ( ˜ r 1 , . . . , ˜ r k ; ⌈ ( k +1)(1 -α ) ⌉ k ) , we can form the uncertainty loss:

After adopting the differentiable ranking and sorting techniques [69], the soft quantile function is differentiable. On the other hand, a diffusion probabilistic model is used to parameterize the learned gradient ϵ θ ( τ n , n ) of the trajectory denoising process. The objective for training the ϵ model can be expressed as Eq. 12:

where n ∼ U{ 1 , 2 , . . . , N } represents the diffusion timestep. ϵ ∼ N (0 , I ) represents the noise target. Finally, τ n denotes the trajectory τ 0 that is corrupted with noise ϵ .

In the context of learning, we hope to use conformal prediction to construct prediction intervals for expected cumulative rewards, which can be used to estimate the uncertainty of the predictions. By incorporating these prediction intervals into the decision-making process, we can make more informed decisions and avoid overly optimistic or pessimistic estimations. Finally, the learning objective for planner training is

Given predicted reward ˆ r and ground-truth reward r , the learning objective for reward model training is

The reward model R ( · ) is used as the condition for the diffusion model as Eq. 4 shows. We can obtain reward r returned by the environment or learned reward model. On the calibration dataset D cal , we compute the reward conformity scores and set Q α i , Q 1 -α i and prediction interval ˆ q using conformal prediction as Eq. 15.

Once the prediction interval is constructed, we assess its performance on the test set D test .

3.5 Our Proposed Procedures

We first iterate over the training set D train and for each expert trajectory τ E i , obtain a predicted trajectory τ i using the learned dynamics model P θ . Then we compute the reconstruction nonconformity score ˜ r for each τ E i and construct the uncertainty C i . The loss L plan and L reward are then computed using Eq.13 and Eq.14, respectively. Finally, the dynamics model P θ and reward model R ( · ) are updated using the computed loss.

Next, we iterate over the calibration set D cal , and for each predicted trajectory τ i , obtain the reward conformity score r i and construct the predicted interval ˆ q i .

Finally, we verify the predicted interval using the test set D test to evaluate its performance. The whole procedure is shown in Alg. 1.

4 Experiments

In our experiments, we aim to answer the following questions: (1) How to construct uncertainty intervals that can capture the dynamics model's prediction error and provide a measure of confidence in the planned trajectories? (2) Can the uncertainty of the dynamics model be reduced during the training process? To answer these questions, we evaluate PlanCP on multiple planning and offline RL tasks. We show planning with our algorithm can update the dynamics model while quantifying uncertainty. In principle, our approach can be applied to a wide range of diffusion dynamics models. For illustrative purposes, we demonstrate our framework using a typical Diffuser [9]. Our primary objective is to assess and minimize the model's uncertainty, while performance improvement is a by-product of our approach and is not necessarily guaranteed. We assess the performance of our models based on several quantitative metrics, including credibility and rewards.

4.1 Training setup

We set the uncertainty weight to λ uncertainty = 5 and the failure probability to α = 0 . 1 . To optimize the model, we use the Adam [73, 74] optimizer with a learning rate of 2 × 10 -4 . We train the diffusion dynamics model on the training set D train for 2 × 10 5 iterations. As we lack ground-truth for evaluation, we generate the evaluation datasets D cal and D test during the evaluation process. We partition the evaluation data into 20% for D cal and 80% for D test . The uncertainty measure ˆ q is derived from the 20% calibration set through conformal prediction.

4.2 Long Horizon Task Planning

We use Maze2D to evaluate long-horizon planning. ˆ q is a measure of uncertainty for the rewards obtained by a planned trajectory. For most variants of the Maze2D environment, we have a smaller uncertainty interval, especially for Maze2D U-Maze, as Tab. 1 and Tab. 2 show. As Fig. 2 shows, Compared to Diffuser, our PlanCP has successfully generated a smoother path with lower uncertainty, which achieves a higher reward. Furthermore, in cases where the previous Diffuser method fails to plan, our PlanCP is able to derive a feasible path.

4.3 Offline Reinforcement Learning

Finally, we evaluate the ability to recover an effective single-task controller from heterogeneous data of varying quality using the D4RL Benchmark [75]. The D4RL benchmark consists of several challenging continuous control tasks, such as the HalfCheetah and Ant environments. For each task, we measured the average return achieved by our approach and the baseline methods over multiple runs. We also computed the standard deviation of the returns to measure the stability of the algorithms. Our experimental results demonstrate that our approach outperforms the baseline methods on most of the D4RL environments, as Tab. 3 shows. For most of the environments, our approach achieved a higher return with a smaller uncertainty interval compared to the baseline methods. We also observed that our approach achieved a more stable performance across multiple runs, as evidenced by the smaller standard deviation of the returns.

Overall, experimental results show that our framework can construct uncertainty intervals that can capture the dynamics model's prediction error and provide a measure of confidence in the planned trajectories.

5 Discussion, Limitations, and Conclusions

In this work, we tackled the problem of obtaining uncertainty estimates for planning and offline RL tasks by proposing a new approach that applies conformal prediction to a diffusion-based dynamics model. Experimental results show that our framework maintains the original model's accuracy while reducing uncertainty. One limitation of our study is that we have not conducted evaluations on real robotic platforms. An intriguing avenue for future research involves gaining a comprehensive understanding and conducting a meticulous analysis of how conformal prediction compares to the extensive body of research in planning under uncertainty, including non-parametric models. Our work focuses primarily on proposing an algorithmic framework. Future work could also explore the use of more recent and effective diffusion models, as well as other types of dynamics models. Moreover, evaluating the proposed method on real-world applications, such as autonomous driving or robotics, where reliable uncertainty estimates are crucial for safety-critical systems, would be an interesting avenue for future research.

Acknowledgments and Disclosure of Funding

The NASA University Leadership initiative (grant #80NSSC20M0163) provided funds to assist the authors with their research, but this article solely reflects the opinions and conclusions of its authors and not any NASA entity.

This research was also partly funded by a gift from Meta's Project Aria.

Parth Nobel was supported in part by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-1656518. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References

Michael Andreas Erdmann. On motion planning with uncertainty. 1984.

Mykel J Kochenderfer, Tim A Wheeler, and Kyle H Wray. Algorithms for decision making . MIT press, 2022.

Jonathan Ko and Dieter Fox. Gp-bayesfilters: Bayesian filtering using gaussian process prediction and observation models. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 3471-3476, 2008.

Christian Soize. Uncertainty quantification . Springer, 2017.

Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization , volume 28. Princeton university press, 2009.

Chancellor Johnstone and Bruce Cox. Conformal uncertainty sets for robust optimization. In Conformal and Probabilistic Prediction and Applications , pages 72-90. PMLR, 2021.

Rinu Boney, Juho Kannala, and Alexander Ilin. Regularizing model-based planning with energy-based models. In Conference on Robot Learning , pages 182-191. PMLR, 2020.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 14129-14142. Curran Associates, Inc., 2020.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104-114. PMLR, 2020.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11319-11328. PMLR, 18-24 Jul 2021.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 69(2):243-268, 2007.

Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. Advances in Neural Information Processing Systems , 34:22313-22324, 2021.

Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems , 34:15682-15694, 2021.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer Science & Business Media, 2005.

Ulf Johansson, Henrik Linusson, Tuve Löfström, and Henrik Boström. Model-agnostic nonconformity functions for conformal classification. In 2017 International Joint Conference on Neural Networks (IJCNN) , pages 2072-2079. IEEE, 2017.

Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli , 29(1):1-23, 2023.

Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets. Journal of the American Statistical Association , 108(501):278-287, 2013.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Yachong Yang and Arun Kumar Kuchibhotla. Finite-sample efficient conformal prediction. arXiv preprint arXiv:2104.13871 , 2021.

Junning Huang, Sirui Xie, Jiankai Sun, Qiurui Ma, Chunxiao Liu, Dahua Lin, and Bolei Zhou. Learning a decision module by imitating driver's control behaviors. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning , volume 155 of Proceedings of Machine Learning Research , pages 1-10. PMLR, 16-18 Nov 2021.

Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou. Neuro-symbolic program search for autonomous driving decision module design. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning , volume 155 of Proceedings of Machine Learning Research , pages 21-30. PMLR, 16-18 Nov 2021.

Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning , pages 345-356. Springer, 2002.

Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76(1):71-96, 2014.

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3), 2008.

Benjamin Kompa, Jasper Snoek, and Andrew L Beam. Empirical frequentist coverage of deep learning uncertainty quantification procedures. Entropy , 23(12):1608, 2021.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415 , 2022.

Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems , 32, 2019.

Vladimir Vovk. Transductive conformal predictors. In Artificial Intelligence Applications and Innovations: 9th IFIP WG 12.5 International Conference, AIAI 2013, Paphos, Cyprus, September 30-October 2, 2013, Proceedings 9 , pages 348-360. Springer, 2013.

Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural networks. In Tools in artificial intelligence . Citeseer, 2008.

Paolo Toccaceli and Alexander Gammerman. Combination of inductive mondrian conformal predictors. Machine Learning , 108:489-510, 2019.

Tuve Löfström, Ulf Johansson, and Henrik Boström. Effective utilization of data in inductive conformal prediction. In International Joint Conference on Neural Networks, Dallas, TX, USA, August 4-9, 2013. IEEE, 2013.

Vineeth N Balasubramanian, Shayok Chakraborty, and Sethuraman Panchanathan. Conformal predictions for information fusion: A comparative study of p-value combination methods. Annals of Mathematics and Artificial Intelligence , 74(1-2):45-65, 2015.

Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence , 74:9-28, 2015.

Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated conformal prediction. In Artificial Intelligence Applications and Innovations: AIAI 2014 Workshops: CoPA, MHDW, IIVC, and MT4BD, Rhodes, Greece, September 19-21, 2014. Proceedings 10 , pages 231-240. Springer, 2014.

Kamile Stankeviciute, Ahmed M. Alaa, and Mihaela van der Schaar. Conformal time-series forecasting. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 6216-6228. Curran Associates, Inc., 2021.

Kegan J. Strawn, Nora Ayanian, and Lars Lindemann. Conformal predictive safety filter for rl controllers in dynamic environments, 2023.

Sander Tonkens, Sophia Sun, Rose Yu, and Sylvia Herbert. Scalable safe long-horizon planning in dynamic environments leveraging conformal prediction and temporal correlations. In ICRA , 2023.

Lars Lindemann, Matthew Cleaveland, Gihyun Shim, and George J. Pappas. Safe planning in dynamic environments using conformal prediction. IEEE Robotics and Automation Letters , 8(8):5116-5123, 2023.

Yuxiao Chen, Ugo Rosolia, Chuchu Fan, Aaron Ames, and Richard Murray. Reactive motion planning with probabilisticsafety guarantees. In Conference on Robot Learning , pages 19581970. PMLR, 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jiankai Sun, Rui Liu, and Bolei Zhou. Hiabp: Hierarchical initialized abp for unsupervised representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9747-9755, 2021.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689 , 2019.

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems , 32, 2019.

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning , pages 3732-3747. PMLR, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Jiankai Sun, Lantao Yu, Pinqian Dong, Bo Lu, and Bolei Zhou. Adversarial inverse reinforcement learning with self-attention dynamics model. IEEE Robotics and Automation Letters , 6(2):1880-1886, 2021.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Modelbased policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 12519-12530. Curran Associates, Inc., 2019.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21810-21823. Curran Associates, Inc., 2020.

Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. arXiv preprint arXiv:2105.07351 , 2021.

Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11) , pages 465-472, 2011.

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 2450-2462. Curran Associates, Inc., 2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations , 2020.

Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-end differentiable adversarial imitation learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 390-399, International Convention Centre, Sydney, Australia, 06-11 Aug 2017.

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31, pages 4754-4765. Curran Associates, Inc., 2018.

Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.02057 , 2019.

Jiankai Sun, Shreyas Kousik, David Fridovich-Keil, and Mac Schwager. Connected autonomous vehicle motion planning with video predictions from smart, self-supervised infrastructure, 2023.

Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg. Plate: Visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters , 7(2):4924-4930, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper combines diffusion modeling and conformal prediction to predict state-action trajectories with uncertainty quantification. The proposed approach then performs uncertainty-aware model-based planning with strong results on several established offline RL benchmarks.\n\nSTRENGTHS:\nThe approach has fairly strong results on several established offline RL benchmarks and outperforms several state-of-the-art baselines. The results show that uncertainty-aware planning does perform better than simply rolling out the underlying diffusion model alone.\n\nThe introduction/abstract are well-written and motivate the approach well. Most of the writing is quite good (some feedback on parts I think are unclear below). The figures are also well done.\n\nTo my knowledge, the proposed approach is novel and technically sound. I think working to incorporate more ideas from uncertainty quantification into offline RL methods is worth investigating deeper as well.\n\nWEAKNESSES:\nThere are no comparisons to other uncertainty quantification methods. Even within the offline RL literature, several existing methods [1][2][3] rely on uncertainty estimates and could be compared to. I think at the very least a comparison to popular uncertainty estimation methods in the model-based RL literature (e.g. bootstrapping disagreement with ensembles, predicting variances) could be helpful.\n\nI struggle with Sections 3.3 and 3.4 a bit and think they could use some improvements clarity-wise.\n\nWhile the approach is novel, it is a fairly straightforward application of conformal prediction to offline model-based planning. While I don't think the result is groundbreaking, it is certainly interesting enough to merit acceptance in my view.\n\n[1] Zhan, Xianyuan, Xiangyu Zhu, and Haoran Xu. \"Model-based offline planning with trajectory pruning.\" arXiv preprint arXiv:2105.07351 (2021).\n\n[2] Yu, Tianhe, et al. \"Mopo: Model-based offline policy optimization.\" Advances in Neural Information Processing Systems 33 (2020): 14129-14142.\n\n[3] Kidambi, Rahul, et al. \"Morel: Model-based offline reinforcement learning.\" Advances in neural information processing systems 33 (2020): 21810-21823.\n\nQUESTIONS:\nI am confused about the role that the expert trajectories play in this approach. The approach is evaluated on datasets without labeled expert trajectories, i.e. the datasets include suboptimal demonstration data. But the Algorithm 1 block references expert trajectories, and I'm not sure why? It seems to me that the approach doesn't really assume optimal demonstrations.\n\nRelated to the above, I'm not sure what Algorithm 1 is actually showing. The \"result\" line indicates that the output is a prediction $\\tau$ with associated uncertainty $C$, but L1-9 describe the complete training loop and there aren't really details on inference in this figure.\n\nI think a brief definition of the $Quantile$ operator would be useful when it's first introduced in Equation 9.\n\nI feel like Equation 5 and the accompanying description of transformers is not necessary to include in the main body, and working to include more of the actual experimental results (currently in the appendix) would be a better use of space in my opinion.\n\n===\nI have read the rebuttal and still lean towards acceptance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper the authors study how to measure uncertainty in a planner with a generative diffusion model, and how to reduce uncertainty during prediction. Results are presented on several MDPs.

STRENGTHS:
+ coupling planners to diffusion models is an interesting idea worth exploring.
+ measuring and reducing uncertainty is a good goal.

WEAKNESSES:
- I found this paper really hard to read and follow. Although the paper appears to address planning, the text refers to expert trajectories the "planner" is supposed to reconstruct. It is not clear if the task is planning or imitation learning. Various aspects are not well defined, for example, how is a quantile directly a loss function? How do you optimize a quantile? At the same time there is a lot of repetition of reconstruction nonconformity and reward conformity. Multiple experimental tasks, such as "test time flexibility", are introduced without adequate discussion of why this approach should be applied to them. The results are in tables that are almost unreadable and there is not enough discussion of the meaning of the results.

QUESTIONS:
n/a

LIMITATIONS:
not an adequate discussion
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the challenge of uncertainty estimation for planning. The authors propose the use of diffusion models for learning dynamics, which have demonstrated effectiveness in overcoming challenges such as multi-modal action distributions. To quantify the uncertainty of these dynamics models, they employ Conformal Prediction (CP), a technique for constructing prediction sets with valid coverage. They introduce PlanCP, a framework that connects conformal prediction to planning and optimizing the model explicity to minimize uncertainty. The effectiveness of uncertainty sets is evaluated through coverage and optimization performance, and the algorithm is tested in D4RL benchmarks and block stacking problem, showcasing reduced uncertainty and outperforming prior algorithms. The authors highlight the flexibility of their method, as it can be combined with different model-based planning approaches and provides uncertainty estimates of the dynamics model.\n\nSTRENGTHS:\n- The paper provides a clear problem statement that is well motivated and clearly outlines the proposed approach with intuitive evaluation metrics.  Overall I found the paper well written and easy to follow.\n    \n- The proposed methodology presents a robust framework for quantifying and mitigating uncertainties within diffusion-based dynamics models, which have garnered significant attention in recent times.\n    \n- The quantitative results on D4RL although not SOTA(in terms of rewards), consistently outperforms the baseline without uncertainty quantification. It is important to note that the limited performance gains could be attributed to the constraints of the D4RL dataset itself, potentially reaching the performance ceiling. Nevertheless, the outlined approach consistently outperforms the baseline methods across the metrics identified.\n\nWEAKNESSES:\n-    The computational complexity of the proposed framework, especially during training and calibration, could hinder its scalability and applicability in real-time or resource-constrained scenarios. It would be beneficial if the authors provided more details regarding the computational aspects of training and inference.\n    \n- The handling of partial observability in the framework is not adequately explained. Naive extensions to partially observed Markov Decision Processes (POMDPs) may lead to unreliable estimations in diffusion models. Further elaboration on how the framework addresses partial observability would be valuable.\n    \n- The paper lacks clarity on how uncertainty estimates will manifest in highly stochastic domains. It would be insightful to include experiments on a benchmark dataset with increasing levels of stochasticity to demonstrate the framework's performance under such conditions.\n\nQUESTIONS:\nI was wondering if the authors had some thoughts on what adapting CP framework to other conditioning approaches will entail, such as temporal condition guidance mechanisms.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes uncertainty quantification for learned dynamics model and imitation learning. The authors incorporates an uncertainty statistic as part of the loss to train their dynamics model that uses a diffusion model architecture. They show that doing so brings performance improvement empirically on common RL simulation environments. The method also performs conformal prediction on the learned reward forecast.\n\nSTRENGTHS:\nUsing UQ methods and optimizing to reduce uncertainty in dynamics modeling is a great idea. The authors are able to make the first step of incorporating conformal statistics directly into the loss function, and is able to show results of (slight) improvement in performance.\n\nWEAKNESSES:\nThere are many areas that the paper needs improvement. \n\n1. English. There are grammatical errors, awkward sentence structures, and confusing statements throughout the paper. Although you don't need perfect English for a CS paper, improving on writing will help you get your message across. Using the abstract as an example, I would edit:\n\n- line 3-5: \"to overcome the xx, yy, and zz challenges\" -> to overcome challenges such as xx, yy, and zz.\n- line 14-15: \"Furthermore, during the test, PlanCP can also measure the model uncertainty\" -> Unclear what you are trying to say. Do you mean that model uncertainty is also used for planning during test time? \n- line 20-21: \"Our method is highly flexible and can combine .... and produces ...\" ->  Our model can be combined with ... to produce .... And what do you mean by flexible? is it with regard to tasks, or underlying models? \n- Figure 1: \"Dynamic model\" -> dynamics model\n\n2. There has been various recent works that uses CP for planning (see list below). You can argue that your setting is different from theirs, but it's important to cite them to provide context. \n\n- Lindemann, Lars, et al. \"Safe planning in dynamic environments using conformal prediction.\" IEEE Robotics and Automation Letters (2023).\n- Strawn, Kegan J., Nora Ayanian, and Lars Lindemann. \"Conformal Predictive Safety Filter for RL Controllers in Dynamic Environments.\" arXiv preprint arXiv:2306.02551 (2023).\n- Tonkens, Sander, et al. \"Scalable Safe Long-Horizon Planning in Dynamic Environments Leveraging Conformal Prediction and Temporal Correlations.\" ICRA (2023)\n\n3. More explanation is needed for the experiments section. \n- The term \"Validity %\" should be changed to \"Coverage %\". Validity is a binary metric; a method either satisfies the validity condition, or not. [1]\n- On the topic of validity - you chose $\\alpha = 0.1$ (line 261), does that mean that your coverage should be at least 90%? (or should it be 80% since you are choosing $[Q^{1-\\alpha}, Q^{\\alpha}]$ in Eq. 15?) Does that mean for all of your experiments, the CP uncertainty interval is _invalid_? That's problematic and means that your data violates the assumptions you are making. You can't say your method captures uncertainty in this case. \n- You need to say in your table captions that bold font means better performance, or the meaning is unclear.\n- How exactly is credibility calculated? it's not explained in either the main text or the appendix. What is the theoretical basis for using the $p$-value as metric?\n\n4. Lastly The term \"learning to reduce uncertainty\" is unsound. The \"reconstruction nonconformity scores\" are calculated on the training set, which violates the algorithm of conformal prediction [1]. The $\\mathcal{L}(\\theta)_{uncertainty}$ in equation 13 is hence not a measure of uncertainty, but just a training heuristics. The only valid CP component is the interval for rewards, which is not used anywhere in training. \n\n5. Contribution is not significant. Form my understanding, the authors used a common CP technique on an existing diffusion dynamics model. The uncertainty results are not new (in CP literature), and improvements in reward are rather insignificant. \n\n[1] Angelopoulos, Anastasios N., and Stephen Bates. \"A gentle introduction to conformal prediction and distribution-free uncertainty quantification.\" arXiv preprint arXiv:2107.07511 (2021).\n\nI think the directions the authors are going in is interesting and insightful; with some work this paper can be a great contribution to the community. However in its current state, the paper is not fit for publishing at NeurIPS yet.\n\nQUESTIONS:\nSee weaknesses 3, 4, and 5.\n\nLIMITATIONS:\nSee weaknesses. I do not think the discussion & limitation section fully address the limitations of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a method for learning a dynamics model that uses conformal prediction for explicit representation of the uncertainty. The dynamics model is used for sequential decision making such as planning in a maze or learning control in one of the D4RL problems.\n\nSTRENGTHS:\n* The primary strength of this paper is the novel use of conformal prediction. The authors correctly describe several possible benefits of conformal prediction as a representation of uncertainty, including not needing a commitment to a particular representation or parameterisation of the uncertainty in the dynamics.\n* The authors give a complete algorithm --- it is (mostly) clear how all the pieces work. \n* The paper is reasonably well written and clear.\n\nWEAKNESSES:\n* The primary weakness of the paper is that it is not clear what exactly the authors have accomplished. Developing a new method for sequential decision making should either show that we can solve problems that we could not solve before ((or find better solutions than was possible), or alternatively, provide some understanding and analysis of the value of a dynamics model that encodes uncertainty using conformal prediction. Unfortunately, this paper does not show that it can solve problems that could not be solved before, or outperform the state of the art. The only comparison is to Diffusion RL which is not the best performing RL algorithms according to the original Diffusion RL paper (Janner et al) or the current leader in the [D4RL benchmarks](https://paperswithcode.com/sota/gym-halfcheetah-expert-on-d4rl).\n\n* It's fine to not have the best performer on D4RL, but then what is conformal prediction buying for the algorithm? A far better comparison would have been to a model-based algorithm that had an explicit parametric representation of uncertainty, and an analysis of what is actually being learned. What is the effect of the loss functions in 12-14 on $P(\\theta)$?\n\n* The experimental results in general are not very compelling, at least in part because there is no systematic evaluation of the uncertainty. The paper needed to remind the reader of the uncertainty models in the D4RL and Maze engines, and even better, show that these uncertainties are difficult to capture with parametric models. Even better would be to show that conformal prediction outperforms non-parametric dynamics models such as Ko and Fox (2009, 2011) in important ways.\n\n* Additionally, while the paper overall is well-written, it is not careful about the distinction between planning and RL. This paper appears to be fundamentally an RL paper, and not a planning paper in the sense that if the model changes or the reward changes, there is no optimisation that can be used to recover from this change. If the loss function was solely equation 13, and did not include equation 14, then at least the model would be robust to changes in the reward function, and a new sequence of actions could be optimised over.\n\nQUESTIONS:\n* The definition of calibration is very unclear. Why is a separate calibration data set is used for the reward conformity loss function, rather than use the training data.\n\n* What is the exact definition of the Quantile function?\n\n* Line 7 of Algorithm 1 says \"update the dynamics model\" but it is not explained in the text how to do this. Figure 1 suggest the update is done through gradient descent ($\\nabla_\\theta \\mathcal{L}_{reward}$, etc.), but it is not obvious to me that you can differentiate through the Quantile function. How is this done? This is a minor point.\n\nLIMITATIONS:\nThe authors provide a limitation section, but I disagree with them that the limitation of the approach is the absence of real robot experiments. Instead, I think the primary limitation is the lack of understanding and careful analysis of what conformal prediction achieves relative to the large body of work in planning under uncertainty, including non-parametric models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}