End-to-End Conformal Prediction for Trajectory Optimization

Authors: Han Wang, Chao Ning
Primary area: applications to robotics, autonomy, planning
Keywords: conformal prediction, end-to-end, trajectory optimization, risk allocation, decision-focused learning

Abstract:
Conformal Prediction (CP) is a powerful tool to construct uncertainty sets with coverage guarantees, which has fueled its extensive adoption in generating prediction regions for decision-making tasks, e.g., Trajectory Optimization (TO) in uncertain environments. However, existing methods predominantly employ a sequential scheme, where decisions rely unidirectionally on the prediction regions, and consequently the information from the decision-making end fails to be transmitted back to instruct the CP end. In this paper, we propose a novel End-to-End CP (E2E-CP) framework for shrinking-horizon TO with a joint risk constraint over the entire mission time. Specifically, a CP-based posterior risk calculation method is developed by fully leveraging the realized trajectories to adjust the posterior allowable risk, which is then allocated to future times to update prediction regions. In this way, the information in the realized trajectories is continuously fed back to the CP end, enabling attractive end-to-end adjustments of the prediction regions and a provable online improvement in trajectory performance. Furthermore, we theoretically prove that such end-to-end adjustments consistently maintain the coverage guarantees of the prediction regions, thereby ensuring provable safety. Additionally, we develop a decision-focused iterative risk allocation algorithm with theoretical convergence analysis for allocating the posterior allowable risk which closely aligns with E2E-CP. The effectiveness and superiority of the proposed method are demonstrated through benchmark experiments.

Decision: reject
Reviewer scores: 5, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730665771515

thread_type:
official_review

<thread_object>
creator: reviewer_igbg

<content>
summary:
This paper studies the problem of online trajectory optimization in an uncertain environment. The proposed algorithm includes a chance-constrained optimization problem that applies conformal prediction (CP) in two ways. First, it uses forward-looking CP to predict a priori confidence intervals for future obstacle positions, which are used for planning. Second, it uses backward-looking CP to compute the posterior confidence intervals of past obstacle positions to estimate how much of the risk budget was actually "used up" compared to the planned amount of risk. The risk budget freed up by this backward-looking assessment is applied to future timesteps, allowing less conservative behavior. The experiments are simple, presenting well-chosen ablations rather than a comprehensive suite of baseline methods, but I found them to be compelling.

strengths:
I really enjoyed reading this paper. The core of the paper is a really interesting idea (understand how much risk you have previously incurred, so as to better decide how much risk to take on in future), and the authors rigorously build a rigorous framework around this idea to design an intuitive trajectory optimization framework.

The paper is clearly written throughout, and although I did not check all proofs, the ones I did check were fairly clear. It is possible that there is some nuance of conformal prediction that I am missing, but overall the approach seems sound.

weaknesses:
The paper is missing some key implementation details (notably: how was the optimization problem solved/what solver was used?). The paper is also missing a discussion of limitations. If there is not space in the main paper, these could be included in the appendix, but I would like to see limitations at least mentioned in the main text.

questions:
1. In section 3.2, is the prediction also conditioned on the history of obstacle positions $Y_{0:t}$ or only on the most recent observation $Y_t$? Your notation implies the latter, but I'd imagine that the former would yield more accurate predictions.
2. Line 90: "level $\beta$ quantile" -- do you mean "level $1-\alpha$ quantile"? If not, what is $\beta$?
3. The empirical results suggest that the proposed method (E2E-CP with IRA) is still conservative. What do you think are the reasons for this, and how would you go about reducing this conservatism?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678213540

thread_type:
official_review

<thread_object>
creator: reviewer_t4jb

<content>
summary:
In this paper, the authors proposed a conformal prediction based posterior risk calculation method to obtain the posterior probability of collision conditional on realized trajectories. This method leverages the realized trajectories to adjust the posterior allowable risk in an online setting. The experiment results demonstrate the efficacy of the proposed method.

strengths:
1) The paper is well organized and easy to follow.
2) This paper utilizes the conformal prediction-based method to achieve the end-to-end adjustments of the prediction region, improving the limitation of conformal prediction.

weaknesses:
1) The prediction regions should be narrow and cover the ground truth data. However, large prediction regions would also achieve a high collision avoidance rate but not meaningful in real-world situations. The authors should provide more details about the prediction region. 
2) Proving the effectiveness of the proposed method is very important. However, the authors merely apply the proposed method on two trajectory prediction models which were proposed in 2001 and 2006, respectively. The authors should provide experiment results on more state-of-the-art models.

questions:
1) Performance of the proposed method on more state-of-the-art methods.
2) Performance of the proposed method under data distribution shift.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716747327

thread_type:
official_review

<thread_object>
creator: reviewer_wqry

<content>
summary:
This paper introduces a novel trajectory optimization algorithm grounded in conformal prediction, which dynamically adjusts posterior risk based on observed trajectories. Benchmark experiments highlight the algorithm’s effectiveness, showcasing its superior performance compared to existing methods.

strengths:
1. The algorithm proposed in the paper is novel, supported by rigorous theorems and proofs that strengthen its theoretical foundation.
2. The presentation is clear and well-structured.

weaknesses:
1. I have concerns regarding the assumptions made in this paper, particularly the strong independence assumptions on obstacle trajectories, which form the basis of the proposed algorithm. In Section 3.1, the authors assume the independence between D and system (1) (lines 134-135) and further assume that the real joint obstacle trajectory Y and the N available joint obstacle trajectories Y^(i) are independent and identically distributed (i.i.d.) (lines 143-145). However, in real-world applications such as autonomous driving, obstacle trajectories—specifically those of other vehicles on the road—are highly interdependent, as each vehicle’s decisions are influenced by the behaviors of surrounding vehicles. The interdependence between obstacle trajectories is a key factor contributing to the complexity of real-world problems. Assuming independence significantly simplifies the problem but may limit the algorithm's applicability in practical, interdependent scenarios. It would be beneficial for the authors to provide a justification for these independence assumptions and to identify any real-world applications where obstacle trajectories can be reasonably assumed to be independent.
2. Another concern is the limited performance improvement observed with the proposed algorithm. In Table 1, the collision avoidance rate shows only a marginal increase, raising questions about the practical impact of the algorithm. Furthermore, in Table 3 (Appendix C.2), the collision avoidance rate actually decreases when using the proposed algorithm, suggesting potential issues with its robustness. Additionally, the benchmark experiments include only two robotic systems, which may not provide sufficient diversity to fully evaluate the algorithm's effectiveness across different scenarios.
3. Additionally, this paper’s focus on trajectory optimization is largely unrelated to machine learning or deep learning. Given the emphasis on trajectory optimization techniques rather than learning-based methods, it would be more appropriately submitted to robotics-focused publication venues rather than conferences dedicated to deep learning.
4. Minor typo: On line 190, "level beta" should be corrected to "level alpha."

questions:
1. As noted in the weaknesses section, it would be valuable for the authors to provide a rationale for these independence assumptions and to highlight any real-world applications in which obstacle trajectories can reasonably be assumed to be independent.
2. The paper would benefit from additional benchmark examples beyond the two robotic systems currently evaluated. Expanding the benchmarks would provide a more comprehensive assessment of the algorithm's effectiveness across diverse scenarios.
3. It would be helpful if the authors could clarify why a paper focused on trajectory optimization, rather than on learning-based approaches, is suitable for a deep learning conference.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732472503

thread_type:
official_review

<thread_object>
creator: reviewer_xxlt

<content>
summary:
In this paper, the authors proposed an E2E-CP framework for shrinking-horizon TO in uncertain environments and the collision risk over the total mission time is always constrained. The proposed method leverages the information of past decisions to adjust prediction regions in an end-to-end fashion. The experiment results demonstrate the effectiveness of the proposed model for trajectory prediction task.

strengths:
1.The paper provides detailed theoretical proof for the proposed method.
2. This paper compares the model performance of sequential CP, average risk allocation setting, and iterative risk allocation setting.

weaknesses:
1.The prediction regions are expected to cover the ground truth trajectory and narrow. Otherwise, overlarge prediction regions would also achieve a high collision avoidance rate. The authors should provide more details about how narrow the prediction regions are.

2. The authors should demonstrate the model performance by implementing more state-of-the-art models but not only the models proposed about 20 years ago.

questions:
Please see the above limitations and address the questions. In addition to the limitations, another question is:

Based on Eqs. 9, 13, and 17, both the upper bound of the posterior violation probability and the allocable risk at specific time step are fixed for the average risk allocation setting, as the upper bound of the posterior violation probability is calculated based on the calibration data D_{cal}^{2}. This violates the real-world situation where the real-world data and calibration data follow different distribution. However, experiment result in Table 1 shows that this average risk allocation method still achieves 89.1% collision avoidance rate. Are there any data distribution shifts in the testing data? If not, what’s the performance of the proposed method under distribution shift?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732890966520

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers and ACs,

We sincerely thank all the reviewers and ACs for your time and efforts on providing valuable review comments and questions, which have helped us to improve the quality of this manuscript. If you have any additional questions or require further clarification, please feel free to let us know. Your insights are highly appreciated.

In our paper, we propose, for the first time in the literature E2E-CP, an end-to-end framework based on Conformal Prediction (CP) for shrinking-horizon Trajectory Optimization (TO) in uncertain environments. The proposed E2E-CP enjoys both performance and validity. Specifically, E2E-CP leverages CP to construct the prediction regions of obstacle positions and adjusts these regions online using the feedback information embedded in decisions. Additionally, such end-to-end adjustments in E2E-CP do not compromise the theoretical coverage guarantees of prediction regions. In the original manuscript, E2E-CP was validated on the kinematic vehicle model and the quadrotor model, demonstrating a reduction in average cost of over 78.5% and 58.0%, respectively, compared to the state-of-the-art method.

Most reviewers are concerned about the performance of our method across a wider range of models and under different scenarios, such as distribution shift and interdependence. So we conduct additional experiments and present the results in the appendix of the revised manuscript. The newly added content in the revised manuscript is highlighted in blue.

In the additional experiments, we implement our method on the dynamic bicycle model (2021) [1] in Appendix C.3, investigate the performance of our method under distribution shift in Appendix D, and examine the performance of our method under the interdependence between the system and obstacles in Appendix E. We provide a brief summary and discussion of the additional experimental results as follows. For detailed results and discussions, reviewers are kindly referred to the appendix of the revised manuscript.

In the experiments on the dynamic vehicle model [1], E2E-CP achieves at least a 40.6% reduction in average cost compared to the state-of-the-art method, further demonstrating the superiority and generalizability of E2E-CP.

In the experiments on distribution shifts, E2E-CP can still satisfy the total risk tolerance constraint while maintaining an average cost reduction of at least 57.1% under moderate distribution shifts. The experimental results demonstrate that E2E-CP exhibits a certain degree of robustness to moderate distribution shifts.

In the experiments on interdependent scenarios, the reduced collision avoidance rate caused by the interaction between the system and obstacles is negligible and remains well within the total risk tolerance. Moreover, the dependence between the system and obstacles has almost no impact on the average cost empirically. In summary, the interactions among agents in multi-agent systems do not significantly affect our algorithm and its performance.

Moreover, we provide more details about how narrow the prediction regions are in Appendix F, and discuss the limitations of our method in the main text.

We hope these new results and discussions can address the reviewers’ concerns.

[1] Astghik Hakobyan and Insoon Yang. Wasserstein distributionally robust motion control for collision avoidance using conditional value-at-risk. IEEE Transactions on Robotics. 2021.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method for online trajectory optimization under uncertainty by applying conformal prediction in a chance-constrained optimization framework. The experiments are basic and are a point of criticism by the reviewers. While two reviewers recommend acceptance and two reviewers think that the paper is marginally below acceptance, I am recommending rejection because the experiments are weak and the baselines are fairly old.