Title: Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Conformal Prediction

Reinforcement Learning

Policy Evaluation

Infinite-Horizon
Abstract: Reliable uncertainty quantification is crucial for reinforcement learning (RL) in high-stakes settings. We propose a unified conformal prediction framework for infinite-horizon policy evaluation that constructs distribution-free prediction intervals for returns in both on-policy and off-policy settings. Our method integrates distributional RL with conformal calibration, addressing challenges such as unobserved returns, temporal dependencies, and distributional shifts. We propose a modular pseudo-return construction based on truncated rollouts and a time-aware calibration strategy using experience replay and weighted subsampling. These innovations mitigate model bias and restore approximate exchangeability, enabling uncertainty quantification even under policy shifts. Our theoretical analysis provides coverage guarantees that account for model misspecification and importance weight estimation. Empirical results, including experiments in synthetic and benchmark environments like Mountain Car, show that our method significantly improves coverage and reliability over standard distributional RL baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages conformal prediction to derive distribution-free predictive intervals (PI) for the return $G^\\pi(s_{test})$ in discounted MDPs for a fixed policy $\\pi$ and a test state $s_{test}$, applicable to both on-policy and off-policy settings. The method is particularly suited for neural network-based predictions due to its distribution-free nature. Specifically, the authors split the data into training and calibration sets: the QTD algorithm is trained on the training set to estimate the value function and generate approximate samples of $G^\\pi(s)$, while k-step TD estimates ($G^\\pi(s_k)$ is sampled from the QTD model) of $G^\\pi(s_{cal})$ on the calibration set are used to compute nonconformity scores (defined as the difference between the k-step TD estimates and the value function estimates). To enhance performance, the authors incorporate weighted subsampling and resampling techniques.\nUnder the assumption that $G^\\pi(s)$ admits a bounded Lebesgue density, the authors establish coverage guarantees for the derived PI. Numerical experiments demonstrate that, compared to PI obtained directly via distributional RL (DRL), the proposed method achieves the desired coverage probability without significantly inflating interval widths.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis work elegantly integrates conformal prediction with distributional RL and replay buffer techniques to derive a theoretically guaranteed predictive interval for $G^\\pi(s_{test})$. The empirical results convincingly validate the method’s effectiveness, making this a compelling contribution.\n\nWeaknesses:\n\nI do not find major flaws, I offer the following refinements to enhance clarity and rigor.\n\n1. Notational Clarity: The symbol m is overloaded for both QTD and resampling parameters.\n\n2. Clarify Methodology for Off-Policy Evaluation (Line 171-178): The current description of estimating $\\eta^\\pi(s,a)$ in the off-policy setting is vague. Explicitly state the distributional off-policy evaluation method used in the paper (e.g., distributional variant of Fitted Q Evaluation) or cite relevant work.\n\n3. Clarify Methodology used in Eqn. (4): Please explain how to estimate P(\\delta | s) (e.g. logistic regression?) and explicitly note that \n P(δ=0)/P(δ=1) can be estimated via frequency ratios (if this aligns with the authors’ intent).\n\n4. In Algorithm 1, on Result line, conformal predictive region can be replaced with conformal predictive interval (because it is indeed an interval). And the impact of the multiple subsampling parameter $\\xi$ is not discussed in the paper (although it is discussed in the Appendix).\n\nQUESTIONS:\nThe coverage guarantees rely on the return distribution having a bounded Lebesgue density. How does the method perform empirically when this assumption is violated which is often the case? A brief discussion or experiment would strengthen the practical relevance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper  proposes a split-conformal‐prediction for infinite-horizon reinforcement-learning (RL) returns. The authors propose a method based on distributional RL to compute pseudo-returns, which are then used to build the conformal scores. For their method, they derive an asymptotic guarantee on coverage.\n    Experiments on a two-state toy MDP, a continuous 2-D benchmark and Mountain-Car show that the method hits the nominal 90 % coverage while a plain DRL quantile interval under-covers.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n- Using an approach based on computing the pseudo-returns make more intuitive sense than prior approaches, as well as using a replay buffer for breaking dependency.\n- Experiments seem to demonstrate the validity of the approach\n- The authors provide an asymptotic coverage analysis for their method\n\n\n**Weaknessess**\n\n- Problematic claims:\n  - The authors in the introduction claim that \"Unobserved Returns\" is a challenge, since full return is not directly observable in the infinite-horizon setting. However, that depends also on the characteristics of the underlying MDP. In some MDPs (e.g., finite and communicating), if one can estimate well the stationary distribution, then it is not necessary to observe the \"entire\" return.\n  - It's not clear why the authors state that \"Model Misspecification\" is a challenge for conformal prediction, since it should be a model-free approach. The overall statement is a bit questionable.\n  - Confusion with DRL in line 125. When the authors write \"DRL-based quantile intervals, referred to as DRL-QR,\", in reality this type of construction is not only limited to DRL, but it's more general than that. It is quite common to use empirical CDFs to build confidence intervals using the lower/upper quantiles (e.g., using bootstrap).\n  - About not using WCP the authors write \"in the RL setting, intermediate transitions along the test trajectory are not available at prediction time, making direct application of WCP infeasible\". But one could use a learned model of the transition, or use an approach base don eq.14 of Foffano et al. 2023\n  - Line 235: not clear why the authors write \" Previous studies have bounded this gap using total variation distance, which235\nfails to capture how different choices of k in k-step rollouts affect the coverage gap. \". Isn't their work the first to use k-step rollouts? Why TV  fails?\n\n- Missing assumptions, notation and possibly not correct equations\n  - The MDP model is missing. The authors write \"we use the standard RL framework\", but what is this standard framework? What are the assumptions on the MDP? Furthemore, also the assumptions on how the trajectories are collected are missing.\n  - The authors define $\\mathcal{P}$ to be the transition function. What is then ${\\rm d}{\\cal P}_0(s)$ in eq 4? Similarly, what do they mean by $P$ in eq 4? They write \" is the distribution of test state\", but a precise definition is missing. Overall, the equation does not seem to be correct.\n  - Off policy setting, lines 171-178: this paragraph just does not seem correct. The goal is to estimate the return distribution from data collected from a different policy, but the section does not seem to explain that (not even uses importance sampling), and I am confused about the intent of the authors here overall.\n  - Theorem 1/2: I don't think this theorem is correct, due to the dubious definition of ${\\rm d}{\\cal P}_0(s)$  (similarly for the calibration) in eq 4. This should depend on the stationary distributions (and we need assumptions on the MDP). Furthermore, what is the limit in $n$ (you mean $N$?) and under which assumptions on the MDP/data/policies we have that the Lebesgue density is bounded by $L$? Also, the boundedness on the importance sampling terms puts assumptions on the data collected, this should be stated more clearly. Also, the statement only holds asymptotically (as the number of trajectories tends to infinity), which affects the strength of the result.\n  - From the proofs is also not clear how the replay buffer should make the samples exchangeable. \n- Experiments are limited to simple examples, and only compare to DRL, but not to other techniques (e.g., Foffano et al 2023). \n- I believe the main challenge in conformal prediction for MDPs is to adequately address the off-policy setting, but as of now,  I believe the paper lacks sufficient clarity/precision to address this point.\n\n- Missing reference on conformal prediction with Markovian data:  Zheng, Frédéric, and Alexandre Proutiere. \"Conformal Predictions under Markovian Data.\" Forty-first International Conference on Machine Learning.\n\nQUESTIONS:\n- Why use the term \"Prediction interval\" instead of \"Confidence interval\"? Furthermore, when writing PI it looks like Policy Improvement.\n- Eq. 2: what if $k$ exceeds the trajectory length?\n- Possible misunderstanding: why it seems like that a small value of $k$ seems to improve the coverage? the second term in the square root will be larger (thm 1)\n- Clarify $N$ and $n$ in the manuscript.\n- Unclear what is ${\\cal I}$ in the algorithm\n\nPossible typos:\n- Line 16: \"As reinforcement learning (RL) are \" -> \"As reinforcement learning (RL) _algorithms_ are\"?\n- Line 93: \"uniform length\" -> \"_constant_ length\"\n- Figure 1, under conformal prediction, (1) is repeated twice\n\nLIMITATIONS:\nLimitations and impact do not seem to be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article considers conformal prediction for policy evaluation in the discounted infinite horizon RL setting. They introduce a new algorithm implementing several trick to address the challenges, namely Pseudo-return construction with Distributional RL to address the unobservable exact returns, Experience replay for the temporal dependence, and Time-Aware Weighted Subsampling for the distribution shift induced. They provide an asymptotic lower bound on coverage as a theoretical guarantee and experiments for empirical validation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\n- Combines ideas stemming from different theories to address all the challenges posed by the problem they consider.\n- The paper is well written, well referenced and all the different ideas are well introduced in the first sections.\n- The experiments are reproducible with the code provided.\n\nWeaknesses :\n- It is not entirely clear how this paper positions itself in the literature of Conformal Prediction for RL. The paragraph \"Conformal Prediction\" in section 1.1 Related Work mentions some other papers but do not explain with them. The sentence \"existing methods are limited by scalability issues, strong modeling assumptions, or the curse of the horizon\" could be worth given more details.\n- In several parts of the paper, such as in Section 1 Introduction line 37, there is a mention of finite-sampling coverage guarantees. Yet to the best of my understanding, this paper only provide asymptotic bounds, as mentioned in the \"Contribution\" paragraph.\n- Minor remarks: \n\t- In the code provided in the Supplementary Material, some headers are written in Chinese. English would be appreciated for better anonymity and understanding of the code.\n\t- In the Algorithms, the sets $\\mathcal I$ are not explicitly defined.\n\t- I don’t understand what the paragraph \"Off-Policy Setting\" in section 3.2 is trying to explain, behind the fact that return distribution can be estimated directly by algorithms such as QTD.\n\nQUESTIONS:\n- Can you detail briefly how this article improves on the current literature ? Mainly:\n  - (1) What are the differences in modeling assumption ?\n  - (2) How does the scalability compares ?\n\n- Algorithms such as QTD are proven to converge (see [1]), with theoretical guarantees on the limit and the true distribution. Isn’t it enough to get asymptotic prediction intervals ? If it isn’t, could you give a brief explanation on why ? If it is, are they are any other reasons to not use it, other than empirical performances ?\n\n[1] Rowland, Mark, et al. \"An analysis of quantile temporal-difference learning.\" Journal of Machine Learning Research 25.163 (2024): 1-47.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a conformal prediction (CP) framework for constructing distribution-free prediction intervals (PIs) for infinite-horizon on-policy and off-policy evaluation in RL. To do so, the authors propose a i) pseudo-return construction using truncated rollouts and learned return distributions (inspired by k-steps TD learning); ii) experience replay (to decorrelate temporally linked transitions) and weighted subsampling (to correct distribution shifts) to restore approximate exchangeability. Moreover, the authors propose coverage guarantees that depend on the Wasserstein distance between true and estimated return distributions. Eventually, empirical evaluations are provided on synthetic and standard environments (Mountain Car, etc.).\n\nSTRENGTHS AND WEAKNESSES:\n1) Strengths\n\nThe research problem is very relevant to the NeurIPS community. Uncertainty quantification in RL is important and few existing methods offer finite sample guarantees. The authors propose an integration of conformal prediction with distributional RL with a plug-and-play method of any return distribution estimator. \n\nThe coverage bounds are derived under approximate exchangeability using Wasserstein distances. These bounds bring nuanced insights of model and weight estimation errors. The difference in the bounds of Theorem 1 and 2 is well explained and the intuition on differences between on and off-policy is well explained. Moreover, the empirical results across on policy and off-policy scenarios show clear improvements in coverage over standard DRL-QR, which are quantile methods. \n\nThe paper is clearly written with very well articulated sections, well defined related work and problem setting. I did not check the novelty w.r.t related work however.\n\n2) Weaknesses\n\nEmpirical evaluation: A few things that to my mind represent weaknesses in the experiment section. The choice of the step size k in pseudo-returns is a little underexplored (experiments run with k=1...5 only). The authors mention aggregating different PIs over k in the conclusion but don’t offer a principled selection method or guidelines. Also, the only comparisons are to DRL-QR and KDE-QR. There is no comparison to recent methods in conformal RL like Foffano et al. (2023), Zhang et al. (2023). Moreover, the settings considered in the experiments are mostly low-dimensional.\n\nQUESTIONS:\nCould you compare against more methods, particularly from recent conformal off-policy evaluation literature and risk-aware RL ?\n\nApart from what you wrote line 228 about Zhang et al 2023, why not implementing Foffano et al. (2023? \n\nCan you further elaborate on adaptive or cross-validated strategies for k, or propose heuristics ?\n\nI am very likely to update my score according to other reviews and answers to all feedbacks.\n\nLIMITATIONS:\nThe authors did not discuss broader impacts of the paper and their answer to that in the checklist (10. Broader impacts) is rather odd: “The manuscript does not contain a Broader-Impact discussion of how autonomous tool-using LLMs might be beneficial or misused.” Is that a copy paste from another NeurIPS submission?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}