Title: Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning theory

online exploration

outcome-based RL

sample efficient RL
Abstract: Reinforcement learning with outcome-based feedback faces a fundamental challenge: when rewards are only observed at trajectory endpoints, how do we assign credit to the right actions? This paper provides the first comprehensive analysis of this problem in online RL with general function approximation. We develop a provably sample-efficient algorithm achieving $\widetilde{O}({C_{\rm cov} H^3}/{\varepsilon^2})$ sample complexity, where $C_{\rm cov}$ is the coverability coefficient of the underlying MDP. By leveraging general function approximation, our approach works effectively in large or infinite state spaces where tabular methods fail, requiring only that value functions and reward functions can be represented by appropriate function classes. Our results also characterize when outcome-based feedback is statistically separated from per-step rewards, revealing an unavoidable exponential separation for certain MDPs. For deterministic MDPs, we show how to eliminate the completeness assumption, dramatically simplifying the algorithm. We further extend our approach to preference-based feedback settings, proving that equivalent statistical efficiency can be achieved even under more limited information. Together, these results constitute a theoretical foundation for understanding the statistical properties of outcome-based reinforcement learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive analysis of outcome-based online reinforcement learning, addressing the fundamental credit assignment challenge when rewards are observed only at the end of a trajectory. The authors develop a provably sample-efficient algorithm (Algorithm 1) for settings with general function approximation, achieving a sample complexity of $\\tilde{O}(C_{cov}H^{3}/\\epsilon^{2})$. A key contribution is the identification of a fundamental exponential separation, revealing that for certain MDPs, outcome-based feedback is unavoidably less efficient than per-step rewards. For the special case of deterministic MDPs, the paper also proposes a significantly simpler and more computationally efficient variant (Algorithm 2) that eliminates the strong completeness assumption required by the primary algorithm. Finally, the framework is extended to preference-based feedback settings, demonstrating that similar statistical efficiency can be achieved.\n\nSTRENGTHS AND WEAKNESSES:\n- strengths:\n\t- Foundational and General Theoretical Framework: This paper provides what appears to be the first comprehensive theoretical analysis for online RL with outcome-based feedback under the challenging general function approximation setting. By developing a provably sample-efficient algorithm, the authors establish a rigorous and foundational contribution that is applicable to complex environments where tabular methods are insufficient.\n\t- Insight on Fundamental Limits: A key strength is the paper's lower bound analysis, which establishes a fundamental exponential separation between outcome-based and per-step feedback. It proves that for certain MDPs, outcome-based feedback is inherently less efficient, requiring exponentially more samples to learn a near-optimal policy. This result clearly defines the statistical barriers of the problem and provides crucial context for when outcome-based learning is tractable.\n\t- Practical Relevance and Extensions: The paper's impact is broadened by its thoughtful extensions and methodological insights. The approach is generalized to handle preference-based feedback, directly connecting the theory to practical Reinforcement Learning from Human Feedback (RLHF) scenarios. For deterministic MDPs, the authors provide a much simpler and more computationally efficient algorithm (Algorithm 2) that eliminates the need for a completeness assumption. Finally, the work offers clear guidance by demonstrating that jointly optimizing the reward and value functions is essential, as simpler decoupled approaches can fail. \n\n\n- weaknesses\n\t- The central contribution, Algorithm 1, hinges on solving a joint optimization problem (Eq. 6) to estimate the value and reward functions. While theoretically elegant, this step poses significant practical challenges. The objective function itself involves a Bellman loss term, $L_D^{BE}(f;R)$, which requires computing an infimum over function class $\\mathcal G$. This creates a nested, max-min-like optimization problem. In the general function approximation setting, where $\\mathcal F$ and $\\mathcal R$ are typically non-convex (e.g., neural networks), solving this outer maximization for a global optimum is generally intractable.\n\nQUESTIONS:\nOn the Computational Feasibility of Algorithm 1: Beyond the special case of deterministic MDPs, could you elaborate on potential pathways to make this optimization computationally feasible, even if it requires approximations? For example, are there specific structural assumptions on the function classes $\\mathcal{F,R,G}$ (beyond determinism) or approximation techniques (e.g., alternating optimization, gradient-based methods for the outer loop) that you believe could make Algorithm 1 practical?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies outcome based (aka trajectory feedback, or aggregate feedback) RL in the general function approximation setup, subject to the standard coverability, realizability and completeness assumptions typically required for sample efficient learning with general function approximation.\n\nIn more detail, the algorithms in the paper receive as input a class of reward functions $\\mathcal R$ and a class of Q functions $\\mathcal F$, and output an $\\epsilon$-optimal policy after roughly $C_{\\rm cov}N/\\epsilon^2$ interaction episodes, where $C_{\\rm cov}$ denotes the coverability coefficient and $N$ the covering number of $\\mathcal R$ and $\\mathcal F$.\nAn additional error term scales linearly with the violation of the Bellman completeness and realizability assumptions. \n\nThe algorithms presented include:\n1. The fundamental trajectory feedback setup, which is computationally inefficient in general.\n2. A simplified algorithm for the case of deterministic dynamics, which is efficient assuming oracle access to $\\mathcal F$.\n3. An extension of the fundamental setup algorithm to the case of preference feedback.\n\nFinally, a lower bound is established that demonstrates standard semi bandit feedback is easier than trajectory feedback in the general function approximation setup (in the sense that the former admits sample efficient algorithms, while the latter does not).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* This work is the first to study trajectory feedback in the general function approximation setup. Prior works only study the tabular and linear setups.\n* Three algorithms of independent interest are studied.\n* The paper is well written with clear presentation, and is easy to follow.\n\n### Weaknesses\n* Nothing stands out.\n\n### Minor comments\n- Line 39: \"where the reward function is assumed to be well structured\" --- some of the works mention consider the tabular setup, and do not make any particular assumptions on the reward shape. What exactly do you mean by \"well structured\" or \"well behaved\"? If I understand correctly the distinguishing feature of the present paper is that it obtains rates independent of $S$. But, this does not relate to well behavedness of the reward function. Works that study the linear setup such as Cassel et al. 24 consider linear MDPs, which is a much stronger assumption than just linear reward.\n- Line 126: \"Bellman operator\" might be a better heading?\n- Line 175: What is $d$?\n- Line 256: \"which is widely used on RLHF literature\" (on $\\to$ in)\n- Line 300: \"Is outcome-based samples are statistically equivalent\"\n\nQUESTIONS:\n* What is the reason for employing the reference policy in the exploration phase of Alg1 (similarly Alg3)? As you explained, you make no assumption regarding the distribution of actions it chooses. If this is the case, why not just stick with $\\pi^{(t)}$ for the entire episode, in favor of simplicity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online reinforcement learning with trajectory-wise reward feedback. While previous works have focused on structured rewards such as linear functions, this paper considers general function approximation. Under standard realizability and completeness assumptions, the authors propose an algorithm and establish an $\\tilde{O}(H^3/\\epsilon^2)$ sample complexity guarantee. They also provide theoretical analysis in the setting of preference-based learning. Finally, the authors present an example MDP in which all outcome-based reinforcement learning algorithms incur exponential sample complexity.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths: \n 1. This work is the first to provide a theoretical analysis of online reinforcement learning with outcome-based reward feedback under general function approximation.\n 2. The authors present a counterexample MDP in which per-step reward RL algorithms are sample-efficient, whereas outcome-based algorithms suffer from exponential sample complexity. This highlights a fundamental difference between outcome-based MDPs and standard MDPs.\n\n### Weakness\n1. In the loop between steps 4 and 7 of the Algorithm 1, the policy $\\pi \\circ h\\,\\pi_{ref}$ is executed once per iteration to collect a trajectory. This means that each iteration yields $H$ trajectories in total, and the entire algorithm generates $TH$ trajectories over $T$ iterations. This differs from standard online RL algorithms, which usually collect only one trajectory per iteration. I haven't carefully checked the proof details, so it's unclear whether a typo error exists at this point. However, it seems that the theoretical guarantees rely on collecting $H$ times more trajectory data. If those additional trajectories were omitted, the sample complexity of the algorithm could increase accordingly.\n\n2. Some notations are not explained, such as $d_h^{\\pi}$ and $\\mu_h$, although they are standard notations in offline learning.\n\n3. In the definition of Bellman completeness, the Bellman operator is applied to vector-valued functions $f$ ($f_1,...,f_H$), whereas in Assumption 2, it is applied to scalar-valued functions $f_{h+1}$.\n\nQUESTIONS:\n1. Could the authors clarify whether the algorithm collects $H$ trajectories per iteration by executing the policy $\\pi \\circ h\\,\\pi_{ref}$?\n2. In offline learning, $\\mu$ usually refers to the data distribution. In the online setting, however, data is generated interactively over time. Could the authors clarify what $\\mu$ specifically represents in online RL?\n3. In the lower bound proof, the ReLU bandit example (Section 6.1.4) demonstrates a linear sample complexity in $T$. Is this because the example violates Assumption 1 and Assumption 2? Clarification on whether the lower bound arises from the failure of these assumptions would be appreciated.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle reinforcement learning with outcome-based feedback in the case of general function approximation. Under the assumptions of coverability and completeness, they show that an optimistic algorithm that models the unknown reward function and balances the estimated value of the Q-function, the TD error, and the reward model error can achieve a sample complexity of $T \\geq O(C_{\\text{cov}} N(1/T) H^2/\\epsilon^2)$ episodes. A simplified algorithm is presented in the case of deterministic MDPs that only requires realizability instead of completeness. They extend the first algorithm to preference-based RL, and conclude by providing a hard instance, showing that learning from outcome-based feedback can be exponentially harder than process reward feedback when coverability is not satisfied but the low Bellman eluder dimension assumption is.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. **Strong theoretical contribution in a highly salient setting**.\n- The paper addresses a highly salient setting with a thorough mathematical analysis, providing trajectory-only feedback theory with matching lower bounds for general function approximation. The proofs appear to be sound.\n- The contributions are varied and thorough, consisting of a statistically efficient model-free algorithm, a simplified algorithm for deterministic MDPs, extension to preference-based feedback, and a lower bound that completes the picture -- while coverability is sufficient for RL with outcome-based feedback consisting of the sum of rewards, the low Bellman eluder dimension assumption is not. \n2. **Original algorithms**.\n- The authors provide two original algorithms for RL theory with general function approximation, that very refreshingly are not derivatives of GOLF. The authors also show that joint optimization of the reward model and Bellman loss is necessary as well.\n- In particular, it is very nice that the authors adopt the MEX approach, instead of maintaining per-step confidence sets as in GOLF, to achieve sufficient optimism in Algorithm 1. \n3. **Clear writing and presentation**.\n- The paper is well-written and very easy to understand. \n\n## Weaknesses\n1. **Claim of first comprehensive analysis of outcome-based feedback in online RL with general function approximation**.\n- Relevant work exists that the authors do not cite or compare their results to. Kausik et al. (2024) view cardinal (good/bad) and dueling (preferential) feedback in RLHF as partially-observed reward states, and provide guarantees for model-based (in the tabular setting) and model-free (a GOLF derivative in the general function approximation setting) RL. The latter uses a history-aware analogue of the Bellman-eluder dimension. In my opinion, the setting is highly similar, even though the approach may be different, and it may not be appropriate to say that this is the first comprehensive analysis of the problem in online RL with general function approximation.\n2. **Outcome model structure**. \n- Further, the claim that this gives a comprehensive analysis of RL with outcome-based feedback may not be appropriate as well. In my view, the contribution is not as broad as the paper claims for the below reason. \n- The authors assume that the outcome-based feedback observed is given by the sum of the rewards in the trajectory. This is not necessarily the case in practice (for instance, one can easily imagine a threshold outcome that yields 1 if the sum of rewards is high enough and 0 otherwise), and is an assumption that other prior work (see Kausik et al. (2024) for an example) does not make.\n- In my view, it is quite possible that this assumption is the reason why this paper achieves stronger results than Kausik et al. (2024) in the case where no intermediate feedback is observed in their setting. To be more specific, the fact that the outcome-based feedback is assumed to be the sum of the rewards in this paper may be the very reason why the credit assignment problem is solvable in this setting (under coverability) but not that of Kausik et al. (2024). \n- However, this is not stated as an assumption, but is presented as part of the setting itself. I do not think this is good practice. I am of the opinion that this should be highlighted and presented as an assumption, due to the above remarks. \n- This ultimately limits the scope of the paper, and one can conclude that this paper therefore may only tackle a special case of outcome-based feedback where the outcome is the sum of the rewards. \n\n## References\n1. Cheng et al. (2022), Adversarially Trained Actor Critic for Offline Reinforcement Learning\n2. Kausik et al. (2024), A Theoretical Framework for Partially Observed Reward-States in RLHF\n\nQUESTIONS:\n1. If the assumption that the outcome-based feedback is given by the sum of rewards is relaxed, how do you think the results would change?\n2. How does your paper compare to Kausik et al. (2024)?\n\nI was going to award this paper a score of 5 (Accept), before noticing the concerns outlined above that affect the significance and originality of the paper. I will be very happy to increase my score if my concerns are addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an algorithm with sample complexity bound for outcome-based online RL. The algorithm can handle general function classes for approximation. The algorithmic framework is further simplified for deterministic MDPs and extended to preference-based RL to demonstrate the generic nature of the approach. Finally, lower bounds are derived for both the per-step reward-based and outcome-based RL settings. This shows exponential deviation among hardness of these two frameworks in some well-crafted MDPs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem setup is interesting and timely.\n2. The algorithm extends the well-studied fitted Q-iteration to this setting, which is nice approach to bridge the per-step reward and outcome-based RL setups.\n3. The analysis of preference-based RL provides a motivating instantiation of the outcome-based RL in the context of modern RLHF applications.\n4. The lower bounds demonstrating the different between per-step and outcome-based RL are insightful for future research.\n\nWeakness:\n1. Limitations of the analysis or the framework are not discussed.\n2. Regret is never introduced in the paper. Also the notation $R$ is confusing with reward (e.g. Eq. 11, 13 etc.).\n3. The description of the comparator class is almost non-existent. I understand that the idea comes from another work but explaining how to choose it exactly, specially in preference-based setting, is necessary for clarity.\n4. The proposed lower bound on T to satisfy the error bounds in Eq 7, 10 and 14 seems intangible to me as both sides depend on T, and I do not see how the left side scales with T for some useful function classes. Can you explain this?\n\nQUESTIONS:\n1. Can you provide some examples of $N(\\alpha)$ and $N_T$ for some function classes? Present description in line 175 is vague to me.\n2. The lower bounds are interesting for future research but no detail or intuition on the \"hard\" MDP construction is provided. Can you summarise why the specific two-layer MDP constructed exposes this gap? What is the bottleneck?\n3. How does the covering number scales with dimensions and structures of policy class? It can go exponential in problem parameters, no? Where can we get a better polynomial type bounds on MDP parameters using this complexity measure? Also do we have to take the min over all base measures $\\mu$? If yes, explain the intuition?\n4. The formal proof for Theorem 3 of preference based optimization passes through proposition 22 and thus, proposition 8, which tries to control the Hellinger distance between the Bernoulli distributions over trajectories induced by different reward models. Why it has to be the Hellinger distance between them? Why not TV or any other metric? Can you explain this choice?\n5. Please address the issues in weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}