Title: On the Sample Complexity Bounds of Bilevel Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bilevel Reinforcement Learning

Sample Complexity

Gradient Descent

Bilevel Optimization
Abstract: Bilevel reinforcement learning (BRL) has emerged as a powerful framework for aligning generative models, yet its theoretical foundations, especially sample complexity bounds, remain relatively underexplored. In this work, we present the first sample complexity bound for BRL, establishing a rate of $\tilde{\mathcal{O}}(\epsilon^{-3})$ in continuous state-action spaces. Traditional MDP analysis techniques do not extend to BRL due to its nested structure and non-convex lower-level problems. We overcome these challenges by leveraging the Polyak-Łojasiewicz (PL) condition and the MDP structure to obtain closed-form gradients, enabling tight sample complexity analysis. Our analysis also extends to general bi-level optimization settings with non-convex lower levels, where we achieve state-of-the-art sample complexity results of $\tilde{\mathcal{O}}(\epsilon^{-3})$ improving upon existing bounds of  $\tilde{\mathcal{O}}(\epsilon^{-6})$. Additionally, we address the computational bottleneck of hypergradient estimation by proposing a fully first-order, Hessian-free algorithm suitable for large-scale problems.


Peer reviews:
comments: SUMMARY:
This paper investigates bilevel RL from a theoretical lens. Their algorithm uses penalty based methods and they prove that their algorithm admits a sample complexity of $O(\epsilon^{-3})$.

STRENGTHS AND WEAKNESSES:
Strengths:

They the first sample complexity bound for bilevel RL.

They improve the sample complexity for general bilevel optimization from $\epsilon^{-6}$ to $\epsilon^{-3}$.


Weaknesses:

They do not prove a lower bound to show that their upper bound is tight.

What are some examples of $G(\theta, \lambda)$? While Assumption 3 is present in [1], it seems this is a strong assumption and not reasonable in RL. Line 133-134 states ``However, such assumptions (unbiased estimates of the upper and lower level objectives) do not hold in BRL, where gradient estimates are inherently biased due to the underlying MDP dynamics.'', which seems to contradict Assumption 3.

QUESTIONS:
What is the purpose of the KL term in the reward function? This seems like a very contrived way of tying BRL to RLHF. Do the results still hold without the KL regularization term?

Can you prove a lower bound in this setting?


I think these may be typos, but disregard if they are not:
- Page 9 line 296, should it be $\eps^{-3}$ instead of $\eps^{-4}$?
- It seems some of the references are duplicated, e.g. 1 and 2, 21 and 22, etc.

LIMITATIONS:
The limitations are discussed in the paper.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-26

comments: SUMMARY:
This paper presents the first finite-sample analysis of BRL, establishing a sample complexity of $\mathcal{O}(\epsilon^{-3})$ in continuous state-action spaces. The contribution is both clear and sound.

STRENGTHS AND WEAKNESSES:
**Strengths**
- This paper is well-written and easy to follow.
- The contribution is clear and significant.
- Establishing sample complexity bounds for continuous state-action spaces is an important advancement.

**Weaknesses**
- Assumption 1.3 requires Lipschitz and smooth Hessians, which is a strong condition.
- While the authors emphasize the contribution of extending from the tabular setting to continuous spaces, it remains unclear what specific technique was developed to enable this extension.
- There are a few typos in the paper. For example, in the conclusion, the sample complexity should be stated as $\mathcal{O}(\epsilon^{-3})$ rather than $\mathcal{O}(\epsilon^{-4})$.

QUESTIONS:
My primary concern pertains to the technical contribution of this work.

Could you elaborate further on Assumption 1.3? Why is it necessary, and has it appeared in prior work? If not, please clarify how this new assumption relates to existing assumptions in the literature.

Furthermore, what limitations in prior work prevented the establishment of sample complexity bounds for continuous state-action spaces? What specific techniques were developed in this paper to overcome these challenges and enable the extension?

LIMITATIONS:
This work requires strong assumptions.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the sample complexity of bilevel RL. In bilevel RL, the goal is to minimize an objective function $G(\\phi, \\lambda^\\*)$, where $\\phi$ is the parameter of the reward model, and $\\lambda^*$ is the parameter of the optimal policy induced by the reward model represented by $\\phi$. Prior work on this problem requires (approximately) evaluating the Hessian/Jacobian matrices and Hessian / Jacobian vector products, or focuses on the tabular setting. In this paper, the authors propose a new first-order algorithm for bilevel RL that works for continuous state-action spaces,  and provide a sample complexity results for their algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThis paper gives the first sample complexity result for bilevel RL. Moreover, the algorithm could handle continuous space. Moreover, even for the standard bilevel optimization setup, the proposed algorithm yields improved sample complexity. \n\nWeaknesses: \n1. I would suggest adding a more detailed introduction on bilevel RL. In particular, I would suggest adding more discussion on the connection between bilevel RL and other topics in RL (like inverse RL and RLHF). Currently, bilevel RL is defined in a very abstract manner, and it would be hard for readers (especially those who haven't worked on bilevel RL) to understand its connection to other RL problems.\n2. A more details comparison with [17], especially in terms of technical aspects, should be added. In particular, it is clear that the result in [17] applies only to bilevel optimization problems without an MDP structure, where for bilevel RL, gradient estimates are inherently biased. However, it is unclear to me why the analysis in the paper, when specialized to the standard bilevel optimization setup, yields improved sample complexity bounds. What are the new technical ideas that allow such an improvement?\n\nQUESTIONS:\nSee Weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies first-order, Hessian-free algorithm for bilevel reinforcement learning. The authors consider two settings, with and without access to unbiased gradients, and for both the settings they establish 1/\\epsilon^{-4} sample complexity. All analysis relies on a PL condition for the lower-level objective and Lipschitz–smoothness of gradients and Hessians.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n-\tThe authors claim to have improved the sample complexity of Bilevel optimization in the case of Non-convex LL and without second order operations, from already known \\epsilon^{-6} to \\epsilon^{-3}.\nWeakness:\n-\tThe convergence results are only to the stationary point. There are no discussions on what is necessary to guarantee global convergence of this algorithm.\n-\tThe analysis is only for a fixed value of \\sigma and step size \\eta, B, N, and T. Hence, the algorithm does not have asymptotic convergence. \n-\tThe final result is only in terms of \\epsilon, and constants which depend on the problem structure are missing in the statement of the theorem. These constants can be specifically important when studying settings like RLHF, where the underlying structure (state, action, and 1-\\gamma dependency) is important.\n-\tSatisfying PL condition for RL problem is not necessarily true. Can you justify this condition for RL setting? Can you provide concrete function approximation settings where all the assumption are satisfied?\n-\tThere are no implementations justifying the sample complexity that is achieved in the theory.\n\nQUESTIONS:\n-\tWhat are s_i , a_i in Eq. (1)?\n-\tIn both Theorems 1 and 2 you assume Assumption 2, which specifies an upper bound on the representation power of the neural network, which models the Q-function. It is not clear for me why the approximation error appears in Theorem 1, but not Theorem 2. Can you elaborate?\n-\tWhat is the implications of your results for the RLHF theory? What is the query complexity of RLHF when studied under the umbrella of Bilevel optimization? How is the query complexity of RLHF affected by the structure of the underlying MDP?\n-\tIn the conclusion section the authors suggest a \\epsilon^{-4} sample complexity, which is in contrast to the rest of the paper. Can you please clarify?\n-\tAre there any known lower bounds for bilevel optimization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}