Title: A theoretical case-study of Scalable Oversight in Hierarchical Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Scalable oversight

theoretical hierarchical reinforcement learning
Abstract: A key source of complexity in next-generation AI models is the size of model outputs, making it time-consuming to parse and provide reliable feedback on. To ensure such models are aligned, we will need to bolster our understanding of scalable oversight and how to scale up human feedback. To this end, we study the challenges of scalable oversight in the context of goal-conditioned hierarchical reinforcement learning. Hierarchical structure is a promising entrypoint into studying how to scale up human feedback, which in this work we assume can only be provided for model outputs below a threshold size. In the cardinal feedback setting, we develop an apt sub-MDP reward and algorithm that allows us to acquire and scale up low-level feedback for learning with sublinear regret. In the ordinal feedback setting, we show the necessity of both high- and low-level feedback, and develop a hierarchical experimental design algorithm that efficiently acquires both types of feedback for learning. Altogether, our work aims to consolidate the foundations of scalable oversight, formalizing and studying the various challenges thereof.

A theoretical case-study of Scalable Oversight in Hierarchical Reinforcement Learning

Abstract

A key source of complexity in next-generation AI models is the size of model outputs, making it time-consuming to parse and provide reliable feedback on. To ensure such models are aligned, we will need to bolster our understanding of scalable oversight and how to scale up human feedback. To this end, we study the challenges of scalable oversight in the context of goal-conditioned hierarchical reinforcement learning. Hierarchical structure is a promising entrypoint into studying how to scale up human feedback, which in this work we assume can only be provided for model outputs below a threshold size. In the cardinal feedback setting, we develop an apt sub-MDP reward and algorithm that allows us to acquire and scale up low-level feedback for learning with sublinear regret. In the ordinal feedback setting, we show the necessity of both high- and low-level feedback, and develop a hierarchical experimental design algorithm that efficiently acquires both types of feedback for learning. Altogether, our work aims to consolidate the foundations of scalable oversight, formalizing and studying the various challenges thereof.

1 Introduction

Next-generation AI models are poised to produce sophisticated outputs such as long-form texts and videos, and execute complex tasks as agents. To build these AIs responsibly, we need to better our understanding of scalable oversight: the ability to provide scalable human feedback to these complex models [2, 8, 15, 5]. An immediate, key challenge to overcome is the size of model outputs, making it time-consuming for humans to parse and provide reliable feedback on, even with AI-assistance [24, 27, 23]. To this end, in this work, we consider human labelers with bounded processing ability such that accurate feedback can only be provided for outputs below some threshold size. We are interested in answering the question:

How can we scale this limited feedback to supervise a model with outputs larger than this limit?

Verily, this task is difficult without further assumptions. If the model output can only be assessed in its entirety, it is impossible for humans to provide reliable feedback. Thus, we investigate a natural setup that gives us hope to overcome the limitation in feedback - when model outputs have hierarchical structure. Hierarchical structure exists in many high-dimensional outputs of interest, including long-form texts (books made up of chapters), videos (movies made up of scenes) and code (main functions made up of helper functions). Indeed, it reflects the way we humans produce many of our most complex creations.

To formalize the setting, we study scalable oversight in a goal-conditioned hierarchical reinforcement learning (HRL) setup. Goal-oriented RL is a popular approach that has seen sizable success in leveraging state space structure to overcome sparse rewards over long horizons [16, 17, 10]. Our aim in this paper differs in using this as an entry-point into understanding how to scale up bounded human feedback, and formalizing the conceptual/technical challenges thereof. It turns out that one known advantage of HRL, besides more efficient exploration and efficient credit assignment, is the ability to enable scalable oversight.

1.1 Preliminaries

We consider a finite-horizon, Markov Decision Process (MDP) M = ⟨ S, A, P, r, s 1 , H ⟩ , with finite state space S , finite action space A , transition probability P : S × A → ∆( S ) , reward r ( s, a ) : S × A → [0 , 1] and finite horizon H . The learner interacts with M starting at state s 1 and the episode ends after H = H h H l time-steps. In this work, policies are trained using human feedback. And so, we assume that a human supervisor is needed to evaluate and provide reward r for trajectories τ ∼ π, P generated by policy π : S → A .

Accompanying Example: Consider the task of learning to generate a long-form, argumentation essay. Providing feedback to an end-to-end policy is difficult as labelers would have to read through entire essays to rate the outputs, after which it may be difficult still to assign a single rating to the entire essay. A tractable alternative is to learn a hierarchical model, with a higher-level policy that generates the essay arguments (goals), and lower-level policies that flesh out these points (realize these goals). It would then be easier for the labeler to rate the shorter-length essay content, and also individual fleshed out arguments, in order to generate a rating on the whole. This approach also mirrors existing rubrics for scoring essays [1].

Bounded Feedback: To formalize the difficulty of human supervisors assessing long-form outputs, we assume that reliable feedback can only be provided for trajectories of length at most max( H h , H l ) . In particular, this means that for the global policy π : S → A , it is infeasible to obtain reliable feedback for its trajectory τ ∼ π, P , as | τ | = H h H l . This thus motivates hierarchical learning, which makes possible the acquisition of reliable feedback in spite of bounded human supervision.

1.1.1 Goal-conditioned HRL

Since we are unable to learn a single, monolithic policy, our goal instead will be to learn a set of smaller policies that make up a hierarchical policy. This set consists of a high-level policy π h : S → ∆( A h ) (outputs a high-level action a h at state s ∈ S ), and a set of low-/sub-policies π l s,a h : S l s,a h → ∆( A ) , where S l s,a h ⊆ S is the set of all states reachable from s after H l steps.

In a nutshell, the high-level policy designates goals by choosing high level actions. The low-level policies then aim to realize these goals, while also trying to achieve a high intermediate return. Importantly, both such policies act over a shorter horizon of at most max( H h , H l ) , making it amenable for human supervisors to evaluate.

Goal Function: in the goal-conditioned HRL setting, we assume access to a function g mapping high-level action a h at state s to a goal-state g ( s, a h ) ∈ S l s,a h . For example, s is the current content of the essay, a h is the action (in natural language) 'add an argument using X' and g ( s, a h ) is the content of the essay with the 'argument using X' included.

Goal-conditioned sub-MDP: Given a high level action a h at state s , this defines the sub-MDP M ( s, a h ) , which has state space S l s,a h ⊆ S , action space A (action space of the original M ), transition probabilities P restricted to S l s,a h , starting state s and finite horizon H l . The sub-MDP reward r l will be defined later and as we will see, an apt choice is important for achieving sublinear regret.

High-level MDP: Given a set of low-level policies, π h may be thought of as operating over a high-level MDP with state space S , action space A h , starting state s 1 and finite horizon H h . Importantly, the high-level transition P ′ of this MDP is a function of the current set of low-level policies Pr ′ ( s ′ | s, a h ) = Pr( s π s,a h H l = s ′ ) , which denotes the distribution over the (final) H l th state that π l s,a h reaches. Furthermore, the high-level reward r h ( s, a h ) = E s j ,a j ∼ π s,a h ,P [ ∑ H l j =1 r ( s j , a j ) | s 1 = s ]

corresponds to the intermediate return of sub-policy π s,a h in M ( s, a h ) . Altogether, this gives rise to a key complication in hierarchical learning. This is that both the transitions and rewards in the high-level MDP are non-stationary, as sub-policies π s,a h are updated over time.

Interaction Protocol: At each time-step t , the high level policy chooses a high level action a t based on current state s t . This defines the sub-goal state g ( s t , a t ) , along with the corresponding sub-MDP M ( s t , a t ) with finite-horizon H l , in which sub-policy π l s t ,a t is used to try to achieve the goal. The overall return of the high level policy π h and low-level policies { π l s,a } s,a ∈ S × A h is the sum of intermediate returns r ( π l s t ,a t ) incurred:

Instantiation in the example: returning to our example, for a cogent essay, the arguments need to be logically related and built on top of each other. This results in a sequential decision making problem corresponding to the one solved by the high level policy π h . Given an argument g ( s, a h ) to flesh out, the low level policy π l s,a h generates up to H l words, whose content aims to realize this argument. Additionally, low-level policies can incur intermediate rewards (return) for eloquent diction and clear structure when fleshing out the argument, all of which add to the essay's persuasiveness.

1.1.2 Learning Task

Our aim is to learn a hierarchical policy, whose return is close to that of the optimal, goal-reaching hierarchical policy, which we define as follows. For brevity, from this point on, we will use a h and a interchangeably to denote high level action.

Assumption 1 (Goal-Reachability) . In every sub-MDP M ( s, a ) , there exists a policy that achieves the goal g ( s, a ) almost surely. That is,there exists at least one policy π ∈ Π s,a in the policy class Π s,a such that Pr( s π H l = g ( s, a )) = 1 .

In other words, we assume that the goal function g is well-defined in that it designates goals that are feasible to reach from the starting state s (e.g. the argument can be successfully fleshed out in H l words or less given the essay content thus far). To motivate this assumption, we note there that there are already many settings of interest, where we have prior knowledge of a good goal function. This is because we humans have often (and successfully) taken the hierarchical approach to build up to and produce these long-form creations. So we know what are good goals to set e.g. we write essays by first writing an outline of arguments, then expanding out each point in the outline. Indeed, this approach of explicitly encoding prior knowledge in the hierarchical learning algorithm has been done in both HRL literature (e.g. we know apriori mazes has hierarchical structure in that it consists of rooms [22]) and scalable oversight literature (e.g. we know that books consists of chapters [27]).

With this assumption, there exists constant C large enough such that if π ∈ arg max π ∈ Π s,a r ( π ) + C · Pr( s π H l = g ( s, a )) , then π is goal-reaching and Pr( s π H l = g ( s, a )) = 1 .

Definition 1. Define optimal low-level policies as π ∗ s,a ∈ arg max π ∈ Π s,a r ( π ) + C · Pr( s π H l = g ( s, a )) . Define optimal high-level policy as π ∗ = arg max π ∈ Π h V π,π ∗ s,a ( s 1 ) .

In words, π ∗ s,a has the highest intermediate return of all goal-reaching policies. Now let π ∗ be the optimal high-level policy fixing each sub-MDP policy to be π ∗ s,a .

Learning Goal: We wish to learn a set of near-optimal high- and low-level polices ( π, { π s,a } ) such that: V π ∗ ,π ∗ s,a ( s 1 ) -V π,π s,a ≤ ϵ .

1.2 Takeaways

The broad takeaway from this paper is that hierarchical structure, if it exists, can be provably used to scale up limited human supervision. That is:

Hierarchical learning can enable scalable oversight.

On a more technical level, this paper studies the challenge of training a set of (instead of a single) policies that work together to form the hierarchical policy. This is the more complicated problem we turn to solve when it is not feasible to train a monolithic policy, due to bounded human supervision. We thus consider learning in the goal-conditioned HRL setup, under both cardinal and ordinal feedback. A key insight that applies in both settings is that an apt sub-MDP reward design (a suitable penalty for non-goal reachability) is needed for bounding regret and controlling the exit state of learned low-level policies. This is so that learned sub-policies do not land at bad states with sizable probability. Doing so would then allow one to compose low-level policies together, and stabilize high-level policy learning in the high-level MDP. More specific takeaways for both types of feedback are as follows:

Under cardinal feedback, we develop a novel no-regret learning, Algorithm 1, that jointly learns a high-level and a set of low-level policies. Notably, Algorithm 1 only requires low-level feedback. Our main structural result in this setting is that hierarhical RL reduces to multi-task, sub-MDP regret minimization. Thus, the regret from the low-level accumulates additively (instead of say multiplicatively) as speculated upon in [15].

Under ordinal feedback, we develop a novel hierarchical experiment-design Algorithm 2, building off of existing work on experiment design in preferenced-based RL [29]. A key observation is that in the ordinal case, low-level feedback may not be sufficient and highlevel feedback may be needed. This introduces complications in human supervision, as the high-level feedback would need to account for the current performance of sub-policies. To this end, we study two natural forms of feedback, requiring differing cognitive loads on the human supervisor. Through the experiment design algorithm we develop, we then analyze the differing sample complexity under the two types of feedback. Finally, we show that high-level feedback should not be used if low-level feedback is sufficient and one form of feedback, with higher cognitive load, leads to better sample complexity.

2 Related Works

HRL under cardinal rewards: There has been sizable interest in understanding of the sample complexity of HRL algorithms, which to our knowledge has thus focused on learning from cardinal rewards. On this subject, the two closest papers to that of ours are [22] and [25]. [22] studies goal-conditioned HRL with the key result being a sample complexity lower bound associated with a given hierarchical decomposition. On the upper-bound side, an algorithm (SHQL) is presented, albeit without theoretical guarantees. By contrast, our work presents a learning algorithm with provable guarantees, and further shows that learning in goal-conditioned HRL reduces to multi-task, sub-MDP regret minimization.

[25] studies HRL under the options framework, providing a model-based, Bayesian algorithm with access to a prior distribution over MDPs that is updated over time. It does not adaptively learn subpolicies based on observed returns, computing instead an option for every exit-profile and equivalence class at each time during model-based planning. By contrast, our work does not assume knowledge of the prior nor ability to update posteriors, and does adaptively explore sub-MDPs via the UCB principle. Additionally, [25] demonstrate that when the size of the set of exit ('bottleneck') states is small, learning is efficient. Our work shed further light on this insight by showing that under a suitable sub-MDP reward, we can induce a small set of exit states with high probability . Thus, even though the total number of possible exit-states may be high, this condition is sufficient for learning with sublinear-regret.

RL under ordinal rewards: There has also been considerable interest in bandits/RL from preferences [26, 28, 18, 14, 30, 29]. Following the demonstrated success of RLHF [9, 31, 19, 4], there has been great interest in studying offline RL from preference feedback, and particularly experiment design for enhanced sample efficiency [30, 29]. Due to the success of RLHF in alignment, we also consider studying scalable oversight in this setup. Please see the Appendix A for further discussions on scalable oversight and goal-conditioned RL.

3 Learning from Cardinal Feedback

We begin by considering the setting when feedback is in the form of cardinal rewards. As noted before, in HRL, the high-level policy performance is dependent on the low-level policies performance. Thus, a naive approach is to learn near-optimal sub-policies in every sub-MDP M ( s, a ) , and then learn a high-level policy on top. However, a more sample efficient approach is to strategically explore sub-MDPs, and discover sub-policies with high intermediate returns in tandem with a high level policy that visits these 'good' sub-MDPs. Please see the Appendix C for all the proofs. Note that in what follows, for brevity, theoretical statements will contain the phrase 'with high probability' and the appendix will contain proofs that formalize this guarantee.

3.1 Sub-MDP reward design for Hier-UCB-VI

We are interested in adaptively learning the necessary sub-policies (the useful goals to achieve) and the associated high level policy that invokes these sub-policies. It is natural then to adopt an upper confidence bound approach and construct an exploration bonus that tracks the best/unexplored sub-MDPs. To this end, we develop an adaptation of the classic UCB-VI algorithm [3]. We highlight two key ingredients needed to construct the Hier-UCB-VI Algorithm 1.

Tradeoffs in sub-MDP reward design: Learned sub-policies in HRL have to tradeoff between two objectives. One is high intermediate returns r ( π s,a ) . The other is that exit-state; sub-policies should not land at 'bad' states, as even if the intermediate return is high, V ( s π s,a H l ) ≈ 0 means the return from hereon out (and hence the overall return) will be low. Thus, in sub-policy learning, we also need to consider the goodness of the exit-state. But how can we incentivize sub-policies to land at 'good' states without being able to calculate V ? Luckily, in the goal-conditioned setting, there is a natural answer for a 'good' exit-state: g ( s, a ) .

To operationalize this, we design a sub-MDP reward that trades-off between intermediate sub-MDP return and goal-reachability. In sub-MDP M ( s, a ) , at time-step h , sub-MDP reward r l,h ( s ′ , a ′ ) = r ( s ′ , a ′ ) + κ 1 ( h = H l ∧ s ′ = g ( s, a )) . Crucially, here we set the weighting κ = max(2 H h H l , C ) , which corresponds to an upper bound on the regret should we not reach the goal-state.

UCB construction: Next, we wish to obtain an UCB for r ( π ∗ s,a ) . Our main observation is that by using a no-regret subroutine for learning in M ( s, a ) , the regret guarantee directly translates to a UCB. Due to our choice of sub-MDP reward r l , the UCB includes a penalty on non-goal reachability.

Lemma 1 (UCB implied by sub-MDP regret) . Let UB ( R n ( s, a )) be an upper bound on sub-MDP M ( s, a ) 's cumulative regret after n rounds. Define β = ( κ + H l )2 log( |C ( S,A ) | H h K δ ) and bonus,

̸

Then, the average reward plus bonus ¯ r n ( s, a ) + b s,a r ( n ) is an UCB for r ( π ∗ s,a ) with high probability.

High-level MDP transition stabilization: An additional benefit of incentivizing goal-reachability is that we know the idealized transition probability in the high-level MDP. As mentioned before, another key difficulty with HRL is that the empirically estimated transitions in the high-level MDP drifts over time. In our algorithm, the key stabilization approach is to avoid estimation and set the transition in the upper bound Q i to be the idealized transition ( g ( s, a ) w.p. 1 ). This allows us to prove our regret guarantee as described below.

3.2 Regret Analysis of Hier-UCB-VI

We start with a definition on clusters of equivalent sub-MDPs. Let there be C ( S, A h ) such clusters. In the most general setting, it is not known apriori if there is any shared structure, in which case each sub-MDP will simply be its own cluster.

Definition 2 (Equivalent sub-MDPs [25]) . Two subMDPs M ( s, a ) and M ( s ′ , a ′ ) are equivalent if there is a bijection F between state space, and through F , the subMDPs have the same transition probabilities and rewards.

Our main structural result is that HRL regret decomposes to multi-task, sub-MDP regret in the cardinal reward setting. This has the implication that only low-level feedback is needed for regret

Algorithm 1 Hierarchical-UCB-VI (Hier-UCB-VI)

1: Initialize D = ∅ , Q H h +1 ( s, a ) = H h H l ∀ s, a , V H h +1 = 0 , κ = max( C, 2 H h H l )

2: for episode k = 1 , ..., K do

3: for timestep i = H h , ..., 1 do

4:

for

(

s, a

)

∈

S

5:

if

(

s, a

)

∈

×

D

A

h

do then

6: Update UCB: UB ( r π ∗ ( s, a )) = ¯ r N k,h ( s,a ) ( s, a ) + b s,a r ( N k,h ( s, a )) 7: Set:

8: for s ∈ S do

10: for time step h = 1 , ..., H h do

12: Traverse sub-MDP M ( s k h , a k h ) with current sub-policy π N k,h s k h ,a k h and transition to s k h +1 , human supervisor provides low-level rewards of the lengthH l roll-out of π N h,k k k .

s h ,a h

13: Feed low-level rewards into no-regret RL algorithm A for sub-MDP M ( s k h , a k h ) . Set the sum of the low-level rewards (the intermediate return of π N h,k s k h ,a k h in M ( s k h , a k h ) ) as the high-level reward r ( s k h , a k h ) = r ( π N h,k s k h ,a k h )

14: Add to dataset D = D ∪ { ( h, s k h , a k h , r ( s k h , a k h ) }

minimization in the cardinal reward case, which as we will see in the ordinal reward case will not always be true.

Theorem 1 (HRL regret minimization reduces to multi-task, sub-MDP regret minimization) . Let UB ( R N K,H h ( s,a ) ) be an upper bound on sub-MDP M ( s, a ) 's cumulative regret over N K,H h ( s, a ) visits:

Proof Sketch. We describe the key regret decomposition. After some manipulation, the regret may decompose into the following form, ∑ K k =1 V k 1 ( s 1 ) -V π k 1 ( s 1 ) ≤ ∑ K k =1 ∑ H h h =1 ρ k h + γ k h + σ k h + ζ k h , which may be parsed as follows.

ρ k h = UB ( r π ∗ ( s, a )) -r ( π N k,h s k h ,a k h ) captures the regret due to sub-optimal intermediate return, the return of π ∗ S,a versus the return of π s k h ,a k h .

γ k h = ( P h -P π k,h ) V π ∗ h +1 ( s k h , a k h ) , σ k h = ( P h -P π k,h )( V k h +1 -V π ∗ h +1 )( s k h , a k h ) captures the regret due to sub-optimal policies missing goal-reachability. Here P h is the idealized transition (goal-reaching), while P π k,h is the transition induced by the current sub-policy.

ζ k h = P π k,h ( V k h +1 -V π k h +1 )( s k h , a k h ) -( V k h +1 -V π k h +1 )( s k h +1 ) is a martingale difference that concentrates via Azuma Hoeffding, and is dominated by the previous three sums.

̸

Focusing on ∑ H h h =1 ρ k h + γ k h + σ k h + ζ k h , we observe that γ k h , σ k h ≤ 2 H h H l P π k,h ( s k h +1 = g ( s k h , a k h )) . The key remaining step is to recognize that ρ k h + γ k h + σ k h resembles the instantaneous regret in M ( s k h , a k h ) , and the result follows after some further bounding and rearrangement.

For a concrete bound, we note that if A is set as the classic UCB-VI algorithm, then we attain the usual ˜ O ( √ K ) regret. Furthermore, we note that our bound is flexible in that one can choose more

11: Take greedy high-level action a k h = arg max a ∈ A h Q h ( s k h , a )

specialized learning algorithms A to leverage prior knowledge. For instance, if it is known that sub-MDPs are linear, one may choose to invoke multi-task RL algorithms that offer more refined regret bounds for UB ( R N K,H h ( s,a ) ) [11].

Goal Selection: An astute reader will note that the return of the learned hierarchical policy is close to V ∗ 1 ( s 1 ) , the return of the optimal hierarchical policy under goal function g . In other words, our learned policy is only as good as the goal function g we choose.

One way to relax the assumption that we have a good goal function g is to assume we have access to multiple goal functions to choose from: g 1 , .., g n . Then, an useful corollary of the sublinear Hier-UCB-VI regret bound, 1 K [ ∑ K k =1 V g i , ∗ 1 ( s 1 ) -V g i ,π k 1 ( s 1 )] ≤ ˜ O ( √ K ) , is that it directly implies an UCB on V g i , ∗ 1 ( s 1 ) (optimal return under goal g i ). Hence, we may apply any UCB-based bandit algorithm on top of this to compete with the return of the best goal out of all the candidates { g j } j ∈ [ n ] .

4 Learning from Preference Feedback

In the previous section, we develop an algorithm to efficiently learn a hierarchical policy, purely from low-level, cardinal feedback. Now, we consider learning from ordinal (preferences) feedback. Our first observation is that the low-level feedback is no longer sufficient for learning a good policy.

Proposition 1 (Non-identifiability of ranking among sub-MDP returns) . For any deterministic highlevel policy learning algorithm with N l samples of low-level feedback, there exists a MDP instance that induces regret constant in N l .

The intuition for this is simply that low-level, ordinal feedback can only identify rankings of low-level policies specific to a goal (sub-MDP), but not necessarily low level policies across differing goals. Thus, no matter how large the low-level sample-size N l , the regret is non-vanishing in N l and hence high-level feedback may be needed to learn. Please see Appendix D for all proofs of results in this section.

4.1 Labeler Feedback and Consequences for Reward Modeling

The canonical approach to learning from preferences is reward modeling. Following previous works, we study offline experiment design and assume we have the ability to collect comparison feedback data, in our hierarchical setting both high and low-level data that are then used to learn the reward model [29]. For tractable analysis, we consider the commonly studied linear reward setup [21, 20, 30, 29].

Assumption 2 (Linear Reward Parametrization) . Suppose we have access to some feature map ϕ : S × A → R d , M has linear reward parametrization w.r.t. ϕ if there exists an unknown, reward vector θ ∗ ∈ R d such that r ( s, a ) = ⟨ ϕ ( s, a ) , θ ∗ ⟩ for all s, a ∈ S × A .

Given trajectory τ = ( s 1 , a 1 , ..., s H , a H ) , we may then define trajectory feature ϕ ( τ ) = ∑ s i ,a i ∈ τ ϕ ( s i , a i ) , and policy feature expectation under transitions P , ϕ P ( π ) = E τ ∼ π,P [ ϕ ( τ )] .

With known feature map ϕ and unknown reward parameter θ ∗ , the preference feedback o t follows the Bradley-Terry-Luce (BTL) model [6].

Assumption 3. For trajectories τ 1 , τ 2 : Pr( τ 1 ≻ τ 2 ) = σ (( θ ∗ ) T ( ϕ ( τ 1 ) -ϕ ( τ 2 ))) .

With the definitions out of the way, we now describe a conceptual challenge that we encounter when learning from high-level feedback, which as we have shown before may be necessary for learning.

What can we assume about the high-level labeler's knowledge?

Consider a high level trajectory τ j = { ( s j i , a j i ) } H h i =1 . ϕ ( τ j ) = ∑ i ∈ [ H h ] ϕ ( s j i , a j i ) ; the key difficulty is that sub-MDP feature expectation ϕ ( s j i , a j i ) is dependent on the sub-policy deployed in M ( s j i , a j i ) . Thus, the high level labeler will have to have in mind some sub-policy π s,a , when making the comparison. We study two natural types of feedback:

Comparisons based on current sub-policy execution: It is natural to first assume that the labeler envisions ϕ ( s j i , a j i ) = ϕ ( π t s j i ,a j i ) at time t . In words, it is equivalent to asking: 'How well does the high level policy do given current execution of sub-goals?'

Current-feedback of this form has the caveat that the labeler will have know about the performance of the current set of sub-policies π t s,a (potentially through AI-assisted means). This knowledge would need to be updated over time as low-level policies π t s,a improve, which introduces a sizable cognitive load.

Comparisons based on idealized sub-policy execution: To reduce the cognitive load on the labeler, it is natural to fix the sub-policies used in the comparisons. A natural choice then is for the labeler to envision ϕ ( s j i , a j i ) = ϕ ( π ∗ s j i ,a j i ) . In words, it is equivalent to asking: 'How well does the high level policy do given perfect execution of the sub-goals?' Instantiated in some examples, this would be: 'how good is the essay if each argument is fleshed out perfectly' or 'how good is the code if each helper function is implemented perfectly'. Idealized-feedback of this form has the caveat that the high-level feedback will be a mismatch of how the current sub-policies actually execute. Although it has the advantage that the labeler is no longer required to (somehow) keep track of low-level sub-policies, thus reducing the cognitive load.

In what follows, we consider both types of feedback, showing that learning from idealized-feedback is possible. As we note, a drawback of idealized-feedback is that it is biased with respect to the realized features (since these are generated under current policies π t s,a ), while current-feedback is unbiased. We present an upper bound on the bias below.

Lemma 2 (Bias of idealized-feedback) . Suppose there are N h , N l high, low-level trajectories, bias b is such that: ∥ b ∥ 2 = ∑ N h t =1 |⟨ θ ∗ , ϕ π Nl ( π i 1 ) -ϕ π Nl ( π i 2 ) ⟩ - ⟨ θ ∗ , ϕ π ∗ ( π i 1 ) -ϕ π ∗ ( π i 2 ) ⟩| 2 = O ( N h /N l ) .

Proposition 2 (Reward model learning) . Let θ MLE = arg min θ ℓ D ( θ ) and let C b denote an upper bound on bias C b ≥ ∥ b ∥ , and γ, B constants. We have that with high probability:

4.2 Hierarchical Preference Learning

Wenowconstruct a hierarchical, preference-learning algorithm that invokes REGIME, a contemporary preference-learning algorithm with provable guarantees, as sub-routine for sub-MDP learning [29].

Sub-MDP reward learning: To start, we again need to incentivize goal-reaching in the sub-MDP reward. As such, given original feature ϕ orig , we introduce an additional feature accounting for goal-reachability. For trajectory τ , define ϕ i ( s τ i , a τ i ) = [ ϕ orig ( s τ i , a τ i ) , 1 ( i = H l ∧ s τ i = g ( s, a ))] and for policy π , feature expectation ϕ i ( s π i , a π i ) = [ ϕ orig ( s π i , a π i ) , 1 ( i = H l ) Pr( s π H l = g ( s, a ))] .

The corresponding reward vector will also change to become θ ∗ = [ θ ∗ orig , κ ] for unknown θ ∗ orig , κ . Assumption 4. Through instructions to the labeler, κ may be raised beyond a threshold of our choosing.

That is, we assume we can provide instructions to the labeler, emphasizing goal-reachability such that κ is higher than some given threshold. As before, we take the threshold to be max( C, 2 H h H l ) . And so while κ is unknown, we know that κ ≥ max( C, 2 H h H l ) . With this set up, we can then bound the regret due to sub-optimal sub-policies, and sub-optimal simulator P ϵ ′ , both of which are needed in the final regret analysis.

Lemma3 (Regret due to sub-optimal sub-policies) . For any high-level policy π , with high probability:

where this bound makes use of the REGIME guarantee on sub-MDP M ( s, a ) that |⟨ ϕ P ( π ∗ s,a ) , θ ∗ ⟩ -ϕ P ϵ ′ ( π N l s,a ) , θ ∗ | ≤ C 1 √ + C 2 ϵ ′ [29].

Algorithm 2 Hierarchical-REGIME (Hier-REGIME)

Require: High-level policy class Π h , low level-policy classes Π l s,a , simulator P ϵ ′ with ϵ ′ -precision 1: for episode n = 1 , ..., N l do

2: ( π n 1 , π n 2 ) ← arg max π 1 ,π 2 ∈ ⋃ s,a Π l s,a ∥ ϕ P ϵ ′ ( π 1 ) -ϕ P ϵ ′ ( π 2 ) ∥ ( ˆ Σ l n ) -1 ▷ explore using policy feature expectation across sub-MDPs

3: ˆ Σ l n +1 = ˆ Σ l n +( ϕ P ϵ ′ ( π n 1 ) -ϕ P ϵ ′ ( π n 2 ))( ϕ P ϵ ′ ( π n 1 ) -ϕ P ϵ ′ ( π n 2 )) T

4: Generate trajectories τ n 1 , τ n 2 and acquire comparison feedback o n ▷ comparison feedback for the pair of lengthH l trajectories

5: Compute MLE ˆ θ l from { τ n 1 , τ n 2 } N l n =1 and { o n } N l n =1

6: Compute π N l s,a = arg max π ∈ Π l s,a ⟨ ϕ P ϵ ′ ( π ) , ˆ θ l ⟩

7: for episode n = 1 , ..., N h do

8: ( π n 1 , π n 2 ) ← arg max π 1 ,π 2 ∈ Π h ∥ ϕ π Nl ,P ϵ ′ ( π 1 ) -ϕ π Nl ,P ϵ ′ ( π 2 ) ∥ ( ˆ Σ h n ) -1 ▷ high-level policy feature expectation generated using π N l s,a

9: ˆ Σ h n +1 = ˆ Σ h n +( ϕ π Nl ,P ϵ ′ ( π 1 ) -ϕ π Nl ,P ϵ ′ ( π 2 ))( ϕ π Nl ,P ϵ ′ ( π 1 ) -ϕ π Nl ,P ϵ ′ ( π 2 )) T

10: Generate trajectories τ n 1 , τ n 2 and acquire comparison feedback o n ▷ comparison feedback for the pair of lengthH h trajectories

11: Compute MLE ˆ θ h from { τ i 1 , τ i 2 } N h i =1 and { o i } N h i =1

12: return high-level policy ˆ π = arg max π ∈ Π h ⟨ ϕ π Nl ,P ϵ ′ ( π ) , ˆ θ h ⟩ , low-level policies { π N l s,a } s,a ∈ S × A h

Lemma 4 (Regret due to sub-optimal simulator P ϵ ′ ) . Let Φ π Nl ,P ϵ ′ ( π ) denote the feature expectation under high level policy π , sub-MDP policies π N l and transitions P ϵ ′ . With high probability, for any high level policy π :

4.3 Hier-REGIME Analysis

Now, we present the Hier-REGIME Algorithm 2. At a high-level description goes as follows. First, we invoke one copy of REGIME across all sub-MDPs with shared exploration (L1-4) and learned reward (L5). Next, we use the learned reward to compute sub-MDP policies π N l s,a for each sub-MDP M ( s, a ) (L6). Finally, we invoke one copy of REGIME for the high-level MDP, where the feature function is defined as ϕ π Nl s,a ,P ϵ ′ (L8). Next, we note two properties about Algorithm 2.

Hierarchical Exploration: A key aspect of experiment design in offline RL is ensuring sufficient coverage with exploration. The difficulty with coverage in the hierarchical setting is that at first glance, we may need to search for pairs of trajectories over ( π 1 , { π 1 s,a } ) , ( π 1 , { π 2 s,a } ) ∈ (Π h , × s,a Π l s,a ) , instead of over π 1 , π 2 ∈ Π h . However, we show that in the goal-HRL case, we can fix the sub-policies to be π N l s,a (for N l large enough), and this is sufficient to compete with the optimal, hierarchical policy.

Additionally, unlike the tabular setting, sub-MDPs now share a common reward parameter θ ∗ , thus allowing us to jointly, instead of separately as in tabular case, explore across sub-MDPs.

Sufficiency of low-level feedback: Through the algorithm, we can observe that low- and high-level exploration generates feature expectations set: { ϕ P ϵ ′ ( π 1 ) -ϕ P ϵ ′ ( π 2 ) | π 1 , π 2 ∈ ⋃ s,a Π l s,a } and

{ ϕ P ϵ ′ ( π 1 ) -ϕ P ϵ ′ ( π 2 ) | π 1 , π 2 ∈ Π h , π s,a = π N l s,a ∀ s, a } . Therefore, when coverage of high level policy is subsumed by low-level features already (the latter is a subset of the former), it suffices to explore only using low-level feedback. As shown before in Proposition 2, it is not always sufficient. However, as we will see below, when it is sufficient, using low-level feedback leads to better rates. First, we derive the regret decomposition and then use it evaluate the sample complexity.

Theorem 2. With high probability, under N h > 0 :

To parse this, the regret decomposes into four terms. The first term is the regret due to sub-optimality in low-level policies π N l . The remaining three terms are derived from sub-optimality due to high-level policy ˆ π , decomposing into the second term on regret due to bias in learned reward ˆ θ , the third and fourth term on regret due to sub-optimality of simulator P ϵ ′ .

A main benefit of developing a learning Algorithm 2 is that we can then quantitatively assess the sample complexity associated with the two types of human feedback. As one may expect, there is a tradeoff between better sample complexity and cognitive load, with current-feedback attaining better sample efficiency but also requiring higher cognitive load on the human supervisor.

Corollary 1. Using Theorem 2, we obtain the following rates in terms of data tradeoffs:

Idealized-feedback and required high-/low-level feedback: the overall rate comes out to O ( N -1 / 4 l + N -1 / 2 h ) . While high level trajectories provide additional coverage, it also incurs bias linear in N h of the bias of the low-level trajectories, thus slowing down the rate (Lemma 2).

Current-feedback and required high-/low-level feedback: the overall rate comes out to O ( N -1 / 2 l + N -1 / 2 h ) . The current-feedback is unbiased and results in more efficient reward learning with ∥ θ ∗ -ˆ θ ∥ ˆ Σ h Nh = O (1) [29].

Only low-level feedback is required due to sufficiency in coverage: the overall rate comes out to O ( N -1 / 2 l ) . In a nutshell, this is because we can explore with just N l low-level samples which is unbiased, resulting in ∥ θ ∗ -ˆ θ ∥ ˆ Σ l Nl = O (1) . Hence, both exploration and reward learning is efficient.

5 Discussion

Our work considers scalable oversight in the context of goal-conditioned HRL, in which we show that one can efficiently use hierarchical structure to learn from bounded human feedback.

Limitations & Future Work: In goal-conditioned HRL, our regret guarantees are with respect to the return of the optimal, hierarchical policy, whose performance is dependent on the usefulness of goal function g . Further research is needed to understand on how to learn good goal functions, using limited supervised or unsupervised learning. Additionally, under current-feedback, the labeler providing high-level feedback is somehow made aware of sub-policy performance. An exciting research direction is how one may provide such knowledge through AI-assistance.

References

AP English Literature Scoring Rubrics. https://apcentral.collegeboard.org/media/ pdf/ap-english-literature-and-composition-frqs-1-2-3-scoring-rubrics. pdf . Accessed: 2024-05-15.

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 , 2016.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International conference on machine learning , pages 263-272. PMLR, 2017.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil˙ e Lukoši¯ ut˙ e, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540 , 2022.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning , pages 1430-1440. PMLR, 2021.

Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575 , 2018.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. Advances in Neural Information Processing Systems , 35:26091-26104, 2022.

Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, and Liwei Wang. Near-optimal representation learning for linear bandits and linear rl. In International Conference on Machine Learning , pages 4349-4358. PMLR, 2021.

Evan Hubinger. Ai safety via market making. https://www.alignmentforum.org/posts/ YWwzccGbcHMJMpT45/ai-safety-via-market-making . Accessed: 2024-05-15.

Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899 , 2018.

Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091 , 2021.

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871 , 2018.

Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948 , 2017.

Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems , 31, 2018.

Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Conference on Uncertainty in Artificial Intelligence , pages 1029-1038. PMLR, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2111.04850 , 2021.

Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems , 33:18050-18062, 2020.

Arnaud Robert, Ciara Pike-Burke, and Aldo A Faisal. Sample complexity of goal-conditioned hierarchical reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 , 2022.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Zheng Wen, Doina Precup, Morteza Ibrahimi, Andre Barreto, Benjamin Van Roy, and Satinder Singh. On efficiency in hierarchical reinforcement learning. Advances in Neural Information Processing Systems , 33:6708-6718, 2020.

Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46, 2017.

Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 , 2021.

Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 33:18784-18794, 2020.

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. Provable reward-agnostic preference-based reinforcement learning. arXiv preprint arXiv:2305.18505 , 2023.

Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k -wise comparisons. arXiv preprint arXiv:2301.11270 , 2023.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

A More Related Works

Scalable Oversight: Scalable oversight is a nascent but important topic in the area of AI alignment [2, 8, 15, 5], wherein the goal is to boost the labeler's ability to provide feedback to complex models. Proposed approaches include (recursive) self-critique, summarization, debate, plain model Interaction and market-making, all of which aim to have the model (or auxiliary models) generate interpretable and/or lower-dimensional forms of outputs for the human to parse [15, 13, 24, 27, 23, 5, 12]. Our work studies how one may leverage hierarchical structure as one approach to scaling up feedback.

Goal-conditioned RL: Further afield, there has been a lot of work demonstrating the promise/success of goal-conditioned RL with examples from the likes of [16, 17, 7, 10]. The sub-MDP reward is often set to incentivize only goal state reachability, as oftentimes the MDP of interest has sparse rewards, making intermediate returns zero. In our setting, rewards need not be sparse, thus bringing into consideration the tradeoff between intermediate return and goal-reachability. This work initiates the study of scalable oversight in goal-oriented HRL, and owing to the success of goal-oriented HRL in practice, it is our hope that it can be stepping stone towards developing practical scalable oversight techniques.

B Concrete Hierarchical MDP Example

The prototypical example in HRL is the maze, as studied in for instance [17, 22]. A maze consists of rooms with doors. The goal is to get to the exit in as few steps as possible. The MDP may be defined as follows:

For the global MDP, S = S h × S l where s h denotes the index of the current room, and s l denotes the position of the agent in the room. Action set A consists of moving (L, R, U, D, Stay).

For the High-level MDP, high-level action A h consists of moving to the (N, S, E, W) door of the room. s is the current location of the agent, and g ( s, a h ) maps the goal (door) to its location.

For the Low-level MDP, it has state space S l s,a ⊂ S and the action set A is the same moving (U, D, L, R, Stay).

As noted in the previous section, HRL algorithms can achieve superior statistical sample complexity when there is lots of repeated sub-MDP structure (there are many isomoprhic rooms) and each room has small state-space size [25].

C Proofs for Section 3

C.1 Sub-MDP Bonus Construction

Sub-MDP Reward Definition: Define the reward in sub-MDP M ( s, a ) at time step h to be: r l,h ( s ′ , a ′ ) = r ( s ′ , a ′ ) + κ 1 ( h = H l ∧ s ′ = g ( s, a )) .

Firstly, since by definition π ∗ s,a ∈ arg max π ∈ Π s,a r ( π ) + C · Pr( s π H l = g ( s, a )) , we have that π ∗ s,a ∈ arg max π ∈ Π s,a r ( π ) + κ · Pr( s π H l = g ( s, a )) . Indeed,

Secondly, using the definition of r l , we have that:

By the reachability assumption, P ( s π ∗ s,a H l = g ( s, a )) = 1 , this implies that

̸

Therefore, summing this across n visits to M ( s, a ) , we have:

̸

This statement is useful because we can compute an UCB on ∑ n i =1 r ( π ∗ s,a ) and, implicitly, a LCB on ∑ n i =1 r ( π i s,a ) (provided we do not bound R n ( s, a ) ).

Lemma 5 (Bonus with 'penalty' for non-reachability) . Let UB ( R n ( s, a )) be any upper bound on the sub-MDP regret, then if we define:

̸

Then, ¯ r n ( s, a ) + b s,a r ( n ) is an UCB for r ( π ∗ s,a ) with probability ≥ 1 -δ 3 |C ( S,A h ) | H h K . Let the event that the above holds be E n s,a .

Proof.

̸

( ⋄ ) : Here we use two applications of Azuma-Hoeffding:

With probability higher than 1 -δ :

̸

̸

̸

We have that E [ P ( s π i s,a H l = g ( s, a )) -1 ( s π i s,a H l = g ( s, a )) |F i -1 ] = 0 .

̸

̸

̸

This is true because P ( s π i s,a H l = g ( s, a )) and 1 ( s π i s,a H l = g ( s, a ) are a function of only the transition probability of the MDP at the i th step conditioned on F i -1 . Thus, P ( s π i s,a H l = g ( s, a )) -1 ( s π i s,a H l = g ( s, a )) is a martingale difference. And we can use Azuma-Hoeffding.

With probability higher than 1 -δ :

This again follows from Azuma-Hoeffding on martingale difference r ( π i s,a ) -ˆ r ( π i s,a ) , as E [ r ( π i s,a ) -ˆ r ( π i s,a ) |F i -1 ] = 0 . And | r ( π i s,a ) -ˆ r ( π i s,a ) | ≤ H l .

̸

Thus,

̸

̸

̸

̸

Remark 1. One choice for UB ( R n ( s, a )) = H 3 / 2 l √ | S l s,a || A | n if we let A s,a be the standard UCB-VI algorithm [3].

C.2 Optimism Lemma

Lemma 6 (Optimism) . Let V k h be the V value as in Algorithm 1 at episode k . Let π ∗ be the optimal hierarchical policy. For a fixed k and h , if ∀ s, a, n , E n s,a holds, then:

Proof. Fix some episode k . We will prove this lemma via induction on h = H h +1 , ..., 1 .

Induction Step: Suppose this is true for up until h = H h +1 , ..., h ′ +1 . Now at time step h ′ and any s, a .

Firstly, if Q k h ′ ( s, a ) = H h H l (e.g. if s, a ̸∈ D k ), then Q k h ′ ( s, a ) ≥ Q ∗ h ′ ( s, a ) . Otherwise, Q k h ′ ( s, a ) < H h H l and we have that:

Thus, V k h ′ ( s ) = max a Q k h ′ ( s, a ) ≥ max a Q ∗ h ′ ( s, a ) = V π ∗ h ′ ( s ) .

Corollary 2.

C.3 Supporting results needed for regret analysis

Proposition 3.

Proof. For any k and h , we consider bounding V k h ( s k h ) -V π k h ( s k h ) , which is equal to:

where we set ρ k h = ¯ r N k,h ( s k h ,a k h ) ( s k h , a k h ) + b s k h ,a k h r ( N k,h ( s k h , a k h ))) -r ( π N k,h ( s k h ,a k h ) s k h ,a k h ) .

Continuing with the original proof and focusing on the second term:

where

In summary,

where we introduce the notation ζ k h = P π k,h ( V k h +1 -V π k h +1 )( s k h , a k h ) -( V k h +1 -V π k h +1 )( s k h +1 ) . Unrolling the recursion starting at h = 1 :

Summing across k ∈ [ K ] , it suffices to bound:

One is the sub-optimality while executing the sub-MDP policies. This is covered by the per-step high level reward bonus (which is also the UCB on the return of the sub-MDP's return) in ρ k h .

The other is the sub-optimality of not landing on g ( s k h , a k h ) , there is covered by γ k h , σ k h , which affects future reward. The martingale difference ζ k h is zero in expectation, so it is not some measure of suboptimality.

We first bound the ζ 's, whose sum is dominated by ∑ K k =1 ∑ H h h =1 ρ k h + γ k h + σ k h .

Lemma 7. With probability ≥ 1 -δ/ 3 :

Let the event that the above inequality hold be E ζ .

Proof. The concentration of ζ k h follows from Azuma Hoeffding, as the following is a martingale difference.

with E [ ζ k h | F k,h ] = 0 , since the expectation is only wrt randomness in s k h +1 . Moreover, this martingale difference is bounded by 4 H h H l

Next, we simplify the sum of remaining terms.

Lemma 8. We have that:

and

Proof.

Similarly,

̸

̸

̸

̸

̸

̸

Lemma 9. With probability ≥ 1 -δ/ 3 :

Let E ρ be the event that this holds.

Proof. We first expand the ρ k h sum:

̸

(Azume-Hoeffding for concentration of ˆ r around r )

̸

Using the two-sided concentration bound we had before (the other way): ∑ i j =1 1 ( s π j s,a H l = g ( s, a )) + ψ i ≥ ∑ i j =1 P ( s π j s,a H l = g ( s, a )) w.h.p:

̸

̸

̸

̸

We continue our derivation:

̸

C.3.1 Overall Regret Bound

Theorem 3. Under events ⋂ s,a,n E n s,a ∩ E ζ ∩ E ρ , we have that:

Proof.

̸

̸

̸

Corollary 3 (Regret under |C ( S, A h ) | clusters of isomorphic sub-MDPs [25]) . Let us set UCB-VI to be the sub-MDP learning algorithm, then we have the following regret bound:

using UCB-VI's guarantee that upper bound UB ( R N K,H h ( s,a ) ) = H 3 / 2 l √ | S l s,a || A | N K,H h ( s, a ) .

Remark 3 (High Probability Bound) . For completeness, we show that the regret bound holds with probability greater than 1 -δ . The regret bound holds under ⋂ s,a,n E n s,a ∩E ζ ∩E ρ , by union bound:

D Proofs for Section 4

D.1 Low-level Feedback is insufficient for learning

To prove the results below, our approach is to construct two MDP instances with identifical low level feedback such that any deterministic learning algorithm picks the arbitrarily worse high level policy.

Proposition 4 (Non-identifiability of ranking among sub-MDP returns) . For any deterministic highlevel policy learning algorithm with N l samples of low-level feedback, there exists a MDP instance that induces regret constant in N l .

Proof. Consider two-horizon MDP with starting state s 1 with H h = 1 , H l = 2 . There are two possible high-level actions a 1 and a 2 at s 1 .

For any policy π 1 in sub-MDP M ( s 1 , a 1 ) , let it have feature expectation ϕ ( π 1 ) = [ ϕ ′ ( π 1 ) , 1 , 0] , and for any π 2 in sub-MDP M ( s 1 , a 2 ) , ϕ ( π 2 ) = [ ϕ ′ ( π 2 ) , 0 , 1] .

Now, we consider two MDP instances with θ ∗ = [0 , 0 , C ′ ] and θ ∗ = [0 , C ′ , 0] for some positive constant C ′ .

Under both instances, we observe identical low-level feedback for trajectories τ, τ ′ in sub-MDPs M ( s 1 , a j ) , j ∈ [2] : the feedback is Bernoulli with parameter σ ( ⟨ ϕ ′ ( τ ) -ϕ ′ ( τ ) , θ ′ ⟩ ) .

Consider any deterministic learning algorithm. WLOG it outputs high level policy π h ( s 1 ) = a 1 with some set of N l samples of low-level feedback.

Then, it follows that its regret under θ ∗ = [ ϵ 1 , 0 , C ′ ] is C ′ , since the reward (and return since H h = 1 ) of π ∗ s 1 ,a 1 is 0 , while the reward of the optimal policy which visits M ( s 1 , a 2 ) is C ′ .

D.2 Hierarchical Experiment Design via REGIME [29]

D.2.1 MLEDefinition:

We first define the MLE expression; note that the MLE is in terms of trajectories only. Define:

High-level trajectories: has realized features,

where ϕ π Nl ,P ( τ i j ) is the feature of the high-level trajectory under sub-policy π N l and transition P (since trajectories are collected from roll-outs in the actual MDP as in [29]). On the other hand, under idealized-feedback, the labeler assumes that each goal-conditioned sub-MDP has been executed perfectly (i.e. by π ∗ s,a ) and so the features correspond to:

Comparison y of high level trajectories follows Bernoulli distribution y i = σ ( θ ∗ · x ∗ i ) .

Low-level trajectories: has realized features,

Note that unlike the high level features, low-level features data are always unbiased. Thus, using high level and low-level comparisons has the same bias from the high level.

Comparison y of low level trajectories follows Bernoulli distribution y i = σ ( θ ∗ · x s,a i ) .

D.2.2 Requisite Lemmas

Lemma 10 (Lemma 5 of [29]) . Let oracle P ϵ ′ be such that with probability 1 -δ/ 5 , the following holds. Let d π h ( s, a ) and ˆ d π h ( s, a ) be the visitation measure of policy π under P and P ϵ ′ , we have for all h ∈ [ H ] and π ∈ Π :

:

This applies across all sub-MDPs M ( s, a ) . Let the event that this expression hold be E s,a . Lemma 11 (Low-level MLE Bound, Lemma 2 of [29]) . With probability at least 1 -δ/ 5

Let the event that this holds for learning from sub-MDP trajectories be E l 1 .

Lemma 12 (Lemma 3 of [29]) . If low-leve trajectories τ 1 , 2 i ∼ π i , P ϵ ′ , then with probability at least 1 -δ/ 5 :

Let the event that this holds for learning from sub-MDP trajectories be E l 2 .

D.2.3 Bias when using idealized-feedback, high level trajectory data in MLE

Proposition 5 (sub-MDP REGIME guarantee of [29]) . For sub-MDP M ( s, a ) , under E s,a ∩E l 1 ∩E l 2 :

Note that for estimation and bias, we have to have both an upper bound and a lower bound (see PbRL example). This requires two-sided bound, where lower bound comes from ϕ ∗ having higher reward than ϕ and upper bound comes from no-regret. Due to optimality of π ∗ , we have the lower bound as well:

Additionally, we have that:

Lemma 13 (Lemma 6 of [29]) . For any s h , a h , ∥ v i ∥ ≤ 2 B , θ ∈ R d and ∥ ϕ ∥ ≤ R under E s,a ∩E l 1 ∩ E l 2 :

With this,

Now, we can analyze the bias of including high level trajectory data in the MLE computation: Lemma 14. Suppose there are N h , N l high, low-level trajectories, bias b is such that, under ⋂ s,a E s,a ∩ E l 1 ∩ E l 2 :

Proof.

Thus,

D.2.4 MLEAnalysis

Under current-feedback, following Lemma 2 of [29], ∥ ∆ ∥ Σ h n + λI ≤ ˜ O (1) . Now, we consider the bias in learned reward under idealized-feedback.

Proposition 6. Let θ MLE = arg min θ ℓ D ( θ ) and let C b ≥ ∥ b ∥ . Then with probability at least 1 -δ/ 5 :

where Σ n = 1 n ∑ n i =1 x i x T i + λI .

Proof. Define ∆ = θ MLE -θ ∗ . As in [30], we have the same convexity result due to ⟨ θ, x i ⟩ ∈ [ -2 LB, 2 LB ] . Suppose we let max x ∥ x ∥ ≤ L and max θ ∈ Θ ∥ θ ∥ ≤ B , then with γ = 1 2+exp( -2 LB )+exp(2 LB ) , we have that:

And so,

Thus,

The key part is bounding ∥∇ ℓ ( θ ∗ ) ∥ (Σ+ λI ) -1 . We have that:

where v i = σ ( ⟨ θ ∗ , x ∗ i ⟩ ) w.p 1 -σ ( ⟨ θ ∗ , x ∗ i ⟩ ) and -(1 -σ ( ⟨ θ ∗ , x ∗ i ⟩ )) w.p σ ( ⟨ θ ∗ , x ∗ i ⟩ ) . And so, entrywise V is such that E [ V i ] = 0 and | V i | ≤ 1 . Note that V i are independent due to the independence of the random variables Y i .

Extra term bias is defined as:

By definition, C b is such that: ∥ b ∥ ≤ C b . As before, define M = 1 n 2 X (Σ + λI ) -1 X T . We use the fact that ∥ M ∥ op ≤ 1 /n . Then, we have that:

This implies that with probability ≥ 1 -δ :

Corollary 4. Let θ MLE = arg min θ ℓ D ( θ ) , then under ⋂ s,a E s,a , with probability ≥ 1 -δ/ 5 :

where Σ N h = 1 N h ∑ N h i =1 x i x T i .

Let the event that this holds for learning from sub-MDP trajectories be E h 1 .

Proof. Firstly,

With this, we have that:

Hence by choosing λ = λ/N h :

Define:

We wish to relate ∥ θ ∗ -θ n ∥ ˆ Σ n to ∥ θ ∗ -θ n ∥ ˜ Σ n .

Lemma 15 (Lemma 3 of [29]) . If τ 1 , 2 i ∼ π i 1 , π N l , P ϵ ′ , then with probability at least 1 -δ/ 5 :

Let the event that this holds for learning from sub-MDP trajectories be E h 2 .

Lemma 16. We have that under ⋂ s,a E s,a ∩ E l 1 ∩ E l 2 ∩ E h 1 ∩ E h 2 :

Proof. Under event E h 2 , as trajectories are sampled from P , we have that:

It remains to upper bound ∥ θ ∗ -θ n ∥ ˆ Σ n by ∥ θ ∗ -θ n ∥ Σ n

We have that under ⋂ s,a E s,a ∩ E l 1 ∩ E l 2 :

Thus,

Plugging in v = θ ∗ -θ n , we have that:

D.2.6 High-level policy regret bound

Lemma 17. For any π , under event ⋂ s,a E s,a ∩ E l 1 ∩ E l 2 :

̸

Proof.

̸

Because for any s h , a h , ⟨ ϕ ( π ∗ ( s h , a h )) , θ ∗ ⟩ - ⟨ ϕ ( π N l ( s h , a h )) , θ ∗ ⟩ ≤ C 1 √ N l + C 2 ϵ ′ .

Lemma 18 (Lower bound on Reachability Probability) . We have that under event ⋂ s,a E s,a ∩E l 1 ∩E l 2 :

̸

and

̸

Proof. Due to the regret guarantee, we have that:

̸

Thus, we have that:

̸

Additionally, we have that from Lemma 5.1:

Thus,

̸

Define goal non-reachability probability to be: δ = 1 κH h + C 1 κH h H l √ N l + C 2 ϵ ′ κH h H l + H l ϵ ′ .

Lemma 19. Let Φ π Nl ,P ϵ ′ ( π ) denote the feature expectation under high level policy π , sub-MDP policies π N l and MDP transitions P ϵ ′ . Under event ⋂ s,a E s,a ∩ E l 1 ∩ E l 2 , we have that, for any high level policy π :

Proof. Let E reach denote the event that roll-out τ ∼ π, π N l , P is such that all high level goals are reached, and similarly event E ′ reach for roll-out τ ′ ∼ π, π N l , P ϵ ′ .

̸

̸

By union bound, Pr( ¬E reach ) = Pr( ∃ s i , a i , s π Nl ( s i ,a i ) H l = g ( s i , a i )) ≤ ∑ H h i =1 Pr( s π Nl ( s i ,a i ) H l = g ( s i , a i )))) ≤ H h δ , and similarly Pr( ¬E ′ reach ) ≤ H h δ .

̸

̸

To finish, we will relate the expression to |⟨ ϕ P ϵ ′ ( π N l ( s h , a h )) -ϕ ( π N l ( s h , a h )) , θ ∗ ⟩| .

Lemma 20 (use of the Elliptical Lemma) .

Proof.

Theorem 4 (Main regret bound) . We have that under event ⋂ s,a E s,a ∩ E l 1 ∩ E l 2 ∩ E h 1 ∩ E h 2 and N h > 0 :

Proof.

(use of Elliptical lemma)

Data Tradeoff: Using the above bound, we can derive the following rates:

Under idealized-feedback and requiring both high- and low-level feedback, the overall rate comes out to O ( N -1 / 4 l + N -1 / 2 h ) .

This is because ˆ Σ N h = O ( N 1 / 2 h N 1 / 4 l +1 ) . Thus, the dominating factor is the bias of the reward learning.

Under current-feedback and requiring both high- and low-level feedback, the overall rate comes out to O ( N -1 / 2 l + N -1 / 2 h ) .

This is because ∥ θ ∗ -ˆ θ ∥ ˆ Σ Nh = O (1) .

Under only low-level feedback (due to sufficiency in coverage), the overall rate comes out to O ( N -1 / 2 l ) .

We have that:

( ⋄ ) : since low-level policy feature expectation is a superset of high-level policy expectation, it follows that by choice of low-level policies π i 1 , π i 2 : ∥ ϕ P ϵ ′ ( π i 1 ) -ϕ P ϵ ′ ( π i 2 ) ∥ ˆ Σ -1 i ≥

Moreover, since low-level feedback is always unbiased, ∥ θ ∗ -ˆ θ ∥ ˆ Σ Nl = O (1) . Thus, the overall rate comes out to O ( N -1 / 2 l ) .

Remark 4 (High Probability Guarantee) . For completeness, we show that the theorem statement holds with probability at least 1 -δ :

D.2.7 Additional Guarantees

In addition, we derive requisite conditions on the constants for idealized-feedback (the most interesting case).

Necessary Auxiliary Parameters Bound: We have that,

Setting the upper bound to be below ϵ , or each term to be below ϵ/ 4 , we obtain the following bounds:

ϵ ′ ≤ O (min( ϵ dH 3 h H 2 l , ϵ d 3 H h BR ) :

Finally, we also require that ( C 2 H h +18 d 3 H h BR ) ϵ ′ ≤ ϵ/ 4 ⇒ ϵ ′ ≤ O ( ϵ d 3 H h BR ) . Thus, we need that ϵ ′ ≤ O (min( ϵ dH 3 h H 2 l , ϵ d 3 H h BR ) .

h l

E Statistical Efficiency of HRL

An useful sanity check for hierarchical RL algorithms is that it achieves improved statisical sample complexity in settings with repeated sub-MDP structure [25]. As in [25], we examine if Algorithm 1 also improves upon algorithms that do not leverage hierarchical structure. We make this comparison with vanilla UCB-VI under the same isomophism assumption.

K,H h

Comparison with vanilla UCB-VI: Standard application of UCB-VI yields the following rate: ˜ O (( H h H l ) 3 / 2 √ | S || A | K ) . Hier-UCB-VI compares favorably to vanilla UCB-VI, if max s,a | S l s,a ||C ( S, A h ) | << | S | . Or in words, there is a lot of repeated/identical sub-MDPs and sub-MDPs have small state space size.

F NeurIPS paper checklist

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our paper contains everything that is covered in the abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is covered in the 'Discussions' section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the proofs for results are included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theory paper that has no experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theory paper that does not involve code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theory paper that has no experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theory paper that has no experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theory paper that has no experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper does conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: To our knowledge, this theory paper has no positive/negative social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the scalable oversight problem in the goal-conditioned hierarchical reinforcement learning setup. The starting point of the paper is that, if the time horizon $H$ is large, there is only a limited amount of feedback that can be given (authors use an example of essay or code writing, which requires the labeller to read through and provide feedback on the whole output, which is is very expensive). Thus, it is crucial to leverage hierarchical feedback (e.g. judging one paragraph, or one helper function, or the high-level plan). Given a tabular MDP, the task of finding a good policy is decomposed into two levels: \n - a high-level policy $\\pi^h$ that computes, in a state $s$, a high-level action $a^h$ which in turn gives a goal $g(s,a^h) \\in S$ and a sub-MDP $M(s, a^h)$ that operates on a subset of states $S_{s, a^h} \\subseteq S$, but has access to all actions $A$\n - a set of low-level policies $\\pi_{s,a^h}$, for each state $s$ and high-level action $a^h$, which attempt to reach a goal $g(s,a^h)$ (almost surely) while collecting as much reward on the way as possible.\n\nIn the first part of the paper, the feedback is assumed to be cardinal. The goal function is assumed to be given and fixed. The task studied is then to find a high-level policy and a set of low-level policies, which give close-to-optimal return. This is done by employing a two-level UCB-VI procedure, where high-level actions correspond to learning UCB over sub-MDPs. The reward feedback is then only given for the low-level trajectories of length $H_l$ (since the high-level reward is computed by adding low-level ones).\n\nIn the second part of the paper, the feedback is assumed to be ordinal, and to follow BTL model. Authors point out that there is a subtle issue with the low-level feedback not always being sufficient for no-regret learning, and high-level feedback depending on what the labeller believes the low-level polices will be (e.g. whether they judge wrt to optimal or actual low-level policies). They develop a hierarchical preference-learning algorithm H-REGIME and analyse its regret in all three cases discussed above.\n\nSTRENGTHS:\nThe paper studies an important problem of scalable oversight, and approaches it from a perspective that could potentially be used as a basis for a practical implementation, both in ordinal and cardinal setup, at least if the problems mentioned in Weaknesses are resolved. The fundamental idea of doing a hierarchical UCB-VI is quite simple, but computing the right form of the bonus, and the insight to operate in the goal-conditioned setup (leading to a particular form of low-level reward functions and simplifying the theory quite a lot) seem novel and non-trivial. The paper looks to be technically sound, and although I did not carefully check very technical proofs of various inequalities, which span 30 pages in the appendix, very detailed notes make it possible to spot-check it without issues.\n\nWEAKNESSES:\nMy main question/objection is about the H-UCB-VI algorithm. For shared learning, the algorithm involves assigning the chosen sub-MDP to an appropriate cluster $C(s,a)$. This means that the transition probabilities and rewards must be known for the whole MDP (well, for all sub-MDPs). But if this is the case, then there's no need for using UCB - we can use VI directly. Moreover, the paper improves over UCB-VI only in case there are many (small) isomorphic sub-MDPs, which makes the dependence on shared learning crucial for this to be useful. (Moreover, the motivating example used throughout the paper is given to be essay writing, which does not seem to enjoy this property. Other than that, authors do not give any other examples, nor do any concrete example calculations - even in the appendix - which makes it more difficult to understand and appreciate the work.) This is where I might be fundamentally misunderstanding the work - please correct me if I'm wrong, this is the main reason for my low rating.\n\nIn general, it was very unclear to me what exactly is the influence of the goal function on the whole setup - see questions below. The discussion was limited to a few lines in \"Goal Selection\" paragraph, but I did not gain a good intuition the interplay between the high-level policy performance, and the goal function. (For example, how much work is good goal function doing when comparing to the baseline UCB-VI.)\n\nQUESTIONS:\nQuestions:\n - The definition of the MDP in the introduction does not contain a reference to high-level actions. How do they fit into the definition, formally?\n - Why are states for sub-MDPs indexed by high-level action as well? Why are all actions $A$ available in sub-MDPs? What if an action leads outside of $S_{s,a}$?\n - In what way do results depend on the goal function? Can it be quantified? Is it possible to derive $W(g)$ of the value of the optimal policy wrt to this goal function, and say something about the sensitivity of of this $W$ on the parameter? How would goal functions be derived in a realistic setup? (Those questions are probably somewhat, or even significantly, out of scope of the paper, but I'd appreciate if the authors provided at least some preliminary answers, since the practicality and a lot of the value of this development hinges on this problem). \n - What are the policy classes $\\Pi^h$ and $\\Pi_{s,a}$? They appear to be assumed in the definitions, but are not introduced otherwise.\n - I do not know the REGIME algorithm, but I could not find it in the provided reference?\n - Does the approach discussed in the paper extend to more than two levels of hierarchy?\n - What is $S_h$ in line 217/218?\n - Algorithm 2, line 12 - shouldn't it say length H^h trajectories?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProvides proofs of the regret bounds for cardinal and ordinal feedback using the sub-MDP framework of goal-conditioned HRL. Sub-MDPs are defined by a starting state, fixed horizon, high-level action and subspace. The high-level policy selects transitions between sub-MDPs. This work first proposes an upper confidence bound algorithm based on lower-level reachability and selection, then proves the regret of this algorithm.\n\nSTRENGTHS:\nThe method provides clear and applicable analysis of hierarchical goal-conditioned RL, even outside of the context of feedback.\n\nThe derivation of the regret due to goal-conditioned HRL is informative and interesting.\n\nThe proposed insight of ensuring hierarchical consistency is meaningful for many applications of GCHRL.\n\nWEAKNESSES:\nAlgorithm 1 is somewhat difficult to follow. In particular, while the components make sense, the core insight of the bounds on the lower-level goals is difficult to locate.\n\nThe regret analysis proof sketch could provide more insight. In particular, the separation of which components occur because of the high-level and low-level errors, which introduce the product of horizons, is not intuitively clear. \n\nThe work for ordinal feedback is not particularly well contained, since it relies on the properties of REGIME, which are not made clear in this work. In particular, it is not obvious without looking into that work where the sub-policy simulator fits in. \n\nIn many ways this seems less to be a work about the nature of scalable feedback, and simply on the nature of hierarchical RL in providing regret bounds. This subject is of interest regardless but calls to question the framing of this work.\n\nQUESTIONS:\nCan the work on regret analysis of GCHRL be separated from the analysis of ordinal rewards?\n\nWhat is the relationship between the analysis of these two?\n\nLIMITATIONS:\nThe analysis of other components of HRL, such as goal sampling, learning procedure, etc. are not that well captured, though this is typically challenging to analyze theoretically.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper analyzes scalable oversight in the context of goal-conditioned hierarchical reinforcement learning. Specifically, it theoretically shows that it is possible to efficiently use hierarchical structure to learn from bounded human feedback.

STRENGTHS:
* The problem is significant and of high importance to the community.
* The theoretical analysis and sub-MDP reward design for Hierarchical-UCB-VI appears novel. Furthermore, the extension to ordinal (preferences) feedback appears novel, specifically in the proposed Hierarchical-REGIME algorithm and analysis.

WEAKNESSES:
* Minor: It would be a nice addition to the paper to empirically demonstrate Algorithm 1 &2 as well.
* Minor: Clarity: Paper had quite a few typo’s. I would encourage the authors to use an automated tool to check for spelling / grammar mistakes.

Typos:

* L61: “policies aims to” -> “policies aim to”
* L183: “approach is avoid” -> “approach is to avoid”
* L218: “identifical” -> “identical”
* L250: “what can we assume the” -> “what can we assume is the”

QUESTIONS:
* Does the analysis for the linear setting generalize to the non-linear / function approximation setting?

LIMITATIONS:
Yes they are discussed in Section 5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study how to scale human feedback, in the context of goal-conditioned hierarchical reinforcement learning. For this work, the authors assume that humans can only provide feedback for outputs with length below a certain threshold. Thus, it is necessary to scale the little feedback provided, in order to improve the complete output. This means, for example, that a human can provide feedback about the high-level actions taken in the hierarchy, but not the detailed steps of the low-level policies. The authors propose two algorithms: one that learns from low-level feedback only, and one that learns by incorporating human preference over high-level trajectories. The authors also present regret bounds for the proposed algorithms.\n\nSTRENGTHS:\nThe paper studies a combination of very important topics: theoretical understanding of hierarchical reinforcement learning and the outcomes of incorporating human feedback. I believe that the field still lacks a strong theoretical understanding of HRL, and this paper provides a step in that direction.\n\nThe paper is generally very well written and easy to follow. The authors clearly state the setting they are dealing with, provide a useful running example, and intuitive discussions about their theoretical results.\n\nWEAKNESSES:\nThe results are based on a very strong assumption that there is a well-defined goal function available (that is, a goal function that only proposes feasible goals). It is very hard to guarantee goal feasibility, so assuming that this function is given is very ambitious. However, the authors do recognize the limitation of this assumption and propose, on a high-level, how this could be dealt with in future work. \n\nWhen discussing the bounds related to the second algorithm (Section 4), it was necessary to introduce several constants, but they were not discussed in more detail, providing, for example, intuitions on the magnitude of such constants. Without it, it is more challenging to understand the significance of the bounds.\n\nQUESTIONS:\nThere were a few variables/subscripts used throughout the paper that, I believe, were not formally introduced. As some examples, $a$ on line 60, $h$ and $i$ on the Interaction Protocol paragraph (no line numbers here), $l$ on line 67, $\\bar{r}$ on line 179, $N(s,a)$ on line 194, $d$ on Lemma 4. Could the authors please add clarifications about these variables?\n\nLIMITATIONS:\nThe authors discuss the limitations of the work. No major concerns.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}