Illustrated Landmark Graphs for Long-horizon Policy Learning
Authors: Christopher Watson
Arjun Krishna
Rajeev Alur
Dinesh Jayaraman
Venue: Accepted by TMLR ()

Abstract: Applying learning-based approaches to long-horizon sequential decision-making tasks requires a human teacher to carefully craft reward functions or curate demonstrations to elicit desired behaviors. To simplify this, we first introduce an alternative form of task-specification, Illustrated Landmark Graph (ILG), that represents the task as a directed graph where each vertex corresponds to a region of the state space (a landmark), and each edge represents an easier to achieve sub-task. A landmark in the ILG is conveyed to the agent through a few illustrative examples grounded in the agent’s observation space. Second, we propose ILG-Learn, a human in the loop algorithm that interleaves planning over the ILG and sub-task policy learning. ILG-Learn adaptively plans through the ILG by relying on the human teacher’s feedback to estimate the success rates of learned policies. We conduct experiments on long-horizon block stacking and point maze navigation tasks, and find that our approach achieves considerably higher success rates (~ 50% improvement) compared to hierarchical reinforcement learning and imitation learning baselines. Additionally, we highlight how the flexibility of the ILG specification allows the agent to learn a sequence of sub-tasks that is better suited to its limited capabilities.

Published in Transactions on Machine Learning Research (03/2025)
Illustrated Landmark Graphs for Policy Learn- ing
Christopher Watson
ecuatson&seas upenn. edu
Deportment of Computer and Information Science University of Pennsyloania
Arjun Krishna
arjk wrjk@scausuupen..c edu
Department of Computer and Information Science University of Pennsylaniia
Rajeev Alur
alur@cis upenn. edu
Department of Computer and Information Science University of Pennsyloania
Dinesh Jayaraman
dineshj upenn. edu
Department of Computer and Information Science Universily of Pernsyloania
Reviewed on OpenReview: https: // openreview. net/ form?? id= 0ADUWC4,s88
Abstract
Applying learning-basee approaches to long- horizon sequential decision-making tasks re- quires a human teacher carefully craft reward functions or curate demonstrations to elicit desired behaviors. To simplify this, first introduce an alternative form of task- specification, Illustrated Landmark Graph (ILG), that represents the task as directed graph where each vertex corresponds to region of the state space (a landmark), and each edge represents an casier to achieve sub-task. A landmark in the ILG is conveyed to the agent through few illustrative examples grounded in the agent's 's observation space. Second, we propose ILG-ne a human in the loop algorithm that interleaves planning over the ILG and sub-task policy learning. ILG-Learn adaptively plans through the ILG by relying on the human teacher's feedback to estimate the success rates of learned policies. We con- duct experiments
long-horizon block stacking and point maze navigation tasks, and find that our approach achieves considerably higher success rates (se 50% improvement) com- pared to hicrarchical reinforcement lcarning and imitation learning basclines. Additionally, we highlight how the flexibility of the ILG specification allows the agent to learn sequence of sub-tasks that is better suited to its limited capabilities.
1 Introduction
How can a human best teach an agent to perform new long horizon task? The two most popular classes of approaches today are reinforcement learning (RL) and imitation learning. In RL, (Sutton & Barto 2018) the teacher specifies the task via reward function that assigns higher scores to more desirable environment configurations. The agent then interacts with the environment to Iearn how to achieve high rewards. For long-horizon lasks, the efficiency of this trial-and-error exploration depends intimately on the human teacher's ability to design well-shaped reward function that encourages incremental progress towards the final goal [Laud] (2004] Ng et al., 1999 (Sowerby et al.) 2022: Gupta al. (022). As concrete example, let us consider the block stacking task (StackChoice) shown Figure [ The task is build a tower; the agent may choose to place block A (red) on block B (green) vice versa. The task itself is fully specified by

---

Published in Transactions on Machine Learning Research (03/2025)
start
grasp
stack
StackAB ILG structure
grasp A
B
start
stack
grasp
B
Initial layout
Grasp B
Grasp
StackChoice ILG structure
(Outwardview visualization) (Outwardview perspective)
Figure 1: Left: Illustrates the Stack task. At the start of each episode, block A (red) starts closer to the robot than block (groen). Both blocks initialize at randomized positions along the centerline of the table. In the Dutwardview condition, block A initially occludes block B. Right: ILG structure for variations of the Stack task. The StackBA IT.G defined analogously.
simple "sparse" reward function that is 1 when the blocks are stacked and otherwise. However, in practice, the human teacher must write more sophisticated reward function that provides "dense" rewards during task exccution. For example, it might incorporate small positive rewards for reaching various milestones lowards the task: moving the gripper close to the block, grasping, lifting, aligning, and finally placing the block. To write such a reward function, the human teacher must be able to interpret the sensor readings available the learner and prudently balance the weight given to cach term in the reward function. general, reward engineering difficult, error-prone and requires considerable expertise, (Booth et al. 2023 Amodei et al. 2016 Skalse al. 2022 limiting RT's applicability long-horizon tasks.
Imitation learning (Hussein et al. 2017 Zare et al. 2023 Ravichandar et al. 2020 allows the human to teach the agent by demonstrating desired behavior. Compared to RL, imitation learning shifts the teaching burden from reward engineering to demonstration. Although demonstrations are an intuitive form of task specification, providing high-quality demonstrations may not always be feasible (Ravichandar et al. 2020 The teacher might not know how best to perform a task, how manually operate the agent to do so, e.g. there may be no good interfaces to mamually operate biped robot to run smoothly. Even without these problems, demonstrations need to teleoperate the robot and gather many demonstrations to adequately cover the states that the agent may encounter during deployment. Finally, demonstrations must account for the learner's limitations: e.g., stacking the blocks in one order may be easier for the robot to perform than the other, because of limitations on its sensing, perception, or actuation abilitics which are sometimes hard to characterize in advance.
be cumbersome to provide: in the StackChoice task, the teacher would
Reward functions and demonstrations occupy opposite ends of a design spectrum: a reward function specifies what must be achieved (perhaps without much intermediate guidance), while a demonstration specifies how to achieve it (perhaps too prescriptively). Ideally, the teacher could provide specification that balances the declarative nature of's reward function (which exposes opportunities for the learner to customize ils approach according to its capabilities) with the prescriptive nature of demonstration (which provides fine-grained guidance). To address this gap, we propose the illustrated landmark graph (ILG) as a form of long-horizon task specification that is inspired by the use of temporal logic to structure reinforcement learning (Li et al. 2017 Jothimurugan al. (002)). Each vertex of an ILG represents an intermediate landmark, which is a subset of the environment' state space, analogous an atomic proposition of temporal logic specification. To show the meaning of a landmark to the learner, the teacher must be able to provide handful of illustrative observations drawn from states within the landmark. Each directed edge (u, v) in the ILG represents the edge task of transitioning the environment from the landmark represented by to the landmark represented by v. Each sink vertex represents a final landmark: The long horizon lask is to reach any final landmark, passing through intermediate landmarks on the way. Importantly, the ILG can contain multiple paths to final landmark, exposing opportunities for the learner find plan that suits its capabilities. Returning to our block stacking task, human teacher can identify that grasping a block before placing is a useful landmark and can communicate this to the learner via the StackChoice ILG shown in Figure
2

---

Published in Transactions on Machine Learning Research (03/2025)
To leverage the ILG for policy learning, we propose ILG-Learn. At high level, ILG-Learn interleaves Dijkstra-style planning over the IT.G with example-based control, (Fu et al. 2018 Singh et al. 2019 Eysenbach et al. 2021) a lightweight alternative to imitation learning that uses goal observations instead of full-length demonstrations, to learn an edge policy for each edge task. This allows the learner to benefit from temporal abstraction, as in hierarchical reinforcement learning, (Hutsebaut-Buyaee et al. 2022) without the need for reward engineering. Concretely, to train an edge policy for "start -> grasp B", the teacher would provide illustrative observations by manipulating the gripper to grasp block B, then letting the learner observe the grasp with its onboard sensors. Example-base based control lets the learner use such illustrative observations in lieu of a reward function as it interacts with the environment to learn a control policy to grasp block B. Beyond providing the ILG and its associated illustrative observations, the ILG-Learn teacher has to only respond to binary success/ aaccess/faiilure /faluure queries during training to inform the learner's graph search. This intuitive teacher-learner interface allows the learner to benefit from exploration without requiring reward engineering.
A key benefit of the IL. specification is the ability to represent multiple paths to overall succoss. The StackChoice IL. includes multiple paths to the final landmark "educk""ddppeddnnng on the learner's capa- bilitties it may be easier stack the blocks in one order the other. For example, if the agent were to perceive the world using an outward-facing camera mounted opposite the robot on the surface of the table (as shown in Figure D, block B would initially be occluded, making it easier to start by grasping the (unoccluded) block A. By querying the teacher for success /failure feedback, the IL.G-eear learner can focus exploration along the most promising paths. Our experiments show that ILG-Learn adapts to the occlusion condition and learns a policy that grasps the unoccluded block first. This capacity for train-time adaptation reduces the burden on the human teacher to predict which approach will be most feasible for the learner. In this way, the ILG differs from the symbolic abstractions used for Task & Motion Planning, (Zhao et al. 2024 in which the human must define which transitions are feasible before planning begins.
Another benefit of the ILG is that the teacher can choose a landmark density (the number of intermediate vertices along a path in the ILG) that is dense enough to guide efficient exploration yet not so dense as to be overly prescriptive. At one extreme, the teacher could provide a landmark graph that comprises a single edge, thus recovering the free-form exploration of ILG-Learn's 's underlying example- basee control algorithm. At the other extreme, the teacher could identify many intermodiate landmarks to tightly scaffold exploration. The ideal landmark density depends the task of interest; we believe that the intuitive structure of the ILG will enable the teacher to apply domain knowledge to select an appropriate landmark density without concerning themselves with the low-level details of the robot's 's perception and motor capabilities. Our experiments show that ILG-n outperforms basclines that use end-to-end cample-based control (which embodics minimal landmark density) and behavior cloning (which uses full-length demonstrationss.. In summary:
We introduce the II.G as a form of task specification that allows a human teacher to decompose a task into a series of easier to achieve landmarks, thereby providing intermediate guidance without reward engineering, full length demonstration, or explicit knowledge of the learner's capabilities.
We propose ILGG-Tearn, a learning algorithm that leverages the ILG's interpretable-by-dee to allow the teacher to provide lightweight guidance as the learner learns policies and a plan to suit its unique capabilitios.
nature
We empirically show that providing a suitable ILG enables ILG-Learn to learn that achieve significantly (~ 50%) higher success rates compared to approaches that do not receive structured guidance (RCE and behavior cloning). Additionally, we demonstrate how a multi-path ILG facilitates learning when the teacher cannot fully anticipate the learner's capabilities.
policies
2 Related work
Hierarchical reinforcement learning, Task & Motion Planning. Hierarchical reinforcement learn- ing (HRI.) (Hutscbaut-Buysse et al.] 2022 Pateria et al.) (021) incorporates temporal abstraction into reinforcement learning. effectively decomposing a long horizon task into easier to achieve sub-tasks. Prior works (Kulkarni et al.) 2016 Nachum et al.] 2018) have shown that incorporating some human priors for
3

---

Published in Transactions on Machine Learning Research (03/2025)
temporal task docomposition enables RL algorithms to find useful abstractions to solve long-horizon tasks. The IT.G specification format allows the human teacher to provide such abstractions; at a high level, each edge in the ILG is analogous to an option in the options framework (Sutton et al.) 1999). Unlike existing
HRL techniques, ILG--earn uscs example-based control and teacher-learner interaction to avoid the neod for reward engineering and explicit specification in terms of environment. states.
In robotics, Task & Motion Planning (Garrett et al. 2021; Zhao et al., 2024) is popular framework to decompose long-horizon tasks into high-level symbolic task planning and low-level continuous motion optimization. Typically, TAMP system designer must specify task-relevant abstractions such motion primitives, their affordances, and constraints, etc., to synthesize task and motion plans under the assumption that the required state is fully observable and the effects of chosen sub-task and action sequences can be modeled deterministically. Contrary to this, to create an ILG the teacher only needs to specify various potentially feasible sequences of sub-tasks and ground them via illustrative observations. During training, IL.G-Laarn uses the ILG to scaffold exploration and discovers a sequence of landmarks that the agent can feasibly achieve based on its capabilities. This directed exploration arrive at the effective task plan is reminiscent of Go-Ex Explore (Ecoffet et al., 2021]. ILG-Learn differs from Go-Explore in that landmarks serve as the task specification (avoiding the need for reward design), include human-defined relationships (in the form of the ILG), and are illustrated to the learner via seacher-pooiidee observations (rather than reached organically).
Specification-guuieed reinforcement learning and reward machines. ILG-Learn's 's use of the human- specified ILG is inspired by specification-gudder reinforcement learning, in which a human provides a temporal logic specification that, can be used to structure hierarchical reinforcement. learning (Jothimurugan el al. 2021 Araki et al. 2021] or reward function generation (Li et al. 2017 Bozkurt et al. 2020). The main benefit reinforcement learning is that some (or all) reward engineering effort can be replaced with logical specification, which may be more intuitive and less error prone. A specficatioon-eriiee reward function can be expressed as a reward machine (Toro Icarte et al. 2022), which is finite-state automaton whose current state determines the reward function. Both the ILG and the reward machine expose the symbolic structure of a multi-step task to learner. Unlike an ILG, however, a reward machine does not include a distinguished "final goal" state; maximizing accumulated reward may or may not require reaching a particular reward machine state. The ILG exposes the high-level objective (reach a final landmark) to the learner, who must combine graph-based planning and example-based control to obtain local guidance. On the other hand, a reward machine immediately exposes local guidance to the learner (in the form of the initial reward machine state's reward function) and the learner may leverage the reward machine's 's symbolic structure to guide exploration as it seeks to maximize accumulated reward.
of specification
T..G-Leam's use of the IL.G is directly inspired by DiRL's (Jothimurugan et al., 2021 use of the analogous abstract graph, which is built according to the structure of a temporal logic specification. ILG-Learn and DiRL both use the specification's structure to scaffold train-time planning; this contrasts with the approach of Yalcinkaya et al. (2024), which uses a neural network to produce an embedding of the specification serves as input to a goal-conditioned policy. The ILG-definable specifications are subsumed by linear temporal logic; ILG-Learn can equivalently be seen as a specífication-guidee reinforcement learning algorithm in which the learner must infer the semantics of each atomic predicate (landmark) from a set of illustrative observations and active success /failure queries.
Imitation learning and example-based control. As we argued in Section [ the price of imitation learning's intuitive teacher-learne interface is rigidity: the learner is vulnerable to compounding deviations from the teacher's demonstrations (Ross et al. 2011 and may not enjoy the flexibility to adjust its approach to suit its own unique cipabilities. To address these challenges, inverse reinforcement learnông (Arora & Doshi 2021 Adamss et al.) 2022) gleans the demonstrator's intent, allowing the agent to learn a policy through environmental interaction rather than rote memorization. In the context of long horizon policy learning, UVD (Zhang et al. 2024) and Relay Imitation Learning (Gupta et al. (020) identify subgoals within demonstrations that can be used for compositional imitation inspired by goal-conditioned RL. Compared to
ITG-Leaar, these approaches remove the need for the teacher to identify landmarks, but in doing reduce
4

---

Published in Transactions on Machine Learning Research (03/2025)
the teacher's ability to scaffold exploration and require the teacher to provide demonstrations rather than illustrative observations.
Brample-based control (Fu et al., 2018 Singh et al. 2019 Eysenbach et al. 2021 allows the teacher to provide single-timestep examples of the environment state after task completion rather than full-length demonstrations. Tt. is often significantly easier for a human to provide a single-timestep example than full demonstration, and sparse guidance allows the learner more flexibility. However, for long-horizon tasks, such low landmark density may not provide the learner enough guidance to complete the task. At high level, our proposed method, ILG-Learn, scales example-based control to long-horizon tasks by incorporating human exploration biases in the form of intermediate landmarks. We use the example-basod control algorithm RCE [Eysebbach et al. 2021 to learn each edge policy that transitions the environment between landmarks.
Human-in-the--le policy learning. Reinforcement learning with human feedback (RLHF) (Kaufmann ct al. 2024 Arzate Cruz & Igarashi [0200] Najar & Chetouani] 2021) elicits guidance from human during training reduce (or completely eliminate) reliance upon the environment's reward function. Some recent applications of RLHF to robotics query human preference over states discover subgoals for HRL (Zhoul et al. 2024) goal-conditioned exploration (Villasevil et al. 2023). Instead of using human preference lo discover landmarks, ILG-Learn requires the teacher to choose which landmarks to include in the ILG before learning begins. ILG-Learn then incorporates human foedback in the form of success saccoss/failure /failure queries to direct exploration towards paths that cater to the learner's capabilities. ILG-Learn's 's use of active queries to steer the learner away from undesirable behavior inspired by VICE-RAQ, (Singh et al., (019) an example- based control algorithm that targets pnccess/failure queries on states that closely resemble human-provided examples, which prevents inferred reward function from encouraging progress towards spurious goals.
Human feedback can also be used to facilitate imitation learning. For example, HI-IRI. (Pan & Shen 2018) requires the human to provide subgoal states alongside a full-length demonstration. During training, the learner can ask for focused demonstrations of particularly hard-to-learn subgoals. Another recent approach, Thrifty DAgger (Hoque et al. 2021) quantifies novelty and risk to make the most of a limited human interaction budget. Another approach the DAgger (Ross et al., 2011] family, RLIF (Luo et al. (024)) lets the human intervene during training tell the learner when it behaves unsatisfactorily and provide a short demonstration of better behavior. The aforementioned techniques begin with a full-length demonstration of the task, which contrasts with our ILG's ability express multiple paths to success. In similar vein, Memarian et al. (2020) combine automata learning and inverse reinforcement learning to discover subgoal decomposition alongside policy to complete a long-horizon task. Their approach allows the learner lo ask for specific guidance while discovering the subgoal decomposition but is not immediately applicable to continuous-state environments with difficult control tasks.
3 Illustrated landmark graphs
We introduce the illustrated landmark graph (ILG) as a form of long-horizon task specification. key feature of the ILG is that the teacher communicates the meaning of each landmark to the learner using illstrotive observations and does not need provide explicit definitions in terms of the environment's state, In Section 3.1 define the environment and policy model before defining the ILG specification format
in Section 3.2 The ILG is a useful form of specification because of the way il facilitates leacher-learner
interaction; we describe our proposed interaction interface in Section 3.3
3.1 Preliminaries
As is standard in RL, we assume the agent interacts with an environment that can be expressed as a Markov Decision Process (MDP). An MDP comprises a continuous set of states S, a continuous set of actions A, dynamics governed by the transition probability distribution p(s s;, a,), and a starting state distribution n. Instead of specifying the task via a reward function (as commonly done in MDPs), we will use an ILG the task specification and incorporate human interaction to guide the learner.
5

---

Published in Transactions on Machine Learning Research (03/2025)
At each timestep i, the agent receives an observation o(s:) E Ω where Q is the observation space. The observation function S + Ω captures the learner's perception capabilities. In real-world setting, o(s:) may represent raw semsor readings or the outputs of a state-estimation module. A policy is a function I : (Ω x A)" x Ω ((A) that maps a trace of observations and actions to a distribution over next actions.
3.2
ILG and satisfaction
Each vertex of an ILG represents a landmark, which is a subset of the state space S, and each edge in the the ILG represents the edge task of transitioning between two landmarks. The ILG must have at least one sink vertex; each sink corresponds to final landmark. The agent's goal to reach a final landmark.
Formallly an ILG is tuple (U, E, uo, 9) with vertex set U, directed edge set E, distinguished source verlex uo: and landmark map B U + 2 that maps each vertex to the set of states that comprise the corresponding landmark. An ILG must have at least one sink vertex. A trajectory & 8ydg$101 ... satisfies the landmark graph E, no, 3) iff there exists a sink vertex u E U and a time such that 8, E 3(u).
Figure | shows the structure of the StackChoice ILG. There are two source-to-sinee paths: "start + grasp + stack" and "start - grasp + stack." Any trajectory that passes through a state the landmark represented by the sink verlex "slack" satisfies the specification. The vertices "grasp A" and "rrsp 8" provide intermediate guidance to policy learner. This guidance takes the form of teacher-learner interaction during training as detailed in the following subsection.
3.3 Interaction and illustration
As is common in RL, we assume that the learner does not begin with knowledge of the environment's transition probabilities; it must learn about environment dynamics through interaction. Similarly, we assume that the learner does not have explicit, access to the II.G specification's landmark map : - 2 During training. the learner begins with access to the structure (U, E, uo) of the ILG (U, E, uo, uo, B). In order to obtain guidance toward the landmark represented by a vertex u, the learner must interact with the teacher to request ilustrative observations drawn from states within ((u). The learner may also request success success/failure feedback from the teacher to see if the current state of the environment lies within a particular landmark.
During training, the learner interacts with the teacher and with the environment using the following proce- dures. Training proceeds as a sequence of episodes. At each timestep t within an episode the learner receives an observation ((s)) and may:
reset() Reset the environment to a state 80 v.. The learner receives the observation o(so).
step(a.) Provide an action a the environment, which causes the environment transition to a state 1 a.)) The learner receives the observation o( 0(s(41).
request[llusttraiioouu.. where U is the vertex of the ILG to be illustrated and the second argument P U* lets the the learner tell the teacher which path it wishes to extend to u. In response, the teacher provides a dataset of illustrative observations O, C {o(s) E 2 E B(u)) of illustrative observations of states within 3(u). Although the path does not affect the contract that the teacher must uphold, it may help the tcacher choose a useful set of illustrative observations: in the StackChoice example, a good leacher would respond to pmpuestIllustrral grasp A) with examples of block A stacked on top of block B (not of B stacked on A).
(slack, starl +
querySuccess(u) where U a vertex of the ILG. The teacher provides binary success/failure feedback of whether the current environment state s, is in the landmark 3()).
Given an MDP M, observation function o, and ILG G the learner tries to learn a policy π that maximizes the probability that a trajectory & drawn from æ interacting with M while receiving observations according to satisfies G.
Importantly, the teacher is never asked to explicitly define the landmark map B. Instead, the teacher must be able to provide set of illustrative observations for each landmark and serve as an oracle for querySuccess.
6

---

Published in Transactions on Machine Learning Research (03/2025)
The teacher can provide set of illustrative observations by, for example, physically positioning robot and allowing its sensors to perceive the environment. This intuitive interaction interface allows the teacher to guide the learner without needing to understand the low-level details of the learner's perception capabilities.
Underlying the ILG's usefulness is its customizability. To permit multiple approaches to complete the task, the human can include multiple paths that. lead to a final landmark. In addition to using branching to expose options for high-level planning, the teacher can adjust the landmark density (the number of intermediate landmarks) along each path to provide the appropriate granularity of guidance. In the context of policy learning, low landmark density cedes resolution of low-level control details to the underlying learning algorithm. This is both convenient for the teacher and can result in well-optimized motion if the learner can successfully transition between landmarks. On the other hand, excessively low landmark density can render policy learning intractable, as the underlying learning algorithm is left with insufficient exploration bias.
4 Learning algorithm
We propose ILG-Learn, numan-in-the-loop approach to policy learning for ILG specifications. At a high level, ILG-Learn performs the following steps:
Policy learning. For an edge (u, v) of the ILG, the learner uses example-based control to learn an edge policy T(u,v) that tries transition the environment from the landmark represented by u the landmark reprosented by
Success estimation. Given a learned path policy (a sequence of edge policies), the learner queries the teacher for binary success/failure feedback to estimate the success probability of the policy.
Planning. The learner tries to find a source-to-sink path of minimal cost, where cost. is defined as the negative logarithm of the associated path policy's success probability.
At the outsel, the learner receives the structure (U, F, uo) of the IL.G specification (U, uo, (3) but does not know how reach any landmark. By interacting with the environment and the teacher (through the interface described in Section 3.3) the learner learns policies to reach intermediate landmarks and plan to satisfy the ILG specification by reaching a final landmark (represented by sink vertex). ILG-Learn iteratively builds tree of lowest-cost. known paths from the source to the other vertices of the ILG. Since the cost of a path depends on the success probability of the learned edge policies, ILG-Learn interleaves graph search and policy learning to obtain the path costs on the fly during training. Learning proceeds as a series of learning infervals, each of which focuses on a single edge the TL... Once LT.G-Learn finds the lowest cost path p to a sink vertex, ILG-Learn returns the plan p and the associated path policy.
Algorithm Sketches the full ILG-Learn algorithm and Figure iillustrtte the steps of learning interval. In the following textual description will denote the ILG specification (U, E, uo, 3) and assume the learner has access to the environment M and the human teacher through the interface described in Section 3.3
To make our description concrele, we will use the StackChoice task introduced in Section 1 as a running example. The structure (U, E, uo) of the StackChoice ILG is shown Figure !
TLG-Learr hyperparameters
illustrationCount:: illustrative observations per request. episodeLength: fixed horizon for edge policy training.
intervalLength: environment steps per learning interval (exclusive of those used during resets). estimationQueries # rollouts (each with snccess/failurr query) at end of each learning interval. success Thrcsholdd lower bound on success probability for an edge to be considered learned. intervalsLimit: max # of learning intervals per edge.
Exploration order (selectEdgo). ILG-Leamn's exploration order is determined by the selectEdge sub- routine. At the outset, IT.G-Learn's tree of known paths contains only the "start" vertex uo. To extend this
7

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm 1: ILGG-Learn
Input: ILG structure (U, E, uo), access to MDP M via reset and step, access to human teacher via requestIllusratioo. and queryšuccess.
Output: Path p and associated path policy
Pust Tuns reachProbur + [], None, 1;
reachProb, +- 0 Yu E V {uo); while selectEdge() returns (u, v) do
(v)) + requsstIlustaationn
(u,,v) - Pe Tu)i < sequencePollcceee(f,,. (i.
successProb + essiaaterobbbblll if successProb reachProb, then reachProb, successProb;
Pv Pu (u, (u,u). v);
TT + π;
< argmax Vertines(G)) (reachProb,.) return Pu. Tu
1.
2.
4.
(1) The learner selects an edge and requests illustrative observations to guide learning. (2) The human teacher provides illustrative observations the destination landmark. (3) The learner uses example-based basod control to learn an edge policy from environmental interaction. (4) The learner executes the edge policy and queries the teacher for success/faili ffaluure feedback.
Figure 2: Stages of a learning interval:
tree, selectEdge chooses an edge leaving the tree for which learn control policy. ILG-Learn implements fine-grained interleaving (mediated by the success TrrcesssTrresolld and intervalsLimit hyperparameters) of policy and plan learning to accelerate learning. For details, see Appendix A
Let us suppose that selectEdge chooses the edge (u, v) = (slarl, grasp A). This marks the start of a learning interval dedicated to learning the edge policy T(u,))-
Human guidance (requestGoals). After selecting an edge (u,v), Learn asks the human teacher to provide a dataset O containing illustration illustrative observations drawn from states within ((r). In our example, the teacher would provide set of observations that illustrate "grasp A".
Policy learning (learnPolicy). ILG-Learn runs example based control for intervalLength environment steps (structured as episodes of episodeLength timesteps) to learn an edge policy T(a,n)- This learning process does not require interaction with the human teacher.
In our example, the learner would interact with the environment learn a policy that tries to reach states that yield observations that look similar to the illustrative observations of the "grasp A" landmark. This similarity is quantified by the example- control algorithm. Our implementation uses RCE, (Eysenbach ct al. 2021) which is inspired by actor-critic reinforcement learning. In RCE, the critic network is trained to predict a lime-discounled success probability for cosseraltionactiinn pairs. Success is defined as a regression task: the success probability of illustrative observations is regressed to while other observations (collected in replay buffer) are assumed to not represent success. The critic's scores are used to optimize the policy, similarly the soft actor critic RL algorithm (Haarnoja et al., (2017). Since this the first edge along a path, each learning episode starts from the environment's reset. distribution.

---

Published in Transactions on Machine Learning Research (03/2025)
In future learning interval, ILG--e may try to learn the next step of the path, namely, how to stack block A on top of block B (assuming block A is already grasped). Since policy learning is sensitive to the starting state distribution, ILG-Learn needs to start learning *(u,r) from the distribution induced by the (alrcady learned) path policy Tu- So in the case of learning to stack block A on block B, each episode would start by executing the (starl, grasp A) policy (for the same fixed horizon episodeLength) to reach the appropriate starting distribution for the (grasp A, stack) edge task
Success Estimation (estimateeProabilityy The last step of each learning interval is to estimate the probability that the learned edge policy T(u,e) successfully reaches ((r)- Just as it was important to start training T(u,o) from the state distribution reached by executing Tu: it is important to evaluate T(u,o) from the distribution reached by Ny. The est.imatterobabilityy subroutine execules estimationEpisodees addi- tional rollouts of the sequenced policy, using querySucceas(u) at the last timestep of each rollout, to ask the human teacher whether the rollout was successful. If this empirical path probability excceds that of the bost known path Pe to v, ILG-Learn updates the best known path and associaled path policy Ne. If the empirical probability exceeds successTThreshen is reached) then ILG-Learn will consider the edge fully explored. At this point, the path cost (the negative logarithm of the success probability) is solidified. No more learning intervals can be allocated to (u, v). Future learning intervals can now be allocated to edges that leave v.
(or the per-edge max number of learning intervals intervals
Since success estimation only occurs at the end of each learning interval, the intervalLength hyperparameter mediates tradeoff between learning efficiency in terms of total environment steps and learning efficiency in terms of human annotation burden. A short intervalLength means that the estimateProbabilit subroutine is executed more frequently, which reduces the likelihood that superfluous training effort will be invested in an edge policy. However, est:imateProbailiie requires the teacher to respond to sucoess/failur queries. We provide advice for selecting ILG-Learn's hyperparameters in Appendix B
Termination. Training is complete when the selectEdge subroutine returns None, indicating that ILG- Learn has found the lowest-cost, path in the ILG and the associated path policy. Otherwise, new learning interval will commence, focusing exploration along the edge chosen by selectEdge.
5 Experiments
Our experiments show that ITG--eearn can learn policies to successfully complete long horizon tasks. To better understand the importance of allowing the teacher choose the ILG's landmark density we evaluate ILG-Learn against RCE, state of the art example-based control algorithm that does not use intermediate landmarks, and behavior cloning (BC), tions. We include two variants of BC, one which predicts actions using an MLP that is architecturally similar to ILG-Learn and RCE, as well as the modern diffusion policy (Chi et al. 2023) architecture. Since the ILG task specification is new, there do not exist direct analogs of ILG-Learn to baseline against; rather, RCE's examples and BC's demonstrations represent two alternatives to IL. specifications.
imitation learning algorithm that receives full-length demonstra-
We also compare against hierarchical reinforcement learning with reward machines (HRM), (Toro Iearte et al. 2022) an HRI. algorithm that exploits the structure of a handerafted reward machine that provides dense rewards. Our reward machine definitions mirror the structure of the ILG for each task; our experiments explore whether ILG-Learn's 's use of example-based control to learn each edge task can rival or exceed the efficacy of reinforcement learning with dense rewards.
Finally, we investigate whether ILG-Learn can discover the path through multi-path (branching) ILG that suits the learner's capabilities. Such train-time time adaptation would reduce the burden on the teacher to predict the best approach to task completion priori. We conduct our experiments in simulation using the following environments:
1 Steps used during resets do not contribute the IntervalLength steps per learning interval, however in our experimental evaluation we limit the total number of environment steps 10 million.
9

---

Published in Transactions on Machine Learning Research (03/2025)
Stack. Our Stack family of environments is customized robosuite Zhu et al. 2020 environment that simulates 7-DoF Franka Panda arm that receives 55-dimensional state observations and uses a 7- dimensional action space (which represents an operational space controller with fixed impedance). As intro- duced in Soction I the learner must stack the blocks to build a tower. The blocks are initially placed along the centerline of the table, with block A closer to the robot. than block B. The exact positions of each block are randomized. We include multiple versions of the task: StackChoice (stack the blocks in either order), StackAB (stack the red block A on the green block B), and StackBA (stack block B on block A). To study ILG- Learn's ability to adapt to a learner's unique capabilities, we further include the StackChoice wwwweeewwwwwwwviviieeeee and StackBA-Outwardyv
variants, which simulate observations collected from an object-detector operating from a camera mounted on the same side of the table as the robot arm. Whenever a block is occluded, we mask the corresponding components of the observation space. Figure iilustrates the Stack lasks; for more details see Appendix E
The "grasp A" landmark comprises the states where the robot gripper fingers are spread approximately the width of the block, and both fingers contact Block A. "grasp B" is defined analogously. The "slack" landmark comprises the states where the blocks are in contact and top of each other. During train- ing, a scripted "human" teacher provides illustration (10 in our StackAB experiment, and 50 in Sanekchhiiee-Outwarrriia
and StackBA (uttarreeo illustrative observations per requestCoals call. The scripted teacher always provides the same set of illustrative observations on subsequent requestGoals with the same desired vertex and path to extend. The scripted teacher responds to querySuccess call perfectly accurately, according to the landmark definitions defined in Appendix E
For the StackAB RCE baseline, we ran experiments with both 100 and 1000 goal examples, all of block A having been stacked on block B. These are drawn from the same distribution as the illustrative observations provided to ILG-Learn for StackAB's "Stack" landmark. Similarly, we ran StackAB BC experiments with 100 and 1000 full-length demonstrations gathered using a scripted policy that includes some random variation (sce Appendix B for details). Each demonstration is 200 timesteps in length, with the task typically being completed by around 150 timesteps.
Point Maze. We use custom layouts of the Point Maze environment from Gymnasium Robotics (de Laz- cano et al. 2023 Fu et al. 2020). The agent is a force-actuated point-mass mass with a 2-dimensional action space and receives 4-dimensional observations that comprise its position and velocity. The agent must navigate from its starting position in the lower-left to the goal position in the upper-right. We include three variants of a diagonal maze (DiagonalMaze3x3, DiagonalMaze5x6, DiagonalMaze7x77) that differ in the length of the task. We also include a DiagonalMazee777- CCorsse variant that has low landmark density and two variants of a 4x4 maze that differ only in their associated ILG specifications. Illustaatoons are shown in Figure 3 the landmark regions are the centers of the rooms that are highlighted in pink.
In each Maze variant, the scripted teacher provides ILG-Learn with 10 illustrative observation per landmark. Similar to the StackAB we conduct DiagonalMaze experiments in which the RCE and BC baselines 100 goal examples or demos (respectively), as well as experiments in which they receive 1000. The RCE baseline with 100 examples all drawn from the final goal region, and the BC bascline with 100 full-length demonstrations gathered by scripted policy.
5.1 Landmark density and learning objective
To disentangle the importance of allowing the tcacher to choose the ILG's landmark density from the importance of multi-path specifications, we first turn our allention to tasks specified with a linear ILG. In the StackAB environment (Figure D the ILG-Learn learner receives a linear ILG that mandates placing the red block A on the green block B. At the end of each learning interval (100k environment steps, exclusive of steps used during resets) the teacher responds to 30 success/faille /failure queries. We allow ILG-Learn and RCE to train for 10 million environment steps and allow BC to train until the validation loss stops decreasing. Appendix contains learning curves.
ILG-Learn achieves an average success rate of 0.975 for the StackAB task (Table D. This is markedly higher than the success rates achieved by the RCE baseline (0.050) or the BC baseline (0.059) when provided
10

---

Maze4x4-Coare Published in Transactions on Machine Learning Research (03/2025)
Table 1: Success rates (mean over 5 trials). ILG-Learn, RCE, and HRM are trained for a maximum million environment steps; see Appendix for experimental details and Appendix for learning curves.
ILa-L
RCE
RCE
BC (MLP)
BC (DifiP
HRM
(ours)
100 example
1000 ex.
100 demo
1000 demo
StackAB
0.975
0.050
0.392
0.059
0.28
0.005
DiagonalMaze3x3
1.00
0.00
0.600
0.991
0.91
1.00
Di DiagonalMaze5x55
1.00
0.00
0.200
0.989
0.68
0.00
DiagonalMaze7277
0.995
0.00
0.00
0.476
0.689
0.00
Diagonallaaze7x7CCoarse
0.00
0.00
0.00
0.476
0.689
0.00
DiagonalMaze3x3
DiagonalMaze7xx
DiagonaaMaaze777-COLL Mazetx4-Fine
Figure 3: The Maze environments with ILGs superimposed (pink). The agent starts the lower-left room and must navigate the upper-right. room. DiagonalMazebx6 analogous to the 3x3 and 5x5 variants.
with 100 goal examples or demonstrations, respectively, Even when provided with 1000 examples, RCE only achieves 0.392 average success rate, and BC (with the more modern Diffusion Policy architecture) only achieves 0.28 success rate. The fact that the RCF baseline achieves low success rate shows that single- frame observations are too sparse to guide the learner towards success; allowing the teacher to specify ILG with the intermediate "grasp A" landmark baseline, which receives very dense guidance towards the goal, yields policies that rarely succeed, showing that carefully mimicking a teacher's detailed demonstrations does not necessarily yield a good policy.
crucial for ILG-Learn's success. On the other hand, the BC
The DiagonalMaze family of enviromments shows that the teacher's choice of landmark density more important for tasks with longer horizons. DiagonalMaze-335 -x23, DiagonalMaze-5xx and DiagonalMaze-Tzz differ only in the length of the path that the learner must travel. Looking again at Table D that ILG-Learn consistently matches or exceeds the performance of the RCE and BC baselines. Importantly, BC achieves high success rate for the relatively short-horizon DiagonalMaze3x3 and DiagonalMaze6x5 but its performance deteriorates dramatically as the size of the maze increases to 7x7. On the other hand, ILG-Learn consistently yiclds near-perfect success rates regardless of maze size. This supports the intuition that ILG-Learn can scale long horizon lasks by (1) allowing the learner flexibility to explore and discover
1.
0.0
grasp A
2.
0.0
grasp A
3.
0.63
grasp
A
4.
0.63
grasp A
start
stack
start
stack
start
stack
start
stack 0.0
(grasp B
0.0
grasp B
0.0
grasp
grasp B
0.86
grasp
6.
0.86
grasp
0.10
7.
0.86
grasp
0.10
8.
0.86
grasp A
0.86
start
stack
start
stack
start
stack
start
stack
0.0
ggasp
grasp
0.0
grasp
0.0
grasp B
Figure 4: Learning intervals for StackChoice- Outwardview. in pink and results of estimateSuccess are annotated.
The most recently explored edge highlighted
11

---

Published in Transactions on Machine Learning Research (03/2025)
suitable policies to reach each landmark and (2) incorporating success/failure feedback during training to ensure each edge policy is learned successfilly before moving on to train successive policies. This is unlike BC and other imitation learning algorithms that are vulnerable to compounding deviations between the demonstrations and the actual trajectories taken by the agent.
To realize II.G-Tearm's benefits, the human teacher must provide an IT.G with appropriate landmark density. Given the excessively low landmark density of DiagonalMazz777--aarre,, ILG-Learn consistently fails to learn successful policy (while the ILG of DiagonalMaze7x7 yields near-perfect policies). Such a failure occurs when the exploration needed to learn an edge policy exceeds the capabilities of ILG-Learn's underlying examplc-based control algorithm. We provide advice for selecting landmark density in Section 0
To investigate the efficacy of example-based control as ILG-Learn's low-level policy learning algorithm, we compare the performance of ILG-Learn with that of HRM. In StackAB and each DiagonalMaze environment, we provide HRM with a reward machine in which each reward machine state corresponds to a landmark in our ILG specification. To encourage progress toward completing the long-horizon task, the reward machine provides dense rewards that encourage progress to the next landmark, which triggers transition to the next reward machine state when reached. The dense reward functions for StackAB are directly inspired by the original environmental reward of the original robosuite Stack benchmark. For the DiagonalMaze tasks, the reward is negative euclidean distance to the center of the next. landmark (see Appendix D.4 for details).
We find that for the relatively short-horizon task DiagonalMazee333, HRM achieves a perfect success rate of 1.00. However, HRM fails complete the larger DiagonalMaze environments. HRM also achieves low average success rate (0.015) for StackAB. While in-depth reward engineering could improve the performance of HRM, the fact that ILG-Learn achieves high success rates with an intuitive ILG specification and 10 illustrative observations per landmark suggests that our approach may be a viable alternative to hierarchical reinforcement learning for long horizon tasks.
5.2 Multi-path specifications
A critical attribute of the proposed ILG specification is that it can include multiple source-to-sink paths, each reprosenting an approach to complete the long-horizon task. To evaluate whether ILG-Learn can discover the path that best suits the learner's capabilities, we lurn to the in Section | and illustrated in Figure 1 The task is to stack the blocks (in either order). Importantly, the learner observes the position of each block via a simulated object detector based on a camera situated on the same side of the table as the robot, facing outward toward the blocks. Since the red block A starts closer to the camera than the green block B, the learner cannot detect the position of block B until block is moved break the occlusion.
task introduced
Table 2: Success rate (mean of 5 trials) of ILG-Learn given multi-path specifications.
StackChoice- OutwardView
StackBA-
Maze4x4-
Maze4x4-
OutwardView
Fine
Coarse
0.952
0.147
1.00
0.800
As expected, in all five trials (random seeds) of ILG-Learn, the learner acquired a policy that follows the path " "start > grasp A >> stack" achieving an average success rate of 0.952. Figure 4 illustrates the exploration process followed by the learner during one representative trial. Initially, the learner explores the ILG by alternating between allocating learning intervals to the (start, grasp A) edge as well the (start, grasp B) edge. However, after 5 training intervals (about 700 thousand total environment steps) the learner successfully acquires policy for the (start, grasp A) edge, and is thus able to reach the " "gmasp 1" landmark. ILG-Learn's best-first search heuristic (detailed in Appendix A) guides further exploration along this path, focusing on the (grasp A, stack) edge. At the end of the 8th learning interval (slightly more than 1 million total environment stops), the learner has acquired a policy that estimates has a success probability of 0.86. Since this exceods the speciiidd successThreshold of 0.8, ILG-Learn terminates. Note that in reality, the success rate of the final policy is 0.947 (estimated from 1000 rollouts)- discrepancy bet
12

---

Published in Transactions on Machine Learning Research (03/2025)
the learner's estimate of its own success probability can arise because (1) the learner only uses modest number (in this case 30, the value of the estimationQueries parameter) of success/ viccess/lailur failure queries and (2) the learner only queries the final timestep of each rollout while we allow success to occur at any point along the trajectory.
The adaptive exploration scaffolded by the II.G described above is crucial for the IT.G-Learn learner to successfully complete the task. To illustrate this, we compare against the StackBA-Outwardvier task, which differs from ceraechoic--Outwaardee forces the learner to start by grasping block B, which is very hard for the learner, since block B is initially occluded. As expected, ILG-Learn achieves low average success rate of 0.147 on StackAB- Outtrdvve.twww.vee...
only in the ILG provided to the learner.
If the human leacher were able to predict which path would be easiest for the learner to follow, the leacher could simply provide a linear ILG (in the the above example, "start + grasp A + stack"). However, such upfront prediction may pose a substantial burden to the teacher. A key benefit. of using illustrative observations to specify landmarks is that the teacher does not need to be intimately familiar with the learner's perception capabilities the same specification might even be used for multiple robots with different camera viewpoints. Moreover, using example- based control as subroutine to let the learner acquire policies to achieve each landmark frees the teacher from having to worry about low-level details of the learner's 's motor capabilities. To fully realize these freedoms afforded to the teacher, the high-level task specification must also be, to some degree, agnostic to the intricacies of the learner's perception, control, and policy learning capabilities. A branching ILG provides this freedom to the teacher: the teacher can provide an ILG to scaffold exploration and allow the learner to interact with the environment to discover a plan that best fits its unique capabilities.
6 Discussion
In Section we observed that IT.G-Learn can learn successful policies for manipulation and navigation tasks. In this section, we discuss how the ILG definition affects the feasibility and efficiency of learning, providing both advice for users of ILG-Learn and highlighting directions for future work.
Human effort. To apply ILG-Learn, the human teacher must provide an ILG decomposition that contains at least one path that can feasibly be learned via iterated example based control. Although allowing the teacher to provide branching ILG reduces the teacher's burden to predict the best landmark decomposition, some tasks may be hard for humans to effectively decompose. An important direction for future work is to conduct a careful user study, in the style of Cui et al. (2021): Biyik et al. (2020); Fitzgerald et al. (2023). to investigate the degree of human effort needed to specify an ILG that facilitates successful and efficient application of ILG-n We envision two future dircctions to reduce the burden of ILG definition: first, we propose to refine the IL. during training by prompting the teacher to provide new intermediate landmarks when the learner struggles. Second, we propose eliminate the need for manual ILG design altogether, instead allowing the human to specify the task using natural language (and perhaps a few final goal images demonstrations) and then prompting foundation model to construct formal specification (taking inspirations from works such as Lang2I.TL. (Liu et al.] 2023) and SayCan {Ahn et al. 2022)), which can in turn be compiled into an ILG (as in DiRL (Jothinurugan et al. 2021)). These directions are complementary; in the context of based sequential decision making, ADaPT (Prasad et al. (024) has shown that an LLM planner can decompose difficult tasks on-the-fly in response to the capabilities of a low level executor. We believe that itcrative refinement of the ILG by foundation model would improve the general applicability of ILG-Learn.
We also plan to incorporate foundation models to reduce the human annotation burden imposed by ILG- Learn's est istiatPrrobbbbii subroutine. For our experiments, we chose parameters such that the human responds to 30 success/ failure queries per 100k training sleps; for our StackChoice- -ottiiarvree experiment this amounted to an average of 360 active queries per trial. While developing ILG-nearn, we tried to reduce this dependence by using a small amount of human-annotated data to train a binary success classifier for each landmark in manner inspired by (Singh et al. (2019). We found that even with hundreds of labeled positive examples herd from successful executions of the policy being trained, there was often threshold on
13

---

Published in Transactions on Machine Learning Research (03/2025)
the output of the classifier neural net that reliably separated success from failure. In the future, we plan to incorporate foundation-model based success detector (e.g. Goko et al. (2024) to reduce the human annotation burden.
Subtask difficulty and learning efficiency. As established in Section 5 decomposing long-horizon task into intermodiate landmarks and providing a branching ILG that permits multiple paths to success allows ILG-Learn to solve long-horizon tasks. Applying ILG-Learn to a new task thus raises the questions "What is the appropriate landmark density?" and "How much branching should the II.G include?" At. a minimum, the teacher should design an ILG that is coarse enough that each landmark is meaningful to the human eye (since the teacher must respond to success faalurr queries). Beyond this, we recommend that the teacher err on the side of providing a dense IL.G with many landmarks and a large amount of branching. The best-first search pattern introduced in Section and detailed in Appendix A is designed to avoid exploring every edge in the ILG.
To gain empirical insight, we compared the performance and efficiency ILG-n on a 2D navigation task using a "finc" ILG (Maze4x4-Fine) versus a "coarse" ILG (Maze4x4-Coarse). The tasks are visualizod in Figure g and learning curves can be found in Appendix F In this case, either ILG suffices to learn a successful policy:
turn our attention to how much efficiency would lose by choosing the excessively fine decomposition of Maze4x4-Fine. Looking at the learning curves, it takes significantly more (compar- ing medians, 7x as many) total environment steps to learn successful policy in Maze4x4-Fime than in Maze4x4-Coarse. Although this loss of efficiency is regrettable, it is preferable to accidentally providing too coarse an ILG and having learning fail altogether.
Finally, we note the amount of training needed to learn a particular edge task varies between random seeds. To reduce this variability future work could develop example-based control algorithms that are sample efficient and learn more robustly. Additionally, in future work we plan to develop version of ILG- that leverages dense rewards from foundation model (laking inspiration from VIP (Ma et al. 2023b) and RoboCLIP (Sontakke et al. 2023)). Foundation- model based low-level policy learning also presents the opportunity to replace the role of illustrative observations with a natural language description of each landmark, which could be combined with rewards from a language-vision foundation model such as LIV (Ma) et al. 2023a)) to enable long horizon policy learning with neither reward engineering nor -roanndrovided goal observations.
Acknowledgments
This work is supported in part by NSF Award 2331783, NSF CAREER Award 2239301, DARPA TIAMAT (HR0011249)) 12490421), and a gift from AWS AT to Penn Engineering's s ASSET Center for Trustworthy AL. The authors would like to thank the TMLR reviewers for their valuable and constructive feedback.
References
Slephen Adams, Tyler Cody, and Peter A. Beling. A survey of inverse reinforcement learning. Artif. Intell. Rev., 4077-44337- aug 2022. ISSN 0269-2821. doi: 0..107//10466--01-110066 URL https: //doi.org//t 101011000001100220111000000091
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Tharz, Brian Tchter, Alex Trpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lec, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language in robotic affordances. In arxiv preprint arXiv:2204.0100 2022.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Conerete problems in ai safety, 2016.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan Decastro, Micah Fry, and Danicla Rus. The logical options framework. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 307 317. PMLR, 18-24 Jul 2021. URL hattps:/ppoceeddie
mlr wwt.p.ersss/v///eeeeee wttpps///1/////rra......aaaaaa hmmmll
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021. ISSN 0004-3702. doi: 2021.103500 URL
Christian Arzate Cruz and Takeo Igarashi. A survey on interactive reinforcement learning: Design principles and open challenges. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, DIS '20, pp. 1195 1209, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369749. doi: 0..1145/3357236
URL https:///oi. org/10. 1145/3357236 3335555352233
Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (ods.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1177-1190. PMLR, 30 Oct-O1 01 Nov 2020. URL https://ppocceddingg
mlr. wwwpsseeeeepaaeeeeeeeeeeeeeeeeee ik20a. html
Serena Booth, W. Bradley Knox, Julie Shah, Scott Nickum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifications. Proceedings of the AAAI Conference on Artificial Intelligence, 37 (7 5929, Jun. 2023. doi: 10.1609/aam v3.5522.......557333. URL https://ojs.saaat wawii.io.aaaaaa.oooaaaa/aaaa.... org/ http:///.aaaaappp/////aaaaaaaaaa
Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, and Miroslav Pajic. Control synthesis from linear tem- poral logic specifications using model-free reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10349 10355, 2020. doi: 10. 111111991/10 /00/ICRAA4945..20 0000044..0009996679999
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adain Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: compos- able transformations of Python-NumPy programs, 2018, URL http://github com/ com/google/jar
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 027836492411236668 2023.
Yuchon Cui, Pallavi Koppol, Henny Admoni, Scott Nickum, Reid Simmons, Aaron Steinfeld, and Tesca Fitzgerald. Understanding the relationship between interactions and outcomes in human-in-the-loop ma- chine learning. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4382 4391. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 20.24963/jjcai.22 2591/599. URL https: http///i...orr//../////////aaaaa 2021/599 Survey Track.
Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Tee, and Jordan Terry. Gymnasium robotics, 2023. URL htttp://iithb. woom//arraa-Fouddaiiooo
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):550 586, Feb 2021. ISSN 1476-4687. doi: 10.1038 111166/41158--0000011----- URL
Benjamin Eysenbach, Sergey Levine, and Ruslan Salak Replacing rewards with examples: Example- based policy search via recursive classification. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (cds.), Advances in Newral Information Processing Systems, 2021. URL https://openrevier: net//forum?iddUVVNd...
Tesca Fitzgerald, Pallavi Koppol, Patrick Callaghan, Russell Quinlan Jun Hei Wong, Reid Simmons, Oliver Kroemer, and Henny Admoni. Inquire: Interactive querying for user-aware informative reasoning. Tn
15

---

Published in Transactions on Machine Learning Research (03/2025)
Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pp- htmm
PMLR, 14-18 Dec 2023. URL,
https://proceeeding" weriiiinnnmm......mmm wwr.press.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. In S. Bengio, Н. Wallach, H. Larochelle, K. Grau- man, N. Bianci, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL. htttp::/prooceddngs.c www.ggs..eurpppo. wttp:ppscccpppppreeeeeeeeeeeeeee
Paper. pdf
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasots for deep data- driven reinforcement learning, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. URL http: http://proceedings. mlr. httppree///0ffffioooooooo//./
Caelan Reed Garreli, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4(1):265 293, 2021.
Miyu Goko, Motonari Kambara, Daichi Saito, Seitaro Otsuki, and Komei Sugiura. Task Success Predic- tion for Open- Vocabulary Manipulation Based on Multi-Level Aligned Representations. In 8th Annual Conference on Robot Learning, 2024.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learn- ing: Solving long-horizon tasks via imitation and reinforcement learning. In Leslic Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1025-1033 PMLR, 30 Oct- 01 Nov 2020. URL https://proceedding: mlr. wtteee//vvoo//gppa2oc. html
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shap- ing: Understanding the benefits of reward engineering on sample complexity. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 15281-15208 Curran Associates, Inc., 2022. URL httpss iewpeeeeedddieeeeenn..ee...e neurips. cc/ Paree/ pdf
paper_
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. Deep Reinforement Learning Symposium, 2017.
Jonathan Heck, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2024. URL htttp://githbb com/google/ffax
Ryan Hoque, Ashwin Balakrishna, Ellen R. Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning, 2021. URL https::/api. swmanticscholaat. org/Corpus D 237371142
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv., 50(2), apr 2017. ISSN 0360-0300. doi: 10.1145/3054911 URL
https://doi.org/ 10. 1145/3054912
Matthias Hutscbaut-Buysee Kevin Mots, and Steven Latré. Hicrarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1):172-221 2022. ISSN 2504-4990. doi: 10.3390/ 10.3390/maaee401 maa///ak4000000/ 0009. URL http::tt///// http://wwwaai.ccc////44444099//0
16

---

Published in Transactions on Machine Learning Research (03/2025)
Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, and Rajeev Alur. Compositional reinforcement learning from logical specifications. In Marc' Aurelio Ranzato, Alina Beygekimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NearIPS 2021, December 6- 14, 2021, virtual, pp. 10026 10039, 2021. URL
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feodback, 2024.
Πya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2022. URL
https::/ggtthub.comm// http://ww...oo//kotttoooo////a/a v2.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacodi, and Josh Tenenbaum. Hicrarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg, Dart: Noise injection for robust imitation learning- In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.). Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 143 156. PMLR, 13 15 Nov 2017. URL htttps://proceedingss mlr. wtw.ppesss//7111skkellllll htmll
Adam Daniel Land. Theory and application of reward shaping in reinforcement learning. PhD thesis, University of Illinois at Trbana-Champaija USA, 2004. AAT3130966.
Xiao Li, Cristian-loan Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3834 1839.3339.20 LEEE Press, 2017. doi:
FEROS..017.82062333 URL https: org/ 1ttp//10..1999/RRI0
.2266244
Jason Xinyu Liu, Ziyi Yang, Ifrah Idrecs, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Lang2ltl: Translating natural language commands to temporal robot task specification. In Conference Robbot Learning (CoRL), 2023. URL
wwtp/rgg///bb//eeeoeee 11649
Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine. RLIF: Interactive imitation learning reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
URL httpp:://openrevieeet wtwp:/ewwwwiieeeeffooeeeeeeeeeee wwt.:fertfofneeeo idooorrrrbioio-LL
Yecheng Jason Ma, William Liang, Vaidchi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Di- nesh Jayaraman. Liv: Languago-image representations and rewards for robotic control. arXiv preprint arXiv:2306.0m 00958, 2023a.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang, Vip: Towards universal visual reward and representation via value-implicit pre-training, 2023b.
Farzan Memarian, Zhe Xu, Bo Wu, Min Wen, and Ufuk Topcu. Active task-inference--uuieed deep inverse reinforcement learning, In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 1932 1938, 2020. doi: 10.1109 BBB9//CCC433222200000002
Ofir Nachum, Shixiang Shane Gu, Honglak Lec, and Sergey Levine. Data-efficient hicrarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Anis Najar and Mohamod Chetouani. Reinforcement lcarning with human advice: A survey. Frontiers in Robotics and AI, 8, 2021. ISSN 2296-9144. doi: 22339/frobb.2021556666 frontiersin. org http://trcleeeee...o. wttp:////..33/3///tttbto 201.222222........
URL https://www.
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99, pp. 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Tne. ISBN 1558606122.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Xinlei Pan and Yilin Shen. Human-interactio subgoal supervision for efficient inverse reinforcement learning. In Proceedings of the 17th International Conference on Aulonomous Agents and MulliAgent Systems, AAMAS '18, pp. 1380 1387, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Comput. Surv., 54(55, jun 2021. ISSN 0360-0300. doi: 10.1145/3453160. URL httpp:://di.
0145/3453180
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-nceded decomposition and planning with language modcls. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Associalion for Compulational Linguisties: Mexico City, Mexico, June 2024. Association for Computational Linguistics. wwwdiinnnnnnniin naacll
NAACL 2024, pp.
doi: 10.18653/v1 fon2...indinss--.. naacl. 264. URL
264/
Harish Ravichandar, Athanasios S. Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robolics, and Aulonomous Systems, 3(Volume 3, 2020):297-336 2020. ISSN 2573-5144. doi: /ttpps//doi.org/ httee//11111111. /ww/////aauewwwooorrooooooooooo/ 190000819--0660 URL https://wwz annualreviess org/ wttp///en//oooaaa//oo...o..... 1146///umueer--ootrro0-001011330
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourleenth International Conference on Artificial Intelligence and Stalistács, volume 15 of Proceedings of Machine Learning Reseorch, pp. 627 635, Fort Lauderdale, FL, USA, 11 13 Apr 2011. PMLR. URL httts::/pproceeddnggg http://cceelnsss.....prrrss///ss ppwpss//1//ossslllaaaaae html
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic rein- forcement learning without reward engineering, 2019.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and charac- terizing reward gaming. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and
A. Oh (ods.), Advances in Neural Information Processing Systems, volume 35, pp. 9460-9471. Cur- ran Associales, Inc., 2022. URL https::/prooceedngg... neurips. cctpspprcpppppprrpeeppeeep filles/paper//222//illll 2affgee6222222655550077777700000
Sumedh A Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Biyik, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: one demonstration is enough to learn robot policies. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Henry Sowerby, Zhiyuan Zhou, and Michael L. Littman. Designing rewards for fast learning, 2022.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdpss A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181-211, 1999. ISSN 0004- 3702. doi: httppp///oiiooo////oooo//ssssss0 scieae//arriccee/////044/0000000
URL httpp:://ww... http://wwww.cieecddreccccoo com/
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 10.1109/TROS..2022.0mm 201...6........810..
IEEE, 2012. doi:
Rodrigo Toro Icarte, Toryn Q. Klasson, Richard Valenzano, and Sheila A. McIlraith. Reward machines: Exploiting reward function structure in reinforcement learning. J. Artif. Int. Res., 73, May 2022. ISSN 1076-9757. doi: 10.1613/jair..1122 1.12.40 URL http://///a.ooog//..1//////////
12440
18

---

Published in Transactions on Machine Learning Research (03/2025)
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicentee Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL
https://zenodb. arggrrecord/812002
Marcel Torne Villasevil, Max Balsells Pamies, Zihan Wang, Samedh Desai, Tao Chen, Pulkit Agrawal, and Abhishek Gupta. Breadcrumbs to the goal: Supervised goal selection from numan-in-the-loe feedback. In shirth-seventth Conference on Neural Information Processing Systems, 2023. URL https://openrevier net/foorum?dd-uuBBall0011
Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chamlatte,, Chanlatte, and Sanjit Seshia. Compositional automata embeddings for goal-conditioned reinforcement learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, vol- ume 37, pp.
www.::s..eurrpp....o..mm Curran Associates, Inc., 2024. URL Confereee pdf
paper
Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Sacid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges, 2023.
Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made easy. In 2024 IEEE International Conference on Robolics and Automation (ICRA), pp. 0973-6980 IEFE, 2024.
Zhigen Zhao, Shuo Cheng, Yan Ding, Ziyi Zhou, Shiqi Zhang, Danfei Xu, and Ye Zhao. A survey of optimization-baseed task and motion planning: From classical to learning approaches. TEEE/ASME Trans- actions on Mechatronics, pp. 27, 2024. doi:
Xinglin Zhou, Yifu Yuan, Shaofu Yang, and Jianye Hao. Mentor: Guiding hicrarchical reinforcement learning with human feedback and dynamic distance constraint, 2024.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martnn, Abhishok Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint ar 200: 12293, 2020.
A Edge selection
The selectEdge subroutine governs how ILG-Learn allocates learning intervals while exploring the ILG. As described in Section 4 II.G-Learn starts at the source of the ILG and iteratively extends tree of best-known policies to the other vertices of the ILG. The key insights that inform the design of selectEdge are:
Training an edge policy ((u,)) can only begin once the policy n. is learned (frozen), so it makes sense lo stop investing learning intervals along an edge once desired success probability successThrest old is reached. This way, future learning intervals can be allocated to downstream edges.
Different edges of the II.G may require vastly different numbers of training episodes to acquire a successful policy, it makes sense to dovetail learning intervals between edges so we do not spend too much effort particularly difficult edge.
Only one feasible path needs to be found, so it makes sense to use a best-first search heuristic to avoid exploring the entire IL..
Our edge selection algorithm is inspired by the exploration order found in Dijkstra's algorithm. In fact, if we set the parameter intervalsLimit to 1, our selectEdge subroutine implements an exploration order suitable for Dijkstra's 's algorithm. When intervalsLimit is greater than 1, selectEdge incorporates dovetailed exploration with early stopping heuristics to try reduce the number of training episodes needed learn satisfactory policy.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Appendix AA.1 describes the high-level structure of selectEdge that implements dovetailed exploration and early stopping. Appendix A.2 details the heuristic scoring function that selectEdge uses to implement best- first exploration. Appendix B provides advice for selection of relevant parameters. The choice of parameters
used for the experiments in Section are detailed in Table 3
A.1 Dovetailed exploration
To implement early stopping and dovetailed exploration, ILG-Learn keeps track of these vertex and edge sets:
explored Vertices. These are the vertices to which ILG-Learn has learned and solidified a path policy.
learnedEdges. These are edges for which we have fully learned and frozen a policy. We froeze a policy once either (1) the empirical success probability exceeds TurcessThreshold or (2) intervals learning intervals have been allocated to the edge.
Edggd These are edges that we will never invest learning effort into because we have already found better path to get all of their successors.
JrontierEdges. These are edges leaving the explored tree (that is, leaving veplocedVertices) Vertices) that are neither fully "learned" nor "abandoned". At each iteration, we need choose one edge from this set to invest learning interval into.
All edge sets are initially empty. The logic for maintaining these sets is shown in Algorithm To reduce notational clutter, we assume that frontierEdges, and learnedEdges, abandonedEdges are global variables that persists across calls to selectEdge. We also assume global access to reachProb (updated in line 15 of Al Amgorithm as well as the existence of the bookkeeping functions:
best.Success PastSucccessrob((u v) returns the highest estimated probability found by line 13 of Algorithm during any learming interval allocated to (u,v). v).
intercals Elapssddu,, v) returns the number learning intervals that have been allocated to (u, v)
Algorithm 2: selectEdge
Input:
IT. structure (U, Fr, uo)
Parameters intervalsLimit and successThresholdd (introduced in Section
Scoring heuristic parameters extensionPenalty and exploitationBonus (introduced in
Appendix A.2
Global access to reachProb (maintainod by Algorithm 1
Output: Edge vw for next learning interval, or None if learning is complete.
learnedEdges + learnedEdges {(u, v) E intervalsElapsedd u, v) intervalsLimit):
learnedEdges learnedEdges U ((u, v) E v bestSuccessProb(u u) > successThresholdy abandonedEdges + abundoneddEdges U {u,, v) Vu', u- n reachProb, < &rechhrrobu/ }; explored Verlices + {oa} {r E U (3u, (u, v) E learnedEEgge) A (Vu, (u,v) E F - (u, E
learnedEdges abandonedEdgge)) };
ProntierEdger + outgoingEdgess explorre wwwloredloeeeeeeoo Vertices) learnedEdges! abandonedEdges; v))
return
A.2 Score
The selectEdge subroutine (Algorithm 2 uses a heuristic score function to implement best-first. explo- ration. We design score to balance the following desiderata:
Exploitation. Perform best-first search by extending paths that have low cost.
20

---

Published in Transactions on Machine Learning Research (03/2025)
Even exploration. Dovetail exploration of multiple edges, trying to assign the same amount of Training intervals to all extensions of the "sufficiently low cost." paths.
Anticipation. Prefer extending paths that have few edges remaining to final vertex.
As in Appendix IntervalEBaapse.
we will assume global access to reachProb and access to the bookkeeping function The score of an edge comprises three terms:
score(u, 2) caplootatioonnncentidddu v.) intercols
Verterr)) edggexxtensionPenally
We describe the components follows:
Term 1: Exploitation. The exploitation incentive strongly priorilizes investing training ellort in edges that could be part of the highest probability extension of some path in the tree explored so far. Concretely, let
bestEatenssinnProb
max (w.w)efrroberrrdgs
reachProb(u)
Now let
exploitationBonus
reachProb(u) 2 bestEanlension
emplotaation/nceetiviqu...
otherwise
wher exploitationBonus is some very large number.
Even exploration. Subtracting intervalsElapsed(u, v) softens the best-first search by trying to eveuly allocate intervals to promising edges. This term will be much less in magnitude than exploitation- Bonus, so it acts as tie- breaker within our soft best-first search.
Term 2:
Term 3: Anticipation. high choice of edgeExteensionPenaly strongly strongly prioritizes extending paths that are close to reaching a final vertex (in terms of how many edges there are).
B Hyperparameter selection
We now provide advice for how to select the ILGG-eern--peeife hyperparameters introduced in Section Hand Appendix A.2. Table 3 presents our choices for each environment. For details about the hyperparameters used for the RCE in ILG-Learn's learnPolicy subroutine, please see Appendix D
IlustrationCount: This should be enough goals so that the underlying example- based control algorithm can efficiently Iearn a successful policy. To provide empirical insight into how to choose a value of illustrationCount we performed multiple experiments for a subset of our environments which we varied the value of illustrationCount, please see Table In the Section value of 10 for all of our enviromments except for the outwardView environments, where we used 50. We note that the choice of illustrationCount is tied to the specifies of the underlying edge tasks; we believe that IL.Gs that include landmarks that admit diverse set of success observations (e.g. a robotic arm can successfully grasp object with a wide variety of arm angles and grip positions) will benefit from a relatively large number of success examples. We also note that a straight extension to ILG-Learn allows the teacher to increase the number of illustrative observations the learner struggles to master an edge task.
5 we used a
The fixed time horizon used when training each edge policy must long enough to allow the example-based control algorithm to explore the environment. The ideal value is usually significantly more than the number of timesteps needed for an expert complete any given edge task.
episodeLength:
21

---

Published in Transactions on Machine Learning Research (03/2025)
intervalLength and intervalsLimit: The quantity intervalLength x intervalsLimit should be enough training steps for example-based control to saturate the success probability of the edge policy for any edge in the ILG. Since the number of learning episodes needed by example based control varies depending on random sced, we recommend over-cstimating this quantity. To avoid wasting Iraining episodes on already-good policies, we recormeed choosing a relatively low value for intervalLength: Since the choice intervalLength governs the frequency of teacher-intervention (in the form of responses to success/ ffailure queries), recommend choosing intervalLength be large enough that there is only modest amount of teacher interaction, yet low enough that learning episodes are not wasted.
successThresholi We recommend choosing value of successThresholk slightly lower than the success rate of an optimal policy. Choosing a high successThreshol causes ILG-Learn to spend large amount of learning effort on an edge before exploring deeper in the ILG, which may increase the success rate the final policy at the cost of more environmental and student-Leacher interactions.
estimationQueries: We recommend choosing a number that gives adequate confidence in the success rate of the learned policy. In practice, we have observed that example based control often yields policics that either succeed much more or much less than our desired maecesssThrcsoolee so relatively few cslimaatonQueriia (e.g. 30) suffice. there are multiple feasible lasks in the ILG, choosing high value of estimationQueries will improve ILG-Learn's ability perform best- first scarch. We chose not to implement statistically-rigooous version estimateProbablll since doing would in general require the teacher to respond to many success /failure queries and not necessary for good end-to-end performance in practice.
exploitationBonus: We recommend choosing very high value, that score function imple- ments "soft" best first seach, in which the second and third terms of the score function serve as tic-breakers. To achieve this, one can choose paramcters such that exploitationBonus > intervalsLimit. + diameler(G)) x edgeeExtensionPenalty where diameler(G) is the diameter of the ILG.
adgeExteessooneenlty: We recommend choosing edgeeExtensionPeualty to be guess of how many learning intervals will be required to train an edge along a feasible path to reach the suc- Theeehodd. Since choosing this parameter requires considerable foresight, we recommend choosing relatively low value.
Table LLG-Lear-specific parameter selection for the experiments shown in Section
Di DiagonalMaze7xtt -oree Coarse
StackChoice Cwwp:hhii---Ouuarrrrraaaa Outwardvies
StackBA Outwardvies
Diagonal Maze3r3
Diagona MaroonaMaazenn Maze5x5
Diagonal DiagonalHaze7xt Haze7x7
Maze4x4-ao Coarse
Maze4z4- Mazo4z4-Fine Fine
StackAB
illustrationCount
10
10
10
10
10
50
LpisodeLength
400
400
400
1200
200
600
80
80
intervalLength
100k
100k
100k
100k
100k
100k
100k
100k
100k
intervalsLimit
100
100
100
100
100
100
100
100
100
successThreehhol
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
0.8
estimation
30
30
30
30
30
30
30
30
30
exploitationBonus
101
101
101
101
101
101
101
101
101
edgeExtennsonneenlly
3
3
3
22

---

Published in Transactions on Machine Learning Research (03/2025)
C Data dependence
In order to better understand how the performance of LeGrn, RCE, and BC depends the amount of numan-provided data, we performed additional experiments in which we varied the amount of illustrative observations, goal examples, or demonstrations provided to each algorithm. We report our findings in Table For ILG-Learn, the choice of LLG-Learn-specifee hyperparameters the same as reported in Table with the exception of IllustrationCoun,, which we manipulate.
Table 4: Final success rates (means over 5 trials). Data count denotes the number of illustrative observations per ILG vertex, number examples for RCE, or number of demonstrations for BC/DiffP. IL.G-Lcarn and RCE are trained for a maximum of 10 million environment steps; see Appendix L for experimental details
and Appendix F for learning curves. Entries are omitted where ILG-Learn achieved near-perfect success rate with fewer illustrative observations.
IL.G-Learn
RCE
BC (MLP)
BC (DiffP)
Data count:
10
50
100
100
1000
1
10
100
1000
StackAB
0.00
0.975
0.752
0.863
0.050
0.392
0.00
0.005
0.059
0.28
DiagMaze3x3
1.00
1.00
0.00
0.600
0.062
0.990
0.991
0.91
DiagMaze5x5
0.991
1.00
0.00
0.200
0.004
0.826
0.989
0.68
DiagMaze7x7
0.200
0.995
0.00
0.00
0.00
0.476
0.689
0.70
Implementation details
D.1 ILG-Learn and RCE
Our implementation is available at htttps///iithub. http://b...oo/cctttto/o/////eeee
We implement ILG- Learn Python. We re-implemented RCE (following instructions Eysenbach al. (2021)) on top of JaxRL's (Kostrikov 2022 implementation of SAC (Haarnoja et. al. 2017). We use this RCE implementation both as the example- ccample-based control subroutine in our ILG-Learn algorithm and as the RCE baseline.
RCE is an actor-critic example-based control algorithm. Rather than train an auxilliary network to provide a loss signal to be used as in reinforcement learning (as is done in earlier example-based control algorit.s such as Fu et al. (2018)) the critic directly predicts a time-discounted probability that an observation, action pair being considered by the learner will lead to success. Success is defined as regression task, in which the usev-provided goal examples represent success and a pool of other transitions (in our case, the replay buffer) represent unlabeled data that is unlikely to represent success.
Hyperparameters. We the same SAC+RCE hyperparameters as Eysenbach et al. (2021), including the SAC-specific hyperparameters that were inherited from Haarnoja et al. (2017), although we increased the width of cach hidden layer in the actor and critic MLPs from 128 to 256. The original RCE implementation varied the "n-step returns" and "Q combinator" hyperparameters depending on the particular task. We use 10-step returns and mar as the "Q combinator" for all tasks, which matches the RCE authors' choices for their "sawyer-bin--e bickiiiiiing" task. We note that the traditional choice "Q-combinator" would be min, following Fujimoto al. (2018).
Our choice of IT.G-Learn-specfff parameters (detailed in Appendix B is shown in Table
Policy sequencing details. Underlying T..G-Tern's compositional approach to long-horizon policy learn- subroutine that sequences edge policies to form path policy, As described in
ing is the sequencePolicia Section 4 each edge policy is executed for a fixed horizon (governed by the episodeLength parameter). To formalize the behavior of requencePolicies it is convenient to associate each (path or edge) policy æ with a horizon denoted horizonz). We assume that the array horizon of such bookkeeping variables in global scope and define the sequencePolicies subroutine in Algorithm
23

---

Published in Transactions on Machine Learning Research (03/2025)
If horizon is maintained such that horizon is episodeLength for each edge policy *(u,v) then Algo- rithm 3 matches the textual description of Section [ In our implementation, we noticed that running cach edge policy for the fixed horizon resulted in non-smooth motion, because the agent would hesitate after having achieved each landmark (waiting for the associated edge policy's horizon to be fully consumed) before moving to the next. To reduce this "waiting time" we slightly enriched the information provided by the teacher: every time the learner uses querySuccess(r) at the end of an episode to assess the success rate of an edge policy T(u,)) if the teacher responds "Success" they also provide the index of timestep which the agent entered Bu. Then we update horizon= to be the max of these success timestep indices, plus an addition 15 timesteps of slack term. The results in Section were collected using the above procedure, although we do not believe that the details of our sequencePolicy heuristic is an important aspect of IT.G-Tearn.
Algorithm 3: sequencePolicies
Input:
Path policy Tu and edge policy ((u,))
Parameter episodeHorizon
Mutable bookkeeping dict horizon
Output: Path policy to reach
Function n. (T):
if | horizon,, then
return (T);
else
return (T);
horizon < (orizon + horizon ( return Tp
Success estimation details. As described in Section 4 the estimateProbabilit estimates the proba- bility that a learned edge policy T(u.r) successfully reaches fv. Since we start each rollout of ((u,e) from the state distribution induced by executing the path policy Tu, we naturally obtain an empirical estimate of the probability that the path policy obtained by sequencing Tu and T(u,,)- We call attention to this because our "Dijkstra-style" planning implemented by selectEdge only receives the costs of paths, not individual edges. If we assume that each edge has an intrinsic fixed cost, then this is an inconsequential bookkeeping detail. However, since ((, could reach ((v) even if not started from (u) (and moreover because are empirical) it is possible for the estimated cost of path to be less than that of one of its prefixes. Such ill-behaved path probabilitics only pose a problem for ILG-Learn if high-cost prefix dissuades selectEdge from exploring what would become the lowest-cost. source to sink path. Such a situation is only likely to arise the teacher provides an ILG that contains landmarks that are both (1) hard to reach and (2) not necessary to scaffold exploration towards subsequent landmarks.
cost estimates
D.2 BC (MLP) baseline
We implemented the BC baseline in Python using Jax (Bradbury et al. 2018) and Flax (Heek et al. 2024). For each task, pooled the (observation, next action) pairs from all the demonstrations and trained MLP to predict the next action given the current observation. Our MLP had 2 hidden layers (cach of width 512), we used the Huber loss and a learning rate of 0.0001. For each task, we had a heldout set of 10 validation demonstrations. We stopped training when the validation loss stopped decreasing.
D.3 Diffusion Policy baseline
We use the implementation of the U-Net based Diffusion Policy as described in (Chi et al. 2023 using Py We train the policy on 1000 expert trajectory demos for the stack and diagonal point-maze tasks. We optimize for 100 epochs on the point maze tasks and 300 epochs for the stack task and report the mean success rates across different seeds (3 for stack and 5 for point-maze tasks) using checkpoints that achieves the lowest action reconstruction loss. We report the other common hyperparameters used for training in Table
D.4 HRM baseline
We bascline against the official implementation of hierarchical reinforeement learning with reward ma- chines. Toro Icarte et al. (2022). A reward machine a deterministic finite automaton whose alphabet is set of proposilions environment's 's state space. In experiments, the propositions the names of
24

---

Published in Transactions on Machine Learning Research (03/2025)
paramcter
valuc
prediction horizon
8
learning rate
3e-4
weight decay
lo-4
input embed dim
256
step embed dim
256
U-Net downsample dims
(256, 512, 1024)
kernel size
5
num diffusion steps
100
EMA power
0.75
batch size
128
Table 5: Diffusion Policy common training hyperparameters
the landmarks, defined formally in Appendix E At each timestep, the current reward machine state deter- mines which reward function used to supply rewards to the agent. Even though the reward functions are conditioned on the current state the reward machine, options correspond to transitions between states. In order to encourage an intra-option policy to reach its specified reward machine state there a tunable bonus hyperparamater (that set 1000) that is recieved by the intra-option policy for successful tran- sition. For our StackAB, StackChoice-Outward//// shown in Figure
and DiagonalMaze3x3 define the reward machines
In the StackAB environment, the reward functions ro is designed to encourage reaching and grasping block A, and rf is designed to encourage completing the stack. The final state has r which gives a constant reward of 1000. The other Stack environments' reward machines are defined analogously, although rō for StackChoice encourages reaching either Block A Block B.
The DiagonalMaze family of environments have reward functions that encourage reaching the next landmark (room center). Each reward machine is defined analogously to the DiagonalMaze3x3 reward machine. The DiagonalMaaeexx-Coorree reward machine has only three states, analogous to the three vertices of the DiagonallMazze777-Coorss ILG.
We define the reward functions as follows:
rs, 5') =(1 tanh (10dist(s', BlockA)) 0.25 0.25(grasp(s BlockA)] r0 (8, a, s') =(1 tanh (10dist(s', BlockA)) 0.25 0.25 BlockA)]
(8, = max (s, (, 8¹) r a, s')
r (s, max ([iftte( + 0.5(1 tanh (horizontalDist(s! BlockA))) 1 (s, a, s') max ([ifteed + 0.5(1 tanh (horizontalDislls )a (look r (s, a, 8') = dist(s', nextRommenneer)
where in the above definitions, dist(s', BlockA) denotes Euclidean distance between the center of the agent's end effector and the center of block A. We also use the notation dist for the analogous concept in the DiagonalMaze. Similarly, horizontalDist calculates Euclidean distance, after both points have been projected onto the horizontal plane. All final state rewards are r(s, s') ) 1000. This design pattern is adopted from the original HRM paper. The preceding rewards for the Stack family of tasks are heavily inspired by the original reward signal from robosuite.
Internally, HRM uses DQN to select options, and DDPG as the intra-option control policy. We use the same hyperparameters that the original authors used for their MuJoCo-based experiments.
25

---

Published in Transactions on Machine Learning Research (03/2025)
grasp A
room 2
stack
r2
grasp
stack
r2
room
ri
10
r
r2
grasp B
T3
stack
10
StackAB reward machine
StackChoice
DiagonalMaze3x3 reward machine
reward machine
Figure 5: Reward machines structure for HRM baseline.
E Environment Details
Our manipulation environment is built in robosuite (Zhu et al.) 2020). Our maze environment is built
de Lazcano et al. (2023) and customized of the Maze2D) environment originally introduced in D4RL (Fu)
et al. 2020). All our experiments use Gymnasium Towers al. (2023) and MuJoCo [odorov et al. (2012).
E.1 Stack
We adapt the "Stack" environment that is included in robosuite. This environment simulates 7-DoF Franka Panda robot arm. We use the provided 7-dimensional continuous action space that includes an operational space controller with fixed impedance. We also use the provided observation space, which is 55-dimensional and comprises the robot's s joint angles, joint velocities, end effector position, end effector quaternion, gripper position, gripper velocity, the position of each block, the quaternion each block, the gripper-object distance of each block, and the distance between the two blocks.
We make the following modfications to the original environment:
1. The side length of each cubical block is reduced from 0.05 meters to 0.04m. We do this include more randomness in the initial block placements.
The blocks always initialize along the centerline (parallel to the x-axis in the simulation, which is depth from the perspective of the robot) of the table with block A closer to the robot than block B. The exact positions are determined randomly in cach reset as follows: we uniformly select two æ coordinate values in the range (-0.2, 0.2). If the values are at least 0.05 meters apart, we let block start at the lower ø value and block start at the greater x value. Otherwise, we repeat this process until we obtain sufficiently spaced x values.
3. For Outwardview we require block B to be stacked on block A.
Outwardview we allow the blocks to be stacked in either order; for StackBA-
4. For StackChoice- Outtairviww and sion, as if the observations are produced by an object detector that is mounted on the same side of the table as the robot. To do this, we position a MuloCo camera at the position indicated by the pink camera icon in Figure []] At each timestep, if no pizel belonging to either block A or block B present in the ground-trun truth object segmentation yielded by this camera we mask all components of the observation corresponding to that cube with the value -2 (which is far from the observation values encountered in normal operation).
we implement simulated occlu-
Landmark definition. To simulate the teacher's response to the querySuccess querics during training, we defined (ggrasp A) to be all states where block A is at least 0.021 meters above the surface of the table (and similarly for 3(grasp B). We defined ((stack) analogously to the success condition of the original robosuite stacking environment: the cubes must be touching cach other, the gripper cannot be touching the block on top of the tower, and the block on top must be both above the surface of the table and aligned horizontally with the other block.
26

---

Published in Transactions on Machine Learning Research (03/2025)
Examples and demonstrations. To generate the landmark examples and demonstrations for behavior cloning, we use handwritten scripted policy. We inject noise into this scripted policy so that it covers a variety of grip locations and arm paths. We also tried using teleoperated demonstrations (and examples gathered therefrom) for the StackAB task and did not find significantly different performance so we elected to use scripted policies to generale all data. While gathering demonstrations, we injected a small amount (uniform with magnitude 0.1) of noise into the simulated actions (but not the action labels in the stored demonstration), similar to DART (Laskey et al., 2017). We 10 held-out validation demonstrations to evaluate loss during BC training.
The example observations for the "grasp" landmarks are taken after the arm has grasped and lifted the block by ~0.02 meters. We found that slightly lifting the block greatly improves RCE's ability to learn a successful policy, even the absence occlusion. We believe this is because grasps that do not lift the block are very close in observation space to "near grasps" that do not make firm contact with the block, yielding difficult discrimination task for the RCE critic.
E.2 Maze
We customized Gymnasium Robotics' Maze2D environment; our layouts are shown in Figure 3 The agent is a force-actuated point mass with a 2-dimensional continuous action space. The observation is continuous and 4-dimensional, comprising the agent's current position and velocity, but not the goal location or the location of any of the maze's walls.
Landmark definition. Each landmark is the center of room, illustrated in Figure 3 Each square room has side-length of 3.6; the landmark region is a circle of diameter 1 centered within the room. Only the agent's position (not velocity) determines membership in landmark.
Examples and demonstrations. To provide illustrative observations, we include one illustrative obser- vation from the center of the landmark, with the remaining illustrative observations drawn uniformly at random from positions within the landmark. All illustrative observations contain 0 velocity.
The demonstrations to better cover the state space. For each task, all demonstrations traverse the same sequence of rooms to reach the final goal. For each environment, we include 10 demonstrations and an additional 5 held-out demonstrations for evaluating loss during training.
gathered with handwritten scripted policies that include small amount of noise
F Learning curves
We report the overall success rate of the entire task; for ILG-Learn this means that the learner must explore a source-to-sink path before achieving nonzero success rate. Note that some trials of IL.G-eearn stop before reaching 10m environment steps, because ILG-Learn allows early stopping. In this case, we extend the success rate of the latest training checkpoint for the rest of the 10m steps (even though no further training occurs). We mark this extension with dashed line.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Macks R.leaan soted dadt)
Tiakkt 10 Waceeeeaeeeeeeeeeeee
1.0
02
44
62
Depfaacttt wiill
Figure Detailed learning curves for ILG-Learn. Each line represents one of 5 random seeds.
DiedMudd
US 1ilie)
2.8
0.5
10
03 65
Figure 7: Learning curves showing success rate over total environment steps for LeGrn, RCE, and HRM The light lines show individual random seeds, the dark lines show the
baseline experiments from Table [ mean of 5 random seeds.
Warld 2CC ww// 196
Saard apist
200
Figure 8: Learning curves showing Huber loss during offline training for the BC (MLP) experiments of Section 5 Each line represents one of 5 random seeds. We chose a number of training steps that was sulficient for validation loss to stop decreasing. For StackAB we found this to occur at about 6000 steps, for the other environments we found this to occur at about 10000 steps.
28


Reviews:
[{"review_id": "sFSgpmbtY0", "reviewer": "Reviewer_dn15", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Innovative Task Specification: ILG provides a flexible and interpretable way to define long-horizon tasks, balancing abstraction and concreteness.\n+ Empirical Evidence: Robust experiments on various tasks illustrate the efficacy of ILG-Learn over baselines like RCE and behavior cloning.\n+ Reduced Human Effort: By focusing on intermediate landmarks, ILG reduces the need for detailed reward engineering or exhaustive demonstrations.\n+ Scalability: The framework accommodates branching paths, allowing exploration of multiple strategies. This gives ILG the ability to incorporate multi-modality by having different paths in the graph that achieve the same goal.\n+ General framework that seems to subsume goal-conditioned RL as a special case where there is just one sink node in the graph. I would recommend highlighting this fact more if this statement is correct.\n\nWeaknesses:\n- Teacher Burden: While reduced compared to traditional methods, selecting appropriate landmarks and densities may still require expertise.\n- Limited Generalization: The method's performance relies heavily on task-specific ILG configurations, potentially limiting generalizability. However, I like the authors ideas for using foundation models to address this in the future.\n- Baselines: Only two baselines is a bit concerning given that there are many types of BC and goal-conditioned RL that could be applied.\n- Human Factors: The paper does not run a user study to explore human factors. I think this is fine for this type of paper that introduces a new algorithmic approach, but having some notion of how usable this method is in practice would significantly strengthen the paper and could be mentioned more in future work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the Illustrated Landmark Graph (ILG), a novel approach for long-horizon policy learning. ILG bridges the gap between reinforcement learning (RL) and imitation learning by allowing human teachers to specify tasks as a graph of intermediate \"landmarks,\" which are illustrated with examples. The proposed learning algorithm, ILG-Learn, leverages these graphs to iteratively plan and learn policies while incorporating teacher feedback through queries. This method reduces the need for manually specifying dense reward functions or providing full-length demonstrations. Experimental results demonstrate that the proposed approach outperforms standard Behavior Cloning and RCE, an example-based control algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Desired Adjustments:\n\n- Provide baseline results for one more imitation learning method (ideally a recent one) and one more example-based or goal-conditioned RL approach.\n- Provide more details on the human interaction. Was the \"human\" a scripted or synthetic agent, one of the authors, etc? \n- Compare and contrast ILG-Learn with hierarchical reinforcement learning methods in the related work.\n- Discuss more advanced versions of DAgger such as \"ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning\" Hoque et al. CoRL 2021 in related work since DAgger is quite old and there have been many improvements since then.\n\nTypos:\n- page 3 \"bahavior cloning\"", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ISjuulHsm", "reviewer": "Reviewer_XYTR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper introduces a novel perspective on task decomposition by enabling a balance between abstract declarative guidance and prescriptive demonstrations.\n\n2. The paper is well-structured, with clear motivation, methodology, and experiments.\n\n**Weaknesses**\n\n1. The method is highly motivated by the research gap between RL and imitation learning. However, the experiments omit RL baselines, such as goal-conditioned RL or hierarchical RL, which are directly relevant. Adding these would strengthen the claims.\n\n2. The comparison with behaviour cloning seems unfair due to significantly less samples provided for behavior cloning. It would be better to show the interactions/demonstrations needed by ILG-Learn and BC to reach a given success rate, say $40$%, $60$% and $80$%. Hopefully, ILG-Learn can reach the same success rate with much less samples.  \n\n\n3. Figure 6 highlights substantial performance variability with different random seeds. This raises concerns about the robustness of ILG-Learn and suggests a need for further discussion or solutions to address instability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Illustrated Landmark Graph (ILG) as a novel framework to address long-horizon sequential decision-making tasks. By using landmarks as intermediate states, the ILG effectively decomposes a complex task into manageable subtasks. The proposed ILG-Learn algorithm integrates this task specification into a policy learning framework that combines planning, policy learning, and teacher feedback. The approach is validated empirically on manipulation and navigation tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The policy learning paragraph in Section 4 states that the policy tries to reach states that *look similar* to the human-provided examples. Could the author clarify how the similarity is quantified?\n\n2. Could the author prepare a table for all hyper-parameters? I would like to know the values of intervalLength, episodeLength, successThreshold, intervalsLimit and the rationale of the choices. \n\n3. In the experiments, more details of the RCE baseline should be provided. \n\n4. Section 5.1 says that each learning interval includes $100$k environment steps, while later it says ILG-Learn is trained for $10000$ environment steps. It is confusing. I think the learning interval should be shorter than the total training. Could the author clarify it?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NewXzUK8zg", "reviewer": "Reviewer_3rxy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- interesting idea to combine an abstract human-designed interpretation of the task (i.e., IL graph) with learning from demonstrations and seems especially relevant for long-horizon tasks\n- solid evaluation on a robotic task and a 2D maze\n\nWeaknesses:\n- the method and contribution are not clear until Sec. 4\n- the similarity to formal methods guided learning is not clear enough from the related work", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes landmark graphs to guide and compartmentalize the demonstration learning process that is guided by examples instead of full-length trajectories. The main advantage is that the definition of landmarks is driven by a human, which makes it easy to provide demonstrations for the landmarks. Policies are learned for each edge of the graph so that the complexity of each subtask is reduced and then combined with a search-like algorithm to obtain a controller that fulfills the entire task. The proposed approach is evaluated on a grasping task and a maze to study landmark density and the effect of different sensor perspectives when providing the examples. \n\nThe paper is mostly well-written.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The abstract does not clearly explain the method proposed, e.g., what is the relation to finite state automata [1] - what are ILGs? what are illustrative observations?\n- The introduction does not clarify the contributions enough, e.g., the difference to finite state automata should be clear, it should be clearer how your approach differs from hierarchical learning and TAMP [3]\n- Fig. 1 can be improved. Especially the right column is not easy to understand - graphs of tasks and graphs for other approaches. Possibly split the Figure in one to explain the tasks regarded and one to illustrate how your approach compares to others. \n- Fig 2. was not really helpful for my understanding and in my opinion, also other methods live in this yellow space [3] [4]\n- The related literature misses a comparison to task and motion planning [3] and also the similarity to automata-guided learning [1,2] should be clearer\n- In my view, your approach uses the human to abstract the problem into multiple subtasks that are hopefully easier to learn. Another approach would be specifying the tasks with temporal logic formula and synthesizing the graph, aka automata, for that, which would be more automated and less human hand-engineering driven. Maybe you can comment on this perspective in the related work.\n- It would be useful to relate the ILG to automata literature in Sec. 3: IL = states, landmark map = labeling function\n- Sec. 5 would be more clear if the experimental setup and results were more clearly separated in subsection\n- It is not clear to me why the chosen baselines RCE and BC are a good choice. You are motivated by being in the gap between imitation learning and reinforcement learning - why not choose an RL algorithm with sparse rewards from a reward machine?\n\n\nMinor:\n- There seem to be typos on page 3 \"bahavior cloning\" page 11 \"StackAB-Inwardview\" and page 12 \"human err\"\n- The style guide specifies that the formatting of Algorithms and Figures should use the full column \n- I was wondering if your approach is extendable to tasks with dynamic obstacles, e.g., autonomous driving. This seems to be not the case since the ILG would likely change depending on a specific traffic scenario\n\n\nReferences:\n\n[1] Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, and Sanjit A. Seshia, “Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning,” to appear in NeurIPS, 2024.\n\n[2] Learning minimal automata with recurrent neural networks BK Aichernig, S König, C Mateis, A Pferscher, M Tappler\nSoftware and Systems Modeling, 1-31\n\n[3] Curtis, A., Fang, X., Kaelbling, L.P., Lozano-Pérez, T. and Garrett, C.R., 2022, May. Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA) (pp. 1940-1946). IEEE.\n\n[4] LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning. Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, Sheila A. McIlraith. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence Understanding Intelligence and Human-level AI in the New Machine Learning era. Pages 6065-6073. https://doi.org/10.24963/ijcai.2019/840", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the use of landmark graphs as a lighter approach for humans to provide demonstrations in sequential decision-making problems (landmarks instead of complete trajectories). The benefits of such an approach go beyond the fact that it makes it less burdensome for humans to provide demonstrations; it also, for example, allows for a modularization of the policy learning problem, as policies are learned for specific edges of the graph.

As can be seen in the initial reviews, the reviewers had concerns about the presentation of the paper (clarity, lack of details on some aspects, a more careful discussion about related work) and the lack of representative baselines. The discussion with one of the reviewers even revealed a bug in one particular set of results! The authors addressed these issues during the rebuttal, and the reviewers all agreed that the new version is substantially improved and should be accepted.
Decision: Accept (poster)