Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data
Authors: Andrew C Li
Toryn Q. Klassen
Andrew Wang
Parand A. Alamdari
Sheila A. McIlraith
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
RL
Formal Language
Reward Machine
LTL
Language
Grounding
Compositional
Primary area: reinforcement_learning
TL;DR: We train RL agents directly from high-level specifications, without reward functions or domain-specific oracles.

Abstract: Grounding language in perception and action is a key challenge when building situated agents that can interact with humans, or other agents, via language. In the past, addressing this challenge has required manually designing the language grounding or curating massive datasets that associate language with the environment. We propose Ground-Compose-Reinforce, an end-to-end, neurosymbolic framework for training RL agents directly from high-level task specifications—without manually designed reward functions or other domain-specific oracles, and without massive datasets. These task specifications take the form of Reward Machines, automata-based representations that capture high-level task structure and are in some cases autoformalizable from natural language. Critically, we show that Reward Machines can be grounded using limited data by exploiting compositionality. Experiments in a custom Meta-World domain with only 350 labelled pretraining trajectories show that our framework faithfully elicits complex behaviours from high-level specifications—including behaviours that never appear in pretraining—while non-compositional approaches fail.

Ground-Compos-RReiffree:: Grounding Language in Agentic Behaviours using Limited Data
Andrew Li, Toryn Q. Klassen', Andrew Wang, Parand Alamdari', Sheila McIlraith
Department of Computer Science, University of Toronto Vector Institute for Artificial Intelligence
Schwartz Reisman Institute for Technology and Society Toronto, Canada
{andrewli, toryn, andrewwang parand, sheila)@cs toronto. eo
Abstract
Grounding language in perception and action is a key challenge when building situated agents that can interact with humans, or other agents, via language. the past, addressing this challenge has required manually designing the language grounding or curating massive datasets that associate language with the environ- Compose-Reinforce, Reinforce, end-to-end, neurosymbolic framework for training RL agents directly from high-level task specifications without manually designed reward functions other domain-specific oracles, and without massive datasets. These task specifications take the form of Reward Machines, automata-base representations that capture high-level task structure and are in some cases autoformalizable from natural language. Critically, we show that Reward Machines can be grounded using limited data by exploiting compositionality. Experiments in a custom Meta-Wone World domain with only 350 labelled pretraining trajectories show that framework faithfully elicits complex behaviours from high-level specifications- iinetiioon-innuinnng behaviours that never appear in pretraining while non-compositional approaches fail.
ment. We propose Ground- Soouund-ooooee---e
Introduction
Grounding language-connectinn language with perception and action within environment-is fundamental challenge when building robots and other agents that are interfaced through language. One popular approach addressing this challenge is to employ manually-desinned domain-specific reward function or success detector (c.g.
interpretation of language, such as canguage-conditionn (T-4). For instance, in the BabyAl AI benehmark ([], successful execution of instructions like "go to the red ball" can be evaluated programmatically in the environment simulator, providing a reward signal for learning language-conding conditioned behaviours. Such instances of grounded language generalize to arbitrary scenarios and controlled subsets of language by design, but are hard to hand-engineer in complex, non- simmlietl settings based on raw perceptual inputs like pixels.
The recent advent of large language models (LLMs) has inspired an alternative pathway to grounding language: training diverse datasets that pair language descriptions with environment trajectories (e.g., To 5. RT-2 language groundings, it typically requires enormous datasets in order to capture the broad scope of language usage within an environment [1] [O]]. For agentic applications that are data-intensive (c.g. robotics) or where access trajectory data is limited, such data-driven language models are prone
[6] LIV (Z), VPT [).. While this obviates the need for manually designed
failure on complex or out-of-distribunt tasks 11:4
We propose Grouund-oopoos-RReineee
end-to-end framework for training reinforcement learning (RL) agents directly from high-level task specifications, without relying on manually
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pretraining
Observation
Supervised Learning
Truth value False
Trajectories
Proposition
Progress 0.2444
Propositional labels
holding
Behaviour Elicitation
(Optional) Natural language command up the blue box, then the green then the red box.
Reinforcement Learning
Action
Self-generated rewards
Autoformalization
Agent
Envronment
Reward Machine
cholding
hholingg
cholding
Observation
Figure 1: Ground-Compont Composs-Reinforre lightweight framework for training RL agents directly from Reward Machine specifications, without oracles like reward functions or feature detectors. In pretraining, learn map propositional symbols context-specific truth values ("is the robot holding the red block?") and progress signals ("how close the robot to holding the red block?"). To elicit behaviours, we prompt the agent via Reward Machine composed of these symbols (or via natural language, if an autoformalizer is available). The agent then synthesizes its own dense reward function and interacts with the environment learn the desired behaviour via RL.
designed rewards or feature detectors. We represent tasks via Reward Machines (RMs) 15 an automata-basee basta task specification language that exposes temporally extended task structure in terms of atomic propositional symbols. Specifically, our framework: (1) grounds these atomic symbols in the environment by pretraining on a limited dataset of labelled trajectories; (2) composes these symbols via RMS express complex tasks; and (3) for any such task, trains an agent solve that task via RL on self-generate dense rewards based on its learned interpretation of the RMM. Overall, Seiadd-ComooeeeeRe
enables the elicitation of a wide range of temporally and logically structured behaviours expressed as RMs (specified directly or in some cases autoformalized from natural language), requires minimal pretraining data, and does not rely on external, domain-specifer oracles for training or execution. In this paper, we present the following novel contributions:
1. conceptual, end-to-end framework for compositionally grounding language in behaviours based on Reward Machines and reinforcement learning (Figure I]. Critically, our framework requires minimal labelled trajectory data for pretraining, and does not require external oracles like reward functions, success detectors, or feature detectors.
2. A compositional reward shaping strategy for Reward Machine tasks that is critical for strong performance Meta- Meta-World [17. Our strategy addresses propositional sparsity, where propositions of interest (e.g. "pick up the block") are rarely satisfied through random exploration.
Experiments across diverse Reward Machine tasks, including temporally extended gridworld navigation and robotic manipulation in Meta-World. Our approach elicits diverse behaviours in Meta- World (including out-of-distributiin behaviours that never appear in the pretraining dataset) from only 350 labelled pretraining trajectories while non-compositional approaches fail.
2 Related Work
Grounded Language Learning. Several past works have explored grounded language learning. Hermann et al. [ show that an RL agent in 3-D environment consistently navigate to target

---

objects described via language by being rewarded for successful trajectories. Hill et al. [3] show that an agent can learn new word-object bindings and apply that knowledge to solve tasks within the same episode. Liu et al. [18] show that meta-RI. agent indirectly learns to interpret language-basee advice embedded within an environment to improve its task performance. Chaplot et al. propose a neural architecture for grounded language learning in a 3-D Doom environment, considering both RL with success-basee rewards and imitation learning from an oracle policy. Brevalier-Boisver et al. I propose the Baby AI platform for learning a synthetic language in 2-D gridworld, finding that existing RL and imitation learning approaches are sample inefficient and generalize poorly. Unlike our approach, these methods require significant manual design of reward functions oracle policies.
Recent works learn grounded language languaag from data, without domain- -ppaaiii--seiiicccie manual design. Black et al. [5]. Zitkovich et al. 6 and Kim et al. [9 present vision- lanoon-lnguag--cttii models for robotics while
Baker et al. [ and Lifshitz al. [3 present models for playing MineCraft. Bahdanau et al. [20] reward functions from data. Works have also directly
and Ma et al. 7 learn
leveraged vangon-language models for rewards in iision-based environments (21 231, but typically do not leverage language compositionality and are prone to failure complex or outtt--distributiin tasks. Shi et al. [1). Yuan et al. (24), Huang et al. (25) and Ahn et al. (26) decompose complex tasks at execution time via language models. While this shares motivation with our work, we represent tasks via RMs and exploit compositional task structure for both training and execution.
Formal Languages for RL. Formal languages like Linear Temporal Logic (LTL) [27] and other associated formal structure such as RMs [66] have a rich history of application in the control 2 29)
verification 32), monitoring, and synthesis of dynamical systems. Recently, they have risen in popularity in deep RL for white-box specification of rich temporally extended reward functions 115 35 and in a number of cases can be automatically generated from natural language commands (e.g., (9). Several works show that formal languages enable compositional generalization to unseen instructions. Vaezipoor et al. [4), Kuo et al. [4] and Yalcinkaya et al. 42 train instruction- conditioned policies that zero-shot generalize to unseen instructions by training on procedurally generated LTL formulas. Qiu et al. (43, Liu et al.
44 León et al. [45] and Jackermeier and Abate [46] train transferable skills that can be invoked by a planner. Nangue Tasse et al. [47] consider how value functions and policies can be composed zero-shot for arbitrary RM tasks, but assume that all the tasks can be captured via a finite, predetermined set of goal states. Our compositional reward shaping approach also builds on prior methods. Camacho et al. (34,, Furelos-Blance et al. [48] and Parać et al. 44 provide potential-based rewards for RMs based on the current RM state, but such methods provide no signal when target propositions rarely occur. Several works propose continuous progress signals for each proposition [33 50-54), but these approaches are limited to tasks with a binary success criterion. While progress signals are typically manually specified, we show how they can be learned directly from data.
Nearly all laamall-angugge based deep RL methods assume access to an external evaluator of symbolic features (a.k.a. a labelling function). To our knowledge, only a few works specifically avoid this assumption, but they instead depend on external reward signal. Li et al. [55] [6] consider the implications of noisily grounding symbols when using RMs. Hyde and Santos [57 and Christoffersen al. (58) infer RMs from the reward signal as inductive bias for RL. Umili et al. (59 60), Kuo et al.
41. Andreas et al. 66] and Oh et al. [62) show that ungrounded formal specifications can improve RL by providing information about the task structure. In contrast to these works, our approach does not rely on an external symbol evaluator or reward function.
3 Preliminaries
3.1 Reinforcement Learning
A reinforcement learning (RL) problem considers environment modelled Markov Decision
Process (MDP) (S,A, T, T,P, P,R, n4 7). where S is a set of states, A a set of actions, T C S a set of terminal states, P S x A AS a transition probability distribution, R S x A x S - R a reward function, μ E AS an initial probability distribution, and y E [0, 1] a discount factor. An episode begins with So μl, and at each time 2 the agent chooses action a,, then observes the next state $141 ~ P(s.. a,) and reward r++1 = R(s,, a, St+1), repeating until a terminal state
'Henceforth "formal languages." " for ease of exposition.
3

---

Go to a red triingle, then a green object, then a blue object that isn' triungle.
Go triangle circle, in any order.
(A,0)
(O.1)
(RAA,)))
(G,0)
(A,
Go to a rod, then green, then blue object. repeatedly.
Go blue object, but never go triangles before circles.
1),1)
(B -1.1)
O AA-A,
(8,1)
(R,0)
0y
Figure 2: Four temporally extended tasks in a gridworld expressed as Reward Machines over the propositions AP = (R, G, B, A, O). An edge labelled (4p, r) indicates the logical condition yp for when the corresponding transition should be followed, and the reward r that yielded as a result. Doubled circles indicate terminal states, and we omit non-rewarding self-loop edges to aid readability.
8T E T is reached. We refer to full episodes as trajectories, denoted by T = (sy, ag 81, aj....). A history h4 (Sp, du, $1, .1,..... st) refers the states and actions up to time The agent's s goal is to interact with the environment to learn policy T (ar]s)) that maximizes the expected discounted return E EEtz y'ra] (where the episode length T can be 2o).
3.2 Task Specification via Reward Machines
Formal languages with sequential or temporal structure- incluuriiinnlinnnggg RMs [T6], regular expressions [63]. and various temporal logics [54] 4ffeoffer
offer an intuitive and expressive interface for specifying tasks in RL while supporting representations that capture compositional task structure. Starting with predefined set of atomic propositions AP that represent abstract, binary features of environment states, such languages can be used to express temporally extended properties or reward functions. In this work, we focus on tasks specified as RMs, which subsume several formal languages of interest 34 such as LTL over finite traces (64 65 and regular expressions.
An RM is an automaton that captures the structure of a reward function over the abstract vocabulary AP (Figure 2. It takes input a sequence of truth assignments (w..w,.... ......) over AP, where each We E 2 denotes the subset of AP that holds true at time f, and outputs a corresponding sequence of rewards (f1, r2.... .). An RM has finite set of internal states U and begins in fixed state uo at time f 0. At each step & 2 1, the RM updates its state to ue E LI and emits a reward based on the current input W/ and the previous state Ut-1- This continues until the RM enters terminal state from a designated subset F Cu
Running Example. The gridworld in Figure wwill serve as a running example. Consider the top-left RM, which describes task composed of three subgoals to be completed in a fixed order. When given an input sequence (Wi, (wi,W2,.. ..)... that identifies the values of all propositions at each time (whether the agent
red object, a green object, a circle, and so on), the RM state us E U tracks which subgoals have been completed and transitions to the next RM state as soon as the agent achieves the current subgoal. When red triangle, a green object, and blue object that isn 't triangle are reached in that order, the RM terminates with reward of
Definition 1. A Reward Machine R is defined as a tuple (U, ug. F,A, du, 8y), where U is f-mitte set of states, with initial state ug l and terminal states C U: AP set of propositions: 6, 11 x AP + U is transition function that updates the RM state based on the current truth assignment: and 8, :u> AP
R is a reward function that emits a reward each step.
As shown in Figure 2 the transition and reward functions of an RM can be intuitively and compactly specified via a set of labelled edges the form (u, w', sp, indicating that if a truth assignment w satisfies the formula io (denoted by += sp), then 6,, (u,w) w) w' and 6,, w)
3.3 Grounded Interpretations
RMs express tasks in terms of abstract symbols (e.g., R, A), but these symbols must be grounded in the environment to be meaningful. This is achieved via labelling function L S - 2 that

---

maps each MDP state s to the set w S AP of propositions that hold true in For an RM R over propositions AP, any such L also grounds R in the environment by inducing a reward sequence for any MDP trajectory т: first, states in are converted into truth assignments wi = ((s)), then the RM is simulated over the sequence (w1,,w..... .) to generate sequence of rewards until termination.
An RM-MDP augments an environment (represented by reward-free MDP) with concrete reward function captured by an RM and labelling function. The resulting reward function is generally non-Markovian with respect to MDP states S, since the reward at time & depends on the internal RM state uq. While one might consider expressing optimal behaviours via a history- based policy I (a)]/gg, this is unnecessary if the agent has oracle access to L since RM-MDPs are Markovian over the extended state space S X 24 [16]. The agent can use L compute wi = L(s)) and recursively simulate the RM to track u, Thus. it is typical to express policies in the form π (a,]s,, u,), where the RM state Ue compactly encodes the history ht and is sufficient for optimal decision making.
Definition 2. An RM-MDP is a triple (M, R, C), where M = (S,A, P. (4,7) is an MDP without rewards or terminal states, R= = (U, uo, A, ou. 8,) is an RM, and L S 2AP is a labelling function. The RM-MDP is equivalent to an MDP with state space U and reward function induced by R and L.
Running Example. The RM at the top left of Figure captures the high-level structure of multi- stage task. To map environment trajectories into concrete rewards for this RM, we need a labelling function L S + 2AP that connects abstract propositions like R and iS to environment states.
4 Problem Setting
Our goal in this work is to faithfully elicit behaviours from agent given only high-level task specification in the form of an RM, R, such as the ones depicted in Figure 2 R can be specified directly, translated from other formal languages like LTL [4), generated from symbolic planner 166 62], or sometimes autoformalized from natural language. Formally, we consider an environment M (S, A. P. pr, 7) (an MDP without rewards or terminal states) and a finite set of propositional symbols AP. For any given RM task R = (U, ug F.AP, 8u, 8,)), we wish to obtain a policy (R(a, that performs well in the RM-MDP (M,R, L*), where L S 24P reflects ground- truth interpretation of the propositions AP in the environment P
Assumptions. We aim to obtain क without online access to L or to an external reward function that evaluates ground-truth performance with respect to R In order to connect symbols AP with environment percepts, we instead assume access fixed pretraining dataset D = ((r w') W J.N. of
trajectories r' = (s ay. s1 ..... with corresponding labels w² = (C" ( (si), In practice, such labels can be obtained via crowdsourced annotations 669 or super-spe learning 770.
For task R, the agent is allowed an arbitrary number of interaction episodes with the environment before committing to a final policy गR. However, during this interaction phase, the agent must learn in self-supervised manner as the environment does not provide a separate reward signal.
5 Ground-Compos--Rnntorr
We propose an end-to-end framework for this setting called Ground-Compose-RRee In the pretraining phase, the agent first grounds propositional symbols AP in environment states via supervised learning D. In the behaviour elicitation phase, the agent is given task composed over symbols AP. The agent then learns a policy मR. by interacting with the environment and synthesizing its own learning signal for RL based on R and its learned interpretation of AP. We hypothesize that this bottom-up approach to grounding language in behaviours- ffrst learning the meanings of individual symbols and then composing them to interpret complex tasks-is an effective strategy. The remainder of this section describes the core implementation of Ground-Compon Reinforce and in Section [6] we raise and address an issue called propositional sparsity where the agent fails to learn in extremely long- ong-horizon tasks.
(Figure I.
RM R.
For the problem statement, we consider the more general form history- policies, rather than policies conditioned on the ground-truth RM state, which depend on L. Such oracles can be notoriously hard to design in practice [68 and often require internal simulator access.
5

---

Core Algorithm. We ground the propositional symbols AP by learn- ing labelling function [(s) R
Algorithm 1 Ground- Coouun-Compose-eee Reieofore for RMs
Input:
MDP M without rewards, Propositional symbols AP,
Dataset D of labelled trajectories, RM task R. over AP
(s) via any binary classification
// Pretraining phase
method on D. Given an RM task R
1: Train labelling function [(s) on D using any binary classifica- tion method
over AP, we solve surrogate RM-
MDP (M, R. L) via RL. This surro- gate task approximates the true RM- MDP (M, R, L for which the agent lacks supervision. Since the agent
// Behaviour elicitation phase
2: Initialize policy गश (a 8,, u) arbitrarily
3: for cach episode do
4: Observe initial state in M, set u to the initial state of R
can query 2 freely, it can simulate
5: while is non-terminal do
6: Sample action ~ TR | 8, u)
rewards (2,,,2,-..)) and RM states
7:
Exccute a in M and observe next state
("1, :2.....) for any trajectory, as de- scribed in Section 3.3 Finally, we use these self-generated signals to train a policy π (a, St, uz) as outlined in
8:
Compute truth assignment @ + L(s')
9:
Updatc RM: u' + Óu, @), r + dr @) Update मæ with RL for transition (s, a,,,,,s'
10:
11: Set 4- s', u'
Algorithm|
Running Example. Suppose our gridworld agent is expected to solve arbitrary RM tasks over the vocabulary {R, C,B,4,O), With Ground- Compose-Reinfonn
the agent first connects these symbols to environment states via DD (i.e. it learns to identify which states have red shapes, which states have triangles, etc). human can then specify a new task as an RM composed over these symbols (e.g. "go to triangle and a circle, in any order") ") without needing to program a task-specific reward function. The agent systematically evaluates its own performance on this task by composing its learned interpretations of {R, G, B, A.A.)) Ob, providing a learning signal for RL.
6 Compositional Reward Shaping
An issue that can arise with the core implemen- tation is sparse rewards. While prior RM works have proposed additional learning signals for these meth- ods fail when RM transitions themselves pose an exploration challenge. This can be caused by propositional sparsity: when (*) is constant across most states under random exploration, the agent struggles to learn to meaningfully af- feet propositional values (e.g. consider a robot manipulation task that involves satisfying the proposition "the box is picked up").
Estimate subtask
Bootstrap with RM-state value
Virls 2
transitioning in the RM 34
48 49
(R A.O)
4
(B -d,0)
Figure 3: An illustration of how we estimate op- timal values in RM-MDP. Suppose the agent is currently in RM state w (green and bolded). To evaluate the expected return for the transition 4 - uB we estimate how close the agent is to satisfying the formula on the transition (reaching the red triangle), and bootstrap with a coarse value estimate for RM state 8 The overall value of a is approximated by the maximum expected return across all outgoing transitions from A
To address this, we extend Reinforce with potential-busee reward shaping 71] Specifically, for a given RM task R, we estimate the optimal value function (OVF) for the surrogate RM-MDP (M, R, L) by decom- posing R into simpler subtasks. The most ba- sic subtasks correspond to satisfying individual propositions or their negations, and we estimate the OVFs for these 2 |PP tasks during pretrain- ing via offline RL on 'D. These 2) ||P OVFs then serve as building blocks that can be composed to approximate the OVF for any of the infinitely many RM tasks over AP.
6.1 Deconstructing RMs into Logical Subtasks
To estimate the OVF of the surrogate RM-MDP (M, R, L), denoted (s, we combine two key ideas: (1) treating each RM transition in R as an independent subtask, and (2) bootstrapping from state-independent value function ex (u). Consider the example in Figure 3 where the agent is in

---

MDP state and RM state wA. Each RM transition is associated with logical condition & (e.g.. RA A or B A A)), and we treat the satisfaction of ip as its own subtask. Definition formmalizzss this: we introduce the class (PPP) of reachabilityy tasks over propositional formulas, where each task Op entails reaching state 8y such that ((T)) F ye, terminating with reward 1 upon satisfaction. Opp is itself an RM-MDP with a single non-terminal RM state (and a terminal RM state), and is therefore Markovian over S. We denote its OVF as (o)(s).
Definition& 3. For any propositional logic formula over AP, define Qip as the task of reach- ing state E S such that L(sT) F The episode terminates and yields a reward of 1 if such a state is reached, and continues indefinitely with reward otherwise. Let (PL(AP)
{ou is a propositional formula over )P) denote the set of such reachability tasks.
The second component of our method is to bootstrap using nate-independent value function uix (u) that approximates the expected return from any RM state while ignoring the MDP state. We estimate oR (u) using variant of Value Iteration over the RM graph, following Camacho et al. (34). Finally, to estimate V (s. u), we evaluate each outgoing transition (u, u', sp,r) from u by combining
the subtask value 1 (s) with the bootstrapped value of the next RM state o (w'). The final estimate is the maximum expected return over all such transitions:
(s u)
max (u.aa (p.r)) (+) (r (r+ rok((())
This approximation assumes no RM self-transitions with non-zero rewards. Appendix A further justifies and explains our approximation while extending it to arbitrary RMs.
6.2 Further Deconstructing Logical Subtasks
Approximation 1 allows us to estimate (5, u) for any RM task R over AP, provided we can estimate (x) for any in o (AP).. However, the number of propositional formulas over AP (up to logical equivalence) is 22 and modelling separate OVF for each such task is intractable. To address this, we further decompose logical formulas based on their structure. Any formula y can be rewritten in disjunctive normal form, i.e., disjunction of conjunctions of literals (where literal is either a proposition x or its negation -r)). We then approximate the OVF of Oy using the semantics of fuzzy logic [72), where max represents disjunction and min represents conjunction
Let yp & ... V Ek. where each & is conjunction of literals. We approximate:
(s)
x max Vo)
(2)
i= l....k
For each conjunctive clause { 4 A A l2, where each 44 is a literal, we approximate:
Vog (5) min kl.....k
(s)
By composing Approximations 1
we can estimate the OVF for any RM task based on only 2|AP]]
OVFs namely, those for Oxr and Q- for each AP. We refer to these as the primitive value functions (PVFs).
6.3 Final Remarks
In this section, we showed that the optimal value function (OVF) of any RM task R. can be approxi- mated using just 2) API primitive value functions (PVFs). From these A| PVFs, we can estimate OVFs for doubly exponentially many logical tasks (22") and infinitely many RM tasks. Each PVF quantifies progress toward satisfying a single proposition or its negation, and can be learned directly from D using any offline RI. algorithm. One might view this approach as trading off expressivity for modularity: directly modelling the OVFs of all RM tasks is infeasible, so we instead model small, reusable set of A|P PVFs at the cost of introducing some approximation error.
Leveraging that we can estimate the OVF for any RM task R, we extend Algorithm I with potential- based reward shaping to address propositional sparsity. Further details on learning PVFs, sources of approximation error, and the potential-basee -bated reward shaping scheme are provided in Appendix A
Fuuzy operators have previously been applied for satisfaction of a temporal formula over quantitative signals 33 53 54 While such tasks are binary in nature, consider RMs, which can express other reward structures.
7

---

7 Experiments
We conducted experiments to evaluate the following research questions:
RQI
With Sosiul--ooooo--einfoooc can we faithfully elicit behaviours given high-level task specifications (RMs)?
Grounding RMs in Behaviours:
RQ2
Compositional Generalization: Can we elicit meaningful, out-of-distribution (OOD) be- haviours beyond those observed in DD
RQ3 Propositional Sparsity: Can the agent operate in extremely long-horizonn environments where
propositional values are hard to alter with random exploration?
Code/videos available at: https: thub. www/anndrewwi71//////co http:///////ooooddoooooooo-eeeee http:///t.w..oooooooooooo/o///ot
7.1 Experimental Setup
We considered two domains: an image-based based gridworld with randomized object locations called GeoGrid (introduced in the running example), and a Meta- Sett--orld-blee World-based robotics environment called DrawerWorke (Figure 4). Full details on the setup can be found Appendix [B.]
Pretraining Datasets. We collected 'D under minimal as- sumptions about downstream tasks. GeoGrid, D con- tains 5000 trajectories generated by random policy. In DrawerWorld, we manually operated the robot to collect 350 trajectories involving generic behaviours (e.g.. opening drawers, lifting boxes). To evaluate OOD generalization, we constrained D to only contain trajectories interacting with at most one box in Drawer World. Finally, trajectories were labelled using a handerafted labelling function.
Figure 4: Drawer World is a custom Meta-World environment where the agent can interact with two drawers and three boxes. Propositions capture whether: each drawer is open; each box lifted by the agent; a given box is in given drawer.
Tasks. We designed diverse set of RM tasks (Table that target behaviours rarely or never seen in D. The GeoGrid tasks evaluate whether the agent can produce fine-tuned behaviours beyond the random-action trajectories observed in D. The TrawerWorld tasks evaluate whether the agent solve complex manipulation tasks that require composing behaviours observed in D (e.g., Pickup- Each- Box requires handling all three boxes, while trajectories in D interact with at most one box).
7.2 Method and Baselines
We benchmarked Hoaundd-oopose-Reiinn
(GCR) against several non-compositional baselines. Methods based on online RL (GCR, Bespoke Reward Model) use PPO 73 to train policy from scratch. During execution, GCR captures memory via the RM state while all Non-RM-basse base- lines encode the observation history using an additional GRU (74). See Appendix B.2 for full implementation details and Appendix B.3 for full training details.
Ground-Composee Reinforce (ours). We implemented GCR with potential-based reward shaping as described in Sections 5 and 6 Both the predicted labelling function & and PVFs are neural networks trained on 'D via supervised Tearning and offline RI, respectively.
Ceifors-Reinforce
LTL-conditioned Behaviour Cloning (LTL-BC) a neural network policy I (agl/i, :p) that directly maps LTL specifications & to behaviours. We labelled each trajectory in 2> with an LTL description xp (based on the propositional labels for r), then trained the policy to maximize the log-likelihood of actions in conditioned on the history h2 and şp. For each downstream task in Table we prompted the policy with an LTL formula that aligns with that task.
We also trained bespoke models with advance knowledge of the downstream RM tasks. Bespoke Reward Model directly predicts rewards, optimal values, and terminations for all downstream tasks simultaneously. We labelled each trajectory in D with ground-truth rewards and terminations for each task based on the propositional labels, then trained the model to directly predict these quantities given the history ht. Value estimates were trained via offline RL in similar manner to

---

Table I: List of RM tasks. For each, report the mean ((D) and max (maxp) undiscounted return over trajectories in D, along with the max achievable expected return of any policy (Max; unknown, report the highest average return observed in experiments). Some tasks involve behaviours that are rarely or never observed in D.
Task
Description
Return
GeoGrid
maxp
Max
Sequence
Go to a red A, then a green A, then a blue A. Repeatedly go to a red A, then green A, then a blue A. Go to all six objects, but always go red objects before
0.04
Loop
0.04
3
5.36
Logic
0.00
1
blue objects, and blue objects before green objects. Go a red object, then blue object, then green object, but always avoid A.
Safety
0.84
DrawerWorld
Hold-Red-Box
Lift and hold the red box as long as possible. Pick up the red box, then the blue box, then the green box.
41.7
736
1538
Pickup-Fach-Bon
0 0
Show-Green-Box
Reveal the green box if it's in closed drawer, then lift it.
0.22
GCR. Finally, we trained policy via RL while using the learned value function for potential-based reward shaping. Bespoke Behaviour Cloning (BC) is a neural network policy that directly imitates successful trajectories in D for every downstream task. Due to the limited number of seward-worhh trajectories in D, we considered any trajectory achieving positive return on that task as successful.
We also compared various reward shaping schemes for GCR. No RS directly uses RM rewards without any reward shaping (i.e. Algorithm[[] High-Level RS, inspired by Camacho et al. [34], uses potential function that only considers the current RM state, but not the current MDP state.
7.3 Results
We ran each method's training pipeline five times and report the average final performances in Table 2 Performance was measured by undiscounted return (averaged over 100 evaluation episodes for GeoGrid or 20 for DrawerWorld), where rewards are with respect to the ground-truti labelling function L. In Appendix B.4 we report RL learning curves, both with respect to the agent's 's own reward model (without shaping rewards) and ground-trus truth rewards under L
GCR consistently solves novel tasks, even when no successful demonstrations exist in On Loop, Hold-Red-Boo and Tickup-aach-a attribute this success to GCR's ability to learn transferable knowledge from D and apply towards novel task compositions, while fine-tuning behaviours with (self-supervised) RL.
it significantly outperforms even the best trajectories in 'D. We
All non-compositional baselines fail to reliably solve any task. Results show that LTL-BC, Bespoke Reward Model, and Bespoke BC do not fare well with limited pretraining trajectories. Learning curves show that the Bespoke Reward Model assigns near-zero rewards to most trajectories in GeoGrid, likely due to the rarity of positive demonstrations in D. In DrawerWorld, it produces misaligned rewards, leading to reward hacking (evidenced by high returns under the learned reward model but low returns under the ground-truth L*).
Reward shaping enables long-houd oong-horizon RL. Our reward shaping strategy yields modest improvements in GeoGrid, but is critical to success in World, where behaviours like opening drawer and picking up box are nearly impossible discover from random exploration alone.
conclude the following. GCR faithfully elicits behaviours from RM specifications, outperform- ing non-compositional approaches (RQ1). Moreover, GCR compositionally generalizes to OOD behaviours beyond those observed in (RQ2). Finally, our compositional reward shaping strategy for GCR enables RL in long-horizon horizon settings involving propositional sparsity (RQ3).
7.4 Extending Ground- Ground-Compose--Rinntee Reinforre with a Natural Language Interface
While natural language (NL) is often argued to have compositional properties 75 exploiting this compositionality in agentic language models (e.g. vision-language-actioo models) remains an open challenge. In Appendix C we show that GeoGrid RMs can be autoformalized directly from an

---

Table 2: Comparison of methods for eliciting behaviours from high-level task specifications. We report performance (undiscounted return with respect to ground-truth rewards) averaged over runs with standard error.
Bespoke Reward Model
Bespoke BC
GCR (Ours)
GCR (Ours) High-Level RS
Task
GCR (Ours)
LTL-BC
No RS
GeoGrid
Sequence
1.00 0.00
0.04 0.01
0
0.05 0.01
0.94 = 0.03
1.00 0.00
Loop
5.36 = 0.08
0.03 0.01
0
0.04 0.01
4.68 = 0.05
5.27 = 0.08
Logic
0.94 = 0.01
0
0
0 0
0.00 = 0.00
0,94 0.01
Safety
1.00 + 0.00
-0.84
0.14
-0.55 = 0.01
0.23 = 0.11
0.97 + 0.01
DrawerWorld
Hold-Red-Box
1538 + 130
0+0
0+0
0+0 0
0 + 0
Pickup-Fach-Bon
1.00 + 0.00
0+0
0+0
0+0
Show-Green-Box
0.61 + 0.06
0+0
0+0
0+0
0 +
NI. reward function description using OpenAl's o3 model, zero-shot-i.e without fine-tuning on trajectories or other forms of grounding in our specific environments. Thus, we posit that leveraging compositional representations like RMs can be an effective way of building NL-interfaced agents in settings with limited labelled trajectory data (i.e. where [D] is small).
8 Future Work and Limitations
Extension to Other Compositional Representations: this work, we propose an end-to-end framework for grounding high-level specifications in behaviours that leverages the compositionality inherent RMs. However, we believe our core insights apply to wide range of compositional representations such as those that deal with objects 776 and relations [77].
Extension to Other Problem Settings: Grounding language is a prerequisite for myriad of unguage-conditioned problem settings. We consider an "RL-in-the-loop" setting, but future works
could extend our insights zero-shot execution of language tasks 40 46), question answering [78 79), and interactive task learning [80,81] [1).
Reward Hacking: Misalignment between agent's interpretation of a task and human intent can lead to harmful consequences, particularly in RL 82 The use of formal specifications like RMs, which unambiguous over the propositional vocabulary, the propositions themselves remains a concern in Ground-Compp Soaadd-o-omooeiinnnnn Reinforce. Prior works suggest that RM structure can be exploited to improve decision making under such ambiguity (55) (6)..
partially mitigate this, but ambiguity in
Assumptions on D: We assume that trajectories propositions. Future works could explore other representations of propositions (e.g. text) well as scalable labelling methods (c.g. crowdsourced annotations [69 or self-supervised learning (70)..
are labelled with values for a fixed set of
Conclusion
This work presents ChounndCompooe-RRinntoon directly from Reward Machine specifications-witieee wiithoutttttttttttt oracle reward labelling functions. A key challenge that we address is grounding these high-level task specifications in executable behaviours, given an agent's perception and action capabilitics. We find that exploiting compositional task structure is critical to faithfully capturing this grounding from limited data. Starting from only 350 labelled pretraining trajectories, we show that our technical approach scales to temporally extended manipulation tasks in Meta-World while generalizing outtof-distribution to behaviours that never appear in pretraining. Moreover, we show that in some cases, Reward Machines can be autoformalized directly from natural language reward function deseriptions to expose this temporal task structure.
end-to-end framework for training RL agents
More broadly, we show that leveraging language compositionality presents a promising pathway to building language-driven agents without relying massive aanguag--labelles data. Future work could explore the extension of these ideas to large-scale agentic language models such as vision- language-action models.
10

---

Acknowledgements
We thank Harris Chan for his insightful and valuable input throughout all stages of this project. We gratefully acknowledge funding from the Natural Sciences and Engineering Research Coun- cil of Canada (NSERC) and the Canada CIFAR AI Chairs Program. Resources used in prepar- ing this research were provided, in part, by the Province of Ontario. the Government of Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence (https //vector iustitute..ttttttt....m..... ai/ httpaatnersshipa// Finally, we thank the Schwartz Reisman Institute for Technology and Society for providing a rich multi diseiiiisypp research environment.
References
[1] Maxime -Bissvrrr- Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: A platform to study the sample efficiency of grounded language learning. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
[2] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world. arXiv preprint arXiv: nee:::10060555.1 2017.
[3] Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.
[4] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018
[5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, et al. No. A vision-language-actto flow model for general robot control. arXiv preprint 2rXin:2410.24164, 2024.
[6] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, et al. RT-2: lision-language-action models transfer web knowledge to robotic control. Proceedings of the 7th Conference on Robot Learning (CoRL), volume 229, pages 2165-2183 PMLR, 2023.
[7] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language image representations and rewards for robotic control. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages 23301-23320 PMLR, 2023.
[8] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. In Proceedings of the 36th Conference Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24639-24654, 2022.
[9] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown. Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[10] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages 265-279, PMLR, 2023.
11

---

[11] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness? An analysis of CoT in planning. In Proceedings of the 38th Conference on Advances Neural Information Processing Systems (NeurIPS), volume 37, pages 29106-291 29141,,, 2024.
[12] Lucy Xiaoyang Shi, Michael Robert Equi, Liyiming Ke, Karl Pertsch, Quan Vuong. James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell al. Hi robot: Open- ended instruction following with hierarchical vision- sasio--lnguage-action models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267, pages 54919- 54933. PMLR, 2025.
[13] Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila Mcllraith. STEVE-1: A generative model for text-to-behavior in Minecraft. In Proceedings of the 37th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 69900-69929, 2023.
[14] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfs- son, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 20di::200.000 102.000.00077 082.07 2021.
[15] Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pages 1107-2116. PMLR, 2018
[16] Rodrigo Toro Icarte. Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73:173 173-28,, 2022.
[17] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. World: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL), volume 100, pages 1094-1100, PMLR, 2020.
[18] Evan Zheran Liu, Sahaana Suri, Tong Mu, Allan Zhou, and Chelsea Finn. Simple embodied language learning byproduct of metarreeiforreement learning. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages PMLR, 2023.
[19] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Open- VLA: An open-source vision- aaiion-anguagc-action model. In Proceedings of the 8th Conference on Robot Learning (CoRL), volume 270, pages 2679-2715 PMLR, 2025.
[20] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
[21] Kate Baumli, Satinder Baveja, Feryal M. P. Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, Clare Lyle, Hussain Masoom, Kay McKinney, Volodymyr Mnih, Alexander Neitz, Fabio Pardo, et al. Vision-language models as a source of rewards. arXiv preprint 23Xiv:2312.0090 2023.
[22] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024.
[23] Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet, FuRL: Visual-language models as fuzzy rewards for reinforcement learning. In Proceedings of the 41 International Conference on Machine Learning (ICML), volume 235, pages 14256-14274, PMLR, 2024.
12

---

[24] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4MC: Skill reinforcement learning and planning for open-world Minecraft tasks. arXiv preprint arXiv:2303. 16563, 2023.
[25]
Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of the 6th Conference Robot Learning (CoRI.), volume 205, pages 1769-1782. PMLR, 2023.
[26] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog. Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, et al. Do as can, not as say: Grounding language in robotic affordances. Proceedings of the 6th Conference Robot Learning (CoRI.), volume 205, pages 287-318. PMLR, 2023.
[27] Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science, pages 46-5 57. IEEE, 1977.
[28] Sertac Karaman, Ricardo G. Sanfelice, and Emilio Frazzoli. Optimal control of mixed logical dynamical systems with linear temporal logic specifications. In Proceedings of the 47th IEEE Conference on Decision and Control, CDC, pages 21 2117-2122. IEEE, 2008
[29] Marius Kloetzer and Calin Belta. A fully automated framework for control of linear systems from temporal logic specifications. IEEE Transactions Automatic Control, 53(1):287-297, 2008.
[30] Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT press, 2008
[31] II Moon, Gary J. Powers, Jerry R. Burch, and Edmund M. Clarke. Automatic verification of sequential control systems using temporal logic. AIChE Journal, 38(1)677-55, 1992.
[32] Amir Pnueli. Applications of temporal logic to the specification and verification of reactive systems: A survey of current trends. In J. W. de Bakker. Willem P. de Roever, and Grzegorz Rozenberg, editors. Current Trends in Concurrency, Overviews and Tutorials, volume 224 of Lecture Notes in Computer Science, pages 510-584. Springer, 1986.
[33] Xiao Li, Cristian Ioan Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3834-3833 2017.
[34] Alberto Camacho, Rodrigo Toro Icarte Toryn Q. Klassen, Richard Valenzano, and Sheila A. Mcllraith. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 6073, 2019.
[35] Cameron Voloshin, Abhinav Verma, and Yisong Yue. Eventual discounting temporal logic counterfactual experience replay. In Proceedings of the 40th International Conference Machine Learning (ICML), volume 202, pages 35137-351 50. PMLR, 2023.
[36] Andrea Brunello, Angelo Montanari, and Mark Reynolds. Synthesis of L.TL formulas from natural language texts: State of the art and research directions. In 26th International Symposium on Temporal Representation and Reasoning (TIME), volume 147 of LIPIcs, pages 17:1-17:19. Schloss Leasstuull-Lbbbii-Zeeetee
für Informatik, 2019.
[37] Jason Xinyu I Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks unseen environments. In Proceedings of the 7th Conference Robot Learning (CoRL), volume 229, pages 1084-1110. PMLR, 2023.
[38] Francesco Fuggitti and Tathagata Chakraborti. NL2LTL-a python package for converting natural language (NL) instructions to linear temporal logic (LTL) formulas. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 16428-16430,
2023.
13

---

[39] Yongchao Chen, Rujul Gandhi, Yang Zhang, and Chuchu Fan. NL2TL: Transforming natural languages to temporal logics using large language models. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing (EMNIPP, pages 15880-15903, 2023.
[40] Pashootan Vaczipoor, Andrew C Li, Rodrigo A Toro Icarte, and Sheila A Mcilraith. LTL2Action: Generalizing L.T.. instructions for multi-task RI.. In Proceedings of the 38th International Conference Machine Learning (ICML), volume 139, pages 10497-10508. PMLR, 2021.
[41] Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Encoding formulas as deep networks: Reinforce- ment learning for zero-shot execution of LTL formulas. In Proceedings of the 2020 IEEE/RS) International Conference on Intelligent Robots and Systems (IROS), pages 1604-5610, 2020.
[42] Beyazit Yalcinkaya. Niklas Lauffer, Marcell Vazquez- Channlate,, and Sanjit A. Seshia Composi- tional automata embeddings for goal-conditioned reinforcement learning. In Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 72933-72966 2024.
[43] Wenjie Qiu, Wensen Mao, and He Zhu. Instructing condiiini reinforcement learning agents with temporal logic objectives. In Proceedings of the 37th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 39147-39175, 2023.
[44] Jason Xinyu Liu, Ankit Shah, Eric Rosen, Mingxi Jia, George Konidaris, and Stefanie Tellex. Skill transfer for temporal task specification. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2024.
[45] Borja G. León, Murray Shanahan, and Francesco Belardinelli. In nutshell, the human asked for this: Latent goals for following temporal specifications. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
[46] Mathias Jackermeier and Alessandro Abate. DeepLTL: Learning to efficiently satisfy complex LTL specifications for multi-task RL. In Proceedings of the 13th International Conference on Learning Representations (ICLR), 2025.
[47] Geraud Nangue Tasse, Devon Jarvis, Steven James, and Benjamin Rosman. Skill machines: Temporal logic skill composition in reinforcement learning. In Proceedings of the 12th Interna- lional Conference on Learning Representations (ICLR), 2024.
[48] Daniel Furelos- Furelos-Blanco, Mark Law, Anders Jonsson, Krysia Broda, and Alessandra Russo. Induction and exploitation of subgoal automata for reinforcement learning. Journal of Artificial Intelligence Research, 70: 1011-111,,,, 2021.
[49] Roko Paraé, Lorenzo Nodari, Leo Ardon, Daniel Furelos-Blanco, Federico Cerutti, and Alessan- dra Russo. Learning robust reward machines from noisy labels. In Proceedings of the 21st International Conference Knowledge Representation and Reasoning (KR), 2024.
[50] Yuqian Jiang, Suda Bharadwaj. Bo Wu, Rishi Shah, Ufuk Topcu, and Peter Stone. Temporal- logic-based reward shaping for continuing reinforcement learning tasks. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages
2021.
[51] Mahmoud Elbarbari, Kyriakos Efthymiadis, Bram Vanderborght, and Ann Nowé. - reward shaping for reinforcement learning. In Adaptive and Learning Agents Workshop 2021: at AAMAS, 2021.
[52] Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. In Proceedings of the 33rd Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[53] Anand Balakrishnan and Jyotirmoy V. Deshmukh. Structured reward shaping using signal temporal logic specifications. In Proceedings of the 2019 JEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3481-3488 2019.
14

---

[54] Derya Aksaray, Austin Jones, Zhaodan Kong, Mac Schwager, and Calin Belta. Q-learning for robust satisfaction of signal temporal logic specifications. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 4575- 2016. doi: 10.1009/DDC.2211..77999 6.7792279.
[55] Andrew C Li, Zizhao Chen, Toryn Q Klassen, Pashootan Vaezipoor, Rodrigo Toro Icarte, and Sheila A Mcllraith. Reward machines for deep RL in noisy and uncertain environments. In Proceedings of the 38th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 113411-11068, 2024.
[56] Andrew C Li, Zizhao Chen, Pashootan Vaezipoor, Toryn Q Klassen, Rodrigo Toro Icarte, and Sheila A Mcllraith. Noisy symbolic abstractions for deep RL: A case study with reward machines. arXiv preprint arXiv:2211 2..002211..00,, 2022.
[57] Gregory Hyde and Eugene Santos, Jr. Detecting hidden triggers: Mapping non Mor-Markov reward functions to Markov. In Proceedings of the 27th European Conference on Artificial Intelligence (ECAT), volume 392, pages 1357-1364. IOS Press, 2024.
[58] Phillip JK Christoffersen, Andrew C Li, Rodrigo Toro Icarte, and Sheila A Mcllraith. Learning symbolic representations for reinforcement learning of non-Markovian behavior. arXiv preprint arXiv:2301.02952 2023.
[59] Elena Umili, Francesco Argenziano, and Roberto Capobianco. Neural reward machines. In Proceedings of the 27th European Conference on Artificial Intelligence (ECAI), pages 3055- 3062. IOS Press, 2024.
[60] Elena Umili, Roberto Capobianco, and Giuseppe De Giacomo. Grounding LTLf specifications in image sequences. In Proceedings of the 20th International Conference on Knowledge Representation and Reasoning (KR), volume 19, pages 668-678, 2023.
[61] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pages 165-175. PMLR, 2017.
[62] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generaliza- tion with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pages 2661-2670. PMLR, 2017.
[63] Ronen I. Brafman and Giuseppe De Giacomo. Regular Decision Processes: A model for non-Markovian domains. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 5516-5522, 2019.
[64] Giuseppe De Giacomo and Moshe Y Vardi. Linear Temporal Logic and Linear Dynamic Logic finite traces. In Proceedings of the 23rd International Joint Conference Artificial Intelligence (IJCAI), pages 854-860, 2013.
[65] Jorge A Baier and Sheila A Mcllraith. Planning with first-order temporally extended goals using heuristic search. In Proceedings of the 21st National Conference on Artificial Intelligence (AAAI), volume 21, pages 788-795, 2006.
[66] León Illanes, Xi Yan, Rodrigo Toro Icarte, and Sheila A. Mcllraith. Symbolic planning and model-free reinforcement learning: Training taskable agents. In Proceedings of the 4th Multi- disciplinary Conference Reinforcement Learning and Decision (RLDM), pages 191-195, 2019a.
[67] León Illanes, Xi Yan, Rodrigo Toro Icarte, and Sheila Mcllraith. Symbolic plans as high-level instructions for reinforcement learning. In Proceedings of the 30th International Conference on Automated Planning and Scheduling (ICAPS), volume 30, pages 540-550, 2020.
[68]
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https Blog post.
/ wttp./////nnneeuuuteeeeeeeeeeeee
[69]
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Proceedings of the 34th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 3008-3024 2020.
15

---

[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning (ICMI.), volume 139, pages 8748 8763. PMLR, 2021.
[71] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In Proceedings of the 16th International Conference Machine Learning (ICML), pages 278 288- 1999.
[72] J. A. Goguen. The logic of inexact concepts. Synthese, 19(3/4):325 373, 1969.
[73] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[74] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 1724-1734, 2014.
[75] Zoltán Gendler Szabó. Compositionality. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, 2024.
[76] Carlos Diuk, Andre Cohen, and Michael L Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference on Machine Learning (ICML), pages 240-247, 2008.
[77] Saso Džeroski, Lue De Raedt, and Kurt Driessens. Relational reinforcement learning. Machine Learning 43:7-52, 2001.
[78] Lynette Hirschman and Robert Gaizauskas. Natural language question answering: the view from here. Natural Language Engineering, 7(4):275-300, 2001.
[79] Robert F Simmons. Natural language question-answering systems: 1969. Communications of the ACM, 13(1):15-00, 1970.
[80] Joyce Y Chai, Qiaozi Gao, Lanbo She, Shaohua Yang, Sari Saba-Sadiya, and Guangyue Xu. Language to action: Towards interactive task learning with physical agents. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), volume 7, pages 2-9, 2018.
[81] John E. Laird, Kevin A. Gluck, John R. Anderson, Kenneth D. Forbus, Odest Chadwicke Jenkins, Christian Lebiere, Dario D. Salvucci, Matthias Scheutz, Andrea Thomaz, J. Gregory Trafton, Robert E. Wray, Shiwali Mohan, and James R. Kirk. Interactive task learning. IEEE Intelligent Systems, 32(4):6-21, 2017.
[82] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Proceedings of the 36th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages
2022.
[83] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[84] Geraud Nangue Tasse, Steven James, and Benjamin Rosman. Boolean task algebra for rein- forcement learning. In Proceedings of the 34th Conference on Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9497-9507, 2020.
[85] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[86] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative Q-learning for offline reinforcement learning. Proceedings of the 36th Conference on Advances Neural Information Processing Systems (NeurIPS), volume 35, pages 1711-1724, 2022.
16

---

[87] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the 31st Conference Advances in Neural Information Processing Systems (NIPS), volume 30, 2017.
[88] Claudio Menghi, Christos Tsigkanos, Patrizio Pelliccione, Carlo Ghezzi, and Thorsten Berger. Specification patterns for robotic missions. IEEE Transactions on Software Engineering, 47 (10):2208-2222) 2021.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims in the abstract and introduction are reflected in the framework
described in Section 5 the compositional reward shaping in Section and the experimental
results in Section 7
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Major experimental details are reported in Section 7, with details like hyperpa- rameters and network architectures in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code and data is released at https://giit wttp:/ccmmaaaaddl/aa.a
ground compose- reinforce)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pblle pwwppppguuuiieeeeep Sodi:////uuddee/deeuuuubbiii SroodeSubiissi.e ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See details in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard error is shown in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:://// http:///eerrriscccceeeee http://wwrg.p.oooooooo.actaaaaaa
Answer: [Yes]
Justification:
The code of ethics has been reviewed and the paper conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are discussed Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and models in this particular paper do not have any possible misuse potential.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our environment is based MetaWorld, which we cite. We also acknowledge the authors of the RL library we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No new assets are released at this time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The methods in this paper do not use LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM for what should or should not be described.
(https: hattaa///aaarrppeecc///offeerrre
24

---

Supplementary Material for Soat--Cooooe-eifoorrere Tasking Reinforcement Learning Agents through Formal Language
In this supplementary material,
we provide further details and analysis on Ground-Compos-RRinnforc (Appendix A
provide experimental details (Appendix [B)
show how autoformalize natural language descriptions of reward functions into
Reward Machines with LLMs (Appendix (
and we discuss the potential societal impact this work (Appendix D
A Supplementary Details on Ground-Coopos--Reiffreee
In this section, we
discuss when Ground-Compose Crouud-Compose-RRinforee
-eeioos-eeeifforeee Reinforce is likely to work well in practice (Appendix A.1
elaborate on Section 6.1 description of how we approximate the optimal value function (s, u),
first under the assumption that the RM does not contain self-loop transitions with non-zero rewards (Appendix A.2)
and then relaxing that assumption (Appendix A.3
discuss approximation errors that can occur (Appendix A.4
describe how we train with offline RL the primitive value functions used in the approximation (Appendix A.55;
show how the approximate value functions for potential-based reward shaping (Appendix A.6).
A.1 Analysis of Advantages and Assumptions
For Grouund-Compos--Riinn Reinforee work well in practice, assume that faithful grounding of propositions can be learned during the pretraining phase from D (i.e. L(s) x (*)).. If this the case, the rewards generated by the core framework (Algorithm[]) [ will faithfully capture tasks for any RM over propositions AAP, by design. To achieve this, D should provide sufficient coverage of the state space S S-but critically, it does not require sufficient coverage of the space of possible trajectories to generalize. Hence, trajectories that are significantly
able to reliably elicit desirable with respect to D.
Unlike many cniitatioo-leariing -bsed methods, Ground-Compose- Reinforce also does rely high concentration of expert demonstrations in D (as evidenced by the fact that our agent reliably solves tasks, even when D contains only random-action trajectories). We can attribute this the RL phase, where the agent fine-tunes its policy using its self-generated learning signal.
A.2 Approximating V (s, u) (No Rewarding RM Self- Salf-Transitions)
To estimate Vi (ss, u), we evaluate each outgoing transition (u, u r) from by combining the subtask value (s) with us (u'). While Camacho et al. [34] treat RM transitions as singular actions, we treat RM transitions as temporally extended options that take variable number of steps to execute. Specifically, for a transition (u, u', r), we make the following assumptions about the corresponding option:
(1) It can be initiated if and only if the current RM state is
It optimizes for the subtask Oyp (i.e. satisfying yp quickly and with high probability).
(3) It either terminates when xp satisfied after variable length time K, or never termi- nates.
(4) The option will never result in different transition in the RM than the one in question.
25

---

Algorithm 2 Value Iteration over RM States (Modified from Camacho et al.
Input: RM R = (L, ug. F, AP, Sns 5,), discount factor "RM Initialize o (u) <- O, Vu U
2: while not converged do
3: for u in 1/ do
4: oiR (u) + (1ax) TRM(T uie (u'))
5: return R
<lava,-1>
<-lava,0>
Figure 5: An RM that produces reward of 1 for each timestep the agent spends in lava, until exits the lava.
To estimate uie (u), we run a variant of Value Iteration, modified from Camacho et al. [34] to reflect that rewards are garnered after the option completes (Algorithm D Note that Algorithm should be run with high-level discount factor 7RM < 7 as it treats RM transitions as singular actions while in the actual RM-MDP, an RM transition may require many low-level steps to achieve.
We estimate the return for an RM transition (u, u', :p, r), given that the agent is in MDP state s and RM state u, follows. First, model the return for immediately achieving the transition as r + 70 (u') - reward r immediately received for achieving the RM transition and o (u') estimates the future discounted return for being in RM state u'. If instead the RM transition is achieved I steps in the future, we estimate the discounted return as >* + yv (u')). Treating (u, u', r) as an option that terminates when the RM transition is achieved, define K to be the (random variable) number of steps it takes for this option to terminate when initiated from MDP state and RM state (where K is oo if the RM transition never achieved). The expected return of initiating the option is:
E (7t ( (v)())) (Eawkh + vok) (8) (r yu (u')
Recalling that the option is assumed to optimize for the subtask Oip, the final step results from the fact that & [>*] is precisely the optimal value for & from state s. We finally estimate the optimal value function for RM task R by maximizing over the choice of option:
(8
max (u'.p.r) E8(())
(V² y0 (w())
A.3 Approximating (s, u) with Rewarding Self-Loop Transitions
To handle RMs that contain self-loop transitions with non-zero rewards, we require few modifica- tions. Intuitively, an option corresponding to a transition (u, u', sp,r) where * u' might receive reward r' at any timestep it is active if there exists another transition (u, v'),"). As a simple example, consider the RM Figure 5 The RM describes a task where the agent needs to exit the lava
quickly as possible, and receives a reward of 1 for each step until it does so. When estimating the expected return of the edge labelled (-lava, 0), we need to consider the accumulation of the -1 reward at each timestep.
This can hard handle in general without more information, particularly when multiple self-loop edges exist for the same RM state. For simplicity, we assume that only non-self-loop transitions are treated as options (for the purposes of estimating optimal values), while all self-loop transitions in the current state are available at each step while option is being executed. Thus, if we define
26

---

Tu,u as the maximum reward for any self-loop transition in RM state we can modify our expression
for the expected return an option corresponding transition (u, u', ip, r) in state (s, u) as follows:
Ex=KKs' [Fa, + Tuua +...
Tau + (r 7V w(
Fa,u
1 Vop(s)
Eg' Fu,u
+
1 Vop(8)
(s) + (uk('))
Tu,a
For the previous lava example, the expected return for the option that corresponds to the transition labelled (-lavo, 0) now correctly reflects a reward of -1 obtained for each step until the option terminates by the agent exiting the lava. We modify our approximation of V (s, u) as follows:
Vo)(o)) 1 1-7
1 1-
V (s. u)
max (a' (6)))
Tuuu
(4)
We also modify the Value Iteration algorithm to consider self-loops with non-zero rewards (Algo- rithm 3). Recall that the Value Iteration algorithm represents the RM as a high-level MDP where RM transitions treated singular actions under discount factor YRM- However, in the actual MDP (with discount factor 7). an RM transition may take several steps to be achieved. One way of viewing the high-level MDP is that it makes a simplifying assumption that all RM transitions take some random variable number of steps k ~ K to complete (where is the same for all RM transitions) and "RM = E [7]]. In other words, 7AM represents the expected amount of discounting upon the RM transition's 's completion.
Now, consider an RM transition from state u to u' that receives reward but that self-loops in state u for K steps (receiving reward Tu,u on each step) before the transition completes. Under the aforementioned assumptions, the expected discounted return of taking this RM transition, including the self-loop rewards, is:
BANK K 1 ('ruw) + 7 o (v'))
(5)
1=0l
Fu,al gg ?
Exnk
2/t v2()
(6)
RRM 1
Tu,u
+ (T + (u)))
(7)
Algorithm 3 Value Iteration over RM States (modified for self-loop transitions)
Input: RM R (41, to. F, AP, 8u, 6,), MDP discount factor y, high-level discount factor 7RM Initialize rzze (u) +- 0, Vu E i
2: Extract maximum self-loop rewards Tu,us Yu li from d,-
3: while not converged do
4: for u in 2/ do
5:
b +*)((u + max(6 (Tu,a T-
L-RMM) ) + 7n + u (u')))
6: return o
A.4 Sources of Approximation Errors
We now discuss how errors can occur when estimating V (s, u) from PVFs.
First, Approximation 4 clearly introduces approximation error from assuming the largest reward among self-loop transitions Ta.. will be garnered until an outgoing transition is reached, and from using bootstrapped value estimate of the next state viz (u') ~ Vit(s,, u²). Another less obvious
27

---

source of error is that the expected return for each outgoing transition (u, u', (P, is estimated by assuming that an optimal policy for reaching yp will be followed. However, this ignores the possibility of some other transition from u occurring before the intended transition. In general, it may not be possible to satisfy &p while achieving value Vop (s) the subtask Oup while also avoiding all other transitions from RM state u that are not (u, u', (P, r).
Estimating the OVF for a disjunction of formulas via Approximation 2 aalways underestimales the true value. We prove this follows. Suppose y E V ... V Ex and recall that the approximation of (8) is max;= Vog, (+). Observe that for each i, Vog, I (s), since for every trajectory T, if & is satisfied at timestep in т, then 4p must also be satisfied at timestep 7 (or earlier) in Thus, max,= Vog, (s) I Vöp The reason this bound is not tight is because there situations where satisfying one of f1..... Ek is easier than satisfying any of fi...., & individually. For example, consider the task O (XV-XX which is always trivially solved on the first step. However, it is possible that OX and Q-X are both non-trivial tasks.
Lastly, estimating the OVF for a conjunction of formulas via Approximation 3 always overestimates the true value, by a similar line of reasoning as for disjunction. In general, knowing Vos, (s) and Vos, (s) does not provide enough information to estimate Vole, (s). For instance, it may be the case that ; f2 mutually exclusive and thus, Vo()) is zero everywhere. However, whether not L1, E2 are mutually exclusive cannot be inferred based only on Vög, (+) and Vog, (s). Nangue Tasse et al. 84 provide a discussion on this topic for a related setting.
A.5
Training Primitive Value Functions
PVFs can be trained directly from the trajectory dataset 'D based on any offline RL approach. Algorithm 4 shows how to train the PVF for single primitive task Or using a simple offline RL algorithm (Fitted Q-Iteration [85). The approach can be easily adapted to negations Q-x as well, and in practice, we simultaneously train all 2| possible PVPs in parallel as a single neural network.
Algorithm 4 Learning PVF V2_ (8) for x E AP from Trajectory Data
Input: Dataset D (T', w0) N
1= discount factor y, proposition AP
1: Initialize Q function Qa x A R
2: Initialize value function V S R
3: while not converged do
4: Sample transition (s, w, a, s', w") D
5: Update é with SGD on Binary Cross)intropy 1[r [w))
6: reward
[ E w/] next_valuue
max Qa(s a'), done + [x Ewl
7: Update 9 with SGD on (Q)(s,a) - stop grop_gradreewadd + 7 +111==+11++1111111 done) next_ valuevalue))"
8: Update ab with SGD on (V'(s') - next_ vaxt_value2
9: return V
A.6
Potential-based Reward Shaping
We extend our core leveraging Approximations 1 and trained PVFs predict V (s. for any MDP state and RM state u. This is shown in Algonthm with changes from the core algorithm highlighted in red.
Ceeppoos-RRiiffrcee algorithm with potential- based reward shaping by
B
Experimental Details
B.1 Domain Descriptions
GeoGrid is an 8 x 8 image-based gridworld depicted in Figure with six objects randomly positioned at the start of each episode. States are x x 6-dimensional images that identify each cell's colour/shape and the agent's location, while propositions (R, G, B, O) identify if the agent at an object with that particular colour or shape. DrawerWorld a MuJoCo environment adapted from Meta- World [17]. The agent controls robotic gripper and can interact with two
Environments.
28

---

Datasets. We carefully curated datasets D in each environment to support our analysis of com- positional generalization (RQ2). In GeoGrid, D is comprised of 5000 trajectories of length 100 generated under a random-action policy. In DrawerWorld, 2D is comprised of 350 trajectories of varying length that we collected by manually controlling the robot the simulator. To ensure sufficient state coverage in D. drawers and boxes were initialized in random configuration when collecting each trajectory. Behaviours that appear in the dataset include opening and closing drawers, picking up boxes, and moving boxes from one location to another. but no trajectory involves direct interaction with more than one box. A small number of trajectories involve incidental (but not prolonged) interaction with more than one box (e.g. bumping into one box while moving another). We intentionally include accidental behaviours in the DrawerWorld dataset such as failing to grip a Algorithm 5 Gtrundd-oomposeRRRiffrre for RMs with Potential-Based Reward Shaping
Input: MDP M without rewards, Propositional symbols AP, Dataset of labelled trajectories, RM task R.
over AP, Shaping potential weighting coefficient À
// Pretraining phase
1: Train labelling function [(s) on DD using any binary classification method
2: Train PVFs Voz (s) and Vo-a (s), Var E AP on D
// Behaviour elicitation phase
3: Initialize policy (w s, arbitrarily
4: for each episode do
5: Observe initial state s in .M; set u to the initial state of R
6: Estimate initial value Rs VR (s, u)) using Approximations and trained PVFs
7:
while is non-terminal do
8:
Sample action a ~ nπ( s, u)
9:
Execute a in .M and observe next state
10:
Compute truth assignment e <- L(s')
11:
Update RM: x' + (u, (),
6, Update value u' Vi (s v²) using Approximations 1 and trained PVFs
12:
13:
Update policy RR with RL for transition (8, a, o, + A - v), xd)
14:
Set 8 < 8', u 4- u', U. f-
drawers (left and right) and three boxes (red, green, and blue). Observations are 78-dimensional vectors representing positions of objects and the gripper. Propositions identify whether particular drawer is open, whether particular block is picked up by the agent, and whether particular block is currently inside a particular drawer.
box, dropping box while attempting to move it, and opening drawer beyond its limit. We also include behaviours not tied to downstream tasks such as placing a box on top of a drawer or throwing a box off the table.
Tasks. We designed diverse set of RM tasks (Table D. For tasks that involve achieving (temporally extended) goal, the RM terminates and provides a reward of I upon doing so. The RM terminates with reward of 0 if the goal becomes logically impossible (e.g. due breaking a constraint), except in Safety, which terminates with a penalty of -1. Loop and Hold-Red- Box involve repeating some desired behaviour, and the RM yields a reward of for each such repetition. For the precise encodings of tasks as RMs,, please see the released code.
B.2 Baseline Implementation Details
All approaches involve supervised training on D, and Ground-Compose-Bie Reinffrce and Bespoke Reward Model additionally require an RL phase in the environment. Network architectures for supervised training on 2 are reported in Table PPO network architectures are reported Table 4 All policy networks (whether trained via RL or behaviour cloning) use GRUs to model temporal dependencies except for GCR, which uses RM transitions. The policy network outputs a probability distribution over actions (in DrawerWorld, the outputs of the network parameterize Gaussian policy's 's mean and standard deviation). For methods relying on potential-bas computed shaping rewards without discounting the next potential; i.e., we issued the shaping reward as X(v' v) rather than X(ye' v). Though this loses some theoretical convergence properties, we found it to significantly outperform the standard shaping reward in all cases.
reward shaping, we
29

---

Table 3: Network Architectures for Supervised Training D.
GCR
LTL-BC
Bespoke Reward Model
Bespoke BC
Labelling Function
PVEs
GeoGrid
Obs Encoder:
C6 3,,1,1,1)
RaL.U
C
(6nv2d 16,3,,111)
Conv2d(6,16,3,.)
Conv2d(6,32,3g
Flatten Linear (2048, (56)
ReLU
ReLU
Comv2a(16,32,5,112
Comv2a(16,323,11.))
33mv2d(32,32,3,1,17
LTTL Encoder:
Rel.U
Rel.Il
Transforma (d_nodel-64, nhead-4,
Flatten Linear (2048, GRU (256, 256)
Flatten
Flatten
Flatten Linear (2048, 256)
Linear (2048,256)
Linear (2048,
(256 256)
din_ red.eerdddddd1125 layeree))
ReLU
ReLU
ReLUU
ReLU
Linear (128,5)
Linear (256,
(256
(256 256)
Linear (768,4) X
Linear (768,
Policy:
(256-644,
(256 256) Linear (768,16)
DrawerWorld
Obs Encoder: Linear (39, 1600)
ReLU
(1near(1600,400)
(39, 1600)
Linear (39, 1600)
LTL Encoder: Transformar (d_model=64,
ReLU
ReLU
Linear (39,)600)
(1600,400)
(1600,400)
Linear 160,)600) 1600)
ReLU
BeLU
(1600,400)
din_ sin__feedforward1128, mun_layers-2)
nhead-4,
ReLU
(4000 (56)
ReLU
(400,00, ,56)
Linear Linear(1600, .1)
ReLU
ReLU
ReLU (256, 266)
Linear (400,
GRU (256, 256)
Policy:
Linear (912,3)
(912, 24)
(864
ReL.U)
GRU( 256)
Linear(976,
GCR consists of the following neural networks: labelling function network that outputs a single binary classification logit for each proposition in AP, a PVF network that outputs an optimal value prediction for each literal in AP, and a policy of the form < (&t, u.) that conditions on the current RM state. The labelling function is trained via binary cross entropy loss on D, the PVFs trained via offline RI on D, and the policy is trained via RI. supported by the labelling function and PVFs to provide learning signals.
In GeoGrid, PVFs were trained using Fitted Qteeraaion [85]. In DrawerWorld, PVFs were trained to directly predict Monte Carlo returns. We also considered a state-of-the--a offline RL method, MCQ (86), but it performed worse than Monte Carlo regression. We attribute this to the relatively small size of D compared to standard offline RL benchmarks.
LTL-conditioned Behaviour Cloning. This baseline models neural network policy ma(a, hh, s)), where the history h; is encoded by a GRU 774 and yo (a goal represented directly LTL) is encoded by Transformer [87]. Only observations (and not actions) are encoded as part of the history. We trained Te by labelling each trajectory 7¹ in D with an LTL formula y based on the sequence of propositional labels o in D and then minimized the behaviour cloning loss Ep [- log тe (a(a)) hes se)]- Finally, we evaluated the model on the downstream tasks by conditioning on the LTL formulas shown in Table
To the best our knowledge, there are no existing approaches that generate LTL descriptions based on single trajectory. We instead used custom approach based on common specification templates to generate diverse LTL descriptions. For each trajectory т in D, we randomly generated a single formula that is satisfied by r for each of the following specification templates found in Table 2 of Menghi al. [8 visit, sequenced visit, ordered visit, patrolling (for this purpose, we consider an event to occur infinitely often if it occurs at least five times within the same trajectory in GeoGrid or
30

---

Table 4: Network Architectures for PPO.
GeoGrid
DrawerWorld
GCR
Bespoke Reward Model
GCR
Bespoke Reward Model
Encoder:
Encoder:
1,,,,,,,,66,,,,1.
Conv2d(6,16,3,,
ReLU
Actor:
ReLU
Conv2d(16,32,3,1)) (on,2d(16,32,3,,
Linear (78-)))))))))
Encoder:
ReLU
ReLU
Linear (78,512)
ReLU
Flatten
Linear (512,512)
ReLU Linear (512, 512)
Flatten
GRU (2048, 128)
ReLU
Linear (512,512)
(R12 512)
Actor Head:
Actor Head: Linear (128, 128)
ReLU
Linear (2048+
Linear Linear(512,8)
Actor Head: Linear (512,512)
ReLU
ReLU Linear(128, (128,64)
Linear (128, 64)
Critie: Linear (78+]44),
ReLU Linear (512,8)
ReLU
ReLU Linear (64,4) 4)
Linear(644 4)
ReLU
Linear (512,512)
Critic Head:
Critic Head:
Critic Head: Linear (128, 128)
ReLU
Linear (512, (12),
Linear (anear(208s+(4)) (2048+}4)),
Linear (512,
ReLU Linear (512,
ReLU
ReLU
ReLU
Linear (12a,r112,,000 (128,64)
Linear (128,64)
Linear (512,1)
ReLU
ReLU
Linear (64,1)
Linear (64,1) 1)
200 times within the same trajectory in DrawerWorld) and global avoidance. These templates were chosen since they correspond to LTL properties that are relatively simple to automatically mine from given trajectory. We then labelled each trajectory with randomly chosen LTL formula from among this set.
Bespoke Reward Model. This baseline is single neural network that directly predicts the reward, termination, and optimal value function for each of the downstream tasks. The neural network consists of an observation encoder, followed by two GRU layers (to encode the history of observations), followed by three linear output heads predict rewards, optimal values, and terminations, respectively, for all downstream tasks simultaneously for that domain. To generate target rewards and terminations for a trajectory т in D. we evaluated the RM each downstream task based on the sequence of propositional labels o'. The optimal value estimates were trained using offline RL in similar manner as the PVFs. Finally, a policy was obtained using RL on the rewards and terminations, while the optimal values were used for potential-basee reward shaping, similar to the shaped version of GCR.
Bespoke Behaviour Cloning. This baseline similar to LTL-BC. except does not condition on an LTL instead, it simultaneously outputs actions for each of the possible downstream tasks. To evaluate the policy on a specific downstream task, only the output for that task is considered. We trained the policy via behaviour cloning on any trajectory that achieves positive return particular downstream task due the limited number of successful demonstrations.
B.3 Experimental Setup and Hyperparameter Details
Details: Supervised Training on D. All experiments were run on a compute cluster. Supervised training on D required a single GPU and CPU, minimal memory resources (24GB of RAM or less) and no more than 30 minutes to train any method to 100 epochs. We tuned hyperparameters via a line search over batch size, learning rate, LI regularization coefficient, and epochs (in that order) using held-out 10% the trajectories in D, and the final hyperparameters are reported in Table Final models were retrained the full data with the tuned hyperparameters. We note that the batch size hyperparameter should be interpreted differently for methods requiring a GRU. For GCR, it refers the number of transitions sampled from D. For LTL-BC, Bespoke Reward Model and Bespoke Behaviour Cloning, it refers to the number of full length trajectories sampled from D. This is because is necessary keep transitions in trajectory the correct order to train the GRU.
31

---

Table 5: LTL formulas used to evaluate LTL-BC.
Task
LTL Formula
GeoGrid (propositions named r.g.,c,1 instead instead of R, C, A, avoid confusion with LTL operators)
Sequence
((r A d) oily A dj A e)))
Loop
no ((r A A O(( A A (
Logic
o(r o(r Olg o ortp:://hh.oa.aaa..ooaaaaaaaaaar B ((r (((
Safety
(66A AO-I
DrawerWorld
Hold-Red-Box
DORodRoxTifted
Pickup-Fach-a
RRed
OfHlueBoxLitteed OCreeenoollftee))
Show-Green-Box
(- Dowwwwrr O(DrawerlOpen GreenBoxt,ifteel
(GreaBBoslaDDawwerr2 O(Drawer2Open'/ GreenBoxLifted))
A-( GreewBoxlnDewweer1 A DrrweeB A OGreenoos
Table 6: Hyperparameters for Supervised Training on DD
Hyperparameter
GCR
LTL-BC
Bespoke Reward Model
Bespoke BC
Labelling Fanction
PVPs
GeoGrid
Batch size
256
1024
100
100
100
Learning rate
3e-4
3c-4
lc-4
3c-3
3c-4
L1 loss
lc:5
0
0
0
0
Epochs
10
100
100
11
Discount factor
n.a.
0.97
n.a.
0.97
n.a.
DrawerWorld
Batch size
256
256
50
50
Learning rate
3e-4
3e-4
1e-3
le-4
lc-4
L1 loss Epochs
1c-5
lc-5
0
0
1c-9
100
100
87
11
4
Discount factor
0.9975
0.9975
Details: RL Training. All experiments were run on a compute cluster. Each RL run used a single GPU, 16 CPUs, and 48GB of RAM. For GCR,
took up to 6 hours GeoGrid (to train to 15M frames) and 16 hours on DrawerWorld (to train to 20M frames). RL training with the Bespoke Reward Model took longer due GRUs up 12 hours GeoGrid (to train to 15M frames) and 18 hours on Drawer World (to train to 20M frames). For RL training, we used the implementation of PPO at httpps: /tttps://githbb www.comm//cc.e homm/ccswill 1lems/ ttr//ttrcchhaaa ac with the hyperparameters in Table The total number of environment steps each method was trained on was 2.5M for Sequence, 4M for Loop, 10M for Safety, 20M for Show-Green-Bor and 15M for all others.
B.4 Learning Curves
We report RL learning curves for each method and task in T and 7 GCR and variants have internal reward model that is better aligned with the ground truth compared Bespoke Reward Model. GeoGrid, GCR's s reward model is near perfect, and in all cases, optimizing its internal rewards improves ground truth performance well. Bespoke Reward Model almost always predicts near-zero rewards on GeoGrid tasks (except Safety), likely since there are few examples of positive demonstrations in D. On DrawerWorld, Bespoke Reward-Mode is highly misaligned, predicting large rewards but garnering near-zero return based on the ground truth.
In terms of sample efficiency, observe that GCR with reward shaping strategy outperforms all baselines. The difference is marginal in GeoGrid, where exploration is less of an issuc, but in DrawerWorld, all other reward shaping approaches fail.
32

---

Table 7: RL Training Hyperparameters.
Hyperparameter
Value for all methods
GeoGrid
Number of parallel environments
Frames update process
1000
Learning rate
3e-4
Discount factor (?)
0.97
GAB parameter (X)
0.95
Clip range
0.2
Entropy coefficient
le-4
Value loss coefficient
0.5
Number of epochs per update
4
Minibatch size
4000
High-level Discount Factor (TRM)
0.97
Shaping potential weighting coefficient (A)
DrawerWorld
Number of parallel environments
16
Frames per update per process
4000
Learning rate
3e-4
Discount factor (?))
0,99
GAE parameter (X)
0.99
Clip range
0.2
Entropy coefficient
0.01 Pickup Tickup-Each-Bal Each-Box, otherwise 0.03
Value loss coefficient
0.5
Number of epochs per update
10
Minibatch size
8000
High-level Discount Factor (TRM)
0.9975
Shaping potential weighting coefficient (A)
0.1 in Hold-Red-Box Box, otherwise
33

---

Sequence
Sequence
1.0
0.06
Undiscorated Return
Undiscounted Return
0..4
0.6
Frames
Loop
Loop
5.0
Undiscounted Return
Undiscounted
25
-3.04
Frames
Frames
Logic
Logic
Undincaunted Ruturn
0.0000
Undiscouned
-0.0004
0M
15M
Frames
Frames
Safety
Safety
Undizcounted Retuam
Undiscountd
0.18
0M
3M
6M
3M
6M
4M
aM
Frames
Frames
Methods
Textures
GCR (Ours)
Actual rewards
High-Level
Bespoke Reward Model
Perceived rewards
Figure 6: RL learning curves for GeoGrid, showing returns under the agent's 's own reward model ("perceived rewards") and under the ground-truth L- ("actual rewards"), Perceived rewards are reported without shaped rewards, and shaded regions show standard error. Approaches based on Ground- Ground-Compose Compose-Reinforce (including No RS and High-Level RS) generate rewards that are closely aligned with the ground truth and lead to an effective final policy, while Bespoke Reward Model produces near-zero rewards in cases and makes little progress.
34

---

Hold-Red-Box
Hold-Red-Box
400
Undisceented Return
1500
Rutum
1003
-000
0M
10M
1MM
12M
Frames
Pickup-Each-Box
Pickup-Each-Box
Undizounted Retur
Undizcounted Ruturn
0.6
0.0
0M
SM
10M
15M
0M
6M
12M
Frames
Frames
Show-Green-Box
Show-Green-Box
0.76
Undisoounted Reburn
Undiscounted
0.50
0.20
0.00
0M
BM
12M
16M
6M
8M
16M
Frames
Frames
Methods
Textures
GCR (Ours)
No RS
Actual rewards
High-Level RS
Bespoke Reward Model
Perceived rewards
Figure 7: RL learning curves for DrawerWorld, showing returns under the agent's reward model ("perceived rewards") and under the ground-truth L ("actual rewards"). Perceived rewards are reported without shaped rewards, and shaded regions show standard error. When evaluated under ground-truth rewards, Ground-Compon Reinffrre with our reward shaping strategy learns strong policies in all cases, while alternative approaches make no progress. Notably, Bespoke Reward Model results in final policy with high perceived rewards, but poor actual performance.
35

---

C
Autoformalizing Natural Language to Reward Machines
While several works have been dedicated to the autoformalization natural language instructions into formal languages such LTL [9). show that modern LLM can sometimes perform this task zero-shot for RMs, without having specifically been trained on it (to the best of our knowledge). This allows directly task the RL agent in framework through natural language, then autoformalize the task description into an RMM
We tested OpenAl's ChatGPT-40 and o3 models the autoformalizer and prompted it with description the autoformalization task (including the output format), a text description of the environment, list of propositions and associated text descriptions, and text description of the desired reward function (Listing IT. For each of the four GeoGrid tasks in Table [ we ran each autoformalizer five times for consistency. We manually evaluated each outputted RM based whether yielded reward function that exactly matched the textual description, with the success rate reported in Table X
ChatGPT-40 correctly produced RMs for all tasks, except for Logic, which requires complex RM (our solution involved 10 states and 18 transitions). However, we note that ChatGPT-4o was nearly correct for all five trials for Logic-each of its outputted RMs deviated by a single transition that changed the behaviour of the resultant reward function. o3 outputted correct RMs on all tasks. Notably, the outputted RMs were identical in structure to the intended RMs we manually constructed in all cases (with the only differences being in the naming of RM states and the representation of equivalent logical formulas).
You are given a list propositional symbols and their descriptions, an environment description, and a description of a desired reward function in English. Your job is to construct Reward Machine representing this reward function.
Revard Machine states should be numbered 0, 1, 2, 3,
with 1 alvays being the initial state, and always being the terminal tuple (i, j. \varphi
state. Transitions should be represented r), where and are the start and end Reward Machine states the transition, respectively, Avarphi is logical formula over the set of propositional symbols (use "!" to represent "not", "&" to represent "and", and "I" to represent "or", and write the formula disjunctive normal form), and transition. Your output should be the transitions Machine, one per line, in the tuple form shoun above, e.g. (0, 1, !XRY, 0.1), with no other punctuation. For brevity, do not list self-loop transitions that provide 0 reward in the output.
the reward for the
the Revard
Environment Description: The environment is gridworld, where some squares have objects. Each object has a single colour (red, green, or blue) and single shape (circle, or triangle).
Propositions:
red: The agent 's current cell has a red object
blue: The agent's 's current cell has a blue object
green: The agent's current cell has green object.
triangle: The agent's current cell has triangle.
circle: The agent's current cell has circle.
Task: <TASK DESCRIPTION
Listing Reward Machine Autoformalization Prompt
Societal Impact
Data-efficient learning lowers environmental cost. Ground-Compose-- Hempoos--Reiffore (GCR) achieves strong generalization from relatively small, task-agnostic trajectory dataset. Because it avoids the
36

---

Table 8: Success rate for ChatGPT-440 and o3 when autoformalizing Reward Machines from natural language description of the desired reward function.
Task Sequence
Description
GPT-4o Success Rate
03 Success Rate
Give reward of I and terminate the episode when triangle, a green triangle, and blue triangle have been reached, in that order. Only give the reward of when the final step has been completed. Give 0 reward and never terminate the episode otherwise.
100%
100%
Loop
Give reward of when a red triangle, green triangle, and a blue triangle have been reached, in that order. Only give the reward of 1 when the final step has been completed. After completing this sequence, the agent may repeat all steps of the sequence to receive the reward again, many times as wishes. The episode never terminates.
100%
100%
Logic
Give reward of and terminate the cpisode as soon as red triangle, red circle, green triangle, green circle, blue triangle, and blue circle have all been reached at some point. However, blue objects should not be visited until both red ob- jects are visited, and green objects should not be visited until both blue objects visited. Circles and triangles of the same colour he reached in either order. any ohjects are reached out order, immediately terminate the episode with a reward of O.
0%
100%
Safety
Give reward of when a red object, green object, and a blue object have been reached, in that order. Only give the reward of when the final step has been completed. However, always avoid squares with triangles- this is violated, immediately terminate the episode with reward of -1.
100%
100%
need for internet-scale demonstrations, the total compute and data collection burden is substantially reduced, which turn diminishes the carbon footprint of training and retraining large decision- decision-making systems.
Transparent, verifiable task specifications. By exposing an explicit formal specification layer (Reward Machines) between the human and the agent, GCR allows auditors to read, simulate, and formally verify the reward logic before deployment. This contrasts with opaque end-to-end reward models and can help regulators trace undesirable behaviour back to concrete symbolic condition rather than a latent neural representation, supporting safer and more accountablc RL pipelines.
Broader access to capable agents. Because symbols are grounded once and then recomposed to create an unbounded task space, domain experts without ML backgrounds can author complex tasks simply by writing RMs, potentially democratizing advanced robotics and simulation tools in education, manufacturing, and assistive settings. The same mechanism lets small-lab researchers prototype complex multi-stage tasks without the costs associated with collecting new labelled rewards.
Reward hacking and specification gaps. If the learned interpretation of propositions in the envi- ronment is erroncous, the resultant behaviour may no longer match human intent. In the paper, we caution that such groundin can lead to harmful behaviours despite the use of precise formal specification.
Labour displacement. Easier programming of general- -urposse robotic skills may substitute for manual labour in logistics or assembly lines, contributing to job displacement without adequate social safety nets.
37


Reviews:
[{"review_id": "PgiTWcV9Dx", "reviewer": "Reviewer_9UHD", "summary": "This paper is a follow-up on the Reward Machines line of research. There are two key innovations to the work with respect to previous papers. First, it learns the label function from data, so it relies on what is true in a reward machine on this learned function. Second, it uses the steps within an RM as subgoals to derive dense reward signals. The method is evaluated on a gridworld problem and a Mujoco task. The results show that the learned label function with a subgoal-based learned reward function outperforms baselines that are variations of the proposed method.", "questions": "- Could you please explain Equation 1 to me? Why is this a reasonable approximation? \n- What are the novel pieces of the system?", "limitations": "The paper addresses the main limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The direction of combining formal languages and RL is quite compelling, so I like the general direction of this work. \n- One of the key issues with this type of neurosymbolic method is the amount of human-annotated data, and this work represents a small step toward streamlining the use of RM in practice. \n- The idea of using the transitions of the RM to generate a denser reward function is also interesting and valuable, although I am not entirely sure how novel it is. \n- The paper has one of the best RM presentations that I have seen. The writing flows smoothly, with examples sprinkled throughout - nicely done! \n\nWeaknesses:\n\n- The experiments presented in the paper are on the weaker side because they use only five random seeds, which is often insufficient to draw conclusions in RL experiments. The paper presents only a table with the return results, while learning curves could also be informative. \n- The paper doesn't provide any rationale for Equation 1. I don't understand why $V*$ multiplies the sum $r + v_R*$. Why would the return of achieving a subgoal be multiplied by the reward observed in one time step, added to the stateless value function? This makes no sense to me. I went to the appendix to try to understand this equation, but still failed, as I don't understand the step from $\\gamma$ to $V^*$.\n- The paper is of moderate novelty at best.", "was_revised": true, "final_justification": "I thank the authors for answering my questions. I better understand the novelty of the contribution and the intuition behind Equation 1. I am thus raising my score.", "rebuttal": "Thanks for your positive evaluation of our work. We appreciate your recognition of the quality of our presentation, and we’re glad that you found the examples in the paper useful. \n\nAn important contribution that we’d like to reemphasize is our novel problem setting—we believe this work provides a new and compelling perspective on how to build language-based agents in domains with limited data. Below, we explain the advantages of our high-level framework over current approaches and why we think our insights are valuable to a broader community.\n\nWe hope our response clarifies the value of this work, and we look forward to a productive discussion period. \n\n## [“What are the novel pieces of the system?”]\n\nWe agree that each component of our approach is based on prior techniques (though our reward shaping approach has key differences from previous works, as we later explain). However, the aggregation of these components solves an important problem in a novel way that yields advantages over prior approaches.\n\nSpecifically, **our primary contribution in this work is an end-to-end framework** that takes a limited dataset $\\mathcal{D}$ of labelled agent trajectories and, via (formal) language prompting, elicits a wide range of useful behaviours. **This is achieved without significant manual engineering or oracle access to any external, environment-specific functions** (e.g. a reward or labelling function). Recall that we only require access to the following:\n\n- a dataset $\\mathcal{D}$ of agent trajectories labelled with propositional evaluations\n- a task, abstractly specified as an RM\n- an environment (without the reward function) with which the agent interacts\n\n**Why is this different from prior works in RL?** To build a language-conditioned agent, one must capture a complex mapping from language to states, actions, and behaviours in the environment. Language-driven RL typically assumes access to a reward function that captures this mapping by incentivizing the completion of any given language task (e.g. [1-4]). Similarly, formal-language-based RL (e.g. with RMs or LTL) assumes access to a labelling function, from which rewards are derived automatically. These approaches are shown to perform well when the reward or labelling function faithfully reflects language under all scenarios, but they rarely address where such a function comes from.\n\nUnfortunately, designing such reward/labelling functions can be notoriously hard [5]. For example, MetaWorld’s manually designed reward function for picking and placing an object: (i) involves complex logic spanning over 100 lines of Python code; (ii) requires access to internal simulator variables; (iii) targets only a single task.\n\nOur end-to-end approach captures complex and meaningful MetaWorld behaviours from only a modest dataset $\\mathcal{D}$ of 350 labelled trajectories. We systematically construct dense reward functions that: (i) require no environment-specific design; (ii) are based on the agent’s raw observations; (iii) can target any RM task over the set of propositional symbols.\n\n**Why is this different from agentic foundation models (e.g. vision-language-action models/VLAs)?** Our end-to-end framework builds language-based agents in a manner that is inspired by VLAs—with emphasis on leveraging existing trajectory data (i.e. $\\mathcal{D}$), and with minimal manual design. However, the key advantage of our approach over VLAs is that we explicitly leverage language compositionality to demonstrate a novel form of generalization.\n\nWe designed our experiments with controlled datasets $\\mathcal{D}$ to evaluate generalization to behaviours that are significantly out-of-distribution with respect to $\\mathcal{D}$. In GeoGrid, $\\mathcal{D}$ is comprised solely of random-action trajectories, but our agent achieved near-optimal performance on a diversity of complex, temporally extended tasks. In our MetaWorld-based domain, each trajectory in $\\mathcal{D}$ interacts with at most one box, but our agent learned to reliably pick up all three boxes within the same trajectory.\n\nTo demonstrate why this kind of generalization is hard, we compared against non-compositional approaches that learn from $\\mathcal{D}$. These methods failed to reliably solve these out-of-distribution tasks, despite having advance knowledge of the tasks when training on $\\mathcal{D}$. \n\n**To provide further evidence, we have run a new baseline that more closely mimics how VLAs are trained.** Specifically, we labelled each trajectory in $\\mathcal{D}$ with a corresponding description in Linear Temporal Logic (LTL) based on the propositional evaluations for that trajectory, and then trained an LTL-conditioned policy via behaviour cloning on that data. For each task in Table 1, we prompted this policy with an LTL description that closely matches the intended RM task. This baseline also failed to solve any tasks, further supporting our claim that non-compositional approaches do not exhibit this kind of generalization.\n\n**We will clarify these points of novelty and add our new results to the manuscript.**\n\n[1]: Chevalier-Boisvert, et al. \"BabyAI: A platform to study the sample efficiency of grounded language learning.\" ICLR (2019).\n\n[2]: Hermann, et al. \"Grounded language learning in a simulated 3d world.\" arXiv:1706.06551 (2017).\n\n[3]: Hill, et al. \"Grounded language learning fast and slow.\" ICLR (2021).\n\n[4]: Chaplot, et al. \"Gated-attention architectures for task-oriented language grounding.\" AAAI (2018).\n\n[5]: Amodei & Clark. Faulty Reward Functions in the Wild. https://blog.openai.com/faulty-reward-functions (2016).\n\n\n## [“using the transitions of the RM to generate a denser reward function is also interesting and valuable, although I am not entirely sure how novel it is.”]\n\nPrior works exploit continuous “distance metrics” towards the satisfaction of propositions (references [30-35] in the paper). However, our approach is different in two ways:\n- Prior works typically manually design these distance metrics. Our entire framework operates without relying on such manual design, and we instead show how these metrics can be learned directly via offline RL techniques from $\\mathcal{D}$. \n- Prior works focus on satisfaction of a formal specification $\\varphi$ (i.e. with a binary success criterion) and estimate a notion of distance to satisfying the overall $\\varphi$. However, it’s unclear how to apply their techniques to general RMs, which are not strictly limited to goals of a binary nature. Inspired by [27] in the main paper, our approach is based on estimating the optimal value function for any RM.\n\n**We will make these differences more clear in the related works section.**\n\n## [“The paper presents only a table with the return results, while learning curves could also be informative.”]\n\nDue to space limitations, the learning curves are currently found at the end of the Appendix (Figures 5, 6) but are referenced in the main text on line 326 and 333-336. \n\nBecause of our unique setting, where part of the challenge is to learn to generate useful rewards based on the dataset $\\mathcal{D}$, the learning curves show performance w.r.t. both these learned rewards as well as a ground-truth notion of rewards. \n\n## [“The experiments presented in the paper are on the weaker side because they use only five random seeds”]\n\nWe agree that more seeds would strengthen the results. The number of seeds is informed by a few factors:\n- Computational cost: Our RL domains are challenging and require up to 20M frames of training. The computational cost is exacerbated by inference costs from the labelling function and PVF networks as well as the internal logic of the RM. These calls must be made on every step of the environment.\n- Stability of results: The error bars in the learning curves and the table already demonstrate consistent performance across seeds.\n\nWe believe five seeds balances the robustness of our claims with computational cost, in line with RL community standards. \n\n## [“Could you please explain Equation 1 to me?”]\n\nNote that there is a minor mistake in Equation 1: the term inside the maximization should read $V^{\\ast}_{\\Diamond \\varphi}(s) \\cdot (r + \\gamma v^{\\ast}\\_\\mathcal{R}(u'))$ (however, based on your comment, this doesn’t appear to be the source of confusion).\n\nRecall that we estimate the return for immediately satisfying the RM transition $\\langle u, u’, \\varphi, r \\rangle$ as $r + \\gamma v^\\ast\\_{\\mathcal{R}}(u’)$.  $r$ is immediately received from the RM transition and $v^\\ast\\_{\\mathcal{R}}(u’)$ estimates the future discounted return for being in RM state $u’$. Again, we believe this part is not under dispute.\n\nSuppose that the number of steps it takes to satisfy $\\varphi$ (which causes the RM transition) from the agent’s current state $s$ is $K$ (where $K$ is a random variable that could be $\\infty$ if the RM transition never occurs). The delayed return for satisfying $\\varphi$ $k \\sim K$ steps in the future is $\\gamma^k ( r + \\gamma v^\\ast\\_{\\mathcal{R}}(u’) )$, or in expectation, $\\mathbb{E}\\_{k \\sim K}[\\gamma^k ( r + \\gamma v^\\ast\\_{\\mathcal{R}}(u’) )]$. Since the second factor is constant w.r.t. $k$, we can rewrite this as $(\\mathbb{E}\\_{k \\sim K}[\\gamma^k]) \\cdot ( r + \\gamma v^\\ast\\_{\\mathcal{R}}(u’) )$.\n\nTo maximize this expected discounted return, the agent ought to maximize $\\mathbb{E}\\_{k \\sim K}[\\gamma^k]$ (which balances the probability of eventually satisfying $\\varphi$ with satisfying it quickly). This is precisely the expected discounted return from state $s$ for the task $\\Diamond \\varphi$ (which yields a reward of 1 on the timestep $\\varphi$ is satisfied), hence the maximum value of $E_{k \\sim K}[\\gamma^k]$ is $V^*_{\\Diamond \\varphi}(s)$, which results in our desired expression.\n\nWe hope this clarifies Equation 1 and we will add this explanation to the Appendix."}, {"review_id": "lLGMOHdpbm", "reviewer": "Reviewer_jbaf", "summary": "This paper investigates whether diverse data is necessary for developing broad capabilities in large-scale foundation models through a data-efficient approach termed *Ground-Compose-Reinforce*. The authors propose a method that first grounds a predefined set of symbols in the environment using a labeled trajectory dataset. This grounding is performed by an agent equipped with the compositional semantics of a formal language. Once grounded, the agent can be tasked through this formal language and can learn to solve a variety of tasks by generating its own learning signals. The approach is evaluated in an image-based gridworld and a MuJoCo robotics domain, demonstrating data efficiency in training from trajectories while supporting diverse, out-of-distribution capabilities through explicit leverage of compositionality. The paper highlights the limitations of current data-driven agents in complex, safety-critical domains like robotics, where data generation is costly and out-of-distribution generalization is challenging.", "questions": "1. Is the dataset D in DrawerWorld manually constructed, and does it also contain failed trajectories? How are the failed trajectories constructed?\n2. Can a performance comparison be made with fully data-driven large-scale foundation models?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strength\n\n1. The authors' goal of building a data-efficient learning paradigm beyond the current mainstream data-driven large-scale foundation models is meaningful.\n2. The running examples provided in the paper, such as Figure 1, are easy to understand.\n\n### Weakness\n\n1. The authors' key claim is that current data-driven large-scale foundation models cannot handle complex multi-stage tasks, and the algorithm designed in this paper is based on this. Although the authors have stated the reasons for this claim in the introduction, they have not compared it with fully data-driven large-scale foundation models in the experiments, which makes the motivation of the paper weak.\n2. The authors state that, \"However, complex multi-stage tasks like building a house remain challenging even for state-of-the-art models.\" In fact, some works, such as [1], have already been able to complete similar multi-stage tasks like building a house in Minecraft, and the constructed house has good functional characteristics.\n\n[1] Guo, et al., “Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification”", "was_revised": true, "final_justification": "Concerns remained about the paper's \"data-driven\" claim and comparison with data-driven models:\n\n- A large amount of data is the core of fully data-driven approaches. The authors cannot conduct data-driven training on a limited dataset and then claim that the proposed method is superior to the data-driven paradigm.\n- The fully data-driven paradigm can accommodate diverse data types, not all of which can be expressed in the form of symbols. This is also a disadvantage of this paper compared to the data-driven paradigm.", "rebuttal": "Thank you for your review. We're pleased that you found the broader goal of our work—developing agents through a more data-efficient pipeline—to be meaningful and a key strength of the paper.\n\nBelow, we address each of your questions and concerns in detail. We hope our responses adequately address your points, and convey the value of our contributions. We welcome further discussion and are happy to engage during the discussion period. \n\nIf we have adequately addressed your concerns, please consider raising your score. If we have not, please let us know of any remaining concerns.\n\n## [“The authors state that, \"However, complex multi-stage tasks like building a house remain challenging even for state-of-the-art models.\" In fact, some works, such as [1], have already been able to complete similar multi-stage tasks like building a house in Minecraft”]\n\nThanks for raising this important point. The scope of our claim in the introduction was too broad and we will revise it to be more accurate. \n\n**However, the value of our work does not ultimately hinge on this specific claim.** Our goal of building language-driven agents through a more data-efficient pipeline is important for domains where agent-interaction data is limited or expensive. This applies to many important domains, such as those that require real-world interactions (e.g. robots) [1], those that are safety-critical, and those that involve atypical observation and action spaces [2]. Prior evidence suggests that standard approaches for training foundation models are not well-suited in these data-impoverished settings [3]. \n\nAs MineCraft does not exemplify such a domain (there exist massive and diverse datasets of gameplay generated by human players [4, 5]) we will remove it as a motivating example. \n\n[1]: Levine. https://sergeylevine.substack.com/p/sporks-of-agi (2025).\n\n[2]: Bommasani, et al. \"On the opportunities and risks of foundation models.\" arXiv:2108.07258 (2021).\n\n[3]: Kaplan, et al. \"Scaling laws for neural language models.\" arXiv:2001.08361 (2020).\n\n[4]: Guss, et al. \"MineRL: A large-scale dataset of minecraft demonstrations.\" IJCAI (2019).\n\n[5]: Baker, et al. \"Video pretraining (VPT): Learning to act by watching unlabeled online videos.\" NeurIPS (2022).\n\n\n## [“Can a performance comparison be made with fully data-driven large-scale foundation models?”]\n\nWe do not compare with existing large-scale foundation models for a few reasons:\n\n- **It is not relevant to the research question at hand.** Our goal is to explore whether we can build language-based agents in settings with limited agent-interaction datasets—and furthermore, whether they can generalize to behaviours beyond this data. Comparing with an existing model that has already been trained on large-scale data does not inform our answer as it violates the premise of our question while making it hard to assess generalization.\n\n- **Our domains are customized.** DrawerWorld is a modified version of MetaWorld with additional objects. This diversifies the possible behaviours in the environment to help us test multi-task capabilities and generalization, but it also means that our observation space is atypical. Similarly, GeoGrid observations are encoded as 8x8x6 arrays in a non-standard format.\n \nHowever, we agree that comparing our approach with the training techniques of these foundation models under equivalent training data conditions is appropriate and informative. \n\n**Thus, we have run an additional comparison with a formal-language-conditioned behaviour cloning baseline.** To train this baseline, we labelled each trajectory in our trajectory dataset ($\\mathcal{D}$ in the paper) with a corresponding description in Linear Temporal Logic (LTL) based on the propositional evaluations for that trajectory, and then trained an LTL-conditioned policy via behaviour cloning on that data. For each task in Table 1, we prompted this policy with an LTL description that closely matches the intended RM task. This baseline failed to solve any tasks, roughly matching the performance of “End-to-End Behaviour Cloning”. We will add these details to our revised manuscript.\n\nNote that the original baselines in our paper, End-to-End Reward Model and End-to-End Behaviour Cloning, already support our claim that non-compositional approaches fail to generalize to significantly out-of-distribution tasks given limited training trajectories in $\\mathcal{D}$. However, this new baseline provides a closer comparison to how agentic foundation models (e.g. vision-language-action models) are usually trained. \n\n\n## [“Is the dataset D in DrawerWorld manually constructed”]\n\nThe dataset $\\mathcal{D}$ in DrawerWorld was constructed by one of the authors manually controlling the robot via keyboard inputs. To ensure a diverse set of scenarios, the initial configuration of the environment was randomized (in terms of locations of boxes, and whether each drawer was open or closed), and the robot was controlled to interact with drawers and boxes (e.g. moving boxes into or out of drawers). In total, 350 trajectories were collected, and importantly, none of these trajectories involved direct interactions with multiple boxes. This allowed us to evaluate out-of-distribution generalization to picking up multiple boxes within the same episode. **We will add these details, as well as those in the response to follow, to the paper or appendix.**\n\n\n## [“does it also contain failed trajectories?”]\n\nThe data includes accidental behaviours such as failing to grip a box, dropping a box when attempting to move it, and opening a drawer beyond its limit. We also included behaviours not tied to downstream tasks, such as placing boxes on top of a drawer, or throwing a box off the table. \n\nThe inclusion of these behaviours in $\\mathcal{D}$ improves the robustness of models trained on $\\mathcal{D}$. For example, we found that only including successful demonstrations leads to increased reward hacking by the downstream RL agent—the agent might deliberately reach out-of-distribution states (with respect to $\\mathcal{D}$), because the model’s outputs are poorly calibrated on such states and often result in erroneously high rewards. \n\nThis presents an interesting problem in itself, but is outside the scope of this paper. For this work, we simply assume the dataset $\\mathcal{D}$ covers a sufficiently diverse set of states to robustly capture the propositions we care about (e.g. whether a drawer is open, or a box is picked up). Nevertheless, as our experiments show, we are still able to demonstrate generalization to tasks significantly out-of-distribution with respect to $\\mathcal{D}$ (e.g. picking up all three boxes within the same episode, or holding a box as long as possible)."}, {"review_id": "FvtDOBpaIK", "reviewer": "Reviewer_J8VM", "summary": "The proposed method inputs a set of propositional symbols and trains a labelling function to ground propositions to state using an input dataset. Given a new task expressed in formal language as a RM, the trained labeling function is then used to generate rewards for reward shaping in RL. The idea is to use provided symbols and RMs for tasks, and learn general labelling and task-specific reward functions to solve those tasks.\n\nContributions claims:\n- Symbol grounding can be learned with predefined set of symbols and a dataset of trajectories and ground truth labels.\n- The proposed reward shaping can be done with offline RL and any provided RMs (without rewards) and improves sample-efficiency and generalization in RL.\n\nAssumptions made by the approach:\n- A set of symbols or propositions are provided.\n- Ground truth labelling function exists (although unobservable) so it can be used along with a dataset of trajectories to train a labelling function to ground predefined symbols to states. \n- The new task is provided as a RM (without rewards but using the same predefined symbols and capturing the temporal structure of the new task).", "questions": "Please refer to the weakness section for questions.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Weaknesses and questions:\n- The trained labelling function learns the symbol mapping for ground states. How is it also used to simulate rewards? It seems the text in the core algorithm is abstracting out necessary details - it mentions L (that returns symbols) but does not explain \\delta_u and \\delta_r functions used to simulate the RM states and rewards. Are \\delta functions what represents given RM and so are. They assumed to be given? They need to defined and explained. Currently, these notations are not referred at all in the text.\n- Currently, the novelty comes from learning a labeling function (which assumes ground truth labels for training) and learning rewards for a RM using offline RL (which assumes no self-transitions which are common in RMs). Other ideas of RMs already exist in the literature. Many reward shaping ideas also already exist (even with RMs). How is the proposed approach h different from them?  \n- What do you mean by unsupervised RL?\n- What exact behaviors in the tasks are not present in the dataset (Table 1)? \n- The empirical evaluation seems more of an ablation study. Have you tried comparing with any state-of-the-art RL approach, such as PPO [1], CAT+RL [2], Dreamer [3]? \n- The related work seems cursory and needs more specific comparisons with most related works. Discuss in more detail about the methods that learn symbol grounding and explore reward shaping. \n\nOverall assessment:\n\nI am of the opinion that the paper is making too many assumptions across different aspects of the approach, and some of them (such as learning a RM instead of assuming it is given or removing self-transition assumption of RMs for generating rewards, etc) should be relaxed for a sufficient contribution. The novelty currently is minimal and insufficient for acceptance - training a labelling function and using provided RMs (with symbols, not rewards) to generate rewards does not seem enough. It is not clear, how the reward shaping ideas are significantly more novel compared to previous reward shaping ideas explored in the RM or LTL or other RL literature.\n\n\n\nReferences:\n\n[1] Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O., 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.\n\n[2] Dadvar, M., Nayyar, R.K. and Srivastava, S., 2023, July. Conditional abstraction trees for sample-efficient reinforcement learning. In Uncertainty in Artificial Intelligence (pp. 485-495). PMLR.\n\n[3] Okada, M. and Taniguchi, T., 2022, October. DreamingV2: Reinforcement learning with discrete world models without reconstruction. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 985-991). IEEE.", "was_revised": true, "final_justification": "I appreciate the authors’ response and have read the other reviews. In my view, assuming access to a reward machine and using it to generate rewards—rather than assuming access to a reward function—feels natural. However, this approach doesn’t seem sufficiently novel to me, especially since assuming the availability of a reward machine for a new task is a strong assumption. The main novelty appears to lie in learning the labeling function from data. I recommend that the authors clearly emphasize the key differences from prior reward machine approaches in the paper (how it accounts for concrete states when estimating the value of RM states). Based on these considerations, I am slightly increasing my score.", "rebuttal": "Thanks for your detailed review and questions. Below we address your questions and comments. We believe our responses address your concerns but if any issues remain, please let us know during the discussion period—otherwise, we kindly ask you to consider updating our score.\n## [“I am of the opinion that the paper is making too many assumptions…such as learning a RM instead of assuming it is given”]\n**We assume the RM is given, not learned**. Note that we do not assume access to an external reward signal to define the agent’s objective. Instead, our goal is to task the agent directly through an RM.\n\n## [“Are \\delta functions what represents given RM and so are. They assumed to be given? They need to defined and explained. Currently, these notations are not referred at all in the text.”]\nThe formal definition of an RM is in terms of $\\delta_r, \\delta_u$ (Definition 1 in the paper), and they govern the rewards and transitions of the RM on each step (as described on lines 89-95). Yes, they are assumed to be given as part of the RM.\n\nHowever, it is typically more convenient to specify an RM’s transitions via edges labelled with a logical condition and a corresponding reward (as in Figure 1). This is ultimately equivalent to specifying via $\\delta_r, \\delta_u$. Lines 107-109 explain how $\\delta_r, \\delta_u$ are defined when an RM is specified via labelled edges.\n\n## [“removing self-transition assumption of RMs for generating rewards”]\n**Our approach does in fact address self-loop transitions in RMs**, and our experiments explicitly test this scenario in the Lift-Red-Box task. \n\nIn the main text, we present Approximation 1 under the assumption of no self-loop transitions to improve the clarity of the exposition—handling self-loops involves several extra steps. This is stated on line 236, and then the reader is pointed to the extension in the Appendix that “handles any RM” on line 237.\n\n## [“The trained labelling function learns the symbol mapping for ground states. How is it also used to simulate rewards?”]\nAn RM is an abstract specification of a reward function, similar to a natural language statement that  \"a reward of 1 is given for picking up a red box, then picking up a blue box\". However, an RM makes the temporal structure explicit. To simulate rewards for an RM expressing this statement, we need a labelling function that, for any environment, determines whether: (1) the red box is currently picked up; (2) the blue box is currently picked up. Then, for any trajectory in the environment, once it has been observed that (1) has occurred at some time $t$, and that (2) has occurred at some time $t’ > t$, then a reward of 1 is issued at time $t’$. This last step is done by systematically updating the RM state according to $\\delta_u$ and issuing rewards according to $\\delta_r$. \n\n## [“What do you mean by unsupervised RL?”]\n\nRL agents typically learn meaningful behaviours by optimizing an external reward function (i.e. the reward signal \"supervises\" the agent). This is also a prominent approach for building language-conditioned agents via RL—often, the agent optimizes a reward function that incentivizes the completion of any given instruction (e.g. [1-4]). These works show that agents can successfully learn to follow language instructions with a reward function that faithfully captures language, but they rarely address where this reward function comes from.\n\nUnfortunately, designing such reward functions can be notoriously hard [5]. For example, MetaWorld’s manually designed reward function for picking and placing an object: (i) involves complex logic spanning over 100 lines of Python code; (ii) requires access to internal simulator variables; (iii) targets only a single task.\n\n**The distinguishing feature of our framework is that it does not assume access to an external reward function** (or any other external, environment-specific functions e.g. a labelling function). Rather, it reflects an end-to-end pipeline for building language-conditioned agents without environment-specific design. Recall that we only assume access to the following:\n- a dataset $\\mathcal{D}$ of agent trajectories labelled with propositional evaluations\n- a task, abstractly specified as an RM (similar to those shown in Figure 1)\n- an environment (without the reward function) with which the agent interacts\n\n**When we say that the agent learns via unsupervised RL, we mean that it trains via interaction with the environment, but does not have access to any external learning signal**. Instead, our problem setting requires the agent to derive its own reward signal for any given RM task based on its grounded understanding of the formal language. \n\n**Why is this setting hard?** The agent is essentially evaluating its own performance and optimizing it with RL—this runs the risk of reward hacking if the agent’s understanding of the formal language does not faithfully generalize across a wide range of scenarios. \n\n[1]: Chevalier-Boisvert, et al. \"BabyAI: A platform to study the sample efficiency of grounded language learning.\" ICLR (2019).\n\n[2]: Hermann, et al. \"Grounded language learning in a simulated 3d world.\" arXiv:1706.06551 (2017).\n\n[3]: Hill, et al. \"Grounded language learning fast and slow.\" ICLR (2021).\n\n[4]: Chaplot, et al. \"Gated-attention architectures for task-oriented language grounding.\" AAAI (2018).\n\n[5]: Amodei & Clark. Faulty Reward Functions in the Wild. https://blog.openai.com/faulty-reward-functions (2016).\n\n## [“Other ideas of RMs already exist in the literature. Many reward shaping ideas also already exist (even with RMs). How is the proposed approach h different from them?”]\n\nOur approach differs from prior RM works because it is end-to-end—we do not require an external labelling function that (in conjunction with the RM) provides the agent’s objective. This allows us to provide a novel perspective on how to build language-based agents with minimal manual design, while only relying on a limited set of labelled agent-interaction data. \nBased on this new problem setting, we demonstrate a novel form of generalization that we argue is not exhibited by non-compositional approaches. \n\nBelow, we also describe the two most relevant lines of work on RM reward shaping and how our approach differs:\n- One idea is to estimate the value of being in any RM state $u$ (without considering the environment state $s$) to generate a shaping signal (e.g. see [27-29] in the paper). Our approach improves on this by also considering $s$ when estimating the value—our contribution is how such a value function can be derived based on $\\mathcal{D}$ for any RM task. We directly compare against this baseline in the experiments.\n- Another idea is to use some external signal that measures how close each proposition is to satisfaction (e.g. Manhattan distance in a gridworld) and to compose them to estimate the distance to the overall goal (e.g. [30-35] in the paper). However, these methods only apply to temporal goals with a binary success criterion. Our work’s novel contributions include: (1) showing how the initial signals can be learned directly from $\\mathcal{D}$ via offline RL and (2) extending this to arbitrary RMs, which can express rich preference structures beyond binary goals.\n\n## [“What exact behaviors in the tasks are not present in the dataset (Table 1)?”]\n- In our grid experiments, $\\mathcal{D}$ consists solely of random-action trajectories, but our agent achieves near-optimal performance on a diversity of complex, temporally extended tasks. \n- In our MetaWorld-based experiments, we controlled the dataset $\\mathcal{D}$ such that each trajectory interacted with at most one box, but the Pickup-Each-Box task requires the agent to pick up the red box, then the blue box, then the green box, all within the same trajectory. \n- The Lift-Red-Box task requires the agent to pick up and hold the red box as long as possible. The longest that the red box is held for in $\\mathcal{D}$ is 736 steps, while our agent learns a policy that holds it for 1538 steps on average (this is close to the upper limit, since episodes last 2000 steps).\n\nOur experiments show that several non-compositional approaches fail to solve these tasks under the limited dataset $\\mathcal{D}$, even when given advance knowledge of these tasks when training on $\\mathcal{D}$.  \n\n## [“The empirical evaluation seems more of an ablation study.”]\n\nWe are proposing a novel problem setting and to our knowledge, few existing methods are directly applicable. Thus, our experiments aim to uncover what elements enable robust language learning and generalization. Most importantly, we show the importance of exploiting the compositional syntax and semantics of the formal language.\n\n**Nevertheless, we have run a new baseline inspired by how agentic foundation models are typically trained**. Specifically, we labelled each trajectory in $\\mathcal{D}$ with a corresponding description in Linear Temporal Logic (LTL) based on the propositional evaluations for that trajectory, and then trained an LTL-conditioned policy via behaviour cloning on that data. For each task in Table 1, we prompted this policy with an LTL description that closely matches the intended RM task. This baseline also failed to solve any tasks. \n\n## [“Have you tried comparing with any state-of-the-art RL approach, such as PPO [1], CAT+RL [2], Dreamer [3]?”]\n\n**These approaches solve an orthogonal problem to our problem setting.** They are concerned with learning a policy to optimize a given reward function. Our setting does not assume access to an external reward function. Rather, learning (from $\\mathcal{D}$) to generate meaningful rewards that align with any given RM task is key to the problem we are trying to solve.\n\nNote that these rewards generated by our framework can be used in tandem with any standard, model-free RL algorithm to elicit behaviours. In fact, all **RL-based methods in our experiments are optimized with PPO** (see lines 320-321)."}, {"review_id": "i5hvieySkQ", "reviewer": "Reviewer_hsjd", "summary": "The paper introduces a data-efficient neurosymbolic framework for RL. The method is the following. First, the agent grounds a fixed vocabulary of symbols in the environment by training a labelling function on a small set of labelled trajectories; then it composes these grounded symbols using the semantics of a formal language (Reward Machines) to specify arbitrary tasks; finally it reinforces itself by generating its own reward signals and learning through standard RL. To address sparse rewards, the authors propose a compositional reward-shaping strategy. Experiments in gridworld and robotics domains show that the method generalizes out-of-distribution for new tasks, obtained by the composition of the simpler ones, encountered in the training set. The compositional reward-shaping allows for successful training.", "questions": "See weaknesses (1).", "limitations": "See weaknesses (2).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n - A clear and good introduction to the subject of Reward Machines, along with a well-defined and formalized approach description.\n - Interesting generalization results.\n\nWeaknesses:\n - The paper would greatly benefit from additional evaluations, for example, the comparison to something like EMMA (https://proceedings.mlr.press/v139/hanjie21a.html), which is conditioned on natural language, deeper investigation into the compositional generalization ability (to what extent does it hold?), and possible cross-domain generalization, i.e., same concepts but in the different domain.\n - The limitation of the number of PVFs that should be fitted is not discussed. It is unclear how this approach scales to the open-ended real-world tasks with a large possible number of concepts.", "was_revised": true, "final_justification": "My concerns are addressed, and I rely on the authors to provide more details in the final version of the paper.", "rebuttal": "Thanks for your positive and thoughtful review. We appreciate that you recognized the paper’s presentation and generalization results as strengths. We believe your summary also accurately reflects the contributions of the paper.\n\n## [“The paper would greatly benefit from additional evaluations, for example, the comparison to something like EMMA”] \n\nWhile works like EMMA are also concerned with teaching agents to understand language in a grounded manner, **they operate under a different set of assumptions and are not directly applicable to our problem setting**.\n\nIn EMMA, the agent learns language by optimizing an external reward signal—this represents a popular strategy for building language-conditioned agents with RL [1-4]. Unfortunately, manually building a reward function that faithfully captures language under all scenarios can be challenging, and this is an important issue that our framework addresses.\n\n\n**Our framework represents an end-to-end pipeline for building language-conditioned agents** from a limited dataset $\\mathcal{D}$ of labelled agent trajectories. **It does not rely on oracle access to any external, environment-specific functions** (e.g. a reward or labelling function). Instead, we ground language directly from the finite number of examples in $\\mathcal{D}$.\n\nRecall that we only require access to the following to elicit behaviours:\n- a dataset $\\mathcal{D}$ of agent trajectories labelled with propositional evaluations\n- a task, abstractly specified as an RM\n- an environment (without the reward function) with which the agent interacts\n\nThis pipeline is inspired by agentic foundation models (e.g. vision-language-action/VLA models) that emphasize the use of existing agent-interaction data (e.g. $\\mathcal{D}$) over manual design (e.g. of reward functions). However, the key advantage of our approach over VLAs is that we explicitly leverage language compositionality to demonstrate a novel form of generalization.\n\n**To provide further evidence of our claims, we have run a new baseline that more closely mimics how VLAs are trained.** Specifically, we labelled each trajectory in $\\mathcal{D}$ with a corresponding description in Linear Temporal Logic (LTL) based on the propositional evaluations for that trajectory, and then trained an LTL-conditioned policy via behaviour cloning on that data. For each task in Table 1, we prompted this policy with an LTL description that closely matches the intended RM task. This baseline also failed to solve any tasks, further supporting our claim that non-compositional approaches do not exhibit this kind of generalization.\n\n**We will clarify these differences with respect to related works and add these new results to the manuscript.**\n\n[1]: Chevalier-Boisvert, et al. \"BabyAI: A platform to study the sample efficiency of grounded language learning.\" ICLR (2019).\n\n[2]: Hermann, et al. \"Grounded language learning in a simulated 3d world.\" arXiv:1706.06551 (2017).\n\n[3]: Hill, et al. \"Grounded language learning fast and slow.\" ICLR (2021).\n\n[4]: Chaplot, et al. \"Gated-attention architectures for task-oriented language grounding.\" AAAI (2018).\n\n\n## [“deeper investigation into the compositional generalization ability (to what extent does it hold?)”]\nThis is an excellent question. The advantage of using a formal language is that the compositional rules are explicit—in our case, we can systematically interpret any RM in terms of its propositional symbols. The key observation that enables compositional generalization is that we can reduce the problem of grounding the formal language of RMs (i.e. capturing how *every* possible RM should be interpreted in the environment) to grounding the finite set of propositional symbols of the language. \n\nStated more concretely, **so long as the learned labelling function $\\hat{\\mathcal{L}}(s)$ represents a faithful grounding of the propositions $\\mathcal{AP}$, then the rewards generated by our core framework (Algorithm 1) will faithfully reflect *any* RM task over $\\mathcal{AP}$, by design**. While one can never ensure that a grounding of propositions learned from data is perfectly faithful, our experiments show that a learned labelling function $\\hat{\\mathcal{L}}(s)$ can indeed faithfully elicit RM behaviours in practice (when evaluated with respect to a ground-truth labelling function). This observation also informs what an ideal dataset $\\mathcal{D}$ might look like in practice—it should contain a sufficient diversity of *states* to learn a robust classifier of propositional values. Crucially, it does not necessarily require a sufficient diversity of *trajectories*. This explains why our framework can faithfully elicit a wide diversity of temporally extended behaviours (expressed as RMs), even when $\\mathcal{D}$ contains no successful demonstrations of such behaviours.\n\n**We will include this discussion in the paper.**\n\n## [“The limitation of the number of PVFs that should be fitted is not discussed. It is unclear how this approach scales to the open-ended real-world tasks with a large possible number of concepts.”] \n\nThe number of PVFs is linear in the number of propositional concepts, $|\\mathcal{AP}|$. We agree that this may scale poorly to more open-ended environments, and in such cases, a promising strategy is to further decompose propositions, e.g. via object and relational representations. Such representations are also compositional and have been shown to be conducive to generalization (e.g. as shown in [5]). \n\n[5]: Hill, et al. \"Environmental drivers of systematicity and generalization in a situated agent.\" ICLR (2020)."}]

Author rebuttal:

Meta-review: The paper introduces ground-compose-reinforce, a neurosymbolic framework for tasking rl agents via formal language. Symbols are grounded from labeled trajectories, composed using reward machines to specify complex tasks, and reinforcement is achieved by exploiting compositional structure to self generate learning signal. A compositional reward shaping strategy predicts value functions for arbitrary tasks, enabling efficient learning in settings of sparse rewards. Exps in gridworld and mujoco show data-efficient training + generalization to diverse, OOD tasks beyond the original trajectories .

In the original round of reviews the committee appreciated the clear and formalized presentation of the reward machines framework and the strong exposition with intuitive examples (reviewers 9UHD, hsjd), the direction of combining formal lang. with rl for data efficiency and compositional generalization (R#jbaf, R#hsjd), and the ability to elicit diverse out-of-distribution behaviors from small trajectory datasets (R=9UHD,J8VM). The overall technical solidity and potential impcat of the neurosymbolic approach was recognized.

During rebuttal and discussion, the authors clarified the novelty of their framework, explained eq. 1 in detail, and added new baselines mimicking foundation-model training, which supported their generalization claims. R-9UHD’s concerns on equation rationale and novelty were resolved, leading to a higher score; R-jbaf acknowledged the advantage in data-limited regimes and raised their score despite comparisons to large-scale models; R-J8VM appreciated clarifications also raised their score; reviewer hsjd’s requests on evaluation and scaling were addressed, and they raised their score to 5. The AC concurs with the majority of the committee’s reviews and post-rebuttal discussions that lean towards acceptance. The authors are encouraged to incorporate constructive discussion in a revised version of the paper.
Decision: Accept (poster)