ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
Authors: Roger Creus Castanyer
Faisal Mohamed
Pablo Samuel Castro
Cyrus Neary
Glen Berseth
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Reward Machines
Foundation Models
Generalization
Exploration
Primary area: reinforcement learning
TL;DR: We introduce a framework where foundation models automatically generate compositional reward machines from language, enabling reinforcement learning agents to solve sparse reward tasks and achieve zero-shot generalization to new tasks.

Abstract: Reinforcement learning (RL) algorithms are highly sensitive to reward function specification, which remains a central challenge limiting their broad applicability. We present ARM-FM: Automated Reward Machines via Foundation Models, a framework for automated, compositional reward design in RL that leverages the high-level reasoning capabilities of foundation models (FMs). Reward machines (RMs) - an automata-based formalism for reward specification - are used as the mechanism for RL objective specification, and are automatically constructed via the use of FMs. The structured formalism of RMs yields effective task decompositions, while the use of FMs enables objective specifications in natural language. Concretely, we (i) use FMs to automatically generate RMs from natural language specifications;  (ii) associate language embeddings with each RM automata-state to enable generalization across tasks; and (iii) provide empirical evidence of ARM-FM's effectiveness in a diverse suite of challenging environments, including evidence of zero-shot generalization.

Published as a conference paper at ICLR 2026
ARM-FM: AUTOMATED REWARD MACHINES VIA FOUNDATION MODELS FOR COMPOSITIONAL REINFORCEMENT LEARNING
Roger Creus Castanyer 1.2
Faisal Mohamed 1.2
Pablo Samuel Castro 1.2
Cyrus Neary
Glen Berseth 1,2
1 Université de Montréal
2 Mila Quebec Al Institute
3 University of British Columbia
ABSTRACT
Reinforcement learning (RL) algorithms are highly sensitive to reward function specification, which remains central challenge limiting their broad applicability. We present ARM-FM: Automated Reward Machines via Foundation Models, framework for automated, compositional reward design in RL that leverages the high-level reasoning capabilities of foundation models (FMs). Reward machines (RMs) - an automata-based formalism for reward specification - are used as the mechanism for RL objective specification, and are automatically constructed via the use of FMs. The structured formalism of RMs yields effective task decomposi- tions, while the use of FMs enables objective specifications natural language. Concretely, we (i) use FMs to automatically generate RMs from natural language specifications; (ii) associate language embeddings with each RM automata- state to enable generalization across tasks; and (iii) provide empirical evidence of ARM-FM's s effectiveness in a diverse suite of challenging environments, including evidence of zero-shot generalization.
INTRODUCTION
A central challenge in reinforcement learning (RL) is the design of effective reward functions for complex tasks. The shape of the reward influences the complexity of the problem at hand (Gupta
cet al. 2022); for instance, sparse rewards provide insufficient learning signal, making difficult for agents to improve (Devidze et al., 2022). Even hand-crafted dense rewards are susceptible to
unintended loopholes 'reward hacking" where agent exploits the specification without achieving the true objective (Fu et al., 2025). The unifying challenge is thus how to communicate complex objectives an agent in manner that provides structured, actionable guidance (Rani al. (025).
While Foundation Models (FMs) excel at interpreting and decomposing tasks from natural language, a critical gap exists in translating this abstract understanding into the concrete structured reward signals necessary for RL. Consequently, high-level plans generated by FMs often fail to ground effectively, leaving the agent without the granular feedback required for learning. To bridge this gap, turn to Reward Machines (RMs), an automata based formalism. By decomposing tasks into a finite automaton of sub- sub-goals, RMs provide compositional structure for both rewards and policies that inherently more structured and verifiable than monolithic reward functions (Tcarte et al. (022). While theoretically principled, their practical application has been confined to fask task-specific applications due to the complexity of their manual, expert-driven design. We posit that the reasoning and code- generation capabilities of modern FMs are well-suited to automate the design and construction of RMs, thereby unlocking their potential solve the broader challenge of communicating complex objectives in RL; the resulting RMs can thus translate abstract human intent into a concrete learning signal for solving complex tasks.
This work makes three primary contributions. First, we develop a novel framework for automatically generating complete task specifications directly from natural language using foundation models,

---

Published as a conference paper at ICLR 2026
introducing language-aligned reward machines (LARMs) which include the automaton structure, executable labeling functions, and natural language instructions for each subtask. Second, we introduce method that leverages the language-aligna nature of the resulting automata to create a shared skill space, enabling effective experience reuse and policy transfer across related tasks. Finally, we provide extensive empirical validation demonstrating that our approach solves complex, long-horizon tasks across multiple domains that are generally intractable for standard RL methods. Specifically, our results show that the framework (i) dramatically improves sample-efficiency by converting sparse rewards into dense, structured learning signals; (ii) scales to diverse set of environments, including grid worlds, complex 3D environments, and robotics with continuous control; and (iii) enables efficient multi-task training and zero-shot generalization.
Reward Machine Generation
Reward Language Specefication
STATES
TRANSITIO PURCTION monann, JUSCTION
Baseline
Prompt
Hie spent his puck up ball in room batisdlo locked door
Foundation Model
Labellng Purction
dul got Juy
Succaa r6t)
dut Jost Jay reav):
doc 0002 (C2P)
Instructions & Embeedings
at opent has the ony, The door ii.."
teleeted
0.25
RL Training
A
Ouni
R2
RL Agent
plotaal RRRM
Rg
Enwironment
@b ((SMM
R2
S
10:0
SRM SEF1
Reward Machine
( FM Embedcingg
Labeiling Functions
100
24
324
44
Figure 1: An overview of our framework (left) and results in a complex sparse-reward environment (right). Reward Machine Generation (top-left): Given a high-level natural language prompt and visual observation of the environment, a FM automatically generates the formal specification of the Reward Machine, the executable Python code for the labeling functi inns, and the natural language descriptions for each RM state. RL training (bottom-left); During the RL training loop, the labeling functions evaluate environment observations to update the Reward Machine's state, which provides a dense reward signal RRM The RL agent's 's policy receives the environment observation along with the embedding ((-) of the current RM state's 's language description, making aware of its active sub-goal. Empirical results (right): Results in a complex sparse-reward Minecraft- based resource- goaturiggatheiing task from Cr ft i um (Malagón al. 2024), where RL agent is unable to make progress (top), while our agent, guided by an FM-generated LARM,, learns to solve the task efficiently (bottom).
2 AUTOMATED REWARD MACHINES VIA FOUNDATION MODELS
We now present Automated Reward Machines via Foundation Models (ARM-FM), a framework for automated reward design in RL that leverages the reasoning capabilities of foundation models to automatically translate complex, natural-language task descriptions into structured task represen- tations for RL training. Figure Diirustrttsss an overview of ARM-FM, which comprises two major components: (i) the introduction of Language-Aligeen Aligned RMs (LARMs), which are automatically constructed using FMs; and (ii) their integration into RL. training by conditioning policies on language embeddings of RM states, enabling structured rewards, generalization, and skill reuse. Figure shows a high-level task description, consisting of a natural language prompt and a visual observation (left), along with its corresponding RM (right). The RM is a finite-state automaton that guides an agent by providing incremental rewards for completing sub- sub-goals, such as collecting keys and opening doors, on the way to the final objective. In the following section, we describe how our ARM-FM framework automates the creation of these reward machines directly from high-level task descriptions.
2

---

Published as a conference paper at ICLR 2026
2.1 LANGUAGE-ALIGNER REWARD MACHINES
We assume the standard RL formalism, which defines an environment
Markov
Task Description
A 0.1
Decision Processes (Puterman 1994] MDPs) (S, A, R. P), where 5 the set
"The egent in pack on a ball room bahind
door apened
not got key -0.1
regures unlock door befouee finding the bult
got R. key
of MDP states, A is the set of possible ac- tions, R S A + is the MDP re- ward function, and P : S A + ((S) a probabilistic MDP transition fune- tion. A Reward Machine (RM) is finite-state automaton that encodes com- plex, temporally extended, and potentially non-Markovian RL tasks (Icarte et al.] 2022). We formally define an RM by the tuple (U, ur Σ, 6, R, F, L). Here, U is the finite set of RM states; uy the initial state of the RM; Σ is the finite set of symbols representing events that cause transitions in the RM; 8: X U the deterministic RM transition function; R : U S A S + R is the RM reward function; F C U is the set of final RM states; and L S x A - E is the labeling function that connects MDP states s E S and actions are useful for describing tasks at an abstract level, especially when said tasks require multiple steps over long time horizons. Each RM state u E U can be thought of as representing a subtask, and the transitions u' 8(u, o)) denote progress to a new stage of the overall objective after particular event a E occurs in the environment. The RM reward function R(u, 8, a, s') assigns a reward based on the current RM state and the underlying MDP transition (s, a, s'). Mcanwhile, the set of final RM states F defines the conditions under which the task described by the RM is complete. Finally, the labeling function L is required to connect the RM's events, transitions, and rewards to states and actions from the underlying MDP.
Each color openz
rolor"
not get 2 bry -0.1
+01
dow opened
622
2ot bull
1.
Reward Machine
Figure 2: ARM-FM leverage FMs to automatically construct RMs:
using the walookToUnloor task description from MiniGrid d (left), RM is automati- cally constructed to solve the task (right).
to the RM event symbols o E E. Intuitively, RMs
We define LARMs as RMs that are addi- tionally equipped with natural-language in- structions Ly for each RM state u. and with an embedding function o(-) that maps such language instructions to an embedding vec- tor Za ((w.) E Rdd We note that by equipping RM states with embedding vec- tors that encode language- based descrip- tions of the corresponding subtasks, we pro- vide the first mechanism for constructing semantically grounded skill space in RMs: policies conditioned on these embeddings can naturally share knowledge across re- lated subtasks, enabling transfer, composi- tionality, and zero-shot generalization.
Text
Image
Roward Machine Generation
Prompt
seif-improvement loop
Gaseratar Prselatie Hull
Feedback
Gftlcc Pazeddiin Hode
Velfication
RL Training
Roward Machine
Language Spacification
Figure 3: A self-improvement loop where generator and critic FMs iteratively refine LARMs, with optional human verification.
We present framework automatically construct LARMs from danguag--ndd-iage--on task descriptions by iteratively prompting an FM, as is illustrated in Figure 3 More specifically, to progressively refine the RM specification, we employ N rounds of self-improvement using paired generator and critic FMs (Tian et al., 2024). A human may optionally intervene by approving the output or providing corrective feedback (see Appendix A..4 for details). In practice, we find that FM-generated reward machines are both interpretable and easily modifiable, as they follow a natural language specification. Figure 4 illustrates an automatticall--conttuuuee LARM for the Unl InlockToUnlor ock task, including text-based description of the RM (left), FM-generated labeling functions L (middle), and natural RM state instructions and embeddings L, (right). All RMs and labeling functions used in this work shown in Appendix (opendix[A.9] and A. While use code to define labeling functions in this work, the ARM-FM framework is general, supporting any boolean predicate (e.g. formal logic, queries to other FMs).
3

---

Published as a conference paper at ICLR 2026
2.2 REINFORCHMENT LEARNING WITH LARMS
The introduction of the LARM uses an augmented state space that is the cross-product the MDP and RM states (S x L.) and reward function that is the sum of the MDP and RM rewards; we will refer to this augmentes augmented MDP as M and illustrated in Figure [ (Bottom). At timestep the environment state and the language embedding of the current LARM state: (8t, Zu,). This language-base policy conditioning is the central mechanism enabling generalization in framework, creating semantically grounded skill space where instructions like "pick up blue key" and "pick up a red key" are naturally close in the embedding space, unlocking pathway for broad experience reuse and efficient policy transfer. During training, after the agent executes action de x (8t; Zu²): the underlying MDP transitions to St+1 and returns a re- ward Rt The labeling function [(5 0t) determines
the agent selects actions conditioned
Language
Instractions Embe deiigg
INTAAL STATE-0
symbolic event has oc-
The ner aitther por didldeors decni
curred, which may induce a LARM tran- sition Mt++1 8(we, (1 ())), as well as an additional reward RR The sum of the MDP and RM rewards are then used for learning: Ri = Rs + R.M. This complete training procedure, adapted for a DQN agent, is formalized in Appendix A.3 The effectiveness of well-designed LARMs yields theoretical guarantees (see Appendix A.5), ensuring that the generated reward structure preserves the optimal pol- icy of the original sparse task.
ust
sENNaR JuNcTIoN
aay meseray adrassl (ual)
pukee be wnee coveiis scatee
con toekialion 96140)
Figure 4: The three core components generated by our method for the Un InlockToUn occ environment: (Left) the RM specification, (Center) the labeling func- tions that drive the state transitions, and (Right) the instructions and embeddings for each RM state.
EMPIRICAL RESULTS
We present series of experiments designed to evaluate the effectiveness and scalability of our method: we test generalization and long-horizon planning in sparse-reward settings with the Mi Minio d and BabyAI suites (Chevalier-Be Boisvertt et al. 2023) (Section3.1]) evaluate scalability with resource- gathering task in a 3D, procedurally generated Minecraft world from Craft ium (Malagón et al.. 2024) (Section and demonstrate the applicability of approach to create RMs that work in continuous control in challenging robotics tasks from PettaWorldd (McLean et al. 2025) (Section 3.3). Finally, we use XLand-Mingg MiniGrid (Nikulin al. 2024) to evaluate the generalization capabilities of RL agents trained with LARMs (Section 3.4). Screenshots of these environments are shown in Figure 3and and environment--peciffc details in Appendix A.2 We used GPT-4o (Hurst et al. 2024) to generate all LARM components for all tasks, with the exception of the 1,000 LARMs for XLand-Ming Minddd-iiiiiid.. These were generated using various open-source FMs of different scales for our ablation study.
(d)
(a) MiniGrid
(b) Crafti fum
(c) Met.a Met.a-Wos Worrld
Figure 5: Evaluation environments. (a) MiniGrid: tests long- horizon planning with sparse rewards. (b) Craftium scaling complexity in a 3D memeeerfft-nniiiree world. (c) Meta-Worll continuous robotic manipulation. (d) Miiand tests multi-task generalization.
4

---

Published as a conference paper at ICLR 2026
DON-RR (Duns)
DON
DON-ICM
--. Reket
MintCind-Joorfeee Rey-bab-vvl
Mintorit-Durr Key- Bib--vl
Minitaid- Minitrid-Deoreey--Be Diibb
2.0
301
MiniGrid DoorKey-Sxd 5x5 vo (CMDP)
MiniGrid Docrsey Sxc vo (CMDP)
MiniGrid DoorKey 8x8-vo (CMDP)
MiniGrid DoorKey- 1616666666 1- (CMDP)
305
1008
1515
2005
250K
200%
300
330K
150K
2006
2005
300K
336
3004 Eritronmmntt Stages
130K
2008
2506
2006
3005 30C
1005
2906
2006
2015
Eneeormment Staps
Enveroreennt Stapps
Ervirommmn Staps
Figure 6: Performance on MiniGrid-Doort shows performance fixed map layout, while the bottom row shows performance on procedurally generated layouts. Our method consistently achieves higher rewards across all tasks.
environments of increasing size. The top row
We report results averaged over 3 independent random seeds, with shaded regions and error bars indicating one standard deviation. Comprehensive details for all environments as well as additional results are presented in Appendix A.2 details on the baselines used and additional ablations in A.7 and hyperparameters A.11
3.1 SPARSE REWARD TASKS
We first evaluate our method on the MiniGrid suite of environments, which are challenging due to sparse rewards. For these experiments, we use DQN (Mnih et 2013) as the base agent and compare our method (DQN+RM) with baseline with intrinsic motivation (DQN+ICM) (Pathak al. 2017), ReAct (Yao et al., (023) an LLM-as-agent baseline which generates reasoning traces as it acts in the environment (Paglieri et al., 2024) - as well as an unmodified DQN. In Appendix A.7.4 we include results comparing to VLM-as -LM-as-reward-a www.wwdwwdodeeeeeoo baseline proposed by Rocamonde ct al. 2023). The LARMs used to train our DQN+RM agent are shown in Appendix A.9
Our agent successfully solves suite of challenging exploration tasks where all baselines fail. We first present results on the Doorkey task in Figure 6 showing performance across increasing grid sizes. Our method consistently outperforms the baselines in all settings, including fixed maps (top row) and procedurally generated maps where the layout is randomized each episode (bottom row). We provide additional analysis of our method's success in Appendix A.7.2] To further test our agent, we select three significantly harder tasks that require longer planning horizons: UnlockTouniow BlockedUnn dUnlockP ockPi only one capable of solving all three tasks and achieving near-perfec reward, while all other baselines fail to make any progress.
ckup, and KeyCor ricoorrdoorr.o As demonstrated in Figure 7 our agent is the
DOM+ SM (DANSO
DON
DOX +EE
... RwAct
Babtyt Stbyyt--unledTTuuullcc
BockesUhbsctPoe..o
MeyCorddorvo
1568
223K
1008
2030
128
Figure 7: Performance on complex, long-horizon horizon MiniGrid tasks. Our method successfully solves all three, while baselines show no learning.
5

---

Published as a conference paper at ICLR 2026
SAC (Sparse reward)
SAC+ (ours)
Success Rate
Assembly
Success Rate
Pick-Place
Success Rate
Stick-Push
0.4
0.4
0.2
0.2
0.25
0.0
0.0
0.00
0.0
1.0 Environment Steps
0.5
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
le7
Environment Steps
1c7
Environment Steps
le7
Figure 8: Performance on Met Wor d manipulation tasks, most tasks, our method achieves high success rates compared to the sparse reward agent.
3.2 SCALING TO COMPLEX 3D ENVIRONMENTS
We assess our method's performance in complex, procedurally generated 3D) environment using Cra ft um (Malaóón et al., 2024). which a Minecraft- based resource- gattreiig task. Here, the agent's s goal is mine diamond by first navigating the world to gather the required wood, stone, and iron. The environment provides a sparse reward only upon collecting the final diamond. For this set of experiments we use PPO (Schulman et al. 2017) the base agent.
As shown in T PPO augmented with our generated LARM consistently completes the entire task sequence. while the baseline PPO agent makes minimal progress. This result is particularly significant, as RMs often require manual, expert-driven design which can become challenging in complex, open-ended environments (lcarte et al. 2022). In contrast, we demonstrate the successful application of a RM that not only automaticaly generated by a FM but is also highly effective in complex 3D, procedurally generated environment. The LARM achieves this by effectively decomposing the high-level goal into progressive subtasks and providing crucial intermediate rewards. This experiment highlights our framework's ability handle increased action dimensionality and visual complexity, and it showcases the capability of FMs to leverage their knowledge to automate task decomposition.
3.3 ROBOTIC MANIPULATION
Our framework can automate the complex task of reward engineering for robotic manipu- lation, providing dense supervision with FM-generated LARM. We evaluate this capability in continuous control domains using the Meta-World benchmark (McLean et al. 2025). where designing dense reward functions typically requires extensive hand-engineering of low-level signals (e.g., joint angles). Our approach bypasses this difficulty entirely. The resulting reward machine offers richer learning signals than sparse rewards, enabling the agent make more progress. As demonstrated in Figure 8 our method achieves higher success rates than learning from sparse rewards alone, using SAC (Haarnjja al. 2018) as the base agent. Additional experiments on Met aWorlld are provided in Appendix A.2.3
3.4 GENERALIZZTION THROUGH LANGUAGE EMBEDDINGS
key design choice in our framework is to condition the agent's policy on the language embeddings of the current RMM state, which contrasts with prior work that uses separate policies that do not permit knowledge sharing (Alsadat al., 2025 In the following, we (i) ablate the roles of the LARM rewards and state embeddings in enabling an RL agent to learn robust, multi-task policies; and (ii) demonstrate how the compositional structure of LARMs leads to shoo-shot generalization on unseen tasks. For clarity, we refer here to zero-shot generalization across novel task compositions within the same domain, rather than cross-domain transfer.
Both structured rewards and language-based state conditioning are essential for learning robust, multi-task policy. To disentangle the benefits of the LARM's reward structure from the state embeddings, we conduct an ablation study in multi-task setting. We train single Rainbow DQN (Hessel et al. 2018) agent on an increasing number of simultaneous XLand-Mind Miiddiiiniiiiii Gr tasks and measure its average success rate. As shown in Figure the baseline agent fails to generalize as the number of tasks increases. Providing the policy with only the state embeddings gives it a weak learning signal that degrades quickly. Conversely, providing only the LARM rewards enables

---

Published as a conference paper at ICLR 2026
multi-task learning, but the policy struggles it is unaware of the active sub-goal Our full method, which uses both the dense rewards from the LARM and the state embeddings to condition the policy, when trained 10 simultaneous tasks.
robust and maintains high performance
The compositional structure of LARMs en- ables zero-shot generalization to novel tasks composed of previously seen sub-goals. The ultimate of compositional approach is whether the trained policy, π (O 9t, Zu,), can solve a novel task without any additional train- ing. We design experiment where ı is trained on a of tasks, (TA, Ta each with an associ- ated LARM, R. and RB. During training, the policy learns skills corresponding to the union of all sub-goal embeddings, {zu ju UA L At evaluation time, we introduce a new, unseen task, Tc. with novel LARM, Rc, generated by the FM. Zero-shot success is possible the of sub-goals in the task is composed of elements semantically familiar from training. i... if for any state u' UC. its embedding Zu/ is close to an embedding seen during training. As illustrated transitions to state '/ the policy receives the input (s;, Z.;). Because the embedding zn₁ (e.g.. for "Pick up a blue key, Position yourself the right of the blue pyramid") is already located in familiar region of the skill space, the policy can reuse the relevant learned behavior to make progress and solve unseen composite task.
Baseline
No RM Reward
No Embeddings
Ours
1.0
Success Rate
0.8
0.6
0.4
0.2
3
5 10
Number of Simultaneous Environments
Figure 9: Ablation study of the components of ARM-FM. A Rainbow agent is trained on an in- creasing number of tasks. While baselines fail to generalize, only our full method (combining LARM rewards and state embeddings) main- tains high success as the number of tasks grows. Figure TO the agent successfully solves Task C. When the LARM for Task C
4 ARM-FM: IN-DEPTH ANALYSIS
We now conduct a fine-grained analysis of our method's key components by evaluating (i) the quality of the LARMs generated by different FMs; and (ii) the semantic structure the state embeddings.
Larger foundation models generate syntactically correct task structures with significantly higher reliability. To evaluate the FM's generation capabilities, we sampled 1,000 diverse tasks from the XLand-Mi mind--iinni Gric environment Niiuulin et al., 2024) and prompted various open-source models to generate the corresponding reward machines, Python labeling functions, and natural language instructions. We compare models of different scales from the Ower. (Qwen| 2025). Genma (Gemmal 2025), Llama (Dubey et al. 2024) and Mistral families. We employed an LLM-as-judgo protoco! score the correctness of the generated artifacts (Gu et al. 2024), using Qwen3-30B-Ar A3B-IIstruct
Task A
Task
Task
Pick
up key
Use the blue
blue
blue
Training
make orange star hald en orange
2) Geab Position
2) Geal: Position
Evaluation
yournelf right
yoursed the right blue pyramid
star inventory
Task C
fllu bay begin requeaee
"RowVino yourself pyramed your
Figure 10: Demonstration zero-shot generalization. An agent is trained on set of tasks (A, B). At evaluation, is given new LARM for unseen composite task (C). Because the sub-tasks in C (c.g.. "Pick up a blue key") are semantically familiar from training, the agent can reuse learned skills to solve the novel task without any fine-tuning.
7

---

Published as a conference paper at ICLR 2026
Soth Conlact
Noither Conect Reward Machine
Laseling Functions
Balilaniee nerow
Owen3 3-38
0.78
6.22
To saat beea
Owen3
0.31
acRimd IECLINE
Ad path
Heare aco
Misocl
spaatimnst Le Ju
Uama3-00
Minitrol 88 0.10
0.79
Lama3.3 83 0.07
0.41
0.32
Gemma3 270M
0.46
0.50
0.0
0.2
0.4
0.6
0.3
10
FSL
(a) FM generation correctness.
(b) Semantic structure of state embeddings.
Figure 11: Analysis FM-generated task components. (a) An LLM-as-judge evaluation across 1,000 tasks reveals strong scaling trend, where larger foundation models more reliably generate correct RM structures and verifier code. (b) PCA visualization of thousands of state instruction embeddings shows a clear semantic structure, with start, middle, and end states forming distinct clusters.
2507 the judge (a FM not used to generate the LARMs). Figure 1la shows clear scaling trend: larger models like Qwen3-32B are significantly more capable of generating fully correct task specifications. Interestingly, some models exhibit different strengths; for instance, Mistral-Small is more adept at generating valid RM structure than correct labeling code, highlighting the distinct reasoning and coding capabilities required.
The "-M-generaeed state instructions produce a semantically coherent embedding space that clusters related sub-goals. Beyond syntactic correctness, the agent's ability generalize depends on the semantic quality of the state instruction embeddings. A well-structured embedding space should group semantically similar sub-tasks together, regardless of the overarching task. We analyze this by visualizing the embeddings of state instructions from the 1,000 generated XLand-MiniGH d tasks using PCA (Abdi Williams, 2010), as shown in Figure TTb We used Qwen3-30B-A3B: Instruct-2507 to obtain the embeddings. The embeddings form distinct and meaningful clusters, with instructions corresponding to the start, middle, and end of a task occupying different regions of the space. Notably, instructions with similar meanings from different tasks cluster together, confirming that the FM produces a coherent representation. This underlying semantic structure is what enables a shared policy treat related sub-tasks in a similar manner, forming the foundation for skill transfer.
RELATED WORK
Reward Machines Reinforcement Learning. RMs are formal language representation of reward functions that expose the temporal and logical structure of tasks, thus enabling decomposition, transfer, and improved sample efficiency in learning (Icarte et al.) 2018 2022). For these reasons, RMs and related formal methods for task specification have been applied to address diverse challenges, from multiagent task decomposition (Neary et al. 2021 Smith et al. 2023) to robotic manipulation and task planning (Camacho al. He et 2015 Cai et 2021). Recent work continues to broaden their applicability, by studying extensions that increase their expressivity (Varricchione et al. 2025). and by addressing uncertainty in symbol grounding and labeling functions (Li al.) [2024] While RMs can be difficult to design for non-experts, Toro Icarte et al. (2019) and Xu et al.
2025
(2020)
propose methods that simultaneously learn RMs and RL policies, if the RM is unknown
priori.
Recent work also explores FM-driven automata. While some approaches treat RM states as isolated symbols Alsadat al. 22025), requiring careful state mapping for policy re-use, others use FMs with classic algorithms for automaton discovery expert demonstrations. Our work differs by generating RMs directly from language descriptions, without behavioral examples. Concurrently, methods like RAD embeddings Yalcinkaya et al. (2024) have been proposed to condition the policy on the automaton's 's topology. We take complementary,
Chanllate et al. (2025). which requires
8

---

Published as a conference paper at ICLR 2026
language-fire firstt approach: our FM generates tanguage--ligneeee embeddings for the meaning of each state, which our results show effectively grounds the policy.
By contrast, ARM-FM not only generates RMs from natural language task descriptions, but also introduces a natural mechanism for connecting RM states by embedding their associated subtask descriptions in shared latent space. Conditioning the policy on these language embeddings can thus enable knowledge transfer across similar subtasks, even when they occur in different RMs.
Foundation Models in Decision-Making, The emergence of FMs has inspired two main lines of work in sequential decision-making. The first uses FMs directly as autonomous agents (Paglieri
al. 2024). Approaches such ReAct, Yao al. 2023) Voyager (Wang et al. 2023) and SayCan Ahn et al. (2022) employ large language models (LLMs) to perform reasoning, planning, and acting. While these systems demonstrate strong capabilities in complex domains, they heavily depend on environment abstractions (e.g., textual interfaces or code as actions) that bypass many of the low-level perception and control challenges central to RL. In contrast, we use RMs to structure policies for learning agents, solving complex sparse reward tasks beyond the reach of non-learning, in-context methods like ReAct which additionally require high-level textual interfaces.
A second line of research integrates FMs with RL training by using them to provide auxiliary signals such as high-level goals or reward feedback. For example, Motif (Klissarov al. 2023) elicits rajectory-leve preferences from FMs and distills them into reward model. ONI (Zheng et al. 2024)
aggregates asynchronous LLM feedback into continuously updated reward function. Eureka (Ma
et al. 2023) leverages evolutionary strategies generate programmatic reward functions, which then used to train downstream policies. ELLM uses pretrained LLMS to suggest plausibly useful goals and trains RL agents with goal-reaching rewards. These approaches illustrate the potential of
injecting FM knowledge to shape RL objectives. However, the outputs are typically limited to an opaque reward model, rather than a structured, compositional representation of the task.
Our work differs in the structure of the FM-RL interface. We employ FMs to generate language- aligned RMs: structured, compositional, and interpretable representations of task reward functions. This formulation combines the expressivity of FMs with the explicit, modular decomposition, and human-in -n--tee-tttoogt refinement enabled by RMs, offering principled path toward hierarchical and interpretable RL. Additionally, our method does not depend on specific environment abstractions (Wang et al., 2023) or the availability of a natural language interface (Klissarov et al. 2023). We provide a detailed comparison with existing methods in Section A.6 Table
CONCLUSION
In this work, we introduce Automated Reward Machines via Foundation Models (ARM-FM), a framework that bridges the critical gap between the semantic reasoning of foundation models and the low-level control reinforcement learning agents. Our central contribution is a method for automatically generating Language-Aligen
Reward Machines (LARMs) from natural language. We demonstrated that by conditioning a single policy on the embeddings of the LARM's 's natural language state descriptions, transform the reward machine from a static plan into a compositional library of reusable skills. Our experiments confirmed the effectiveness of this approach. We showed that ARM-FM solves suite of long-horizon, sparse-reward tasks across diverse domains from 2D grid worlds to a procedurally generated 3D crafting environment - that are intractable for strong RL baselines. Our analysis revealed that this performance is underpinned by a coherent semantic structure in the state embedding space and that both the structured rewards and the state embeddings are critical for robust multi-tas, learning. The ultimate validation of our compositional approach was the demonstration of zero-shot generalization to a novel, unseen task without any additional training.
Ultimately, this work establishes language-aligned reward machines as a powerful and versatile framework connecting foundation models, RI. agents, and human operators. The modular, language- based structure allows FMs to generate accurate plans, agents to learn generalizable skills, and humans to easily inspect and refine the task specifications. While this paradigm is promising, one tradeoff of our approach is the human verification step during RM generation. On one hand, this step may be viewed as feature the language- based reward structures output by ARM-FM provide an interface for humans to interpret and refine task specifications. On the other hand, this step presupposes access to human verifiers. We note, however, that such verifiers are not strictly required,

---

Published as a conference paper at ICLR 2026
although they can improve output quality when available. Furthermore, future work may reduce eliminate this dependence by exploiting the automaton- structure of RMs to enable automated self-correction for example, through formal verification. More broadly, believe this work paves the way for a new class of RI. agents that can translate high-level human intent and FM-generated plans into competent, generalizable, and interpretable behavior.
ACKNOWLEDGEMENTS
We want to acknowledge funding support from Natural Sciences and Engineering Research Council (NSERC) of Canada, Google Research, and The Canadian Institute for Advanced Research (CIFAR) and compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia.
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, ((4):433-59, 2010.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
Shayan Meshkat Alsadat, Jean-Raphas Gaglione, Daniel Neider, Ufuk Topcu, and Zhe Xu. Using large language models to automate and expedite reinforcement learning with reward machine. In 2025 American Control Conference (ACC), pp. 201. IEEE, 2025.
Mingyu Cai, Mohammadhosein Hasanbeig, Shaoping Xiao, Alessandro Abate, and Zhen Kan. Modular deep reinforcement learning for continuous motion planning with temporal logic. IEEE robotics and automation letters, 6(4):7973-7980, 2021.
Alberto Camacho, Jacob Varley, Andy Zeng, Deepali Jain, Atil Iscen, and Dmitry Kalashnikov. Reward machines for vision-based robotic manipulation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 14284 14290. IEEE, 2021.
Maxime Chevalier-Boisvert,, Bolun Dai. Mark Towers, Rodrigo De Lazcano Perez- Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and K Terry. Minigrid miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Thirty- seventh Conference on Neural Information Prucessing Systems Datasets and Benchmarks Truck, 2023. httttt
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guideer reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829-5842 2022.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Picter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pp. 8657 8677. PMLR, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2409 2024.
Jiay Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint M.XXi::5522.877770 2025.
Gemma. Gemma technical report. arXiv preprint arXiv:2503.19788 2025.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on aujase--judge. arXiv preprint arXiv:2411.1559, 2024.
10

---

Published as a conference paper at ICLR 2026
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, S Kakade, and S Levine Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Adv. Neural Inf. Process. Syst., abs/2210.09577 October 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learming. pp. 1861-18701 Pmlr, 2018.
Keliang He, Morteza Lahijanian, Lydia E Kavraki, and Moshe Y Vardi. Towards manipulation planning with temporal logic specifications. In 2015 IEEE international conference robotics and automation (ICRA), pp. 346-352, IEEE, 2015.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila Mcllraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference Machine Learning, pp. 2107-2116. PMLR, 2018
Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73:173-208 2022.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00166 2023.
Martin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C Machado, and Pierluca D' O'Oro. Maestromotif: Skill design from artificial intelligence feedback. arXiv preprint arXiv:2412.08542, 2024.
Andrew Li, Zizhao Chen, Toryn Klassen, Pashootan Vaezipoor, Rodrigo Toro Icarte, and Sheila Mcllraith. Reward machines for deep rl in noisy and uncertain environments. Advances in Neural Information Processing Systems, 37:1 1011110441110666 2024.
Andrew C Li, Toryn Q Klassen, Andrew Wang, Parand A Alamdari, and Sheila A Mcllraith. Ground- compose-reinfore:: Tasking reinforcement learning agents through formal language. arXiv preprint arXiv:2507.10741, 2025.
Yecheng Jason Ma, William I. Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023.
Mikel Malagón, Josu Ceberio, and Jose A Lozano. Craftium: An extensible framework for creating reinforcement learning environments. arXiv preprint N1V:22407.03366 2024.
Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K.R. Zentner, Ryan Julian, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Meta-world+: An improved, standardized, RL benchmark. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
URL /tttps::/opeereevie. n foreet/forummmmmm 2ot
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 112.562,,,, 2013.
11

---

Published as a conference paper at ICLR 2026
Cyrus Neary, Zhe Xu, Bo Wu, and Ufuk Topeu. Reward machines for cooperative multi- agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp. 934 924-942,242 2021.
Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid: Scalable meta-reinnorceement learning environments in jax. Advances in Neural Information Processing Systems, 37:43809-4338 2024.
Davide Paglieri, Bartlomiej Cupiat, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, al. Balrog: Benchmarking agentic Ilm and vim reasoning on games, arXiv preprint arXiv:2411.13544 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity -driven exploration by self-supervised prediction. In International conference on machine learning, pp- 2778-2787. PMLR, 2017.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastie Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
Qwen. Qwen3 technical report, 2025. URLL/ttpps///rr....rrr/bbb/////
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Septia Rani, Serena Booth, and Sarath Sreedharan. Goals vs. rewards: Towards comparative study of objective specification mechanisms. In 2025 20th ACM/EEEE International Conference on Human-Rolee Interaction (HRI), pp. 1558 1562. IEEE, 2025.
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint 1rXiv223101129211 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.00007,, 2017.
Sophia Smith, Cyrus Neary, and Ufuk Topcu. Automatic decomposition of reward machines for decentralized multiagent reinforcement learning. In Proceedings of the 62nd IEEE Conference on Decision and Control, pp. 5423-5430. IEEE, 2023.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of IIms via imagination, searching. and criticizing. Advances Neural Information Processing Systems, 37:52723-52777 2024.
Rodrigo Toro Icarte, Ethan Waldie, Toryn Klassen, Rick Valenzano. Margarita Castro, and Sheila Mcllraith. Learning reward machines for partially observable reinforcement learning. Advances in neural information processing systems, 32, 2019.
Giovanni Varriechione, Toryn Q Klassen, Natasha Alechina, Mehdi Dastani, Brian Logan, and Sheila A Mcllraith. Pushdown reward machines for reinforcement learning. arXiv preprint MrXiv:2508.0684/ 2025.
Marcell Vazquez-Chanlatte Karim Elmaaroufi, Stefan Witwicki, Matei Zaharia, and Sanjit A Seshia. Im: Learning automata from demonstrations, examples, and natural language. 2025.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 2023.
Zhe Xu, Ivan Gavran, Yousef Ahmad, Rupak Majumdar, Daniel Neider, Ufuk Topeu, and Bo Wu.
Joint inference of reward machines and policies for reinforcement learning. In Proceedings of the International Conference Automated Planning and Scheduling, volume 30, pp. 590-598, 2020.
12

---

Published as a conference paper at ICLR 2026
Beyazit Yalcinkaya, Niklas Lauffer, Marcell Yazquez-Chanlatee and Sanjit Seshia. Compositional automata embeddings for goal-conditione reinforcement learning. Advances in Neural Information Processing Systems, 37:72933-72965 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Mingqi Yuan, Roger Creus Castanyer, Bo Li, Xin Jin, Wenjun Zeng, and Glen Berseth. Rlex- plore: Accelerating research in antrinsicallymmotiaale reinforcement learning. arXiv preprint arXiv:2405. 19548, 2024.
Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, and Brandon Amos. Online intrin- sic rewards for decision making agents from large language model feedback. arXiv preprint arXiv:2410.2302,, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LLM USAGE
In the preparation of this manuscript, used large language models (LFMs) writing assistants. Their role was strictly limited to improving the grammatical correctness of our text.
The LLM prompted review author-writter drafts and provide feedback on phrasing flag passages that were potentially unclear. No standalone text was generated by the LLM for inclusion in the paper. All core scientific ideas, experimental results, and analyses the original work of the human authors, who take full responsibility for the final content.
A.2 ENVIRONMENT DETAILS
A.2.1 MINIGRID AND BABYAI ENVIRONMENTS
This section provides detailed description of the MiniGrid and BabyAl environments used in experiments. These tasks are selected to test distinct agent capabilities, ranging from basic exploration and generalization complex, long-horizon planning and reasoning.
In the DoorKey task, the agent must find a key within the observable room, use it to unlock a door, and navigate goal location. The sparse reward, given only upon reaching the goal, makes this a classic exploration challenge. We use procedurally generated versions of this task to evaluate generalization novel map layouts.
The BlockkUUnlockiikku task significantly increases the planning complexity. The agent must first move a blocking object (a ball), retrieve a key from the main room, unlock door, and finally pick up target box in separate room. This requires long and precise sequence of actions to solve.
is BabyAl task that tests hierarchical reasoning and memory. The agent must find key for first door to navigate to different room, which in turn contains key for second, final door the goal room. This creates nested dependency structure with extremely sparse rewards, making it exceptionally difficult.
UnlockToUUne
The KeyCorridor environment is a difficult exploration task. The agent starts corridor with multiple rooms, one of which contains a hidden key. It must explore the side rooms to find the key, return the corridor to unlock the correct door, and reach the final goal.
g
(a) DoorKey
(b) BlockedUnnlockPiccup
(c) UnlockToUnlock
(d) KeyCorridor
A.2.2 CRAFTIUM
Craftium is nigh-performance, open-source voxel platform designed for reinforcement learning research. Inspired by Minecraft, Craftium offers rich, procedurally generated open worlds and fully destructible environments. Built on the C++-based Luanti engine, it provides significant performance advantages over Java-based alternatives and integrates natively with modern RL frameworks through the Gymnasium API. This makes an ideal testbed for assessing agent performance on tasks requiring generalization in visually complex, dighediimnsional settings.
Within this platform, we designed challenging open-world task where the agent's 's sole objective is mine diamond. The environment procedurally generated for each episode, and sparse reward
14

---

Published as a conference paper at ICLR 2026
ShetePace
1.30
0.79
030
00
0.2
54
03
10
12
Environment
Let
Figure 13: Performance the Meta- World Shelf-Place task: With careful hyperparameter tuning, an agent that maximizes the sum of sparse task and reward machine rewards significantly outperforms the sparse reward agent.
only awarded upon successful collection of the diamond. This task implicitly requires long and complex sequence of actions: gathering wood, stone, iron and diamond in this order.
As shown in Figure (Right), baseline PPO agent fails learn meaningful policy and makes negligible progress on the task. In contrast, PPO augmented with our generated reward machine consistently learns the full sequence of behaviors required solve the task. This result demonstrates that our framework effectively scales to visually complex. procedurally generated 3D environments with extremely sparse rewards.
A.2.3 META-WORLD
evaluate our method on a subset of Meta-World, robotic manipulation benchmark originally created for evaluating multi-task and meta-RL algorithms. We adapted this benchmark to our setting by replacing the dense reward with a sparse reward signal, and we compare an agent that maximizes only the sparse reward signal to agent that maximizes the sum of the sparse reward and the reward from the reward machine (Figure 14p. We evaluated our method on the following tasks:
Assembly: The agent task is to pick a nut and place into a peg
Bin Pin-iic The agent's s task to pick puck from one bin and place it in another bin.
Pick Pick-Place: The task is to pick puck and place it in specific goal location.
The agent's 's task is to pick a puck and place it on a shelf.
Stick-Push: The agent's task is to grab stick and push a box using the stick.
The observation and action spaces share the same structure among the tasks. The observation vector consists of the robot's end-effector 3D coordinates, scalar value indicating whether the gripper is open or closed, and the position and orientation information of objects the environment. At each time step, the current observation is concatenated with the observation from the previous time step, along with the goal position, resulting a 39-dimensional vector. The action vector consists of three displacement values (da,, dy, and dz) of the end effector, with additional action for opening or closing the gripper.
The result of the main experiment is shown in Figure We also show in Figure 13 that, with more careful hyperparameter tuning, the agent augmented with the reward from the reward machine can solve the task with a high success rate. Moreover, the reward machine off-the-shelf intrinsic exploration rewards, such as RND (Figure 15). This results in overall better performance in most environments compared to the results in Figure g
be combined with
A.2.4 XLAND-MINIGRID
To evaluate our agent's generalization capabilities and its ability to adapt to novel situations, we the SLand-MiniGria benchmark. This suite of environments is specifically designed for meta-reinforeement learning research, combining the procedural diversity and depth of DeepMind's XLand with the minimalism and fast iteration of MiniGrid.
The entire framework implemented from the ground up in JAX, a design choice that enables massive parallelization and makes large-scale experimentation accessible limited hardware. Its
15

---

Published as a conference paper at ICLR 2026
(Sparse reward)
0.4
Assembly
Success Rate
0.04
Bin- Bin-Pickinng
success Rate
Flck-
Successs Rate
0.05
Shelf-Place
Success Rate
Stick Push
0.2
0.02-
0.0
0.05
0.00
0.5
1.0
0.5
1.0
Enviranment Stople)
Environment Stople?
Environ ment Stople
Environment Stapke?
Enviranment
Stepke?
Figure 14: Performance on the Meta-World on five tasks, method offers richer reward signal than sparse reward.
SAC-AND iSparse reward)
SAC RMD (ours)
Assembbyy
Successs Rate
SinsPicking
0.50
Pick-Place
5905 sore
Shelf-Place
Succerss Ratz
Stick-Push
Successs
0.2
025
05
0.0
0.0
00
100
0.0
0.5
1.0
6.0
0.5
1.
Envirenment Stepdo7
Environment
Stepse?
Stepso7
Environment
Step207
Enviranment Stepde?
Figure 15: Performance on the Meta-World: When combining the reward machine with the RND
exploration term, our method make of exploration bonuses, resulting in better overall performance.
core feature compositional system of rules (e.g., "keys open doors of the same color") and goals (e.g., "go to the blue box") that can be arbitrarily combined to procedurally generate vast and diverse distribution of distinct tasks. This allows for the creation of structured curricula and rigorous tests of an agent's is ability to infer the underlying rules of new environment and adapt its strategy accordingly.
In our experiments in Section 4 we leverage Manndd-iinGrid to assess how effectively our framework can adapt across this wide distribution of tasks. The primary challenge in this setting is not to master a single, static task, but to develop policy that can quickly recognize the objectives and constraints of newly sampled environment and formulate successful plan on the fly. This makes it powerful testbed for evaluating the adaptability and generalization of our approach.
Rules
NEAR
NEAR
Goals
NEAR
Figure 16: A sample task from Xl.nnd-Mini distribution, with the optimal solution path highlighted. The agent must infer that placing the blue pyramid near the purple square creates a red circle, which must then be moved the green goal. A distractor object (yellow circle) can render the task unsolvable. The agent is unaware of these rules, and object positions are randomized to test for adaptation.
Xlind-MMiirrii provides a formal language for procedurally generating tasks from combination of goals, rules, and initial object placements. This allows for the creation of a vast and diverse task space. The complete sets of supported goals and rules, adapted from the original XLand-MiniGrid paper, are detailed in Tables [ and 2 Figure [7 illustrates our framework's zero-shot generalization capabilities within this formal language, mirroring the experiment from Figure IO An agent trained on tasks A and B can successfully solve the novel composite Task C, demonstrating its ability to understand and execute policies based on the underlying formal structure of the environment.
16

---

Published as a conference paper at ICLR 2026
Table Supported goals in the XLand-MiniGrid formal language.
Goal
Meaning
ID
EmptyGoal
Placeholder goal, always returns False
0
Agenllol)GGollaa
Whether agent holds
(gcntOnT)l
Whether agent
tile
NgantNaar
Whether agent and a are on neighboring tiles
TileNearGoal(a, b)
Whether and b are on neighboring tiles
4
AgentOn Position
Whether agent on (x, y) position y) position
5
TileOmPositionnGaalaa x, y)
Whether
TileNearUpGioa(a, b)
Whether one tile above a Whether is one tile to the right Whether is one tile below a Whether is tile to the left of tile above agent
TileNeaRRighrGoalaa.. b)
TileNeaDDonnGoald b)
TileNearl Lele b)
10
A Agent
Whether
A genttNearRighhGnalla
Whether one tilc to the right of agent
A gentNcarDDownGGatta
Whether is one tilc below agent Whether one tile to the left of agent
13
(genNearLLefGGaa((a
14
Table 2: Supported rules in the XLand-MiniGria formal language.
Rule
Meaning
ID
EmptyRule
Placeholder rule, does not change anything
AgentHoldRule(a) +
If agent holds replaces with
AgemtNcarRule(a) +
If agent is neighboring tile with replaces it with
TileNcarRuleaa, b) c
If and b on neighboring tiles, replaces one with and removes the other
TileccaUUpRulcaa, +
b one tilc above a, replaces one with and removes the other
TilNNeaRiihhRuulea.. b)
If b is one tile to the right of a, replaces one with and removes the other
TilnNeatDownRule(a, b)
If b one tile below replaces
with and removes the other tile the left of replaces with and removes the other
TileNearLefiRule(a, b) c
If b
AgentNcarUpRuulc(a) +
If a one tile above agent, replaces with
AgentNcarRightRullcaa) +
If a one tile to the right of agent, replaces it with
AganteerrDownRulcaa) +
If a is one tile below agent, replaces it with c
10
AgentNearl.effRuledaa
If a is one tile to the left of agent, replaces it with
11
For the experiments in Section we evaluate performance on the first 1,000 tasks from the medium- Im benchmark in Xland-MiniGrid. The specific seeds visualized in Figure 9 are: 197 (1-task); 212, 197, 260 (3-task); 212, 197, 260, 859, 594 (5-task); and 212, 197, 260, 859, 594, 571, 602, 751, 660, 616, for the 10-task setting.
17

---

Published as a conference paper at ICLR 2026
Task A
Task
Task
GOAL: AgentNearRightGaal blue pyramid
GOAL:
GOAL
Aggnts-oldforange star)
AgentNeerRightCall pyramid
RULES:
AgentNecrt aftiull( star
Training
orange
AgentHoid]
RULES: AgentHold(
Evaluation
AgentHold(
purple
blue key - green star
- blue pyramid
blue key - blue pyremid
INIT TILES:
INIT TILES:
INIT TILES blue
yellow pyremic red goal.
Task
uo 'Pick up the olue key to begin the sequence."
u1 "positio yourself so that the biue pyremid is one tile to your right."
Figure 17: Demonstration zero-shot generalization. An agent is trained on a set of tasks (A, B). At evaluation, it given new reward machine for an unseen composite task (C). Because the sub-tasks in C (e.g., "Pick blue skills solve the novel task without any fine-tur tne-tuning.
key") are semantically familiar from training, the agent can reuse learned
18

---

Published as a conference paper at ICLR 2026
A.3 DQN TRAINING WITH LARMS
This section provides a detailed description of the reinforcement learning training procedure used in work. We adapt the standard Deep Q-Newwork (DQN) algorithm (Mnih al., 2013) to incorporate Laaguage-Alignee
Reward Machines (LARMs). The core idea is to augment the agent's state representation with the current state of the LARM and use the LARM to provide a dense, structured reward signal.
Algorithm T formalizes this process. The key modifications to the standard DQN algorithm are highlighted in blue. These changes include:
1. Augmented State Input: the environment state s, but also the language embedding of the current LARM state, o ((+)). This allows the agent to learn state- and task-dependent skills.
The policy, represented by the Q-network, takes as input not only
2. LARM State Transitions: After each environment step, the LARM is updated based on environment state $1+1 and the action a, taken. The labeling function L determines if relevant event occurred, which turn may cause a transition to a new LARM state U4+1-
The agent learns from composite reward signal that sum of the base environment reward R, and the reward from the LARM, RRM. This provides dense, incremental feedback for completing subtasks.
Combined Reward Signal:
4. Augmented Experience Replay: The transitions stored in the replay memory D are augmented to include the LARM states, i.e., (8t, ut. at, Rottal 88+1, We+1). This ensures the agent learns the Q-values over the joint state space.
By conditioning the policy semantic embeddings of LARM states, the agent effectively generalize across related subtasks, leading to improved sample efficiency and performance on complex, long-horizon tasks.
Algorithm DQN Training with Language-Aligned Reward Machines (LARMs)
1: Initialize: Replay memory 'D to capacity N.
2: Initialize: Q-network Q with random weights
3: Initialize: Target Q-network Q with weights 0- +
4: Initialize: Update frreuency C for the target network.
5: Input: LARM (U, Uj, 6, R, L) from Section 2.1
6: Input: State instruction embedding function (+))
7: for episode = to M do
8: Reset environment to get initial state 80-
9: Reset LARM to its initial state, +- uy.
for 0 T-1 do
11:
With probability select random action a;
Otherwise, select a, arg max, Q(st, Φ(u), a)
13:
Execute action in the environment, observe reward R. and next state 8
Get LARM event via labeling function: e ((8+11,0).
15:
Get next LARM state: Mt-1 5(u4, Et).
16:
Get LARM reward: 1RR R(u,, et).
17:
Compute total reward: Relctal = R₁ + RRM
18:
Store transition (8t, We, de, Riooal s111, u4+1) ) in D.
19:
Sample random minibatch of transitions (sj, Uj aj, Rioial 88+1: Uj+1) from D.
if episode terminates step j
20:
Set target yj = R + max ((5))), -') otherwise
21:
Perform gradient descent step on (y, Q plu;)(1 a(; Every C' steps, update the target network: 0 A.
23:
end for
24:
end for
19

---

Published as a conference paper at ICLR 2026
A.4 HUMAN-IN-THE-LOO LARM GENERATION
In Figure [ we show the self-improvement loop used to generate reward machines, where we instantiate both generator and critic foundation models iteratively refine the LARMs. key advantage of our LARM framework is that the interface to define and refine them is natural language, which allows human operators easily interpret and intervene the generation process. This section provides a transparent breakdown of the specific numan-in--the-oog efforts involved for each environment presented in this paper, summarized in Table To facilitate this, we implemented an interactive interface where a human operator could replace the critic foundation model during any round of self-improvement. In this mode, the generator model would receive the full history of LARM attempts and critic feedbacks, followed by a new refinement comment provided directly by the human. This design allowed us to seamlessly integrate both FM-generated and human- pronnddd feedback within the same improvement loop.
Table 3: Summary of Human-in-the-LLogg Effort for LARM Generation.
Environment MiniGrid-DoorKey (all sizes)
Human?
Description of Intervention
No intervention. The FM self-improvement loop was sufficient.
Human check confirmed correctness after 3 iterations.
MiniGrid-UnlockPickupp
Yes. The initial LARM missed edge case: the agent dropping key after pickup. human provided feedback add this transition (reflecting loss of progress). The FM incorporated this, and the task was solved.
MiilGrd-BlockcUUnlodPckkuup
No intervention. The FM self-improvement loop was sufficient.
Min@Gri--KyyCoriddS5333
Yes. The originally generated LARM too sparse. A human provided high-level advice to "define intermediate rewards" and suggested "crossing doors" "entering new rooms" as progress signals. The FM then generated a denser, effective LARM.
Craftium
X
No intervention. This was notable, as FM successfully leveraged its latent knowledge of Minecrafi-like game mechanics without guidance.
XLand-MiniGrid
X
No human intervention on any of the 1,000 generated LARMs. Conrectness was validated automatically using the LLM-as-judge method (as shown in Figure 11, left).
MetaWorld (all tasks)
Yes. The initial reward values in LARM were leading the agent to local minima, for example, grasping the object without moving it to the specified location. A human scaled the reward values for specific events in the LARM avoid the local minima, without changing the events themselves which assessed appropriate.
20

---

Published as a conference paper at ICLR 2026
A.5 THEORETICAL PROPERTIES OF GURM-GUIDED RL
In this section, we formalize the relationship between the original sparse-reward environment and the dense-reward objective created by the LARM. We show that under the conditions met by our generated LARMs, optimizing the LARM-augmented reward preserves the optimal policy of the original task.
Preliminaries. As defined in Section 2, let the environment be an MDP M = (S,A,P,)) The original task is defined by a sparse reward function (5) = Rgeal if s E S gual (a terminal state), and O otherwise. The LARM is A = (U, uo. F, 6, RLARM), where 8: U x - U is the transition function and RLARM is the LARM reward function. This induces the cross-prus pross-product MDP MLAAM. where the agent's &s reward is determined by ALA based transitions in A.
[Optimality Preservation] Assume the generated LARM A contains no positive reward cycles (i.e., for any cycle u, - the final reward Rgoot (obtained on transition to an accepting state u E F) is strictly greater than the cumulative reward of any non-terminal trajectory. Then, policy n² that is optimal for the cross-product MDP MLARM also optimal for the original sparse MDP M.
ui, the sum of rewards RIARM along the cycle is S O). Assume also that
Proof Sketch. The condition of positive reward cycles is key. It ensures that the value of any non-terminal looping trajectory is bounded and not preferable to progressing toward the goal. Any eycles in the LARM (e.g.. for losing progress) must have non non-positive cumulative reward, which prevents the agent from creating reward traps. Because the terminal reward Rgoal is set to be strictly dominant, the optimal policy for MMLAR will always maximize value by finding a path an accepting state F. The intermediate rewards from PLARM thus act as potential-base shaping to guide exploration, densifying the sparse signal without altering the set of optimal policies.
This proposition holds for the LARMs used in this paper (see Appendix A.9 For instance, the LARM for UnlockPico ckup contains cycles, such as losing key ((u1, lost_y, lost_y_key_ kee) u0). However, this transition has negative reward (-0. 1) that exactly cancels the positive reward from acquiring the key (+0.1). This "potential-basee
structure ensures no positive cycles are
created, satisfying the proposition's 's condition. The Doorkey LARM contains a similar zero-sum cycle. The Craft t um LARM is a Directed Acyclic Graph and thus trivially satisfies the condition. All other LARMs used in our experiments adhere to this property.
21

---

Published as a conference paper at ICLR 2026
A.6 COMPARISON WITH RELATED WORK
To further clarify our contributions, we provide a detailed comparison with prior work in Table The table is split into categories: (1) methods that use FMs synthesize or interact with automata and (2) general FM-guided RL frameworks. This comparison highlights that ARM-FM is unique in its ability directly generate complete, semantically grounded automaton from language without requiring expert demonstrations, and then use it to train a learning agent.
Table 4: Comparison of ARM-FM with FM-driven automata and FM-guided RL frameworks. Our method's advantages are highlighted in bold.
Methodl
Geneests RM?
Require Domoc?
Agrmt Type
Avampption
Primary FM Output / Rule
FMG for Automate Synthesis
LLLM Nanguee- Chaniate at 2005
Yes
Yes
No agents trained
Expert Demonstatios
Anowers memberstip queries for the 1." sigo- fill..
RAD Filmuuy the
No
RL (Leuzzed)
RMs are given SAT bused RM learning
Alsadar or
Yes
No
RL (Leuzzed)
FMs generate text leedback to a SAT based
algornti leam RMs
Yes
RL. (Learned)
Language Specification
FMs generate LARM + semuntiic embed- dings from language end-to-end.
PMM-Guided
Ytnetal
In-Cantest (CoT)
Text Interfice
SayCar
Yes (Skills)
Prestnined Skills
Pro-defined
pro-defined
Voyager
No
In- Conteat (Code)
Code Interface
Generates Python code for explotatio
(Mincraff-qpraiee
Panoka Sareni: Party
No
No
RI. (1 carmod))
Rewand Sez Code
Eadoo ohzookhencee progemmntic rownd funsiion
Mair Knoveor
2023
No
No
RL. (Letanal)
Tex Captions
Disiils FM-gencratal injocter prefesences
into rewadd model.
Macstoooar Kinsaro a. 2024 2014
No
No
RL (Leunned)
Manuall-deelted delined dills
Manually
Uses LLM feedback to design rowards foe peo. define skills
FLLM Du 20a3
No
(Iemrod))
FM query at eschhattie atrec
Suggeat plancilly wachul gouls leoud on the
sgent ' cunest stat.
22

---

Published as a conference paper at ICLR 2026
A.7 ADDITIONAL RESULTS
A.7.1 MINIGRID EXPLORATION BASELINES
For clarity, the main paper presents results against the best-performing exploration baseline from our evaluation, the Intrinsic Curiosity Module (ICM). this section, we provide a detailed comparison of the three intrinsic motivation methods we tested: ICM, Random Network Distillation (RND), and Disagreement. All baseline implementations are adapted from the well-tested RLeXplore library (uaan al. 2024).
Figure 18 shows the comparative performance of these methods on the DoorKey tasks. The results demonstrate that ICM consistently outperformed the other methods in our tested environments, justifying its selection as the primary exploration baseline for our main analysis.
DXR Dteegreament
DEN-EEM
DON-FIN
LLM Agent
Minitiaa Senntrrid-DDeonneeeetttto
Minitz Stti://rddennnaaaaaaeaheeee
Epicote Seclerd
0.6
00
Mirica ad--ooo (CMDP))
Min Grid--ooe bey-bxo-UU (LML3))
Manitrid Dww.naay--s----eeeee Sett------- (CMDV)
Kiniie ad Kea--DDnnKK--------e Max.oroe---i (CMOF)
Estodde Rooord
08-
50k.
100K.
2006
2001
200K
2006
30C
200K Envtroneean Stepp
150K
200K
200K
2006
50k
L00K
120K
2006
2006
2004
506
2034
200K
2008
200K
2006
Ensmmement Sheps
Emvironment Steps
Enstrnmment Stepe
Figure 18: Comparison of exploration baselines (ICM, RND, Disagreement) on the MiniGrid DoorKey tasks. ICM demonstrates the strongest and most consistent performance, establishing it as the most competitive exploration baseline for our experiments.
A.7.2 MINIGRID ANALYSIS OF LARM REWARDS
This section provides fine graiigraied analysis of how LARM-generatee rewards guide agent toward solving complex, sparse-reward tasks. The LARM effectively decomposes sparsely rewarded problem into sequence of sub-goals, providing dense, structured reward signal that serves learning curriculum.
Figure illlustrraee this process for the UnlockToUnlock task (see Appendix (xrenndixAA9Pipm for the full RM). The plot shows that during training, the agent first learns to make incremental progress by maximizing the LARM reward (blue curve), which is awarded for completing key sub-goals like collecting keys and opening doors. Once the agent has reliably learned to follow this reward curriculum completion (indicated by the dashed line), the final task success rate (orange curve), which corresponds to single sparse reward for reaching the goal, rises sharply. This demonstrates that the LARM successfully bridges the credit assignment gap, enabling the agent to solve task that would otherwise be intractable due to the sparse environment reward.
A.7.3 MINIGRID LONGER TRAINING
For completeness, we provide extended training results for the DoorKey experiments. Figure 20 shows the learning curves for the same agents when trained for 1M and 10M steps. These results confirm that performance saturates relatively carly. validating our decision to focus on the initial phase of learning in the main paper's analysis.
23

---

Published as a conference paper at ICLR 2026
Success rate
0.8
0.6
g
0.2
0.0
Figure 19: Analysis of LARM rewards during training in the UnlockToUnlock environment. The agent first learns maximize the structured reward provided by the LARM for completing sub-goals (blue curve). Once the sub-goal sequence is mastered (dashed line), the agent rapidly achieves high success rate on the sparsely-rewardee final objective (orange curve).
Mir1Gtt-DDonnee-5555550
MiniGnd-DDoctkee---bbb
Mrilnid-Doorley-1ee1-
1.0
Edtodde Reware
044
Doortey
Doonleyy-
Dow.teyy 16:16-00 (CMDF)
Epccde SeaMe
Envirorment Staps
ment
Environment Staps
Environmennt
(a) Training for IM environment steps.
DON--RM RM (Dun)
DQN
DQM+1CM
LLM Agent
MinsGrid-Tharrteyy.oo
Min Gri-----Doeeeey- 1oct-e 18c166-00
Ed0ode Seuard
0.6
MiniGrd-Dborreyy 55 ([MEP)
Minicii-Demmaapdssbee (CMDP)
Mintnnd-Das Bo ([OBP)
MivGrid-DDerr B8 (CMDP)
Eshoote Revirr
0.3
02
2003X 4000K G000K 2008 106
2008
42204
60.04
DOOCK
12M
2006e
4000
$200K
2000K 46008.
GOCOK BOOKK 11M
Pistroomett Steps
Suumement Strce
Feummment Steae
Fnvinrment Strpp.
(b) Training 10M environment steps.
Figure 20: Extended training for the DoorKey experiments shown in Figure 6 The plots show performance up to IM steps (a) and 10M steps (b). As agent performance plateaus early in training (around 300k steps), we present the shorter horizon the main paper for clarity.
24

---

Published as a conference paper at ICLR 2026
A.7.4 MINIGRID- VFMS ZERO-SHOL REWARD MODELS
To provide more comprehensive comparison, we evaluated the performance of Vision-Language Model (VLM) used directly as a zero-shot reward function, following the methodology proposed by Rocamonde et al. (2023). For this baseline, we employed CLIP (Radford al. 2021 to generate a dense reward signal. The reward at each timestep is calculated as the cosine similarity between the CLIPP embedding of the current visual observation (an image of the environment state) and the embedding of a target language description specifying the task goal. We implemented the goal-baseline regularization technique from the original work to stabilize training, using a negative description as the baseline.
The positive goal descriptions and the shared baseline description for each MiniGrid task were specified as follows:
"The agent (red triangle) has opened the door (color-outlined square) and reached the goal room (green square)."
"The agent (red triangle) has moved the ball (circle) away from the door (color-outlined square), has picked up the key, opened the door, and is now in the goal room (box square)."
MiniGrid-Unlock Tolnllooe "The agent (red triangle) has picked up both keys, opened both doors, and is now in the goal room (box square)."
Baseline (Negative Description): "The agent (red triangle) is far from the goal, has not picked up any key, and has not opened any door."
The results of this baseline are presented in Figure 21 The CLIP-based reward model failed to make any progress across all evaluated tasks. Consequently, we omitted these results from the main paper for clarity. We hypothesize that this failure stems from known limitations of current VFMs, particularly their challenges with spatial reasoning and their struggle to interpret visually abstract or out-of-distribuu failure modes are common when applying general-purpose VFMs to specialized domains that require nuanced visual understanding.
environments like MiniGrid. As noted by Rocamonde et al. (2023), such
DOK+AN
DOK-ICH
DQN-CCJP
MiniGrid-Dood Mir-orri--DoorRcy--55S--
Mink Siniarrs-DDonncy--dd---
02
Doortey (CMDP)
DoorKey vo (CNDP)
Doortey vo (CMDP)
MiniGrld Doorley (CMDP)
000
2034 300
J0CK
150K
2334
30K
300K
3306
2006
239C
3004
506
100K
180K
200K
250K
3004
106K
1596
2004
1558
300
Envirommnt Stops
Enviromment Staps
Enwromment Staps
Environment: Steps
Figure 21: Performance of an agent trained using CLIP embeddings as a direct reward signal on MiniGrid. The VLM-based reward fails to provide a sufficient learning signal for the agent make progress.
25

---

Published as a conference paper at ICLR 2026
A.8 PROMPTS
Below are the prompts for the generator and critic Foundation Models (FMs) for the DoorKey environment. This same prompt structure is used for all tasks, varying only the mission description for each environment and details on the specific environment API to generate the python labeling functions.
Prompt: Reward Machine Generator
Environment:
Agent: A colored triangle.
Key: Unlocks a door the same color.
Door: A color-outlined square within a wall.
Goal: A colored square in room (e.g., green).
Episode ends: Upon reaching the goal (+1 reward) reaching the step limit (0 reward).
Mission:
"This environment has a key that the agent must pick up order unlock door and then get the green goal square."
Your Role: Reward Machine Generator
Generate concise, correct, and compact reward machine in plaintext, wrapped in "plaintext" tags.
Your machine must:
Densify the reward signal to guide the agent effectively towards the goal.
2. Use Boolean--redicet events that are functions of the environment state. Do not use raw actions as events.
3. Maximize compactness with the fewest states and transitions possible, collapsing irrelevant events into per-state (state, else) state transition.
4. Adhere to the strict format provided below. Do not add comments extra text.
5. Use clear event names that are valid Python function names.
Action Set (for reference): turn_left, turn_right, move_forward, pickup, drop, toggle
BEWARD_MACHINE: MACHINE: STATES: u0, ul, INITIAL_STATE: u0 TRANSITION COANIITIONFUUCTION (u0, <event>) -> ul (u0, else)
REWARD REWARD_FUNCTIN FUNCTION: (u0, <event>, u1) -> X
Only list non-zero rewards in the REWARD_FUNCTION All other transitions assume reward 0. ().
Generate the reward machine now.
26

---

Published as a conference paper at ICLR 2026
Instructions:
Reward Machine Critic
Your Role:
Reward Machine Critic Evaluate candidate reward machine for the MiniGrid environment. Focus on correctness, compactness, completeness, and format.
0. Compactness
Ensure the fewest possible states and transitions are used.
All irrelevant or zero-reward events in state must be collapsed into a single (state, else) -> state transition.
1. Boolean-Predicate Events
Confirm that each transition's 's event is a Boolean predicate, not a raw action.
These predicates must reflect meaningful state conditions.
2. Coverage of Events
Every possible change in key predicates must either explicitly handled or aggregated under that state's e e transition.
Identify missing edge-case predicates
3. Dense Rewards + Penalties
Check for positive rewards transitions that signify progress.
Verify that penalties zero-rewards are used for regressions
Suggest additions for under-penalized failure modes.
Ensure reward magnitudes do not allow for reward hacking
4. Mission Logic
Ensure the sequence of states correctly enforces the logic required to solve the task.
Verify there are no unreachable states or unintentional loops.
5. Format Clarity
The submission must strictly follow the specified format:
REWARD MACHINE:
STATES: u0, u1,
INITIAL_STATE: CRANSTTIO_FUNCTIO (u0, <event>) ul (u0, else) -> u0
(u0, <event>, u1) -> x
Only non-zero rewards should be listed in the REWARD SOWWARD_FUNCTONNN FUNCTION.
There must be comments extra text within the plaintext block.
Event names must be descriptive Boolean predicates
Your Response Format:
Cite specific transitions or sections of the machine in your evaluation.
List concrete, actionable changes
Be concise and to the point, while not missing any important details,
End your response with one of the following two verdicts:
NO CHANGES NEEDED
CHANGES REQUIRED followed by bullet-list of the necessary fixes.
27

---

Published as a conference paper at ICLR 2026
Labeling Function Generator
Task:
Implement each event from the reward machine below Python boolean function. Each function must return True if the event condition holds in the current state (env), otherwise False.
Reward machine: (REWARD MACHINE)
Guidelines:
Function Naming and Signatures
Define one function per event in the RM.
Each function name must exactly match an event name.
* Each function should take only env as its argument.
Implementation Rules
Use only the environment attributes and methods below:
hww.grid..et.. (i, j) Access object at (i.
env.agent_pos Agent's position
env. env.agent_dir Agent's direction (0-3)
env. cav.cirry - Object agent is holding, e.g., a Key or None
env. env.width, env weight - Grid dimensions
Object Information
* WorldObj is the base class.
A Door has _ie_open and is _ockeed attributes.
Key has type == "key".
Goal has type "goal".
You cannot import classes. Instead. check object attributes (e.g..
bype == "key").
Output Rules
Only output clean, valid Python code.
No comments, explanations, extra output.
*
Do not define function for the else event.
*
Wrap your final output in triple backticks with python tag for formatting.
28

---

Published as a conference paper at ICLR 2026
Labeling Functions Critic
Your Role:
Event-Function Critic
You are evaluating Python functions that implement Boolean event predicates for given reward machine (RM). Your job is to verify that the logic is correct, complete, and aligned with the RM specification.
Task that the given RM should solve: "This environment has a key that the agent must pick up in order to unlock door and then get the green goal square."
Evaluation Criteria
1. Boolean Predicate Fidelity
Each function name must exactly match an event name from the RM.
Each unique event in the RM have a corresponding function.
The function must return True if and only if the corresponding predicate becomes true in the current environment state.
Coverage & Scope
Every event in the RM must have corresponding function.
There should be no extra functions that are not used in the RM.
3. Correct Use of env API The following attributes and methods are available from the env object.
env.grid.get (i, 5) env.agent_pos
# Access object at (i, j)
# (x, y) position of the agent
env.agent_dir
# Integer: direction the agent is facing # Object being carried (or None)
env.carrying
env. env.width, env height
# Dimensions of the grid
Object types must be checked by attribute, classes cannot be imported:
A Door has .is_open and .is locked attributes.
A Key has tyyp == "key".
A Goal has type == "goal".
Functions must inspect these properties determine predicate truth.
4. Clarity & Format
Each function must be standalone, containing only executable code.
Do not include comments, extra print statements, or surrounding explanations.
The code should be clean, idiomatic Python.
When You Respond
Point out any missing, misnamed, or extraneous functions.
Highlight any logic that incomplete, incorrect, or inefficient.
Suggest precise code fixes where needed.
End your review with one of the following two verdicts, exactly shown:
CHANGES NEEDED
or
CHANGES REQUIRED
[List of bullet-pointed issues and suggested fixes]
29

---

Published as a conference paper at ICLR 2026
A.9 REWARD MACHINES
DoorKey
REWARD MACHINE:
STATES: u0, ul, u2, u3
INITIAL_STATE: ub
TRANSITION FUNCTION: : ul
(u0, has_key)
(u0, else) -> u0
(ul, is_door_in_env_pee
u2
(ul, not has key) -> u0
(ul, else)
(u2, at_goal) -> u3
(u2, else) -> u2
(u3, else) -> u3 REWARD_FUNCTION
(u0, has key, has_key, ul) -> 0.2
(ul, is _e_door_n_eev_oppee _ u2) -> 0.3
(ul, not has key, u0) -> -0.2
(u2, at_goal, u3) -> 1.0
BlockedUnlookKiccup
REWARD_MACHINE
STATES: u0, ul, u2, u3, u4
INITIAL_STATE:
TRANSTTION_FNE FFNTTTOONFUNTTON:
(u0, has_ball) -> ul
(u0, else) -> u0
(ul, has_key) -> u2
(ul, else) -> ul
(u2, door_um _onloccd)) -> u3
(u2, no key)
ul
(u2, else) -> u2
(u3, has_box)
u4
(u3, else) -> u3
(u4, else) u4
REWARD FUNCTION:
(u0, has_ball, u1) -> 0.2
(u1, has_key, u2) -> 0.2
(u2, door_unlocked, u3) -> 0.2
(u2, no_key, ul) -> -0.3
(u3, has_box, u4) -> 1
30

---

Published as a conference paper at ICLR 2026
UnlockToUnloce
BEWARD_MACHING
STATES: u0, ul, u2, u3, ud, u5 ANITIAL_STATE: uo
TRANSITION_FUNCTOON (u0, got_y_key) ul
(u0, else) -> u0
(ul, door_y_opened, -> u2
(ul, lost_y_key) -> u0
(ul, else) ->
(u2, _______ey) -> u3
(u2, else) -> :2
(u3, door_r_opened) -> u4
(u3, lost_I key) -> u2
(u3, else) -> u3
(u4, entered leaa_golllloooooo -> u5
(u4, got_ball)
u5
(u4, else) >> u4
(u5, else) u5
(u0, got_y_key, ul) -> 0.1
(ul, door_y door_y_opened, u2) -> 0.2
(u1, lost_y_ken u0) -> -0.1
(u2, got_r_key, u3) -> 0.1
(u3, door_r_opened u4) -> 0.2
(u3, lost_r_key, u2) -> -0.1
(u4, entered_goall room, u5) -> 0.3
(u4, got_ball, u5) ->
KeyCorridor
REWARD_MACHINE
STATES: u0, ul, u2, u3, u4
INITIAL_STATER
TRANSITION CAANSITIONFUUCTIOON
FUNCTION:
(u0, on cn_purple_ddr (u0, else) -> u0
setee_doo___at_e not doo___ard_oot__ta__eeee key)
(ul, got_key) -> u2
(ul, else) -> ul
(u2, on wtopppuppddoor....___eeee aetoo_arn_has_eeee -> u3
(u2, opened red door) -> u4
(u2, else) -> u2
(u3, opened_red_ rod_roor)
u4
(u3, else) -> u3
(u4, else) u4 EEWARD_FUNCTION: (u0,
ul) 0.1
(u1, got got_key, u2) -> 0.2
(u2, on on_purple_door_ete door_ has key, u3) 0.25
(u2, opened_red_door, u4) >> 0.5
(u3, opened_red_door,, u4) -> 0.5
31

---

Published as a conference paper at ICLR 2026
Craftium
REWARD_MACHINE.
STATES: u0, ul, u2, u3 u0
TRANSITION FUNCTION:
CIPCTTIIOOOFUCIONN (u0, get_wood) -> ul
(u0, else) u0
(ul, get_stone) >> u2
(ul, else)
(u2, get_iron -> u3
(u2, else) ->
(u3, get diamond) -> u4
(u3, else) -> u3
REWARD_FUNCTION
(u0, get_wood, ul) -> 0.25
(u0, get_stone, ul) -> 0.5
(u0, get get_iron, u1) -> 0.75
(u0, get get_diamond, ul) -> 1.25
Metaworld
STATES: u0, ul, u2, u3, u4
INITIAL_STATE
TRANSITIO_FUNCTTON
(u0, near_object) -> vl
(u0, grasp_success) -> u2
(u0, else) -> u0
(ul, grasp_success] >> u2
(u1, not_mear_object) -> u0
(ul, else) -> ul
(u2, not_grasp_success] >> u0
(u2, object_near_goal)
u3
(u2, success) -> u4
(u2, else)
(u3, not not_object near_ -> u2
(u3, success) -> u4
(u3, else) -> u3
(u4, else) > u4 REWARD_FUNCTION (u0, near mear_object, u1) -> 0.20
(ul, grasp_success, u2) -> 0.40
(u0, grasp_success, u2) -> 0.40
(ul, not_near_object, u0) -0.20
(u2, not_grasp__uccess, u0) -> -0.40
(u2, object_near_gpal,, u3) 0.80
(u3, not_object_pear_goal. u2) ~> -0.80
(u2, success, u4) 1.50
(u3, success, u4) -> 1.50
32

---

Published as a conference paper at ICLR 2026
A.10 LABELING FUNCTIONS
Labeling Functions for DoorKey
def has key (env)
return cav.carrying is not None and env. carrying. "key"
def is_door_ ___door_in_ev__ppee (env)
for i in range (env. height):
for in range (env.wicth) obj gww.grid.get (j, i)
if obj is not None and cbj.type "door" and abj.
is_open: return True
return False
def not not_has has jot_has_ke((en))
return not (env. cenvccgryyigg is not None and env. anv.carrying.te. type -- key")
def at goal (env)
x, = env anwnttentta..... pos obj env cnw.grid.gg.. (x, y)
return obj is not None and type "goal"
Labeling Functions for PlockeuU
def has_ball (env):
return env. env.carrynn is not None and env, carrying. type "ball"
def has has_key (env):
return env. env.carrying is not None and env, www.carrying.ta.. type == "koy"
def door ueno
for range (env.width):
for j in range hannhheigh)): obj env. anv.grid.get (i, j)
if obj is not None and hasattr (obj, "is "is_eec and
obi.is locked False:
2
return True
return False
def no_key (env):
return not has key (env)
def has box (env)
return env. carrying is not None and env, anw.carrying.typ. type "box"
33

---

Published as a conference paper at ICLR 2026
Labeling Functions for UnlockToUnloer
def got_y_ kokyy_ (env) return {
eny.cirryng is not None
and (enat
"type", None] "key"
and getattr (env. (env. carrying, "coler", None) "yelloo cow
def door Seer___oppee opened
for in range width) for j range (env. height))))) obj = env. env.grid.get (i, j)
if
obj is not None
and getattr (ob), "type", Nonc) "door" and getattr (obj, "color", None) "yellow"
and getattr (obj, "is_open", False)
return True
return False
def (env
return not
env. env.carrying
None
and getattr (env. carrying, "type", None) "key" and getattr (env.carrying, "color", None) == "yellow"
def got_r_key (env)
return {
env.carrying is not None
and getattr (env. carrycarrinngg and getattr (env.carrying, "color", None] "red"
"type", None) -- "key"
def door r opened
for in range (env.width)
for j range (env height) obj = env. env.grid.ger (1, j) if
obj is not None and getattr (obj, "type", None) "door" and getattr (obj, "color", None) "red"
and getattr "is_open", False)
):
return True
False
def lost I k_y (env): return not env.carrying is not None
and getattr (env. caryccarryin,, and getattr (env. canvccarrying, "color", None) "red"
"type", Nono) "key"
def nettered_goal_ooe (env))
Example check: agent is the leftmost columns. return env.agent_pos [0] <
def got ball (env):
return
env. hww.carrying is not None
and getattr (env. carrying, "type", None) -- "ball"

---

Published as a conference paper at ICLR 2026
Labeling Functions for KeyCorridor
def on or__our_ppp
_at___o__h____ pot_hass_eey(enr)) (env)
2
i, = env, env.agent_pos
obj = env. env. www.grid.get (i, j)
if obj is not None and hasattr (cbi, 'type') and obj. type door' and getattr (obj, 'color', None) 'purple': if env.carrying is None or (hasattr (env. (env.carrying, 'type')
and env. serring.type !- " return True return False
def got_ key (env):
return {
env. cny.cingg is not None and hasattr (env. carrying, 'type')
and env.carrying type 'key'
def purple
and has key (env)
i, = env www.agent_par. pos
obj = env.grid. env.grid.get
j)
if obj is not None and hasattr (obj, 'type') and obj. -- door' and getattr (obj, 'color', None) 'purple';
if env.carrying is not None and hasattr (env.carrying, 'type ') and env. www.ccarriggtp... return True
"key':
return False
def opened_red_door (env):
i, j = env aaw.aagee__ps
obj = env.grid.gett (i, j)
if obj is not None and hasattr (ebj, 'type') and door' and getattr (obj, 'color', None) rrdd
--
if getattz 'is 'is_open', False) is True: return True
return False
def reached_ (env):
i, = cnv.agent_pp obj = env. www.grrd.get...... (i, j)
if obj is not None and hasattr (obj, 'type') and == goal':
return True
return False
35

---

Published as a conference paper at ICLR 2026
Start
uo
<near_obfect,O.2
U1
jgrasp_object,y
<success,1.5>
U2
u3
u4
<object <object_near_goall 0oar.
<success,1.5>
visualization of the Meta- World reward machine. Red arrows indicate a negative reward when the state transitions in the opposite direction, moving further away from the success state For example, if the agent in state 4 and transitions back state ug. will receive a reward of -0.2. When the agent does not trigger any transition event, the state remains the same, indicated by the self-loop arrows.
Figure 22:
A.10.1
VISUALIZATION OF META- WORLD REWARD MACHINE
Figure 22 shows visualization the Meta-Wor Mera-World reward machine generated by the FM. Red arrows indicate a reversed path in which the reward machine's 's state transitions further away from the success state.
A.11 HYPERPARAMETERS
A.11.1 DQN (MINIGRID & BABYÁI)
The hyperparameters listed in Table 5 were used for all DQN, DQN+RND, and DQN+RM agents in the MiniGrid and Al environments.
Table 5: DQN hyperparameters used for all MiniGrid and BabyAl experiments.
Hyperparameter
Value
Total Timesteps
1 x 107
Learning Rate
1 x 10-4 4
Replay Buffer Size
106
Learning Starts
80, 000
Batch Size
32
Discount Factor (7)
0.99
Target Network Update Frequency
2,500
Target Network Update Rate (r)
1.0
Train Frequency
Greedy Exploration
Initial Epsilon (Cstart)
1.0
Final Epsilon (Cend)
0.01
Exploration Fraction
0.35
Double Q-Learning
False
36

---

Published as a conference paper at ICLR 2026
Table 6: PPO hyperparameters used for the Craftium experiments.
Hyperparameter
Value
Total Timesteps
10?
Number of Parallel Environments
4
Steps per Environment (Rollout)
128
Number of Minibatches
4
PPO Update Epochs
4
Optimizer and Learning Rate
Learning Rate
5 10
Learning Rate Annealing
True
Max Gradient Norm
0.5
PPO GAE Parameters
Discount Factor (ry)
0.99
GAE Lambda (A)
0.95
Clipping Coefficient
0.1
Value Funetion Loss Clipping
True
Advantages Normalization
True
Loss Coefficients
Entropy Coefficient
0.01
Value Function Coefficient
0.5
A.11.2 PPO (CRAFTIUM)
For the more computationally demanding Craftium environment, we use PPO leverage vectorized rollouts for faster training. The hyperparameters for the PPO agent, which were kept consistent for both the baseline and our method, are detailed in Table
A.11.3 RAINBOW (XLAND-MINIGRID)
For the experiments in Xlannd-MinGrid,, consistent for both the baseline and our method, are detailed in Table 7
Rainbow DQN agent. The hyperparameters,
A.11.4 SAC (META- WORTA-
In Meta- World experiments we use SAC (Haarnoja et al. 2018) to train the sparse reward and the agent augmented with the reward machine. The hyperparameters are detalied in Table
37

---

Published as a conference paper at ICLR 2026
Table 7: Rainbow DQN hyperparameters used for the XLand-Mmiirid experiments.
Hyperparameter
Value
Base DQN Parameters
Total Timesteps
x 106
Learning Rate
6.25 10 10-5
Replay Buffer Size
1 106
Learning Starts
80,000 000
Batch Size
32
Discount Factor (?)
0.99
Target Network Update Frequency
5, 000
Train Frequency
4
Epsilon-Greedy Exploration
Initial Epsilon (Estort)
1.0
Final Epsilon (Eend)
0.05
Exploration Fraction
0.1
Rainbow Components
N-step Learning
3
PER Alpha (ox)
0.5
PER Initial Beta (3,)
0.4
Distributional Atoms
51
Distributional Value Range ( (Vmsn Vmaz)
10, 10]
Table 8: SAC hyperparameters used for Meta-Worke experiments.
Hyperparameter
Value
Base SAC Parameters
Total Timesteps
1.5 x 10?
Learning Rate
x 10
Replay Buffer Size
1 x 106
Learning Starts
000
Batch Size
512
Discount Factor (7)
0.99
Target Network Update coefficient
0.005
Policy Train Frequency
2
Critie Train Frequency
Exploration
Intrinsic Reward model
RND
Intrinsic Reward Coefficient
0.01
38


Reviews:
[{"review_id": "Rf7BIrVYJ6", "reviewer": "Reviewer_egA8", "summary": "This paper considers the problem of synthesizing reward machines from natural language. The aim then is to use the synthesized reward functions for single and possibly multi-task RL. This works in three parts. First a reward machine language specification is created. This describes the underlying finite state transducer structure of the reward machine, e.g. the states, transitions, alphabet. The second part describes the grounding functions for each input symbol. Finally each state is tagged with a natural language description describing the abstract state of the agent in it's policy. The abstract description is additionally encoded as a text embedding. A key contribution of the work is to propose using this text embedding as the representation of the automata state to the RL-agent. The insight is that this provides two features:\n\n1. If the state is sufficiently descriptive, if provides strong clues as to what the agent should do at that state, e.g., go collect the key for a door.\n2. If the state descriptions semantically (or exactly) align, then the policy can re-use experience from other tasks.\n\nEmpirical results show improvements against sparse feedback baselines and signs of multi-task generalization.", "questions": "Have you explored compositional tasks which aren't sequences, but have conditional logic? E.g., If x occurs, then you need to do y. My concern is the state embedding sharing becomes less useful since the local and global automata structure can non-trivially change when adding a task.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Formal representations of tasks such as reward machines offer a powerful mechanism to unambiguously/explicitly provide additional structure and memory to an RL agent. Being able to easily generate such representations is a worth while research program.\n1. The exact method used in this paper is to my knowledge novel. While the translation of natural language to an automaton has been explored in other works, this work adds an interesting contribution of a natural language embedding to each state.", "weaknesses": "1. The approach for constructing the automaton seems rather ad-hoc and relies somewhat blindly on the LLM's ability to provide a faithful translation. This is in contrast to techniques which seek to reduce the learning of the automaton structure to classic learning algorithms based on membership oracles [1].\n2. The language embeddings are the only source of automata structure embedding. it's unclear how reliably the automata's structure is actually relayed. While likely enough for short horizon tasks, the concern is that for complicated automata with many loops, explaining what the state is in natural language is non-trivial. On the one hand, compared to domain agnostic embeddings, e.g., RAD embeddings [2], these provide vital domain specific clues for what the policy should do. I would be curious what would happen when combining such approaches.\n3. Perhaps the largest issue is it's unclear how to test for alignment in the work aside from simply optimizing the reward and checking the outcome. The grounding function in particular seems non-trivial to verify without a large amount of human in the loop work.\n\n[1] L*LM: Learning Automata from Demonstrations, Examples, and Natural Language\n[2] Compositional automata embeddings for goal-conditioned reinforcement learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "spvbIcP4vN", "reviewer": "Reviewer_4WaP", "summary": "The paper introduces ARM-FM, which uses a large FM to automatically generate RMs from natural language task descriptions. The generated LARM includes state-level natural language descriptions and labeling functions. Policies are trained on the product MDP of the environment and RM, conditioned on the embedding of the current RM state instruction. Experiments on MiniGrid, Craftium (3D), and other benchmarks demonstrate improved zero-shot generalization.", "questions": "1) Precisely what is novel beyond FM-generated RM structure/code and conditioning on language embeddings of RM states? What core technical insight would not be obvious from prior RM + product-MDP and FM-guided RL work?\n2) Can you provide a comparison table clarifying how your pipeline differs from closest work that already uses LMs to synthesize automata/RMs/LTL specs, and from FM-guided RL?\n3) Is there a theoretical claim you can formalize?\n4) What fraction of tasks required human verification/edits? \n5) How are RM-state language embeddings obtained?\n6) How large are the domain gaps actually used (measured by a concrete metric)? Can you report performance as a function of controlled gap magnitude and type?\n7) Can you compare against stronger FM-guided baselines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ I believe the main strength of the paper is the use of language embeddings of RM state descriptions for conditioning the policy. \n+ Using FMs to automate RM construction from natural language specifications is useful since manual RM design requires expertise.\n+ The experiments and visualizations across multiple environments are commendable.", "weaknesses": "- My main concern is the limited novelty. RL over RMs and product MDPs is well established. The main addition here is automating RM generation with an FM and embedding the RM states, which is conceptually incremental in my view for ICLR.\n- Another concern I have is that relation to prior work is not sufficiently clear. The distinction needs to be well articulated. This includes a large body of work on FM-guided RL frameworks such as ReAct, SayCan, Voyager, Eureka etc and the list goes on. \n- Most tasks considered seem to involve small domain variation rather than truly different domains. It remains unclear whether the learned embeddings generalize under larger domain or modality gaps. So is the paper over claiming generalization? \n- There are missing comparisons in evaluation with recent vision language or FM-guided RL frameworks \n- The human verification weakens the claim of full automation.\n- Reproducibility may be an issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "anaehWvfl4", "reviewer": "Reviewer_3qmz", "summary": "The paper presents a novel methodology for tackling complex, long-horizon tasks in reinforcement learning (RL). At the core of the approach is the use of a foundational model (FM) to produce a high-level task decomposition, which is formalized as a Language-Aligned Reward Machine (LARM). In this framework, each state of the reward machine is annotated with a natural language description of the corresponding subgoal, enabling the agent to condition its policy on these instructions. This alignment facilitates goal-directed behavior and supports zero-shot generalization to previously unseen tasks.\n\nAs the agent transitions through the reward machine states, it receives structured intermediate rewards that guide learning and enhance sample efficiency. Notably, the FM also produces executable code to define the labeling function, which determines when a subtask has been completed. In doing so, the FM provides all essential components—task decomposition, subgoal descriptions, intermediate rewards, and labeling logic—allowing the agent to leverage LARM in novel environments with minimal human supervision.\n\nExperimental results across a diverse set of benchmarks demonstrate the effectiveness of the proposed approach, outperforming strong baselines such as RL with intrinsic motivation and LLM agents.", "questions": "1. Could you elaborate on the specific role and responsibilities of the human verifier during the experimental evaluation?\n2. How frequently was the LARM generated by the FM deemed unsatisfactory by the human verifier?\n3. What were the most common issues or failure modes observed in the LARM generation process by the FM?\n4. When an LARM was rejected by the human verifier, what steps were taken next? Was the prompt revised or refined, and if so, how was this process carried out?\n5. Does the labeling function rely solely on information observable by the agent, or does it access privileged information from the environment's internal state? If it uses privileged information, this could pose challenges for real-world deployment—particularly in sim-to-real scenarios—since such information may not be available outside the simulation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "I thoroughly enjoyed reading this paper. It presents a compelling case for the use of foundational models (FMs) and reward machines (RMs) as effective tools for communicating task structure to reinforcement learning (RL) agents. Particularly noteworthy is the FM’s ability to autonomously generate all necessary components—from the RM structure to the labeling function—based solely on a natural language task description and an image of the environment. The idea of associating natural language instructions with each RM state is especially interesting, as it enables zero-shot generalization and, depending on the quality of the subgoal descriptions, can even lead to near-optimal task execution despite the decomposition into potentially myopic subtasks.\n\nThe paper also offers a broad and thorough empirical evaluation, spanning environments from 2D grid worlds to complex 3D settings in robotics and Minecraft. Across these diverse domains, the proposed method consistently demonstrates strong performance. The authors further validate their approach by testing generalization capabilities and conducting a sensitivity analysis, highlighting the importance of both intermediate rewards and state embeddings to the agent’s success.\n\nFinally, I believe this work opens up several promising avenues for future research, some of which I have outlined in the weaknesses section of my review.", "weaknesses": "While I found the paper insightful and well-executed overall, there are some areas where further clarification or experimentation would strengthen its contributions. \n\nFirst, the role of human verification in Figure 3 is underexplored. The only mention of this “human-in-the-loop” process appears in the final paragraph of the conclusion. From what I understand, the human reviewer inspects the LARM generated by the FM and may reject it. However, it remains unclear what happens after a rejection: Does the human refine the prompt, or simply initiate additional rounds of self-improvement? Moreover, how frequently were LARMs rejected or refined during the experiments? What were the typical reasons for rejection? For instance, were there cases where bugs in the labeling function or poorly designed intermediate rewards prevented task completion? Clarifying whether such issues arose—and how they were handled—would help contextualize the robustness of the method. Alternatively, if no human verification was performed during experiments, that should be explicitly stated.\n\nSecond, in Section 3.1, the paper attributes the superior performance of DQN+RM primarily to the intermediate rewards provided by the reward machine. While this is a plausible explanation, I believe the state embeddings may also play a significant role. Since the environments are partially observable, the RM state effectively acts as a form of memory, allowing the agent to retain contextual information across time steps. In contrast, the baseline agents are memoryless, which may explain their underperformance—not necessarily due to the absence of intermediate rewards, but because they lack the capacity to maintain relevant state information. To better isolate the contribution of intermediate rewards, it would have been helpful to include the “No RM rewards” ablation from Section 3.4 in Figures 6 and 7. Additionally, comparing against an LSTM-based agent would help determine whether memory alone accounts for the observed performance gap. The same considerations apply to the experiments discussed in Section 3.2.\n\nThird, while the zero-shot generalization result is promising, it currently feels somewhat anecdotal. The paper presents a single instance in which the agent successfully generalizes to a novel task. A more systematic evaluation—spanning a broader range of tasks—would strengthen the case for the generalization capabilities of LARMs.\n\nFinally, a discussion on optimality would be valuable. The FM can describe RM states using either myopic or non-myopic natural language instructions. For example, a myopic instruction like “Get the key” may lead the agent to complete subtasks sequentially without considering long-term consequences, potentially resulting in suboptimal behavior (e.g., choosing a key that doesn’t lead to the door). In contrast, a non-myopic instruction such as “Get the key and then go to the door” could guide the agent toward more globally optimal behavior. It would be interesting to explore whether prompting the FM to produce non-myopic subgoal descriptions could improve long-term performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HcqvskvjFd", "reviewer": "Reviewer_tffC", "summary": "This paper presents a framework using foundation models (FMs) to automate Reward Machine (RM) generation from natural language specifications for reinforcement learning (RL). The core idea is to leverage FMs to produce not only the RM's automata structure but also the executable labeling functions to address sparse-reward, long-horizon embodied control tasks. The framework also proposes conditioning the RL policy on language embeddings of RM states to enable compositional generalization. The proposed method demonstrates its effectiveness by solving several challenging benchmark tasks (MiniGrid, Craftium, MetaWorld) intractable for standard RL baselines.", "questions": "1. How sensitive is the framework to errors or inefficiencies in the FM-generated labeling functions, especially for more complex environments? What is the typical failure rate, and how much manual correction was needed during experiments?\n\n2. What are the concrete next steps or necessary innovations required to extend ARM-FM to environments where only raw observations (e.g., pixels) are available, requiring the labeling function predicates themselves to be learned?\n\n3. While RMs offer structure, manually designing them becomes intractable for complex tasks. Does the FM-based generation approach scale effectively? Were there task complexities or types where the FM struggled significantly to produce a useful RM structure, even with the critic loop?\n\n4. Regarding the zero-shot generalization results (e.x., Figure 10), success relies on the new task's sub-goal embeddings being 'close to an embedding seen during training.' How can we quantify this 'closeness' in the embedding space? For example, does a specific cosine similarity threshold between a new sub-goal embedding and the closest training sub-goal embedding correlate with successful transfer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Effective FMs and RM pipeline with strong empirical results on sparse, long-horizon RL benchmarks (MiniGrid, Craftium, MetaWorld) where baselines (ICM, PPO, SAC) fail.\n2. By generating an explicit RM automaton, the framework offers a degree of interpretability and debuggability compared to opaque, end-to-end FM reward signals.\n3. Robust automated labeling code gen through LLM generator-critic loop for plan grounding.\n4. The use of language embeddings for the policy conditioning enables zero-shot generalization on a compositional task.", "weaknesses": "1. The grounding mechanism (using labeling functions) relies on access to the simulator's high-level, symbolic state representation (env.carrying, obj.type, obj.is_open). This might limit the framework's direct applicability to real-world scenarios or learning from raw pixels/sensory data, a common limitation but still noteworthy.\n\n2. While LLMs-driven RM generation and labeling functions shows novelty in automation, the strong performance on non-Markovian tasks feels incremental, as RMs' efficacy in such settings is well-established (e.g., [1, 2, 3]). Expected results limit surprise, though the proposed framework remains valuable for practitioners.\n\n[1] Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A McIlraith. Reward machines:\nExploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence\nResearch, 73:173–208, 2022.\n\n[2] Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. Using reward machines\nfor high-level task specification and decomposition in reinforcement learning. In International\nConference on Machine Learning, pp. 2107–2116. PMLR, 2018.\n\n[3] Andrew Li, Zizhao Chen, Toryn Klassen, Pashootan Vaezipoor, Rodrigo Toro Icarte, and Sheila\nMcIlraith. Reward machines for deep rl in noisy and uncertain environments. Advances in Neural\nInformation Processing Systems, 37:110341–110368, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)