MaestroMotif: Skill Design from Artificial Intelligence Feedback
Authors: Martin Klissarov
Mikael Henaff
Roberta Raileanu
Shagun Sodhani
Pascal Vincent
Amy Zhang
Pierre-Luc Bacon
Doina Precup
Marlos C. Machado
Pierluca D'Oro
Venue: ICLR 2025 Oral (2025)
Keywords: Hierarchical RL
Reinforcement Learning
LLMs
Primary area: reinforcement learning
TL;DR: A method for AI-assisted skill design via Motif and LLM code generation, solving tasks zero-shot from language descriptions on NetHack.

Abstract: Describing skills in natural language has the potential to provide an accessible way to inject human knowledge about decision-making into an AI system. We present MaestroMotif, a method for AI-assisted skill design, which yields high-performing and adaptable agents. MaestroMotif leverages the capabilities of Large Language Models (LLMs) to effectively create and reuse skills. It first uses an LLM's feedback to automatically design rewards corresponding to each skill, starting from their natural language description. Then, it employs an LLM's code generation abilities, together with reinforcement learning, for training the skills and combining them to implement complex behaviors specified in language. We evaluate MaestroMotif using a suite of complex tasks in the NetHack Learning Environment (NLE), demonstrating that it surpasses existing approaches in both performance and usability.

Published as a conference paper at ICLR 2025
MAESTROMOTIE: SKILL DESIGN FROM ARTIFICIAL INTELLIGENCE FEEDBACK
Martin Klissaroy Mikael Henaff Roberta Raileanu', Shagun Sodhani', Pascal Vincent". Amy Zhang*3 Pierre-Luc Bacon Doina Precup Marlos C. Machado 6,,,, 7,77,8 Pierluca 1, Mila, Meta, University of Texas Austin, Université de Montréal, McGill University, University of Alberta," Amii, Canada CIFAR AI Chair
ABSTRACT
Describing skills natural language has the potential to provide an accessible way to inject human knowledge about decision-makinn into an Al system. We present MaestroMotif, a method for Al-assisted skill design, which yields high- performing and adaptable agents. MaestroMotif leverages the capabilities of Large Language Models (L..Ms) to effectively create and reuse skills. It first uses an LLM's feedback to automatically design rewards corresponding to each skill, starting from their natural language description. Then, it employs an LLM's code generation abilities, together with reinforcement learning, for training the skills and combining them to implement complex behaviors specified in language. We evaluate MaestroMotif using suite of complex tasks in the NetHack Learning Environment (NLE), demonstrating that it surpasses existing approaches in both performance and usability.
INTRODUCTION
Bob wants to understand how to become versatile AI researcher. He asks his friend Alice, respected Al scientist, for advice. become a versatile Al researcher, she says, one needs to practice the following skills: creating mathematical derivations, writing effective code, running and monitoring experiments, writing scientific papers, and giving talks. Alice believes that, once these different skills are mastered, they can be easily combined following the needs of any research project.
Alice is framing her language description how to be versatile researcher as the description of set of skills. This often happens among people, since this type of description is convenient way exchange information on how to become proficient in a given domain. Alice could instead have suggested what piece of code or equation to write, or, at an even lower level of abstraction, which keys press; but she prefers it, because it would be inconvenient, tine-consuming and likely tied specific circumstances for be useful to Bob. Instead, describing important skills is easy but effective, transmitting large amounts of high- level information about domain without dealing with its lowest-leve intricacies. Understanding how to do the same with AI systems is still largely unsolved problem.
Navigation
do
Interaction
Composite
Zero-Shot
Task Training
MaestroMotit
Motif
LLM Policy
Embedding Sim.
Recent work has shown that systems based on Large Language Models (LLMs) combine sets of skills to achieve complex goals (Ahn et al. (2022) Wang et al.) 2024). This leverages the versatility of L.MMs solve tasks zero-shot, after the problem has been lifted from
Figure 1: Performance across NLE task categories. MaestroMotif largely outper- forms existing methods zero-shot, includ- ing the ones trained on each task.
Equal supervision. Correspondence martin issa @ma mcgg

---

Published as a conference paper at ICLR 2025
the low-level control space, which they have difficulty handling, to a high-level skill space grounded in language, to which they are naturally suited. However, humans cannot communicate skills to these systems naturally as Alice did with Bob. Instead, such systems typically require humans to solve, by themselves, the skill design problem, the one of crafting policies subsequently used by the L.M. Designing those skills typically entails very active involvement from human, including collecting skill-specific data, developing heuristics, or manually handling reward engineering (Ahn ct al. 2022). Thus, existing frameworks for designing low-level skills controlled by LLMs require technical knowledge and significant amounts of labor from specialized humans. This effectively reduces their applicability and generality.
In this paper, we introduce the paradigm of Al-Assisted Skill Design. In this paradigm, skills are created in process of human-Al collaboration, in which human provides natural language description of the skills and an Al assistant automatically converts those descriptions into usable low-level policies. This strategy fully leverages the advantages of both human-base based and Al-based skill design workflows: it allows humans to inject important prior knowledge about task, which may enhance safety and performance for the resulting agents even in the absence of optimal Al assistants; at the same time, it automates the lowest-level and more time-consuming aspects of skill design.
Based on this paradigm, we propose MacstroMotiif, a method that uses LLMs and reinforcement learning (RL) to build and combine skills for agent to behave as specified in natural language. MaestroMotif uses an I.M'' feedback to convert high-level descriptions into skill-specific reward functions, via the recently-proposed Motif approach (Klissarov et al., 2024). It then crafts the skills by writing Python code using an LLM: first, it generates functions for the initiation and termination of each skill; then, it codes policy over skills which is used to combine them. During RL training, the policy of each skill is optimized to maximize its corresponding reward function by interacting with the environment. At deployment time, MaestroMotif further leverages code generation via an LLM to create a policy over skills that can combine them almost instantancously to produce behavior, in zero-shot fashion, as prescribed by a human in natural language.
MaestroMotti thus takes advantage of RL from Al feedback to lift the problem of producing policies from low-level action spaces to high-level skill spaces, a significantly more automated way than previous work. In the skill space, planning becomes much easier, to the point of being easily handled zero-shot by an LLM that generates code policies. These policies can use features of a programming language to express sophisticated behaviors that could be hard to learn using neural networks. In essence, MaestroMotii crafts and combines skills, similarly to motifs in a composition, to solve complex tasks.
We evaluate MaestroMotti on a suite of tasks in the Nethack Learning Environment (NLE) (Küttler eet al. 2020), created to test the ability to solve complex tasks in the early phase of the game. We show that MaestroMottif is a powerful and usable system: it can, without any further training, succeed in complex navigation, interaction and composite tasks, where even approaches trained for these tasks struggle. We demonstrate that these behaviors cannot be achieved by baselines that maximize the game score, and we perform an empirical investigation of different components our method.
2 BACKGROUND
A language-conditionee Markov Decision Process (MDP) (Liu et al. 2022p is a tuple M = (8,A, r.,, 7: 1), where S is the state space, A is the action space, G is the space of natural language task specifications. r S x g + R is the reward function, p S x A + A(S) is the transition function, y E (0, 1] is the discount factor, μ E A(S) the initaal state distribution.
A skill can be formalized through the concept of option (Sutton et al. 1999 deterministic Markovian option w E J2 is triple (In, Tw. (w)), where Ie
Precup 2000). A
1) is the initiation function, determining whether the option can be initiated or not, To : S - (AA) is the intra-option policy, and 8_, : S + {0, 1} is the termination function, determining whether the option should terminate or not. Under this mathematical framework, the skill design problem is equivalent to constructing a set of options l that can be used by an agent. The goal of the agent is to provide a policy over options π: G x S + Ω. Whenever the termination condition of option is reached, selects the next option to be executed, conditioned on the current state. The performance of such a policy is defined by its expected return J(T)) E D rr))..)
2

---

Published as a conference paper at ICLR 2025
D
Dwi
P1
Dataset
Agent Dasigner
LLM Coder
Tui
Reward
Inf./Tarm. functionss
training
Generation of Skills initiation Intlistion/Terminnn Termination
Annotator
Ph
Per-skil Annetated Dataset
Per-skil
T
Reward Function
Agent Designer
Training Policy
Agent Designer
LLM Coder
Automated Skills Reward Design
Generation Training Policy Over Skills
Tuc.. Iriliation
(Tug Muag
Far Termination Function
Available Skills
Reinforcement learning maximize Tip.
7V
Policy Over Skils
Active
Environment
Skill training via Reinforcement Learning
Al-assisted Skill Design with MaestroMMtti.. 1. An agent designer provides skills de- scriptions, which get converted to reward functions Teg by training on the preferences of an LLM dataset of interactions. 2. The agent designer describes initiation and termination functions, Iwi [w.......) and pw....... to the LLM, which instantiates them by generating code. 3. The agent designer describes a train-time policy over skills тт which the LLM generates via coding. 4. Each skill policy Tw, is trained to maximize its corresponding reward Typ,- Whenever skill terminates (see open/closed circuit), the policy over skills chooses new one from the of available skills.
Figure 2:
In the Al-Assisted Skill Design paradigm, an agent designer provides a set of natural language prompts {#1 #2. ..... Vy }. Each prompt consists of a high-level description of a skill. An AI X' - Ω to convert each prompt into an option. Note
system should implement a transformation that the ideas and method presented this paper generalize to the partially-observable setting and, in our experiments, we learn memory conditione policies.
3 METHOD
MaestroMotif leverages Al-assisted skill design perform zero-shot control, guided by natural language prompts. To the best of our knowledge, it is the first method that, while only using language specifications and unannotated data, able to solve end-to-end complex tasks specified in language. Indeed, RI. methods trained from scratch cannot typically handle tasks specified in language Touati al. 2023), while I.I.- -aaed methods typically feature labor-intensie methodologies for learning low-level control components (Ahn al. 2022 Wang al.) 2024). MaestroMotif combines the capability of RL from an LLM's feedback to train skills with an LLM's code generation ability which allows compose them at will. We first introduce MacstroMotif as general method, describing its use for Al-assisted skill design and zero-shot control, then discussing its implementation.
3.1 AI-ASSISTED SKILL DESIGN WITH MAESTROMOTIF
MaestroMotif performs Al-assisted skill design in four phases shown Figure 2 It leverages L.MSs in two ways: first to generate preferences, then to generate code for initiation/termiatioonn functions and for a training-time policy over skills. It then uses these components to train skills via RL.
Automated Skills Reward Design In the first phase, agent designer provides description for each skill, based on their domain knowledge. Then, MaestroMotti employs Motif (Kiissrrov et al. 2024) to create reward functions specifying desired behaviors for each skill: it elicits preferences of an LLM on pairs of interactions sampled from a dataset D, forming for each skill a dataset of skill-related preferences Da,, and distilling those preferences into skill-specific reward function Pe
3

---

Published as a conference paper at ICLR 2025
"Vou we will cocl whinn defiess the methed "seeect skd" Alethtack Player that
videagerne Natrlook, [-]
dollawing betweon mice levels S...""
LLM Coder
Stule
Action
π
Environment
Skill naural network
Policy Over Skills
Figure 3: Generation of policy over skills during deployment. The LLM takes a task description and template input, and implements the code for the policy skills skill selection function. Running the code yields a policy over skills that commands skill neural network by sending the appropriate skill index. Initiation and termination functions, determining which skills can be activated and when skill execution should terminate, are omitted from the diagram.
by minimizing the negative log-likelihoo,, i.e., using the Bradley-Tery model:
C(px) = -E 9)) 1yy lo (s s2] 1 2] log Ppl + srs
(I)
where is an annotation generated by an LLM annotator and P Palsa > st] e" (ta) +e"v(e)) is the
estimated probability of preferring state to another (Bradley & Terry 1952).
Generation of Skill Antiiaiion/erminattinn a skill when it is active, it does not prescribe when the skill can be activated or when it should
While a reward function can steer the behavior of
be terminated. In the options framework, this information is provided by the skill initiation and termination functions. MotitroMotii uses an LLM to transform a high-level specification into code that defines the initiation function Iw; and termination function Bw, for each skill.
Generation of training-time policy over skills To be able to train skills, MaestroMotif needs a way to decide which skill to activate at which moment. While skills could be trained in isolation, having an appropriate policy allows one to learn skills using a state distribution closer to what will be needed during deployment, and to avoid redundancies. For instance, suppose the agent designer decided to have a two-skill decomposition, such that skill A's goal can only be achieved after skill B's goal is achieved; if they not trained together, skill A would need to learn to achieve also the goal of skill B, nullifying any benefits from the decomposition. To avoid this, MaestroMotii leverages the domain knowledge of agent designer, which gives language specification of how interleave skills for them to be learned more easily. From this specification, MacstroMotif crafts policy over skills to be used at training time, .T, which, as with the previous phase, is generated code by an LLM.
Skills training via RL In the last phase of Al-assisted skill design, the elements generated in the previous phases are combined to train the skill policies via RL. Following the call-and-return paradigm (Sutton et al. 1999), the training policy decides which skill to execute among the ones deemed as available by initiation functions Iuja.....) Then, the skill policy Tw; of the selected skill gets executed in the environment and trained to maximize its corresponding reward function Tep until termination function B_, deactivates Initialized randomly at the beginning of the process, each skill policy will end up approximating the behaviors originally specified in natural language.
3.2 ZERO-SHOT CONTROL WITH MAESTROMOTIE
After Al-assisted skill design, MaestroMotif has generated a set of skills, available to be combined. During deployment, user specify task in natural language; MaestroMotif processes this
4

---

Published as a conference paper at ICLR 2025
language specification with a code- gedeeeeeerttingg LLM produce and run policy over skills that, without any additional training, can perform the particular task.
The policy over skills is then used, together with the skill policies fu initiation functions Iu........ and termination functions S fwwy......" built through Al-assisted skill design, to compose the skills and implement the behavior specified by the user. This process follows the same call-and-return strategy, and recomposes the skills without any further training. It is illustrated in Figure 3 which shows concrete examples of prompts and outputs. More examples are reported in appendix.
3.3 MAESTROMOTIF ON NETHACK
We benchmark MaestroMotif the NetHack Learning Environment (NLE) (Küttler et al., 2020). In addition to being used in previous work on Al feedback, NetHack is a prime domain to study hierarchical methods, due the fact that it is long-horizon and complex open-ended system, containing a rich diversity of situations and entities, and requiring vast array of strategies which need to be combined for success. To instantiate our method, we mostly follow the setup of Motif (Klissarov et al.) 2024), with some improvements and extensions. We now describe the main choices for instantiating MaestroMotii on NetHack, reporting additional details in Appendix[A]
Skills definition Playing the role of agent designers, we choose and describe the following skills: the Discoverer, the Descender, the Ascender, the Merchant and the Worsh i pper. The Discoverer is tasked to explore each dungeon level, collect items and survive any encounters. The Descender and Ascender are tasked to explore and specifically find staircases to either go up, or down, a dungeon level. The Merchant and the Worshippez are instructed to find specific entities in NetHack and interact with them depending on the context. These entities are shopkeepers for the Merchant, such that it attempts complete transactions, and altars for the Worshipper iippp, where it may identify whether items are cursed or not. The motivation behind some of these skills (for example the Descender and Ascender pair) be traced back classic concepts such as bottleneck options (Iba. 1989 McGovern & Barto 2001 Stolle & Precup 2002).
Datasets and LLM choice To generate a dataset of preferences Dw,, for each one of the skills, we mostly reproduce the protocol of Klissarov et al.] (2024). and independently annotate pairs of observations collected by a Motif baseline. Additionally, we use the Dungeons and Data dataset of unannotated human gameplays (Hambro et al. (0022b). We use Llama 3.1 70B (Dubey et al. 2024) via vLLM (Kwon 2023) as the LLM annotator, prompting with the same basic mechanism employed in Klissarov et al. (2024).
Annotation process In the instantiation of Motif presented in Klissarov et al. 2024). preferences are elicited from an L.M by considering a single piece of information provided by NetHack, the me messages. Although this was successful in deriving an intrinsic reward that was generally helpful to play NetHack, our initial experiments revealed that this information alone does not provide enough context to obtain a set of rewards that encode more specific preferences for each skill. For this reason, additionally include some of the player's statistics (i.e., dungeon level and experience level), as contained in the observations, when querying the LLM. Moreover, we leverage the idea proposed by Piterbarg al. (2023a) of taking the difference between the current state and state previously seen the trajectory, providing the difference between states 100 time steps apart as the representation the LLM. This provides compressed history (i.e. non-Markovian representation) to LLM and reward functions, while preventing excessively long contexts.
Coding environment and Policy Over Skills A fundamental component of MaestroMotif is an LLM coder that generates Python code (Van Rossum & Drake Jr. 1995). MaestroMotif uses Llama 3.1 405b to generate code that is executed in the Python interpreter yield initiation and termination functions for the skills, the train-time policy over skills, and the policies over skills employed during deployment. practice, we find it beneficial to rely on an additional in-context code refinement procedure to generate the policies over skills. This procedure uses the LLM to write and run unit tests and verify their results to improve the code defining a policy over skill (see Appendix A.2 for more details). In our implementation, policy over skills defines function that returns the index of the skill to be selected. For the training policy, the prompt given to the LLM consists of the list of skills and a high-level description of an exploratory behavior of the type "alternate between the Ascender and the Descender; you see a shopkeeper activate the Merchant... ", effectively transforming minimal domain knowledge low-level information about skill's desired state distributions.
5

---

Published as a conference paper at ICLR 2025
RL algorithm and skill architecture To train the individual skills, we leverage the standard CDGPTS baseline based on PPO (Schulman et al. 2017)) using the asynchronous implementation of Sample Factory (Petrenko et al. 2020). Instead of using a separate neural network for each skill, we train a single network, with the standard architecture implemented by Miffyli (2022), and an additional conditioning from one-hot vector representing the skill currently being executed. This enables skills to have a shared representation of the environment, while at the same time reducing potential negative effects from a multi-head architecture (see Section 4.3)
EXPERIMENTS
We perform a detailed evaluation of the abilities of MaestroMotti on the NLE and compare its per- formance a variety of base- lines. Unlike most existing meth- ods for the NLE. MaestroMotif is a zero-shot method, which pro- duces policies entirely through skill recomposition, without any additional training. We empha- size this in our evaluation, by first comparing MaestroMotif to other methods for behavior spec- ification from language on a suite of hard and composite tasks. Then, we compare the resulting agents with the ones trained for score maximization, and further analyze our method. We report all details related to the experi- mental setting in Appendix [A.5 All results are averaged across nine seeds (for MacstroMotif, three repetitions for skill training and three repetitions for software policy generation), with error bars representing the standard error. All MacstroMotii results obtained by recombining the skills without training, and the skills themselves were learned only through LLM feedback, without access to other types of reward signals.
Overview of the early game of NetHack
Dungeons of Doom
Gnomish Mines
Shopkeeper
Altar
Minetown
Delphi
Figure 4: Simplified depiction of the early NetHack game where
significant areas (such as branches) and entities are labeled.
Evaluation suite As NetHack is complex open-ended environment (Hughes et al. 2024). allows for virtually limitless possibilities in terms of task definition and behavior specification. To capture this complexity and evaluate zero-shot control capabilities beyond what has been done in previous work, define comprehensive benchmark. We consider set of relevant, compelling, and complex tasks related to the early part of the game, deeply grounded in both the original NLE paper (Küttler al. 2020) and the broader NetHack community (Mouli 2022). Our benchmark includes three types of tasks: navigation tasks, asking an agent to reach specific locations in the game; interaction tasks, asking an agent to interact with specific entities in the game; composite tasks, asking the agent to reach sequences of goals related to its location in the game and game status. In navigation and composite tasks, we evaluate methods according to their success rate; interaction tasks, we evaluate methods according to the number of collected objects. Figure 4 presents an overall depiction of navigation and interaction tasks, and Appendix A.6 provides providdd more details. Our evaluation is also inspired by SkillHack (Matthews al. 2022) a benchmark for evaluating skill transfer NetHack.
4.1 PERFORMANCE EVALUATION
Baselines We measure the performance of MaestroMotif on the evaluation suite described above. For MaestroMotif to generate a policy, it is sufficient for a user to specify a task description in natural language. For this reason, language: first, to using Llama as a policy via ReAct (Yao et al.) (022), which is an alternative zero-shot method; second, methods that require task-specific training via RL, with reward functions
mainly compare MacstroMotif to methods that instructable via

---

Published as a conference paper at ICLR 2025
Zero-shot
Task-specific training
Reward Information
Task
MacstroMotif
LLM Policy
Motif
Emh. Simil.
RL w/ task reward score
Gnomish Mines
46% + 1.70%
0.1% = 0.03%
9% + 2.30%
3% = 0.10%
3.20% + 0.27%
Delphi
29% + 1 1.20%
0% +: 0.00%
2% 1: 0.70%
0% + 0.00%
0.00% 1 0.00%
Minetown
7.2% 1 0.50%
0% i 0.00%
0% 1 0.00%
0% 1 0.00%
0.00% 1 0.00%
Transactions
0.66 0.01
0.00 0.00
0.08 0.00
0.00 + 0.00
0.01% 0.00%
Price Identified
0.47 0.01
0.00 0.00
0.02 + 0.00
0.00 = 0,00
0.00% 0.00%
BUC Identified
1.60 1 0.01
0.00 0.00
0.05 +: 0.00
0.00 i 0.00
0.00% 1 0.00%
Table 1: Results on navigation tasks and interaction tasks. MaestroMotif and LLM policy are zero-shot methods requiring no data collection or training on specific tasks; task-specific training methods generate rewards from text specifications (based on AI feedback or embedding similarity) and train an agent with RL; the last column reports the performance of a PPO agent using privileged reward information. a combination of the task reward and the game score (not accessible to the other methods). MaestroMotif largely outperforms all baselines, which struggle with complex tasks.
generated by using either AI feedback or cosine similarity according to the embedding provided by a pretrained text encoder (Fan et al.] 2022). In addition, we also compare to an agent trained to maximize a combination of the task reward and the game score (as auxiliary objective), which has thus access to privileged reward information compared to the other approaches. For all non-zero-shot methods, training runs of several GPU-days are required for each task before obtaining a policy.
Results on navigation and interaction tasks Table T shows that MaestroMotii outperforms all the baselines, which struggle to achieve good performance, in navigation and interaction tasks. Notice that this happens despite the disadvantage to which MaestroMotii is subject when compared to methods that are specifically trained for each task. The poor performance of the LLM Policy confirms the trend observed by previous work (Klissarov et al. 2024]: even if the LLM has enough knowledge and processing abilities to give sensible Al feedback, that does not mean that it can directly deal with low-level control and produce a sensible policy via just prompting. At the same time, methods that automatically construct single reward function that captures language specification break apart for complex tasks, resulting in difficult learning problem for an agent trained with RL. MaestroMotii,, instead, still leverages the ability of LLMs to automatically design reward functions, but uses code to decompose complex behaviors into sub- sub-behaviors individually learnable via RL.
Results on composite tasks A feature of language its compositionality. Since, in the type of system we consider, a user specifies tasks in language, different behavior specifications can be composed. For instance, an agent can be asked to first achieve a goal, then another one, then a last one. The composite category in our benchmark captures this type of task specifications. In Table 2 we compare MaestroMotti to other promptable baselines, showing the task description provided to the methods and their success rate. MaestroMotif has lifted the problem of solving a task to the
COT.OIETOIIOOOOOOO Discovery Hunger
Golden Exit
Level Up & Sell
"Alternate between the first three levels of the Dungeons of Doom least once)
"Do leave the first dungeon level until you achieve XP level then find shopkeeper and sell an item that you have collected: finally survive for another 300 steps."
"Reach the oracle level (the Delphi) in the Dungeons of Doom, but before discovering the Gnomish Mines and eating some food there after getting hungry."
Tasks
until you collect minimum of 20 gold pieces and defeat monsters: finally try to quit NetHack"
Methods
MaestroMotii
24.80 % 1 1.18 % 0% 0.00%
7.09% 0.99%
7.91% + 1.47%
LLM Policy
0% 1: 0.00%
0% 0.00%
Motif' Embedding Similarity
0% 0.00%
0% 0.00%
0%% 0.00%
0% 0.00%
0% 0.00%
0% 0.00%
Table Description of the composite tasks and success rate of MaestroMotif and baselines. Using a code policy allows MaestroMotii to compose skills by applying sophisticated logic, requiring memory or reasoning over a higher-level time abstraction. This is impossible to achieve for a zero-shot LLM policy, and hard to learn via single reward function, which explains the failures of the baselines.
7

---

Published as a conference paper at ICLR 2025
1.40
1.69
Completed Pransactions
Identification
B/U/C Identification
0.09
Gnomish Mises
Delgäi
MaestroMoetf
Motr
Hlerrcctial BC 290
BC PPO
P20
Figure 5: Performance of MacstroMotif and score- score-maxine maximiziixiin baselines in interaction tasks (first row) and navigation tasks (second row). Despite collecting significant amounts of score, score-mase more-maxiiizing approaches only rarely exhibit any interesting behavior possible in our benchmarking suite.
one of generating a code policy: thus, even if the tasks entail extremely long-term dependencies, simple policies handling only few variables can often solve them. In contrast, defining rewards that both specify complex tasks and are easily optimizable by RL is extremely hard for existing methods, because exploration and credit assignment in such a complex task become insurmountable challenges for a single low-level policy. To the best of our knowledge, MacstroMotii is the first approach to be competitive
recision-makinng tasks of this level of complexity, while simultaneously learning to interact through the lowest-level action space. reporrs example of complex behavior exhibited by an agent created by MaestroMotif while solving one of the tasks. The overall results, aggregated over navigation, interaction and composite tasks, are presented in Figure [
Minimap
Mininap
Minimap
agent
leading found.
fights Gnomes
collects items
Theg
the
Minimap
Minimap
Minimap
After feeling hungry, the agent ents
It ascends back the Dungeons of Doom.
Finally, the deseends the Delphi.
Figure 6: Illustration of MaestroMotti on the composite task Hunger Discovery. We show screenshots from the game as well as an accompanying Minimap, where the agent's 's position shown as red dot To complete the task, the agent needs to find the entrance to the Gnomish Mines, which is a staircase randomly generated anywhere between the levels to 4 in the main branch, the Dungeons of Doom. After exploring the first few levels, the agent finally finds the hidden entrance and descends into the Mines, where it fights monsters and collects items to help it survive. After a few hundred turns, the agent's hunger level increases hungry, prompting it to eat comestible item. Finally, it has to ascend back into the main branch, before beginning the perilous journey down the Delphi, which appears anywhere, randomly, between the level 5 in the Dungeons of Doom.
4.2 COMPARISON TO SCORE MAXIMIZATION
The vast majority of previous work on the NetHack Learning Environment has focused agents trained to maximize the score of the game (Sypetkowski & Sypetkowski] 202TT: Piterbarg et al. 2023b
Wolezyk et al. 2024). Although the score might seem like potentially rich evaluation signal,
8

---

Published as a conference paper at ICLR 2025
0.4
Ablado
Mtalaaa lesii
Necrualion SU
0.2
Mortalted
0.0
Global Stop
Global Stop
(a) Effect of choices in hierarchical architecture and learning strategy.
(b) Skill reward learning curves
(c) Skill reward learning curves
(skills learned in isolation).
(skills learned simultaneously).
Figure 8: (a) Coal-conditioning as an architecture for skill selection and synchronous alternation of the skills using an exploration policy is essential for obtaining good performance. (b) When learning skills asynchronously (alternating them in different episodes), some important skills do not manage to be learned. (c) Learning skills synchronously automatically induces emergent skill curriculum, in which basic skills are learned before the most complex ones.
has been observed by previous work that a high-performing agent terms its score does not necessarily exhibit complex behaviors in the game (Hambro al., 2022a To illustrate this fact in the context of our work, we compare MaestroMotif's performance in the navigation and interaction tasks to the one achieved by agents trained to maximize the in tn-game score via different methods.
Figure D reports the performance of these methods, showing that, even if maximizing the score might seem good objective in NetHack, it does not align to the NetHack community's preferences, when the source of training signal is an expert, such as in the behavioral cloning case.
4.3 ALGORITHM ANALYSIS
Having demonstrated the performance and adaptability of MacstroMotif, now investigate the im- pact of different choices on its normalized performance across task categories. Additional experiments can be found in Appendix A.8
Scaling behavior Central to the approach behind MaestroMotif is an LLM producing policy over skills in code, re-composing set of skills for zero-shot adaptation. It is known that the code generation abilities of an LLM depend on its scale (Dubey et al. 2024): therefore, one should expect that the quality of the policy over skills generated by the I.LM coder will be highly dependent on the scale of the underlying model. We verify this in Figure 7 showing clear trend of performance improvement for large models. In Appendix A.4 on the performance of MaestroMotif.
we also investigate the impact code refinement
Hierarchical architecture As illustrated Figure 10 of
Appendix A.7 the neural network used to execute the skill poli- cies follows almost exactly the same format as the PPO base-
Normllize Performace
04
line (Miffyli 2022), with the only difference of an additional conditioning via a one-hot vector representing the skill currently being executed. We found that this architectural choice to be crucial for effectively learning skill policies. In Figure Sal we compare this choice to representing the skills through ditferent policy heads, as is sometimes done in the literature (Harb et al. 2017, Khetarpal al.) 2020). This alternative approach leads to collapse in performance. We hypothesize that this effect comes from gradient interference as the different skill policies activated with different frequencies.
0.0
75t
4078
Scale LLM Coder
Figure 7: Impact of scaling for the LLM code generator on final per- formance across tasks.
Emergent skill curriculum In Figure Sal we also verify the importance learning all the skills simultaneously. We compare this approach to learning each skill in a separate episode. We notice that without the use the training-time time policy over skills, the resulting performance significantly degrades. To better understand the reason behind this, plot in Figure 8b and Figure &c] for each skill, the corresponding reward during training. Learning each skill in isolation leads to a majority of the skills not maximizing their own rewards. On the other hand, learning multiple skills in the

---

Published as a conference paper at ICLR 2025
same episode leaves space to learn and to leverage simpler skills, opening the possibility of using those simple skills to get to the parts of the environment where it is relevant to use more complex and context-dependent ones, such as the Merchant or the Worshipper. This constitutes skill curriculum, which is naturally induced by the training-timv time policy over skills. The curriculum emerges because of the data distribution in which each skill is initiated: skill expressing a more advanced behavior will only be called by the policy over skills when the appropriate situation can be reached, which will only happen once sufficient mastery of more basic skills acquired. We discuss in Appendix A.9 how modifying the skill selection strategy, for example by adapting it through online interactions, could further improve the ability to learn skills.
emergent
5 RELATED WORK
LLM-based hierarchical control methods Our method relates to a line of work which also uses LLMs to coordinate low-level skills in a hierarchical manner. SayCan and Palm-E (Ahn et al.) 2022 Driess et al. 2023) also use an LLM to execute unstructured, natural language commands by recomposing low-level skills in a zero-shot manner. A key difference in our work is how the skills are obtained: whereas they leverage a combination of large human teleoperation datasets of language- conditioned behaviors and hand-coded reward functions, we train skills from intrinsic rewards which are automatically synthesized from unstructured observational data and natural language descriptions. MaestroMotif is particularly related to those approaches in which a high-level policy is generated as a piece of code by an LLM (Liang et al.) 2023). Voyager (Wang et al. 2024) also uses an LLM to hierarchically create and coordinate skills, but unlike our method, assumes access to control primitives which handle low-level sensorimotor control. LLMs have also been used for planning in PDDL domains (Silver et al. 2023). see Appendix A.10 for a detailed discussion.
Hierarchical reinforcement learning There is rich literature focusing on the discovery of skills through a variety of approaches, such as empowerment- based methods (Klyubin et al. 2008 Gregor et al. 2017). spectral methods (Machado et al. 2017]] Klissarov & Machado 2023) and feudal
approaches (Dayan & Hinton, 1993] Vezhnevets et al. 2017) Most of these methods are based on learning representation, which is then exploited by an algorithm for skill learning (Machado et al. 2023). In MaestroMotii, we instead work in the convenient space of natural language by leveraging LLMs, allowing us to build on key characteristics such as compositionality and interpretability. Interestingly, some of the skills we leverage in our NetHack implementation are directly connected to early ideas on learning skills, such those based on notions of bottleneck and in-betweeness (Ibal et al.) 2002 Şimşek & Barto| 2004).. Such intuitive notions had not been scaled yet they hard to measure in complex environments. This is precisely what
1989 McGovern & Barto|
MaestroMotif provides: a bridge between abstract concepts and low-level sensorimotor execution.
DISCUSSION
Modern foundation models possess remarkable natural language understanding and information processing abilities. Thus, even when they are not able to completely carry out task on their own, they can be effectively integrated into human-Al collaborative systems to bring the smoothness and efficacy of the design of agents to new heights. In this paper, we showed that MaestroMottif is an effective approach for Al-assisted skill design, allowing us to achieve untapped levels of controllability for sequential decision making in the challenging NetHack Environment. MaestroMotif takes advantage of easily provided information (i.e., a limited number of prompts) to simultaneously handle the highest-level planning and the lowest-level sensorimotor control problems, linking them together by leveraging the best of the LLM and the RL worlds.
Like other hierarchical approaches, MaestroMottif is limited in the behaviors can express by the set of skills it has at its disposal: given a set of skills, a satisfactory policy for a task might not be representable through their composition. Therefore, an agent designer should perform Al-assisted skill design while keeping in mind what behaviors should be eventually expressed by the resulting agents. Despite this inherent limitation, we believe our work provides a first step towards a new class of skill design methods, more effective and with a significantly higher degree of automation than existing ones. More broadly. MaestroMotii also constitutes evidence for the benefits of a paradigm based on human-Al collaboration, which takes advantage of the complementary strengths of both.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian lchter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not as say: Grounding language in robotic affordances, 2022. URL https: /ttp:://arvvvv.. http:::://axxi..or//abbb... wtgg /ataaaas/22442 220.200699 01691
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Confer- ence on Machine Learning, ICML 2017. Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 166 166-175. PMLR, 2017.
Akhil Bagaria, Jason K. Senthil, and George Dimitri Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, 2021. URL
https://aptl http://appssmaamiii..aemm www.:e/ssssnnniccchhooeoeoee
www//oorgouuooooooo 22210099 221094080
:21
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Processing Systems, 2016. URL wttp::tt///// //ttps://ap5. /api URL semant: caccholasssc org/ Corp/srrpuuuso 83100655
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL http:httt//seee ptttps://ase semanticscho
org/CorpusTTi ID: 125209808
Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei,
Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. URL
https::/api. semant
org/ CorppsiI:piii:i :
Özgür Şimşek and Andrew G. Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings the Twenty-First International Conference on Machine Learning, ICML '04, pp. 95, New York, NY, USA, 2004. Association for Computing Machinery, ISBN 1581 138385. 1015330. 1015353
doi: 10. 11144//001533000153333. /RChhppss///oo..orr/00011111
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Robotics: Science and Systems, 2014. URL https: //api. semant. eacho ar. orrg
Corpus IerpusID:2 6043466)
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learnnn In Advances in neural information processing systems, pp. 271-278, 1993.
Gary L. Drescher. Made-up minds a constructivist approach to artificial intelligence. 1991. URL
https://api. www.etttscntnnooeen.... httpsss/aap...emmniiccchh.... wttp:echho.aaa.ooorrrroos...s.se wtt/Corrps ID: 3099707
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:23033. 03378, 2023.
Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey al. The llama 3 herd of models. ArXiv, abs/2407.21788. 2024. URI htthttpp////aaa semant i cscho ar orgg wwwo/oo/oooooooooooooooooooooo cprsss DD
271571434]
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar, Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thitty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022, URL httthttp:// http:////pe//eeeeee/// /02/:///ttps:///eneeeeee/ net fortffrrm??
dd=rcco__8888 T8X
11

---

Published as a conference paper at ICLR 2025
Richard Fikes, Peter E. Hart, and Nils J. Nilsson. Learning and executing generalized robot plans. Artif. Intell., 3:251-288, 1993. URL http:: //apps::///pp.. semanticscholar www.aaccww...aa..oorrro...... org/CorpusID;
17260619
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra, Variational intrinsic control. In 5th International Conference on Learning Representutions, ICLR 2017, Toulon, France, April 24-26. 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL /stps::/openzeeierr
net pot//oose Ett.e/oornnndi---eeeeee
Eric Hambro, Sharada Mohanty, Dmitrii Babaev, Minwoo Byeon, Dipam Chakraborty, Edward Grefenstette, Mingi Jiang, Jo Daejin, Anssi Kanervisto, Jongmin Kim, Sungwoong Kim, Robert Kirk, Vitaly Kurin, Heinrich Küttler, Taehwon Kwon, Donghoon Lee, Vegard Mella, Nantas Nardelli, Ivan Nazarov, Nikita Ovsov, Jack Holder, Roberta Raileanu, Karolis Ramanauskas, Tim Rocktäschel, Danielle Rothermel, Mikayel Samvelyan, Dmitry Sorokin, Maciej Sypetkowski, and Michat Sypetkowski. Insights from the neurips 2021 nethack challenge. In Douwe Kiela, Marco Ci- ccone, and Barbara Caputo (eds.), Proceedings of the NeurIPS 2021 Competitions and Demonstra- fions Track, volume 176 of Proceedings of Machine Learning Research, pp. 41-52. PMLR, 06-14 Dec 2022a. URL BtLLhtts::// http:::://prooceeding ngs. ml www.pressssee vll hht.m//aaa222aa.. htmmmml...mmmmmmmm.mmmmm
Eric Hambro, Roberta Raileanu, Danielle Rothermel, Vegard Mella, Tim Rocktäschel, Heinrich Kuttler, and Naila Murray. Dungeons and data: A large-scale nethack dataset. In Thirty-sixte Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022b.
URL htt.httpss//oppneeeiieee net.: /wt/tete/oorreeee
forum? H 0xN
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not option Learning options with a deliberation cost. ArXiv, (bb///70900451,, 2017. URL https::/app
semanticscholar org/ Corpus ID: 19247295
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. In NeurIPS, 2022.
Ex-
ploration via elliptical episodic bonuses.
URL
http:///ppers.mpps..cpppppp
html
C. R. Hoare. An axiomatic basis for computer programmin. Commun. ACM, 26:53-56, 1969.
URL https://api. Semant i icscoo wttpaarrrrrooooooorpa 6059550
Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. Open-endedness is essential for artificial superhuman intelligence, 06 2024.
Glenn A. lba. A heuristic approach to the discovery of macro-operators. Machine Learning, 285-317, 1989. URL httph:tts://// semant cscho ar. ar. orr.org//e /tt.or//crrruss111333499999
Khimya Khetarpal, Martin Klissarov, Maxime Bervalier-Boisver Pierre-Luc Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1444-4451, 2020.
Martin Klissarov and Marlos C. Machado.
Deep laplacian based options for temporally-extender exploration. International Conference on Machine Learning, 2023. URL https::/app)
semant i cscho 1 ar org/ DD:
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. In The Twelfth International Conference on Learning Representations, 2024. URL
https: http:://ppeeeevieeeeeee .ee wwt.:eewwwetttttiieettoom
Alexander S Klyubin, Daniel Polani, and Chrystopher L. Nehaniv. Keep your options open: An information based driving principle for sensorimotor systems. PloS one, 3(12)xc4018 2008.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
12

---

Published as a conference paper at ICLR 2025
George Dimitri Konidaris, Leslie Pack Kaelbling, and Tomas Lozano- Perez. From skills symbols: Learning symbolic representations for abstract high-level planning. J. Artif. Intell. Res., 61: 215-289, 2018.
Nishanth Kumar, Tom Silver, Willie McClinton, Linfeng Zhao, Stephen Proulx, Tomás Lozano- Pérez, Leslie Pack Kaelbling, and Jennifer Barry. Practice makes perfect: Planning to learn skill parameter policies. In Robotics: Science and Systems (RSS), 2024.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK. May 29 June httpts
2, 2023, pp. 9493-9500, IEFE, 2023. doi: 10.1109/CRA4889) 202..10160599
10. 11 T9/TCRA4T000 8891 911-911111 2023. 101 60591
B. Liu, Yuqian Jiang, Xiaohan Zhang, Qian Liu, Shiqi Zhang, Joydeep Biswas, and Pe- ter Stone. LIm+p: Empowering large language models with optimal planning proficiency. ArXiv, Abs/2304.11477/ 2023. httpttts://pi. 258298051
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In Luc De Raedt (ed.), Proceedings of the Thirty- -irst International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 5502-5511, ijeai.org, 2022.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, pp. 2295-2304 PMLR, 2017.
Marlos C. Machado, André Barreto, and Doina Precup. Temporal abstraction in reinforcement learning with the successor representation. J. Mach. Learn. Res., 24:80:1-80:66 2023. URL
wtag///rrsssiie ID: 238634579
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self- feedback. ArXiv, abs/2303.17651. 2023. URL https: www.essisnnnccchiir............. org
CorpusID:2533
Bhaskara Marthi, Stuart Russell, David Latham, and Carlos Guestrin. Concurrent hierarchical reinforcement learning. In Leslie Pack Kaelbling and Alessandro Saffiotti (eds.), IJCAI-05, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30 August 5, 2005, pp. 779-785. Professional Book Center, 2005. URL 05/ Papers
pdf
Michael Matthews, Mikayel Samvelyan, Jack Parker-Holder, Edward Grefenstette, and Tim Rocktäschel. Hierarchical kickstarting for skill transfer in reinforcement learning, 2022. URL
https://arxiv: http:///aaxxi..oo//ab//////////
wttg/ass/g///000 11584
Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998. URL //api. www.eeeinnnnnolaeeeeeeeeeeeeeeee wtt.olaar..rr/ooopsuuuiu..
59656859
Amy McGovem and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In International Conference on Machine Learning, 2001. URL https: 7/ 7/api.. sema ccchoo ar. wwwpollla.rrroooooooe/oe ptt..orr//orrpuss:223777
13

---

Published as a conference paper at ICLR 2025
Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut dynamic discovery of sub-goals in reinforcement learning. In European Conference on Machine Learning. 2002. URL https: //api. //apl.semanticschoo...
azg/CorpusSD::83010
Jorge Mendez-Mendez, Leslie Pack Kaelbling, and Tomás Jozano-Pérez Embodied lifelong learning for task and motion planning. In 7th Annual Conference on Robot Learning, 2023. URL https:
/ //openreview.o www.forumeeemeeeeeeemmmmm d=2F
Miffyli. nlc-ssmple-facttoy--bccllin 2022. URL GitHub repository.
nle sample-
Dion Moult. Nethack: an illustrated guide to the mazes of menace, dec 2022. URL https: //
thinkmoult, www.woow.......eee...e hwtp/nethacc. http://eetaac-cllsssreeeeeee goiddeguieeeeee mazes of menace html Ac-
cessed: 2024-09-02.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville, Film: Visual reasoning with general conditioning layer. In AAAI Conference on Artificial Intelligence, 2017.
URL https: /app. https://ap...emmn......m www.atiisscciiieeeeeee
org/ http///rppsDD::: 9119291
Aleksei Petrenko, Zhehui Huang. Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learn- ing. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learming Research, pp. 7652- 7662. PMIR, 2020. URL html
ngs ml www..pppee/.wlleeemmm http://v111//petteeeeeeeeeee
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. diff history for neural language agents, 2023a.
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. Nethack is hard hack, 2023b.
Doina Precup. Temporal Abstraction in Reinforcement Learning. PhD thesis, University of Mas- sachusetts Amherst, 2000.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv 2017.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Neural Information Processing Systems, 2023. URL https: www.nticscho..n........
258833055
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In AAAI Conference on Artificial Intelligence, 2023. URL.tttpe //api. semanticschi www.chcc..hcooooooooooooooo..eo. org/
Corpus 662760
Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Symposium on Ab- straction, Reformulation and Approximation, 2002. URt.https://app. oll.itttpe//ppp...emnttcccchhh semannicccccinn
org/CorpusID: : 16398811
Shao-Hua Sun, Te-Lin Wu, and Joseph Lim. Program guided agent. In International Confer-
Learning Representations, 2020. URL
net wwt.:/w..tteteeeee//e.em forum?id=
BkxUvnEYDH
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps:
framework for temporal abstraction in reinforcement learning. 1999. URL https://app
Demanticschi ar. wttp:/oorrossi:255333333
Maciej Sypetkowski and Michat Sypetkowski. Autoascend 1st place nethack agent for the nethack challenge at neurips 2021. https: com www/ htt//macieii-ee httmm//aeeee-sspstoossssss autoascend 2021. GitHub repository.
14

---

Published as a conference paper at ICLR 2025
Andrea Lockerd Thomaz, Cynthia Breazeal, al. Reinforcement learning with human teachers: Evidence feedback and guidance with implications for learning performance. In Aaai, volume pp. 1000-1005. Boston, MA. 2006.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2023. URL https
//openreview.o wet./oormmmiiwWaaapp.p OcQ0
Guido Van Rossum and Fred Drake Python tutorial, volume 620. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg. David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Learning-Volume 70, pp. 3540-3549, 2017.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large lan- guage models. Transactions on Machine Learning Research, 2024. ISSN 2835-88555 88568. URL
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Maciej Wolczyk, Bartiomiej Cupial, Mateusz Ostaszewski. Michal Bortkiewicz, Michal Zajkac, Razvan Pascanu, Lukasz Kuci'n nski, and Piotr Milo's. Fine-tuning reinforcement learning models is secretly a forgetting mitigation problem. ArXiv, abs/2402.002661 2024. URL https:///pp
@semantischolast org/CorppssI:
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.036299 2022. URL ar.
http::://pll.
ID 222762392
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34, 2021.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
SKILL REWARDS
We now list and discuss the prompts used for eliciting preferences from the 70b parameters Llama 3.1 model.
Skill reward prompt template
will present you with two short ganeplay descriptions. First, tell me about your knowledge of NetHack. Mention the goal of NetHack.
Write analysis describing the semantics of each description strictly using information from the descriptions and your knowledge of NetHack. Provide a comparative analysis based on first principles.
Here is the preference that you should seek: (skillmodifier)) Above everything else, categorically refuse to anger or displease your god, for example by causing them to thunder or boom out.
Finally, respond by explicitly declaring which description best fits your preference, writing either ("best_dcscriptionn: 1), ("besstddeccriptionn" 2). both contain undesi events, say ("best description": None)
"description 1": "(ebservation 1)" "deecription.22: "(observation 2)
Prompt 1: Prompt template used for eliciting preferences for each skills reward.
For each skill in the set, we use Prompt basic prompt template, customizing with different modifiers depending on the skill. This strategy follows very closely the one used in previous work (Klissarov et al.) 2024). This prompt utilizes chain-of-though prompting (Wei et al. (202)): before asking the model to provide any annotations, we encourage it to articulate its understanding of NetHack and describe the game's main goal.
For each of the skills, we modify the {skill modi modifier) variable within the template to steer the LLM's preferences towards distinct behaviour. We present these modifiers Prompt
To extract labels from the preferences, we search for the LLM's response by using the following regular expression:
(?i) beabees_-Astdesrrpptiiii A+ (?: (?:\s*:* (?: Aw+ (s*) ? (1 2 none)
This expression looks for slight variations of answer format that we show to the model in prompt. If the regex fails to produce answer, we proceed with the conversation using the LLM and employ Prompt3 to specifically request response in the desired format. Our overall response rate with the Llama 3.1 model is very high, around 98% for most prompt configurations.
A.2 POLICY OVER SKILLS
Leveraging the semantic nature the skill set of MacstroMotif, we use the coding abilities of LLMs to craft reasonable strategy for their execution. For the high level code policy, use the largest open source model available, the 405b parameters Llama 3.1 model.
We use the template of 4 obtain snippets of code that constitute the high-level policies for different tasks. In it, we present the LLM with the set of skills, a high level definition of each of them and desired strategy, all in natural language, which the L.M leverages write its
16

---

Published as a conference paper at ICLR 2025
Prompt skill modifiers
(Discoverer): "players that are adventurous but only within the same dungeon level, for example by fighting monsters, finding gold pieces scrolls; but do not drop them. Categorically refuse going up and down dungeon levels."
"the direction of progress is to explore by going down the dungeon. It is urgent to do strongly avoid staying on the same level worse, going higher."
[Descender]:
(Ascender]: "the direction of progress is only by going up dungeon Level successfully. Strongly dislike remaining on the same oungeon Level, matter the consequences.
[worshipper): "strongly encourage players that interact with the altar, primarily for identifying whether items are cursed or blessed, rather than for praying to pleasing their god."
[Merchant]: "prefer players that negotiate, sell and interact with shopkoepers. Be careful not to steal from stores."
Prompt Skill modifiers used for instantiating variations of Prompt
Retry prompt
8o, which one is the best? Please respond by saying ("best description": 1), ("best description". 2), or ("best description": None)
Prompt 3: Prompt provided to the LLM to continue the conversation when the regular expression does not find valid annotation in the LLM's ss answer to the original prompt.
NetHackP 1 aaer class, representing the high level policy. We employ form of chain-of-thought prompting to obtain an initial version of this policy defined in code, an example is given in Output [
In many situations, this initial attempt does not capture an adequate strategy for skill execution. To avoid such undesirable outcomes, we leverage the LLM's capability for self-refinement through the help of self-generated unit test. An initial attempt is passed through unit test, producing trace of exccution as shown in Output 2 We then ask the model whether the produced trace satisfies the strategy. If the answer is yes, the self-refinement procedure stops. the answer no, ask the LLM to reflect on the code it has previously proposed, identify potential flaws in its logic and write an improved version (as shown in Prompt 5). This process repeated for maximum of iterations.
Such process is similar to standard refinement prompting strategies for LLMs (Shinn et al. 2023 Madaan et al. 2023).
A key element of the self-refinement strategy is to leverage unit test that generates traces of execution. This unit test itself crafted by the L.M through Prompt which presents the LLM with the same list of skills, their description in natural language and blueprint of the unit test's structure.
This strategy employed to generate the code exploration policy used to learn the skill policies. This is done by first defining general select_skii 11 method for selecting skills. This method is then leveraged to define the beachhdungeons.o wf of_doom and reach-gnomi sh mi ne methods which steer the agent between the different branches of NetHack (as shown in 4.. We present in Output one of the obtained explorative code policies. In Output 4 we present one the code policies for achieving the Dí scovery Hunger composite task.
17

---

Published as a conference paper at ICLR 2025
Prompt for the train-time policy over skills
You are to write code which defines the method "select.skill skill" the NetHack Player class that selects amongst a aet of skills in the videogame of NetHack. The set of skills corresponds to ("discoverer", "descender", "ascender", "merchant", "worshipper").
when activated, the Discoverer fully explores the current dungeon, while fighting off enemies. The Descender makes its way to 8 staircase and goes down. The Ascender makes its way to staircase and goes up. The Merchant interacts with shopkeepers by selling its items. The Worshipper interacts with altars by identifying its itens.
Find strategy that will let the player explore fully each of the first few dungeon levels, alternating directions between going all the way down towards the maximum depth, then going up towards the first dungeon. This might get interrupted by the end of the loop or if the preconditions of worshipper and merchant allow for it.
You can keep track any other information by assigning values to other class attributes, but only if that really helps.
Your code will be verified through this unit test.
44#
(anit.test)
###
Before writing code, write few questions for yourself, and answer them to make sure you understand the assignment. In your response make sure to include the code for the class definition and the unit test in python markdown.
Prompt 4: Description of the coding task for the LLM to code the train-time policy over skills.
Retry prompt for code policy
Go through your code, 1ine by line, and reflect on where the issue arises from. Use this to modify your code, remove unnecessary parts add elements.
your response nake sure include the code for the class definition and the unit test in a python markdown.
Prompt 5: Prompt used for code self-refinement while producing code policies.
As shown in Prompt the L.LM allowed to create additional attributes to define the code policy over skills. When the LLM defines such attributes, it is afterwards queried to write code for these attributes such that their values are gathered from the NLE.
The average amount of tokens produced for policy over skills is 9030 tokens according to the Llama tokenizer. Similarly, the average amount of tokens used for the termination and initiation functions is 810 tokens. To query the 405B model, there exists many solutions online and locally, with throughput as high as 969 tokens/second (generating policy in largely less than a minute even including the refinement process) and cost as low as S3/1M (generating policy for few cents).
18

---

Published as a conference paper at ICLR 2025
Initial policy over skills
class NetHackPlayez:
def _init__t__ (self, max_depth);
self.max_deptae = max_depth.
self. ("discoverer", "descender", "ascender",
self.direction "merchant", "xorshipper"] for down, for up
def serchant_prrcondiitie (self):
For the purpose of this example, it always returns False return Palse
def worshipper cetpiipee_rrecniiiion (self):
# For the purpose of this example, it always returns False return False
def select_skee (self, current_skill, Langeon_deptiinn
merchant_ proconnt_prcoondtiion vorshipper.prrcondtttee
if merchant merchant.prccondliio.:
veturn'merchant
elif vorshiipee_prreondttin return 'worshipper'
elif current_skill - 'di discoverrr';
if dungeon_deptb. self _aa__depxhhhhhaaeeeeeeaepeee depth and self self.direction return 'deccender'
elif dungeon__eppa depth 1 and seleciiroction = -1: return 'ascender'
else:
self.direction delf.direction 1:
return 'deccende''
else:
return 'ascender'
elif current_skill 'descender's return discoverer''
elif current_skill -- 'ascender': return 'disccrerrr'
Output Example of generated policy over skills before any refinement.
A.3 INITIATION AND TERMINATION
Finally, we leverage the coding abilities of the LLM to also define the termination and initiation functions of the skills. These quantities, together with the skill policies, define the option tuple from the options framework (see Section The termination function indicates when a skill should finish its execution and the initiation function when it can be selected by the high level policy. As these functions are significantly simpler than the high level policy, we do not leverage the same self-refinement through unit tests. In Prompt we present the prompt used define the termination function and in Prompt 9 the one to define the initiation function.
A.4 CODE REFINEMENT
In Figure we further compare the importance of leveraging code refinement through self- genf-gene unit tests. We notice that this leads to improved results when using the 405b LLM, however no significant difference observed for the smaller models.
19

---

Published as a conference paper at ICLR 2025
Unit test execution trace
Turn 1: Skill = discoverer, Dungeor: depth =
Turn 2: Skill descender, Dungeon depth 2
Turn 3: Skill discoverer, Dungeon depth 2
Turn 4: Skill descender, Dungeon depth
Turn 5: Skill discoverer, Dungeon depth 3
Turn 6: Skill descender, Dungeon depth = 4
Turn 7: Skill discoverer, Dungeon depth 4
Turn 8: Skill descender, Dungeon depth
Turn 9: Skill discoverer, Dungeon depth 5
Turn 10: Skill ascender, Dungeon depth = 4
Turn 11: Skill discoverer, Dungeon depth 4
Turn 12: Skill ascender, Dungeon depth
Turn 13: Skill discoverer, Dungeon depth 3
Turn 14: Skill ascender, Dungeon depth = 2
Turn 15: Skill discoverer, Dungeon depth 2
Turn 16: Skill ascender, Dungeon depth
Turn 17: Skill discoverer, Dungeon depth
Turn 18: Skill descender, bungeon depth 2
Turn 19: Skill discoverer, Dungeon depth 2
Turn 20: Skill descender, Dungeon depth 3
Output 2: Example of output from a unit test written by the LLM.
Nomalized Perfomance
0.4
valurat Keiromttn
0.2
0.0
8b
70b
405b
Scale of LLM Coder
Figure Refining the LLM Coder outputs through self-generated unit tests yields significant improvements for the 405b parameter Llama model.
A.5 ENVIRONMENT AND METHOD DETAILS
We base our implementation on the NetHack Learning Environment (Küttler al. 2020) and Chaotic Dwarven GPT-5 baseline (Miffyli 2022), which itself was defined on the fast impiementation of PPO (Schulman al. 2017) within Sample Factory (Petrenko et al. 2020p. As discussed in Klissarov et al. (2024), although some actions are available to the agent (like the "eat' action), it is not possible for the agent actually eat most the items in the agent's inventory. This limitation is also other key actions such as the action for drinking, the 'quaff" action in NetHack terms. To overcome this limitation, we make a simple modification to the environment by letting the agent eat and quaff any of its items, at random, by performing particular command (the action associated with the key y). We also include standard actions such as pray, cast and enhance. All agents that we train are evaluated using these same conditions, except the behaviour cloning based agents in Figure which have access to an even larger action set.
for
For the skill reward training phase of MaestroMotiif, Bonus baseline (Henaff et al. (2022). Similar to Klissarov et al. (204)). we train the intrinsic reward ra with the following equation,
the message encoder from the Elliptical
20

---

Published as a conference paper at ICLR 2025
Unit test prompt
You are to write code for a unit test the NetRackPlayer class and its "select.skill" method. This method takes as input the skill, "dungeon.depth" and "branch.number" arguments and outputs skill. You must write code that simulates how the environment reacts to the "select.skill" method.
The skills consist of "discoverer", "descender", "ascender", "merchant", "worshipper". when activated, the Discoverer fully explores the current dungeon, while fighting off enemies. The Descender makes its way to staircase and goes down. The Ascender makes its way to a staircase and goes up. The Merchant interacts with shopkeepers by selling its items. The Morshipper interacts with altars by identifying its items.
Here is the template:
max.depth =
player NetHackPlayer (max.depth)
skill discoverer' dungeon.depth =
for turn in range (20)
print (f"Turn [fturn 1)): Skill [[skil1)), Dungeon depth
[[dungeon depth))")
merchant precondition = alayer.merchan precondition()
vorshipper.orecondtiin player. wow.sopprppprrrrrereddiiiiaea
skill weasyerrseeettssiile (skill, dungeon.depth, werchant.precoddtiion worshipper.preeoonitiiol
I the environment updates the dungoon.depth
Code here
You are to write the unit test only in its current form, net the NetHackPlayer class. Do not create new classes, functions or import anything.
Prompt 6: Prompt given to the LLM code generator for coding up the unit test used during refinement.
[(4) (2 Daat
1 = 1]log Pylo2 > o2) + 1ly = 2og Po x 0il
+ 1ly = log (P ~ 0z) Polo2 > 0i]
where Poloa oz] chatta) fe (og) is the probability of preferring observation another. This the Bradley-Terry model often used in preference-based learning (Thomaz et al. 2006 Knox
Stone 2009 Christiano et al. 2017). The work on Motif adopted this reward transformation.
Tine (on(erservti =1 ( (orrrriinnnnnlllrbrrrrbrrraraaer (observvato) 2 c] h/wpa/////o/aaaaaa////////a//aaa
where N(observation) was the count of how many times a particular observation has been previously found during the course of an episode. We adopt the same reward transformation, although we relax the requirement that N() is a function over the full course the episode, but rather over the last 20 steps. This opens the opportunity leverage this transformation larger spectrum
21

---

Published as a conference paper at ICLR 2025
of environments by keeping short memory of transitions rather than functional forms of counting which are difficult to achieve in many practical settings (Bellemare et al. 2016).
Hyperparameter
Value
Reward Scale
0.1
Observation Scale
255
Num. of Workers
24
Batch Size
4096
Num. of Environments per Worker
20
PPO Clip Ratio
0.1
PPO Clip Value
1.0
PPO Epochs
1
Max Grad Norm
4.0
Value Loss Coeff
0.5
Exploration Loss
entropy
Table 2: PPO hy perraaarameees..
To obtain the LLM-based reward, we train for 20 epochs using learning rate of x 10-5, As Equation Baattioonnsoowss. we further divide the reward by an episodic count and we only keep values above certain threshold. The value of the count exponent was 3 whereas for the threshold we used the 85th quantile of the empirical reward distribution for each skill, except the Discoverer which used the 95th quantile. For the Motif and Embedding Similarity baseline, we perform a similar transformation on their reward, using a count exponent was 3 whereas for the threshold we used the 30th quantile. For all methods, before providing the LLM-based reward function to the RL agent, the Motif paper, the authors
normalize
by subtracting the mean and dividing by the standard deviation. additively combine both the I.LM-base intrinsic reward and reward coming from the environment with a hyperparameter o, leading to different trade-offs for different values. In MotitroMotif we completely remove this hyperparameter and instead learn completely through the intrinsic reward coming from the LLM. Finally, in Table hyperparameters.
we report the remaining standard values of the RL agent's s
A.6 BENCHMARK DESIGN AND MOTIVATION
We note that out of the original tasks from the NLE paper, the Staircass (and closely related Pet) tasks have by now been solved (Zhang et al. 2021 Klissarov et al. 2024). The Score task is effectively unbounded, but noted in (Wolczyk et al. 2024), it is possible to achieve very high scores by adopting behaviors which correlate poorly with making progress in the game of NetHack (for example, by staying early levels and killing weak monsters). This is also observation corroborated by our experiments in Section 4.2
To define set of compelling and useful tasks in the NLE, take inspiration from the NetHack community, in particular, from the illustrated guide to NetHack Moult (2022). This guide describes various landmarks that every player will likely experience while making progress in the game. Some of these landmarks were also suggested in the original NLE release Küttler et al. (2020). The first such landmark is the Gnomish Mines which constitutes the first secondary branch originating in the main branch, the Dungeons of Doom (see Figure 4). The second landmark is Minet own, deeper level into the Gnomish Mines which players might interact with Shopkeepers and gather items. The third landmark is the Delphi, which is level that appears somewhere between depth and in the main branch and is the home to the Oracle, famous character the game. is not necessary interact with the Oracle to solve the game of NetHack, but reaching the Delphi is necessary step towards it, which is the reason we include it and not the Oracle task.
As these tasks are navigation oriented, we additionally include set of tasks that require the agent to interact with entities found across the dungeons of NetHack. The interactions we select are chosen because they key the success to any player playing the NetHack game. For this reason, focus on interactions that will give the agent more information about its inventory of items. In NetHack, most items that are collected have only partially observable characteristics. For example, ring
22

---

Published as a conference paper at ICLR 2025
that is found could be blessed or cursed, and its magical effects are not revealed (it could be ring of levitation, a ring of cold resistance, etc.).
The first type of interactions are those where the agent interacts with altars associated with the NetHack gods. These offer many benefits, the most common one is the possibility to identify the blessed/curseedunncrssee (B/U/C) status of an item. The difference between cursed and uncursed item can have deadly consequences in NetHack. The second type of interactions are those where the agent finds a shopkeeper either sell item and collect gold, or attempts sell an item get an offer from the shopkeeper. When getting price offer from the shopkeeper, it is possible to identify the kind of item that the agent has in its possession (i.e. a wand of death or wand of enlightenment).
Overall, we believe that these tasks are well-aligned with making progress towards the goal of NetHack. is also important to note that even though these tasks are very hard for current Al agents, they only represent fraction of the complexity of NetHack.
A.7 HIERARCHICAL ARCHITECTURE
5g 1
St-1
CNN
VLE
MLP
MIP
LSTM
LSTM
MLP
Ot
bp
MLP
St
Skill Index
(a) Skill-conditioned policy
(b) Multi-head policy
Figure 10: Neural network architectures. The architecture on left, used throughout the paper, was key for the the successful training of the skill policies.
In Figure T0a we present the architecture used to learn the skill policies, which simply consist of single neural network conditioned on one-hot vector. This one-hot vector represents the skill index (i.e. the first entry this vector is associated with the Discoverer skill and the last one with the Me Merchant skill). This implementation is not only efficient in terms of the number of parameters needed to represent diversity of behaviours, but also was also crucial for successfully learning these behaviours. We explored alternative architectures, such as adding multiple heads to the network, each for one of the skills, as shown in Figure 10b] Results in Figure show that this lead to a collapse in performance which different skills. It is important to notice that the skills are activated with very different frequencies (for example the Discoverer is activated almost 50 times more often than the Worsh Worshippen r).. Another possibility in terms of architecture would be to consider more sophisticated conditioning mechanism such as FiLM (Perez al. 2017) which has been successful in various applications.
attribute to catastrophic interference between the gradients coming from
A.8 ADDITIONAL ABLATIONS
we have presented the ways in which the annotation process used in the NetHack implementation of Maestro differs from the one presented in K Klissarov al. (2024). In Figure [] we verify how each of these choices affects the final performance of our algorithm. The importance of providing the player statistics within the prompt eliciting preferences from the LLM is made apparent, as without such information the performance drops to almost 30% of its full potential. When the player statistics are provided but no information about how they differ from recent values (i.e. diffStats), the resulting performance similarly decreased. This is explained by the non-Mare non-Markovian nature of observations in NetHack: as example, a status shown as hun arygry could be the result of being previously sat iat ed or fainting; which present two quite different ways of behaving and would produce difference preferences. Finally, our preference elicitation phase integrates episodes from the Dungeons and Data dataset (Hambro et al.) (0226), which provides greater coverage of possible interactions and observations of NetHack. We notice that this choice is
Preference elicitation In Section 3.3
23

---

Published as a conference paper at ICLR 2025
Normalizer Performance
0.4
MotstroMatif
w/o diffStats
w/o player statisties
0.3
wyo human gameplay
0.2
0.1
0.0
Figure 11: Ablation studies on MotstroMotif's design choices.
important obtain the full performance MaestroMotii. This result illustrates how Al feedback can be an effective strategy for leveraging action-free and reward-frre datasets.
A.9 CONSIDERATIONS FOR THE SKILL SELECTION
In this work, we have leveraged LLM to define training- tiimin high-level policies, termination and initiation functions in order to learn the skills. These components defining the skills selection strategy were then fixed during the skill learning process. As have seen in Section 4.3 this led to an emerging curriculum over skills, where easier skills developed first and harder skills developed later on. However, could see significant improvements in skill learning efficiency if the high-level policies, termination and initiation functions were instead adapted online. This could be done, for example, by deciding what skills to select and how to improve them (Kumar et al.) 2024). Ideas from active learning (Daniel et al., 2014] Mendez-Mender et al. 2023) would be of particular value for pursuing this research direction. Another consideration with respect to the high-level policy its robustness. Currently, before the high-level policy is deployed, it is verified through self-generated unit test. This strategy was generally successful to avoid particular failure modes and obtain good strategies. However, it is not full-proof strategy, and adapting the high-level policy through online interactions could be significantly more robust. One way to approach to adapt the high-level policy would be to provide in-context execution traces from the environment through which the L.M could iterate on a proposed strategy. Another approach would be through RL, for example through intra-option value learning (Sutton al. 1999). We are then faced with the following question: what reward would this high level policy optimize? A possible answer would be to apply Motif to define such reward function on a per-task basis.
A. ADDITIONAL. RELATED WORK
Connections to the Planning Literature MaestroMotif learns skills through RL and, when faced with a particular task, re-composes them zero-shot through code that defines the execution strategy. To do so, the LLM writing the code needs to specify where skills can initiate, where they should terminate and how to select between them. MaestroMottif is in fact an instantiation of the options formalism (Sutton al. 1999 Precup 2000), which defined the necessary quantities for learning skills in RL. However, the idea to abstract behavior over time in the form of skills has a long history in AI. for example through STRIPS planning (Fikes et al. 1993). macro- oprrrtorr Iba (1989). Schemas Drescher (199T) and Planning Domain Definition Language (PDDL) (McDermott et al.) 1998). The structure behind the option triple can also be seen in related fields, such as formal systems through the Hoare logic (Hoare) 1969). Silver et al. (2023) recently investigate how LLMSs can be used as generalized planners by writing programs in PDDL domains, which similar to how MaestroMottif write code to sequence skills. Their results show that LLMs are particularly strong planners. Another promising direction would be to use LLMs to convert natural language into PDDL, to then leverage classical planning algorithms (Liu et al.) 2023). Further investigating the connections between the options framework and symbolic representations would be particularly promising (Konidaris et al. 2018 Bagaria al. 2021), in particular in the context of L.LMs.
24

---

Published as a conference paper at ICLR 2025
Zero-shot
specifi training
Task
MaestroMotif
LLM Policy
LLM Policy (Fq.. Prompting)
Motif
Motif (Eq. Prompting)
Gnomish Mines
46% 1.70%
0.1% + 0.03%
0.3% :: 0.03%
9% t: 2.30%
9% = 2.30%
Delphi
29%
10%
0% 0.00%
0% = 0.00%
0.70%
1.7% 0.70%
Minetown
7.2% :: 0.50%
0% 0.00%
0% = 0.00%
0% 0.00%
0% 0.00%
Transactions
0.66
0.00 +: 0.00
0.00 0.00
0.08 0.00
0.09 :: 0,00
Price Identified
0.47 0.01
0,00 + 0.00
0.00 0.00
0.02 = 0.00
0.02 + 0.00
BUC Identified
1.60 0.01
0.00 + 0.00
0.00 0.00
0.05 +: 0.00
0.04 + 0.00
Table 3: Results navigation tasks and interaction tasks. We provide all prior knowledge given to MaestroMotif to two additional baselines, LLM Policy (Equivalent Prompting) and Motif (Equivalent Prompting). Results indicate that this additional information does not increase the performance. Learning how and when to leverage this information, from context, makes very challenging.
HRL approaches with code policies MaestroMotif is particularly related to approaches that combine code to define policies over skills and RL to learn low-level policies, such as concurrent hierarchical Q-learning (Marthi al. (005), policy sketches (Andreas et al. 2017). and program- guided agents (Sun et al. al., (2020). MaestroMotif employs LLMs as generators of reward functions, sermination/iiitiation functions, and policies over skills, significantly simplifying the interaction between humans and the Al system which is used in existing hierarchical RL. methods.
A.11 ADDITIONAL PROMPTING EXPERIMENTS
We further verify the hypothesis that the hierarchical structure of the MaestroMotif algorithm is key to obtain performance. In Table 3 we present two additional baselines. LIM Policy (equivalent prompting) based the LLM Policy baseline but its prompt contains all the information that used within the different prompts of MaestroMotif. This includes skill descriptions, high-level descriptions of the task and also the generated code by the policy-over-skills that is used within MaestroMotif. We also investigate Motif (equivalent prompting). which similarly builds on the Motif baseline but provides all the prior knowledge given to MaestroMotif. Despite giving significantly more information to both baselines, the performance does not improve. Although additional information provided, the burden on how and when to leverage this information, from context, makes it very challenging.
25

---

perahtpper_precondee Published as a conference paper at ICLR 2025
Code for train-time policy over skills
Nat
de:
taiit_(sel (self, mex_depth, branch_depth):
depth max_depth
self. wwwaccrnnoppteeeeeeeeeeeeeeeeeee branck_depth
#e15 exploreeeeeeeeell set
self.cirection "down' Start geing down
merchani_precoontiion (sel:)))
Placeholder for actual merchant precondition logic recurn
de korshipper porrhhiperr_pproooddt poeponditionn (sel
Placeholder for actual worshippar precondition logic recurn Palse
def (sel,,,, current_ dengeoa_depth,
merchant_prrecodiriin Perchippe__preconddite
if srcchant_rrccodd-tt
return merchant"
if www.hipper_preeeeeee precondition: return "orshiiper''
if current_skill =- 'discovezer':
if ael r.direction =ы 'down':
if duageon_depth sel5.max_dep.hr 'descander"
elae:
self.direction 'up' return ascender'
self.direation 'up': it dungeon_depti 1: return 'ascender'
seif.direction 'down' return 'descendes' clif cursent_skai skill 'daccender'; return discoverer'
eli. cursent_skill skill -- 'ascender'; return discoverer''
elae:
return discoverer''
def select_ake 11 laakkkllluggeooe www.ddunnsssaaaeo daoonndddddm_dd__e__ curren=_akill, dungeon_depthl
branch_number
cetiaaattrreoonttiioo worshipper_precondee
if dungeon_depth, =- branchhhhhehhhh
branch_number 2:
'sscander'
else:
seturn 'descender'
el1* Crannh__nmbba ss 2 and langeon_depch ss sel.. hallfaaranchdeptte retuzn "ascender'
else:
return seffsseeec__killl (halsectttilll cecurrent_skkll. dungeon_dapth,
merchant_ Becchant_rreoondtoo warshippar
select_aki anomish_minas iself, currens_akill, dungeon_dppte
branch_number,
vorshippar_preeondit.ooo
if branch_number 6:
if dangeon_depth self self.braach_dep:h:
raturn 'descender'
elff dungeon_depth
self. hitt.bbrnch_dedee
return 'ascender'
611* branch_nunber _umben
return self, selec_skkilllleeeeeeee (eureeliiiitrttettakkki. dungeon_depth,, prrcchent_precondiii porehippes.rreodditt.oo
self.select__kee seurrent_ekilll dungeon_depti.
Output 3: Code of the train-time policy over skills generated by MaestroMottif.
26

---

Published as a conference paper at ICLR 2025
Prompt for the policy over skills for Discovery Hunger
You are to write code which defines the method "select.skine skill" the NetHack Player class that selects amongst a aet of skills in the videogame of NetHack. The set of skills corresponds to ("discoverer", "descender", "ascender", "merchant", "worshipper").
when activated, the Discoverer fully explores the current dungeon, while fighting off enemies. The Descender makes its way to 8 staircase and goes down. The Ascender makes its way to staircase and goes up. The Merchant interacts with shopkeepars by selling its items. The Worshipper interacts with altars by identifying its itens.
Here is your current implementation of the NetHackPlayer class:
444 ccurrent.class_iigg itt..llssiilmmmmttttttionon
###
You are faced with the task following task.
Reach the oracle Level (the Delphi) in the Dungeons of Doom, but not before discovering ng the Gnomish Mines and eating some food there after getting hungry.
The Dungeons of Doom (branch number 0) constitute the main branch starting from depth 1. At branch.depth, second staircase appears opening the door towards the Gnomish Mines (branch number 2) branch.depthtl. Therefore, the Gnomish Mines constitute a second branch from branch.depth+l.
You should make use of the two existing methods. The sreach.gnomish.miness method brings the player the Gnomish Mines from the Dungeons of Doom. The "reach buancendungeoss.c of doom"" method brings the player to the Duageons of Doom from the Gnomish Mines.
Finding food happens by activating the Discoverer skill in the Gnomish Mines. Inmediately after finding the food, head fast possible for the Oracle level in the Dungeons of Doom (branch number 0), without more exploration.
Your code will be verified through this unit test.
444
funit.test} ###
Before writing the code, write few questions for yourself, and answer them to make sure you understand the assignment.
Identify if you need to define any additional attributes. If you define additiona. attributes, for each of them you must write methods that will simulate changes to their values. These methods must be called upon in the unit test (not in perform.task), as if the environment is updating them. The values should also be printed each iteration.
In your response make sure to include the code for the class definition and the unit test in python markdown.
Prompt 7: Description of the coding task for the LLM to code the policy over skills at deployment time when attempting to solve Discovery Hunger.
27

---

Published as a conference paper at ICLR 2025
Code policy for Discovery Hunger
cloas NotRackFloyer:
de
init inin_(aall max_depth, branch_depth):
sels.max_depth max_depth
sel bell.branhh_oee
branch_depth
seli.skilla ."discoverer", "descender", "sacendez", "merchant", "worshipper")
for down, -1 for self.caten_food False
self.direction
dell..xxllred__nnome
False
b.o]
select_shill curzent_skill, denteoa_depth,
sorahipper_preconnttiioo
if mezchani__peeconddiio
'eturn'merchent'
clif
return 'worshipper'
eli: curzent_skill =- 'diacovezer'
dangeon__epth self.max_depth ceneciino
return 'descender'
elif dungeon_depth and self.direction -1:
return 'aszender'
else:
self.ddreettoon if self.direction =- 1: return 'descendes'
elae:
return 'ascender'
clif cursent_ckill 'dascender';
return discoverer' elif
'ascender'; return discoverer'
merchant precondition, def marchant._recoodditio worsigger_preennitiin
cucrent_shill, dungeon_ depth, brabec,,
if dungeon depth belf.brrach_ddpth. return descender''
dungeon_deptl, self.branch_depth branch_numbe:
retuzn 'ascender'
clae:
(currren dangeon_éepth, merchant_preconddi sarchippar_prccocittte
de: reach_
current_akili, dungeos_depthy, branch_numbed
pazehigger_preccnditt if dungeon_depte depth == self.branck and branch_number == 0:
srrchant__recondiite
return
eli: dungeon_depth self. beht:bbranc_depth 1 and branch_numbe: 2:
return 'ascender'
elif cungeon_depth bil.bbrach_depts branch_numbe:
return neschant_preconditiin vorshigper_precondis.
still_sseelllerree (earrent__kkil dangeon_depth,,
clae:
return selff.elect_ee [eiect_skill] (ourrent_skill nangeon_depthl merchant_precondiiiinn vocehipper_preconciil
pertorm ourzeni_skill, dengeos_depth,, branch_miaber, merchant
merchant__precmmiition
wreeiiper_ppreccnditc.
eww.ooodd_gnnniihheee miinnsiiiiies if tranch_number =s 2:
selff.explordd_gnoml 'discoveres'
True
else:
hea..reecl_gnonishhuoo (eurrent_ski:l, dusgeon depth,
branch_number, srcchant_precodittiinn vozthigper_preecoditiio
not aelf.eaten_$oom
selffeaten_fon food True retuzn "dscovvree''
else:
1f Dranch_numberi 0: seturn branch_number, prrchant_preconditie aeahigger_preecoditiie
(ew.eeedeooreerree_____... dungeoa_ depth,
alif dungean_degp depth
return ddssoender"
else:
'discoveres'
Output 4: Example of code generated by MaestroMotif solve the Discovery Hunger compos- ite task.
28

---

Published as a conference paper at ICLR 2025
Termination function prompt
You implement the cerminatin methed of the NetHackPlayer class. This method determines when any of the skills should terminate.
Here is description of the skills. When activated, the Discoverer fully explores the current dungeor, while fighting off enemies. The Descender makes its way to a staircase and goes down. The Ascender makes its way to a staircase and goes up. The Merchant interacts
with shopkeepers by selling its items. The Worshipper interacts the any of preconditions of
with altars by identifying its items. the preconditions is true, the current skill should terminate no matter what.
the Merchant or Worshipper
def skill.cermination (self, skill, skill.time, current.depth, previcus.depth. preconditions)
Prompt 8: Prompt given to the LLM code generator for the generation of the termination function for each skill.
Initiation function prompt
You are implement the precondition. method of the NetHackFlayer class. This method determines when any of the skills can initiate.
Here is description of the skills. When activated, the Discoverer fully explores the current dungeor, while fighting off enemies. The Descender makes its way to staircase and goes down. The Ascender makes its way to a staircase and goes up. The Merchant interacts with shopkeepers by selling its items. The Worshipper interacts with altars by identitying its items. Define the preconditions only for the last two skills. Before writing the code, identify the entities that will be useful to identify: mention their ascií characters and their ascii encoding number. To correctly identify an entity, you also have to make use of the the aaarascci.coole that represents the color of the ascil character. Refer to color.map fetch the right color.
def skillprcconditionneell. char.ascliencodiing char num.items, color.map);
numpy array representing the ascii encoding of the characters surrounding the player # char asci colors a numpy array representing the colors of the characters surrounding the player num.items the number of itens the agents has
ihar.ascii.encodings
colormmap a map from common characters to their color
Prompt 9: Prompt given to the LLM code generator for the generation of the initiation function for each skill.
29


Reviews:
[{"review_id": "tIt55qI0zu", "reviewer": "Reviewer_KM3c", "summary": "The paper introduces a method called MaestroMotif for designing and using AI-assisted skills that enhance agent performance and adaptability, particularly for complex tasks in environments like NetHack. MaestroMotif leverages Large Language Models (LLMs) to translate natural language skill descriptions into reward functions, which are then used to generate code for skill initiation and termination and to create a policy over these skills. This allows the AI to perform specified behaviors through a combination of reinforcement learning (RL) and LLM-based code generation. \nThe paper evaluates MaestroMotif on the NetHack Learning Environment (NLE) and demonstrates its effectiveness in handling complex navigation, interaction, and composite tasks. MaestroMotif’s zero-shot approach, using skills without additional training, outperformed both LLM-only and reward-based RL baselines, highlighting the value of hierarchical skill composition and the LLM’s role in skill design.", "questions": "Could you please eloborate more about the concerns that I made on the Weakness sections? Especially, follwoing three.\n\n- Source of Performance Gains\n- Concerns about Generalizability\n- Need for Pseudocode", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Overall, I believe MaestroMotif presents an effective approach to incorporating human prior knowledge to solve specific tasks. Given that NetHack tasks involve a variety of challenges, including understanding natural language descriptions, planning over high-level abstractions, and exploration, it is notable that the authors demonstrated improved performance in the NetHack environment using their method.\nAdditionally, I find it interesting that the authors showed that learning skills simultaneously is essential for acquiring skills more efficiently, thanks to the emergence of curriculum-based learning.", "weaknesses": "- **Source of Performance Gains**\n\n  I wonder whether the performance gain primarily comes from the human domain knowledge or the method itself. It appears that the method relies heavily on human effort and domain knowledge to solve individual tasks. For instance, humans select a set of skills such as *Discoverer, Descender, Ascender, Merchant,* and *Worshipper,* and they even required to modify prompts for acquiring each of these skills. Therefore, when comparing this work with baselines, it’s difficult to determine whether the performance gain is due to the method itself or the expertise of human contributors. While the core of this work aims to incorporate human priors, the baseline algorithms should ideally leverage human priors at a comparable level.\n\n- **Concerns about Generalizability**\n\n  I'm not entirely convinced that the proposed five skills are general enough to tackle any downstream task in the NetHack environment, or if they are instead a highly targeted set of skills specifically suited for the benchmark tasks. To better demonstrate the method’s generalizability, one approach could be to train reusable skills across a broader range of environments, such as robotics tasks. For instance, training foundational skills on a robotic platform, like a humanoid or robotic arm, and performing zero-shot evaluations on new tasks that require combinations of these learned skills could provide stronger evidence of generalizability.\n\n- **Need for Pseudocode**\n        \n  It is challenging to understand the entire concept concretely without pseudocode for the full process, and some aspects remain unclear. For example:\n        \n    - If certain agent states satisfy instantiation conditions for more than one skill, how does the LLM prioritize among different skills?\n    - What happens if none of the skills satisfy the instantiation conditions?\n    - Does the training-time policy over skills adapt/evolve during training? I wonder because, as each individual skill is being changed along the training process, it might be helpful for a policy-over-skills adapts to the current set of learned skills.\n\n  I understand that the specific prompts in the Appendix may partially address these questions; however, I believe it would be beneficial to clearly state them in the main sections.\n\n- **Relationship to Related Work**\n     - It would be helpful if the relationship between this work and Motif (Kilssarov, 2024) were more explicitly stated.\n\n- **Minor Corrections**\n     - *Line 196:* \"crafts aploicy\" → \"crafts a policy\"\n     - *Line 239:* *Llama 3.1 70b* → *Llama 3.1 70B*\n     - *Figure 8(c):* It is difficult to compare the performance of the two different approaches, as they are visualized in separate plots.\n     - *Appendix A.2 and A.3* : title of these subsections are same\n     - *Line 269:* What negative impact? Additional clarifications would be helpful for better understanding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBA9DZl4eG", "reviewer": "Reviewer_ygS6", "summary": "The paper introduces MaestroMotif, an extension of the recent Motif framework that accepts datasets of unlabeled demonstrations to first learn a set of labeled options, then query an LLM to write Python programs that trigger these learned options to satisfy an arbitrary objective in the given domain. MaestroMotif is compared to Motif, ordinary PPO, and expert behavior-cloning to see how well each method can satisfy arbitrary unseen objectives, and the results indicate that MaestroMotif is better than these baselines at solving long-horizon, complex requests (e.g., \"go to area X, obtain item Y, wait until condition Z is met and then take action A\").\n\nThe authors also present an ablation of MaestroMotif for different approaches to learning options/skills for the framework, showing the importance of learning their skills jointly and without separate policy heads. The authors also look at the scaling performance of MaestroMotif, showing that smaller LLMs produce predictably smaller returns on the complex tasks.\n\nI thank the authors for their thorough rebuttal and for their comments on the work. After reviewing the rebuttal, I have updated my score to 8 and advocate for acceptance of the work.", "questions": "* Are the baselines in any way capable of accepting goals, or is their failure to generalize to new tasks fairly expected behavior?\n* What is the cost of running MaestroMotif for a single task (i.e., how much time and memory is required, or what type of hardware is needed?)\n* How would MaestroMotif be applied to a new domain? My interpretation is: 1) identify necessary core skills (ask an expert), 2) gather a large unlabeled dataset of gameplay, 3) label that dataset with preferences for each task, 4) learn options over the data, 5) prompt an LLM to combine the learned options. Have I missed something?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* MaestroMotif fits in well with the presented prior work and literature, and is a logical extension of much of the \"code-as-policies\" literature from LLMs, combining human demonstrations and data with AI planning and code-synthesis.\n* The baselines used (LLM as policy, PPO, etc.) are sensible baselines and suitable cover the different methods that one might employ to attempt to learn arbitrary NetHack policies for zero-shot transfer to new skills or tasks.\n* The paper presents an approach that combines large labeled datasets, human expertise, and LLM code synthesis/planning into one framework, neatly exemplifying human-AI collaboration for task completion with LLMs.\n* The full details for reproducibility are in the appendix (importance of unit tests, specific prompts, re-prompting strategies, etc.), which is a useful resource for future work.", "weaknesses": "* Learning the options networks seems to require intense manual effort, as a human must label different states with preferences/task-alignment to learn different options.\n* While MaestroMotif outperforms the baselines for arbitrary tasks, it does seem to be pitted against methods that have no real way of generalizing to such tasks (for example, are the score-maximization approaches capable of receiving sub-goals or goals in any way?). The success of MaestroMotif is noteworthy, but the delta over prior work is less impressive given the prior work is, at times, apparently severely hamstrung.\n* The overall framework relies on significant prompting/re-prompting of a 405b model, which could quickly become prohibitively expensive. It would be nice to see results or metrics on the cost (in time, memory, or money) of running MaestroMotif for each of the specified tasks, or for generalization to a new task or domain.\n* Much of this paper is tailored to NetHack, so the line between \"core requirement/contribution of MaestroMotif\" and \"tweak made to adapt to NetHack\" is blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nt57c0IFLb", "reviewer": "Reviewer_QXfj", "summary": "This work presents a system for learning skills that humans describe in language. The system, called MaestroMotif, (1) trains per-skill reward functions using an LLM annotator of interaction datasets, (2) generates code describing the initiation and termination functions of skills, (3) generates code describing how to choose between skills during \"training time\", and (4) trains a skill-conditioned RL policy based on PPO. There are multiple experiments on the complex NetHack environment, where the proposed system is significantly stronger than other baselines.", "questions": "- MaestroMotif performs very well relative to all baselines, but still performs poorly in settings like Minetown, where only a 7.2% success rate is achieved. What are the bottlenecks to better performance? I.e. is the RL policy unable to complete the skills in this setting or is the generated \"skill chooser\" code bad for some tasks?\n- What architecture is used for the reward functions? Prior RLHF literature initialized RMs from smaller LLMs, but it would be interesting to see if different sized RMs would lead to different results from RL training.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The experiments are comprehensive, covering most baselines one would expect and extensive ablations (like comparing skills learned in isolation vs simultaneously, LLM scale, and the architecture). Furthermore, the results clearly demonstrate the effectiveness of MaestroMotif.\n- To me, the results of this paper mark a big improvement in the domain of hierarchical RL, where LLM-generated code is used to conduct high-level planning while a small RL network controls low-level actions.\n- The proposed technique is quite general, being applicable across many domains outside of NetHack.\n- The writing is very clear and the presentation is excellent overall.", "weaknesses": "- The technique only trains reward functions on an existing offline dataset, which may be out of distribution with respect to the distribution of states encountered by the trained RL agent. This choice is borrowed from the original Motif paper (which said that the reward model was not fine-tuned during RL training due to simplicity), but based on the poor performance of the RL baselines (used to collect the original dataset) it seems like there may be a greater distribution shift in this work. Results determining whether the trained reward model generalizes to the online distribution would be appreciated (i.e. checking how much the annotator agrees with the reward model on trajectories generated by the final policy).\n- The technique relies heavily on the LLM's ability to generate code. However, even with unit tests and code refinement this seems to be difficult for many LLMs according to Figure 9. It is unclear from the paper whether the 405B model has achieved close to \"perfect\" performance or if there are still key issues with the generated code. Further analysis of the failure modes of the LLM coders would be useful for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8t27CYu3H", "reviewer": "Reviewer_BKw5", "summary": "The paper presents MaestroMotif, a novel method for AI-assisted skill design in RL that enables zero-shot adaptation to log-horizon tasks in NetHack. The key idea is using LLMs to map human-generated description of skills and high level policies to corresponding to skill specific reward functions, initiation and termination functions on the first hand, and high-level code-based policies on the other hand. \nThe method is evaluated on NetHack, where it significantly outperforms existing approaches on a series of tasks.\n\nOverall I really like the approach and I think this paper provides an interesting contribution to the field. This said, there are several missing information and several points I'd like to see discussed in the paper (see weaknesses and questions fields).\n\nI'm going to give a rating of 6 for now and I'm willing to increase it if the authors address my concerns below.\n\nUpdate: I increased the score to an 8 after the rebuttal (see discussion below)", "questions": "These are questions and comments\n\nExperiments comparisons:\n* MaestroMotif is first training the skills, which costs GPU time too. So we should ideally compare this training resources and sample efficiency to the one of training a policy for the final downstream task. For fair comparisons, the \"RL from scratch\" approaches should be given the same training time and environment interactions as is used for MaestroMotif pretraining; is that the case?\n* I do get the argument that when targeting a new task, there is no additional cost for MaestroMotif and there is one for the \"RL from scratch\" approach, but this also assumes that the test task can be decomposed with the existing skills, which is conveniently the case here, but might not be in general.\n* Is the performance reported for task-specific training the training performance (with exploration noise), or performance measured outside of training with no exploration noise? In some environments where one mistake can be costly, using training performance with exploration noise would under-estimate true performance (eg crossing a bridge requires walking straight but exploration noise could throw you off). \n\nOnline skill learning\n* The approach here assumes access to a list of skill descriptions that when combined, allow to achieve any testing task, this is mentioned in the discussion. \n* This is an important assumption and I would like to see a discussion of how to move past it. \n  * Can we imagine the LLM generating candidate skill descriptions instead of using human-generated ones, and learn these? How good would that be? Could that only work if conditioned on a set of test tasks, or could an LLM intuitively figure out a good set of skills that can easily be combined to solve many downstream tasks?\n  * What happens if a test task requires skills that were not pre-trained? The current architecture cannot recover from that, but could we imagine letting the LLM generate a missing skill description, pre-training a new skill and then adapting to that OOD task? Would that work?\n* Providing feedback about how to best execute the skills in the training phase seems rather unnatural for a human user, could we imagine a way of automatic this process?\n* Maybe we could learn a high-level transition model: which termination state distribution leads to which initiation state distribution, ie a graph model of skill chaining. Once we have that we could generate paths in that graph optimizing for some intrinsic metrics (active learning)? This paper does something a bit in that vein: https://arxiv.org/pdf/2402.15025 \n\nData annotation to synthesize reward functions: \n* \"we independently annotate pairs of observations collected by a Motif baseline\" → this sentence sounds like the authors did some of the annotation, is that the case?\n* The fact that we need a dataset of expert / exploratory trajectories to train the reward functions so we can explore better seems like a limitation of this approach that should be discussed. How could the reward functions be synthesized without such data?\n* How many trajectories are required to enable the training of the reward functions? What are the requirements in terms of data quality / diversity?\n\nRobustness of the approach:\n* The high-level policy is generated zero-shot (with couple of refinements), but what happens if it fails? Could the authors discuss how one could train this high-level policy? Maybe the algorithm sometimes needs to go back to the pretraining phase and add a new skill or refine a skill, maybe it needs to try another high-level strategy, how could it detect what's wrong and improve?\n* There is no discussion about the impact of human-generated feedback on the success of the approach. How much time and effort has been put in the design of this feedback? Would it work with feedback generated by non-AI researchers NetHack experts? Here the skill descriptions seem rather finetuned. \n\nResults:\n* Why is the Reward Information baseline performing worse than the Motif task-specific training? Doesn't it have access to the true task reward? What are the reward functions Motif generates here and how do these explain the higher performance?\n\nLinks with the planning literature\n* The paper makes interesting connections with the option framework but doesn't mention connections with the planning literature, which I feel might be just as relevant. I especially think of PDDL approaches and task and motion planning (hierarchical planning)\n* MaestroMotif indeed resembles approaches using LLM to generate PDDL domain descriptions: where skills are operators with initial and terminal functions, and the LLM is used to generate the high-level policy (bypassing planning) conditioned on code-based conditioned (equivalent to PDDL predicates), eg https://arxiv.org/pdf/2305.11014\n* Chaining issues: although this does not seem to occur in the experiments, pretrained skills can often have the problem where the distribution of final state of an operator (/option), is different from the initiation states of the skill we may want to chain it with, such that the next skills stochastically can or cannot be applied. How would your approach handle this? Would it require coming back to the definition of termination function to have them be more constrained?\n\nTypo: implemenetation (L508)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and the approach is well motivated. \n\nThe approach allows to construct high-performing policies for complicated, long-horizon tasks with high-level human inputs, which is quite impressive. \n\nMany connections with the option framework, makes the paper easy to parse and understand.\n\nI liked the experiments comparing different methods for skill learning and the emphasis on transfer learning across different skills learned in the same network. \n\nMethods are rather clear and the appendix contains a lot of important information.", "weaknesses": "There is not much discussion about how robust the system is to the quality of human feedback, and my guess is that the system could be quite brittle.\n\nThe reliance on pre-collected expert / exploratory trajectories to generate the skill-specific reward function is a limitation of the approach that is not discussed.\n\nResults are limited to a single environment, and the authors do not discuss how this approach would work in other environments, where it might be useful or not, what the cost of generating this human feedback is in the first place, etc.\n\nThe authors did not specify whether the code will be released publicly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)