Kati8/ICLR_2023_2025 huggingface dataset:
authors: Martin Klissarov
authors: Mikael Henaff
authors: Roberta Raileanu
authors: Shagun Sodhani
authors: Pascal Vincent
authors: Amy Zhang
authors: Pierre-Luc Bacon
authors: Marlos C. Machado
authors: Pierluca D'Oro
reviews: 
Comments: The paper presents MaestroMotif, a novel method for AI-assisted skill design in RL that enables zero-shot adaptation to log-horizon tasks in NetHack. The key idea is using LLMs to map human-generated description of skills and high level policies to corresponding to skill specific reward functions, initiation and termination functions on the first hand, and high-level code-based policies on the other hand. 
The method is evaluated on NetHack, where it significantly outperforms existing approaches on a series of tasks.

Overall I really like the approach and I think this paper provides an interesting contribution to the field. This said, there are several missing information and several points I'd like to see discussed in the paper (see weaknesses and questions fields).

I'm going to give a rating of 6 for now and I'm willing to increase it if the authors address my concerns below.

Update: I increased the score to an 8 after the rebuttal (see discussion below)
Strengths: The paper is well written and the approach is well motivated. 

The approach allows to construct high-performing policies for complicated, long-horizon tasks with high-level human inputs, which is quite impressive. 

Many connections with the option framework, makes the paper easy to parse and understand.

I liked the experiments comparing different methods for skill learning and the emphasis on transfer learning across different skills learned in the same network. 

Methods are rather clear and the appendix contains a lot of important information.
Weaknesses: There is not much discussion about how robust the system is to the quality of human feedback, and my guess is that the system could be quite brittle.

The reliance on pre-collected expert / exploratory trajectories to generate the skill-specific reward function is a limitation of the approach that is not discussed.

Results are limited to a single environment, and the authors do not discuss how this approach would work in other environments, where it might be useful or not, what the cost of generating this human feedback is in the first place, etc.

The authors did not specify whether the code will be released publicly.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This work presents a system for learning skills that humans describe in language. The system, called MaestroMotif, (1) trains per-skill reward functions using an LLM annotator of interaction datasets, (2) generates code describing the initiation and termination functions of skills, (3) generates code describing how to choose between skills during "training time", and (4) trains a skill-conditioned RL policy based on PPO. There are multiple experiments on the complex NetHack environment, where the proposed system is significantly stronger than other baselines.
Strengths: - The experiments are comprehensive, covering most baselines one would expect and extensive ablations (like comparing skills learned in isolation vs simultaneously, LLM scale, and the architecture). Furthermore, the results clearly demonstrate the effectiveness of MaestroMotif.
- To me, the results of this paper mark a big improvement in the domain of hierarchical RL, where LLM-generated code is used to conduct high-level planning while a small RL network controls low-level actions.
- The proposed technique is quite general, being applicable across many domains outside of NetHack.
- The writing is very clear and the presentation is excellent overall.
Weaknesses: - The technique only trains reward functions on an existing offline dataset, which may be out of distribution with respect to the distribution of states encountered by the trained RL agent. This choice is borrowed from the original Motif paper (which said that the reward model was not fine-tuned during RL training due to simplicity), but based on the poor performance of the RL baselines (used to collect the original dataset) it seems like there may be a greater distribution shift in this work. Results determining whether the trained reward model generalizes to the online distribution would be appreciated (i.e. checking how much the annotator agrees with the reward model on trajectories generated by the final policy).
- The technique relies heavily on the LLM's ability to generate code. However, even with unit tests and code refinement this seems to be difficult for many LLMs according to Figure 9. It is unclear from the paper whether the 405B model has achieved close to "perfect" performance or if there are still key issues with the generated code. Further analysis of the failure modes of the LLM coders would be useful for readers.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 4