Informing Reinforcement Learning Agents by Grounding Language to Markov Decision Processes

Authors: Benjamin Adin Spiegel, Ziyi Yang, William Jurayj, Ben Bachmann, Stefanie Tellex, George Konidaris
Primary area: reinforcement learning
Keywords: Language Grounding, RLang, RL, Formal Language, LLM

Abstract:
While significant efforts have been made to leverage natural language to accelerate reinforcement learning, utilizing diverse forms of language efficiently remains unsolved. Existing methods focus on mapping natural language to individual elements of MDPs such as reward functions or policies, but such approaches limit the scope of language they consider to make such mappings possible. We present an approach for leveraging general language advice by translating sentences to a grounded formal language for expressing information about *every* element of an MDP and its solution including policies, plans, reward functions, and transition functions. We also introduce a new model-based reinforcement learning algorithm, RLang-Dyna-Q, capable of leveraging all such advice, and demonstrate in two sets of experiments that grounding language to every element of an MDP leads to significant performance gains.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729892096636

thread_type:
official_review

<thread_object>
creator: reviewer_8geu

<content>
summary:
The paper proposed RLang-Dyna-Q, which can ground any language advice to all components in MDP, compared to grounding only to individual components before. The solution uses in-context learning to first select the grounding type, then translate the advice to RLang program. The enhancement outperforms prior approaches.

strengths:
1. The algorithm automates language-to-MDP component translation, and streamlines the process of learning human decision-making for robots
2. The authors conducted extensive experiments and described the algorithm clearly

weaknesses:
1. In-context learning limits the capability enhancement of the language model. It might be better if we could make the LM trainable and train the language model and the RL system end-to-end
2. Human language might not be expressive enough to be translated to RLang. In the experiment section, it stated that some advice cannot be converted to RLang. Could we have a more natural intermediate representation for the advice and agent?

questions:
1. How to go beyond in-context learning?
2. How to handle the inexpressiveness of human language for RLang?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729993335145

thread_type:
official_review

<thread_object>
creator: reviewer_3iuj

<content>
summary:
This paper studies the problem of leveraging natural language advice for training effective/efficient RL agents. The authors hypothesize that some natural language advice is more suitable to be grounded in reward function while others are better captured in transition functions. They further suggest that leveraging natural language advice by grounding them in a formal language (RLang) that is capable of describing all aspects of the system, is better than advising a subset of the system. 

The authors adapt Dyna-Q to work with grounded advice in RLang. They evaluate this method in Minigrid and VirtualHome with mostly positive results to support their hypothesis.

strengths:
1. I appreciate the range of experiments included, as well as a comparison with SayCan in the appendix.
2. I also enjoy reading the section on user studies, and section 4.3 on automated semantic labeling for disambiguation advice. In general, I agree that gradually removing the need for expert annotations is important, let it be dense natural language advice, crafted vocabulary, or RLang grounding files in this case.
3. This is an important research topic and the contribution is contextualized nicely.
4. Most of the paper is quite clear. A few improvements can be made - see weakness 3.

weaknesses:
1. Expert advice seems much more dense, verbose, and low-level (almost program-like) than non-expert advice. It is not completely surprising to me that LLMs can ground them to predefined symbols that are approximately defined on a similar level of abstraction.
2. It might help to have a paragraph discussing results and how advice on effect/policy/plan each contributes to the combined policy. Are they the same? Is it task-dependent? I think this can help better justify that an approach to encode "information about every element of an MDP" is necessary.

(The two concerns above are why I gave 2 for contribution and not higher. Would be happy to improve scores if they are addressed)

3. Stylistic nit-picking: could you please increase the font size in Figure 1, and reward curves in Figure 2-6? The text in Figure 7 looks much better. Perhaps capitalize "Perception" in the title in the left figure for consistency. Consistent legend colors and orders for different methods on page 8 would improve comparability across figures.
4. Broken reference on line 164.

questions:
1. How was the expert advice (textboxes on page 8) collected for the main experiments (who are the experts, how are they trained, what's the protocol)?
2. Do you have ablation studies for HardMaze in Figure 4?
3. Why is RLang-Dyna-Q-combined worse than RLang-Dyna-Q-plan curve in Figure 6?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730131125307

thread_type:
official_review

<thread_object>
creator: reviewer_ebyn

<content>
summary:
This pager introduces RLang-Dyna-Q, an extension of prior work, RLang, that transforms natural language into a formally specified language for solving tasks. Rather than focusing on policy-centric translations, as in much prior work, the authors observe that much of the advice or coaching offered by human experts will come in the form of statements that describe a transition function (e.g., "If you miss, you will fall down"), a reward function (e.g., "You get 10 points when you touch the key"), or a plan (e.g. "Go to X, then pick up Y, then return to Z"). RLang-Dyna-Q is a combination of Dyna-Q and RLang that uses the learned world model/transition functions of RLang to further refine the Q function learned in Dyna-Q. 

The proposed RLang-Dyna-Q algorithm is compared to Dyna-Q and to a random policy in a handful of tabular RL domains, showing that it significantly outperforms Dyna-Q. The authors also perform an ablation study in which they test only subsets of RLang-Dyna-Q (only policy advice, only transition advice, or only plan advice).  Finally, the authors conduct a small user study in which 10 undergraduate students provide advice to warm start an RLang-Dyna-Q, with each student contributing one piece of advice, and 5/10 pieces of advice leading to policy improvements over the baseline, un-advised policy.


After reviewing the rebuttal, the authors have clarified the scope of the paper and their intended contributions and research area. Given the research direction and goals are more in line with language grounding and leveraging language for tasks, rather than improving task-learning performance or RL/IL efficiency, I will raise my contribution score to 2 and my overall score to 5.

strengths:
* The paper is well written and provides a clear overview of the motivation, problem setting, and proposed solution.
* The paper proposes a blend of conventional planning literature and formal specification with the advancement of LLMs, leading to a significant improvement over conventional tabular RL solutions.
* The authors conduct a small scale user study, which solicits and leverages advice from untrained human coaches for a planning task.

weaknesses:
* The method is not entirely clear, particularly given the heavy reliance on prior work (RLang) in this paper. It is not clear how the Q table relates to the RLang vocabulary files or RLang declarations, and this information must be obtained by referring to and reading prior work (meaning that RLang-Dyna-Q does not entirely stand on its own, but feels more like the "second half" of the RLang work).
* The results for RLang-Dyna-Q are not very convincing, and the comparison to a method that is nearly three decades old is also not very convincing. Comparisons to more modern RL baselines would improve the work. In particular, comparing to an LLM that generates Python/programming solutions seems like a very important baseline (even if there is no RL refinement, it would be useful to simply see to what extent an advanced LLM can solve these tabular domains out-of-the-box).
* The advice required to make RLang-Dyna-Q actually improve over baselines seems very particular. For example, looking at the advice in Figures 3-6, there is a combination of plans, general advice, and state transition advice. There is not a discussion or written analysis on what types of advice work best, or why. Similarly, the success of different types of advice seems extremely finicky. Comparing advice from participants 5 and 10 in the user study, the written advice is nearly identical. However, the performance deltas are quite significant (from a 33% increase to just a 2% increase).

questions:
* Why not compare to conventional RL methods (e.g., PPO with a small neural network), to RLang itself, or to LLMs that generate code for plans? 
* Why cut training off at 30-75 episodes, which is quite a small budget given that these are not expensive or safety-critical domains? It seems that one argument for RLang-Dyna-Q is that it could be is significantly more efficient than modern RL baselines by leveraging human advice, but if so then this should be shown by empirical comparisons (e.g., how many episodes does each method require to achieve maximum returns?).
* What differentiates good vs. bad advice for RLang-Dyna-Q? The user study provides great insight into the effects of different natural language prompts for the method. However, at times the prompts appear semantically identical, but they yield different results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731256932176

thread_type:
official_review

<thread_object>
creator: reviewer_q6ru

<content>
summary:
The paper proposes a framework to leverage natural language-based advice to accelerate RL learning process. The RLang-Dyna-Q algorithm extends the original RLang framework and combines it with the traditional Dyna-Q. Empirical results over two sets of experiments help verify the effectiveness of propose algorithms.

strengths:
1. The writing is overall good and easy to follow.
2. The idea of translating natural language advice to RLang and using RLang to generate synthetic transitions makes sense.
3. The writing flow in the experiment is great – sec 4.1 and 4.2 present two effective cases with assumptions on semantically-meaningful labels, while sec 4.3 also presents efforts to try to address this assumption. Also, user study has been completed in Table 2.

weaknesses:
1. Only Q-learning-based RL is tested in the experiment. More advanced and modern RL algorithms are needed to show the generality, e.g. PPO.
2. More LLM + RL baselines are needed. There are a few simple alternatives to directly leverage LLM to process natural language advice to help RL training. For example, what if we don’t use any RLang-based program, and only treat LLM’s as the generator for actions and transitions?
3. Another important assumption (and limitation) in the paper is that each environment will be provided with human-annotated natural language advice. This is a strong prior compared with all RL baselines. The author needs to discuss more about this assumption and whether we can use any other ways to bypass the need for human labels. For example, could LLMs directly generate advice for any given environment?
4. More qualitative results are needed for section 4.3 (a demo is not enough)

questions:
1. Any idea why the DynaQ baseline doesn’t work in Figure 6’s experiment?
2. Typo in line 164.
3. If we are going to extend the algorithm to high-dimensional continuous RL problem, what could be the biggest challenges?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733198480574

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Review and Rebuttal Process for ACs and SACs

comment:
Some in the community have suggested that a concise summary of reviews and rebuttals would assist reviewers, ACs, and SACs during the next part of the review process. We have tried our best to summarize the main strengths and weaknesses mentioned in our reviews, as well as our responses to them.

In terms of readability, all of the reviewers remarked that the paper was well-written, easy to follow, well-contextualized, and well-motivated, with one praising the writing flow in the experiments section in particular (reviewer q6RU, score: 5) and another stating, “This is an important research topic and the contribution is contextualized nicely” (reviewer 3iuj, score: 6). Some of the reviewers were impressed with the quantity, variety, and quality of experiments performed (reviewer 3iuj, score: 6), with one reviewer highlighting our “extensive experiments” (reviewer 8geU, score: 6), which included experiments done across two sets of environments (6 total environments with our agent, multiple ablations, and a baseline), a user study, a two-tiered symbol-grounding demo using a VLM, and an additional experiment comparing directly to an additional baseline (SayCan). Others were unimpressed by the experiments, with one citing DynaQ as a poor point of comparison that is “nearly three decades old” and suggesting a comparison to a deep-learning based agent and an LLM without any RL (reviewer EByn, score: 3). Our work, however, is precisely on how to integrate language into RL, and our language-informed RL agent called RLang-DynaQ is based on DynaQ, which, due to its simplicity, is a prime candidate for a clear and minimal example showing how language can impact performance in classical model-based RL. Another stated that our VLM demonstration—an addendum demo designed to show that a VLM can replace the need for human-generated labels by using a VLM to label objects directly—needed more quantitative results, “a demo is not enough” (reviewer q6RU, score: 5). We believe a demo is sufficient to show this relatively straightforward image-labeling capability.

Overall we found the reviews to be very constructive, and made a substantial number of updates to the paper based on helpful feedback. This included an additional full paragraph elucidating the methodology (suggested by reviewer EByn), another full paragraph dissecting the impact of various kinds of advice on agent performance (suggested by reviewers EByn and 3iuj), a sentence elaborating on how advice was collected in our user study (suggested by reviewer 3iuj), and we increased the font size on all reward plots (suggested by reviewer 3iuj). All reviewers responded positively to our updates, with reviewer 3iuj increasing their contribution from fair (2) to good (3). While most reviewers responded to our rebuttals, the author of our paper’s most critical review (EByn, score: 3), whose review we responded to in detail on November 12th (the day that reviews were released), has not yet responded to our rebuttal at the time of this comment. We note that the average score reported for all reviewers who engaged with us in the rebuttal process is (6+6+5)/3 = 5.666.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces RLang-Dyna-Q, an extension of the RLang framework, which grounds natural language advice in a formal language (RLang) and integrates it into the Dyna-Q algorithm to improve reinforcement learning (RL). The method leverages human-provided language to enhance the learning of RL agents, particularly in transition and reward functions. Empirical results show improvements in RL tasks using natural language advice from human experts.

Reasons to accept
- The idea of grounding natural language advice to improve RL performance is novel and well-motivated.
- The paper presents multiple experiments across domains like Minigrid/BabyAI and VirtualHome, demonstrating the benefits of the proposed approach.
- The inclusion of a user study, where human participants provide advice, offers valuable qualitative insights into how different types of language inputs influence performance.
- The method works towards reducing the reliance on dense human annotations, which shows potential for real-world application.

Reasons to reject
- The experiments primarily focus on Q-learning, which is considered somewhat outdated. Comparisons to more advanced and widely used algorithms, such as SAC and PPO, are missing.
- Performance improvements are tied to specific types of advice, and it is not clear which types of advice are most effective or how to generalize them across tasks.
- A major limitation is the assumption that high-quality advice must be provided by human experts.
- The reliance on in-context learning for language grounding may restrict the model's ability to adapt and refine its understanding. Making the language model trainable alongside the RL agent could yield better results.
- The method's connection to prior work (RLang) is not fully clear, and the paper requires readers to reference previous work for a complete understanding. The algorithm is presented as an extension rather than a standalone solution.

While this paper studies a promising research direction and presents an interesting approach, I believe its weaknesses outweigh its strengths. Consequently, I recommend rejecting the paper.