Policy Learning with a Language Bottleneck
Authors: Megha Srivastava
Cédric Colas
Dorsa Sadigh
Jacob Andreas
Venue: Accepted by TMLR ()

Abstract: Modern AI systems such as self-driving cars and game-playing agents achieve superhuman
performance. But they often lack human-like generalization, interpretability, and inter-
operability with human users. This paper introduces *Policy Learning with a Language
Bottleneck* (PLLB), a framework enabling AI agents to generate linguistic rules that capture
the high-level strategies underlying rewarding behaviors. PLLB alternates between a *rule
generation* step guided by language models, and an *update* step where agents learn new
policies guided by rules. Crucially, PLLB enables this kind of language-guided learning
even when a natural language rule is insufficient to completely describe the target policy.
Across five diverse tasks, including a two-player signaling game, maze navigation, image
reconstruction, and robot grasp planning, we show that PLLB learns more interpretable
and generalizable behaviors than standard policy learning methods. In three additional
human subject studies, we show that show the learned rules significantly improve human
task performance, enabling more effective human-AI coordination

Published in Transactions on Machine Learning Research (02/2026)
Policy Learning with a Language Bottleneck
Megha Srivastava
megha&@cs.stannordd edu
Stanford University
Cédric Colas
ccolas @mil edu
Massachusetts Institute of Technology
Inria
Dorsa Sadigh
dorsa wwt.sa....aao..ooccccc...oo edu
Stanford University
Jacob Andreas
jda edu
Massachusetts Institute of Technology
Reviewed on OpenReview: https: oppenevveee. net/ forum? id= SKBuEqaQPU
Abstract
Modern AI systems such as sclf-driving cars and game-playing agents achieve superhuman performance. But they often lack human-like generalization, interpretability, and inter- operability with human users. This paper introduces Policy Learning with a Language Bottleneck (PLLB), a framework enabling Al agents generate linguistic rules that capture the high-lovel strategies underlying rewarding behaviors. PLLB alternates between rule generalion step guided by language models, and updale step where agents learn new policies guided by rules. Crucially, PLLB enables this kind of language-guided learning even when a natural language rule is insufficient to completely describe the target policy. Across five diverse tasks, including a two-player signaling game, maze navigation, image reconstruction, and robot grasp planning, we show that PI.LB learns more interpretable and generalizable behaviors than standard policy learning methods. In three additional human subject studies, show that show the learned rules significantly improve humman task performance, enabling more effective human-Al Al coordination
Select high /low reward examples
FAMC REAH
Ieeaatio
Lx I Blue,
update:
gen gen_rule rule:
explore and learn new policy regularized by L,
describe successful strategy with LM
go Right or Left
Iteration
L2 : should always
go Right on Blue.
Figure 1: Policy Learning with Language Bottleneck (PLLB) alternates between two steps: 1) gen_rwr rule generates linguistic rule L; explaining the agent's best behaviors by prompting a language model with contrastive (positive and negative) episodes; 2) update learns policy conditioned L. PLLB strengthens human-AT coordination by constraining policies to be more interpretable and generalize better.
We provide source code experiments http::///shhha..ccc/mmaaayaaaaee

---

Published in Transactions on Machine Learning Research (02/2026)
1 Introduction
As AT systems play increasingly central role in automation and decision-making, their success depends not only on their performance, but also on their ability to model and aligu with human behavior. To be truly effective assistants, these systems must act and generalize in ways that align with human expectations. However, many of today's AT systems do not meet these standards. Self-driving cars or game-playing agents like AlphaZero may achieve super-human performance, but they lack interpretability {McIlroy- Young ct al. 2020) and often act unpredictably, especially outside their training distrrbuttin (Wang el al. 2023).
Unlike Al systems that are trained in isolation, humans acquire most of their skills and knowledge from interacting with others, often using language- via instructions, advice, explanations that improve their decision-making capabilities (Carruthers & Boucher 1998 Mesoudì & Thornton 2018). Language acts as a communicative medium, enabling us to teach, learn from, and coordinate with others to solve complex problems. It also supports other cogriliie functions: even when not used for communication, it allows us to represent abstract concepts (Hesse) 1988
Lakoff & Johnson 2008). and plan (Vygotsky) 1965 Clark 1998);
it guides our attention (Waxman 1994; Yoshida & Smith, 2003). and prompts relational thinking (Gentner & Loewenstein ).02).
Consider driver learning to navigate novel social conventions (e.g., triangle-shaped stop signs). While adapting to the environment, they might verbalize strategies to themselves to avoid future mistakes (e.g., If the sign is trianguarr, I should slop, a cognitive use), or transmit this convention to others (e.g., telling a friend In Japan, slop signs are triongles, a communicative use). Representing learned information in language helps humans solve problems and transmit knowledge by effectively capturing abstract problem structures that facilitate learning and generalization (Boutonnet & Lupyan 2015 Chopra et al.] 2019 Tessler et al., 2021). Importantly, language remains useful even when it cannol fully encapsulate an entire stralegy (e.g.- drivers' reflexive actions, or fine- ane-grained driving mechanics).
There exist many recent examples in the litcrature of AI systems leveraging language- based representations or linguistic leedback, but these often rely on external supervision: humans in the loop or hard-coded feedback functions (Luketina et al. 2019 Colas et al., 2022 We argue that more human-like AI systems should not only use banguage-based supervision but also generate their own language-bat based feedback to leverage both the communicative and cognitive functions of language. Importantly, this capability should also extend to tasks that require updating an underlying policy that is only partially expressible in natural language, such as low-level control or reflexive actions.
This paper introduces Policy Learning with Language Bottleneck (PLLB), framework that provides artificial embodied agents the ability to generate linguistic rules that capture the strategies underlying their most rewarding behaviors. As shown in Figure [ PLLB alternates between a rule generation step that explains the agent's experiences by prompting a language model (LM) with contrastive episodes, and a policy update step that learns a new policy guided by these rules. Unlike past work that solely leverages LMs for modeling agent behavior and multistep reasoning (Park et al., 2023 Wei et al., 2023 PLLB is applicable even when aspects of the target policy cannot be expressed with language.
Policy Learning with a Language Bottleneck can be applied to a wide range of agent types, from RL policies to LLM-based learners to robot pose estimators, and the core mechanism of PLLB remains the same: using contrastive reward signals to extract linguistic rules via an LLM, and then conditioning policy updates on these rules depending on the type of learning agent. We investigate the role of PLLB in shaping more human-like policies across five distinct tasks. They perform better: in two image reconstruction tasks, PLLB agents generate instructions increasing the listeners' performance compared to non-linguistic baselines (Section 7 and PLLB agents also help more efficiently learn robot grasping policies (Section g They improve few-shot generalization: in a maze task, PLLB rules uncover abstract problem structure that improve learning similar mazes (Section 6 and PLLB also reduces reliance on non-generalizable visual features in robotic manipulation (Section R They are more interpretable: in a coordination task, agents converge on humans' preferred policy when multiple optimal policics exist (Soction 5 They are more inter-operable: in maze and image reconstruction tasks, humans achieve better rewards when interacting with PLLB agents compared to agents trained without a bottleneck (Sections 6 and 7
2

---

Published in Transactions on Machine Learning Research (02/2026)
2 Background & Related Work
PLLB is inspired by the dual use of language as both a communicative and cognitive tool for docision-making in both humans (Carruthers & Boucher 1998) and machines (Colas et al. 2022
Language for communication. Language facilitates cooperation and coordination between humans and machines via instructions (Hermann et al. 2017 Chevalier-
et al. 2019p, advice (Watkins et al. 2021 explanations (Zhong et al. 2020 Lampinen et al. 2022), or the formation of conventions (Hawkins ct al. 2020 Hu & Sadigh 2023) Such communicative functions increase the fidelity and breadth of cultural a process of social learning that underlies hurman ecological success (Mesoudi & Thornton PLLB agents not only learn from language, but also generate their own to be shared with others.
transmission
2018).
Language for cognition. Language also augments learner's cognitive abilities. Language-augmenteed RL agents represent more abstract goals (Jiang et al., 2019), generalize beller (Hill et al. 2020; Colas el al. Wong et al.)
2020
2021], explore more efficiently [Colas et al.) 2020 Tam et al. 2022
Klissarov et al.
2023
and can decompose complex goals into simpler ones (Chen et al. 2021 Ahn et al.
2022 Hu ct al.
2022
Sharma et al., 2021 Hu & Clune 2023). Our work extends these benefits to agents that learn from
self-generated linguistic leedback.
Inner speech. Generating linguistic rules for oneself is a form of inner speech. In the Vygotskian tradition, inner speech is seen as the internalization of the social speech generated by caretakers to help children solve
problems (Vygotsky 1965 Luria
1959). As a result, it is thought to support our capacities for complex,
long-term behaviors (Vygotsky 1965
Luria 1959
Vazquez et al., 2001 Spelke 2003 Al agents
endowed with forms of inner speech (explanations, descriptions or subgoals) have been lound to perform
and generalize better than agents trained with purely neural representations (Wong et al., 2021 Lampinen
ct al. 2022 Roy et al. 2022 Hu & Clune 2023 Kim et al. 2020 Unlike these approaches, PLLB approach
generates language in an unsupervised way and maximizes downstream performance as well interpretability
and inter-operability.
Multi-step reasoning and text agents. Recent works have proposed guiding LMs' reasoning by prompt-
ing them to step through sequences of "thoughts" (Wei el. al. 2023 Yao et al. 2022 Li el al. 2023b Shinn
et al. 2024 Our approach similarly uses a language bottleneck to concisely express intermediate informa- tion useful for later behavior (i.e. policy learning). However, unlike text-based reasoning agents such as ReAct (Yao al., 2022), Rememberer ęZhang et al.] 2023), and Reflexion (Shinn et al. 2024). PLLB does not require the underlying policy to be fully expressible in text. For example, we show that text-only maze solving agent completely fails, whereas PLLB can improve classical Q-learning style agents (Section G). Moreover, instead of describing actions for a particular state, PLLB rules capture high-level strategies over a sequence of states and actions, enabling generalization. We also show the learned rules can be shared to improve human task performance across multiple user studies (Sections and 7
3 The Language Bottleneck
PLLB builds on the standard RL. framework to train agents to solve decision-making tasks in human-like ways. We formalize decision-making tasks (e.g,, solving a maze) as Markov decision processes (S, A, f, R, T) with reward function R: : S x A + R (e.g., solving speed) over states S (c.g., cell coordinates) and actions A (e.g., directions), finite time horizon T, and a deterministic transition function f S A S that maps state-action pairs (s, a) to next states s (c.g., moving between cells). Standard RL then involves training policy æ to maxinize expected reward (e.g., learning to solve the maze faster). This is usually done by alternating between two steps: (1) collecting data with the current policy: D +- Ti and (2) updating the policy using the data: xi+1 +- update(m; D), where update implements an RL algorithm (Sutton & Barto. 2018). This procedure often fails to yield interpretable, human inter-operable, or generalizable behaviors.
Our key idea is to introduce a language bottleneck between data collection and policy update. We extract linguistic rules explaining past rewarding behaviors and then use them to regularize the policy's behavior in the next learning iteration (e.g.. I should go right in blue cells, see Figure D. Our experiments will show that this improves the interpretability and inter-operability (communicative use), and performance and
3

---

Published in Transactions on Machine Learning Research (02/2020)
SAYSELECT
MAZE
BUILDER
BIRDS
GRASP
00000
211.2.3..455
Dringe
Image GMP are. bran target and dfffsslon
A.grasp keypoint A Furquu
find blue balls
X
28 Levenstedn recouutuction loss
Dutput rule should fallas when providinn descr ipt
Output a rule I should follow when providing descripctons
Outoutt a ru I should fol Tom when grasping the object
stralegy follew
should follow
REWARD ROATROATIN::
HEGH REHARD white pel binnd
HIGH REMARD
(Witte, Faxt), (Atue, hor th) (Milte, nor thi,
gen gen_rule rule (LN)
ACTION:
Red dots 19,49, 243, (0.2, 0.24)-
ACTION: (54, G8)
LOV REWARD
LOV REMARD (Wiifie, South?, (whilte, RCWARD: 10
LOw REMARD At Least five dota acettered across canvas
LoW RevARD perched tree branch.
LOW REWARD
COSERVATION:
RIWARD:
TOXQUE
L
shou, setect the sane
obstere Alue, NOKTH.
Avord using vagus terms "scasterep".
Descrrbe the bird's
Grasp c cleser to the cencer for atable pickup.
[observation] solect
[observation] should select
L+ [observation] Provide descriptton
[observation]
[observation] shou select
update (LM)
Provide desce ipl ion
0 11:0, 2: 2: 3: 4: D.1, 5:0)
Q((0,2)]
Four blue triangles (0.8, 0.7)...
A red hird, likely
Q [[ing]
(North: 0.8, South: 0,
(... (60,50): 0.7,
East: D, West:0.2
cardinal
(60,60): 0.1, ...
Figure 2: PLLB be applied to diversity of domains. In all domains, it iterates between gen_ruu rule, function that prompts an LM (lop gray boxes) to extract a linguistic rule (C, blue) by contrasting high and low reward episodes from past experience (green and red boxes), and update, function for updating an agent's policy through interaction with the environment conditioned on L. PLLB can be applied to multi-step decision making tasks and visual or robotics tasks alike with minimal implementation variations (replacing the LM by VLM gen_rule, or replacing policy regularization by simple instruction condi- tioning update. Gray boxes represent prompts gen_rule and update, and full prompt details are Appendix Section B.7
generalization (cognitive use) of our agents over standard RT baselines in variety of tasks (see Figure 2 and Sections 5 to 7 The resulting algorithm thus alternates between three steps: (1) data collection, D, 2 T; which is implemented just as in ordinary policy learning approaches; (2) language bottleneck generation,
C, + ren_r)le(D,) and (3) policy updating, T+11 + update(T.. D., C,) (see Figure D.
The core mechanism underlying PLLB is consistent across all instantiations: gen_rule prompts a language model with contrastive examples (high- vs. low-reward trajectories) to extract a rule L capturing what distinguishes successful from unsuccessful behavior, and update conditions the agent's subsequent learning on L. What varies across domains is only how update incorporates the rule, which is depends on the learning agent's modality in that domain (e.g. Q-value regularization (Sections 5 6 prompt conditioning for LMs (Section 7) or visuomotor control (Section more detail.
We next describe these the ren_rule and update steps in
3.1 Rule Generation (gen_ruil) rule)
Using all the experience D, collected by the policy n. in the current iteration, gen_rule aims to infer abstract rule L, that best explains the agents' successful behaviors: L, +- gen_ruu ren_rule(D). This is done by prompting an LM with contrastive episodes from D; (top-N highest vs. top-N lowest total rewards) and asking it to provide the rule that. should be followed to oblain high rewards (see first row of Figure 2 and Appendix Section B.7 for full prompts). Importantly, this requires the first iteration of gen_rule to start only once we observe pair episodes with sufficiently different rewards. We found this contrastive approach, inspired by [hhong et al. (2023) and Dunlap et al. (202)), provide more precise rules than simply summarizing high-reward strategies.
4

---

Published in Transactions on Machine Learning Research (02/2026)
3.2 luie-Guided Policy Update (update)
Given rule C,, the update step produces new policy TH+1 t- update(TI, Dr, L.) that is better aligned with L. There exist many methods for leveraging language instructions to update agents' policies, though these methods traditionally focus on instructions provided by human experts. Crucially, which method to implement PLLB with depends on the underlying agent and action space representation, and will improve with advances in multi-modal modeling-
For RL policies, we leverage InstructRL (Hu & Sadigh 2023), which regularizes the learned policy with another policy induced by the linguistic rule Tg. For instance, in the maze example shown in Figure []] if the rule is 1 should go right every blue cell, the induced policy Te should assign probability 1 to the right action in blue cells, but equal probabilities to all actions in every other situations. In the Q-lcarning algorithm, this approach simply adds a regularizing term (orange) lo the standard Q-learning update rule
(t), t T++1 +
where 0c+1 arg max (Q(++11,a) + Alog mc(a 8t)]-
Here 7 is a discount factor and À controls the strength of the rule-induced regularization and could be made time-dependent with a pre-defined or learned schedule (e.g., stronger regularization early).
But how do we induce Ve from L? Since L is expressed in natural language, we may obtain a rule-conditional policy by prompting an LM. In particular, our experiments condition the LM on both the current rule and the current state of the agent St, and instruct the LM to generate the next action to obtain a probability distribution over admissible actions (e.g directions), as in Hu & Sadigh (2023). While these prompted LMs may perform tasks poorly their own (c.g. because of their inability to perform long-range planning process complex visual input), the regularizer may nonetheless guide Q-learning in the right direction. Running Q-learning updates with the regularization term from experience data gives us a new Q-table Q++1 from which we can derive the new policy æ;+1 by taking actions with maximum expected value in every state (441(5)) = max, 1 (s), a).
Finally, there exist some domains where the policy can be directly implemented by an LM or VLM, such as text-based games explored by methods like ReAct (Yao et al.) 2022) or Rememberer (Zhang ct al., 2023) In these cases, update can be implemented by conditioning the policy on the rule L and adding it to the prompt Te, ultimately steering the agent's: 's: Mi+1 <- <c.- Given new policy Ti+1, we can now close the loop and generate a new rule Che1.The choice between InstructRL-style regularization and prompt conditioning depends on the action space and policy architecture. In settings with large, continuous action spaces (e.g. reasoning over free-text), conditioning text-generation on rules likely more tractable, whereas Q-learning style updates may be more appropriate for interactive environments with feedback (c.g. game playing). We implement PLLB with tabular RL agents in Sections 5 and 6 with LM/VLM policies in Section 7 and visuomotor robot controllers in Section g
4 Experiment Set-Up
We run experiments on a diverse set of five tasks, with different agent architectures and action spaces, to showcase how PLLB can train more human-like agents. These include a simple two-player communccation game called SAYSELECT (Section 5) a MAZE solving task (Section 0 two collaborative image reconstruction tasks with synthetic (BUILDER) and natural (BiRDS) images (Section 7 and grasp planning for a 7-DoF robot (Section For all tasks, we include hyperparameter details (Appendix B.I), the exact prompts used in gen_rule and update (Appendix B.7). and examples of generated rules. For our main experiments, we prioritized using accessible and low-cost open weight language models such as 1lama-2-7bb-chat. However, we analyze the effect of changing the underlying language model on gen_rule in Appendix [BS]] and B.9
To evaluate the cognitive function of PLLB, we first evaluate the policies learned by artificial agents in each environment based on performance metrics such as task reward, similarity to known human-like policics,
Thhis regularizing term can also applied in the function approximation setting MMih al. 2013
5

---

Published in Transactions on Machine Learning Research (02/2026)
a. Standard (Reward)
b. Standard (Interpretability)
c. Fixed Speaker (Reward)
1,0
1.0
Bottleneck
Reward
0.5
Interpetablliy
0.8
Reward
0.8
TabularQ (Baseline)
0.6
0.0
0.4
InstructRI. (Oracle)
0.2
0.4
Adversarial Bottleneck
-0.5
0.2
2000
2000
4000
6000
0
2000
4000
6000
Episode
Episode
Episode
Figure 3: SAYSELECT. a) Bottleneck agents learn as fast as TabularQ and Instruct and faster than agents using adversarial rules Adversarial. b) Unlike TabularQ, they learn human-interprotable policies; this without relying on an external instruction like InstructRL. e) When faced with speakers enforcing a non-human nuuman-nterpretabbl policy, Bottleneck converges faster than InstructRL.
generalization. For all tasks, we compare our Bottleneck approach with baseline methods (e.g., standard RL, base LM) and an Adversarial ablation of PLLB that leverages invalid linguistic rules (e.g., obtained by labeling high reward examples as low reward) in gen_ rule. The purpose of Adversarial is to test whether PLLB gains arise from meaningful rule content rather than merely introducing any linguistic signal.
To evaluate the communicative function of PLLB, we conduct several human subject studies showing that generated rules L can help improve human task performance on a variety of metrics (e.g., reward, solving time). Participants were cither recruited in-person (for MAZE) or on Prolific (BUILDER, BIRDS), compensated at rate of US$12/hous All studies were approved by our institution's IRB. To the best of our knowledge, our work the first to validate self-generated agent rules from LMs for training human participants.
Finally, we conduct additional sensitivity analysis the affects of variations in prompt, model temperature, and non-contrastive episode selection in the Appendix.
5 SaySelect
As a proof-of-concepl, we first consider SAYSELECT, simple collaborative game introduced in Hu & Sadigh (2023) Here, a speaker can see a hidden set of five balls including two blue ones and must. help a listener find the two blue balls by communicating with them only via numbers, see Appendix Figure g In principle, the speaker and listener could converge on any bijective convention associating messages balls, and this is what we find with RL agents (multiple optimal policies). Humans, on the other hand, empirically prefer the mapping + 1, 2 + etc. which we call the human-interpretton showed that regularizing the listener's policy with the instruction I should select the same number as wy partner successfully guided the two RL agents to the numan-interpretabll policy, we ask whether PLLB could learn a similar rule and achieve a similar interpretability on its own?
policy. Whereas Hu & Sadigh (2023)
Task Overview. The speaker and listener are RL agents trained with Q-Learning that receive positive rewards when the listener collects blue balls, negative rewards otherwise. After training both agents for 200 episodes, we prompt llama--70bb-cha with the task description and contrastive set of high- and low- reward episodes to generate a rule L, e.g., should choose action 4 whenever the obsercation is 2, 3, 4 5. fgen_rule, see Figure 2
We generate new rules every 500 training episodes. In between, we regularize the listener's policy with a another policy æg induced from the current rule L. This is done by applying a softmaz on the LM's logits (see method in Section 3) across all possible actions and obtaining a distribution. In the following experiments, we compare our Bottleneck method, with the Adversarial version (corrupted rule), a TabularQ baseline,
well as an InstructRL upper bound using the same predefined instruction as Hu & Sadigh 2023
Shttes//huugginffarr.o//mmetttt -alanaaamaa2222 wll.amaa---m-mm-haaa 2-70b-chat-hf

---

Published in Transactions on Machine Learning Research (02/2026)
Standard
b. Generalization
c. Adaptation
1.00-
1.00
Bottleneck
Reward
0.77
Reward
0.75
Reward
0.75
Adversarial Bottleneck
0.80
0.50
022
0.25
"Lincar3 Agent exalasted oiter eptecdes
0.25
TabularQ (Baseline)
0.00
0.00
LinearQ (Baseline)
-0.25
-0.25
-0255
0
20
80
50
Episode
Episode
Episode
Figure 4: Results in MAZE. a) Bottleneck agents learn as fast as the non-linguistic Baseline agents, but faster than Adversarial and LinearQ agents. b) When faced with a new maze with similar structure, Bottleneck agents learn faster than TabularQ (which does not perceive color) and LinearQ (which does, but learns slower). c) When faced with maze of a different structure, Bottleneck agents adapt swiftly while LincarQ cannot recover. We do not evaluate Adversarial for the Gencralization or Adaptation experiments due to its poor performance in the standard setting.
5.1 PLLB helps learn human-interpretable policies
We first compare the four methods along two dimensions: their reward and their interpretability mcasured as the proportional (0 to 1) similarity between the listener's policy and the optimal human-interpretable policy (the intuitive message-bal described above). All methods but Adversarial quickly solve the task (Figure 3a), showing that corrupted rules hinder learning. Bottleneck and InstructRL both converge on the human-interpretable policy thanks to their linguistic rules while TabularQ deprived of such rule- does not (Figure Bp). Interestingly, Adversarial policies are more interpretable than TabularQ ones, suggesting that even corrupted language may provide inductive biases towards more human-like behavior. Qualitativcly, we observe that generated rules converge towards describing the human-interpretable policy (e.g., I should follow the strategy of choosing the same action as Agent 1, see examples in Section B.8
5.2 PLLB can learn counter-intuitive policies
A potential confound in the preceding experiment is the possibility that PLLB converges to a human- interpretable policy because this is the only behavior an LM can describe. To test this hypothesis, we fixed the speaker to use a counter-intuitive policy using a random mapping between balls and messages (e.g. 1 - 3, 5 + 2, etc.). Here, the human- human-interpretablee policy is mis-aligned, and would lead the listener to fail or learn slower. In Figure we observe that while InstructRL and Adversarial, both regularized by misaligned rules (the human-iiterppetabble one for InstructRL, another random permutation for Adversarial), eventually adapt to the fixed speaker, they converge at slower rate than both TabularQ (no rule) and Bottleneck. This effect was more pronounced at higher values of X. Overall, Bottleneck is the only method that adapts to a fixed speaker policy while also convorging on the human-like policy when the speaker is learning, without human supervision.
6 Maze
In the MAZE domain, agents must navigate a maze to find the goal using four directional actions (N/S/E/W We study mazes in which environment cues provide hints about the optimal path. This setup helps us study the impact of the language bottleneck: can PLLB agents infer the underlying structure and use it to generalize across mazes (cognitive use)? Can they transmit this knowledge to humans and help them perform better in novel settings (communicative use)?
We randomly generate different 7x7 mazes with the gym-maze code base 1 Agents receive as reward the inverse of the number of steps they needed to reach the goal as an incentive to learn the optimal solution. Although all mazes are random, we introduction additional structures by coloring cells for which the optimal action south (red) or north then east (bluc) with probability 50% (or they are left blank).
Task Overview.

---

Published in Transactions on Machine Learning Research (02/2026)
Therefore, agents that leverage color information can better generalize to new mazes where the coloring is preserved, even if the optimal action sequence differs.
The first iteration of PLLB corresponds to training standard RL algorithm to obtain Th, in our case a tabular Q-learning agent (TabularQ). After the agent observes at two solved mazes, we run gen_rule r_lueee by prompting a 1 LM with contrastive episodes and task description. This gives us linguistic rule Li (e.g., Upon observing RED, take SOUTH) that we can use to update the policy, We first induce the regularizing policy Tc, with rule L by obtaining probability distribution over the actions from the LM, and then run the RL algorithm for episodes to obtain the new policy m2 (procedure described in Section 3 We repeat these steps every 5 episodes of interactions with the environment.
Baselines include the labular Q-learning algorithm without language bottleneck (TabularQ) and a variant of Bottleneck generating rules from
episode samples (Adversarial). We also evaluate LinearQ. an agent learning a lincar model Q-function with an additional feature for cell color. We train LinearQ with batch size of 10 and learning rate 0.001, after performing hyperparameler sweep. We found LinearQ took significantly longer to converge than the other methods, likely due to the increased state representational complexity.
6.1 PLLB improves few-shot generalization
We next evaluate PLLB's ability to improve the few-shot generalization capabilities for policies on unseen mazes with similar underlying structure. For fair comparison, start with the fully converged policy for each method (requiring 100 episodes for LinearQ), Figure shows that, in the Standard setting, learning valid rule (Bottlencck) does not increase learning speed of single maze over TabularQ. However, using a corrupted rule (Adversarial) or learning from linear features (LinearQ) slows down learning. Furthermore, Bottleneck outperforms TabularQ with respect to few-shot generalization: when we switch to a new 7x7 maze sharing the same underlying color semantics, but with a different optimal action sequences. Bottleneck senueneeuueeeee leverages the learned rule and generalizes the new maze more effectively than all other agents (Figure While TabularQ cannot generalize because it does not perceive colors, LinearQ does generalize faster first, but converges slower.
The generated rules improve over time to better capture the underlying structure of the maze (e.g., if I observe BLUE, then lake the NORTH action; see other examples in Appendix B.Sp. Across all trials, 100% of the final rules mention the red + south rule and 60% uncovered the more complex blue + north-then- east rule. Finally, we find that policy that does not update Q-values (that is, purely implemented via the LM) achieves success rate 0% within the same time limit, even when using the final rules generated from PLLB. This highlights the importance of leveraging experience that cannot be expressed with language, which text-based reasoning agents cannot do.
6.2 PLLB learns adaptable policies
In Figure 4 we reproduce this same experiment. on maze with a different underlying structure (red now indicates west while blue indicates east then south). Although the rule Bottleneck learned does not apply anymore, it can still adapt faster than the baselines. We find that all trials end with rules capturing the new mapping (100% red + west, 50% blue + souuhhhh-sst)),, Mcanwhile, LinearQ overfits to the first maze and struggle to adapt. Overall, our results show that PLLB supports human-like cognitive functions by finding a tradeoff between the efficiency TabularQ and the generalization capabilities LinearQ, while remaining more adaptable.
6.3 PLLB is more interpretable and inter-operable
Can generated rules be useful to humans as well? We asked 50 university students to solve a 7x7 maze in the fewest steps possible using arrow keys. They could only observe the cells they had already visited and the walls they had already bumped into (sce Figure 10 in Appendix). We split them into three groups: a Control group receiving no assistance, and two others receiving information about a similar but different maze sharing the same underlying color semantics. The Visual group is shown visual representation of
8

---

Published in Transactions on Machine Learning Research (02/2026)
the optimal policy (arrows in each cell, see Figure in Appendix), while the Bottleneck group is provided randomly sampled language rule & learned by PL.BB This set-up lets us evaluate how generalizable the two different aids are to a new maze, as well as how quickly participants can account for incorrect information deponding on modality.
100
00000
Participants using PLLB rules solve the new maze with fewer steps than others and find this aid significantly more useful than the visual one in average (see Figure 5). Participants in- deed found it harder to extract the relevant information from the visual aid, which contained great deal of non-transferable information (optimal actions in all non-colored cells). PLLB rules focus on the useful and transferable information learned by the previous agent, which much more readily usable by humans. Overall, the results in MAZE demonstrate the abil- ity of PLLB to train agents that are more generalizable and adaptable (cognitive use) well more interpretable by, and inter-operable with humans (communictiive use).
Usefalness Rating
of Steeps
0000000
00000
0000
000
0000
Control Bottleneck Visual
Bottleneck Visual
Figure 5: Participants given PLLB rules solve new mazes faster than those given ei- ther visual or no aid (control). They also self-reported linguistic feedback more useful than visual. marks statistical sig- nificance at level 0.05 with two-sided Mann- Whitney test after Bonferroni correction.
7 Collaborative Image Reconstruction
Blaos
Badigrund
Species
Bewod Ty
2.1
100
Rowund
Humaa
(Tocn Scuich
Sporke
Bottlenock
Baseline
Figure 6: BUILDER: Bottleneck listeners improve upon Baseline descriptions over time, unlike an (Adversarial) listener that inaccurately represents rewards. Human users reconstructed target images with b. higher accuracy, and c. less time, when provided Bottleneck instruction vs. original Basclino descriptions. BIRDS: PLLB rules (3 iterations) help automated and e. human speakers generate better descriptions for naturalistic image reconstruction for three distinct reward functions.
PLLB is not restricted to the training of RL agents. This sec- tion introduces two collaborative image reconstruction tasks in- spired by the collaborative assembly task of McCarthh| (2020). BUILDER and BIRDS both consider two agents: a spenker, who can see hidden target image, and a listener, who must accu- description from the speaker. Both agents aim to converge
rately reconstruct. the targel. image based to description style leading to high reconstruction accuracy. We consider both synthetic images built from sequences of discrete actions by the listener (BUILDER), and natural images of birds that the listener reconstructs using text-to-image generation model (BIRDS). Here, PLLB combines with state-of-the-a multi-modal models to let. agents perceive semantic visual features.
7.1 Task Overview
Datasct. In BUILDER, we construct datasct of synthetic images with a variable number of shapes (tri- angle, square, circle) in different colors (magenta, blue, red, green) and locations on 2D grid. Each image is created by sequence of discrete actions, each representing particular combination of shape, color, and
9

---

Published in Transactions on Machine Learning Research (02/2026)
location (e.g., [ACT12 ACT2 ACT4], where ACT2 adds magenta circle in the lower right area of the grid). In BIRDS, we construct dataset of natural bird images by selecting 5 images for each of 10 bird species from the CUB-200-2011 dataset provided by Wah et al. (2011) (see Appendix Section B.4 for species list).
Reward. In BUILDER, since target and roconstructed images are fully determined by their action sequences, we measure task success via the Levenshtein similarity between the (sorted) corresponding sequences. In contrast, the BIRDS listener outputs text-to-image generations with varied properties, so we define three reward functions: Color, Background, and Species. To evaluate a roconstructod image Ty against a target 26, we form a contrast set from three CUB-200-2011 images differing in the target property (e.g.. color) but most similar in others. Reward is then r= E.cc d(CLiP(s;)) (5r); (L))( d(CLiP(=*)) CLIP(a+)); where CLIP denotes the CLIP image embedding (Radford et al. 2021 For both BUILDER and BIRDS, rewards are reported on held-out validation set.
Models. As open-source VLMs struggle with describing synthetic images, we use the llama-2-70b-cha LM as our speaker for BUILDER, representing images in raw text that list all shapes' type, color, and exact coordinates individually in sequence. On the other hand, we use the open source Llava VLM as our speaker for BIRDS (Liu et al. 2023). , We prompt both speaker models using the target image and a general task description, as well generated rules after the first iteration.
We implement the listener for BUILDER with a neural soquence-lo-squennce model pre-trained on English text [Lewis et al. 2019), which we fine-tune at each episode on training set of (description, action sequence) pairs using descriptions provided by the speaker. Meanwhile, we use the Stable Diffusion text-to-image diffusion model as the listener for BIRDS.
For both tasks, we consider two settings: From Scratch, where the listener initialized using the original pre-trained model weights at every episode, and Continual Training, where the listener is continuously updated over time using descriptions by the speaker. We create separate held-out splits for training, early stopping, selecting samples for gen_ gen_rule, rule, and evaluation.
Implementation. We first generate base image descriptions for both tasks by providing the speaker prompt containing a general task description and target image, but without any rule L. For the BUILDER task, an original image description is: At least five dols in lolal four red and one green scallered across the canvas, with two sets of matching locations (x= = 0.49 / = 0.24 and I = 0.5 / 0.24), another set at x = 0.52 / = 0.3, and finally, one dot located near x = 0.76 / = 0. This description is overly complex, mixes different coordinate formats, and difficult for the listener to use to reconstruct the target image, leading to low reward. Meanwhile, an example original image description for the BIRD task is: "a bird on a branch". While simple, this description does not contain sufficient information about the Color, Background, and Species, causing the listener to generate an image reconstruction with low reward.
We next evaluate these base descriptions using each task's respective reward functions, and select the (image. description) examples with the 5 highest and lowest rewards. Using the prompts shown in Section B.7 we implement cen_rule using the 1ama-2-70b-che and llava models for BUILDER and BIRDS, respectively. For BIRDS, we observe that output rules L are specific to the reward function: an example rule we get for the Species reward is: "Identify the bird's species if possible, and include any distinctive characteristics that set it apart from other birds.", while an example rule we get for the Background reward is: "Describe the setting or background of the image, such the presence of snow, water, other elements".
We implement update for both tasks by simply appending the output rule L to the original prompt to the speaker, as shown in Figure using the speaker's s output as the new description for a given target image. We repeat this cycle of eliciting rule C. appending to the same prompt used to generate the base instructions in order to re-label our data, with new instructions, and training and evaluating the listener model for a total of iterations for BUILDER and 3 iterations for BIRDS, We qualitatively show this leads to improved image reconstruction from the listener in Appendix Figure 13 and next discuss our experimental results.
10

---

Published in Transactions on Machine Learning Research (02/2020)
7.2 PLLB helps speakers provide more usable instructions
For both tasks, listeners following rules generated by PLLB (Bottleneck) outperform listeners trained from uninformed (Bascline) or misinformed (Adversarial) speakers (Figure (). This holds true for all three dif- ferent reward functions in BIRDS, as well as both training settings, with Continual Learning learning enabling Bottleneck to have an even stronger improvement over Baseline image descriptions by leveraging learned task experience. Continual Learning does not significantly help the Adversarial method, empha- sizing that the linguistic rules C in Bottleneck do capture succesfull task strategies. However, Continual Learning is not required for Bottleneck to oulperform Baseline image descriptions. The success of the From Scratch training setting can be interpreted as the speaker generating abstract rules to guide its own learning, leading to improved performance and showing evidence for a cognitive use of language.
Finally, we observe that the generated rules f. encourage the speaker to reduce vague and general language, and describe properties relevant to the corresponding underlying reward function in BIRDS, although this might take several cycles to fully converge (sce example rules in Appendix Section B.8 Furthermore, rules C become more complex over time, and across different trials we observe path dependency where different trials converge to different rules (e.g., (2) vs. O.rr. y 0.yy for coordinates in BUILDER). Overall, these rules help guide the speaker and listener agents in both tasks to iteratively improve image reconstruction over time, which can also be seen qualitatively in Appendix Figure 13.
7.3 PLLB can collaborate with human listeners
Do human listeners benefit from image descriptions provided by speakers following Bottleneck rules? We conduct human subject study for the BUILDER task, where we replace the listener with study participants at the end of one iteration of PLLB, and evaluate how quickly they can reconstruct target images from a held-out test set. We recruit 20 crowdworkers on Prolific to reconstruct 5 images using descriptions from a speaker following a rule L generated with (Bottlencck), and 5 images using the original (Bascline) descriptions. We provided participants an interface that included a drawing canvas and buttons controlling shapes and colors (Section B.3), allowing us to use Levenshtein similarity between (sorted) user actions and the underlying action sequences for target image reward.
Participants using Bottleneck descriptions built target images faster and more accurately than partici- pants using control descriptions (Wileoxon signed-rank test, p < 0.05, Figure 6 In a post-study survey, 55% preferred Bottleneck descriptions, describing them as "more direct and less ambiguous". ". 10% found both descriptions types equally easy to follow while remaining participants preferred Baseline descriptions because they gave more flemibility, leading to a more unstructured user study. However, we note that that user preference for customization does not correlate with any notion of improved performance in our original goal of image reconstruction.
7.4 PLLB collaborates with human speakers
Can humans benefit. from rules generated by PL.BB? We asked 12 Prolific crowdworkers to act as speakers and provide descriptions for our datasct of BIRDS images. Half of the participants were provided rules corresponding to one of the three reward functions (La) while the other half were not. Figure t (bottom) shows that listeners instructed by human speakers provided with PLLB rules outperform those instructed by uninformed human speakers (Bascline) (Bonferroni--orreette t-tests < 0.05 for species and colors).
The descriptions from users not provided a rule are more diverse and less focused e.g., A barbed wire is an uncomfortable stop for a bird. With PLLB rules on the other hand (e.g., Describe the bird's coloration accurately), they generate more specific descriptions: Red and black bird on barbed wire. Bright red chest, red top of head, black wings and beak for the same target image (see other example rules in Appendix Table PI.LB agents easily transmit, what they learned from experience (the rule) to humans.
11

---

Published in Transactions on Machine Learning Research (02/2020)
Standand
c. Zero Shot VLM Ouiput
d. Boedlencck Esamples
-30
reward
Egrande
Gomerallzation
Where should the tobof graup the object?
Gäven the controsting sxureplzs ( nskerr showld the robof grasp the object?
A'lenaa Motaa-28: The should grasp the 1os The rost suttable heypoint for the robot to bag would
aLLena (oo
OLLemat//olmooTee tiraet stable flexiolo sections
grosp the bug by its sidee, awar sring
15
Epioode
Vanilla RI. (Boseline)
Adversarial Bottleneck
Zero-Shot VLM (Oracle)
Bottleneck
Figure 7: Results in GRASP. a) Bottleneck agents learn faster than the non linguisti RL Baseline agents, as well as Adversarial and an Oracle zero-shot prompt of a VLM without any internal policy update. b) Bottleneck agents generalize faster than both Baseline (which over-relies on image details) and Oracle to unseen bag designs and locations, (c) shows the fixed linguistic rule d) Example grasps, shaded by reward.
8 Robot Grasp Planning
Can PLLB rules improve policy learning in embodied tasks with complex action spaces such as robot planning? Existing works on LM-based agents (e.g., ReAct (Yao et al.] 2022) or Reflexion (Shinn et al. 2024)) and robot policies (e.g., Code Policies) leverage pre-trained models for action generation, but cannot be applied in domains where (non-LM-based) policy must. be updated using real-world RL signals.
Consider training a robot to lift paper bag. The most intuitive grasp would be at the bag's 's handles; indeed, zero-shot Oracle prompt of the molmo-7b multi-modal language model outputs grasp keypoints on bag handles (Figure 7). However, if the bag is too heavy, a robot should adapt to less common grasps (e.g. by the side) to avoid tearing. Because VLMs are not grounded with actual physical quantities, such adaptation is impossible without leveraging an external reward (Gao et al.] 2024). On the other hand, existing Rasbaaod approaches for robotic manipulation are often inefficient (Kalashnikov et al. 2018). We show that PLLB rules can be used to efficiently learn non-intuitive robot grasping policies.
8.1 Task Overview
In the GRASP lask, our agent a Franka Emika Panda Arm robot that needs to lift and move a paper bag filled with unknown objects off of a shelf, as shown in Appendix Figure [2] Due to the complex action space, follow Kalashnikov et al. (2018) and decompose the robot's policy into (i) perceiving the scene and identifying an appropriate grasp location, and (ii) planning path to this location.
We begin with a fired keypoint-condditoneed path-planning policy which is pre-trained on a dataset of 50 trajectories with 3D-point cloud observations using Sphinx (Sundaresan et al. 2025). This model outputs a trajectory of 7-DoF' end-effector poses ht/////w//pitl/// http://ithho/t///eee (to/gripper) for the robot to execute given the keypoint. We then use PLI.B to train (via RL) a grasp prediction policy that selects a grasp keypoint (a 2-D position) given an image of the environment. We define a successful grasp as one where the robot can lift the bag without it breaking, and so set reward to be 0 for successful grasp, -100 if the bag is not grasped at all, and use the normalized downward torque (e.g. -20Nm) applied to the robot's shoulder joint as dense reward for unsuccessful grasps. Initially, the agent performs random grasps. After it observes one successful grasp, we run gen_ can_rule with the open-source MOLMO-7B vision language model with contrastive images of successful and unsuccessful grasp keypoints. This gives us a linguistic rule (e.g. grasp the left side of the bag) that we can use to update the policy. We regularize the initial grasping policy toward the policy ne induced from rule by prompting molmo-7b while conditioning on L, then instructing it to generate random keypoints. By executing Sphinx with these keypoints, the overall policy is updated to condition on L.
12

---

Published in Transactions on Machine Learning Research (02/2026)
We compare our Bottleneck method with Adversarial version (swapping high and low reward grasps and an Oracle that always guides the policy with the rule generated from zero-shot prompting molmo-7b (no constrasting examples based on a reward). We call this Zero-Shot VLM an Oracle as it represents the VLM's privileged prior knowledge about intuitive grasping strategies, or knowledge a human expert would also possess. We also evaluate a Vanilla RL baseline where the policy only uses image observation data to predict a grasp keypoint based on task reward after exccution, which helps us isolate PLLB's incremental value for keypoint selection.
8.2 PLLB can learn counter-intuitive policies
As shown in Figure 7 when averaged across 20 random seeds, our Bottleneck converges to higher reward compared to all other methods. Because the Oracle method suggests a rule encouraging the policy predict key points near the bag's handles, it requires more episodes to learn that this behavior leads to lower reward. Qualitatively, common locations (e.g,, Grasp the boltom of the object, see examples in Appendix Section B.8 This shows that PLLB can help improve over standard RL methods even for learning counter-intuitive policies.
observe the generated rules converge towards guiding the robot to grasp at less
8.3 PLLB improves few-shot generalization
To evaluate fewOshot generalization, create environment with new bag, location, and set of visual distractions (see Appendix 12] We then evaluate Bottleneck, Zero-Shot VLM (Oracle), and Vanilla RL (Baseline) when intialized with the policies learned in the original environment, allowing for updates from the environment over time. As shown in Figure 7 the rules learned by Bottleneck outperform alternative methods for generalizing to this new environment. Because the Vanilla RL policy can only rely on image features, it is not able to leverage the more generalizalle stralegy PL.LB rules capture and select grasp keypoints that hold semantic meaning (e.g. the bottom of the bag) captured by pre-trained LMs. Overall, the GRASP results show that PLLB can improve policy convergence even for the complex dynamics of embodied tasks.
9 Limitations & Discussion
PLLB helps agents becomes more interpretable by and inter-operable with humans. Tn our current imple- mentation, rule_gen requires converting cpisodes into LM-compatible representations (text/images), which may limit use with long-horizon sensorimotor trajectories. Nevertheless, our Grasp lask successfully uses pretrained robotic models, and advances in long-context modeling can address this limitation. A natural question is whether PLLB's gains arisc specifically from language or from reward-aligned abstraction more generally. Comparing against non-linguistic bottlenecks (e.g., learned latent codes) is challenging: such rep- resentations require training across many task variations to become meaningful, whereas language provides pre-structured abstractions via pretrained LLMs that transfer single tasks. Our Adversarial ablations across experiments confirm that rule content, not merely the presence of bottleneck, drives improvements with PI..B.
Our human studies also demonstrate uniquely linguistic benefits (interpretability, human transfer) that would be unavailable with other latent representations (including code). However, rule generation can fail when the LM either abstracts too much (e.g., BIRDS we once found the rule: "Use descriptine language that conveys the mood, such as the serenity of a snowy day or the freedom of flight") or, more scriously, causes harm in safety-critical situations by creating a false trust in generated rules. We believe improvements in multi-modal models can help mitigate these issues (Appendix B.9), as well additional verification steps of rules for high-stakes situations.
Our work opens up a set of interesting questions around designing intelligent sampling of contrastive episodes in rule_gen. For example, sampling episodes that are not outliers can help with handling stochastic envi- ronments, where high reward trajectorics may not necessarily be optimal. Sampling of contrastive episodes can also adjust known propertics of how LMs attend to long context (i.e. more attention to episodes at the start and end). Another interesting direction for future work is applying PL..B to tasks with complex
13

---

Published in Transactions on Machine Learning Research (02/2026)
reward functions that reflect hard-to-articulate human preferences, such image captioning for accessibility (Nie et al. 2020) and personalizing language models (Li et al. 2023ab. Although one concern is whether the linguistic rules from PLLB might exacerbate spurious reward signals, any reward misspecification be- comes more detectable and diagnosable through interpretable rules, whereas black-box policies offer no such transparency.
Our experiments with PLLB focus on environments with tractable action spaces (e.g. discrete choices, text captions, keypoints for robotic grasping). In like open-ended text generation in math reasoning code synthesis, this approach would require adaptation. Possible extensions include: (1) using the rule verifier or reward bonus rather than an action prior, (2) discretizing the action space into semantically meaningful categories that the LM can reason over, or (3) using the rule to filter or re-rank candidate actions from a base policy. Such adaptations would extend the cognitive and communicative benefits of taaguage-guiden learning a broader class of sequential decision-making problems (Colas et al. 2022 and we leave exploration of these extensions to future work.
10 Acknowledgements
We would like acknowledge support from NSF Awards #2006388, #2125511, #2238240 and #2212310, AFOSR YIP, ONR #N0001421122988 and DARPA. Research sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Megha Srivastava was additionally supported by an IBM PhD Fellowship. Cédric Colas received funding from the European Union's Horizon 2020 research and innovation program under the Marie Sklodowska-Cuunn grant agreement No. 101065949.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as i say: Grounding language in robotic affordances. 2022.
Allen Institute for Al. Molmo blog.
brll Accessed: 2025-02-25.
Bastien Boutonnet and Gary Lupyan. Words jump-start vision: A label advantage in object recognition. Journal of Neuroscience, 35(25) 9335, 2015. doi: (0.152//NBUURSCC11111-1200
URL https:
wwww.jneurosci.co org/ wttpeontttttnn//////oo/
Peter Carruthers and Jill Boucher. Language and Thought. Cambridge University Press, 1998.
Valerie Chen, Abhinav Gupta, and Kenneth Marino. Ask your human: Using human instructions to improve generalization in reinforcement learning. Proc. of ICLR, 2021.
Maxime Chevalier-Boisver Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. Proc. of ICLR, 2019.
Sahil Chopra, Michael Henry Tessler, and Noah Goodman. The first crank of the cultural ratchet: Learning and transmitting concepts through language. Proc. of CogSci, 2019.
Andy Clark. Magic Words: How Language Augments Human Computation. In Peter Carruthers and Jill Boucher (eds.), Language and Thought. Cambridge University Press, 1 edition, 1998. ISBN 978-0-521- 63108-2 978---921-63758-9 978-0-511-59990-9.
Códric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Fricr, Peter F. Dominey, and Pierre- Yves Oudeyer. Language as cognitive tool imagine goals in curiosity driven exploration. Proc. of NeurIPS, 2020.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Cédric Colas, Tristan Karch, Clément Moulin-Frier, and Pierre- Yves Oudeyer. Language and culture inter- nalization for human-like autotelic ai. Nature Machine Intelligence, 4(12):1068-1077 10766 2022.
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gon- zalez, and Serena Yeung-Levy. Describing differences in image sets with natural language, 2023.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024. URL https: org/ abs/
2309.025611
Dedre Gentner and Jeffrey Loewenstein. Relational Language and Relational Thought. Erlbaum, 2002.
Robert D Hawkins, Minae Kwon, Dorsa Sadigh, and Noah D Goodman. Continual adaptation for efficient machine communication. Proc. of ACL, 2020.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepes- vari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Mareus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world. 2017.
Linda Hermer-
Anne Mosfet, and Paul Munkholm. Language, space, and the development of cog- nitive flexibility in humans: The case of two spatial memory tasks. Cogmition, 29(3):263-2990 2001.
Mary Hesse. The Cognitive Claims of Metaphor. The Journal of Speculative Philosophy, 1988.
Felix Hill, Andrew K. Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClel- land, and Adam Santoro. Emergent systematic generalization in situated agent. Proc. of ICLR, 2020.
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-Al coordination, 2023.
Hengyuan Hu, David J Wu, Adam Lerer, Jakob Foerster, and Noam Brown. Human-Al coordination via human-regularized search and learning, 2022.
Shengran Hu and Jeff Clune. Thought cloning: Learning to think while acting by imitating human thinking. Proc. of NewrIPS, 36, 2023.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Proc. of NewrlPS, 2019.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep rein- forcement learning for vision-based robotic manipulation. arXiv preprint Xrx:::1866. 10293, 2018. URL
/1tp../1.....00 10293
Jinkyu Kim, Suhong Moon, Anna Rohrbach, Trevor Darrell, and John Canny. Advisable learning for self- driving vehicles by internalizing cnerrattio-to-acttion rules. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2020. doi: (..109//VVR42660 011e422600.2000.000 2020..0902..2....
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Tue Bacon, Pascal Vincent, Amy Zhang, and Mikacl Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. 2023.
George Lakoff and Mark Johnson. Metaphors We Live By. University of Chicago press, 2008.
Andrew K. Lampinen, Nicholas A. Roy, Ishita Dasgupta, Stephanie C. Y. Chan, Allison C. Tam, James L. McClelland, Chen Yan, Adam Santoro, Neil C. Rabinowitz, Jane X. Wang, and Felix Hill. Tell me why! -explanations support learning of relational and causal structure. Proc. of ICML, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeeeee pre-training for natural language generation, translation, and comprehension, 2019.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. 2023a.
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei- Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language podel-auggentee code emulator. 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselincs with visual instruction tuning, 2023.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foenster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. 2019.
Aleksander R Luria. The directive function of speech in development and dissolution. Word, 15(2), 1959.
William McCarthy. Emergence of compositional abstractions in human collaborative assembly. 2020. URL
https://appl httpppssaaaacccccooooooooooo///o wtw.oooaaa..rr/oppsosssoao :33117999
Reid McTlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhuman ai with human behavior: Chess model system. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Dala Möning, KDD '20. ACM, August 2020. doi: (0.1145/339446 3403219. URL http://dx.dee doi wttp://o....ooooooooooo//
340321
Alex Mesoudi and Alex Thornoon. What cumulative cultural evolution? Proceedings of the Royal Society B, 285(1880):20157712 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Allen Nie, Reuben Cohn-Gordon, and Christopher Potts. Pragmatic issue-sensitive image captioning, 2020.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Nicholas A Roy, Junkyung Kim, and Neil Rabinowitz. Explainability via causal self-talk. Advances in Neural Information Processing Systems, 35:7655 7670, 2022.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. Proc. of ACL, 2021.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Lan- guage agents with verbal reinforcement learning. Proc. of NeurIPS, 2024.
Elizabeth Spelke. What makes smart? core knowledge and natural language. Language in mind: Advances in the study of language and thought, pp. 277-311, 2003.
Priya Sundaresan, Hengyuan Hu, Quan Vuong, Jeannette Bohg, and Dorsa Sadigh. What's the move? hybrid imitation learning via salient points. In Proceedings of the International Conference on Learning Representations (ICLR), February 2025.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Allison Tam, Ncil Rabinowitz, Andrew Lampinen, Nicholas A Roy, Stephanic Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. Proc. of NeurIPS, 2022.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Michael Henry Tessler, Jason Madcano, Pedro A. Tsividis, Brin Harper, Noah D. Goodman, and Joshua B. Tenenbaum. Learning to solve complex tasks by growing knowledge culturally across generations, 2021.
L.S Vygotsky. Thought and Language. MIT Press, 1965.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, 2011.
Tony T. Wang, Adam Gleave, Tom Tseng, Kellin Pelrine, Nora Belrose, Joseph Miller, Michael D. Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, and Stuart Russell. Adversarial policies beat superhuman go ais, 2023.
Olivia Watkins, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Jacob Andreas. Teachable reinforcement learning via advice distillation. A. Beygelzimer, Y. Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Proc. of NewrIPS, 2021.
Sandra R Waxman. The development of appreciation of specific linkages between linguistic and conceptual organization. Löngua, 92:229 257, 1994.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models, 2023.
Catherine Wong, Kevin Ellis, Joshua B. Tenenbaum, and Jacob Andreas. Leveraging language to learn program abstractions and search heuristics. Proc. of ICML, 2021.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. 2022.
Hanako Yoshida and Linda B Smith. Sound symbolism and early word learning in two languages. In Proc. of CogSci, 2003.
Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. Large language models are semi-parametric reinforcement learning agents. arXiv preprint arXiv:2306007900 2023.
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahm, Dan Klein, and Jacob Steinhardt. Goal driven discovery distributional differences via language descriptions, 2023.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: Generalising to environment. dy- namics via reading. In Proc. of ICLR, 2020.
A Appendix
B Appendix
B.1 Hyperparameters
1. SAYSELECT: We use the same default parameters used in InstructRL (Hu & Sadigh 2023 including setting the regularization strength À = 0.25. Additionally, each time we invoke gen_rulé we create an ensemble of rules, which we aggregate over when construct the probability distribution over actions during update.
2. MAZE:
We use the InstructRL objective with tabular Q-lcarning agent, but introduce a CLM parameter that controls whether the regularization strength is O or 1 at each time-step in an episode. Although we did not observe a strong effect on modifying cLMs we did not explore values larger than GLM = 0.4 as that led to increasod inference cost and experiment latency. Finally, each time we invoke gen_rule we create an ensemble of rules, which we aggregate over when construct the probability distribution over actions during update.
17

---

Published in Transactions on Machine Learning Research (02/2020)
wsll puck
Till pakk ssl
Figure 8: Overview of SAYSELECT game, reproduced from Hu & Sadigh 2023
For the listener agent we linetune BART for a maximum of 100 epochs each iteration of PLLB, employing early stopping
BUILDER:
held-out validation set and using the default arguments pro- vided by the HuggingFace Transformers library (Lewis et al. 2019h. Each time we invoke gen_rule, we same 3 rules and select the rule with the highest aggregate likelihood across all tokens. Likewise, for each image description (the speaker's action), we sample possible descriptions under the given rule and select the one with the highest probability.
BIRDS:
For the fine-tuned version of the listener agent, finetune StableDiffusion for 1000 steps on separate finetuning dataset. When generating images, simply sample one image per description, using 10 inference steps and guidance scale of 7.5.
5. GRASP:
For grasp keypoint selection, we train an upper confidence bound agent on a neural contex- tual bandit using image features as context, with learning rate 0.01, with an (100, 100) action space containing integer coordinates of the environment image. During update, we sample 20 coordinates following a given rule from gen_rule, which we use as the probability distribution over actions, set- ting À 0.2 as the regularization strength. To execute the grasp, we use a grasp model built by training Sphinx (Sundaresan al. 2025) over 50 waypoint trajectories with learning rate 0.0001 for 100 epochs.
B.2 SelectSay Overview
B.3
Maze
See Figures and
10
Figure 9: Visual aid provided to participants in the human subject study for the MAZE task.
B.4 Dataset Details for Birds.
We consider images of the following bird species the CUB-200-2011 dataset from Wah al. (2011 for the BIRDS task: [Indigo Bunting, Cardinal, Yellow Breasted Chat, American Crow, Vermillion Flycatcher, California Gull, Blue Jay, Tropical Kingbird, White Pelican, Horned Puffin].
18

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 10: Interface used in the human subject study for the MAZE task. Cells not visited by the participants are hidden in gray.
8/10
follow instruction Each instruction be belongs other Type or Type you should follow al instrcctoon the come way but wa ask you after the study choose whether you found "ype inst uctions or Type 8 instuctions
right create
bu buttons right
(Type B) Three red shapes in the top left quadrant, including two triangles and located (0.2, 0.73), another (0.27, 0.72), third at (0.31, 0.84). Additionatly, there is a red square positioeed (0.24, 0.25) and two more triangles, one (0.23, 0.68) and the other (0.21, 0.73).
squaro,
will receive
post-study survey
Please complets the task one sittnng your best to fallow the instructons, We reserve the right
reject your subm ission you cieate random or empty images.
Red
Green
Blue
Magenta
Cirdl
Triangle
Square
Clear
Finish
Figure 11: Interface used in the human subject study for the BUILDER task.
B.5
Grasp
B.6
Builder and Birds Iterations
B.7
Full Prompts
We first provide the full prompts used for gen_rule rule. For space constraints, we do not include example samples, but note they follow the format shown in Figure
SAYSELECT: You will be given a list of (OBSERVATION, ACTION, REWARD) examples collected from agents learning to salve task. Possible ACTTONS agent can take 1, 2, 3, 4, 5, and quit. Each
19

---

Published in Transactions on Machine Learning Research (02/2026)
Fueelard Ca
Figure 12: Task environment for GRASP, showing new location and distractors for the Gencralization setting.
Test Target
Listener Reconstructions
BUILDER
BIRDS
1=3
Figure 13: Both the BUILDER and BIRDS tasks consist of speaker and listener agents. At test time, the speaker needs to provide a language description the listener that helps them recreate the image accurately. For both tasks, PLLB helps improve listener accuracy over time.
OBSERVATION describes the ordered sequence of actions that AGENT 1 picks, and each ACTION describes the ACTION that AGENT 2 picks based the given OBSERVATION. The examples separated into HIGH REWARD and LOW REWARD examples.+ [samples] examples++ssamppess) + a language rule that best summarizes the strategy AGENT 2 should follow to receive HIGH REWARD, not LOW REWARD, based the examples. Start the instruction with the prefix should'.
MAZE: You will be given a list of example (OBSERVATION, ACTION) trajectories collected from an Each trajectory receives REWARD. Possibles ORSERVATIONS agent see are: WHITE, RED, BLUE Possible ACTIONS an agent can take are: NORTH, SOUTH, EAST, WEST. The examples are separated into HIGH REWARD and LOW REWARD examples + [samples] + Output language rule that best summarizes the strategy the AGENT should follow when picking a sequence of ACTIONS solve the maze and receive HIGH REWARD, not LOW REWARD, based on the examples. Start the instruction with the prefix 'I should..
AGENT learning solve
3. BUILDER: There are two agents. The goal of Agent is to provide instructions to Agent 2 that helps Agent 2 to successfully recreate the image. You will be given list (ORIGINAL, AGENT INSTRUCTION, RE- WARD) values where ORIGINAL. is the original description of an image, INSTRUCTION is the instruction provided by Agent 1 to Agent 2, and REWARD is the reward Agent 2 receives when trying to re-create the image (higher is better). The examples separated into HIGH REWARD and LOW REWARD examples. + [samples]+
Based on the examples above, output list of 2 RULES for Agent 1 to follow when generating INSTRUC- TION order receive HIGH REWARD, instead of LOW REWARD. Write the rules after prefix RULES:
4. BIRDS:
The top row of three images have the following HIGH REWARD pnteeriptions:+hhii reward samples+ battom row three images have the following LOW REWARD descriptions: reward samples+ Provide rule should follow in order to provide image descriptions with HIGH REWARD, not LOW REWARD. Provide the rule after the prefix RULE:
5. GRASP:
The top image shows a grasp keypoint with HIGH REWARD. The boltom image shows a grasp keypoint with LOW REWARD. Based these images, provide rule the robot should follow in order to select grasp keypoint that results in HIGH REWARD, not LOW REWARD. Provide the rule after the prefix RUL.F:
20

---

Published in Transactions on Machine Learning Research (02/2026)
We next provide the full prompts used in update for each task.
SAYSELECT: Agent 1 selected [observation]. So should select
MAZE:
You are an agent solving a maze following a provided RULE. You will be given a list of PREVIOUS ACTIONS and the CURRENT OBSERVATION. Follow the RULE to select your NEXT ACTION (East, West, South, North):
RULE: [4]+ PREVIOUS ACTIONS: [_....-]] + CURRENT OBSERVATION: + [observation]
What is the NEXT ACTION you should take? Output one of (East, West, South, North) after the prefix
NEXT ACTION:
3. BUILDER: You will be given DESCRIPTION of image. Your goal is to use this description to provide short INSTRUCTION to help someone else, who cannot see the image, accurately re-construct You will also be given a list of RULES you must follow when providing the instruction.
DESCRIPTION: + observation
RULES: +L +
Please provide a short instruction following the prefix INSTRUCTION:
4. BIRDS: Provide a one-sentence description of this image, using the following RULES: C
5. GRASP: Provide a keypoint in the image where the robot should grasp the object, following the RULE: C.
21

---

Published in Transactions on Machine Learning Research (02/2020)
B.8
Examples of Generated Rules for All Environments
See Tables
4 and
5
Setting
L1
L2
Ls
Standard
should follow the strategy of choosing action whenever the observation 3, 4,
should follow the strategy of choosing actions based on the observation provided by AGENT 1. If the observation 1, should choose action 1. the observation is 2, should choose action 3. If the obser- vation is 3, I should choose ac- tion 3. If the observation is 4, I should choose action 4. the observation is 5, should choose action 5.
I should follow the strategy of choosing the action AGENT for observations 1,
2, 3, 4, and
Fixed Speaker
should choose action when
I should choose action when observation is 1, 2, 3 5.
should choose action 1 when observation is 1. should choose action when obser- vation is I should choose action when observation is 3. I should choose action 2 when observation chouse action 3 when observa- tion
observation is 1 or 2 5. should choose action 2 when obscrvation 3.
should choose action 2 when observation is 4.
T should
Table 1: Example rules generated for the SAYSELECT environment, for the Standard setting (both Listener and Speaker agents are RL agents trained from random initialization) and Fixed Speaker agent.
B.9 Effect of Model on Generated Rules
We additionally compare the effect of different language models, in particularly those of different sizes, our results. We did not observe significantly strong quantitative difference in performance when gen_ gen_rule rule is instantiated with models of different sizes (e.g. llama-2-7bb-chat vs. llama-2-13b-chat. However, we did notice interesting qualitative differences across samples that are likely due to the additional fine-tuning step using reinformmenet learning from human leedback (RLHF, see https://llama, meta. com mommmlama2/ for more information). We describe these differences per environment below.
1. SAYSELECT: Because of the simplicity of this environment, it is possible for rule L to summarize the full optimal human-inlerprettbbe policy a sequence of if-statements (e.g. "If the currend stale is '1', I should take action 1.") we observed that smaller language models (e.g. llama-2-7b-chat and 1lama-2-13--hhe instruct) were better able to generalize and use more efficient language, such as should take the same action as the observation". The smaller models also often included superfluous language, such as "I should always take the action that leads the highest reward".
always did this, while larger models (c.g. Ilaba-2-70b-haat and mistral-8x-7b.
2. MAZE: The majority of generated rules captured the underlying color semantics of the maze, enabling generalization. However, smaller model sizes (e.g. 1lama-2-7b-chat) resulted in more superfluous language (e.g. "I should always prioritize taking actions that lead to the most recent reward, and avoid taking actions that lead low reward.") "), and the generated rules focused more on the first few actions the agent should take, which may not always generalize.
3. BUILDER: While we do not observe any model-speciiic differences, there exists variation across samples in the type of formalting and synlax generaled rules encourage (e.g. provide coordinates using the format (I, y)* vs. a using the format 'x=0.XX, y=0. YY""), leading to agents converging to different descriptions. Furthermore, some rules encourage list formats (e.g. 1.Drow a green dot (0.72, 0.21). 2. Draw a green dot at (0. 10,78, 0. 0.72).) while other rules encouraged clustering of identical shapes (e.g. Draw two green dots al. (0. 72, 0.21) and (0.73, 0.72).)
image descriptions
22

---

Published in Transactions on Machine Learning Research (02/2026)
Size 7x7
E1
Ls
Ls
should start by mov- ing north, then con- tinue moving north un- til reach a wall, then turn cast and move un- til reach a wall, then turn south and move un- till reach a wall, then turn west and move un- til I reach the end of the maze.
should prioritize moving North, East, and South over moving West, as the HIGH REWARD EX- AMPLES show a higher frequency of these ac- tions. 2. should avoid moving West unnecessar- ily, as it is rarely seen in the HIGH REWARD EX- AMPLES and often leads to LOW REWARD EX- AMPLES
should take the EAST action when WHITE, then repeat the EAST action until see RED, then take the SOUTH action, and then repeat the FAST action until I see BLUE, then take the NORTH action, then take the EAST action until I see RED again, then take the SOUTH action, and finally take the WEST action.
I should take the EAST action when I observe WHITE, and keep tak- ing EAST actions until I observe RED, then take the SOUTH action. At ter taking the SOUTH action, I should take the EAST action again un- til observe BLUE, then take the NORTH action. After taking the NORTH action, T should take the EAST action until I ob- RED again, then take the SOUTH action.
see
Table 2: Example L rules generated for the MAZE environment for the 7x7 maze size.
Setting
L2
Re- Cee-Tnitialinn Iiatiin
1. Be specific with location detaila: Agent 1 should pro- vide detailed location informa- tion for each element in the image, such as x and coor- dinates. 2. Use descriptive language for elements, such as "red dot' or 'green triangle'.
Use specific coordinates when instructing Agent to draw shapes. 2. Use descrip- tive language to specify the color and shape of cach elc- ment. For example, 'a green triangle" instead of 'a green thing'.
Be specific with location co- ordinates: provide specific co- ordinates
the location each shape, using format x 0.XXX, y=0.YY. 2. Use descriptive shape names: In- stead of using generic terms like "dot" or "square," use more descriptive namess that indi- cate the shape's color and size, such 'green triangle' or "red square."
Continual Training
1. specific and detailed your instructions. High reward examples specific coordi- nates and shapes, while re- ward examples have more gen- eral descriptions. 2. consistent format for your in- structions. High reward ex- amples have a consistent for- mat for listing coordinates and shapes, while low reward ex- amples have more free-form format.
Provide explicit coordinates for each element the image, using format (x, y). Use specific colors when referring clements in the image, such as 'red", 'green', or 'blue'. Avaid using vague terms like "colored' or "shadod".
1. Use consistent for- for describing shapes, such as always listing the coordinate first, followed by the y-coordinate. For example, instead 'one green square at the point x=0.53, y=0.24", use "one green square at (0.53, 0.24)'. 2. Avoid using vague terms like "various shades of green'. Instead, use spe- cific colors, such as "green" or "blue". Additionally, use spe- cific shapes, such as "square' "triangle' rather than vague terms like "rectangle'.
a
Table 3: Example L rules generated for the BUILDER environment.
Rules demonstrated more reward-specificity (i.e. specific to background, color, or species rewards) when generated with larger VLMs (e.g. Ilava-13b) versus smaller models (e.g. llava-v1.6 vicuna-7b), with the latter primarily proposing rules that encouraged more detailed descriptions (c.g. "Avoid using vague general terms").
BIRDS:
5. GRASP:
We found that most VLMs other than molmo-7B performed poorly al recognizing keypoints, and instead relied heavily on task information. For example, one rule generated by Ilava-13b was "A robot should grasp a keypoint that is visible and not obstructed by other objects." " We believe the superior performance of the molmo series of models are due to specifically training on the PixMo dataset. with 2D-poists (Allen Institute for AT).
23

---

Published in Transactions on Machine Learning Research (02/2026)
Reward
Li
L2 Describe the bird's coloration accurately.
color
Describe the bird's color, species, and any distinctive markings patterns.
Describe the bird's coloration accurately.
background
Include details about the bird's surroundings, such as the type of branch or post it is on, and any additional elements in the background.
Include the bird's action (perched, flying, standing) and its location (on a branch, railing, pole, etc.)
Describe the bird's action (fiy- ing, perching, standing) and the environment it is in (sky, tree, water).
species
Describe the bird's color, markings, and any distinctive features.
Describe the subject's unique features, such as coloration, beak shape, other distin- guishing characteristics.
Include specific details about the bird's appearance, such
the color of its feathers, beak, eyes, and any distinctive markings or patterns.
Table 4: Example & rules generated for the BIRDS environment demonstrate reward-specificity time.
Reward
L10
C30
Eso
Shoulder Torque
Select keypoints that are away from sharp edges or corners of the object to avoid potential damage and improve stability during grasping.
Prioritize grasp keypoints that are closer to the center of the object for more stable and pre- cise pickup.
Prioritize grasping at the mid- dle of the object's surface rather than the top edges or handles to ensure a stable grip.
Table 5: Example C rules generated for the GRASP environment.
B.10
Sensitivity Analysis of Rule Generation
We next compare the sensitivity PLLB to the implementation of gen_rule, specifically variations in the prompt and temperature for sampling. We present results for SAYSELECT, as its discrete action space and deterministic dynamics provide the clearest signal for detecting sensitivity effects.
B.10.1 Prompt Variations
For SAYSELECT, we evaluate prompt sensitivity in gen_rule by comparing the following variations with the Adversarial bascline described in 5
(Original) You will be given list of (OBSERVATION, ACTION, REWARD) examples collected from two agents learning to solve a task. Possible ACTIONS an agent can take arc: 1, 2, 3, 4, 5, and quit. Each OBSERVATION describes the ordered sequence of actions that AGENT picks, and each ACTION describes the ACTION that AGENT 2 picks based on the given OBSERVATION, The examples are separated into HIGH REWARD and LOW REWARD examples. [[smmlles] + Output a language rule that best summarizes the strategy AGENT should follow receive HIGH REWARD, not. LOW REWARD, based the examples. Start the ruction with the prefix 'I should:.
(No Format Instruction) You will be given list of (OBSERVATION ACTION, REWARD) examples collected from two agents learning to solve task. Possible ACTIONS agent can take are: 1, 2, 3, 4, 5, and quit. Each OBSERVATION describes the ordered sequence of actions that AGENT picks, and each ACTION describes the ACTION that AGENT 2 picks based the given OBSERVATION. The examples are separated into HIGH REWARD and LOW REWARD examples.+
[samples] (sample) +utput a language rule that best summarizes the strategy AGENT 2 should follow to receive HIGH REWARD, not LOW REWARD, based on the examples.
(Low Context) You will be given list of (OBSERVATION, ACTION, REWARD) examples collected from two agents learning to solve a task. Output a language rule that best summarizes the strategy AGENT 2 should follow to receive HIGH REWARD, not LOW REWARD, based on the examples. Start the instruction with the prefix 'I should'.
(Rephrase) You've been given a list of (OBSERVATION, ACTION, REWARD) triples from two agents learning to solve task. Possible ACTIONS each agent might take are: 1, 3, 4, 5, and quit.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Each OBSERVATION refers to the ordered sequence of actions that AGENT 1 selects, and each ACTION refers to the ACTION that AGENT 2 selects based on the seen ORSERVATION. The examples are divided into HIGH REWARD and LOW REWARD examples.+(samples Desccribe the strategy AGENT uses in HIGH REWARD examples that differs from LOW REWARD examples. Start it with 'I should?.
Table 6 and show that PLLB is mostly robust to variations in prompt and always outperforms the Adversarial bascline, showing that PLLB indeed places higher weight on the content of the contrasting episodes rather than relying on any specific syntax for rule generation.
B.10.2 Temperature Variation
For SAYSELECT, our default temperature for sampling with gen_guin is 0.5. We compare with temperatures 0.1 (low diversity) and 0.9 (high diversity), and find only slight decrease in reward at the higher temperature of 0.9 (see Table 6). Importantly, across all temperatures the average reward and human- intnan--ttrrpetabllly scores are stronger than basclines, indicating robustness towards sampling temperature.
Episode
500
1000
6000
Reward (Original, temp=0.5)
0.63 0.1
0.8 + 0.1
0.96 0.03
Reward (Original, temp=0.9)
0.47 0.2
0.68 0.1
0.96 0.05
Reward Originall, temp=0.1)
0.66 0.1
0.70 + 0.09
0.96 0.02
Reward (No Format Instruction)
0.51 + 0.2
0.77 0.05
0.96 0.02
Reward (Low Context)
0.56 0.2
0.73 + 0.1
0.96 0.02
Reward (Rephrase)
0.64 1 0.1
0.81 0.2
0.96 0.02
Reward (Adversarial)
0.21 0.1
0.45 + 0.03
0.8 0.05
Table Reward across ablations to probe sensitivity of rule generation to prompt variation and temperature. Results are averages across 5 trials.
Episode
500
1000
6000
Interpretability (Original, temp=0.5)
0.77 = 0.03
0.87 + 0.04
0.97 = 0.02
Interpretability (Original, temp=0.9)
0.72 = 0.05
0.80 I 0.05
0.93 1 0.03
Interpretability (Original, temp=0.1)
0.87 = 0.04
0.90 0.1
0.95 = 0.02
Interpretability (No Format Instruction)
0.67 = 0.06
0.91 0.03
0.94 = 0.02
Interpretability (Low Context)
0.77 = 0.04
0.84 0.05
0.97 = 0.03
Interpretability (Rephrasc)
0.8 0.04
0.97 0.04
0.97 = 0.02
Interpretability (Adversarial)
0.55 = 0.03
0.57 0.03
0.65 + 0.06
Table 7: Human interpretability across ablations to probe sensitivity of rule generation to prompt variation and temperature. Results are averages across trials.
B.11 Non-Contrastive Variant
A core feature of PLLB is using contrastive episodes (examples of both low and high reward episodes) to gen rene. Here we report a non-contrastive ablation for the SAYSELECT (RL) and BIRDS (diffusionn- model based communication game) environments, where only use high reward examples in the prompt to gen ren_rule. Our results show that contrastive episodes are important for strong performance. Concretely, for SAYSELECT, the non-contrastive ablation decreases both task reward and humanninterpretabllie of the RL policy, across different lotal number of episodes seen (see Table g. Likewise, for the BIRDS lask, the non- contrastive ablation results in lower reward than PLLB, although still leads to a non-trivial improvement over the Bascline and Adversarial methods (see Table Dp. Upon closer look, we find that without contrasting episodes (i.e. negative episodes that highlight what behaviors are shared between high and low rewarding strategies), PI..B rules often capture spurious features of the tasks, such as Include details about the
25

---

Published in Transactions on Machine Learning Research (02/2026)
bird's actions. This helps in creating more vivid and engaging description.". This subsequently results in highly-specffic captions such as bird 'possibly displaying terrilorial or mating display" looking at the cameru', neither of which are relevant for any of our reward functions. Overall, these results highlight the importance of using contrasting episodes to improve PLLB rule equality.
Episode
500
1000
6000
Reward (Original)
0.63 + 0.1
0.8 + 0.1
0.96 + 0.03
Reward (Non-contrastive)
0.53 + 0.02
0.7 = 0.2
0.91 +: 0.03
Reward (Adversarial)
0.21 + 0.1
0.45 + 0.03
0.8 + 0.05
Interpretability (Original)
0.77 + 0.03
0.87 + 0.04
0.97 + 0.02
Interpretability (Non-contrastive)
0.75 + 0.02
0.8 = 0.06
0.88 + 0.02
Interpretability (Adversarial)
0.55 + 0.03
0.57 = 0.03
0.65 + 0.06
Table 8: Reward and human intrpeebttrryttaallityy scores for contrastive and non-contrastive versions of PLLB in
SAYSELECT. Results averages across 5 trials.
(a) Color
(b) Background
(c) Species
Method
Score
Method
Score
Method
Score
Baseline
1.69 + 0.5
Baseline
1.27 + 0.5
Baseline
1.59 + 0.5
Bottleneck
2.93 + 0.4
Bottleneck
2.30 + 0.3
Bottleneck
2.69 + 0.5
Non-contraasivee
2.64 + 0.3
Non-contrastive
1.78 + 0.4
Non-contrastive
2.51 + 0.5
Adversarial
1.43 = 0.5
Adversarial
1.01 = 0.4
Adversarial
1.21 = 0.4
Table 9: Noncontrastive ablation in comparison with main results for BIRDS task across 3 reward functions.
Results averaged across 5 trials.
26


Reviews:
[{"review_id": "Wfy4H5j65d", "reviewer": "Reviewer_h2d6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide a framework for learning RL agents to generate linguistic rules that capture the high-level strategies underlying rewarding behaviors, and use these rules to learn a policy that solves the task\n\nThe proposed procedure involves two steps: \n- **Rule generation:** a LM is used to generate rules that explain a set of observed trajectories. This LM is shown the task description and contrastive examples, specifically, the the top N and bottom N rewarding trajectories from the collected data. The LM is then asked to produce a concise rule of the form “I should …” that explains what differentiates successful from unsuccessful behavior.\n- **Policy Update:**  The agent then uses the rule to shape behavior in a domain-appropriate way: in the RL settings, the rule induces a rule-conditional action prior from the LM (via a softmax over logits) that regularizes Q-learning; in the non-RL collaborative reconstruction settings, the rule is iteratively appended to the speaker’s prompt to regenerate more reward-aligned instructions and improve downstream listener performance over successive rounds.\n\nBy inserting a self-generated language bottleneck between experience and learning, the authors argue that PLLB improves performance across diverse settings, promotes the discovery of reusable high-level abstractions, and yields rules that are directly useful for human-agent coordination.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper shows the following core contributions/claims: \n- Language can act as a useful bottleneck even when the optimal policy is only partially expressible in text → The strongest evidence in support of this claim is the failure of the pure LM based policy that does no Q value updates on the Maze task. \n- PLLB can steer coordination toward human-preferred conventions, increasing interpretability comparable to an instruction-oracle baseline without external instruction → This claim is supported in the SaySelect task, where PLLB and InstructRL both converge to the human-interpretable policy, while TabularQ does not, indicating that the self-generated rule plays the same steering role as the oracle instruction.\n- PLLB yields better generalization and adaptation in structured environments relative to tabular and linear baselines, with rules converging toward the latent color-to-action semantics → This claim is supported in the Maze experiments, where the authors compare PLLB to tabular Q that has no notion of color, and linear Q which uses a feature related to color. When the authors keep the semantics the same, but change the optimal policy, PLLB is the best. When the color semantics are changed, PLLB adapts faster to the new rule. Post-hoc analysis of the rules validates that rules converge to color-to-action semantics\n- Providing users with PLLB-generated rules improves performance and perceived usefulness, showing that self-generated agent rules have communicative value → This claim is supported by the human-based study on the Maze task where human subjects learn to solve the maze in fewer steps when given PLLB rules as compared to no help (control) and visual aid (visual baseline)\n- The observed gains arise from reward-aligned rule extraction rather than adding language per se → This claim is argued for mostly through the adversarial baseline for the experiments\n- Self-generated rules improve multi-agent coordination quality. → This claim is primarily shown in the non-RL settings, specifically Builder and Birds. Here, the learning is not Q-learning, but rather the  learned rule is iteratively appended to the speaker’s prompt, re-generating descriptions, and re-training or updating the listener. In such cases, PLLB rules help the speaker produce more usable and less ambiguous instructions than baseline prompting for both synthetic and neutral images.\n\nThe evidence, as it stands, supports the claims to a great extent. However, it can be strengthened by addressing the changes requested.", "audience_yn": "Yes", "audience_explanation": "I think the paper will be interesting to some communities in TMLR. It explores the intersection between iterative learning loops in sequential decision making and the use of self-generated language as an intermediate representation within that loop. The framing of language as a bottleneck that can shape subsequent learning, coordination, and generalization is likely to resonate with readers interested in representation learning, agent communication, and the role of LMs in RL.", "requested_changes": "- **Pitch:** Currently, I am currently struggling to identify a crisp core novelty that unifies the broad PLLB instantiations. Because the paper spans regularized Q-learning, instruction-augmented image reconstruction, and a bandit-style learner in Grasp, I suggest the authors articulate the common mechanism that is invariant across these settings more clearly. It would significantly strengthen the central pitch.\n- **The necessity of language as a bottleneck:**  The results currently show that a bottleneck derived from reward contrast can help learning and coordination. In this work, this bottleneck takes hte form of language rules. However, they do not yet convincingly show that language is the essential form this bottleneck must take. I would request a baseline that uses a similarly sized non-linguistic bottleneck derived from the same contrastive data (e.g.; a learned contrastive classifier or compact latent code). This would directly test whether the gains are language-specific or reflect a broader reward-aligned abstraction effect.\n- **The link between the PLLB rules and rewards:** The paper’s framing is that gains arise from reward-aligned rule extraction rather than adding language per se. I think this is the right hypothesis to test, and the adversarial bottleneck is a good start. However, given that reward design heavily shapes what the rules should look like, I worry that PLLB may also inherit and potentially amplify reward mis-specification. The paper notes that some participants preferred baseline descriptions in communication tasks because they offered more flexibility, which may indicate the reward function does not fully reflect user preferences. I would like the authors to either argue more carefully why this does not undermine the reward-alignment claim, or provide additional analyses demonstrating how rule quality tracks meaningful reward structure without simply reinforcing reward artifacts.\n- **Rule quality and robustness:** Rule generation relies on prompting an LM with contrastive trajectories. I would like to see sensitivity analyses w.r.t. the number of positive/negative examples, the sampling choice, and modest prompt variations. This would help establish how robust the method is to design decisions in the rule-generation pipeline.\n- **Contrastive Ablation:** Reward contrast appears to be the centerpiece of PLLB’s rule quality. I would request an explicit ablation comparing contrastive rule generation to a non-contrastive variant (e.g., high-reward-only summaries) across at least one representative RL setting and one communication setting. This would clarify whether contrastive prompting is essential or simply beneficial.\n- **Grasp decomposition clarity:** The grasp experiments rely on a decomposition framework. I would like a clearer breakdown of which parts of the observed gains are attributable to PLLB versus the underlying decomposition and priors. A targeted ablation demonstrating PLLB’s incremental value within this pipeline would strengthen the claim of generality.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4vR6Q0U3Ar", "reviewer": "Reviewer_oXZv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method called Policy Learning with a Language Bottleneck (PLLB) which combines reinforcement learning with LLM-based control. The authors argue that this combination is better than either technique alone. RL can struggle to generalize to new environments and often produces convoluted, uninterpretable policies. LLM-based control can generalize better and is more interpretable, but struggles to learn policies that are difficult to express with language. PLLB alternates between prompting an LLM to propose rules that explain the difference between high- and low-reward trajectories, and then updating the current policy with Q-learning steered towards actions proposed by the generated rule. The authors test PLLB across five environments: a collaborative tabular environment, a single-agent tabular environment, two visual environments, and a robotics environment. They find that PLLB generally produces more interpretable and generalizable policies compared to RL, while succeeding in some environments where LLM-based control struggles.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experiments in the paper are quite comprehensive and mostly back up the authors' claims that PLLB produces policies that are more performant, generalizable, interpretable, and inter-operable. I think the shakiest claim is \"generalization,\" which seems to be evaluated in the Maze and Grasp environments. In the RL literature, generalization often refers to the ability to perform well in new environments *with no additional training* [1, 2]. However, the experiments in this paper evaluate \"generalization\" performance after training in the new environment. I think a more accurate description for this would be \"adaptation\" or at least \"few-shot generalization.\"\n\n[1] Cobbe et al. Leveraging Procedural Generation to Benchmark Reinforcement Learning (evaluates generalization to new environments with no additional learning)\n\n[2] Fu et al. Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (refers to GAIL failing to generalize in terms of zero-shot performance in a different environment)", "audience_yn": "Yes", "audience_explanation": "There has been quite a bit of research interest in combining LLMs with classical control and RL techniques in recent years, so it seems like there would definitely be an audience for this paper.", "requested_changes": "My main request is to rework the claim of \"generalization\" to be in line with other deep RL papers.\n\nAdditionally, it seems like an undiscussed limitation of PLLB is its extension to stochastic environments. In general, methods that try to construct a policy by contrasting low- and high-reward trajectories can lead to overoptimism in stochastic environments. For example, if an agent has a choice between taking action A for a sure $10 or action B for a 10% chance of $20, the optimal action is to take action A. However, the trajectories with the highest rewards will all involve taking action B. It's unclear if PLLB will work in such a setting, and it seems worth at least mentioning that stochastic environments could be a challenge.\n\nBesides the above, I think there are a few areas where the clarity of the paper could be improved:\n * Figure 2 shows an overview of how PLLB learns in each environment, but it doesn't actually describe the environments, making it quite confusing before reading through the rest of the paper. It would be helpful to have some visual representation of the environment, what the actions are, what the objective is, etc. in order to understand what's going on. At first it's also hard to understand what the top row of the figure is (the boxes that say \"Output a rule that best summarizes the strategy I should follow\", etc.).\n * The \"adversarial bottleneck\" condition is briefly described in Section 4 but there is little explanation for why it is included in the experiments. I think I understand that the idea is to make sure that PLLB is actually using the language guidance, since if it was ignoring the language rules then the adversarial setting wouldn't hurt performance. It would be helpful to make the purpose of this condition more clear in Section 4.\n * I also found the \"oracle\" label to be unclear in the Grasp experiments. I think it makes sense for SaySelect, because in this case the oracle is given access to the rule it should follow. However, in Grasp, the paper says the oracle “always guides the policy with the rule generated from zero-shot prompting molmo-7b.\" This doesn't really seem like an oracle setting since it's not given additional information that the PLLB policy can't access. Also, how exactly does the rule guide the policy? Via InstructRL? It would be helpful to have more details.\n * Small typo: there is a missing space after LinearQ in Figure 4's caption.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vve4CLlV8C", "reviewer": "Reviewer_4Yfh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Policy Learning with a Language Bottleneck (PLLB), a framework that incorporates a language-based feedback loop for policy improvement. The core idea is to alternate between two steps: (1) `gen_rule`, where a language model (LM) is prompted with contrastive (high vs. low reward) episodes to generate a linguistic rule summarizing successful strategies, and (2) `update`, where the agent's policy is updated, regularized by the generated rule. The authors evaluate PLLB across five diverse tasks (SaySelect, Maze, Builder, Birds, and Robot Grasp Planning) to demonstrate its benefits on interpretability and generalization, etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. Here are a few claims by the paper and their evidences:\n\nClaim 1. PLLB learns more interpretable and generalizable behaviors.\n\nEvidence: Interpretability (SaySelect): PLLB agents converge to the human-preferred policy (e.g., \"choose the same number\") without external supervision, while a standard RL baseline does not. Generalization (Maze): PLLB agents transfer knowledge to new mazes with the same color semantics much faster than baselines. The \"Adaptation\" experiment further shows they can unlearn an old rule and learn a new one more effectively than a linear function approximator, which overfits.\n\n---\n\nClaim 2. PLLB enables more effective human-AI coordination (inter-operability).\n\nEvidence: This is validated through three separate human subject studies: (a) Maze: Humans given PLLB-generated linguistic rules solved mazes faster than those given visual aids or no aid, and found the rules more useful. (b) Builder: Human \"listeners\" reconstructed images more accurately and faster using descriptions from PLLB-guided \"speakers.\" (c) Birds: Human \"speakers\" provided with PLLB rules generated descriptions that led to higher-reward image reconstructions than uninformed humans.\n\n---\n\nIn all experiments, the authors use an \"Adversarial\" ablation (using corrupted rules) as a control group. It shows that the quality of the generated rule is crucial, not just the presence of any linguistic signal. This rules out the possibility that the LM is merely providing a useful inductive bias regardless of content.", "audience_yn": "Yes", "audience_explanation": "Yes, this paper sits at the intersection of several high-interest areas for the TMLR audience and would be of significant interest to a broad segment of its readers. To name a few:\n\n* Reinforcement learning researchers: They would be interested in a novel training paradigm that imbues RL agents with interpretability and generalization without sacrificing performance.\n* Natural language processing researchers: The paper presents an innovative application of LMs not just as reasoners or policy executors, but as tools for rule abstraction and distillation.", "requested_changes": "To further improve its clarity and rigor, the following changes are requested:\n\n* Clarify the `update` step. From 3.2 I understand that the update step involves a Q-learning objective. How is it implemented for LLM? And how the regularization term works in this context? More rigorous technical details are needed to improve the clarity of the method.\n\n* Scalability. Can the method be extended to domains such as math reasoning where the action space is not limited to a few choices?  Please illustrate how this method can be extended and whether the Q-learning objective is still applicable.", "broader_impact_concerns": "One potential concern is the misleading or incorrect rules. If a PLLB agent generates a flawed rule that is nonetheless convincing to a human, it could lead to the human making errors. The \"Adversarial\" ablation shows that bad rules can hinder learning, but the risk of a plausible-but-wrong rule causing harm in a safety-critical domain (e.g., a self-driving car's rule being misinterpreted by a engineer) is a non-trivial risk that should be acknowledged. Future work should include safeguards, such as human-in-the-loop verification for high-stakes rules.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)