Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
Authors: Shengran Hu
Jeff Clune
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Reinforcement learning
Imitation Learning
AI Safety
Interpretability
TL;DR: We introduce Thought Cloning, a novel imitation learning framework that enhances agent capability, AI Safety, and Interpretability by training agents to think like humans.

Abstract: Language is often considered a key aspect of human thinking, providing us with exceptional abilities to generalize, explore, plan, replan, and adapt to new situations. However, Reinforcement Learning (RL) agents are far from human-level performance in any of these abilities. We hypothesize one reason for such cognitive deficiencies is that they lack the benefits of thinking in language and that we can improve AI agents by training them to $\textit{think like humans do}$. We introduce a novel Imitation Learning framework, Thought Cloning, where the idea is to not just clone the behaviors of human demonstrators, $\textit{but also the thoughts humans have as they perform these behaviors}$. While we expect Thought Cloning to truly shine at scale on internet-sized datasets (e.g. online videos with transcripts), here we conduct experiments in a domain where the thinking and action data are synthetically generated. Results reveal that Thought Cloning learns much faster than Behavioral Cloning and its performance advantage grows the further out of distribution test tasks are, highlighting its ability to better handle novel situations. Thought Cloning also provides important benefits for AI Safety and Interpretability, and makes it easier to debug and improve AI. Because we can observe the agent’s thoughts, we can (1) more easily diagnose why things are going wrong, making it easier to fix the problem, (2) steer the agent by correcting its thinking, or (3) prevent it from doing unsafe things it plans to do. Overall, by training agents $\textit{how to think}$ as well as behave, Thought Cloning creates safer, more powerful agents.

Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
Shengran Hul.2 srhuêcs. ubc.ce
Jeff Clune
j jclune@mmail 11. com
'Department of Computer Science, University of British Columbia
Vector Institute
Canada CIFAR Al Chair
Abstract
Language is often considered a key aspect of human thinking, providing us with exceptional abilities generalize, explore, plan, replan, and adapt tions. However, Reinforcement Learning (RL) agents are far from human-level performance in any of these abilities. We hypothesize deficiencies is that they lack the benefits of thinking in language and that we can improve AI agents by training them to think like humans do. We introduce novel Imitation Learning framework, Thought Cloning, where the idea is to not just clone the behaviors of human demonstrators, but also the thoughts humans have as they perform these behaviors. While we expect Thought Cloning to truly shine at scale on internet-sized datasets of humans thinking loud while acting (e.g. online videos with transcripts), here we conduct experiments in a domain where the thinking and action data are synthetically generated. Results reveal that Thought Cloning learns much faster than Behavioral Cloning and its performance advantage grows the further out of distribution test tasks are, highlighting its ability to better handle novel situations. Thought Cloning also provides important benefits for AI Safety and Interpretability, and makes it easier debug and improve AI. Because we can observe the agent's is thoughts, we can (1) more easily diagnose why things are going wrong, making it easier to fix the problem, (2) steer the agent by correcting its thinking, or (3) prevent it from doing unsafe things it plans to do. Overall, by training agents how to think as well as behave, Thought Cloning creates safer, more powerful agents
situa-
reason for such cognitive
Introduction
Language may be the key to what separates humans from all other animals, endowing us with an amazing level of general intelligence (T/). Crucially, the benefits of language are not confined to improving our ability to communicate with others: language also helps think better 2 We first describe the benefits of agents that can understand language (a common topic in AI) before moving to the benefits of agents that think in language (a topic that has received far less attention).
There are many benefits that arise if our agents can understand language. Doing so is erucial for agents to generalize to new tasks we want them to perform. This is because it is drastically more sample efficient if one can tell agent what the task is, rather than requiring the agent to figure the task through trial and error 5 6]. Moreover, agents that can understand language allow us to detine new tasks at test time without having to anticipate every wish we might eventually have for
'The source code, model weights, and dataset are available
Thought - Cloning
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

our trained agents 77. That is in contrast to conventional dand-designed task descriptions, which can be vast, but still place constraints on what we can ask an agent to perform [].
While the benefits of agents that can understand language are commonly discussed, there has been relatively little discussion in AI, especially in Reinforcement Learning (RL), regarding the many benefits of agents that think in language. While it remains debated whether humans exclusively think in language 22 [0], many scholars believe that natural language is intricately woven into our thought processes, conferring distinct cognitive advantages: thinking in language helps humans generalize, extrapolate, adapt to new situations, combine old knowledge in new ways, explore, plan, replan when necessary or beneficial, transfer knowledge via analogies, and the list goes on [2 Despite these benefits, AI agents rarely, if ever, think, at least not human language. While neural networks have internal vector activations that can be considered thinking, many hypothesize that there are specific benefits to thinking in the discrete, symbolic form of language (e.g. combining ideas in an exponential number of ways) 66 1 12.. meaning that agents that think in language might learn faster, perform better, and generalize better than non-lingual agents.
In addition to agents being more capable, there are major benefits regarding AI Safety and Inter- pretability that arise when agents think in our language. If one can watch an agent think during training, one can recognize deficiencies in skills or values that can be improved, or one could decide the agent is not ready to be deployed. During testing, one can constantly scan the thoughts of the agent and intervene when the agent plans to do something undesirable. For example, if an agent thinks "My goal is to take my passenger to the store as fast as possible so I will run through this red light without stopping" one could intervene to stop that behavior ahead of time. Furthermore, watching agents think enhances the steerability of agents. If an agent confused when solving challenging tasks, one can inject their thoughts into the agent to help it solve the task in a desired way, A final major benefit of agents that think in human language is it makes it easier to train more capable, safer AI agents. One can spot why things are not working, instead of just seeing that they are not working, and that provides ideas for how to debug and or improve AI training.
For all these reasons, adding the ability of AI agents to think in language could produce many significant advantages, and we suggest that the most effective way to achieve this goal is by imitating human thinking. Humans do not acquire thinking skills in isolation: instead, they are learned in part through demonstrations and feedback provided by teachers 2 13-15) As such, promising method is to have agents learn from demonstrations where humans think out loud while acting. This approach is distinct from existing works that leverage pre-trained Large Language Models (LLMs) for planning [16] because such LLMs are not trained on data where humans think out loud while acting. Thought data, such as YouTube videos and transcripts [18 19], contains millions of hours of people talking out loud while performing tasks, revealing the thinking behind their actions, planning, decisions, and replanning. such as when they play video games [9] This thought data is greatly valuable and widely available (Section 2). but has not yet been extensively explored, and this work hopes to encourage further research into the use of thought data to teach thinking skills to agents.
Provided we can solve the real, significant challenges of AI Safety and existential risk (0 24 there are tremendous gains to be had by creating more powerful Al or even AGI. In this paper, we propose a novel Imitation Learning framework, Thought Cloning, where agents not only learn to act from human demonstrations, as in Behavioral Cloning 225, but also learn to think from demonstrations where human think out loud while acting. Although we expect Thought Cloning to truly shine when trained on vast online datasets of synchronized human thoughts and actions, this paper validates the concept with synthetic thought data in challenging domain, Baby Al (6). Our experimental results illustrate that Thought Cloning outperforms Behavioral Cloning, even when Behavioral Cloning agents have the ability to think (in latent vectors), but have to learn that skill without the supervision of thinking provided by Thought Cloning. We also demonstrate that Thought Cloning generalizes better than Behavioral Cloning in -uitt---istribuutinn tasks in both zero-shot and fine-tuning settings. Finally, we provide empirical evidence for the previously discussed advantages of Thought Cloning in terms of Safety and Interpretabilityy where unsafe behavior can be near perfectly stopped before execution. All told, the results are promising and offer a glimpse of the enormous potential of Thought Cloning to not only make AI smarter, but also safer and more interpretable.
2

---

Thought History
Thought Cloning Agent
Lipper the unean and
Next Thought
Laplor the unain aec
Thought Generator
Thoupht
Cloning
Mission
Mayl
Diseevottin
Generator
Action
Forward
Actien
Action
Lona
Figure 1: Overall framework for Thought Cloning (TC). The TC agent has two components: the Thought Generator and Action Generator. At each timestep, the TC agent receives an observation, a mission, and a history of thoughts as inputs. The Thought Generator generates thoughts, and the Action Generator generates actions conditioned on these thoughts. Generated thoughts and actions are compared to the ground truth from the demonstration dataset to calculate the loss.
2 Proposed Method
Conventional Imitation Learning methods [27 288 such as Behavioral Cloning 225 strive to construct a policy that accurately replicates the distribution of behavior in given dataset of demon- strations. However, our proposed framework, Thought Cloning, diverges from this approach by aiming to teach agents how to also think while acting, utilizing synchronized dataset of hu- man thinking. The thought dataset, denoted as D M comprises series of trajectories, D, (m, {(o,, lhe, at) {=1)) Each trajectory encompasses mission, m, defined in natural language, along with an observation Oz, an action dN and a corresponding thought Lhg at each timestep, L. Such datasets are widely available online. For example, by inferring action labels from Youtube videos with VPT [19] and then retrieving the corresponding transcripts, we can obtain a thought dataset that contains both human thinking and action 19 I8 In such a dataset for Minecraft, a thought like "I need to gather wood to build a shelter before nightfall" might correspond to the player moving towards tree and collecting wood. It is worth noting that noise an inevitable aspect of online data. For instance, commentary like "please subscribe to my channel" would be commonly found in YouTube videos. However, we believe this challenge could be effectively mitigated with proper data preprocessing. An example from MineCLIP [8] shows that careful heuristic-based preprocessing of the captions can make data clean enough to train multi-modal models (e.g. a simple heuristic focusing on domain-specifer vocabulary could remove most off-topic text). More promisingly, one could use language models to filter out off-topic data, either by removing any off-topic comments or determining that some videos should be excluded because not enough of the commentary is helpful. Additionally, even if noise is present, recent ML history shows that-at scale-such "noise" does not prevent learning the "signal": examples of this occurring internet-scale data include GPT [29]. CLIP (30), and VPT 19). While not a perfect simulation of online caption noise, as described later, add noise to the synthetic thought data and Thought Cloning works well despite it (Section 3.1)
In the Thought Cloning training framework, agents learn to produce natural language thoughts at each timestep and subsequently condition their actions based on these generated thoughts. This learning process gives rise to a bi-level architecture (Fig. TT. The architecture comprises an Thought Generator responsible for thought generation, and Action Generator tasked with executing actions based on the thoughts generated by the Thought Generator. While different choices of what to condition the Thought Generator and Action Generator are possible, in this work, for particular trajectory of length in the thought dataset we minimize:
min Pi
log no,, (thelm, for (= log ne, (a, Im, (or)
(1)
Here, 8y and O represent the weights for the Thought Generator and Action Generator; a represents the coefficient for Thought Cloning loss; th, o, a, and m denote thought, observation, action, and mission, as previously described. The first part the loss the Thought Cloning loss, where the Thought Generator is conditioned on the history of thoughts, observations, and the mission, to predict the thought. That thought is then compared with the ground truth thought in the dataset. The second part is the action loss, where the Action Generator is conditioned on the current thought, the history
3

---

BabyAl grid maze room
Agent's Thoughts Actions
Stop
(..after steps)
Thought: open blue door explore
Actions: (forward, ... open]
purple complete mission Actions: [forward]
Thought: remove blocking object purple Actions: Actions:[pickup, drap)
Thought: go to purple box complete goto mission Actions: [left, left]
purple box go the green door on your
Figure 2: Left: BabyAl [26] environment example. The enviromment contains various colored items (ball, key, box, door). The agent can pick up, drop, and move objects or open and close doors, while locked doors can only be unlocked with color-matched keys. The agent can observe the x grid cells in front of it, which can be blocked by walls and closed doors. Right: An example from trained Thought Cloning agent planning and replanning. The mission requires reaching the purple box (highlighted), but purple ball blocks the way. The agent's thoughts and actions show replanning when encountering the obstacle, removing it, and resuming the previous goal.
of observations, and the mission and predicts the action to do, and we then calculate the loss by comparing the predicted action the ground truth action in the dataset.
For more complex or large-scale scenarios, the Thought Generator can be implemented with pre-trained Vision-Language Models (VLM) either zero-shot fine-tuned BT. while the Action Generator can be trained from scratch or adapted from existing language-condiigg conditioned con- trollers in the target domain 6 16] In this paper, base both components on the BabyAI 1.1 model architecture [32], which utilizes a memory-augmentee
architecture- an LSTM 63 - address the partial observability challenge. The model also employs FiLM [34] for modality fusion, effectively combining visual and text input. The detailed architecture adopted in this paper can be found in Supplementary Material A While all models in this paper trained from seratch, we anticipate that the utilization of pre-trained models in complex domains will be beneficial.
Experimental Results
3.1 Domain and Synthetic Thought Data
This paper employs BabyAl [26], simulated partially observable 2D gridworld domain. We focus on the most challenging environment, BossLevel, BabyAI. An overview of the domain is shown in Fig. (left. Each BabyAl environment consists of randomly generated room layout, item configuration, and mission described in natural language, sampled on an environment distribution. Colored items (balls, keys, boxs, doors) and the initial position of the agent are randomly distributed across x 20 grid world containing nine 6 rooms. Missions comprise four possible tasks (GoTo, PickUp, OpenDoor, PutNextTo), connected by then/after and and (with or without ordering constraints). GoTo and PickUp require agents go or pick up an object; OpenDoor requires agents to open or unlock door; PuINex/To requires the agent to pick up object A, find object B, and drop A next to B. The mission may implicitly require the agent to open or unlock doors to find the target objects. Relative directional instruction in the mission, e.g., your right, is based on the agent's initial position. An environment is solved when all tasks the mission are completed. The agent's 's observation consists of the 7 x grid cells in front of the agent, except the agent cannot see through walls (Fig. yellow square). This work features the slate-based observations provided by Baby 266. Each grid cell in the 7 x 7 observation is represented by three integers: [the item ID, the color ID. and a status code], resulting in a x 7 x observation matrix. The status code is for closed doors and for locked doors, with 0 for open doors and other items. Occluded grid cells assigned an item ID of 0. The agent's 's action space includes [left, right, forward, pickup, drop, toggle door (unlock, open, close)].
The key challenges in Baby revolve around partial observability, hard-to-explore mazes, complex missions in natural language, and long honghoorizon planning. The 7 x 7 observation field is limited

---

compared to the 27 x 27 maze, and the agent cannot see through walls and closed doors. The maze containing multiple closed rooms is difficult to navigate and explore as the agent needs to find target items across multiple closed (even locked) rooms. The missions are challenging hecause (1) they are described in natural language and (2) they can consist of multiple tasks, each requiring complicated navigation and actions. Combining all these factors results in long horizon, with hundreds or even thousands of steps needed to solve a single environment.
One significant advantage of Baby is that it provides an Oracle Solver (named BOT in 26 capable of generating step-by-step solutions for any given environment. This is achieved through hand- coded rules and an internal stack machine to generate plans for solving environments. In our work, we translate the Oracle Solver's internal states into natural language thoughts with pre-defined rules. For example, if the inner logic is to open a red door to explore the room, the translated thought will read, "open red door to explore". This translation process is combined with the generated demonstrations to synthesize the thought dataset with million trajectories. To make the dataset more realistic, noise is added, with a % chance of adding a random noisy segment at each timestep. consisting of a random thought and several random actions, with random length sampled from 1-6. A trajectory with example noise is shown in Supplementary Material
3.2 Experiment Setup
To verify the effectiveness of learning to think, we compare our Thought Cloning (TC) approach to the classic learning algorithm, Behavioral Cloning (BC). BC shares most of its architecture with the Action Generator of TC (Fig. [. and because it is trained only on action loss, it does not encode thought like the Action Generator of TC. Additionally, since BC has fewer parameters than TC, we introduce an ablation variant called TC w/o Imitating Thought that is trained without the Thought Cloning loss demonstrate that TC's superiority is not solely due its larger number of parameters. This variant's architecture is identical to the TC architecture, except for a minor architectural difference where the latent vector from the Thought Generator is directly input to the Thought Generator as thought. This adjustment is necessary because this variant is not trained on the Thought Cloning loss, so we do not have per-word supervision. To train these parameters, we thus need to train them based on how these "thoughts" contribute to the action loss (i.e. they are trained end-to-end to predict actions). If we sampled words (as in Thought Cloning), we could not train these parameters end-to-end because hard sampling of words is non-differentiable, could not flow backward through this operation. Thus, we make one small change order to allow the parameters to be trained, which is to pass the logits of the Thought Generator directly into the Action Generator, which is a differentiable operation. We feel this is the closest and fairest control possible to Thought Cloning, allowing virtually the same architecture and the same number of parameters, but not including the Thought Cloning innovation of exploiting human thought data.
gradients
The training iterates for 8 epochs on the 1 million episode dataset, corresponding to a total of 7 x 102 training frames. The Thought Cloning loss parameter a (Eq. [T) is set to 2. During training, we employ teacher-forcing when decoding thoughts. It conditions the Action Generator on the ground truth thoughts from the dataset. The reacher-forcing ratio linearly decreases from 100% to 0% during the training process. Producing all the main results in the paper took about ten A40 GPUs for one week. More details on training can be found in Supplementary Material A
Our training setup is based on BabyAl 26 32
[35], which is adopted
In our experiments, the performance of agents is evaluated based on their success rate in held-out test environments. Success for an environment is defined as the completion of all specified tasks in the mission. By controlling random seeds, all test environments are unseen during the training process. All experiment results from Sections 3.3 3.4 and 3.5 are calculated from five independent runs. The success rate results in Section 333 are obtained by testing agents on set of 512 sampled environments. In Section 3.4] agents are tested in a larger set of 1,024 test environments. During the testing phase, the TC agent has identical observations as the BC agent, i.e. it has no extra information.
3.3 Imitation Learning
In this section, we show the main performance results of training TC, BC, and TC w/o Imitating Thought. The results illustrate that TC learns faster than BC, where BC requires orders of magnitude more time to achieve a performance similar to TC's early-stage results, and TC ultimately outperforms BC at the end of training (Fig. 3). The outperformance of TC compared to BC at 25%, 50%, 75%, and
5

---

100%s
80%
Rate
60%
40%
Thought Clarinng
Behavioral Cloning TC w/o Imitating Thought
20%
Training Frames
Figure 3: Training progress comparison of Thought Cloning (TC), Behavioral Cloning (BC), and TC ablation variant without the Thought Cloning loss. The BC architecture is identical to the Action Generator of TC and the TC w/o Imitating Thought has the same architecture as TC, without the TC loss. BC and the ablation variant are trained solely with the action loss (which leads to some minor architectural differences, see Section 3.2) The error hars are the 95% confidence interval from five runs of experiments. The results indicate that TC learns faster than BC and also outperforms it. Furthermore, the comparison between TC and TC w/o Imitating Thought demonstrates that the superiority of TC is not simply due to having more parameters.
100% of the way through training is statistically significant, as confirmed by the Mann- Kann-Whitney U test, with p = [0.012, 0.008, 0.021, 0.008] < 0.05. These results support our hypothesis that natural language can help the agent learn explore and plan.
Another comparison is between TC and an ablation variant TC w/o Imitating Thought that shares the same architecture with TC, but without the Thought Cloning loss in training. The results show that TC also substantially outperforms TC w/o Imitating Thought (Fig. 3b. Similar to the previous, the results statistically significant (p [0.008, 0.012, 0.008, 0.008) 0.05). The results reveal that TC's superior performance is not solely due to a larger number of parameters than BC, and also supports argument that learning from human thought boosts agent's ability think.
An example of a TC agent planning and replanning is shown in Fig (right). After opening the blue door, the agent discovers the target (a purple box) within its observation and thinks about going to to complete the task. However, the agent realizes that purple ball is blocking its path. A smart replan emerges here, with the agent inserting a plan to remove the ball. The agent achieves this subgoal by picking up the ball in its way, finding an empty space, and then dropping it. After completing this new, necessary, intermediate task, the agent resumes its original mission to go the purple hox and successfully solves the environment. From this example, we can see that by thinking like humans in natural language, the agent demonstrates successful planning and replanning abilities. We also see the interpretability benefits, as it easy to follow along and understand why the agent executes certain actions.
3.4 Generalization to Out-of-Distribution Environments
This section compares the generalization abilities of the TC and BC agents by testing them on environ- ments that increasingly out of distribution. We define the distribution of environments with two difficulty dimensions: Behavioral Difficulty and Cognitive Difficulty. Behavioral Difficulty is based on the length of the action sequence required to solve the environment (provided by Oracle Solver, see Section 3. The simplest environments require about 20 steps, while the most challenging environments require more than 500 steps. Cognitive Difficulty reflects the complexity of the mis- sion, with more difficult environments requiring stronger planning abilities to complete complex BabyAI tasks. The definition of Cognitive Difficulty is adapted from the maxStep parameter environments (26) and is given by (# of [ [ooTo, PickUp, OpenDoor) + 2 x # of { (PutNexITo} - # of ordering constraints). The PutNextTo task is assigned a higher weight because it involves combination of picking up, navigating, and dropping, making it the most challenging task among the four. The range of cognitive difficulty spans from (simplest) to (most difficult). In the training distribution, the environments exhibit means and standard deviations of Behavioral and Cognitive Difficulties 84.2 + 68.8 and 2.7 1.6, respectively. In this paper, we define -ustr--disttibuttion (OOD) environments as those with Behavioral Difficulty > 175 Cognitive Difficulty > 4, each being approximately more than one standard deviation away from the mean. The furthest OOD environments, with Behavioral Difficulty greater than 425 Cognitive Difficulty of 9, had

---

80%
60%
40%
20%
Behavioral Dimicuity
Cogritive Dificuty
(a) Zero-shot success rates of agents on environments that are increasingly out of distribution.
TC zero shot
BC zero shot
TC TCe tune improvement
BC fine tune improvement
100%
80%
Success Rato
60%
20%
175
225
275
325
375
425+
Behavioral Difficllty
Cognitive Difficulty
(h) Success rates after fine-tuning agents on out-of-distribution environments
Figure 4: The zero-shot and fine-tuning success rate of Thought Cloning (TC) and Behavioral Cloning (BC) agents on environments that are increasingly out of distribution. Behavioral and Cognitive Difficulties defined by the length of the solutions environments and the mission complexity of environments respectively (Section 3.4 The error bars in bar and line plots are the 95% confidence interval from five runs of experiments. (a): The gray region indicates the training distribution. The Oracle Thought + TC Learned Control refers to the TC agent with oracle high-level thoughts. The results demonstrate TC generalizes much better than BC. They also illustrate that with a more powerful Thought Generator trained from vast human thought data, the agent should become drastically more capable. (b): TC much better at adapting to novel situations than BC.
less than 5.7 10-5 and 1.6 10 probability of being sampled during training (calculated with rejection sampling). For testing both in-distribution and out-of-distributio environments, we sample various sets of environments that extend away from the distribution in terms of both Behavioral and Cognitive Difficulty, and then evaluate agents on these sets. For Cognitive Difficulty, we sample sets of environments across the full range of Cognitive Difficulty levels 1-9. For Behavioral Difficulty, sample sets of environments within intervals of 50 (e.g.. 125-175, 175-225, etc.), starting from 25. Environments with Behavioral Difficulty >> 425 are grouped into
First, we test the zero-shot performance of TC and BC agents in OOD environments. The results show that the TC agent substantially outperforms the BC agent with environments being increasingly out of distribution (Fig. 4aab, and the results are statistically significant across all testing difficulties (Mann-Whitne) U test 0.05), thereby supporting hypothesis that language utilization can enhance agents' generalization capabilities. Moreover, we observe that the Oracle Thoughts TC Learned Control achieves near-optimal performance
the most challenging environments. This indicates that the current limitation of TC performance lies high-level thinking. As we seale approach to leverage internet-sizer datasets of human thinking, the high-level thinking capability expected to improve substantially, thereby enhancing the power of the TC agent.
Next, we investigate how well the agents adapt to new situations by fine-tuning them OOD environments. We fine-tune the TC and BC agents on the corresponding environments for 15 epochs, with the same settings described in Section 3.2 The results demonstrate that the TC agent is better at adapting to OOD environments (Fig. 4bb. The superiority TC over BC statistically significant across all testing difficulties, as supported by the Mann-Whitne U test 0.05, with the exception of Cognitive Difficulty 4, where both methods already achieve near near-perien perfect performance. The results support our argument that language can better assist agents in adapting novel situations.
3.5 Safety and Interpretability
The ability to observe the agent's thought process gives our model a high degree of interpretability. To empirically assess the interpretability of TC, we introduce metric named the Future Action

---

F-aure Puur-Adio Deduraton Scone
60%
No MaareeSee
50%
Cognittee Dificuyy
40%
Thinkng Daaticion
30%
10%
120% Tocching rad hara
COEN
11a zie zha vin
cha aie se ide
Pidiingpp up ba
Picking up ro/qquetted Rtmm
Dehastoral dificcayy
Unoate Detaviors
(a) Interpretability of Thought Cloning agents
(b) Effectiveness of the Precrime Intervention
Figure 5: (a): A heatmap illustrating the Future Action Declaration Score, a metric designed to evaluate the interpretability of Thought Cloning agents (Section 3.5). The I and y axes denote various levels of difficulty. Each cell represents region of sampled environments, with the color intensity reflecting the mean score. Brighter cells indicate a higher degree of match between the agent's declared thoughts and subsequent actions. The green square denotes the training distribution, while the rest of the regions are out of distribution (Section 3.4). The results illustrate the robust and consistent interpretability of Thought Cloning agents. (b): A bar chart demonstrating the effectiveness of the Precrime Intervention mechanism, which is to halt the Thought Cloning agents upon detecting dangerous plans in their thoughts and thus prevent unsafe hehaviors. We show three tests (x axis) where (1) touching red items, (2) picking up balls, and (3) picking up requested items were declared unsafe. We report the fraction of episodes where unsafe behaviors occurred (y axis). The error bars are the 95% confidence interval from five runs of experiments. The results show that Precrime Intervention effectively eliminates almost all unsafe behaviors.
Declaration Score. This metric quantifies the fraction of times when an agent, preparing to execute an action other than navigation, declares this impending action in its thoughts beforehand. In the training distribution, TC agents performed exceptionally well (green square in Fig. Sap. Interestingly, TC agents also scored near-perfectly across all out-of-distribution environments (rest of Fig. Sap, demonstrating the robust and consistent interpretability of our model even under novel, out-of- distribution situations, which an important property for AI safety and interpretability.
Due to its high degree of interpretability, Thought Cloning allows a simple method that can consider- ably enhance AI safety. We call it Precrime Intervention. In practical settings, an agent might employ dangerous or undesirable strategies to accomplish challenging tasks. However, because Thought Cloning features such strong interpretability, we can simply halt the agent upon detecting dangerous thoughts and thereby prevent the unsafe behavior it was planning to conduct. Additionally and impor- tantly, Precrime Intervention does not require any changes to the weights of the model. If we learn or decide after training that a certain behavior is unsafe or undesirable, Precrime Intervention can still prevent it. The same flexibility allows different definitions what allowable and unsafe behavior in different settings (e.g. in the presence of adults vs. children or customized to the preferences of different countries with different regulations). To demonstrate this flexibility and test what extent Precrime Intervention works, we conducted three separate tests, where we declared three different behaviors as unsafe (1) touching any red item, (2) picking up any ball, and (3) picking up the object the agent is being asked to pick up in its mission. The last one is particularly interesting because the agent has a strong prior to want perform that action, which Precrime Intervention has combat. We report the fraction of episodes where such unsafe behaviors occurred with and without Precrime Intervention (Fig. (bb. Remarkably, Precrime Intervention almost entirely eliminates all unsafe behaviors, thereby demonstrating the promising potential of TC agents in advancing AI safety.
Moreover, the interpretability of the model also greatly aids in diagnosing problems, thus simplifying the development of more capable and safer AI. This feature actually proved beneficial during the development of this paper. Initially in our development, the TC agent showed promising performance in training, but frequently failed during testing, repetitively oscillating between incorrect thoughts (plans) without actively exploring new ideas. This observation helped us to recognize that, because we had trained with teacher forcing throughout with oracle (i.e. perfect) thoughts. the agent had never practiced having incorrect thoughts, and thus had never practiced recovering from them by trying alternate ideas. Thus, at inference time when it had to generate its own thoughts, which are sometimes incorrect, it did not know how to recover. We then instead tested an immediate transition from reacher-forcing 100% auto-regressive sampling and training (i.e. from 100% reacher-forcing
8

---

one training step to 0% on the next). but the agent generated too many nonsensical thoughts. Thanks to the model's 's interpretability, we were able to recognize the situation and try an alternative strategy that worked well, and is the one we report results for in this paper (Section. (.3): we gradually decay the teacher-forcing rate (fraction) during training, which dramatically improved performance. Supplementary Material [
contains more details about this example.
Lastly, TC enables steerability, including helping agents when they are stuck. That is because the actions of TC agents are conditioned on their thoughts, and we can manually inject alternate thoughts to have the agents do what we wish. The TC agent, when provided with Oracle high-level thoughts, is capable of near perfect performance across almost all environments (Fig. 4aj, which provides evidence that steering agents is possible and effective.
4 Related Works
4.1 Planning in RL with Language
Recent work leverages the reasoning capability, the ability to flexibly combine abstractions, and the interpretability offered by natural language to address high-level planning challenges in real-world domains. We augment this approach by enabling agents to think in language, faciiiitatigg the capability of agents, Al Safety, and Interpretability. There are two major categories of works in the literature that enable language planning. The first involves Hierarchical RL methods, where the language represents the hierarchy [36,40] 40]. However, the planning space in these works is constrained to a pre- subgoal set, limiting their generalization to novel scenarios and preventing them from utilizing the reasoning and powerful commonsense found in pre-trained L.LMS 29 41 42). The second category of work involves pre-trained L.LMs that generate plans in language for RL systems. Earlier works allow the LLM to predict step-by-step plans for a specific task. However, these works are
[6 17 open-loop methods, as the LLM cannot perceive the environment while acting, and thus cannot adapt and change once things do not go according to plan, which is a crucial capability in complex environments. Some recent approaches have developed closed-loop methods to provide LLMs with dynamic information for planning [17
43 44]. While these works show exciting performance in oracle from the environment or complicated captioning models. The work most relevant to our vision is PALM-E 31, in which pre-trained Vision-Languse it to recognize patterns from observations directly. However, the works mentioned above were not trained data of humans thinking and acting, meaning they do not benefit from learning from human thought demonstrations how to do things like plan, replan, create high-level goals and the subgoals required to achieve them, and the many other benefits of thinking intelligently during acting.
different environments, their closed-loop loop feedback mechanisms for the LLMs either rely
Model is adopted the planner, allowing
Apart from embodied domains, many studies utilize LLMs as agents [45] operating in text domains, such coding 46 47 or using API tools 48). Many LLM agents benefit from the reasoning and planning capabilities of LLMs [8 49]. Some even allow LLMs to replan based on new information [46 477. However, none of the methods described above directly learn to think while acting by imitating human thought data. Thus, unlike Thought Cloning, they do not benefit from learning from human thought demonstrations. Within LLM agent research, ReAct (50) is the most relevant to approach. It prompts LLMs to first generate reasoning and then conditions actions based on such reasoning text. The paper also includes an experiment that fine-tunes LLMs with demonstrations that contain both reasoning and actions. However, obtaining such demonstrations is challenging, as highlighted in the ReAct paper (50), and the paper does not provide insights on overcoming this data bottleneck to scale the method.
4.2 Learning from Dataset Aligning Action and Language
Several studies have recognized the value of datasets that align action with language. DIAL [5] employs such a dataset to train iagguage-condttoons agents with Behavioral Cloning, achieving impressive results in real-world robotic tasks. However, it limited by a pre-defined instruction set. Another work, (SL)² [52], generates hierarchical dataset for agents to learn from, demonstrating superiority challenging 3D simulation domain, but has the drawback discussed in the previous section of being loon- Finally, in the study most similar to our own, Hu et al. 5x collected a dataset from two human players collaborating on an RTS game, However, the agent (53) not language conditioned, which limits its potential to learn to do any task (c.g. arbitrary tasks

---

requested of it in natural language). Similarly, work concurrent to ours constructed a dataset with BabyAl oracle plans [54]. However, their architecture, unlike ours, is not compatible with most pre-trained models, making ours more able to harness new, powerful foundation models to tackle increasingly complex challenges. Additionally, although previously mentioned two methods [53] employ learning frameworks similar to ours, they do not explore the full potential of learning from datasets that align action with language, particularly in terms of the resulting benefits in terms of generalization, AI Safety, and Interpretability.
Discussion and Conclusion
Our research findings are focused on two main areas. First, our Thought Cloning (TC) agent demonstrated superior performance compared Behavioral Cloning (BC), effectively showcasing its capabilities in generalization, exploration, planning, replanning, and adaptation to various situations. Second, we presented empirical evidence underscoring the benefits Thought Cloning provides in Al Safety and Interpretability. The robust interpretability of the TC agent not only help developers in diagnosing AI systems, but also contributes to Al safety, as evidenced by mechanisms such as Precrime Intervention. Our empirical results on steerability further spotlight the potential of TC agents in effectively collaborating with humans to tackle complex tasks.
We utilized a synthetic dataset and trained a model from scratch as a proof of concept. However, the full vision for the TC framework will be when Thought Cloning agents are trained on internet-scale datasets of humans thinking out loud while acting, such as YouTube videos and their narration [19] (whether in closed caption transcripts or directly from audio). Consider the prospect of an agent that has both learned to think and act like humans in huge variety of settings. Much like the thoughts of human children are guided by teachers, our agents could become skilled at planning, replanning. reasoning, and explaining their thinking to us (either via their outputs or because we have the unique ability to observe the thoughts in their minds). The vision for utilizing internet-scale datasets is also supported by the empirical results in Section 3.4 which suggest that the current bottleneck in agent capability is its high-level thinking, a skill that could be enhanced by scaling to vast online data. Additionally, we trained all models from scratch in this paper. In more complex domains, we anticipate that the utilization of pre-trained Transformer-bas based models could be beneficial.
Moreover, Thought Cloning can also improve Foundation Models (FMs) by enabling a separate "thought channel" where they can output thoughts that get fed back in when they are planning and answering. Recent studies indicate that FMs can benefit from contexts they self-generate, such as reasoning steps [ [50 Thought Cloning can amplify this advantage by training FMs with Thought Cloning data which includes not only the answers or solutions (action data) but also the reasoning behind them (thought data). For example, a human programmer needs the ability to think when developing software, but in a way where those thoughts are not part of the final output. However, current LLMs are not trained this way. In addition to performance gains, adding such a thought channel to FMs allows all of the Al Safety and Interpretability advantages of Thought Cloning, including Precrime Intervention, to be added to the world changinh FM technology.
Of course, there are also risks associated with such agents. As occurs with LLM pretraining and other forms of Behavioral Cloning, Thought Cloning could inadvertently inherit undesirable human flaws, such as speaking falsehoods, providing false yet persuasive rationalizations, or being biased. Alignment techniques are being constantly improved to address these challenges (55). However, even improving AI agent safety up to the level of a (flawed) human would be a major advance, even if the resulting system is not perfect. Additionally, distinguishing feature of Thought Cloning provides the ability to interpret and prevent these flaws from culminating into actions, making TC a more favorable approach in this regard. (See the experimental analyses on the effectiveness of Precrime Intervention Section 3.5). Our intent was to inspire researchers to advance this method and or test it in real-world scenarios, but the method itself should not be considered as a comprehensive safety solution for real-world systems.
In conclusion, this paper has introduced Thought Cloning, where agents not only simply learn to act from human demonstrations, as in Behavioral Cloning, but also learn to think from demonstrations where humans think out loud while acting. Through Thought Cloning, we have illustrated how an agent can become more capable, interpretable, and safe by imitating human thinking. This work facilitates the training of increasingly powerful agents and opens up numerous avenues for future scientific investigation in Artificial General Intelligence, AI Safety, and Interpretability.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Vector Institute, the Canada CIFAR AI Chairs program. grants from Schmidt Futures and Open Philanthropy, an NSERC Discovery Grant, and generous donation from Rafael Cosman. We thank Aaron Dharna, Ben Norman, and Jenny Zhang (sorted alphabetically) in our lab at the University of British Columbia for insightful discussions, and Michiel van de Panne for helpful commentary.
References
[1] Robert C Berwick and Noam Chomsky. Why Only Us: Language and Evolution. MIT press. 2016.
[2] David Premack. Is language the key to human intelligence? Science, 303(5656):318-320) 2004.
[3] Noam Chomsky al. Language and Mind. Cambridge University Press, 2006.
[4] Steven Pinker. The Language Instinct: How the Mind Creates Language. Penguin UK, 2003.
[5] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, 2021.
[6] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Em- bodied reasoning through planning with language models. In arXiv preprint 2iXi::207.006000 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901 2020.
[8] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, al. Open- ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021.
[9] Ben Alderson-Day and Charles Fernyhough. Inner speech: development, cognitive functions, phenomenology, and neurobiology. Psychological bulletin, 141(5):931, 2015.
[10] Romain Grandchamp. Lucile Rapin, Marcela Cerrone-Bertolotee Cédric Pichat, Célise Haldin, Emilie Cousin, Jean-Philippe Lachaux, Marion Dohen, Pascal Perrier, Maëva Garnier, et al. The condialint model: Condensation, dialogality, and intentionality dimensions of inner speech within a hierarchical predictive control framework. Frontiers in Psychology, 10:2019, 2019.
[11] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-Thowe prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
[12] Gary Mareus. The algebraic mind, 2001.
[13] Lev Vygotsky et al. Interaction Between Learning and Development. Linköpings universitet, 2011.
[14] John D Bransford, Ann L Brown, Rodney R Cocking, et al. How People Learn, volume 11. Washington, DC: National academy press, 2000.
[15] Steve Olusegun Bada and Steve Olusegun. Constructivism Learning Theory: paradigm for teaching and learning. Journal of Research & Method in Education, 5(6):66-70, 2015.
[16] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi. Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang,
11

---

Kuang-Huee Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jomell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do As I Can and Not As Say: Grounding language robotic affordances. In arXiv preprint arXiv:2204.00000 2022.
[17] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pages 91 9118-9147. PMLR, 2022.
[18] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL
https:///peerrevier. www.nee//oouiie ott.
[19] Bowen Baker, llge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35: 24639-24654, 2022.
[20] Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended Al: Tensions between control and creativity. In Conference Artificial Life, pages 27-35. MIT Press, 2020.
[21] N Bostrom. Existential Risks: analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology, 9, 2002.
[22] Kaj Sotala and Roman V Yampolskiy. Responses catastrophic AGI risk: survey. Physica Scripta, 90(1) )0010000,,,, 2014.
[23] Eliezer Yudkowsky et al. Artificial Intelligence as a positive and negative factor in global risk. Global catastrophic risks, 1(303):184, 2008
[24] Thomas G Dietterich and Eric Horvitz. Rise of concerns about Al: reflections and directions. Communications of the ACM, 58(10)338-40, 2015.
[25] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, pages 103-129, 1995.
[26] Maxime Chevalier-Boisvert Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa- haria, Thien Huu Nguyen, and Yoshua Bengio. BabyAl: First steps towards grounded language learning with a human in the loop. Internationall Conference on Learning Representations. 2019. /ttpptttt////peerreeee wtt.ee/weeeetooomm//oooooooooooo
[27] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: survey of learning methods. ACM Computattion Survey, 50(2), apr 2017. ISSN 0360-0300. doi: 10.1 145/3054912. URL https:://oi..rgg/ 10. 1001/ /054492
[28] Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in Neural Information Processing Systems, 1, 1988.
[29] OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages
PMLR, 2021.
[31] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaI.-FF: An embodied multimodal language model. In arxiv preprint arX3iv:303.03377 2023.
12

---

[32] David Yu-Tung Hui, Maxime Chevalier-Boisvert,, Dzmitry Bahdanau, and Yoshua Bengio. Babyai 1.1. arXiv preprint arXiv:2007.12770, 2020.
[33] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, (8):1735-178)) 1997.
[34] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018, doi: 10. htt99/aai.v32.1.110
htRps:/p:://o88 aaai
org/ brg/index.c php/
11671
[35] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, (2))27002200 1989.
[36] Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisi- tion in multi-task reinforcement learning. In International Conference on Learning Representa- lions, 2018.
[37] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[38] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In Conference Robot Learning, pages 706-717. PMLR, 2022.
[39] Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4202-4213, Online, August 2021. Association for Computational org/
Linguistics. doi: N0.18653/v1/2022 12055/////22......nnn-aaa..l... findiings-acl.368. 2021 findings-nnssii acl. 368
URL
[40] Michal Zawalski, Michal Tyrolski, Konrad Czechowski, Tomasz Odrzygóddž, Damian Stachura, Piotr Piçkos, Yuhuai Wu, Lukasz Kuciński, and Piotr Mitos. Fast and Precise: Adjusting planning horizon with adaptive subgoal search. In International Conference Learning Representations, 2023. URL https://opperreeiee
[41] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[42] Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense knowledge in large language models. In Empirical Methods in Natural Language Processing, pages 11838-11855 2022.
[43] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, Explain, Plan and Select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint urXiv:2302.01560 2023.
[44] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2Motion: From natural language instructions feasible plans. arXiv preprint arXiv:2303.12155. 2023.
[45] Lilian Weng. Lim Llm-powered autonomous agents. illianwengggithubbi Jun 2023. hhttss
//lilianweng, www.gggtthh.ooo http:toss//poso-.////0. 00-2022222002---222 ageatt
[46] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri
Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.1765/ 2023.
[47] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXivv2303.11366 2023.
13

---

[48] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends huggingface. arXiv preprint arXiv:2303. /7580, 2023.
[49] Timo Schick, Jane Dwivedi-Yi Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.06 222.62000,, 2023.
[50] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
[51] Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquistion via instruction augmentation with laiiunla models. In Workshop on Language and Robotics at CoRL 2022, 2022.
[52] Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1713-1726, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18655/11/2022.ac--oo 120. URL attps://aallanthology wwa.logg..rggg///
120
[53] Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions. Advances in Neural Information Processing Systems, 32, 2019.
[54] Lina Mezghani, Piotr Bojanowski, Karteek Alahari, and Sainbayar Sukhbaatar. Think Before You Act: Unified policy for interleaving language reasoning with actions. In Workshop on Reincarnating Reinforcement Learning at ICLR, 2023.
[55] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[57] Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations vector space. In International Conference on Learning Representations, 2013.
[58] Gi-Cheon Kang, Jaeseo Lim, and Byoung- Tak Zhang. Dual attention networks for visual reference resolution in visual dialog. In Proceedings of the 2019 Conference Empirical Methods Natural Lunguage Processing, pages 2024-2033, 2019.
[59] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X1v:1412.6980, 2014.
[60] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint arXiv: 1706. 02677, 2017.
[61] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.
[62] HF Canonical Model Maintainers.
gpt2 (revision 909a290), 2022.
URL https://
huggingface ct/ co/gpt2
14

---

Supplementary Material
A Architecture and Training Details
For full transparency, replicability, and to facilitate future research building on our work, we are releasing the source code, model weights, and the dataset used in our experiment Additionally, we provide key details necessary for the evaluation and replication of our work in this supplementary information. The architectural details of the Thought Cloning models adopted in this paper are shown in Fig. As in [32]. all missions and thoughts are encoded with Gated Linear Units (GLUs), with separate encoders employed for the missions and thoughts respectively. After the encoding process, we apply an attention mechanism [56] to dynamically weight the importance of different parts of the text encoding, based on the state history. The observation is encoded with a Convolutional Neural Network (CNN) and Bag-of- Bag-of-Words Words [57] encoding approach. In the Thought Generator, a Transformer encoder [56] 588 is adopted to embed the thought history and mission, with the thought history as the query and the mission as the key and value. This Transformer encoder consists of two layers, each with two heads. The Action Generator is identical to the Behavior Cloning Baseline, except with the additional encoding of thoughts. Key architectural parameters, such as memory size and embedding size, are consistent with the baseline in [32. as shown in Table D
The pscudocode for Thought Cloning (TC) training framework shown Algorithm In the
loss function, we follow [32] by including an entropy term for actions. The Adam optimizer [5] is adopted to train TC and TC variant, with batch size of 180 and a learning rate of 5e 54, Similar to the setting in baseline [2 26). we train BC with batch size of 296 and a learning rate 5e-5. The learning rate schedule begins with warm-up phase of 5T training steps, where T 51200, linearly increasing from le to 5e for every T step, and then decaying by 50% at 1207 training steps, similar to the practices in 26 60). The teacher-forcing ratio linearly decreases from 100% to 0% from the 10T training step to the end for every 7 step. In line 5 of Algorithm [ the input thought could be the ground truth from the dataset (th,) or the generated thought from the Thought Generator (th,), depending on with or without teacher forcing. For training efficiency, Backpropagation Through Time truncated 20 steps in TC. The mix precision in Py Torch is also adopted during training, which speeds up training without sacrificing much performance [6]. In fine-tuning experiments, due the increased difficulty of the levels and longer steps requiring more memory, we reduced the batch size from 180 to 40 and trained with an auto-regressive strategy. Detailed hyperparameter settings are shown in Table T
Table 1: Hyperparameter Settings
Hyperparameter
Value
Adam B1
0.9
Adam B3
0.99
Adam
10-5
Entropy Coefficient
0.01
Image Embedding Dimension
128
Text Embedding Dimension
256
Memory Dimension
2048
B Ablation Study: BC with the same parameter count or data as TC
We designed the control TC w/o Imitating Thought (Section 3.2 to address the concern that TC outperforms BC because TC has more parameters. Although TC w/o Imitating Thought allows the control to have the same number of parameters (and architecture) as Thought Cloning, for completeness, we try to address the concern in another way. Instead of holding the architecture the same, we create a BC control with a more canonical BC architecture, but with the same number of
15

---

Toughh Hi
Thought Generator
Action Generator
Obe
Monori
Bet Product Jiteetion
Meverrd
rrdkuct
Laben
Nerway
LSTM
Nerweys
Hol
Croern
Figure Detailed architecture for Thought Cloning (TC) agent adopted in this paper. At each timestep t, the inputs to the TC agent include a natural language-defing defined mission m, and an observation All preceding thoughts {(h, ) is embedded with an LSTM in the Thought Generator. The generated thought the from the Thought Generator will be the input to the Action Generator and an action ag is predicted by the Action Generator. (Left): The Thought Generator. We employ an LSTM [33 embed the thought history and a transformer encoder process both the mission and thought history. The text input is then fused with the visual observation input using FiLM [34]. (Right): The Action Generator largely similar the Baby agent [26], with the primary difference being the additional embedding of the thought generated by the Thought Generator.
Algorithm 1 Thought Cloning
1: Input: thought dataset D = (Di where each D; (m, {(or, the (
Thought Generator Te_ (thlo, m, (history_th) Action Generator Te, (alo, m0,,m,th
2: while training do
3:
for each D, = (m, {(o the a ()))(1)) in D do for each (o,, the, a1) in D, do
4:
5:
Generate thought sentence th me_ (-1m, for Mt-i { }==1)
6:
Predict action probability distribution à no, (-Im, {o+)i=1 the)
7:
Compute the loss: L(O,, 0t) = (aE(a, à,) + aLce(thy, the 3({(a.) Update the policy network parameters 0., 02 by minimizing [(O,, 01)
8:
9:
end for
10:
end for
11: end while
parameters TC. Results show it does not perform nearly well TC (Fig. and Table "Pure BC architecture (and matched num parameters)"".
Also, one might argue that TC gets more data (one set of actions and one set of words per episode), and thus that a proper BC control is to give BC twice as much data. counter that such a control is unnecessary because noticing and harnessing this additional (often free) and heretofore ignored data stream is central contribution of this paper, so showing that using this data improves things the main comparison to be made. However, for completeness, much data: results show BC with twice as much data is still far slower to Iearn and has significantly worse performance at convergence (Fig. 7 and Table 2 "BC w/ 2x data"). Additionally, prior work Think Before You Act [54] has a similar number of episodes of actions and language data in the same domain, and their model has more parameters than ours. Results show that TC also outperforms that method (Table 2.
ran experiment giving BC twice as
Finally, to further confirm that the superiority of TC did not come from both more data and parameters, an ablation study was conducted on a pure BC architecture with the same number parameters as TC but also with 2x data. The, results show that it also underperformed TC (Fig. 7 and Table 2 "Pure BC architecture (and matched num parameters), 2x data").. All of the aforementioned results highlight the effectiveness of the TC framework and the importance of the thought data.
16

---

100%
80%
Success Rate
60%
40%
Thought Cloning
Behavioral Cloning
BC w/ 2x Data
Pure BC architecture (and matched num parameters)
Pure BC architecture (and matched num parameters), 2x data
TC w/o Imitating Thought
Training Steps
Figure 7: Training progress comparison of Thought Cloning (TC), Behavioral Cloning (BC), and other ablation variants. The BC architecture is identical to the Action Generator of TC and the TC w/o Imitating Thought has the same architecture minor architectural change that is required since it does not have this loss; see response to reviewer gnSR). BC w/ 2x Data trains BC with 2x more episodes worth of actions than TC and BC. Pure BC architecture (and matched num parameters) is to augment BC architecture to have matched parameters with TC. Pure BC architecture (and matched num parameters), both 2x episodes of actions and a BC architecture scaled up to match the number of parameters of TC. Table this document for more details on the number parameters and amount of data for each treatment. The error bars are the 95% confidence intervals from five runs of experiments. The results show that TC learns dramatically faster and has significantly higher performance at convergence compared with BC and all ablation variants, which indicates the advantages of TC are not solely due to additional training data or model capacity.
TC. but without the TC loss (and has one
data trains BC with
C Synthetic Human Thought Dataset
Fig. 8 presents an example trajectory. We translate the inner state of the BabyAl Oracle Solver (called "Bot" in (26)) into natural language thoughts. These thoughts outline the current plan for task completion and also describe the underlying intentions behind these plans, the same low-level plan can serve different stated bigh-level purposes. For instance, the plan could be to "open the red door" with the intention of "completing the open mission" or "exploring", The segments with inserted noise are marked red Fig.
D Example on Diagnosing Agents by Observing Thoughts
In this section, we provide example of one time when were able diagnose Thought Cloning (TC) agents by observing their thoughts during the development phase of this paper. In the early stages of development, that during testing, the agents often got stuck persisting with incorrect thoughts and did not actively explore new ideas. For instance, in the top right example in Fig. 9 after t=53, the agent persistently attempted to implement the incorrect thought "pickup blue box to complete putnext mission" " until reached the maximum step limit, without seeking new ideas. This observation led realize that, as we exclusively trained the agent with oracle thoughts via maacher-forcing strategy, the agent had never practiced dealing with incorrect thoughts and consequently had learned recover from them by trying alternative ideas. Subsequently before this realization, we had attempted to transition directly to auto-regressive training after the teacher-forcing training stage. However, the agent then started to generate nonsensical thoughts. The trajectory in Fig. (left) shows nonsensical thoughts such as open blue (I=24) and pickup door door (I=75) being generated when constant ruto-regressive strategy is applied. Because of the realization from being able to observe the agent's
trained the TC agent with constant teacher-forcing strategy. We observed
17

---

mission: put a key next to the blue ball and pick up the green key
t=1-5: Thought "pickup key to complete PutNext mission", Action [forward, right, forward]
t=6: Thought "pickup blue key complete PutNext mission", Action [pickup]
t=7-12: Thought "explore unseen area", Action [right, right, rightt
t-13-14: Thought "open green door to explore", Action [left, left, ..., togzle]
t=15-22: Thought "explore unseen area', Action [forward, forward, .., left]
t=23-28: Thought "open red door explore", Action (forward, forward, ... toggle]
t=29: Thought "explore unseen area", Action [forward]
Action t-39-44: Thought "explore unseen area', Action [right, right, rightt....Iettt
t=30-38: Thought "drop blue key complete PutNext mission", Action [forward, forward, ... drop!
t=45-48: Thought "open green door to explore", Action [left, left. .... toggle]
t=49-56: Thought "explore unseen area', Action [forward, forward, left]
t-57-69: Thought "open green door to explore", Action [forward, forward, toggle]
t=70-77: Thought "explore unseen area', Action [forward, forward, .... right]
t=78-88: Thought "open grey door explore". Action (forward, forward, ... toggle]
t=89-95: Thought "explore unseen area', Action (forward, forward, ... right]
t=96-101: Thought "open purple door to explore", Action (forward, forward, ..., toggle]
t=102-110: Thought "explore unseen area". Action [forward, forward, .... right]
t=111-122: Thought "open red door to explore', Action [forward, forward,... toggle]
t=123-130: Thought "explore unseen area", Action (forward, forward, right]
t=131-134: Thought "open grey door to explore", Action [right, right, .., forward]
t=135-137: Thought "pickup green box to complete PutNext mission", Action lleft, left, left)
t=138-204: Thought "open grey door to explore", Action [right, right, .... toggle]
t-205: Thought "explore unseen area", Action [forward]
t=206-213: Thought "pickup green key complete pickup mission", Action [forward,...ppckup)
Figure 8: Example trajectories of the synthetic human thought dataset. The inserted noisy segments are highlighted in red.
mission: put the green box next the purple door after you go the purple door and put blue box next to the purple door
Constant beacher fording traiting
Constont suto-rngrentin training teacher-frrchg trainiing
untinn pherggeennt Serwer Lnpor
sealore actal Newerd fema'
4. deo Join Saaiid wrkk
100 29 des cide
Stuskwht
pio redees adine berwann
Sarerd
1d0 et mutore adia Serval Aimaa
incomct
:houghts
Seacher-frrting gridusll
Nomenatica Booghts
1411 peheo bas oiet prter mucee tour Hrghh
I-th now frww whe wart farmer reppht
rove mithes Flevard, cniing
echupttn tour complee, crewe
1:10
firet sevien after 3-sh rich
wi B1 mater eocoe ane, onler loownd lumeri (994))
agare
e-45 agen bos door onhoos of Brrves forverd .meere
1i1 11e cose and cow ancodiry un on aor fomert wauld
thought but
fives them.
Inwemmere soheol
Figure 9: Example trajectories of agents trained with different strategies. Constant teacher-foreing training refers to exclusively training with the teacher-forcing strategy. In this scenario, the agent does not learn to recover from incorrect thoughts. Once it adopts an incorrect thought, it continues to follow this thought for thousands of time-steps until reaches the maximum step count (top right from t=53 to t=2880). Constant rugo-regressive training after teacher-forcing training implies directly transitioning to auto-regressive training following an initial phase of teacher-forcing training. In this case, agents begin to generate nonsensical thoughts, shown on the left, such open blue t=24 (left) and pickup door door at t=75 (left). Gradual decay of teacher-forcing rate involves gradually reducing the ratio of teacher-foreing during training. This strategy adopted in the final version of Thought Cloning. In this setting, the agent might generate some incorrect thoughts as shown t=53 (bottom right), but it can recover from these errors explore new ideas, evidenced at 1=131 (bottom right).
18

---

Table 2: Success rates of TC, BC, ablation variants, and the related work Think Before You Act 54 on BabyAI BossLevel. See the descriptions of ablation variants in the Fig. Caption. The Success Rate is the mean and standard deviation from 5 experiments (except Think Before You Act, which did not include the number of experiments in their paper). The num (nubmer of) parameters of Think Before You Act estimated from the number of parameters of the Huggingface GPT-2 model 62] The results show TC outperforms all other methods, illustrating the superiority of TC, and supporting our argument that the advantages of TC are not solely due to additional training data or model capacity.
Method
Num Parameters
Data
Success Rate ( (%)
Behavioral Cloning
20.6M
1M Episodes Actions
91.2 + 0.9
Thought Cloning w/o Imitating Thought
82.5M
1M Episodes Actions
65.5 + 12.4
BC w/ 2x Data
20.6M
2M Episodes Actions
91.4 + 1.7
Pure BC architecture
83.9M
IM Episodes Actions
91.9
(and matched num parameters)
Pure BC architecture
83.9M
2M Episodes Actions
92.7 1.0
(and matched num parameters), 2x data
1M Episodes Actions
Think Before You Act
124M (cstimated)
85.2 = 0.5
Language
IM Episodes Actions
Thought Cloning
82.5M
96.2 + 0.8
Language
thoughts, we adopted a gradual decay schedule for teacher-foreiigg rates during training. As shown in Fig. 9 (bottom right), the agent in this setting was able to explore new ideas after failing incorrect thought, and rarely generate nonsensical thoughts. For example, the agent generates an incorrect thought at f=53, but it can recover from these enrors explore new ideas, e.g. open grey door to explore. Because we can observe the TC agents thinking out loud, we are able to identify the issue and improve the agent's performance. Without this visibility into the agent's thoughts, simply observing their actions would have made it much harder to pinpoint the underlying problems.
19


Reviews:
[{"review_id": "ZjNoLjITWd", "reviewer": "Reviewer_gnSR", "summary": "This paper presents an approach incorporates discrete intermediate-level descriptions and goals  to train RL agents to perform higher level actions. These intermediate-level descriptions and goals are described in natural language, and the authors refer to them \"thoughts\" that the RL agent learns which then influences a desired action, in a process called Thought Cloning (TC). The paper contrasts this approach with conventional Imitation Learning that rely on Behavioral Cloning (BC). The paper claims that TC learns faster than BC and generalizes much better than BC across cognitive difficulty and behavioral difficulty.", "questions": "What is the architecture of the TC w/o imitating Thought and what are its control signals?", "limitations": "A major limitation to this approach mentioned in the introduction is the availability of applicable thought data.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: This paper appears to propose a novel method, but I am not certain of that. \n\nQuality: The collected experimental data and the collection of synthetic training data are are all sound, and several valid metrics are provided to support the papers claim, and this is a complete, but, I believe, flawed work due to the construction of the ablation study.\n\nSignificance: If the ablation study can be properly addressed and modified to show a clear advantage of TC over BC, then this paper would be very significant.", "weaknesses": "Writing: Prose used in the abstract and introduction is superfluous at times. Many sections speculate why the proposed method is superior supported by biological comparison, but the paper lacks evidence for any such comparison. The introduction could possibly be halved in length and still convey the most important aspects of this paper. Specifically, lines 47-81 could easily be condensed. \n\nQuality: The ablation study performed, as I understand it, is the weakest component of this paper. The implementation of TC w/o Imitating Thought is described opaquely, and leaves a lot of inference to the reader. As I understand it from the paper, and copy of the TC architecture is made, however the control signals for Next Thought and Thought History are left null. While this does create a model with an equivalent number of parameters as the original TC, only the parameters for the action agent are trained. \n\nClarity: See above issues with interpreting the ablation study.\n\nExperiment Design: Comparing BC to TC is an imperfect comparison as noted by the paper and hence the inclusion of the TC w/o Imitating Thought, however, this model does not adequately address difference between model performance either. My intuition tells me that TC significantly outperforms BC because it has access to more training data (the intermediate thoughts) and comprises of more parameters which then allows the model to train faster and generalize better. The paper does not do an adequate job to address this concern. A possible new experiment that could address this is to ensure training data is equivalent between the models, and allow the BC model to approximate the same number of parameters as TC. Without this additional comparison, it cannot be confidently known where the true cause for the improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and acknowledging the strengths of our work, including its novelty and saying it is ***“very significant”*** (provided your requested experiments confirm TC outperforms BC, which they all did!).\n\nWe have addressed each of your concerns, significantly improving the manuscript as a result. We feel this paper makes an important contribution the ML community will benefit from if published. Your current score is low and likely will prevent publication. We hope you will keep an open mind to changing your score in light of our improvements, and since two other reviewers rated the paper a 7 (accept) and 8 (strong accept).\n\n***“The implementation of TC w/o Imitating Thought is described opaquely ... As I understand it …[a] copy of the TC architecture is made, however the control signals for Next Thought and Thought History are left null. While this does create a model with an equivalent number of parameters as the original TC, only the parameters for the action agent are trained.”***\n\nAll parameters were trained in the 'TC w/o Imitating Thought' control (described in L158-162).\n\nWe apologize it was not clearer. We will add the following to make it more clear.\n\n“we introduce an ablation variant called TC w/o Imitating Thought to demonstrate that TC’s superiority is not solely due to its larger number of parameters. This variant’s architecture is identical to the TC architecture, except for a minor architectural difference where the latent vector from the upper level is directly input to the lower level as thought. This adjustment is necessary because this variant is not trained on the Thought Cloning loss, so we do not have per-word supervision. To train these parameters, we thus need to train them based on how these “thoughts” contribute to the final performance (i.e. they are trained end-to-end). If we sampled words (as in Thought Cloning), we could not train these parameters end-to-end because hard sampling of words is non-differentiable, so gradients could not flow backward through this operation. Thus, we make one small change in order to allow the parameters to be trained via SGD, which is to pass the logits of the Upper-level Component directly into the Lower-level Component, which is a differentiable operation. We feel this is the closest and fairest control possible to Thought Cloning, allowing virtually the same architecture and the same number of parameters, but not including the Thought Cloning innovation of exploiting human thought data.”\n\n***\"My intuition tells me that TC significantly outperforms BC because it has access to more training data (the intermediate thoughts) and comprises of more parameters which then allows the model to train faster and generalize better.\"***\n\nWe designed the TC w/o Imitating Thought control to address this concern, allowing the baseline to have the same number of parameters (and architecture) as Thought Cloning. We hope now that you know these parameters were trained, your main objection to the paper is removed. However, for completeness, we try to address it in another way. Instead of holding the architecture the same, we create a BC control with a more canonical BC architecture, but with the same number of parameters as TC. Results show it does not perform nearly as well as TC. (see new results PDF)\n\nTo address if “has access to more training data (the intermediate thoughts)”, we will add this text: \n\n“One might argue that TC gets more data (one set of actions and one set of words per episode), and thus that a proper BC control is to give BC 2x data. A counter is that such a control is unnecessary because noticing and harnessing this additional (often free) and heretofore ignored data stream is a central contribution of this paper, so showing that using this data improves things is the main comparison to be made. However, for completeness, we ran an experiment giving BC twice as much data: results show BC-2xData is still far slower to learn and has significantly worse performance at convergence.” (See new results PDF)\n\nSince you said TC’s performance advantage might come from having extra parameters or more data, we wanted to make sure it did not come from having both. We tried BC with 2x data and a pure BC architecture with the same number of parameters as TC, and it too underperformed TC.\n\nAlso, a prior work “Think Before You Act [Mezghani et al. 2023]” had a similar number of episodes of actions and language data in the same domain, and their model has more parameters than ours. TC also outperforms them. (See new results PDF)\n\nYou wrote “If the ablation study can be properly addressed and modified to show a clear advantage of TC over BC, then this paper would be very significant.”\n\nThe original control, a new comparison to prior work, and all of the new experiments you requested do indeed show a clear advantage of TC over BC (dramatically faster learning and better performance at convergence), providing strong evidence that the advantages of TC are not solely due to additional training data or model capacity. Given that you said that would make this paper “very significant” were this true, we hope you are open to substantially increasing your score. Thank you for considering it!\n\n***\"Many sections speculate why the proposed method is superior supported by biological comparison, but the paper lacks evidence for any such comparison.\"***\n\nThank you for pointing this out. We will be happy to revise the writing. While we used human thinking as a source of inspiration, and we do think it suggests this direction is powerful, one could jettison that analogy and our method would stand alone on its empirical results (higher performance, debugability, safety benefits, etc.). We will eliminate superfluous language and make these issues clearer in the paper.\n\n***“A major limitation to this approach mentioned in the introduction is the availability of applicable thought data.”***\n\nAddressed in the Response to All Reviewers."}, {"review_id": "TfLHDXbSPs", "reviewer": "Reviewer_QGuE", "summary": "The paper studies the problem of imitation learning and attempt to improve existing IL algorithms by training the agent to think like the expert that is being imitated. Authors propose Thought Cloning - an extension to behavior cloning that seeks to imitate thoughts expressed in natural language. They evaluate the proposed method on thought-augmented data from solving tasks in the BabyAI gridworld. Authors compare the proposed algorithm to standard behavior cloning and observe faster training and better generalization. The paper also contains extended discussion about the benefits of thought cloning with respect to AI safety.", "questions": "> we can improve AI agents by training them to think like humans do.\n\nDisclaimer: i am no expert on the subject matter, feel free to ignore the question.\n\nIn the beginning of the paper, (L5-L6) it is presented a fact that humans think in a language.\nTo the best of my knowledge, there is no scientific consensus that all humans think in a language, or that humans think only in language - though I am no an expert on that account. Is this really true? If so, please cite the relevant work (e.g. in S1). If not, please paraphrase the sentence to avoid accidentally misleading the reader.", "limitations": "Authors have adequately adressed most of the limitations of their work. As for the sociental impact, I have highlighted several safety concerns for the proposed system in the \"weakness\" section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The Thought Cloning method appears easonable and exploits the paper's high level idea in a natural / intuitive fashion.\n2. The paper contains a reasonable evaluation of the proposed idea by augmenting popular environments with expert's \"thoughts\". Typical caveats are addressed with a reasonable a reasonable ablation analysis.\n3. The paper is well-written and easy to follow. The general presentation quality is high.", "weaknesses": "### Concern 1: lack of comparison with prior work\n\nThere are several previous papers that also attempt to teach an agent to think using natural langauge. Authors address planning in natural langauge in section 4.1, but there is a line of work that is not covered by planning. From a surface-level analysis of existing work, there are papers that\n\n* Improve reinforcement learning by performing self-feedback - Madaan et al, Self-Refine: Iterative Refinement with Self-Feedback\n* Ask a language model to reflect on it's performance in natural language, improving learning - Shin et al, Reflexion: an autonomous agent with dynamic memory and self-reflection\n* Produce natural langauge explanation - parts of Wang et al, Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents\n\nSince this is only a surface-level analysis by a non-expert (me), there could be more related works that study this problem.\n\nI would argue that the paper could benefit greatly from comparing against all related works. This should include both a theoretical comparison of the proposed approach (e.g. which parts of TC are novel), and an empirical evaluation of how this translates to training speed, generalization, etc.\n\n\n\n### Concern 2: Safety Implications of Thought Cloning\n\nOne of the two main rationales from the introductory section is that thought cloning could make the resulting agent more safe.\nIn S 3.5 / Fig.5, authors evaluate a thought-based safety technique that they call \"precrime intervention\" and find that:\n> remarkably, Precrime Intervention almost entirely eliminates all unsafe behaviors, thereby demonstrating the promising potential of TC agents in advancing AI safety\n\nWhile this sounds intuitively correct, there seems to be (at least) two caveats that could result in a less safe system due to the false sense of safety.\n\nFirst, under the formulation proposed in Section 2, Thought Cloning is incentivized to copy any cognitive errors from the expert, including lies, false rationalizations and general ignorance.\nUsing the car driver example from L48-49, an agent could (falsely) argue that it is legal to ignore some traffic rule in this specific circumstance. It is easy to imagine that human \"expert\" drivers will avoid \"testifying against themselves\" without a dedicated system to control for that. A sufficiently powerful language model can mimic their plausible explanations to mislead the passenger.\n\nSecond, if the proposed system is improved upon -- either directly with Reinforcement Learning or indirectly with manual updates, the agent would be incentivised to produce explanations that satisfy the user, so as to avoid being stopped. Note that these explanations do not have to be true - they simply have to be convincing. This can be seen as a special case of the \"stop button problem\" (see, e.g. , https://arxiv.org/pdf/1611.08219.pdf or https://www.lesswrong.com/posts/wxbMsGgdHEgZ65Zyi/stop-button-towards-a-causal-solution )\n\nIn other words, the initial system has a risk of inheriting potential flaws of (human) experts and any improvements to the system run a risk of incentivising less truthful explanations. From my (limited) point of view, it appears to early to claim that TC precrime intervention can truly increase the real-world system safety. One way to improve this would be to add a dedicated safety analysis, formulate and prove safety guarantees.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for noting our method is reasonable and supported by our evaluations and arguments. \n\nWe sincerely believe our paper presents a significant and beneficial contribution that would enrich the ML community upon publication. Reflecting upon your concerns, we believe they either have been fixed, or are areas for future work rather than fundamental flaws that should prevent publication. Overall we would have predicted a higher score based just on your comments alone. Your current score recommends not publishing this work, but we hope you will keep an open mind to reconsidering in light of our revisions, replies, and since two other reviewers rated it as 7 (accept) and 8 (strong accept).\n\nWe have attempted to address each of your concerns, and significantly improved the manuscript as a result. \n\n**1. Concern about the lack of comparison with prior work**\n\n***“There are several previous papers that also attempt to teach an agent to think using natural langauge. Authors address planning in natural langauge in section 4.1, but there is a line of work that is not covered by planning.”***\n\nWe will add the following text in the related works section to address this:\n\n“There are many works that use LLMs to improve the ability of agents to act [Wang 2023], but their approaches are quite different. They generate thoughts in addition to actions [Yao et al. 2022] or let LLMs replan based on new information [Madaan et al. 2023, Shinn et al. 2023], but none directly learn to think while acting by imitating human thought data. Thus, unlike Thought Cloning, they do not benefit from learning from human thought demonstrations how to do things like plan, replan, create high-level goals and the subgoals required to achieve them, and the many other benefits of thinking intelligently during acting.”  \n\nPlease note that the area and the work you mentioned “Wang et al, Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents” is covered in L308.\n\n***“I would argue that the paper could benefit greatly from comparing against all related works. This should include both a theoretical comparison of the proposed approach (e.g. which parts of TC are novel), and an empirical evaluation of how this translates to training speed, generalization, etc.”***\n\nWe compared TC to what we feel is the closest baseline (behavioral cloning). As just discussed, there are other works that may seem related (since they use LLMs to think in some ways), but they are actually quite different, and direct comparisons are thus apples to oranges and not very scientifically informative. However, we did add a performance comparison to one similar related work (Think Before You Act, Mezghani et al., 2023), which also learns from action + language data in the same domain (BabyAI). Thought Cloning outperforms it, showing that the main idea of Thought Cloning (imitating human thinking) is beneficial. (See Table 1 in new results PDF)\n\n**2. Concern about the Safety Implications of Thought Cloning**\n\n***“under the formulation proposed in Section 2, Thought Cloning is incentivized to copy any cognitive errors from the expert, including lies, false rationalizations and general ignorance.”***\n\nThank you for highlighting this potential concern. We added the following text as a result:\n\n“As occurs with LLM pretraining and other forms of Behavioral Cloning, Thought Cloning could inadvertently inherit undesirable human flaws, such as speaking falsehoods, providing false yet persuasive rationalizations, or being biased. Alignment techniques are being constantly improved to address these challenges [Ouyang et al. 2022]. However, even improving AI agent safety up to the level of a (flawed) human would be a major advance, even if the resulting system is not perfect. Additionally, a distinguishing feature of Thought Cloning is it provides the ability to interpret and prevent these flaws from culminating into actions, making TC a more favorable approach in this regard.”\n\n***“if the proposed system is improved upon -- either directly with Reinforcement Learning or indirectly with manual updates, the agent would be incentivised to produce explanations that satisfy the user, so as to avoid being stopped. Note that these explanations do not have to be true - they simply have to be convincing.”***\n\nWe agree, but that is a reason to avoid training the system to not be stopped by precrime intervention. One should instead train agents to do desirable things and avoid undesirable things directly, and reserve precrime interruption as a safety mechanism (and not training reward function). Additionally, one could use ever-smarter LLMs (or, better, multi-modal models like vision-language models) to monitor the thoughts and actions to attempt to detect such issues, a capability that is only possible because Thought Cloning provides interpretable access to the agent's thoughts. We will add these nice insights to the paper. Thanks! \n\n***“In other words, the initial system has a risk of inheriting potential flaws of (human) experts and any improvements to the system run a risk of incentivising less truthful explanations. From my (limited) point of view, it appears to early to claim that TC precrime intervention can truly increase the real-world system safety. One way to improve this would be to add a dedicated safety analysis, formulate and prove safety guarantees.”***\n\nOur paper contains arguments for and experimental analyses on the effectiveness of precrime intervention, which can serve to inspire others to advance this method and or test it in real-world scenarios. We’ll ensure the writing does not guarantee it will be effective, but instead (like much helpful scientific publications) simply provides evidence of a new, promising method. \n\n**3. It is debatable whether humans exclusively think in language.**\n\nAddressed in the Response to All Reviewers."}, {"review_id": "yExc2wCUt1", "reviewer": "Reviewer_9iZD", "summary": "This paper provides Thought Cloning (TC), an imitation learning method that clones not only behaviors but also thoughts. Here, thoughts are descriptive texts for each behaviors. The basic idea is that language can help agents to better plan their actions and adapt to a new environment. More specifically, the TC agent consists of two components: the Upper-Level and Lower-Level Components. The Upper-Level Component learns to generate a thought conditioned on an observation, a mission, and a history of thoughts. The Lower-Level Component learns to generate an action conditioned on an observation, a mission, and the generated thought. This paper demonstrated the effectiveness of TC on BabyAI, since synthetic thoughts can be easily generated on BabyAI. The experiments showed that TC can be trained much faster than Behavioral Cloning (BC), and TC can better generalize to out-of-distribution environment than BC.      ", "questions": "Q1. How can we effectively collect thoughts for actions to train TC agents?\n\nQ2. How much robust is the TC agent with regard to the quality of training thoughts data? What is the effect, if we control the quality of synthetic thoughts on BabyAI?\n\nQ3. Is LSTM proper as a base model for TC agents? If we use Transformers instead of LSTMs, is the performance improved or not?\n\nQ4. How well does the TC agent perform on other environments in addition to BabyAI?", "limitations": "- The effectiveness of TC agents seems to be highly dependent on the quality of thoughts for actions. It would be better to provide a way of efficiently collecting high-quality thought data. Also, it would be interesting to show how much robust the TC agent with regard to the level of quality of thoughts.\n\n- It would be better to provide more results on other environments.\n   ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper has some strong points as follows.\n\n- First of all, this paper is very well written and organized.\n\n- The idea of generating actions conditioned on thoughts (descriptive texts for actions) is simple but effective.\n\n- Thought Cloning (TC) seems have some advantages compared to Behavioral Cloning (BC). First, TC is trained faster than BC. Second, TC may improve the ability to generalize to out-of-distribution environments. Third, TC also enhance the interpretability, since human can see descriptive texts for each actions. ", "weaknesses": "This paper has some weak points as follows.\n\n- One of my main concerns is how can we effectively collect thoughts for actions to train TC agents. For the purpose of demonstration, this paper trained the TC agent on BabyAI where thoughts for actions are synthetically generated. However, in real world scenarios, it is not easy to collect (high quality) thoughts for actions. Even though the authors mentioned that Youtube videos and vision-language models (VLM) can be used for collecting thoughts for actions, such thoughts would be very noisy compared to synthetic thoughts on BabyAI. Accordingly, the training efficiency and the performance may be deteriorated.  \n\n- This paper implemented the TC agent mainly based on LSTM. I am not sure that LSTM can effectively generate thoughts conditions on a history of thoughts. It would be interesting to show the performance of a Transformer-based TC agent.\n\n- The main experiments of this paper was performed on only one environment, BabyAI. It would be better to provide experiment results on other environments.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and feedback on our manuscript. We are delighted that you consider our idea simple yet effective, and recognize our contribution to the capabilities and interpretability of agents. Below we address each of your concerns and questions.\n\n***“How can we effectively collect thoughts for actions to train TC agents?”***\n\nThank you for your comments. Please see the general reviewer response, where this is addressed. \n\n***“Even though the authors mentioned that Youtube videos and vision-language models (VLM) can be used for collecting thoughts for actions, such thoughts would be very noisy compared to synthetic thoughts on BabyAI.”***\n\nWe recognize the noise in internet data and that it potentially could be an issue. However, we believe this challenge could be effectively mitigated with proper data processing. An example from MineCLIP [7] shows that careful heuristic-based processing of the captions could make data clean enough to train multi-modal models (e.g. a simple heuristic focusing on domain-specific vocabulary could remove most off-topic text like “welcome to my channel”). More promisingly, one could use language models to filter out off-topic data (they are quite capable and will only get better), either by removing any off-topic comments or determining that some videos should be excluded because not enough of the commentary is helpful. Additionally, even if noise is present, recent ML history shows that–at scale–such “noise” does not prevent learning the “signal”: examples of this occurring on internet-scale data include GPT, CLIP, and VPT (all of which one might have argued ahead of time would not work due to training on noisy data, but in fact work very well despite noise). We view these as fascinating topics of future research inspired by our paper and will say so in the revised paper.\n\n***“How much robust is the TC agent with regard to the quality of training thoughts data? What is the effect, if we control the quality of synthetic thoughts on BabyAI?”***\n\nWe added noise to our training data and found TC is robust to it (in fact, it helps because it trains the agent how to get back on track if it makes a mistake in its thinking). Currently, about 3.5% of data is noisy data. If you like, we would be happy to do an experimental sweep of different fractions of noise. Given how long it takes to perform TC runs, it was not possible to do that sweep during the short amount of time in the rebuttal window. If you request it, we could also do tests with other types of noise, including adding phrases like “please subscribe to my channel” and additional tests using LLMs to filter out such off-topic phrases, but we believe such a substantial additional set of experiments is better left to future work where there is space to properly document it..\n\n***“Is LSTM proper as a base model for TC agents? If we use Transformers instead of LSTMs, is the performance improved or not?”***\n\nWe recognize the superiority of the Transformers architecture in recent work. If it comes to real-world scenarios with large-scale language or multimodal tasks, a Transformer would likely be a more powerful model and would be a natural choice. However, in our test domain (BabyAI), it is still a middle-scale problem compared to the internet-scale dataset where Transformers truly shine. Thus, the data-hungry nature of Transformers could make it not the most powerful architecture, or at least not necessary, in BabyAI. One example is from Think Before You Act (Mezghani et al. 2023), where they use GPT-2 like Transformers in the BabyAI environment, but their performance (85.2% Success Rate on BossLevel) is outperformed by BabyAI1.1 (Hui et al. 2020), an LSTM-based architecture baseline (90.4% Success Rate on BossLevel), which has fewer parameters than Think Before You Act. We will mention the value of testing with transformers in future work. \n\n***“How well does the TC agent perform on other environments in addition to BabyAI?”***\n\nWe think this is a great direction for future work. However, we believe the best domain to do the science for this paper introducing the method was BabyAI. It is in fact a quite challenging domain, where there are complicated language-described tasks and long-horizon action controls, but it also has other key attributes, such as the ability to generate synthetic thought data. We believe the core principles of Thought Cloning hold promise in more challenging domains, and that much interesting future work exists to see how it performs in such domains. \n\nThank you once again for your review and positive comments. Your score is already high and we appreciate that, but we wonder if you feel the paper is stronger having read our responses if you might consider increasing it further to help its chance of being published and shared with the ML community. We deeply thank you for your consideration."}, {"review_id": "DjM6cpKL7f", "reviewer": "Reviewer_HBeY", "summary": "This paper develops a method for \"thought cloning\", which involves imitating a human's thought process during task performance. The authors apply this to a partially observable 2D grid world domain. They show that thought cloning yields superior performance compared to behavior cloning.", "questions": "I think the paper is basically good as is. If it were a longer format, I would ask about how the authors would scale up their approach (as they discuss briefly in the paper).\n\nUPDATE: the authors have addressed my comments.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The approach is innovative, and could be highly impactful once it is scaled up.\n\n- The paper is clearly written.\n\n- The methods are sound, as far as I can tell.\n\n- The application to AI safety and interpretability is interesting and increases the significance of the contribution.\n\n- The paper makes a strong empirical case that their approach works, at least for gridworlds.\n\n- I liked the cognitive science motivation, even if I might quibble with some of the arguments.", "weaknesses": "- It would have made a stronger contribution if the authors could show that this method scales up to more complex domains, though I appreciate that introduces many new difficulties and they wanted to show that the approach works on simpler domains first.\n\n- I think the authors are a bit loose with their arguments about human cognition. It is hotly debate to what extent cognition depends on language in a strong way. It also important to note that one can endorse a \"language of thought\" hypothesis about high-level cognition without endorsing the hypothesis that this corresponds to natural language. In any case, I appreciate that these points have little bearing on the substance of this paper.\n\n- Eq. 1 could benefit from more explanation.\n\n- Please state what error bars show in figures.\n\nUPDATE: the authors have addressed my comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review of our paper and your acknowledgment of the strengths of our work, including the novelty and soundness of the method, the strong empirical case, and the contribution to AI Safety and Interpretability. We appreciate the depth of your feedback and are pleased to note your positive assessment. We address your concerns and suggestions below. \n\n***“It would have made a stronger contribution if the authors could show that this method scales up to more complex domains, though I appreciate that introduces many new difficulties and they wanted to show that the approach works on simpler domains first.”***\n\nWe think this is a great direction for future work. As you say, that would introduce additional difficulties and we wanted to show (and study) the approach on a simpler (and easier to analyze) domain first. We believe the best domain to do the science for this paper introducing the method was BabyAI. It is in fact a quite challenging domain, where there are complicated language-described tasks and long-horizon action controls, but it also has other key attributes, such as the ability to generate synthetic thought data and being easy to analyze. We believe the core principles of Thought Cloning hold promise in more challenging domains, and that much interesting future work exists to see how it performs in such domains. \n\n***“I think the authors are a bit loose with their arguments about human cognition. It is hotly debate to what extent cognition depends on language in a strong way. It also important to note that one can endorse a \"language of thought\" hypothesis about high-level cognition without endorsing the hypothesis that this corresponds to natural language. In any case, I appreciate that these points have little bearing on the substance of this paper.”***\n\nThank you for your comments. Please see the general reviewer response, where this is addressed.\n\n***“Eq. 1 could benefit from more explanation.”***\n\nThank you for your comment. We will add the following text in L104 in the manuscript.\n\n“The first part of the loss is the Thought Cloning loss, where the Upper-level Component is conditioned on the history of thoughts and observations, and the mission, to predict the thought. That thought is then compared with the ground truth thought in the dataset. The second part is the action loss, where the Lower-level Component is conditioned on the current thought, the history of observations, and the mission to predict the action to do, and we then calculate the loss by comparing the predicted action to the ground truth action in the dataset. ”\n\n***“Please state what error bars show in figures.”***\n\nThank you for pointing this out. The error bars are the 95% confidence interval from five runs of experiments. We will add the explanation in the related figure captions.\n\nThank you once again for your review and positive comments. Your score is already high and we appreciate that, but we wonder if you feel the paper is stronger having read our responses if you might consider increasing it further to help its chance of being published and shared with the ML community. We deeply thank you for your consideration."}]

Author rebuttal:
We are deeply grateful to the reviewers for their comprehensive evaluations and thoughtful feedback on our work.

We are encouraged by the reviewers' positive comments including:
- “The approach is innovative, and could be **highly impactful** once it is scaled up.” (HBeY) “The idea of generating actions conditioned on thoughts (descriptive texts for actions) is simple but effective.” (9iZD) 
- “If the ablation study can be properly addressed and modified to show a clear advantage of TC over BC, then **this paper would be very significant**.” (gnSR) [The new, requested ablation studies do show such a clear advantage]
- “The paper makes **a strong empirical case** that their approach works” (HBeY) “The paper contains a reasonable evaluation of the proposed idea… Typical caveats are addressed with a reasonable ablation analysis.” (QGuE) “The collected experimental data and the collection of synthetic training data are all sound.” (gnSR)
- “The application to AI safety and interpretability is interesting and increases the significance of the contribution.” (HBeY) “TC also enhance the interpretability, since human can see descriptive texts for each actions.” (9iZD)
- “TC is trained faster than BC… TC may improve the ability to generalize to out-of-distribution environments” (9iZD)
- “The paper is well-written and easy to follow.” (QGuE) “This paper is very well written and organized” (9iZD) 

We feel this paper makes an important, helpful contribution that the ML community will benefit from if published. **We appreciate that two reviewers rated the paper a 7 (accept) and 8 (strong accept).** But the two other scores are low and likely will prevent publication. We hope all reviewers will keep an open mind to increasing your scores in light of our improvements to improve the chances that the ML community will be able to benefit from learning about this work

In response to your requests, we have produced many new results (see results PDF), all of which strongly empirically support the benefits of Thought Cloning. We also improved the paper in the ways you suggested. Here is a quick summary of the major revisions:
- We clarify that all parameters were in fact trained in the “Thought Cloning w/o Imitating Thought” control, and thus there was already a behavioral cloning control for the number of parameters (and architecture), eliminating one reviewer’s major objection. While this was described in the original paper, we have modified the paper to make the description of this control much clearer. For completeness, we also performed additional experiments requested by reviewer gnSR, all of which show that the superiority of TC does not come from a larger number of parameters or from (arguably) having more training data. Instead, it is the main idea of Thought Cloning that drives the performance gains.
- We modified the language in the paper’s motivation making it clear that it is simply a hypothesis (and one up for debate) that humans think in natural language (and have added citations that the claim is debatable). We also point out that this biological comparison is simply motivation, and that the method can independently stand on its empirical results even if one does not believe in this hypothesis or analogy. 
- We addressed the safety concern raised by reviewer QGuE.
- We expanded and improve comparisons to prior work.
- We've expanded the explanation for Eq. 1 and explained the error bars more clearly in the corresponding figure captions.

The revisions have made the paper stronger and we deeply thank all the reviewers for their help. Below we respond to common questions, and the reviewer-specific responses answer unique comments. 

**1. Regarding our motivation text saying humans think in natural language:**

Reviewer QGuE, HBeY: **It is debatable whether humans exclusively think in language.**

Thanks for mentioning this. While (as you noted) this issue is not essential for judging whether Thought Cloning is a valuable method (since it performs well), the claim is a central inspiration for the work. To recognize the claim is debated, we will update the text as follows: 

“While it remains debated whether humans exclusively think in language [Grandchamp et al. 2019, Alderson-Day et al. 2015], some argue that natural language is intricately woven into our thought processes [Premack 2004, Chomsky 2006], conferring distinct cognitive advantages. [Pinker 2003]”

**2. Concern about the availability of thought data:**

Reviewer 9iZD: ***“How can we effectively collect thoughts for actions to train TC agents?”***

Reviewer GnSR: ***“A major limitation to this approach mentioned in the introduction is the availability of applicable thought data.”***

We apologize for the confusion. It appears there may have been a misinterpretation regarding our stance on the availability of thought data. Our introduction intended to convey the opposite: that thought data is widely available, but it's an avenue that hasn't been sufficiently explored by the community. As an example, we could collect action demonstrations from unlabeled Youtube videos with VPT [Baker et al. 2022, Fan et al. 2022] and pair them with related video transcripts (the closed captions of the video where people narrate what they are doing and why while acting). This gives us a dataset encompassing both human cognition and action [Fan et al. 2022]. We proposed this approach in L62-66 and L91-96 in our paper. A contribution of this paper is introducing a method that can mine that heretofore under-exploited treasure trove of data. There are also other (albeit more expensive) ways to get data, such as human volunteers, contractors, employees, or vision-language models that create such narrations for videos.

Meta-review: Imitate both the behaviors of an agent and any language that agent generates while performing those actions. While other work attempts to add LLMs to RL agents in various ways, this method focuses on learning to generate the same "thoughts" as a task is carried out.

Reviewers were unanimous that this is an interesting direction that could lead to better imitation learning algorithms and richer settings. Raising the abstraction level of reasoning performed by agents is likely to pay off.

If I could give a conditional acceptance to this manuscript, I would. Throughout the manuscript the authors make the point that they imitate human thoughts. But they do not. They imitate language generated from rule-based descriptions of a planner. There is a big gap between the two, which is the underlying reasons for reviewers pushing back on the general applicability of this work. No dataset where humans describe their thoughts in this manner exists. The authors contend that one could use narrated YouTube videos, but there is no evidence that this would work. It is also why this work leads to complex discussions about the nature of thoughts.

It is far simpler to describe what the manuscript actually does: the method does not mirror human thoughts. It mirrors self-reports from a planner in natural language that describe what the planner was doing. I would encourage the authors to update their manuscript to make this clear. This actually makes their method stronger: any time you can have a planner generate descriptions you can apply their method to more quickly align your agent with that planner; perhaps in a way that generalizes better. The applicability of the method as originally described, while sounding far more grandiose, is actually very limited.

That being said, the method is novel, it is a promising method to incorporate language into RL, and so as the reviewers point out, it deserves to be published.
Decision: Accept (spotlight)