Policy Improvement using Language Feedback Models
Authors: Victor Zhong
Dipendra Misra
Xingdi Yuan
Marc-Alexandre Côté
Venue: NeurIPS 2024 poster (2024)
Keywords: instruction following
language feedback
language grounding
learning feedback model
imitation learning
Primary area: natural_language_processing
TL;DR: We train small and efficient language feedback models to identify productive behaviour in grounded instruction following environments, then imitate this behaviour to improve policy performance.

Abstract: We introduce Language Feedback Models (LFMs) that identify desirable behaviour --- actions that help achieve tasks specified in the instruction - for imitation learning in instruction following. To train LFMs, we obtain feedback from Large Language Models (LLMs) on visual trajectories verbalized to language descriptions. First, by using LFMs to identify desirable behaviour to imitate, we improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments (Touchdown, ScienceWorld, and ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict actions, when controlling for the number of LLM output tokens. Third, LFMs generalize to unseen environments, improving task-completion rate by 3.5-12.0% through one round of adaptation. Finally, LFMs can be modified to provide human-interpretable feedback without performance loss, allowing human verification of desirable behaviour for imitation learning.

Policy Improvement using Language Feedback Models
Victor Zhong" University of Waterloo Microsoft Research
Dipendra Misra Microsoft Research
Xingdi Yuan Microsoft Research
victor. w who.ooooooooguuuuaerooooooo
Marc-Alexandre Côté
Microsoft Research
Abstract
We introduce Language Feedback Models (LFMs) that identify desirable behaviour actions that help achieve tasks specified the instruction - for imitation learning in instruction following. To train LFMs, we obtain feedback from Large Language Models (LLMs) on visual trajectories verbalized language descriptions. First, by using LFMs to identify desirable behaviour to imitate, improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments (Touchdown, Science World, and ALFWorld). Second, imitation learning using LFMs outperform using LLMs as experts to directly predict actions, when controlling for the number of LLM output tokens. Third, LFMs generalize to unseen environments, improving task-completion rate by 3.5-12.0% through round of adaptation. Finally, modify LFMs provide human-interpretablee feedback without performance loss, allowing human verification of desirable behaviour for imitation learning.
Introduction
Sample-efficieney and generalizability are two primary challenges in learning instruction following agents in grounded environments [26] 23 2). First, we want an agent that is sample-efficient it learns from few demonstrations of how to act according to instructions. Second, we want an agent that is generalizable: it should act successfully in novel environments according to new instructions after training. Reinforcement learning (RL; Sutton and Barto (40) and imitation learning (IL: Schaal 32 Abbeel and Ng (T] are two techniques for learning agents for instruction following in grounded environments. These techniques often require large numbers of trials and errors or expensive-to-obtee expert demonstrations. Recent work show that pretrained large language models (LLMs) exhibit sample-efficient learning through prompting and in-context learning for textual [10] and grounded problems such as robotic control (2). However, for instruction following in grounded problems, current methods rely on LLMs on-line during inference, which is impractical and expensive.
We develop a sample-efficiett and cost-effective technique that uses LLMs to train Language Feedback Models (LFMss) for policy improvement in instruction following. Figure D illustrates policy improvement using LFMs. Consider the task of interacting with objects in a kitchen to follow instructions shown Figure [(c) First, in Figure ((a) given grounded environment and base policy (i.e. behaviour cloned policy), we roll out the base policy to collect small set of trajectories for different instructions. Next, we verbalize observations in the trajectory by describing scenes in language. For each instruction and verbalized trajectory pair, query LLM to provide feedback identifying which behaviour the trajectory is productive to solving the task identified
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

train
Initial
Rallouts
query feedback
LLM
batched
Feedback
feedback
Feedback
feedback
model
(a) Learning a small and cost-efiective Language Feedback Model from LLM feedback. We roll out an initial policy, then prompt LLM to provide feedback on what actions the policy took during the rollout were productive in achieving the task outlined in the instruction. We then this data to train feedback model that predicts whether an action productive given the instruction.
Dataset of desirable
Base policy
Rollouts
query
Feedback model
step-wise prediction
imitation
Improved policy
feedback
leamingg
behaviour
(b) Policy improvement by imitating desirable behaviour identified by a learned feedback model. Given the instruction, we roll out a base policy, then identify productive actions that help achieve tasks specified in the instruction using the trained feedback model. Finally, we update the base policy by imitating productive actions.
Instruction: clean some lettuce and put them the fridge
(c) Example of desirable behaviour identified ALFWorld, a kitchen instruction following benchmark.
Figure Given an environment and instructions follow, we assume verbalization procedure that converts observations to language descriptions. Policy improvement using Language Feedback Model involves (a) training a feedback model, then (b) using it to identify desirable behaviour for policy improvement via imitation learning. The feedback model is yellow, other models purple, and generated intermediate data green. An example LFM-iidetifiid behaviour is shown in (c).
in the instruction (i.e. answer yes or no). For instance, given an instruction "put a clean slice of lettuce in the refridgerator", GPT-4 [29] is able to deduce that key milestones are 1) find the lettuce. 2) slice it 3) wash in the sink, and 4) put it in the fridge. Consequently, such an LLM able to identify when an agent is exhibiting desirable behaviour conducive to solving tasks outlined in the instruction, for instance by taking the lettuce the sink, versus undesirable behaviour, for instance by cooking the lettuce. We define desirable behaviour as productive actions that are constructive, bekebbenf,,,a,,, and effective in following the instruction. In other words, taking the action brings the agent closer (in terms of trajectory length) to accomplishing the task specified in the instruction. After collecting LLM feedback, we distill this world knowledge into a small and cost-effective LFM. Finally, in Figure (b) given policy to improve on potentially new environments and instructions, we use the learned
identify desirable actions on-line, then update the policy to imitate these actions. Crucially, this technique is sample-efficient in that it improves policy with additional human-labeled demonstrations. Furthermore, this technique is cost-effective in that it requires few LLM interactions to collect off-line dataset during LFM training (i.e. before deployment), as opposed to many LLM interactions on-line during policy improvement (i.e. after deployment).
Our findings are as follows: first, LFM policy improvement achieves consistent gains over strong be- haviour cloned base policies on three grounded instruction following benchmarks in Touchdown [22 ScienceWorld [42) and ALFld (38) Second, we compare LFMs against prompting LLMs to directly predict what actions to take, then imitating this L.M--preditedd behaviour. On bench- marks, using LFM feedback outperforms using L.MSs as experts for imitation learning, given fixed allocation LLM output tokens. This gain is especially pronounced environments with larger action spaces, such ScienceWorld, where is much easier to critique than to generate the correct action. Third, we show that learned feedback models generalize to unseen environments with new tasks and new transition functions. After training LFMs training environments, identify desirable behaviour on test environments, which we then imitate to adapt the policy. A single round of adaptation achieves significant gains (3.5-12.0% task cisk-complttion across all environments.
them to
In addition to policy improvement, using LFM feedback offers two advantages over existing tech- niques such as using LLMs as expert policies for imitation learning. First, L.FM improves policies on-line without additional expensive calls to L.MSs. Second, LFM can offer human-interpretable feedback when identifying desirable behaviour to imitate. We show in Section 5.4 that LFMs can

---

busisi
crowistonn bullisi
bwiggn Burow iarownstonn busiss
claseest gray buslding
the ref "settroke"" buildbi
celored for encope
guildinng
vosblo esoetis
bollding
uotnwall builddings
buildiings
placo
pia secorii Cenniia pointing
far prad
ppertet blec
menb
verbalization
Instruction
Action
Verbelized Observation
straight ahead
Turn the scaffolding is your left and with the
Behind you, see: white jeep, large red brick building, the scaffolding, two trees, the next comer. To your left, you see: a large red brick buildingg the next corner. Straight ahead, you
flww of traffic to the next corner and turn right. When you there are orange cones in the road to go through both Keep going, through first...
building, large
An example verbalization for Touchdown. We align CLIP image embeddings of panorama patches and language embeddings common noun-phrases to populate language template. Ap- pendix D describes this procedure in detail. The blue arrow at the top indicate the agent's orientation while the green arrows indicate valid directions to proceed in.
Figure 2:
be easily modified to provide not only desirable behaviour but why they were desirable, thereby allowing humans to inspect and validate imitation data used for policy improvement. Source code for our environments and experiments are available at Videos LFM feedback are available
Background
Language grounded instruction following. In language-grounded instruction following, an agent given an instruction x specifying the task to achieve the environment. Each turn, the agent receives potentially partial observation O,, and takes action a, which causes the environment to transition to a new state. In the Figure[[[c]]e example, the agent observes a counter with objects such as toaster, some lettuce, and knife top. To follow the instruction "put clean slice of lettuce in the refridgerator". an effective agent may choose to grab a piece of lettuce. In the reinforcement learning setting, the environment additionally give the agent reward after desirable (positive reward) or undesirable (negative reward) action [40]. In this work, we consider long-horizon settings with only sparse and delayed task-completion rewards. Consequently, focus on imitation learning from demonstrations as opposed to reinforcement learning from rewards 32
Online imitation learning. In online imitation learning for instruction following, are given expert policy x "alz, 0) and learn a policy (a o) with parameters We first roll out the policy (8) of the rollout т.. optimize 0 to imitate the action at" chosen by the expert Te. For each step oz
o (a|z, (2) when given the same observations: arg ming E,,- [L (Taa, o(")), dz Here, L is step-wise cross-entropy between the policy's 's action distribution and the action chosen by the expert given the same observation: (x) - a
a!") In to a' x (
Behavioural cloning. Online imitation learning assumes expert policy that can be executed online produce expert actions. For instance, given an expert, imitation learning assumes that this expert π (a xr, 0t) provides corrective actions a, the policy π [aa or) runs. In many cases, this is impractical - a human-in-the-logg expert is expensive and inconvenient while an LLM expert expensive and, we show in our experiments, inaccurate. Alternatively, in behaviour cloning (BC), we instead collect an offline dataset of expert trajectories from which clone expert behaviour [6 BC (or offline imitation learning) only asks the expert to perform the task N times to collect N
3

---

trajectories [T.) Each consists of M; steps of observations and associated expert actions: (1) (4)
T = [of" servation of (0) We train policy Th to imitate the expert action, given the same observation seen by the expert, by minimizing the following objective: arg ming N
0 M. M. where a the action chosen by the expert x ((| of given the ob-
1 Σi M. E. L [T(aaz ).)), aq.")
The key distinction between BC and imitation learning is that the former optimizes over trajectories under the expert policy while the latter optimizes over trajectories under the learned policy. Conse- quently, while BC is offline and easily batchable, it suffers from covariate shift/exposure bias B 7
Like prior work in long-horizon instruction following in grounded environments [ 12), we use BC to warm-start a strong base policy 4 which we then improve using imitation learning.
3 Language Feedback Model
How can we leverage world knowledge in LLMs to make policy learning more sample-efficient and generalizable? In this work, we use LLMs to distill a small and cost-effective Language Feedback Model to identify desirable behaviour from a base policy (Figure I(a)). We then improve the base policy by imitating this desirable behaviour through batched imitation learning, without need for on-line LLMs (Figure l(b). Appendix AppeendixEEprooveeeee provides pseudo-code for the entire procedure for policy improvement using LFMs. A natural question is why not directly use LLMs as experts for action prediction. Section 5.4 shows that the using LLMS to learn feedback models results in higher policy improvement than using LLMs experts for action prediction. Moreover, LFMs generalize to new environments unseen during training, thereby allowing policy improvement on new environments.
3.1 Verbalization
To leverage world knowledge in LLMs, we convert raw observations to language descriptions using a verbalization procedure V. Figure FigureZillustrate illlstrates verbalization for Touchdown [12]. where the agent navigates Google Street View panorama images based on given natural language instruction. First, we extract all noun-phrases (NPs) from instructions in the dataset and compute their CLIP language embedding. Given visual observation, compute CLIP visual embedding for each image patch, and align it to the NP with the highest cosine similarity between CLIP embeddings. We then combine aligned NPs with agent orientation formulate egocentric language description of the scene. This is described in more detail in Appendix D
3.2 Learning a feedback model
Naively learning from LLM feedback. Given verbalization procedure V, instruction x, an LLM, and a policy To, we now describe a procedure to use the LLM''s knowledge to improve Ty. First, prompt the LLM to provide feedback on whether a particular action taken by the policy [a(a v) is productive in achieving the tasks outlined in the instruction æ. We then improve the policy mo by updating its parameters to imitate desirable behaviour determined by the LLM. Let : denote "such that". Let LLM(x, v, a) return yes if and only if the LLM feedback indicates that action taken in verbalized state and instruction is productive, Given set of instructions X {a+) the optimization procedure is then arg ming Ee,a' ":ai(,,,ei(( L (Ta (a|æ, v) a'). Here, instruction x is sampled from X. Observations v and actions a' are sampled from rollouts of the policy no.
Efficiently learning a language feedback model. While the previously described naive learning is a reasonable procedure for using I.IM feedback to improve the policy, it requires calling LLMS at each step during policy improvement. This is prohibitively expensive both in terms of query cost, because I.MMs capable of giving desirable feedback are expensive to run, and training time, because generating feedback using large LLMs is slow. Instead of using the LLM each step, we make modification to collect LLM feedback over long horizons in batch [14] in order to train a small and cost-effective language feedback model.
First, for instructions (2)
we roll out the base policy xa to collect set of trajectories (TI, ....... consisting of verbalized observations and actions taken: т
{uf" (1) ((x(),))))) (1) (1)
0/2 (1((1), o22 ..... For each t,, we prompt the LLM for feedback on which
steps were productive in achieving the instruction z(t), Table 2 LFM row shows an example of

---

requesting feedback from GPT-4 on a rollout in ALFWorld, which is an instruction following bench- mark in verbalized 3D kitchens. This LLM feedback then parsed to identify the precise steps in which the base policy took a productive action towards achieving the goals outlined in the instruction. The set of desirable behaviour compiled into dataset F. Let y L.M,( v, a) denote the feedback given by the LLM for the instructions observations v, and action a. We use the dataset F = f2x(t) U, a, y Vee a E Tzxx(1) T to train a small Language Feedback Model f:
arg min
Σ
L (y (,,,a),9").
(1)
(B.U.0.3")EP
Here, L is the cross-entropy between the feedback model output fo and gold label y from the LLM.
Learning from language feedback. The naive learning procedure in Eq (.2) updates the policy after each step using slow and expensive LLM feedback. Here, we instead update the policy in rounds using fast and cost-effective LFM feedback. In round k, we rollout the base policy x (R) and use the
(k) denote the action chosen feedback model f to collect a dataset D2 of desirable behaviour. Let di by policy π ( æ, v.). Let DESIR DESTRABLER ABLE( v, a) = I (y = yes | x; U, a) > f (y = no xt, v, a) return whether the feedback model predicts that action desirable. In the keth round, we collect the dataset of desirable behaviour DR- = (r, Vt, dz (k) vt: DESIRABLE(Z, Ut, dj a(())) } which we
combine with previously collected behaviour to update the policy via imitation learning:
0" arg min
Σ
L (m (A) x, v.),)
8
P PU P4
In the next round, we set the base policy (k++1) parameters to fr Should demonstrations be available, we initialize the base policy at k = 1 to the BC policy, and train both demonstrations and identified desirable behaviour during subsequent rounds (i.e. Do where Do are demos used to train BC).
4 Related Work
Instruction following in grounded environments. Instruction following in grounded environments has been explored in settings such as navigation [1 18 12), game-playing 48), and robotics [8 37
9. However, most prior work model environment observations separately from language instructions by using specialized encoders (e.g. RESNET [19), BERT [6), CLIP (30), then learn from data how to associate raw observations with language instructions. Instead of solely using raw observations, more recent work verbalize raw observations to describe environments in language [38] 49 34]. In doing so, observations and instructions can be directly jointly reasoned over using language models to achieve more efficient and generalizable learning through large-scale pretraining. We build on this last direction by verbalizing raw observations into language descriptions to train language policies. However, unlike prior work that train language models to predict next actions, we develop language feedback models that critique verbalized observations and behaviour.
LLM agents in language settings. LLMs exhibit reasoning abilities after pretraining on vast quantities of text IT
number of recent work on LLMs language agents exploit this reasoning Yao et al. [46] Deng et al. [15] train instruction following language agents to interact with web browsers to answer questions or interact with web pages. Ahn et al. [2] show that a language agent can be connected with verbalized robots via API interfaces for robotic control. Xie al. [4]] use large visual language models (VLMs) for instruction following in virtual machines, but show that LLMS with verbalization outperform VLMs. While powerful, these prior work are limited in that they require querying an expensive LIM on-line. In contrast, our work examines settings where an L.M is not available on-line. Specially, we use LLMSs to collect a small set of off-line data for training LFMs. The small and cost-effective LFMs are then used to identified desirable behaviour for on-line policy improvement without additional interactions with the LLM.
43
ability. Nakano et al.
28
Learning from feedback. Recent work enhance language agents by augmenting them with feed- back. Ziegler et al. (50], Stiennon et al. [39] and Bai et al. [5) learn reward models from human preference to improve policies via reinforcement learning (RL). Instead of using human feedback, Bai et al. [5] and Lee et al. [4] use LLM feedback to train separate reward model for RL for textual
5

---

Table 1: Examples of verbalization. We abbreviate long verbalized observations using "..."
Benchmark
Context
Action
ALFWorld
Task:
heat some egg and put it in diningtable.
go to microwave
Observation: You arrive at loc 12. On the sinkbasin 1. you sec...
T-1 Observation: You are in the middle... Action: go to sinkbasin
T-2 Observation:
ScienceWorld
Task:
Your task to find a(n) living thing. First, focus on the thing. Then. open door move it to the purple box in the bathroom.
to outside
Observation: You move to the kitchen. This room is called the kitchen, In it, you see: the agent substance called air a chair. On the chair is... In your inventory, you see: an orange... T-1 Observation:
The door is now open. Action: go to kitchen
T-2 Observation... Action: open door to kitchen
Touchdown
Task:
Follow the flow of traffic, with the row of flowers on your left and straight make a left at the intersection. There will be a white billboard...
ahead
Observation: behind you. you sec: the right lane intersection, large... behind you, slightly... Action: slightly to your left
T-1 Observation:
use LLMs to reason about potential resolutions to failed actions. Yuan et al. use LLMs to generate new prompts and corresponding re- sponses, then use an LLM reward model to identify good prompt-response pairs for self eef-impprvemmnt in text generation alignment. Unlike these approaches, we do not use LLMs during on-line policy improvement. We train an initial small language feedback model from offline LLM data, then use this small feedback model for policy improvement. Additionally, we focus on-line improvement via language feedback for oon-hhoiioon, sparse reward, grounded environments instead of text generation alignment. Our procedure for batched, on-line imitation learning is similar to DAGGER [31], which we compare to in Appendix FF However, we collect batched expert feedback to identify desirable behaviour instead of corrective actions. Klissarov et al. 22 and Du al. [17 recent works that
alignment. Huang et
Yao et al. [66 and Shinn et al. 35
describe learning from feedback approaches complementary to ours. The former learns preference models based on pairwise observations while the latter uses LLMs suggest exploratory goals during training. Unlike these works, which assume that the underlying goal is the same between training and inference, we consider settings where training and evaluation goals are different. That said, one can expand these approaches to generalize to unseen environments by adapting a preference model during inference (22) and by goal-conditioned subgoal generation during inference (17). However, unlike LFM, these modifications would then rely on calling LLMs during inference.
5 Experiments and Analysis
We evaluate using LFM for policy improvement on three distinct language grounding benchmarks. Formally, the environments from benchmark are distinct partially-observed Markov Decision Processes that share some (or all) of the environment dynamics but have different instructions, observations, and/or action space.
5.1 Evaluation benchmarks
Table shows examples of verbalized environments and tasks from each benchmark. Each benchmark provides distinct training and test environments to test generalization. each environment, the agent takes actions to perform tasks outlined in language instruction. The task is considered completed if and only if the agent solves the tasks within the preallocated number of steps. We evaluate using task-completion rate over test environments. The statistics from each benchmark is shown in Appendix D Table 6 These three benchmarks share challenges in sparse, delayed reward, partial observability, and compositional generalization to unseen tasks and environments.
FLFWorld is a verbalization of ALFRED 36), a natural language instruction following benchmark set in 3D simulated kitchen. Here, the agent interacts with objects in kitchens to achieve composi- tional goals such as cleaning then microwaving potatoes. FLFWorldd 38) raw state information from ALFRED are used populate language templates that describe observations in language.

---

Table 2: LLM prompts used to collect desirable behaviour. ACTPRED uses LLMs directly generate actions for each step, whereas LFM uses LLMs to generate batch feedback that identify which taken actions productive. For brevity, abbreviate long verbalized observations using ... "Before" contains the observation before the first step in the batch.
ACTPRED
Prompt
Your task is: look at alarmclock under desklamp.
You see: you are in the middle of room. looking quickly around you, you see a bed I... what do you decide to do? available actions: examine shelf 1, examine shelf 2, go to bed...
You decide to: go to desk 1.
You see: you arrive at desk what you decide to do? available actions: examine desk...
You decide to:
LLM Output
examine desk
LFM
Prompt
You will be shown playthrough for solving a task.
Task:
put two candle in drawer.
Before:
You open the drawer 6. The drawer is open. In it, you see nothing.
Step 27. Your action: close drawer Result: You close drawer
Step 28. Your action: put candle in/on drawer 1. Result: You put the candle 3 in...
Is the player on the right track to solve the task?
Answer yes
yes, list the helpful steps by the step number in bullet form.
LLM Output
Yes Step Step 29...
ScienceWorld is textual simulation benchmark for basic science experiments 42 The agent interacts with objects to conduct experiments specified in natural language, such as determining the boiling temperature of a material. Science World uniquely challenging to due the large amount of variations in task types (30), and parametric variations (10-1400) such as the specific substance to be melted. Furthermore, ScienceWorkd has a substantially larger action space and longer horizon tasks.
Touchdown is navigation benchmark where the agent navigates Google Street View images to follow long, compositional instructions [[2]. Touchdown requires jointly reasoning over natural images from Google Streetview with occlusion and multi-sentence natural language instructions that describe long-how oag-horizonn goals. We introduce a new verbalization procedure for Touchdown based on matching noun-phrases and image patches with CLIP embeddings to populate egocentric language templates. Behaviour cloning using our verbalization is detailed in Appendix [] Touchdown considers multiple subtasks, in this work we only test the agent's 's ability to arrive at the correct. location according to the instruction.
5.2 Methods
We train BC baseline policies using existing demonstrations. We examine three different techniques for improving the BC policy. Table 2 shows examples of LLM prompts used for each technique.
imitation learning from LLM experts. We compare directly using L.MMs experts to predict actions for imitation learning. First, we execute steps of the base policy, then query the LLM for the next action given the instruction x and the verbalized observations v. We repeatedly collect examples (I, a), then train the policy using this collected data and BC demonstrations.
ACTRRED:
LFM: imitation learning using feedback models. We learn small and cost-effective feedback model described in Section 3.2 to identify desirable behaviour for imitation learning. First, we learn feedback model on the training environments. Second, we use the feedback model to identify desirable behaviour in the training environments for policy improvement via imitation learning. To collect LLM feedback for training LFMs, we collect one rollout for each enviromment in a benchmark and sample 10k 20-step windows from the rollouts. Crucially, we limit the amount feedback data collected from the I.IM such that the number of output tokens produced by the LLM is identical to ACTPRED (we use 100k GPT-2 tokens for benchmarks). For LFM, we collect feedback for as many windows possible until we exceed 100k output tokens, then use this feedback to train the
7

---

LFM. For ACTPRED, we label actions until we exceed 100k output tokens, then combine this labeled set with demonstrations to train the policy. This limitation on LLM interactions answers whether the feedback model is more cost-effective than direct action prediction for imitation learning.
LFMA: adaptation using feedback models. LFM only imitates desirable behaviour in training environments. In contrast, LFMA adapts the policy to test environments. Given new test environments, we identify desirable behaviour using feedback models trained on the training environments, then perform one round of imitation learning to adapt to new test environments. This experiment tests whether language feedback models generalize to new environments, and whether we can use their feedback to adapt policies to new environments without using LLMs nor additional demonstrations.
5.3 Experiment details
We use the GPT-4 (2023-03-15) for tion prediction and feedback, and finetune 770M FLAN-T5 [3] for policy and feed- back models. Verbalized observations v contain the most recent 20 steps. We train models for 10k steps with batch 20, learn- ing rate 5e-5, and early stopping over val- idation demos. For ACTPRED and LFM, we limit the amount of LLM usage 100k GPT-2 tokens. Touchdown verbalization uses vit- aet-large-patch14 Appendix H details GPU usage.
Table 3: Task completion rates of behaviour cloning Bc, imitation learning (IL) using LLM expert ACTPRED, and IL using LFM. On held-out test environments, LFM outperforms other methods on all benchmarks. ACTPRED and LFM are limited to 100k output tokens of GPT-4 interactions. Further adaptation to the new en- vironments using LFM results in significant additional gains (LFMA).. Errors are standard deviations across seeds. Previous SOTA are Micheli and Fleuret [27] for ALFWorld, Lin et al. 25 for ScienceWorld, and Schu- mann and Riezler (33 for Touchdown. Unlike Lin et al. (25), our methods do not use MorldceWWrdd-speeeee cus-
tom room tracking nor action reranking.
Feedback model training and inference. We train LFMs using I.LM feedback over 20-step windows. We then parse feedback to identify whether the action taken in each step was productive solving the tasks outlined in the instructions. We subsample the feedback data to obtain an even split of productive and not-productive actions. This data is split into a 80% train/20% val- idation dataset to train the LFM.
ALF
SciWorld
TD
Prev SOTA
57.6
45.8
29.3
GPT-4 zeroshot
3.0
1.3
3.2
BC ACTPRED LFM
0.4
45.8
0.3
56.0
39.0
58.0
64.1 0.3
47.1
59.7 + 0.1
LFMA I md LFMA rnds
7..6 1.3
49.3
12.8
1.0
1.2
Policy training and inference. To train policies, we fine-tune language models to minimize token- wise cross-entropy of the ground-truth verbalized action. During inference, we consider (potentially very large) set of plausible actions given by the environment. For each action, we evaluate the policy's language model perplexity, and choose the action with the minimum perplexity averaged over tokens.
5.4 Results and discussion
Table 3 shows the performance of the policy behaviour cloned from demonstrations BC, imitation learned from LLMs using action prediction ACTPRED, and imitation learned from LFM. For LFMs, show zero-shot (LFM) well adaptation (LFMA) results.
LFMs improves policy performance across all benchmarks. Table shows that LFM improves upon the strong behaviour cloning baseline policy BC in all benchmarks. Table 5 shows examples of LFM-identified desirable behaviour. This shows that LFMs are an effective means to leverage the knowledge in pretrained LLMs for policy improvement in language-groundee
environments, which agree with -identiidde desirable behaviour. Appendix G alsoo compares GPT-4 to the open-source LLAMA 2 70B for training feedback models using human evaluation. We find that GPT-4 consistently outperforms LLAMA 2, which tends to identify spurious desirable behaviour.
Learning LFMs is more cost-effective than using LLMs for action prediction. Assuming the same L.M output-token quota, Table compares using L.LMs to train feedback models (LFM) to using LLMs to predict actions (ACTPRED) for policy improvement. Specifically, ACTPRED tends
8

---

Table 5: Example of detailed numan-interprette feedback. The prompt and output differences between succinct feedback (Table 2 and detailed feedback are highlighted in red. Note that the second row shows shows batched L.LM inference, which used to train a step-wise feedback model. The wise inference of the feedback model is shown in the last row. We leave providing summaries and suggestions with the learned feedback model to future work.
Prompt
You will be shown playthrough for solving task. Task: put two candle in drawer. Before: You open the drawer 6. The drawer 6 is... Step 21. Your action: close drawer 6. Result: You... ... Is the player on the right track to solve the task? Next under heading #Summary, summarize in one sentence what the player doing at high level. Finally under heading #Improvement, describe how the player can improve their strategy to solve the task.
LLM output Yes
Step 28: The player locates the candles by arriving at countertop 1, where a candle available.
Step 29: The player successfully takes candle 3 from countertop I, demonstrating the ability to collect... #Summary
The player gathering candles and placing them in drawer, working towards completing the task of... #lmprovement
The player can improve their strategy by: not closing drawers unnecessarily...
Learned feedback model output
Yes. The player successfully locates the candle... Yes. The player correctly takes the candle from...
to predict spurious actions, especially for complex environments with large actions spaces such as ScienceWorld, In contrast, the difficulty in identifying productive actions independent of the action space, and LFM consistently improves policy even with large action spaces. This shows that LFMs is a more cost-effective means use LLMs for policy improvement compared to using LLMs as expert policies for imitation learning.
LFMs generalize to new environments, allowing for policy adaptation without additional LLM usage nor demonstrations. Table&[shows that LFMs trained during language feedback learning can accurately recognize desirable behaviour in new environments. Table shows that imitating this behaviour obtains significant policy improvement across all benchmarks. This shows that LFMs generalize to new environments, which allows for policy adaptation to new environments despite not having demonstrations nor LLM access. Appendix shows additional experiments that demonstrate correlation between LLM/LFM feedback and true state values.
LFMs can provide human-interpretahle feedback, al- lowing moman-in-thh-lopg verification during policy improvement. LFMs improve policy performance with succinet feedback. Here, we extend them to additionally provide detailed explanations. Consider an instruction "turn left when you see the stop sign then go to the second building the right". Suppose that in the current step the agent proceeds straight, arriving at the stop sign. Instead of feedback saying "yes" (i.e. the action was productive), the LFM can provide a human-interpretable explanation for why this action was productive (i.e. "yes because you found the stop sign where you are supposed to turn"). Ta- 5 shows that we can enhance LFM to produce detailed feedback by training detailed feedback prompted from LLMs. Specifically, we train a detailed LFMD to simul- taneously identify productive actions, summarize agent intent, and suggest potential high level recovery strategies. Sable|[showr that surprisingly. LFMD that produce detailed feedback perform similarly to those that provide succinct feedback. This shows that Language Feedback Models can be used to provide accurate feedback interpretable to humans. While interpretable feedback requires more costly LLM usage, it allow for human-in-the loop verifi- cation of desirable behaviour identified by the LFM. Consequently, interpretable LFMs promotes user trust in the quality of the imitation learning data and subsequent policy behaviour.
Table 4: Feedback performance of LFM. We measure F1 score of the productive/not-proouciii made by the learned LFF using the LLM predictions as ground truth. We observe no significant performance degradation when using a much more detailed feedback model (LFMD) that also provides explanations behind the feedback, summaries of agent behaviour, and strategy suggestions.
predictions
ALF
SciWorld TD
LFM LFMD
93.2
83.7
43.9
92.0
82.5
42.5

---

Conclusion
We introduced Language Feedback Models that identify desirable behaviour for imitation learning. On three instruction following benchmarks, small and cost-effective LFMs consistently outperform BC baselines and using LLMs as experts for imitation learning, without using LLMs during policy improvement. In addition, LFMs generalize and provide significant policy adaptation gains on new environments, without using LLMs nor new demonstrations. Finally, LFMs, can provide detailed human-interpretable feedback that human verification of imitation data. We advocate for future exploration of how to exploit detailed LFMs, such as learning dense, subgoal-aware reward models for RI., and trustworthy policies with human verification.
References
[1] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
[2] M. Ahn, Brohan, N. Brown, Chebotar, Cortes, B. David, Finn, C. Fu, Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, Xia, T. Xiao, Xu, S. Xu, Yan, and A. Zeng. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, 2022.
[3] J. Andreas and D. Klein. Alignment baased compositional semantics for instruction following. In EMNLP, 2015.
[4] J. T. Ash and R. P. Adams. On Varm-Starting Neural Network Training. In NeurlPS, 2020.
[5] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernande7, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Lar- son, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield -oddds Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional Al: Harmlessness from AI Feedback, 2022.
[6] M. Bain and Sammut. framework for behavioural cloning. In Machine Intelligence, 1995.
[7] Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In NeurlPS, 2015.
[8] V. Blukis, Y. Terme, E. Niklasson, R. A. Knepper, and Y. Artzi. Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight. In CoRL, 2019.
[9] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-2: LiooonLLngaag-Acttionn Models Transfer Web Knowledge Robotic Control, 2023.
[10] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger. T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, Sigler, M. Litwin, Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. In NeurlPS, 2020.
[11] D. Chen and R. Mooney. Learning to Interpret Natural Language Navigation Instructions from Observations. AAAI, 2011.
10

---

[12] H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi. Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments. In CVPR, 2019.
[13] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling Iestruction-Finetunee Language Models, 2022.
[14] C. Colas, L. Teodorescu, P.-Y. Oudeyer, X. Yuan, and M.-A. Côté. Augmenting autotelic agents with large language models. In Proceedings of The 2nd Conference Lifelong Learning Agents, PMLR. PMLR, 2023.
[15] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2Web: Towards Generalist Agent for the Web. In NeurIPS, 2023.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACI, 2019.
[17] Y. Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas. Guiding Pretraining in Reinforcement Learning with Large Language Models. In ICML, 2023.
[18] D. Fried, J. Andreas, and D. Klein. Unified Pragmatic Models for Generating and Following Instructions. In NAACL, 2018.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition, 2015.
[20] M. Honnibal and I. Montani, spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
[21] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, Levine, K. Hausman, and Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. In CoRL, 2022.
[22] M. Klissarov, P. D'Oro, S. Sodhani, R. Raileanu, P.-L.. Bacon, Vincent, A. Zhang, and M. Henaff. Motif: Intrinsic Motivation from Artificial Intelligence Feedback. In ICLR, 2024.
[23] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In HRI, 2010.
[24] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, and S. Prakash. RLAIF: Scaling Reinforcement Learning from Human Feedback with Al Feedback, 2023.
[25] B. Y. Lin, Fu, K. Yang, Brahman, Huang, C. Bhagavatula, P. Ammanabrolu, Y. Choi, and X. Ren. SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks, 2023.
[26] M. MacMahon, B. Stankiewicz, and Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI, 2006.
[27] V. Micheli and F. Fleuret. Language Models are Few-Shot Butlers. In M.-F. Moens, X. Huang, L. Specia, and W.t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
[28] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. WebGPT: Browser-assisted question-ansseriigg with human feedback, 2022.
[29] OpenAl. GPT-4 Technical Report. http::/aarivooraaass333.887777 2023.
[30] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning Transferable Visual Models From Natural Language Supervision, 2021.
11

---

[31] S. Ross, G. J. Gordon, and J. A. Bagnell. A Reduction of Imitation Learning and Structured Prediction No-Regret Online Learning. AISTATS, 2011.
[32] Schaal. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 1999.
[33] R. Schumann and S. Riezler. Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
[34] R. Schumann, W. Zhu, W. Feng, T.-J. Fu, Riezler, and W. Y. Wang. VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View. In AAAI, 2024.
[35] N. Shinn, Cassano, B. Labash, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: language agents with verbal reinforcement learning. In Neural Information Processing Systems, 2023.
[36] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020.
[37] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In CoRL, 2021.
[38] M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[39] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and Christiano. Learning to summarize from human feedback. In NeurIPS, 2020.
[40] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[41] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation, 2018
[42] R. Wang, P. A. Jansen, M.-A. Côté, and P. Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader? In EMNLP, 2022.
[43] Wei, X. Wang, Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of- Prompting Elicits Reasoning in Large Language Models. In NeurlPS, 2022.
[44] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In NeurlPS, 2024.
[45] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang. and T. Yu. Rewa2Reward: Automated Dense Reward Function Generation for Reinforcement Learning. In ICLR, 2024.
[46] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR, 2023.
[47] W. Yuan, Y. Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. Weston. Self-Rewarding Language Models, 2024.
[48] V. Zhong, T. Rocktäschel, and E. Grefenstette. RTFM: Generalising to Novel Environment Dynamics via Reading. In ICLR, 2020.
[49] V. Zhong, A. W. Hanjie, S. I. Wang, K. Narasimhan, and L. Zettlemoyer. SILG: The Multi- environment Symbolic Interactive Language Grounding Benchmark. In NeurIPS, 2021.
[50] D. M. Ziegler, N. Stiemnon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and Language Models from Human Preferences, 2020.
G. Irving.
12

---

A Limitations
This work proposes using LLM feedback to improve policies for long-horizon planning in grounded environments. It assumes access to a verbalization module that faithfully describes observations in language. For some practical problems (e.g. rich scenes in operating systems Xie et al. (45), verbalization has been shown to be difficult problem, which limits the capability of language feedback models. Moveover, LLMs have been shown to hallucinate, especially grounded settings that are uncharacteristic in internet pretraining data [42. Consequently, LLMs may provide inaccurate feedback, which limits policy improvement gains.
Our empirical results demonstrate consistent policy improvement using LFMs on three distinct language grounding benchmarks. These benchmarks cover kitchen interactions (AFLWorld), scientific experiments SSeienceWordd), and real-scene navigation (Touchdown). Future work should investigate the application of LLM feedback learning to real-world robotics, as an important test-bed for LLM ability to provide precise feedback on real-world observations.
While this work significantly reduces the computational burden of using language models policies in grounded environments by extracting world knowledge from LLMs into small LMs using language feedback, the LM policies we use still require large GPUs train. Specifically, we note significant performance degradation (e.g. 10% task completion) when changing from 770M Flan-T5 to the smaller 250M variant. Future work should investigate techniques to further reduce model size that LMs can be tractably used as policies on small devices such as phones.
B Broader Impacts
Although this work develops LM as policies grounded environments, the outputs these policies generate are specific (i.e. plans) to the benchmark environments used. Consequently, the generated outputs are not suitable for malicious use (e.g. disinformation, fake profiles). One potential misuse of proposed method lies in central party providing a malicious feedback model, which provides negative reinforcement to good behaviour and positive reinforcement to bad behaviour. Learning from such feedback model can potentially result in malicious downstream policies.
C Licenses for existing assets
We will release our code and resources under the MIT license. This work uses assets from the Flan-TS series of models, OpenAl GPT-4, Llama-2, and the three benchmarks. The licenses follows:
Flan T5: Apache 2
OpenAl GPT-4: fair use policy
Llama-2: community license
ALFWorld: MIT
WorlnceWorld: Apache
Touchdown: Attribution 4 International
13

---

Table 6: Statistics from benchmarks as measured by training demonstrations. The are the average number of GPT-2 tokens in the instruction. verbalized observation, and action; the average demon- stration steps; the average number of plausible actions a state; the number of unique actions, instructions, and observations; and finally the number of training demonstrations.
ALFWorld
SciWorld
Touchdown
Ins len [a]
8.8
64.7
93.4
Obs len [
23.9
239.4
284.9
Act len |a]
4.5
6.0
2.4
Traj len ||]
19.7
55.1
34.2
Act space
29.9
1.9k
2.1
# act [a]|
2.6k
2.4k
8
# ins
{-)
1.0k
1.2k
6.5k
# obs {w}
18.2k
157k
34.3k
# demos
3.5k
3.6k
6.5k
D Verbalization of visual environments
How can we leverage the world knowledge learned by LLMs from pretraining on vast quantities of text? In many instruction following problems, environment observations are inherently visual. In this section, we describe a verbalization procedure that converts visual observations to language descriptions, that LLMs can make inferences by jointly referring to the instruction and environment observations. Specifically, we use Touchdown as an example.
As shown in Figure 2 Touchdown [1] is primarily a visual navigation problem. Given set of connected Google Streetview panoramas that represent neighbourhoods Manhattan, an agent must follow long, complex natural language instructions to navigate the correct location. Crucial to this task of navigation are landmarks referred to by the instructions. For instance, the instruction "turn so the scaffolding is your left and... to the next corner and turn right..." " refers to the land- marks scaffolding and next corner. Prior work in verbalization use LLMs to identify landmarks (34). In this work, take the set of common noun-phrase ahunssessses in the corpus of instructions to be landmarks.
Extracting aligned noun-phrase annotations for visual patches First, we identify all noun- phrases using SPACY (20]. Given visual scene, we divide the scene into 300x300 pixel non- overlapping patches. For each patch, we identify the noun moun-phrase with the highest cosine similarity between the noun-phrase text embedding and the image patch embedding. We use text and visual encoders from CLIP [30] to extract embeddings for each modality. For patches with no aligned noun-phrase with cosine similarity greater than 0.2, we do not provide annotated a noun-phrase.
Converting to verbalized observations To obtain verbalized observations in the form of an egocentric scene description, we consider the direction the agent is facing (shown in blue) well the directions of possible next steps (shown in green). The noun-phrases identified the scene are then categorized into 8 directions in 45-degree increments, relative to the agent's 's current orientation: straight ahead (337.5 to 22.5), slightly to your right (22.5 to 67.5), to your left (67.5 to 1 112.5), behind you, slightly your right (1 12.5 to 157.5), behind you (157.5 to 202.5), behind you, slightly your left (202.5 to 247.5). to your left (247.5 to 292.5), and slightly to your left (292.5 to 337.5). A scene is then rendered as follows:
Straight ahead, you see white van Slightly your right, you red brick building, scaffold...
It is important to note that Touchdown consists of multiple subtasks, such as finding the correct scene, stopping at the correct scene, and then finally orienting to face a hidden object in the scene. this work, similar to Zhong et al. [49], we only consider the first task of finding the correct scene according to the instruction. To compare our verbalization with prior work, we also evaluate a separate setting (e.g. used in Chen et al. [2], Schumann et al. 34 where the agent must identify when to stop and is credited so long as it stops within one panorama of the target scene. Behaviour cloning using our verbalization technique results in 63.0% task-completion rate.
14

---

Statisties of verbalized environments In A Table 6 environments as quantitative evidence of their challenges.
we show statistics of verbalized
15

---

E Pseudocode for Policy Improvement using Language Feedback Models
In this section we detail, using pseudocode, the procedure for policy improvement using Language Feedback Models. Algorithm [ describes learning model from LLMs. Algorithm 2 describes identifying desirable behaviour that are productive for solving tasks specified in the instruction, and then using this behaviour for imitation learning. Algorithm 3 describes the iterative policy improvement procedure using these two algorithms.
Algorithm 1 I'RAINFFEDDRCKKOOEEL Training Language Feedback Model using LLM feedback.
1: Inputs: initial policy LLM LLM, environment E
2: Feedback dataset F <- 0
3: for = N...N do
4: x < SAMPLEINSTRUCTION
5: T; f- ROLLOUT(Z, E..)
6: for window w in т; do
7: Y OUFRNILLNFORFFEEDACCCCLLMM w)
8:
for verbalized observation ve. LLM feedback yi in each step of y do
9: F FUk yk)
10: end for
end for
12:
end for
Feedback model f f (RAINLM(F)
Algorithm 2 IMITATBUSSINGFEEDDACCK Imitation learning using desirable behaviour identified by feedback model.
Inputs: base policy n, environment F., feedhack model ʃ
2: Imitation dataset G +- behaviour cloning dataset
3: for = NN..NN do
4:
SAMPLEINSTRUCTION
5: т; + ROLLOUT(T, E,
6: for verbalized observation tk. action dş in each step of т; do
7: Us = f
8: if yk is desirable then
9:
G - G ((v,, aR)
end if 11: end for
end for
13:
Improved policy n' + TRAINLM(G)
Algorithm 3 Policy improvement using Language Feedback Models.
1: Inputs: base policy environment E
2: Feedback model f + TRAINFLEDBACKMODEL (T, LLM, E)
3: To
4: for 1...N. do
5: TTk <- IMIITTTEUSSNNGFEEDRACCCCI--TL
6: end for
16

---

Table Task completion rate on evaluation benchmarks, including DAGGER.
ALFWorld
ScienceWorld
Touchdown
BC
62.6
45.8
57.5
ACTPRED
56.0
39.0
58.0
DAGGER
55.2
22.5
50.2
LFM
64.1
47.1
59.7
LFMA
74.6
49.3
62.8
Comparison to DAGGER
Our main experiments in Section 5.4 illustrate the diffieulty of using LLMs expert to predict actions. Specifically, we show that when these predictions are used for imitation learning, the resulting policy improvement is worse than using Language Feedback Models. This performance degradation is exacerbated in environments with larger action spaces, such as ScienceWorld.
DAGGER [31 is an intermediate method between Language Feedback Models and using LLMs as an expert policies for imitation learning. Specifically, DAGGER, we also use L.MSs as experts to predict action. However, instead of using LLMs during each step, in DAGGER to provide batched retroactive action prediction similar to how Language Feedback Models we use LLMs to provide batched retroactive feedback. Here, apply DAGGER action prediction to the exact same number of examples as when we collect feedback data for LFMs. In Table 7 we compare DAGGER performance to those using LLM expert (ACTPRED) and using Language Feedback Models (LFM). We find that although DAGGER is more efficient than ACTPRED in that annotates synthetic examples in batch, it underperforms ACTPRED (and consequently LFM) across all benchmarks.
use LLMs
17

---

Table 8: Agreement between GPT-4 and LLAMA across the benchmarks. We collect steps from rollouts on the training environments where either GPT-4 or LLAMA 2 identified productive action. This table shows percentage of of those actions that are identified exclusively by GPT-4, exclusively by LLAAMA 2, and identified by both models. The total number of steps identfied are 40569 for ALFWorld, 68565 for ScienceWord and 90529 for Touchdown.
GPT-4 only
LLAMA 2 only
both
ALFWorld
14.4%
49.3%
36.2%
Science World 10.2%
62.3%
27.5%
Touchdown
22.3%
67.3%
10.4%
Table 9: Human verification of LLM feedback in terms of percentage of true positives and false positives. A true positive (TP) is step that is correctly identified by the LLM as being productive to solving the task. A false positive (FP) is a step that is wrongly identified by the LLM as productive. We manually evaluate 10 examples from each benchmark, each with up 20 steps. Support (# of steps) is shown in brackets.
GPT-4
LLAMA 2
TP
FP
TP
FP
ALFWorld WoillceWorld
100% (22)
0
32% (18)
68% (38)
78% (38)
22% (11)
48% (38)
52% (41)
Touchdown
81% (22)
19% (5)
39% (24)
61% % (38)
G Quantitative and Qualitative Analyses of Learned Language Feedback
How much difference is there between language feedback obtained from the open-source LLAMA 2 vs from GPT-4? Table 8 shows that, surprisingly, there is a large degree of disagreement between GPT4 and LLAMA 2. Specifically, LLAMA 2 identifies significantly more actions as being productive achieving the goal.
Comparison of GPT-4 to LLAMA 2 70B
We perform a manual analysis of language feedback by GPT-4 and LLAMA to characterize qualitative differences between feedback collected by these two models. First, we roll out BC policies, then ask each model for feedback. Each example contains a segment of up to 20 steps extracted from rollout, and the LLM is prompted to list productive steps. For each step the LLM identifies as productive to solving the task, we manually verify whether the step is indeed productive. We manually inspect 10 examples from each model for each benchmark, for a total of 10 x 3 60 examples. Table shsows the number of true and false positives predicted by both models in this manual evaluation. We find that a significant number of steps are incorrectly determined by LLAMA 2 as desirable. When we train the policy on a combination of LLAMA 2 data and demonstrations used to learn the BC policy, we obtain worse task completion percentage than using GPT-4 data and demonstrations. Specially, performance drop from 64.1% (GPT-4) to 56.0% (LLAMA 2) on ALFWorld, from 47.1% % 47.0% on Science World, and from 59.7% to 56.5% on Touchdown.
Table T0 shows some examples of steps identified as productive by these models that illustrate LLAMA 2's tendency to identify spurious actions being productive. In the ALFWorld examples, for instance, LLAMA 2 has a strong tendency to identify opening and closing cabinets and drawers as productive, even though they have nothing to do with putting a clean soap bar on the counter top (the first instruction) or putting clean spatula on the side table (the second instruction). Similarly in Science LLAMA 2 identifies unnecessary actions such as going outside (example 1) and going to the bedroom (example 2) as productive, even when the instruction explicitly details that the aluminum foil is found in the kitchen (example 1) and that the unknown substance found in the workshop (example 2). Finally, LLAMA 2 also tends to identify spurious actions as productive in Touchdown. the last example, the instruction asks to take a left after the first intersection, but LLAMA 2 rewards the left turn during the first turn, before the agent even arrives at the first intersection. GPT-4, on the other hand, correctly identifies Step when the agent finally encounters the first intersection, as productive.
18

---

We show in Section 5.4 that small and cost-effective Language Feedback Models are able to replicate LLM feedback through training. Our comparison between GPT-4 and LLAMA 2 show that less powerful model such as LLAAM 2 are unable to provide high-quality feedback. The summary from this experiment are then that 1) powerful L.LMs are necessary provide good feedback, but expensive to run during on-line policy improvement 3) consequently, learning small LFMs is an effective solution to achieve high feedback performance while reducing inference cost during policy improvement.
19

---

H GPU Usage
We train feedback models and policies using 80GB A GPUs. To produce rollouts at in parallel, we use a cluster of 200 32GB V100 GPUs. For all environments, feedback model training takes under 24 hours using one 1100 GPU while inference can be performed locally using 32GB GPU under hours. Policy training requires day for ALFWorld, 2 days for ScienceWorld, and 3 days for Touchdown. For all environments, policy rollout over the entire evaluation environments can be performed over the cluster of 200 32GB VI00 GPUs in under 6 hours.
20

---

I Correlation between LLM/LEM feedback and true state values
To investigate whether LLM feedback is correlated with state values, we obtained partial rollouts for environments in ALFWorld and asked GPT4 to score from 1-5 whether the partial rollouts are on the right track solving the task. We then ran a planner (with full observability) to complete these partial rollouts in order to obtain ground truth optimal values. With no training, the LLM's predicted score has strong correlation (0.61 Pearson) with the optimal values. This means that GPT as feedback model has strong zero-shot generalization on environments it was not trained on.
In addition, regressed FLAN-TS model to estimate state values from language feedback using states from a random policy, then evaluated its predictions against true state values mixed policies where an expert is select p fraction the time (and a random policy is used other times). When we uniformly sample states from the random policy, for O, 0.2, 0.4, 0.6, 0.8, the regression model achieves Pearson correlations of 0.3 0.4. When we subsample training states evenly across values, we achieve Pearson 0.65 - 0.75. This shows that state coverage, for which state value coverage is proxy, indeed important. Conversely, when we use mixed policies for training as well, we obtain 0.78 0.85 Pearson. This shows that optimality is indeed important.
21

---

Table 10: Example steps identified as productive by GPT-4 LLAMA 2, and both. Touchdown steps are truncated for brevity.
ALFWorld
Task:
clean some soapbar and put it in countertop.
Step 3 Your action: take soapbottle from toilet Result: You pick up the soapbottle from the toilet 1.
Step 6 Your action:
go to countertop Result: You arrive at loc 11. On the countertop 1, you see candle 2...
Step 11 Your action: open cabinet Result: You open the cabinet 1. The cahinet is open. it, you...
Step 17 Your action: open cabinet 1 Result: You open the cabinet 1. The cabinet 1 is open. In it, you see...
Step 19 Your action:
go to cabinet Result: You arrive at loc 12. The cabinet 3 is closed. Task: put clean spatula in sidetable.
Step 3 Your action:
go to sidetable Result: You arrive at loc 22. On the sidetable you see nothing.
Step 10 Your action: go to drawer 3 Result: You arrive at loc 19. The drawer 3 is closed.
Step 11 Your action:
open drawer 3 Result: You open the drawer 3. The drawer 3 is open. In it, you... Step 14 Your action: open drawer 4 Result: You open the drawer 4. The drawer 4 is open. In it, you...
SeienceWorld
Task: Your task is to determine if aluminum foil is electrically conductive. The aluminum foil is located around the kitchen. First, focus the aluminum foil. If it is electrically conductive, place it in the yellow box. If is electrically nonconductive, place in the purple box. Do not focus on ohjects not central to completing the task!
Step 21 Your action: open door to greenhouse Result: The door is now open...
Step 23 Your action: open door outside Result: The door is now open...
Step 25 Your action: open door to foundry Resulc: The door is now open...
Step 29 Your action: go to outside Result: You move to the outside...
Step 31 Your action:
go to kitchen Result: You move to the kitchen...
Step 32 Your action: pick up aluminum foil Result: You move the aluminum foil to the inventory...
Step 33 Your action: focus aluminum foil Result: You focus on the aluminum foil...
Step 37 Your action: go to workshop Result: You move to the workshop...
Step 38 Your action: move aluminum foil to yellow box Result: You move the aluminum foil to the yellow... Task: Your task is to determine if unknown substance N is electrically conductive. The unknown substance N is located around the workshop. First, focus on the unknown substance N. If it is electrically conductive. place it the blue box. If is electrically nonconductive, place in the orange box. Do not focus on objects not central to completing the task!
Step 1 Your action: open door to art studio Result:
The door is now open...
Step 4 Your action: go to hallway Result: You move to the hallway...
Step 6 Your action: go bedroom Result: You move to the bedroom...
Step 9 Your action: open door to hallway Result: The door is already open...
Step 11 Your action: open door to workshop Result: The door is now open...
Step 12 Your action: go to workshop Result: You move to the workshop...
Step 13 Your action: pick up unknown substance Result: You move the unknown substance N to the...
Step 15 Your action: move unknown substance to blue box Result: You move the unknown substance N to...
Touchdown
Going with the flow of traffic, proceed to the intersection and take left. There should be some blue scaffolding on your right hand side, take left here. You should now have tall solid green fence on you right hand side, go straight to the next intersection and take a left. There's 's a bunch of scaffolding on the left hand side, go straight. Pass the first intersection you come across. You will pass mail trucks parked on both sides of the road.
Task:
Step I Your action: slightly your left Result: behind you, you see: parked, white box trucks, two wider sidewalks. this narrow two lane road, the right sidewalk buildings. behind you, sightly to your left, you see: three air-conditioners. , three awning a smaller yellow taxi to your left, you see: . the theater awning a yellow cab car, the second purple awning slightly to your left, you see: a white-capped hydrant, ornate gray balconies, , the purple wayfair truck. straight ahcad, you see: a median strip, some tall, brick buildings parked, white box trucks. blue bus lane sign, the right sidewalk buildings...
Step 8 Your action: straight ahead Result: behind you. you see: a brown storefront, surface streets the right sidewalk buildings, parked black box truck, some unremarkable brick buildings. behind you, sightly to your left, you see: then a storefront white/grey van, a large, blocky, gray building. to your left, you see: a large white store sign, construction vehicle.. long gray and white 5 story building slightly your left, you see: some tall, brick buildings, fedex van white/grey van straight ahead, you see: a large red brick apartment building. an orange and white traffic object. 3rd and 4th intersections. a small blue car, the right sidewalk huildings. , the parked yellow suv taxi. slightly to your right, you see:
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: we outline a new method for policy improvement via Language Feedback Model. In the abstract and in the introduction, we claim that this method improves policy performance. Our experiments show that this method improves base policy performance on 3 distinct benchmarks. Furthermore, our experiments show that this technique outperforms alternative methods such as using LLMs as zeroshot policy, using LLMs to provide action labels, and using LLMs to provide DAGGER-Style annotations. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
23

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We outline experimental details in this paper, including steps and prompts required to produce LLM feedback to train LFM. Furthermore, we outline hyperparameters and compute resources used train feedback models and policies. Finally, we release our code and checkpoints for reproducing experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We open source data and code required to reproduce experiments. This release also contains documentation on how to run the code to reproduce data and re-train models described in this work. At this time, the release is anonymized at the link described in the introduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
//nips.cc/put wwtpcccccppccccceee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Our experiment section describe hyperparameters used to train models and collection feedback data. They also contained in the source code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide standard deviation task completion rates in our main results table in Table[Jacross 3 random seeds.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe compute usage in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwaap....paaaaaaaaaa/aaa
Answer: [Yes]
Justification: we have reviewed the NeurIPS Code of Ethics and made sure that this work does not deviate from the Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
26

---

Justification:
We discuss the broader impact of this work in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not release data models that have high risk of misuse. We discuss limitations and impact of released models in Appendix and Appendix
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We detail license and terms of usage models released for this work as well as assets used to develop this work in Appendix C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
27

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This work does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
28

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "zPwX5JSKK1", "reviewer": "Reviewer_8DkG", "summary": "The paper proposes to train a language feedback model and leverage the language feedback model to conduct policy improvement in language-based task. The authors also propose a pipeline that can apply CLIP to convert images into textual description. Experiments over ALFWorld, ScienceWorld and Touchdown validate the effectiveness of proposed algorithms.", "questions": "1. There are a few literatures that might be highly relevant to the paper, including:\n[1] Shinn, Noah, et al. \"Reflexion: Language agents with verbal reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024), which also studies how to use verbal descriptions as feedback to guide the LLM's policy.\n[2] Feng, Xidong, et al. \"Natural Language Reinforcement Learning.\" arXiv preprint arXiv:2402.07157 (2024), which is also motivated by verbalizing policy learning process.\n[3] A bunch of work covering llm-as-judge, like: Wang, Yidong, et al. \"Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization.\" arXiv preprint arXiv:2306.05087 (2023).\n\n2. Why choose GPT-4-0315 to conduct experiments? This is a relatively old GPT4 model considering it's mid 2024 now? And is there any other ablation studies covering different types of llm?\n3. By checking the prompt template shown in table 2, I have a question about: why there is no COT process between the model judge yes or no? Is it done on purpose or is there any explanation for that since in most setting COT can enhance GPT-4's performance -- and I believe this yes/no is very important since it directly influence the performance of policy improvement.\n4. The comparison between LFMD and LFD or other baselines seems a bit unfair? Since you still require to run rollouts on test set while the other results are zero-shot.", "limitations": "See Weakness and Questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, The paper is overall clear and easy to follow.\n2. The idea of training a separate language model to serve as language critic function for policy learning is overall novel in decision-making task.\n3. The experiments cover a large range of tasks, including one visual task -- which is great to demonstrate the generality of the proposed algorithms.", "weaknesses": "1. The improvement is mainly from distilling language feedback from stronger models (such as GPT-4), which somehow limits the technical contribution of proposed algorithm. Is it possible to derive language feedback from exactly the same model? (FLAN-770M might be impossible, but what about larger one like llama-3-8b?)\n2. The experiments are not comprehensive enough. For instance, the table 3 presents ALF-world's SOTA from results in paper from 2021 -- which I believe is quite out-dated and there are plenty of work that improve ALF's performance a lot. Also, the authors miss several important baselines, 1. RL -- since the policy evaluation + policy improvement is the basic foundation for RL. It is essential to compare the proposed algorithms with RL. 2. a bunch of work starting after reflection, that also uses verbal feedback to improve the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the strengths of our paper, including its clarity, novelty, and demonstration of generality across a range of tasks.\n\n## W1: improvement from distilling language feedback from stronger models\nOur primary contribution is a novel framework that combines language feedback with policy improvement. While it is true that using a single model for both policy learning and language criticism is possible, our results show that a weaker feedback model results in insignificant policy improvement (we show results using Llama2 70B in the Appendix, which is better than Llama3 8b on most benchmarks: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md#benchmark).\n\nWhat our findings suggest are that\n\n1. a very large model (let’s say an LLM) is difficult to use as a policy because it is expensive to train and slow to inference.\n2. a small model can be trained to provide a reasonable tractable policy, however it is not capable of providing high quality feedback (without training on such feedback).\n\nOur framework uses existing large LLMs to provide high quality feedback, without further training, to improve small tractable policies for specific environments we care about.\n\n## W2: ALFWorld baselines\nWe also appreciate the feedback on the comprehensiveness of our experiments. Although Table 3 presents a paper from 2021, it is the best-performing imitation learning method without external knowledge on ALFWorld according to the ALFWorld authors. As mentioned in W1, we consider the setting where the LLM is not available during test time. Consequently we did not use reflective techniques nor CoT techniques that tend to show strong benefits with very large models (50B+, Li et al https://arxiv.org/abs/2306.14050). Are there specific works the reviewer would like comparisons to?\n\n## W2: reinforcement learning\nWe reference prior results that use RL on ALFWorld and Touchdown (https://arxiv.org/abs/2110.10661). Crucially, we show that in these challenging environments, RL underperforms LFM (and the baseline model) after training for 10 million steps. In contrast, ALFWorld episodes typically have <30 steps. Training on the demonstration dataset amounts to 30 * 3.5k ~ 100k steps. LFM improvement using one rollout per training environment, as is the case in our experiments, results in another 100k steps. For Touchdown, episodes are typically <200 steps. Demonstration steps and LFM improvement steps are consequently 1.5 million steps each. For both these cases, imitation learning and LFM improvement require substantially fewer steps than RL and achieve substantially higher task success rate (e.g. 64 vs 23 ALFWorld, 60 vs 15 Touchdown).\n\n## W2: reflection\nWe are very interested in adapting LFM to RL and with reflection, as the reviewer suggested. In this preliminary work, we wanted to scope our problem to investigate the most fundamental setting where the large model (the LLM LFM) is not used during policy improvement. Improving policies with LLMs in the loop (e.g. to provide rewards, to provide reflection) results in hundreds of steps per rollout, and is very expensive for environments we consider. However, we would like to investigate this direction in future work, for instance by learning a small language reflection model in parallel with the policy.\n\n## Q1: references\nWe appreciate the reviewer’s suggestions of several relevant works, including [1], [2], and [3]. We agree that these papers share similar motivations and approaches with ours, and we will include them in our related work section. In the current manuscript, we specifically discuss the ReAct, Reflexion, and InnerMonologue line of work in the last paragraph of our related works section.\n\n## Q2: GPT-4-0315\nUnfortunately due to internal infrastructure policy it was the only GPT-4 model widely available to us at the time of our experimentation. We also experiment with Llama2 70B, the results for which are in our appendix. In ongoing and future work, we are investigating more recent LLMs, including VLMs, as policy critics.\n\n## Q3: Chain of Thought\nIn this work, we make the assumption that the LLM is not available during test time, only training time. Current evidence suggests that CoT is only significantly helpful when the base model is sufficiently large (50B+, Li et al https://arxiv.org/abs/2306.14050). As mentioned in response to W1, a very large model (let’s say an LLM) is difficult to use as a policy because it is expensive to train and slow to inference. Consequently, in this preliminary work, we do not investigate CoT. In future work, we would like to study how to distill large base models into small models such that they are capable of providing high quality CoT. Our result on LFM-D is a first step towards this direction, where we ask the model to provide evidence for its critique, akin to asking a question answering model to provide reasoning steps for its answer.\n\n## Q4: LFMD and LFD and baseline comparison\nWe want to make an important clarification that LFM and LFM-D do not perform rollouts on the test set - they perform rollouts only on the training set for policy improvement. The only method that performs improvement via test-set adaptation is LFM-A, which achieves significant improvement via adaptation compared to LFM. The comparison to LFM-D (D for descriptive) serves only to show that we can increase the interpretability of the feedback model without suffering performance degradation.\n\n## Summary\nWe sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJ7iQqnvhB", "reviewer": "Reviewer_u2Y5", "summary": "The papers present a method to essentially filter which actions should be used to learn a policy via imitation learning. The method follows the online imitation learning setting and replaces the expert policy with a language feedback model (LFM) distilled from a LLM. The LFM evaluates which transitions from the policy's rollouts should be used to train the policy. A LFM is used instead of LLM to reduce the computational complexity of the task, and both consumes and produces text. The method is evaluated on several benchmarks. The method is compared against several ablations over the design decisions.", "questions": "- It is unclear how the method is marking states as desirable in section \"learning from language feedback\" is different from that in \"naively learning from LLM feedback\". Is there a connection missing that links how the LFM is \"efficiently\" learned and how the policy learns? Instead of querying the LLM at each step, the LLM is queried of a trajectory of steps and responds by saying which steps were desirable?\n- Is there also a reward function? \n- What is an iteration/round in this method? i.e. \"In round k, we rollout the base policy....\" (Section \"Learning from language feedback\")\n- You state, \"Unlike these works ... we consider settings where training and evaluation goals are different.\" It is not clear to me how the training and evaluation goals differ in your set up.\n- How do you decide when to summarize what has happened with a \"Before\" in the LLM prompt? e.g. Table 2 - LFM prompt.\n- What does it mean that you \"...limit the amount of LLM usage to 100k GPT-2 tokens\"? Did you constrain the number of input tokens? The number of output tokens? How did you go about the constraints?\n- How did you select 20 as the number of timesteps over which LLM feedback was collected?\n- You say you subselect feedback data to have an even split of productive and non-productive action. How much data do you actually end up with? What were the originally ratios? How diverse are the samples?\n- For the results section on generalization to new environments, what makes a new environment new? Table 4 holds results for ALFWord, ScienceWorld, and Touchdown, which are the environments you report training on.", "limitations": "There are no limitations in the main body.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes an interesting idea to filter out the data that should be used for imitation learning, and then proposes to do this with a language model. \n- The method demonstrates clear improvements to methods evaluated against.", "weaknesses": "- The authors claim that their LFM method is better than using a LLM as an expert policy, because it scan provide human interpretable feedback. However, the authors do not provide any results to suggest that the LFM can produce outputs humans can do something with.\n- The only comparisons are to ablations of the proposes method. At a minimum, some of the mentioned related work (like MOTIF (Klissarov et al [22])) should probably be a baseline.\n- The LMFA 1 rnds and LMFA 2 rnds in Table 3 do not seem to be discussed in the main body.\n- There are some gaps writing that have left me with a lot of questions/uncertainties (see below).\n\n- Small things: \n     - you have places in the PDF with weird formatting, e.g. lines 156 - 157.\n     - The reference to Figure 1(b) in Section 2 line 72 should probably be 1(c)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the novelty of our method, as well as clear demonstrations of improvements using our method.\n\n## W1: interpretable feedback\nTo clarify, we have 2 claims. First, policy improvement from LLMs feedback is better than using LLM as policy. There are two experiments that support this. a) we compare to using GPT4 (0315) on-policy zeroshot. Table 3 shows that our method achieves significant improvement. b) we use GPT4 to label what the agent should do, then train a policy using demos as well as GPT-4 labeled actions. This method of using LLM as expert policy to label actions (ActPred), underperforms using LLMs to train feedback models (LFM).\n\nThe interpretable feedback experiments show that instead of producing LFMs that only identify good behaviour, we can train descriptive LFM-Ds that also state why a behaviour is good. We show that LFM-Ds perform similarly to LFMs, achieving both interpretability and high levels of policy improvement.\n\n## W2: related work\nThank you, we will add this. LFM differs from MOTIF in that the former improves policies via imitation learning while the latter derives a reward model for RL. While MOTIF shows results on a single game NetHack, we show results across three different settings in household (ALFWorld), scientific experiments (ScienceWorld), and real-scene navigation (Touchdown). In the latter two settings, RL is  substantially worse than LFMs (https://arxiv.org/abs/2110.10661). In the future, we will investigate RL using language feedback for multimodal grounded settings.\n\n## W3: adaptation rounds\nThank you for the comment. We describe one-round adaptation in section 5.2 on line 249. We will describe two-round adaptation in the manuscript. Round-wise adaptation is described on ​​line 148: we use the policy from the previous round to perform rollouts, then filter for good behaviour using the trained LFM, then imitate them to improve the policy.\n\n## W4: small things\nThank you, we will make the corresponding corrections to the manuscript.\n\n## Limitations\nWe discuss limitations and broader impacts of this method in Appendix Section A and B, we will move them before references. \n\n## Q1: efficient vs. naive feedback learning\nLet’s suppose we have 20 steps. The \"naive\" method queries the LLM each step, resulting in 20 queries. The kth query has steps 1…k-1 in the context and asks the LLM whether the kth action is productive. The \"efficient\" method batches feedback requests into 1 query, which asks the LLM to list which steps were productive (Figure 2). The efficient method is much cheaper, requiring 20x fewer API calls to the LLM (line 130). Once feedback is collected, the subsequent training would be identical.\n\n## Q2: reward function\nWe do not train a reward function - we only use imitation learning for policy improvement, not reinforcement learning.\n\n## Q3: iteration/rounds\nAfter training a LFM, we can improve the policy in rounds. In round 1, we start with the base policy P1 trained on initial demos. We roll out P1, and then identify its good behaviour using our LFM. We then add the good behaviours into the demo set and train the policy P2. We then repeat this in round 2, where we identify good behaviours using P2, then use those to train P3, and so on.\n\n## Q4: training vs evaluation goals\nLet’s consider NetHack. NetHack has 1 implicit goal for the agent, which is to obtain the highest score it can. In contrast, we consider settings w/ different instructions between train and test, which require generalization to new scenes (like NetHack) and new instructions (unlike NetHack). For ALFWorld, the agent may be trained to “find and wash glasses” and “put apples in the fridge”, but during test it may be asked to “find apples, wash them, then put them on the dining table” in new rooms. Similarly, in ScienceWorld the agent is required to follow new instructions in new spaces (e.g. the determine the boiling temperature of a new substance). In Touchdown, the agent is required to navigate between new starting and end points in new neighborhoods.\n\n## Q5: summarize with “Before”\nIn Table 2, the “Before” is not a summary, it is the observation of the step right before the window starts. In this case step 20. We will clarify this in the manuscript.\n\n## Q6: Token limitation for LLM usage\nAs described in Table 3, we limit LLM interactions to 100k output tokens. We do not limit input tokens as they are much cheaper than output tokens. We collect feedback for as many windows as possible until we run out of 100k output tokens, then we use this feedback to train the LFM. For ActPred, we label actions for as many steps as possible, until we run out of 100k output tokens, then we use this annotated set along with demos to train the policy. We will clarify this in the manuscript. Thank you!\n\n## Q7: 20 steps\nEmpirically, 20 steps fits over 90% of observation windows into the model’s context length (8k for GPT4 0315). We will investigate using new LLMs w/ longer context (128k) to train LFMs.\n\n## Q8: Data ratios and sample diversity\nThis ratio differs between settings (60% not productive for ALFWorld, 70% ScienceWorld, 90% Touchdown). All settings use 10k 20-step windows as feedback data.\n\nWe have diversity across tasks, instructions, and envs. LFM data collection is biased by how good the base policy is. If the base policy is bad at task A and good at B, then we tend to identify more “good behaviour” from trajectories from B. In this work, we do not do sophisticated filtering to rebalance on a task level, but we are interested in exploring this in the future.\n\n## Q9: new environments\nPlease see Q4.\n\n## Summary\nWe sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5IBbu4Emk", "reviewer": "Reviewer_hZ53", "summary": "This paper provides a policy improvement method with a Language Feedback Model (LFM) for decision making tasks. \n\nThe proposed method mainly consists of two stages: (1) training a Language Feedback Model (LFM) and (2) improving a policy model with the trained LFM. In the first stage, to train a LFM, the initial policy is used to generate rollouts by interacting with the environment. Then, a LLM is used to generate text feedback on each action in the rollouts. Then, a LFM is trained on this feedback dataset. In the second stage, the trained LFM is used to generate feedback on each action generated by the initial policy. Then, actions that are predicted as desirable from the LFM are collected as a dataset. Finally, the policy is trained on the dataset that contains desirable actions.\n\nThis paper evaluates the proposed method on three decision making benchmarks: Touchdown, ScienceWorld, and ALFWorld. This paper empirically demonstrates that the proposed method provides better scores than baselines such as BC or ActPred.", "questions": "Q1. Regarding the weakness W1, how many rollouts are needed to train a LFM? Also, how many revised rollouts are need to train an initial policy? \n\nQ2. Regarding the weakness W2, if we only select desirable actions from a rollout, does the revised trajectory correctly reflect or follow the environment dynamics or the transition function?", "limitations": "The authors provide limitations of their work in Appendix A (i.e., Limitations).", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1. The idea of learning a language feedback model (LFM) to provide text feedback on each action sampled from the initial policy is interesting.\n\nS2. This paper provides experimental results on three representative benchmarks: Touchdown, SecienceWorld, and ALFWorld.", "weaknesses": "W1. In the introduction section, this paper mentions that sample-efficiency and generalizability are important in instruction-following agents. I am not sure that the policy improvement with LFM is sample-efficient and generalizable. The proposed method trains the initial policy on a rollout dataset where desirable actions are collected by the LFM. Training on many rollouts does not seem sample-efficient. Also, the policy trained on a specific environment may lose generalizability.\n\nW2. In Section 3.2 (i.e., Learning a feedback model), it is rather unclear how to collect desirable behavior D_k. \n\nW3. It would be better to provide ablation study that shows effectiveness of the proposed method. For example, the authors may compare the policy improvement with an expert LLM to the policy improvement with the LFM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the novelty of our method, as well as clear demonstrations of improvements using our method across three representative benchmarks.\n\n## W1: sample-efficiency and generalizability\nWe define sample-efficiency to mean that the agent requires few human labeled demonstrations in order to achieve high performance. In this work, we assume access to a small set of human labeled demonstrations to train a base policy. With no further human labeled demonstrations, we can synthesize demonstrations using the LFM automatically, with which we can improve policy performance. Consequently, we describe the proposed method as sample-efficient because it improves policy performance with no additional human labeled demonstrations.\n\nWe define generalizability to mean that the agent generalizes to new environments not seen during training. Let’s consider the game NetHack. In NetHack, there is one implicit goal for the agent, which is to obtain the highest score it can, given the rules of NetHack. In other words, every NetHack agent is given the implicit instruction “get the highest score you can in NetHack, subject to the rules of the game”. In contrast, all three environments we investigate have different instructions between training and evaluation. These environments require generalization to new scenes (like NetHack) as well as to new instructions (unlike NetHack). For ALFWorld, the agent may be trained to “find and wash glasses” and “put apples in the fridge”, but during test evaluation it may be asked to “find apples, wash them, then put them on the dining table” in new rooms. Similarly, in ScienceWorld the agent is required to follow new compositional instructions it has not seen before during training, in new spaces (e.g. the determine the boiling temperature of a new substance). In Touchdown, the agent is required to navigate between new starting and end points in new neighborhoods. Consequently, we describe the proposed method as one that generalizes because it improves performance on new environments with new scenes and new instructions not seen during training.\n\nWe will clarify these two points by precisely defining sample-efficiency and generalization in the manuscript. We thank the reviewer for identifying this point.\n\n## W2: desirable behaviour\nWe want to emphasize that the proposed framework consists of two components. The first component involves learning a language feedback model (LFM). The second component involves using this LFM to improve policy. The D_k the reviewer refers to has to do with the second (improving policy), not the first (learning a feedback model). We will summarize both components here.\n\nWe start from a base policy trained from a small set of human demonstrations. In the first component of learning a LFM, we roll out the base policy to collect trajectories. These trajectories are given to an LLM to create a dataset (this is not D_k) to train the LFM. This dataset consists of tuples of (agent behaviour, LLM feedback). We train a smaller LFM to imitate LLM feedback. That is, given the agent behaviour, the LFM critiques in natural language whether this behaviour is productive. We then freeze this LFM.\n\nIn the second component, we use the trained LFM to improve the base policy. First, we roll out the base policy to collect trajectories. We then use the LFM to predict whether each step in the trajectory is productive. We take the subset of productive behaviour (eg. context and agent action) and add it to our collection of productive demonstrations. Then we update the policy by training on this collection of demonstrations. We do this in multiple rounds, in each round K, we roll out the policy, identify good behaviour, and add it to the collection of demonstrations. We refer to the collection of demonstrations during round k as D_k.\n\n## W3: compare expert LLM to policy improvement with LFM\nWe do perform two such experiments in our ablation study. Experiment A compares to using GPT4 (0315) on-policy zeroshot. Table 3 shows that our method (and other methods that train a smaller policy model) achieves significant improvements over zero-shot GPT4. Experiment B uses GPT4 to label what the agent should do, as opposed to critique how the agent performed. We then train a policy using demonstrations as well as GPT-4 labeled actions. This method of using LLM as expert policy to label actions (ActPred), underperforms using LLMs to train feedback models (LFM).\n\n## Q1: number of rollouts\nDataset statistics are shown in Appendix C Table 6. To train the initial policy, we use 3.5k demos for ALFWorld, 3.6k for ScienceWorld, and 6.5k for Touchdown. These correspond to 1 demo for each unique environment. When rolling out, we perform one rollout per unique environment. Consequently we have an identical number of rollouts as initial demos. That is, we perform 3.5k rollouts for ALFWorld, 3.6k for ScienceWorld, and 6.5k for Touchdown.\n\n## Q2: desirable actions vs. transition function\nThe revised trajectories correctly reflect env dynamics because they are steps that took place during the rollout, according to env dynamics. This observation also explains underperformance of ActPred. In ActPred, an LLM is used to label what actions the agent should take. However, the LLM can hallucinate actions, resulting in demonstrations that do not correctly reflect env dynamics. In contrast, our proposed method only identifies real actions that took place as productive behaviour. Consequently all demos collected using LFMs correctly reflect environment dynamics. This also explains why LFM improvement is larger than ActPred.\n\n## Summary\nWe sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHTVLJHOvY", "reviewer": "Reviewer_B45L", "summary": "The authors introduce Language Feedback Models (LFM), a method filtering out desirable transitions (i.e. transitions considered as helping to solve a task) collected by an agent in an environment to then improve this agent by doing Imitation Learning on these transitions. The LFM method has three phases: 1) First, a fairly large number of transitions are collected by the initial policy, and GPT4 is used to determine which transitions are desirable. Then, 2) given this dataset of desirable transitions, a smaller LLM is trained to reproduce GPT4 in saying these transitions are desirable. Finally, 3) the policy is used to collect new transitions, which are then filtered out by the smaller LLM, taking only the ones considered as desirable, and the policy maximizes the likelihood of the actions chosen in these transitions.\n\nExperiments in two TextWorlds and one visual navigation environment are performed, showing that LFM successfully improves the initial policies (learned with Behavioral Cloning from expert demonstrations) and leads to better results than using GPT4 to directly propose the best action to perform and imitate it. Additionally, experiments show that the learned feedback model can be used to improve the policy in unseen test tasks which further increases the policy's performance even though the feedback model has not been trained on the test tasks. The authors also show that one can go further and not only train the smaller LLM to select desirable transitions but also to explain why these transitions are desirable (also by imitating GPT4), leading to better explainability.", "questions": "- In Table 3, BC results are different from \"Prev SOTA\" on ALFWorld and Touchdown. Why is this the case, and how did the authors collect expert demonstrations that led to a policy better than SOTA?  \n- When describing the ACTPRED baseline l237, it is mentioned \"First, we execute k steps of the base policy, then query the LLM for the next action\". What are these k first steps and why are they necessary?\n- The authors computed the perplexity averaged over tokens of each possible action with the policy (that is an LLM) and selected the action with the minimum perplexity. Why did the authors not use the (log) probability of the sequence of tokens as in SayCan (Ahn et. al, 2022), GLAM (Carta et. al 2023) or TWOSOME (Tan et. al 2024)?\n- Also, computing the perplexity (or probability) of each action to follow the prompt can be very extensive when the action space grows, given that the policy is still a fairly large model (770M). Did the authors use anything to make this fast enough, especially in ScienceWorld where the set of possible actions can be very large?\n- In LFM, an initial phase uses the policy to collect trajectories that are then given to GPT4 in order to obtain the dataset to finetune the feedback model. In the experiments, 10k trajectories of 20 steps have been collected if I understood well. Were these trajectories also appended to the dataset used to improve the policy later on?\n- The authors showed an example of an explanation provided by the feedback model (based on Flan-T5 770M) in Table 5. Given the limited abilities in text generation of Flan-T5 770M, did the authors perform a deeper study of the explanations provided by the model, especially on the unseen test tasks?", "limitations": "The authors discuss the limitations of the introduced method in the appendices. They identify that, while LFM improves compute efficiency by distilling feedback abilities from GPT4 into a smaller LLM, Flan-T5 770M is still a fairly large model to be called at every step, both for the policy and the feedback. \n\nAlso, Appendix G shows that using Llama 2 70B instead of GPT4 can significantly impact the results. This is not a strong limitation, in my opinion, given that GPT4 is only used in the initial phase to train the smaller LLM.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method relies on distilling the feedback ability from GPT4 to a much smaller LLM (Flan-T5 770M) to reduce the cost of asking for feedback at every step. The authors show that they successfully transferred this ability and that the obtained feedback model generalizes to unseen tasks. The experiments also show that their method is not only less compute-effective but also much more efficient than directly imitating GPT4 used as the policy.\n\nThe authors also show the method can be trained to provide explanations of the feedback given, leading to better explainability of the method.", "weaknesses": "One of the main weaknesses I see is the lack of baselines to properly assess the efficiency of the method. The LFM method acts like a rejection sampling algorithm where only transitions considered desirable by the feedback model are kept to fine-tune the policy. Similarly, If all environments provide a reward if would have been interesting to have a baseline relying on this information for the rejection sampling. If the reward is sparse, one could wait for the end of the episode and only keep desirable trajectories (e.g. whose episodic reward is above some threshold or if the goal has been reached). \n\nThe authors also say \"In this work, we consider long-horizon settings with only sparse and delayed task-completion rewards. Consequently, we focus on imitation learning from demonstrations as opposed to reinforcement learning from rewards.\" It could also be interesting to see how RL performs with the same amount of collected data.\n\nFinally, it seems the authors use a dataset of expert demonstrations to first train the initial policy with BC before applying their method and the compared baselines. However, they do not provide any insights on how important this preliminary phase is. Indeed, on key advantage of LFM is that it relies on BC while not requiring an expert policy. However, results from Table 3 show that LFM mostly seems to substantially improve the initial policy learned with BC from expert demonstrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our paper and provide us with valuable insights. We appreciate the acknowledgement of the generalizability and efficiency of our method, as well as improvements to model explainability.\n\n## W1: Rejection sampling\nWe thank the reviewer for drawing connections between our method and rejection sampling (RS). We discuss 2 key differences between the two techniques.\n\nFirst, LFM removes 50-80% of non-productive steps whereas RS includes entire trajectories (20-100 steps for our envs). That said, we experimented with training on entire rollouts w/ positive reward. This did not significantly improve over the base policy.\n\nSecond, LFM filtering can be done during testing, where no reward is present. In this case, reward-based RS cannot be directly applied to perform adaptation during testing. That said, one could train a model to predict rejection, using data from the training environments, then use said model on test environments for adaptation. This is precisely what our proposed method does.\n\n## W2: reinforcement learning\nWe reference prior RL work on ALFWorld and Touchdown (https://arxiv.org/abs/2110.10661), which shows RL underperforms LFM after training for 10 million steps. ALFWorld episodes typically have <30 steps. Training on demos amounts to 30 * 3.5k ~ 100k steps. We train LFM using 1 rollout per training env for another 100k steps. Touchdown episodes are typically <200 steps. Demos and LFM steps are then 1.5 million steps each. For both envs, LFM requires fewer steps than RL and achieve higher task success rate (e.g. 64 vs 23 ALFWorld, 60 vs 15 Touchdown).\n\n## W3: Importance of initial demonstrations\nWe experimented w/ a random base policy, but found collecting trajectories that demonstrate productive behaviour difficult, as random policies are mostly not productive. Consequently, most data given to the LLM for feedback annotation do not exhibit productive behaviour, therefore increasing the cost of LLM usage. We can alleviate this using reward-based RS, as the reviewer suggested in W1. That is, we sample many trajectories, then give those w/ non-trivial reward to the LLM for feedback annotation. We will explore this in future work.\n\n## Q1: BC vs Prev SOTA\nWe use the same demos as prior SOTA, provided by the env designers. However, we use stronger, newer base models. For instance, ALFWorld prior SOTA fine-tuned a GPT2 model. Touchdown prior SOTA trained a CNN/RNN network. In contrast, we fine-tuned a FLAN T5 Large model, which performs well in language grounding tasks (https://arxiv.org/abs/2210.11416). We perform similarly to prior SOTA for ScienceWorld using the same model but without ScienceWorld-specific prompts (e.g. tracking object types, rooms visited).\n\n## Q2: ActPred k steps\nThe significance of this k-step is due to the token limitation. We want diverse trajectories over many envs. However, each trajectory may consist of >100 steps (ScienceWorld and Touchdown are often 50-200 steps). Predicting actions from the start using LLMs is biased towards early-trajectory steps - we will exceed the token limitation before encountering late-trajectory steps. Completely rolling out one env at a time will exceed token limitation before processing many of the envs. To balance between coverage of envs and trajectory depth, we first roll out k steps, then predict action using LLM. k is sampled from the max trajectory length of demos.\n\nNote that this coverage problem is less significant for LFMs. Because LFM data is queried in windows of 20-steps, we can maintain high coverage of both environments and of trajectory depth. We also experimented with asking GPT4 to retroactively relabel actions in 20-step windows, however the labels were significantly worse than labeling actions one step at a time.\n\nWe will add explanations of the k-step significance to the manuscript.\n\n## Q3: average perplexity vs logprob of sequence\nWe did explore with log probability, which performed slightly worse than averaged perplexity.\n\n## Q4: Scoring large action spaces\nWe emphasize that training strictly maximizes log probability of tokens of the correct action, so its complexity is independent of the size of the action space. Inference requires scoring large action sets, however is cheap due to lack of gradient-tracking. The only optimization we did was compute scores in chunks, which are concatenated to select the optimal action.\n\n## Q5: initial phase trajectories\nWe clarify that it is 10k windows of 20 steps sampled from trajectories. These windows contain many negative examples, consequently we do not add them to the dataset for policy improvement. The data used for policy improvement are only steps from base policy rollouts that identified by the trained LFM as productive.\n\n## Q6: analysis of explanations\nThe examples in Table 5 are actually from the held-out test set tasks. Unfortunately, we did not perform quantitative analyses. Our qualitative findings are that a 770M model, trained on LLM feedback for a specific domain, is able to provide accurate feedback and explanations for said domain. One limitation was that LFMs are sometimes not consistent in evaluating productiveness of ambiguous actions such as exploration. For instance, if the instruction is to find cups and wash them, then even a good policy will spend time searching the room for cups. To some observers, this search procedure is productive, to others, it is not productive. In our experience, LLMs and LFMs are sometimes not consistent in assigning productiveness to such actions. In future work, we would like to analyze the limits of what types of feedback LLMs (and VLMs, for that matter) are able to provide in multi-modal grounded environments.\n\n## Summary\nWe sincerely thank the reviewer for taking their time to help us improve this work. We hope we have addressed the reviewer’s concerns and questions. If so, would the reviewer please consider increasing their score to show support for our work?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers' concerns focus on evaluation, particularly limited comparison with baselines and ablations, and on potential limitations of the approach. These type of concerns are common in this emerging field and it's good to see the community push for such improvements. At the same time, it does speak to the innovative nature of work in this field, including this paper, and the authors do satisfactorily address the concerns. What remains is a subjective assessment of significance, which I find sufficient, thus leaning to accept.
Decision: Accept (poster)