Continual Learning for Instruction Following from Realtime Feedback
Authors: Alane Suhr
Yoav Artzi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: continual learning
interaction
instruction following
user feedback
natural language processing
language grounding
situated interaction
collaboration
TL;DR: We design and deploy a learning approach for improving instruction-following agents from feedback provided in real time during collaborative interactions with human users.

Abstract: We propose and deploy an approach to  continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting  user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.

Continual Learning for Instruction Following from Realtime Feedback
Alane Suhr
Yoav Artzi Corelll University yoavêcs. cornee1.ccccellccccoco
University of California, Berkeley* suhrbberkeley. edu
Abstract
We propose and deploy an approach to continually train an instruction-follloingg agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback they observe the agent following their instructions. We design a contextual handit learning approach, converting user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.
Introduction
The dynamics that arise situated language interactions between human users and automated agents expose plethora of language learning signals. A prime example of such signal is explicit feedback: when users convey their intent via natural language instructions for an agent to follow, they are well positioned to provide feedback, for example, through stream of binary signals as the agent acts.
Learning from this type of signal has significant potential. It shifts the burden of learning from annotated data to learning through interaction with users, which not only reduces data costs, but also enables continua improvement through interaction with users. This signal also fundamentally differs from gold gold-standard annotated data: it directly targets the current agent behavior, rather than reflecting optimal human behavior that may be of less relevance to the agent's 's current policy. This approach stands in contrast with most methods for learning to follow instructions, where the use of demonstration data entails high data costs, and the clear separation between training and deployment passes over opportunities for learning through interaction with users [e.g., 3 38
In this paper, study the problem of continually improving automated instruction-follont agen Fl by learning from user feedback in human-agent interactions. We situate the interaction in collaborative environment, which provides conditions necessary for this type of learning: users are incentivized to improve the agent's 's capabilities by providing feedback, and are continuously present as the agent executes their instructions. We use the CEREALBAR scenario [41], where two participants collaborate towards a common goal in shared world, using natural language instructions coordinate their actions. During interaction, human users write instructions for an agent, and provide feedback to the agent as it executes them via a stream of binary signals. Figure illustrates
"Work done while Cornell University.
"e use the term continual learning to refer to a learning setting where an agent continually improves its task performance [14, in our case following instructions. The term is used at times describe the domain adaptation challenge of continually learning handle new tasks. Our work does address this problem. Throughout the paper, we use agent to refer to an automated instraction-followinn system.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

User- Agent Interactions
Learning from User Feedback
Rocads P 1,2,3
Instruction- Following Policy m(: s. 0pp
Raw Feedback
Aggregated Training Data
Contextual Bandi: Training
Data
0pt1
Figure 1: Illustration of our continual learning process. The including deployment and training. In deployment, we sample and execute actions from policy parameterized by 8p. conditioned on ussr-written instruction E and agent observation o. Concurrently, users provide binary feedback they observe instruction execution. We convert this feedback to rewards, and estimate parameters 0 of next round's policy with a contextual bandit objective.
The process progresses in rounds p. each
the setup and our learning process. To the best of our knowledge, our work is the first to demonstrate continual learning from human feedback for natural language instruction following.
A key challenge is the complexity of the learning signal. Users provide feedback their discretion and under time pressure as the agent executes their instructions, resulting in an unpredictable, and often noisy, learning signal. While the concurrency of agent actions and user feedback provides a weak alignment, delays in human response and lack of clarity about which action or actions they critique make it challenging to attribute learning signal specific actions. Further complicating learning is the feedback loop created by the continually changing agent behavior and constant adaptation by human users. These factors create scenario that particularly difficult for complex methods as they often make various assumptions about the learning signal and the interaction dynamics [20] 31 We opt for simple approach with minimal assumptions, and formulate learning as contextual bandit scenario. We alternate between deployment, where the agent interacts with users that provide feedback, and training, where we compute immediate rewards from feedback signals, and weigh examples using the current policy to account for the continually changing agent behavior.
We experiment with approach through interactions with humans users, where we repeatedly deploy, train, and re-deploy the agent. We study the effects of human adaptation and process design decisions the efficacy of learning. We design our experiments to tease apart genuine agent improvements from human user adaptation, and observe dramatic improvements in agent behavior beyond what user adaptation explains, 15.4% absolute improvement in instruction execution accuracy in our longer running experiment. Our code and data is released under the MIT license 2023
2 Technical Overview
Interaction Scenario We situate user-agent interactions in CEREALBAR [41, collaborative scenario where two participants, a leader and a follower, collect sets of matching cards together in a shared, 3D environment. The two participants collaborate towards a shared goal, act in the world, and coordinate using natural language. The leader plans what the pair should do, acts according to the plan, and describes the follower's part of the plan using natural language instructions. The follower's 's role is to follow the instructions. The pair receives a point for each set of cards they successfully collect. Their goal is to maximize their score. We add to the original CEREALBAR setting the ability for the leader to provide binary feedback signals to the follower as they execute instructions. our study, the agent controls the follower and the leader is always human user. We use CEREALBAR as it is the only research environment designed to support the real-time collaborative interaction scenario we study: alternatives (e.g., Thor 23 are not designed to deploy scale for model-user interaction studies, and would require extensive modifications to support multi-agent collaborative interactions.
Task The agent's 's task is to map natural language instructions and observations to follower actions. Actions include moving FORMARD or BACKWARD, turning LEFT or RIGHT, and completing in- struction execution with STOP. The agent observation is a partial view of the world state from the
'Our code and data is available here: htttps///iithu> http:s////ggtttoo.ommm/aaa/a//// wwt../i-la//////........ cb
The original
CEREALBAR game and data 41 is licensed under MIT License 2022.

---

first-person perspective of the follower. Formally given an instruction E and an initial observation o1. our goal generate a sequence ((o), a1). .......... (Om am))) where o an agent observation, a, the action taken, and (mm = STOP.
Inference and Learning We define the follower agent as a policy (I o; 0 parameterized by 0 that maps instruction I and state observations o to distribution over actions 6 During interaction with users, for each user-written instruction I. we sample and execute actions a; ~ .(- Σ., og; 0) until we sample the instruction completion action. Concurrently, the user may provide positive and negative feedback signals f. Feedback signals are given in realtime, and are not timed to coincide with a specific action. Between any two agent steps, there could be any number of feedback signals.
We optimize parameters through rounds of continual learning (Figure Tp. At each round p. deploy the policy with parameters 0p to interact with users, and then estimate parameters 0--1- During deployment, collect agent instruction executions and user feedback. The execution of an instruction during interaction creates a trace made of two sequences (((,,, Oes ) (C,, wy) ))jm)) where
each action a; and feedback signal f, are paired with wall times wf and w/. We compute reward from the feedback construct training examples (z, a, r), where r is reward. We optimize policy parameters by solving contextual bandit problem, maximizing immediate expected reward.
Evaluation Our main metric is instruction execution accuracy for user-written instructions. Because we only have access to agent instruction executions, and not ground-trute demonstrations, we cannot compute accuracy automatically. We use crowdsoureing to acquire human judgments of follower accuracy after each round of deployment. We also evaluate trends in user feedback and ratings.
3 Continual Learning
We estimate the policy parameters from user feedback during interaction with users. The process, illustrated in Algorithm progresses in rounds. Each round p includes: (a) deploying the agent policy parameterized by o to interact with users (Section 3.1b, (b) computing rewards from user feedback to construct training data DDp (Section 3.2). and (C) optimizing the policy parameters using all data observed so far Upr=0 Dr compute 0p+1 (Section 3.3). We initialize the process with policy parameterized by 01 estimated on human demonstration data Du (Algorithm) T Line T.
3.1 Deployment Interactions
During deployment in round p. users collaborate with the agent, and delegate to it tasks by typing natural language instructions. For each instruction H, we sample a sequence of actions from the policy a; (- T, o,; 0g) (Lines 1 Executing an action in the environment results in new observation 0:+1 (Line TTp. For each action, we record the wall time w as the time when the user
starts seeing the follower starting to execute the action (i.e., the action animation starts) (Line 12)
terminate instruction execution when the stop action STOP is sampled (Line 15).
At any point in time during the agent's instruction execution, the user may provide binary (positive or negative) feedback signals, creating stream ((f,, w/ )> where w/ is the user's wall time when feedback button is pressed (Line (3}. We also allow the user to reboot the follower at any point during an instruction execution (omitted from Algorithm D] This is meant to terminate very bad executions before they damage the interaction, for example by leading the follower too far astray. Rebooting adds negative feedback signal to the end of the stream, and removes all queued instructions
The execution of an instruction by the agent and the feedback given by the user are combined together to create a trace made of two sequences ((o,, ai w and (ffs w )
'Suhr et al. [41] provide complete formal notation for CEREALBAR. We use a simplified notation to focus on the follower instruction following task only. 'Our policy takes as input a single instruction. Considering interaction history, including previous instructions. is an important direction for future work.
CEREALBAR allows the leader queue multiple instructions: the follower only the current instruction.
3

---

Algorithm Continual learning for instruction following from realtime user feedback.
Input Human demonstration data Do; number of interactions per round T; dataset construction process convert Sunvert_feedback feedback__tt.. ttr_rraarraaaeeeee reward that maps from instruction and feedback- execution trace a set reward-annotated training examples (Section 3.2): learner train train_poleey that optimizes a contextual bandit policy gradient objective (Section 3.3).
1: 01 wtratrinnn.... prredn_pooocc(Pegg
>Iiitialize policy parameters with human demonstration data.
2: for round p= 1,.11,.... 1.... do
3: D> < {
4: for T interactions do
>-Deploy the current policy for T interactions.
5: repeat
T7 01 f- observe instruction from user current world state
7:
a. F < ()
DInitialize empty action and user feedback sequences.
8:
i <- 1
repeat
>Generate actions for instruction. D-Sample action from policy.
10:
0g (- 0(;p))
0:41 ~ T(oi, a.)
&Concureentl:: execute action and observe feedback.
J. f observe feedback
12:
a +- a+ (for, walltime ())
13:
F ++ F F+ J,
&Collect user feedback given during action execution.
i+1
15:
until 0 STOP
16:
D, D, convert feedback_te to_ aeweearrdtooewwdeeeeeeo (I,0,F)
17:
until interaction terminates
18:
0pt1 t tttrainnpplice (U>
(U>=o Dp)
DTrain policy for next deployment using all data.
3.2
Dataset Construction
We use all traces collected in round p to construct training dataset Dp. Each example in D, is tuple (I, r), where T is an instruction written by the user in round is the agent observation when action was selected during the execution æ in the interaction, and r a numerical reward we compute from the feedback stream. For each instruction written during round p we may create no examples if no feedback was given, or multiple examples if computing the rewards from feedback resulted in attributing rewards to multiple actions.
Simple Reward We consider the value of each positive feedback as +1 and each negative feedback as f = -1 We compute the reward r; for an action a, as the sign of sum of feedback signals that occur between it and the next action after correcting for human response delay time 31]
T: sgn E W < w/ - d w
(1)
where Wi is the wall time of action a,, w/ is the wall time of feedback signal fy, and d is a human response delay constant (d = 0.2 seconds). If the set is empty with no feedback signals within the time window, or if r, O, we do not
reward or create example for the corresponding action.
Heuristically Propagated Reward As users are not required to follow any specific feedback schedule, the simple reward computation is likely to result in relatively sparse reward assignment, with many actions not receiving any learning signal and thus essentially thrown out as training data. Indeed, in experiments,
observe that roughly only 63.1% 1% of actions receive feedback. However, human
feedback may not be about the most recent action only, but can potentially be attributed to multiple recent actions. For example, the user may not be certain if to discourage behavior, such as turning and going in a certain direction, until the trajectory is clear after several steps.
We address this by taking inspiration from the use of eligibility traces for credit assignment 42 and create heuristics to propagate the reward to actions that otherwise receive no reward. If an action assigned no reward, we propagate to it a reward from a later action by assigning to it the reward of the next action that is assigned one, if one exists within the next eight actions. The only exception that we do not propagate reward from STOP action that receives negative reward.

---

We additionally prevent noisy feedback from being propagated by considering the scenario dynamics. We remove all training examples for actions following an action that both results a card selection (or de-selection) and receives a negative reward. If an action with positive reward results in invalid set (i.e., the selected cards do not match), we remove from the training data and do not propagate its award. These acenario-ppecific heuristics were developed by identifying common feedback errors in the data. Following propagation, we observe that 82.2% of the actions have reward assigned to them, up from 63.1, without increasing the reward error rate of about 7.0% S
3.3 Parameter Optimization
The complexity of human feedback complicates reward attribution, We adopt simple approach and maximize the immediate reward (i.c.. a discount factor of zero) [ scenario. Because of the costs of human interaction and our aim to rapidly improve through few interactions, we emphasize simplicity and use a REINFORCE-styl policy gradient objective [47]. At the end of each round, we train from scratch using all the data observed so far Up - Dpr We process the initial human demonstration data Do to the same form as the reward data we get from user interactions, so we can use it seamlessly with the data we get from the rounds. For each action and input instruction x and observation in Do, we create an example (Ir, o, a, +1); we consider all demonstration data as gold-standard, and set the action reward to +1.
This creates a contextual bandit
During round p. the gradient for an example aT,,, from dataset D' p' p is:
A = c
,')'r log (a I, 0)
(2)
π (a E. 0,
min
1,
p' >
da,
क (a
Σ, 0,
p 0==0
where c(-) computes a clipped inverse propensity scoring (IPS) coefficient of the policy gradient 17 and 0pr are parameters with which the examples in Dyr were generated. IPS is commonly used for debiasing policies during off off-policy learning. We also use IPS to avoid the problem of exploding gradients while training on negative examples (r < 0) [22). We clip the value of the IPS coefficient to maximum of I to avoid overfitting positive examples [12
4 Experimental Setup
Initialization Data Except when specified otherwise, the demonstration training dataset Dn includes 8,790 instructions from 456 Pilots studies showed estimating 01 using this data provides a good balance of numan-human data amount and initial system performance for interaction with human users.
human- human interactions from Suhr al. [41).
Model We implement the policy π as neural network with parameters 0. The network design is based on the design of Suhr et al. 41 except that we account for partial observability. Roughly, the instruction and observation o are embedded independently. We use instruction-conditione convolu- tions on the embedded observation to mix features from both inputs, and modified LINGUNET (6) generates a distribution over actions. Appendix B provides formal model details.
Deployment In each round of deployment, we collect up to a fixed number of interactions per user. We instruct users use the reboot button sparingly, and only when they anticipate the follower will make a mistake that cannot be casily corrected. We do not enforce any specific feedback patterns. Users are aware that their instructions and feedback will be used to train the agent for future rounds.
Evaluation The main measure of performance is the agent's instruction execution accuracy during interactions with human users. Because have human demonstration annotation for this data, we perform post-hoc manual evaluation for each round. We randomly sample instructions from each round. We only sample instructions the agent completed (i.e., executed the STOP action) annotate.
$ EError rate determined by manually annotating actions and feedback from 100 randomly sampled instructions. WW consider reward propagation (Section 3.2) to be part of the mapping of feedback to reward, so not considered as discounting in the learning objective.
5

---

85
100
15
Proportion of Actions with Positive and Negative Feedback
Execution Accuracy
Feedback Ratio
Positive Negative
Instrictionn
20030 suu Cunns
10
5
20
65
2
4
4
Number of Observed Instructions
-10*
Number of -10* Observed Instructions
Figure 2: Left: Mean estimated instruction execution accuracy and game scores across 11 rounds. The x-axis shows the number of instructions observed. We mark with x the accuracies and scores from the post-hoc study of user adaptation for 0 and 011 Dotted lines illustrate the share of the improvement due to user adaptation. Right: Proportion of actions receiving positive and negative feedback from users over 11 rounds, and the ratio of frequency between positive and negative feedback. The x-axis shows the number of instructions observed. In both plots, dashed lines show round boundaries.
We assume all instructions rehooted by the user are failures, and adjust down the accuracy as estimated from the annotations according to the ratio of rebooted instructions, Appendix D formally defines the evaluation metric. We show the instruction, cards that were reached by the follower during execution. We ask whether the follower correctly executed the instruction, and whether the follower made any number of errors in a pre-defined list. For performance measurements and statistics, we report the mean and confidence interval 10 except where noted. When emphasizing the relationship between the amount of training data and each measure, we plot the metric by the number of observed instructions rather than by round index.
animation of the agent's execution, and list of
Crowdsourcing We use MTTurk for both user-agent interactions and post-hoc instruction evaluation. We recruit workers from Engoiih-majority locales using qualification quiz and tutorial. (Appendix contains crowdsourcing management and compensation details. We maintain novice and expert groups of workers. Expert workers have higher compensation, and access to the post-hoc instruction accuracy tasks. To become an expert, workers must show they understand the task and annotation expectations during interactions the novice pool. We qualify 108 workers, including 65 expert workers. During deployment, we balance the number of interactions between workers by limiting each worker to about eight interactions per agent and round.
5 Results and Analysis
We conduct two experiments: an 1 1--round experiment to observe long-term learning and user behavior trends, and a shorter 5-round experiment to compare several learning design decisions.
5.1 Long-Term Experiment
We evaluate over eleven rounds of deployment and training. This experiment uses the heuristically propagated reward, with the goal of obtaining more training examples compared to the simple reward (Section 3.2). In total, we collect 3,368 games and 46,573 instructions, at cost $15,944.45 USD.
Agent Accuracy Figure 2 (left) shows instruction execution accuracy (estimated with 800 instructions per round) and number of points across the eleven rounds. Accuracy improves from 66.7 + 1.5 in the initial round to 82.1 1 + 1.1 in the final round. The continual learning process also results in increased
10 CConfidence intervals are computed using bootstrap sampling. n 10,000 samples for interaction metrics, and 1,000 samples for instruction- and action-level metries.

---

The follower completed all the tasks instructed them 10 do.
Error
p=
p= 11
Trend
Missed card
11.1
6.1
Missed card that should have been selected.
Round
2 Extra card Selected a card that should not have been selected.
11.4
4.6
The follower only things that asked them to do, and no more than that.
3 Wrong stop Stopped in the wrong position.
4.5
2.1
MA
Round
11
4 Wrong turn
2.6
1.3
0
20
40
60
80
100
Turned the wrong way when told go left, right, backward, etc.
The follower executed instructions efficiently, using the minimum number of steps required.
5 Wrong order
0.6
0.1
Accomplished the right tasks, but in the wrong order.
Round
6 Inefficient
5.8
6.3
VW
The follower took unnecessary, extra, redundant steps.
11
20
40
60
80
100
7 Other
1.1
0.4
Strongly Disagree
Disagree
Neutral
Agree
Strongly
Agree
Figure 3: Left: Distribution over interactions post-hoc user agreement with three statements about agent performance. Right: Instruction execution errors during 11 rounds for seven common categories. Center columns show prevalence of each error type as proportion of analyzed instructions, for the initial (p 1) and final (p 11) agents; the rightmost column shows trend in prevalence over time.
interaction success: the game score increases from 3.3 + 0.2 to 5.3 0.2 over time. Performance begins to plateau after the sixth round, which we hypothesize is due to model performance limitations. Appendix H provides additional evaluation
static, held-out set of human demonstration data.
Influence User Adaptation on Performance User adaptation over time a confounding factor, as improvements in agent performance may be due to users adapting what tasks they delegate to the follower agent, or how they phrase their instructions. We conduct an additional deployment with both the initial agent (with parameters 0,) and the last deployed agent (01.). ). The agents are deployed concurrently a randomized experiment, Users not know which agent they are collaborating with. We collect 571 interactions, with total of 7,784 instructions. Points the righthand side of Figure 2 (left) show the performance of both models during this deployment. Exccution accuracy (cstimated with 400 instructions) for the initial model 0, improves from average of 66.7 to 70.4 from user adaptation alone, while the final agent achieves an accuracy rate of 83.6 during this side-bye dede-by-side comparison. This demonstrates that while roughly 20% of the improvement is due to user adaptation, genuine agent improvement through continual learning is extremely effective.
User Perception of Agent Behavior User perception of the agent improves over time. In-game feedback (Figure 2] right) improves: the rate of positive feedback per action increases from 44.9 to 60.3% and the negative feedback rate decreases from 10.1 to 4.7%. The reboot rate per instruction also decreases from 15.2 to 8.7%. After each interaction, we ask the leader for their agreement with several statements. Figure (left) shows the distribution of these post-interaction Likert ratings, which improve significantly over the rounds. For example, the rate of agreement with The follower completed all the tasks instructed them do increases from 27.7 to 59.8% of interactions.
Error Analysis(Figurs 3 (right) shows trends of seven agent error types across rounds the long-term experiment. Error types are annotated by workers as part of accuracy estimation. All but one error type decreases significantly over time; the rate of inefficient executions remains stable (Error 6). Manual analysis shows that more than third of remaining missed-card errors (Error 1) could be attributed porrty-written instructions, and most extra-card errors (Error 2) occur when the incorrect card shares at least one property with the target, or is closer to the starting position than the target.
7

---

Language Change Users are likely to change their language over time, as observed in human-human interactions, including in CEREALBAR [T]] In our study, instruction length remains relatively stable over time at an average of 8.6 tokens per utterance. This contrasts with existing findings in reference games, where convention formation facilitates reduction in utterance length over time [24 9), and shows that interaction dynamics and incentives shape the dynamics of language change. We observe changes in instruction content over time, using heuristics to estimate the number of target cards specified in each instruction. There is a decrease in the rate of instructions that specify no cards to interact with (i.e., cards to select or deselect) from 12.0 in the first round to 7.7% in the last. The rate of instructions specifying a single card increases from 81.5 to 88.7%, while the rate of instructions specifying multiple cards decreases from 6.5 to 3.7%. This is potentially because no-card instructions relatively inefficient, while multi-card instructions are relatively hard and have higher risk of cascading errors that require corrections (i.e., deselecting a wrongly selected card). We also find that the rate of instructions containing reference an object (e.g., tree, a pond) decreases over time from 17.9 to 13.5% of instructions. We find that users increasingly shift labor to the agent over time: the average number of steps per set taken by the follower increased from 14.8 to 15.3, while steps taken by the leader decreased from 10.0 to 8.9 steps. These changes illustrate the impact of users developing game strategies and, potentially, model of the follower's instruction following ability.
5.2 Comparison of Learning Design Choices
We conduct second deployment experiment to study the impact of the amount of initial demon- stration data, negative feedback, and reward propagation heuristics. We also compare the feedback learning signal to supervised demonstration data. We design and deploy five system variations:
REWARDPROP uses the exactly same process as in the long-term experiment (Section 5.1), includ- ing reward propagation (Section 3.2).
SIMPLEREWARD measures the effect densifying and de-noising user feedback with heuristics by using only the simple reward (Section 3.2).
NONEGATIVE simulates using positive feedback only. We discard all negative feedback, including reboots, from the data, and then apply the reward propagation heuristics (Section 3.2) |TI
FEWERDEMO uses only 144 human-human interactions (2,114 14 instructions) for initialization, about 25% of the supervised data used in REWARDPROP.
SUPONLY compares training with human demonstrations vs. feedback data given the same number of interactions. SUPONLY does not use data from human-agent interactions, though we deploy it for evaluation. Instead, after each round, we add a fixed number of additional sumannhumann interactions to the training set, equivalent to the amount of data collected with the other variants. Due to the limited amount of demonstration data, we only deploy this approach for three rounds.
We deploy the five variations for five rounds. In each round, all variations are deployed concurrently. each for 200 user interactions. To ensure that each worker interacts with all agents and reduce ordering bias, we assign each worker a unique, consistent, and random ordering of agents; as they play multiple games, they cycle through agents this order. Because all variants except FEWER DEMO are pre-trained on the same initial data, in the first round only, we deploy two agents: one agent for REWARDPROP, SIMPLEREWARD, NONEGATIVE, and SUPONLY, and another for FEWER DEMO. After the first round, we deploy five agents, one for each variation. This experiment cost $16,769.46 .46.
Figure 4 shows results for the five agent variants over five rounds. Overall, learning from feedback is robust to the different design choices. Reward propagation (REWARDPROP) has minor benefit relative to the simple reward (SIMPLEREWARD), REWARDPROP shows slightly faster learning, achieving significantly higher accuracy (80.3 + 1.0) than SIMPLEREWARD (78.1 + 1.0) in Round 3, which means SIMPLEREWARD gives a poorer user experience early on, though catches up quickly [2]
We observe that negative feedback is important: compared to REWARDPROP and SIMPLEREWAN NONEGATIVE learns slower, receives less positive feedback, and more negative feedback and reboots.
We do not remove the option to provide negative feedback during games in this experiment. Leaders are not informed that negative feedback will be discarded for training. We leave the direct comparison between positive-only and positive-and -egattve user interfaces for future work.
"Though initial early experiments showed more striking promise of using heuristies to densify the reward signal, this experiment shows that densification via heuristics is not influential as initially thought.
8

---

REWARDPROF SIMPLEREWARD
70
30
NONEGATIVE FEWERDEMO SUPONLY
80
60
Execution Accuracy
70
50
25
Instruction
Feedback
40
Reboot Rate
60
20
30
50
15
20
40
10
30
Round
Round
Figure 4: Results for five agent variants over five rounds. Left: mean estimated instruction execution accuracy. Right: rate of actions with positive (/) and negative (+) feedback, and instruction reboot rate (/) Across five rounds, all variants exhibit increased execution accuracy, decreased reboot rates, and increasingly positive (and decreasingly negative) action-level feedback.
This difference does not show in early rounds, but appearing around round three. It particularly stands out round four, when it shows an accuracy of 76.8 1.0, compared to 81.0 0.9 of REWARDPROP.
The experiment illustrates that learning can start with a much weaker agent. FEWER uses roughly quarter the demonstration data available to the other variants, and indeed starts with much lower performance, 34.8 + 1.8 accuracy, compared to 70.6 + 1.1 for the others. Even this weak model is sufficient for effective learning: it leads to much steeper learning curve, improving to 64.2 1.5 accuracy after five rounds. However, this comes at cost to the initial user experience: in the first round, only 2% interactions with FEWERDEMO were rated positively by users.
Finally, we observe the feedback data is roughly equivalent to supervised data as a learning signal, showing similar accuracy and feedback trends. REWARDPROP achieves equivalent performance to SUPONLY with less than half the amount of demonstration data (i.e., used for initialization). This shows the benefits of training through user-agent interaction: not only do learning signals directly target current agent behavior, but the training process is significantly less expensive, requiring no human-human demonstrations, but instead just user-system interactions.
Related Work
Learning for instruction following commonly relies on various levels of supervision, such as gold- standard demonstrations [e.g.. 7 43
32
1
5
6
41
g 38), or goal annotations [e.g.. 3 33 40
collapse the distinction between learning and deployment, shifting training and evaluation into human-agent interactions, relying only
small amount of demonstrations data to initialize.
There has been limited work on continual learning for language-related tasks. A major challenge studying this problem is designing a scenario that allows for scalable non-expert feedback and itera- tively improving model and deploying it. CEREALBAR is unique supporting real-time interaction with situated natural language between agents and human users. Alternatively, continual learning has been studied via synthetic deployment by simulating feedback using supervised datasets fe.g.. 34 29, [2]. We focus on studies with real human instructors, which create dramatically different dynamics and data cost constraints. For example, this line of work usually uses significant amount of supervised data, while we emphasize rapid learning from limited interactions. Kojima et al. 22 studied the problem of continual learning for instruction generation by observing human following behavior, including deployment in iterative study. Similar to us, they used CEREALBAR AR because of

---

its ability for scalable deployment and real-time model-human interaction. Beyond this environment choice, our work differs significantly: we study human feedback and the task of instruction following.
Human feedback through annotation or selection of intended output has been used for semantic parsing 45 18 This is related to soliciting partial annotations
post-interaction feedback [30 in dialogue. Recent work has also employed preference based user judgments for reward model learning fine-tune language models using RL [RLHF; 26
48
25
49 39
35]. In contrast
to this existing work, we focus on sequential execution of instructions in embodied interactions, using realtime feedback provided by human users continually train our language- understanding agent. While work in this line showed improving a learned model once, it did not study iteratively improving and deploying model multiple times, core focus of our work.
Continual learning has been studied more extensively outside of a language context, e.g., for robotics. Our work is inspired by TAMER (0 21 46) and COACH [31 4 where humans provide binary feedback to an agent acting in a world. While these methods were developed in the context of an agent learning single task, we focus on generalization previously unseen natural language instructions.
7 Discussion
We propose approach for continually learning to follow instructions through interaction with human users that provide realtime feedback. We demonstrate its effectiveness through multiple rounds of training and deployment, including thousands of interactions. Experiments with various learning design decisions show our approach is robust, and can start from weak initial agent.
Our work sets the stage for several avenues of further study. First, future work could reconsider number of our design choices, including the simple contextual bandit objective and the neural architecture. Application of a user simulator [13] could further improve performance; however, this approach fails to capture dynamics that arise in real interactions through user adaptation. important direction for future work is incorporating additional observational interaction signals (c.g.. Such orthogonal signals can accelerate learning. Another promising avenue for research is more expressive feedback mechanisms, such as through natural language and even complete bi-directional dialogue. Although reasoning about such signals significantly complicates both reasoning and learning, they are also much more informative. A potential direction is to use simple feedback signals like ours to bootstrap the language capabilities of the system, and then gradually switch to more complex natural language feedback.
by observing the instructor behavior) 10 22
Acknowledgments
This research was supported by ARO W911NF21-1-0106 NSF under grant No. 1750499, and a gift from Open Philanthropy. Suhr supported by the NSF under grant No. DGE-2139899 2139899, and via Facebook PhD) Fellowship. We thank Noriyuki Kojima for discussions and sharing code, Suyi Diao for contributing the CEREAL BAR implementation, and the participating MTTurk workers for their work and help in refining the user experience. We also thank Xiang Lorraine Li and Faeze Brahman, well the anonymous reviewers, for providing valuable feedback.
10

---

References
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan navigation: Interpreting
Reid, Stephen Gould, and Anton van den Hengel. visually-grounded navigation instructions in real environments. In The IEEE Conference on Computer Vision and Pattern Recognition, pages 3674 3683, 2018.
[2] Yoav Artzi and Luke Zettlemoyer. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 421 432, 2011 1. UtL /tw.t:://wwwwww.. ac acl web www.:e.....ggaaooooooooooooooooe
[3] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association of Computational Linguistics, 1:49-62, 2013. URL httt:://eclwebb.org//mbboo hgppholllgglggg131 005
[4] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L. Littman. Deep reinforcement learning from policy-dependent human feedback. ArXiv, abs/ 0e0/1902044577 2018.
[5] Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, and Yoav Artzi. Follow- ing high-level navigation instructions on simulated quadcopter with imitation learning. In Proceedings of the Roboties: Science and Systems Conference, 2018.
[6] Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi. Mapping navigation instruc- tions to continuous control actions with position visitation prediction. Proceedings of the Conference on Robot Learning, 2018.
[7] David I.. Chen and Raymond J. Mooney. Learning to interpret natural language navigation instructions from observations. Proceedings the National Conference on Artificial Intelli- gence, 2011.
[8] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. IEFE Conference on Computer Vision and Pattern Recognition, 2019.
[9] Herbert H Clark and Deanna Wilkes-Gibbs. Referring as a collaborative process. Cognition, 22 (1):1-39, 1986.
[10] Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Nickum, and Bradley Knox. The EMPATHIC framework for task learning from implicit human feedback. In Conference Robot Learning. PMLR, 2020.
[11] Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi Analysis of language change in collaborative instruction following. In Findings of the Association for Computa- tional Linguistics: EMNLP 202/, pages 2803-2811, November 2021. doi: 10. 16233/2211//01 findings-emnlp... URL https: http::///aaaaaaaoaaa.o...oom http///////////.aaaaaooooooooo/p wtw.:///....iinnnnneeeemem
239
[12] Ge Gao, Eunsol Choi, and Yoav Artzi. Simulating bandit learning from user feedback for extractive question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 5167-5179, May 2022. doi: 10.
URL BtRLhhtts::/ htttps::/aclaantollll. http:://wa..aaoooooooooo///22/e/
wwtp://g..2.... aal.d22...l-ooon..33 long. 355
[13] Kallirroi Georgila. James Henderson, and Oliver Lemon. User simulation for spoken dialogue systems: Learning and evaluation. In Proceedings of Interspeech, 2006. doi: 10.21437/
iiterspeech22066- 160.
[14] Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. Continual adaptation for efficient machine communication. In Proceedings of the 24th Conference on Com- putational Natural Language Learning, pages 408 419, Online, November 2020. Associ- ation for Computational Linguistics. doi: 00.18653/v1/2020.com 1.33. URL https!
//aclanthology. org.ooggg/200 2otooooooo.orr222220o..111.1.... con11-1.33
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.

---

[16] Emiel Hoogeboom, Jorn W.T. Peters, Taco S. Cohen, and Max Welling, HexaConv. In Interaational Conference Learning Representations, 2018. /
net/ foruum?
[17] D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Associatio,n,,,,, 47(260):663-683 1952. ISSN 01621459. URL htRt: /www. jstor wtt/ssaabeee22877888888
[18] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning neural semantic parser from user feedback. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, pages 963-973, 2017. doi: 10.18653/v1/
P17-1089. BRLLhtp::// httt:::/aalaattoollooooooo P17-1089.
coli. uni saar and. de http/dff//P//
[19] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2014.
[20] W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: the TAMER framework. In Proceedings of the International Conference on Knowledge Capture, 2009.
[21] W. Bradley Knox, Peter Stone, and Cynthia Breazeal. Training robot via human feedback: A case study. In Proceedings of the International Conference on Social Robotics, 2013.
[22] Noriyuki Kojima, Alane Suhr, and Yoav Artzi. Continual learning for grounded instruc- tion generation by observing human following behavior. Transactions of the Association for Computational Linguistics, 9:1303-1319 2021. doi: 10.1162/tac_ a a_00228. URL
https::/aclanthollee org -acl........02...11...11
[23] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An interactive 3D environment for visual AI. arXiv, 2017.
[24] Robert M. Krauss and Sidney Weinheimer. Changes in reference phrases function of frequency of usage social interaction: a preliminary study. Psychonomic Science, 1:113-114, 1964.
[25] Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural ma- chine translation be improved with user feedback? In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 92-105, June 2018. doi: 10.18653/v1/11883011 URL 3012
https://acll http///lllattoooo.....oooom
[26] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit reinforcement learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1777-1788, July 2018. doi: 10. 10.18653/v1/P18-. 165. URL https: : tttt:p:://acaanthollle httph//ooooo..or///-111115
feedback for sequence-to-sequens
[27] Akshay Krishnamurthy, Alekh Agarwal, and John Langford. PAC reinforcement learning with rich observations. In Advances in Neural Information Processing Systems, 2016.
[28] John Langford and Tong Zhang. The epoch- grech- algorithm for multi-armed bandits with side information. In Advances Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver British Columbia, Canada, December 3-6, 2007, 2007.
[29] Carolin Lawrence and Stefan Riezler. Improving neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Axsoci- ation for Computational Linguistics (Volume I: Long Pupers), pages 1820-186 1830, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 8653/v1/P188 1169. URL https: / wewanntology.oogeoooooooooooo
12

---

[30] Bing Liu, Gokhan Tür, Dilck Hakkani-Tün Pararth Shah, and Larry Heck. Dialogue learning with human teaching and feedback end end-to-end trainable task-oriented dialogue systems. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pages 2060-22000 206000206,,920 June 2018. doi: 10. (0.18653/11/118-11338
wrg. N1 B----1111111
[31] James MacGlashan, Mark K. Ho, Robert Tyler Loftin, Bei Peng, David L. Roberts, Matthew Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In Proceedings of the International Conference on Machine Learning, 2017.
[32] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1004 1015, 2017. doi: 10. 10.18653/1//D17-1000 URL
http://ac web. org/ http:/aanoooooooolo111/
[33] Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2667-2678, 2018. URL https: / //ww:../wwwwwww.. aclweb. org/ ant Mathologgy/18-1120
[34] Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neu- ral machine translation with simulated human feedback. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1464-1474 1474, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. N0.18653/11/117-113
BtLhttps:// www wwwwww.wwwwb....... wwtp:/cwwwww..cra.aatttooooo///.
[35] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations, 2023. URL https://openrev//
net wttee//oooommmdddm001.. htt.:/orrmmmm110000222222222
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. ArXiv, abs/1 707.06347, 2017.
[37] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages
August 2016. doi: 10. 20118633v1/P166 1162. URL
https: antha /tttps///acllatthooooo...ooo
www.:l..org/p... www.olaa..oooo.ooooooo.oooo 1162
[38] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In IEEE Conference on Computer Vision and Pattern Recogni- tion, June 2020.
[39] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with hu- man feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008- 3021, 2020. URL htttps://proceedingg 1 119985555999983555555 pawp pdf
[40] Alane Suhr, Srinivasan Iyer, and Yoav Artzi. Learning map context-dependent sentences to executable formal queries. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2238 2249, 2018. URL http:: www.eee....gaaaaaeaaeee wtt./attoooogy//1BBBB.
[41] Alane Suhr, Claudia Yan, Charlotte Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. Executing instructions in situated collaborative interactions. In Pro- ceedings of the Conference on Empirical Methods Natural Language Processing, 2019. doi: 10.18653/v1/DI 1/D19- 1218. htts::/ / www. wwwwwbewbb...c http:/ewww.rr..tttoooooo///////e
13

---

[42] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. 1EEE Transactions on Neural Networks, 9: 9:1054-1054 1998.
[43] Stephanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence, 2011.
[44] Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the International Joint Conference on Artificial Intelligence, 2015.
[45] Sida I. Wang, Percy Liang, and Christopher D. Manning. Learning language games through interaction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2368 2378, 2016. doi: 10.18653/v1/P16-1 1224. URL hEtps://ww> aclweb. org/ http:/annhhooooy/116-1122
[46] Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter R Stone. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the Conference on Artificial Intelligence, 2018.
[47] Ronald J. Williams. Simple statistical tradient-followin algorithms for connectionist reinforce- ment learning. Machine Learning, 8, 1992.
[48] Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. An imitation game for learning semantic parsers from user interaction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6883-6902, November 2020. doi: 10.18653/v1/2020.
Emnlp-main.555 URL /tt.Ltttee///a..attooooooooo////
emr.111000........mmmmmmm main. 559
[49] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 085939,,, 2019.
08593
14

---

A Limitations and Broader Impacts
Our study was conducted with group of 108 workers, all recruited from English-majorig locales, due to the complexity of recruiting and training workers given the complexity of the task. The group size limits the variation language use we observe. Its composition restricts our ability to evaluate generalization to other languages, an important direction for future work. Another question for further study is the dynamics created when completely new users join the community in later stages.
Because of the complexity of our studies we kept our architecture close to previous work on CE- REALBAR, and did not study more contemporary architectures or using pre-trained models. We hypothesize using both could lead to better performance, and more consistent improvement trends. This is an important direction for future work.
Our learning setting involves deploying our agent in interaction with human users, which raises time and interaction considerations not present in RL scenarios without human in-the-loop. This is especially stressed by working in a complex dynamic system where agent observations are coming from nolicy-dependent distribution, interactions they have with the agent. Thus, we prioritize sample efficiency when designing our learning algorithm, and opt the simplicity of the contextual bandit optimization problem, which is known to be sample efficient compared to methods maximizing total reward [28 27]. This also gives us an objective that mathematically similar over more complex RL approaches in terms of stability and predictability in learning. Exploring more complex RL algorithms in this setting, e.g., PPO (6), is compelling direction for future work.
users adapt their language and behavior to the previous
supervised learning objective, which is a benefit
We do not vary the settings of CEREAL Effenberger et al. [[]] find that the interaction design and incentives influence the process of language change. Studying the impact of the scenario design decisions would have significantly complicate our experiments, and increase our costs. We decided not to focus on this research question in this work, but treat these parameters as fixed. Although the analysis of Effenberger et al. [I]] shows that CEREAL creates interesting and complex language dynamics, further study of the impact of interaction design decisions on continual learning is important, and currently under-studied, Our work does not answer these questions, but we hope it will stimulate further research into them.
The data and models we release are not designed to be directly deployed beyond the CEREALBAR scenario. In general, deployment of continually learning systems requires guardrails and monitoring to avoid various undesired outcomes, including acquiring behaviors that may harm users.
B Model
We implement our policy as a neural network based on the design of Suhr et al. [41]. The inputs are an instruction and observation o, and the output is distribution over actions. The policy architecture is composed of several modules that combine to a single network.
Embedding Instructions We embed the instruction E (#....., zn) of length with a bidirectional recurrent LSTM [15]. This results in a sequence of hidden states (b.....,h,) The embedding of x is the final hidden state of the sequence h,,
Embedding Observations Each agent observation o includes information about the observable environment and the instruction execution so far. The follower agent in CEREALBAR has partial observability. We use a representation similar to that of Suhr et al. [41) but without making the simplifying assumption of full observability. The environment state W is tensor representing the properties of each position in the environment as embedding indices. The properties represented in W also encode information about the follower's trajectory so far, the presence of obstacles in the environment, and the follower's observability. Due to partial observability, each position's 's representation is derived from its most recent observation; any information that changes about the world may be outdated in W. We embed W into a dense tensor W'.
Fusing Embeddings After independently embedding the instruction and observation into h, and W', we compute a joint representation of both inputs using text-conditioned (i.e., via h2) convolutions over W'.
15

---

Transforming the Coordinate System Predicting actions requires interactions between represen- tations of multiple positions. W represents the environment using offset coordinates, which do not precisely represent the structure of hexagonal grid in CEREAL BAR. We transform W to axial coordinates [16], and translate and rotate the tensor such that the center position represents the agent's current location, and the agent is facing in consistent direction. These transformations are not parameterized.
LINGUNET We use LINGUNET to predict the policy distribution over actions T(- T o; 0), with slight modifications the design of Suhr al. [4T]. For all convolutions, we apply hex-based convolutions with kernels that operate only on voxels within hex diameter of d around the center voxel, for a kernel size of d. We apply instance normalization the last LINGUNET layer of the imput and bastd convolutions. Finally, we do not perform the final transposed convolution. Instead, we directly predict distribution over the action space given the output of the transposed convolution.
B.1 Inference
We use ensemble-based inference. Given sets of model parameters 0 (01..... .a......0mue 0m), we construct policy π over executable actions using voting!
(a I, 0,
(3)
exp
aaag max of ((10,188)))
1sism
Actions are sampled and executed from .(. o; 0). Executing action in the environment results observation according to the transition function T. We continue to sample actions until the stop action STOP is sampled, until the leader manually reboots the follower. The STOP action marks the current instruction as complete, which either results in the follower's turn ending, or it receiving the next instruction follow.
C Implementation Details
We lowercase and tokenize instructions using BPE [37 with maximum vocabulary size of 4,096 and a minimum wordtype occurrence of [ We learn size-64 word embeddings from scratch. We encode instructions with single-layer LSTM RNN [15] with 128 hidden units. We embed each position's 's properties into vectors of size 16. We use the same LINGUNET hyperparameters as Suhr et al. and did not perform additional hyperparameter search.
use an ensemble size of m = 10. We do not train ensemble, but train ten separate models and apply ensemble-based inference during deployment. When using reward propagation, we use maximum distance of 8 for propagating to previous actions that received no feedback. For training, use batch size of 16 agent steps, learning rate of 0.001, and ADAM [19] for optimization. We re-initialize model parameters from scratch at the beginning of each round of parameter optimization. We use held-out subset of the original CEREALBAM training comprising 5% of the original split. After each epoch, we evaluate model performance using SWSD (Appendix D) the validation set. We use patience for stopping; if ten epochs have passed since the last epoch where the validation SWSD surpassed the previous global maximum, we terminate the training process and choose the model parameters that maximize validation SWSD. We single GeForce RTX 2080 Ti for training each model. Training a single model takes about 28.9 hours on average. We inference CPU during deployment.
validation set for early stopping,
Comparison of Learning Design Choices In our second experiment comparing different lcarning design choices, deploy for five rounds. This number of rounds was chosen because after five rounds in the long-term experiment (Section 5.Ip, learning trends were clear; this choice balances
We assign zero probability inexecutable actions, ie., that would result in intersection with an obstacle.
We use the implementation provided by HuggingFace at http:://lsggsngfacccco//// tokenizers/
16

---

experiment cost and insight. If we acquire more than 200 interactions per round because of the crowdsourcing process, we select exactly 200 games for training and analysis by preferring earlier games played by each worker. We discard the other games.
D Evaluation
Instruction Execution Accuracy For each deployed agent, we randomly sample instruction execution traces & C & S & for manual evaluation. & contains all instructions marked as complete by the agent, and & contains instructions that were either marked as complete rebooted [S Excluding rebooted instructions from this evaluation creates a biased sample, as reboots nearly always reflect incorrect instruction execution, so we re-adjust accuracy estimates based on reboot rates. We assume all rebooted instructions are incorrect executions. The adjusted correctness rate is:
Exse, [lcoron( & E
correctness
3
where correct( Z) is user judgment of execution ē = ((o,, ds, w for instruction T.
Static Evaluation Data We also evaluate on the development split from Suhr et al. 41, with success
weighted by stopping distance (SWSD). SWSD is computed per instruction execution:
, 1+ ट E-
SWSD(E', 2--
where e' is the trace of the agent's execution of an instruction and e* is the human demonstration. रーニ و" only 7' results in the set of cards selected in ट" ج| -²_ || is the hex distance between stopping positions. SWSD is stricter than simple card-state accuracy [41], as it gives only partial credit to instructions where an execution stops in an incorrect position.
E
Crowdsourcing and Data Details
This study received an exemption from the Institutional Review Board of the institution where the research conducted. Worker identities anonymized in the data we release
We qualify workers through a tutorial and a short quiz about the game rules. Workers are also required to reside in an Anglish-majority locale and have HIT approval rate of over 90% with at least 100 approved HITs during their time on MTTur.. The base pay for completing the qualification task $0.50 USD, and qualified workers receive $2.00 bonus. We qualify 108 workers. Following Suhr al. [41). we pay workers bonuses per point earned each game, increasing the compensation per point the game score increases. On average across all experiments, each game costs $2.91 and workers are paid an average $21.85 per hour.
We split workers into two pools: expert and novice. Expert workers earn a 50% higher bonus per game than novice workers. Workers are moved the expert pool after playing least two games with score greater than zero, as long as their rate of giving feedback is greater than 75% of instructions 16 Workers return to the novice pool if they play for two rounds with feedback rate of less than 75% of instructions. 65 workers achieve and maintain expert status throughout the experiments. Only expert workers are qualified to provide post-hoe instruction execution judgments, where they are paid $0.07 per judgment of instruction execution. Worker IDs are anonymized in the distribution of interaction data. Figure shoos the instructions provided to workers in MTurk, and Figure 6 shows the CEREALBAR interlace during interaction.
Agreement For about 20% of instruction execution receiving post-hoc judgments, we acquire judgments from three workers; we find that for only 3.6% of these no consensus is achieved among the three workers, which indicates very high overall agreement.
In this evaluation, we ignore all instructions that were not completed due to game ending. 16e rate of feedback per instruction measures the proportion of instructions where at least one action in the follower's instruction execution is given positive negative feedback. including reboot.
17

---

HIT, you'l collaborative
be Leader and will with astomated Polnwer it not another workers
You should abto connect quickty pertne:. you sound more than minute two maiting for partner, plvise contect
you
garn,
game page, page
SUBMISSION)
Check the HET lite see whather you ard in the novice or arpert group.
decals
wadsers
We will evauuto workers in the export batch perfodically make sure they are still adieering our oxport standards, and wo will oxcluste norice workers aher overy batch see
Some things take account Leader garnes
You should move around the map with your porther to collect sets.
You shousd under stand what makes vold set (thee cards with cifferent colors, Shapes, and numbers)
You be actively giving good/tted levetback pariner lesal
insircciion.
forming pootty
Give feedback for over 75%, Instructions (feedback can include rebocting the follower).
pool,
least
with
feedback
instructions
Please report any issues you
box below, take screenshots where possible PREDACTED FOR SUBMISSION].
Don't uttempt play than one game time. Our server will only accept single connection from each warker at time. We appreciate actentioon while playing
Figure 5: Instructions provided to workers for the main task on MTurk. Detailed instructions about gameplay were provided separate set of webpages, and will be available alongside our code when released.
pog
Lader
Feedback
Folowerss View
Reboot
[DONE] turn around and get the red triangles next to the mountain
In this turn:
[CURRENT]
orange card
Time Left:
SEnO
Figure The CERRAL BAR interaction interface. Users provide instructions in the command box, and feedback via either buttons in the GUT keypresses. The follower's partial view of the environment is visible in the bottom righthand corner of the interface.
Additional Results
F11 User Perception of Agents for Comparison of Learning Design Choices
Figure shows the Likert distribution for the three post-interaction statements users are asked about for the experiments comparing learning design choices, where we concurrently deployed five systems for five rounds.
18

---

F.2 Evaluation on Static Data
Evaluation through human-agen aggett interaction is the main focus of our work. However, we also evaluate cestruction-oolloiing agents against held-out, static data from Suhr et al. [41]. This evaluation does take into account how the actual data distribution shifts over the agent's lifetime, because of the dynamics between the agent and human users. Figure showws average SWSD for the models deployed in each round. SWSD begins 39.7 for the initial model, and peaks at 46.4. This improvement due entirely to adding training data acquired from human-agent interactions.
The follower completed all the tasks I instructed them to do.
REWARDPROP
20
SIMPLEREWARD
20
40
60
100
NONEGATIVE
20
40
60
FEWERDEMO
20
SUPONLY
20
The follower only did things that asked them to do, and no more than that.
REWARDPROP
SIMPLEREWARD
20
NONEGATIVE
20
FEWERDEMO
20
SUPONLY
20
The follower executed my instructions efficiently. using the minimum number of steps required.
REWARDPROP
SIMPLEREWARD
20
NONEGATIVE
20
FEWERDEMO
20
40
SUPONLY
20
40
100
Figure 7: Distribution of post-interaction user agreement with three statements about the follower's performance our approach comparison experiment.
19

---

46
Development Set SWSD
42
2
3
4
5
Number of Training Instructions
-10
Figure 8: SWSD on the held-out development data, averaged over five runs of sampling-base inference.
20


Reviews:
[{"review_id": "kClqxAnejB", "reviewer": "Reviewer_zvAL", "summary": "The authors work on the CEREALBAR setting, where two agents (one human and one computer) cooperate using natural language to achieve a shared goal. Specifically, the authors propose a new setting where the human agent can provide binary feedback to the computer agent. In their new setting, the authors follow the contextual bandit setting and use a REINFORCE-like algorithm to train the agent. Overall, experiments show that the agent achieved an improvement by learning from human signals.", "questions": "1. Why do you use a contextual bandit rather than the full RL setting? This is a bit unintuitive since you have a sequential decision-making process.\n2. In Figure 3, it seems the performance is the highest at around round 5. Is there a reason for this?\n3. Do you plan to release the collected human feedback data? I checked the data folder in the supplementary file but didn't see them.\n4. What models are you using? The paper mentions neural networks, but could you please give more details?", "limitations": "Limitations are discussed in the Appendix.\nFor example, the authors openly disclose that they did not use a more modern architecture for their agent.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors explore a setting that involves real-time human feedback, which may be under-explored.\n2. The experiments involve significant human labour and may be beneficial to the community if the authors share their data.", "weaknesses": "1. Limited contribution. The authors simply applied existing algorithms to their settings.\n2. Experiments show little insight. Their experiments demonstrate that the agent improves by receiving more human feedback, which is expected.\n3. (Minor) Lack of concrete numbers. Results are mostly shown in figures, and it's difficult to see the real performance numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We are looking forward to answering any follow up questions during the discussion period.\n\nContributions: As the reviewer suggests, indeed, learning from real-time human feedback in embodied interactions is certainly an underexplored area. There is a small amount of existing work in robotics that used real-time feedback to train non-language policies, from both explicit (e.g., TAMER – Knox et al. 2009, COACH – MacGlashan et al. 2017) and implicit (e.g., EMPATHIC – Cui et al. 2020) human feedback. But all this is not focused on natural language. Reinforcement learning from human feedback (RLHF, as in Ziegler et al. 2019 and more recently) is focused on preference-based feedback (i.e., comparing two possible outputs) provided by annotators outside of the interaction context; this work is also not embodied. While our work builds on existing techniques, our learning process as a whole, including experimenting with different ways of mapping feedback to rewards, is a new contribution. The objective we use is a variation of the well known policy gradient REINFORCE objective (although we use it in a bandit setting), but it's part of a complete learning process that is our contribution.\n\nData sharing: We will share all our data publicly under the MIT license for maximum usability (briefly noted in the paper). The data is included in our supplementary material, in the `game_recordings/` directory; the `data/` directory includes classes and utilities for managing the data. We apologize for the confusion. \n\nNumbers in figures: We specify key numbers in the text. We don’t discuss concrete numbers from Figure 3 (left) or Figure 4 (right) and only highlight general trends in the text due to space limitations. The figures try to balance readability of trends with accuracy. We will add concrete discussion of these two subfigures into the main text, with the exact key numbers. We will also release the complete raw data underlying each of the plots.\n\nQuestions:\n\nQ1, contextual bandit (CB) objective: Generally, there's a significant credit assignment challenge in a sequential decision problem like this one, and naively computing discounted returns can lead to many wrong reward assignments. CB avoids this, even if at the expressivity cost of restricting the reward to a single step. Studying better solutions to the credit assignment problem is an important problem for future work. Note that most recent RLHF work also uses a contextual bandit reward, albeit even more restricted because reward is assigned to the complete output (even though text generation can be cast as a sequential decision process). So we are not the only ones opting for this simplification. Also, theoretically, CB has sample complexity advantages with much tighter sample complexity bounds when comparing upper bounds for contextual bandits (Langford and Zhang, 2007), even with an adversarial sequence of contexts (Auer et al., 2002), to lower bounds (Krishnamurthy et al., 2016) or upper bounds (Kearns et al., 1999) for total reward maximization. This is critical in our scenario where we learn rapidly from few examples. \n\nQ2, performance in Figure 3: Figure 3 shows likert scores for post-interaction user questions and error categories, so we are unsure of where this conclusion is being drawn from – could you please clarify where in Figure 3 there’s evidence that performance is highest around Round 5?\n\nQ3, feedback data release: Yes, please see above. The data is included in the `game_recordings/` directory of the supplementary. It will be released under MIT license. \n\nQ4, models: We use a model based on prior work (Suhr et al. 2019), where the inputs (instruction and observation) are embedded and a modified LingUNet (Blukis et al. 2018) is used to predict an action. This is specified in Section 4, and details of the model architecture are available in the Supplementary Material."}, {"review_id": "kD6RyOCbP4", "reviewer": "Reviewer_hEYa", "summary": "This strong work presents a systems contribution in a fully-fledged system for continual learning from language feedback, in the context of situated human-to-robot instruction following tasks. Using the CerealBar environment (roughly inspired the card game SET, with an embodied flair), this work starts by learning an agent's instruction following policy from a set of offline demonstration data. This agent is then deployed against a crowd of *real human users* who work together with the agent to complete more and more task instances in the CerealBar environment, with humans providing (sporadic) binary feedback in real-time.\n\nAfter each learning round (a fixed number of episodes), the agent policy is updated from the real-time reward data, where the authors define a heuristic to assign the real-time feedback to individual (state, action) pairs taken by the robot; this heuristic is derived from empirical \"delay in human response\" data, as well as from the traditional RL literature for assigning credit to actions that aren't explicitly provided feedback. The agent update is a contextual bandit style update, using a variation of the simple REINFORCE policy gradient update (training effectively a 1-step RL policy).\n\nAcross 11 rounds of updates from real-world human data, the proposed system obtains impressive results, showing a constantly growing trend in both individual instruction execution accuracy, as well as total CerealBar score (with the steepest increase in performance happening across the earlier rounds). The work also has a series of systematic ablations (comparing the binary feedback based approach with \"full supervision\" - showing that the proposed bandit approach can almost match the same performance), as well as qualitative Likert scale results from the real crowd users actually interacting with this system.", "questions": "What motivated the choice of using clipped IPS coefficient for the policy gradient update vs. a PPO-style update (with a learned one-step value function estimator for debiasing); given the choice of using REINFORCE policy gradient anyway for the 1-step update, this would've been an easy addition, and possibly more stable?\n\nI'd be curious to see a more detailed breakdown of the types of language instructions (and *when* in an episode a user usually provides feedback) across the different rounds; are there more complex instructions/abstractions learned over time? Do users intervene differently in different rounds? What are the failure modes?\n", "limitations": "The paper provides a thorough limitations section (in the appendix) that is well thought out and clearly states weaknesses in the current approach. It would be nice to move some of the punchlines to the main body in the final version of the paper though!\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a well-written and extremely well-executed systems contribution - the task of learning to improve situated agents from language and binary feedback is incredibly timely, and every part of the proposed system is implemented carefully and studied thoroughly. The evaluation is strong (both qualitative and quantiative), and definitely show that not only is the agent able to effectively improve with feedback, but it's able to improve *efficientyly* and actually grow its capabilities over time. I am a huge fan of this work.\n\nSeparately, I think the dedication of this paper to open-sourcing all parts of the pipeline, including the multi-round crowdsourced interaction data is admirable, and will be an incredible resource to the community.", "weaknesses": "In a paper such as this, I can understand it's very hard (costly + time-consuming + introduces a lot of confounds) to evaluate multiple different learning approaches at the full-scale of the real-world evaluation (though I really appreciate the experiments in section 5.2 that does this for a limited number of rounds). That being said, I would love to look at other agent learning paradigms, and justify the use of the 1-step contextual bandit style reward, vs. a multi-step RL approach that automatically learns how to perform credit assignment given the sporadic binary feedback. Especially using some of the more recent tools in the RL toolbox (e.g., learning a value function/advantage function, PPO-style clipped updates, or even off-policy/offline RL methods).\n\nA general worry I have is that much of this paper hinges off the design of the reward/utility function, which starts from the principled binary feedback provided by users, but is then further processed through a series of heuristics that may or may not really capture what's going on (for example, how the current heuristic labels transitions when there isn't an explicit binary reward tied to that timestep). I would love for the authors to address these choices in a bit more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We are looking forward to answering any follow up questions during the discussion period.\n\nOther methods: We agree that experimenting with different learning methods is a good direction for future work; in this case, we opted for simplicity especially in light of only having a small amount of data to learn from (methods which require learning multiple models, such as a value function, may have stronger requirements in the amount of data used for training), and working in a complex dynamic process where agent observations are coming from a policy-dependent distribution (i.e., because users adapt their language and behavior to their interactions with the agent). The paper also explores uncharted territory, so we figured simplicity is the best way to approach it, even if higher performance is likely with more complex methods. As the reviewer notes, cost was also a factor, as more complex methods often require more tuning. We hope our work paves the way for future work in this area, including along the directions the reviewer mentions.\n\nHeuristics: We manually developed heuristics initially after early experiments showed they were useful. In general, the rate of “errors” assigned by the heuristics was very low (i.e., nearly always reflected the instruction’s intent), so heuristics serve to densify reward in our contextual bandit setup. Surprisingly to us, in our comparison experiments (Section 5.2) we found they were actually not as influential as initial pilot experiments showed; however, there is some evidence they sped up learning early on. So, an important takeaway is that the effectiveness of the approach does not necessarily rely on very involved heuristic reward design (although future work might reveal better approaches that do make significant difference). \n\nPolicy gradient vs. PPO-style learning: see above under “other methods”. This was motivated by preference to simplicity, especially given costs of studies and the goal of rapid updates with a very limited amount of data. Early on, we conducted pilot studies with more complex methods (e.g., COACH by MacGlashan et al. 2017, which is designed to learn from feedback), but they didn’t work well. That said, there were too many confounding factors to draw strong conclusions about the algorithm, except that getting them to (potentially) work is more complex. An interesting direction for future work now that we showed the whole process works is to apply more localized changes to the optimization algorithm (e.g., switching to PPO as you suggest) and understanding the implications. The rationale for using IPS is (a) to avoid exploding gradients and (b) de-bias (end of Section 5). \n\nAnalysis: We didn’t find evidence that users built abstractions or relied on more complex instructions over time; in fact, there is some evidence that they simplified their instructions (e.g., decreasing the rate of multi-card instructions), and reducing the number of references to objects in the environment. We don’t think the CerealBar stimuli is designed to elicit abstractions. In terms of interventions, we found that the rate of user intervention via reboots goes down significantly over time (as the agent gets better). Also, while this is not reported in the paper, we found that users shifted more labor to the agent over time: the average number of steps the follower took per set increased from 14.8 to 15.3 over the long-term experiment, while the number of steps for the leader decreased from 10.0 to 8.9 steps. We can add this analysis to the main paper. If by failure modes you are referring to common errors in instruction execution, Figure 3 (right) reports such categories and their trends via manual analysis, with discussion in Section 5.1.\n\nThank you for the suggestion to move limitation highlights to the main text! We will certainly do this if the paper is accepted.\n"}, {"review_id": "4FSBDTk3RS", "reviewer": "Reviewer_d6RP", "summary": "This work demonstrated a simple yet effective framework for continual learning in instruction following task utilizing human feedback. Using CEREALBAR as testbed, this work demonstrated the framework in abundant details, and show effectiveness through experiment results. This work also conducts various analysis related to human feedback during and post-hoc the interactions. Finally, various decision choices are compared.", "questions": "1. Figure 3: left, the x axis range indicates number of turk workers right? aren't they 108 in total?\n2. Maybe I missed something, but how do you annotated the demonstration data? Also, since reward propagation is one of the main design choice, does the gold data's feedback distribution similar as those annotated by mturkers?\n3. Do you have any clue on how user adaptation contribute to specific error decreasing? (figure 3, right)\n4. line 271: not sure what \"we deploy a single agent for all but FEWERDEMO in the first round\" means. Can you explain more?\n5. (Figure 4, right)There seems to be a golden ratio between positive and negative feedback emerging. What do you think of that?", "limitations": "Limitations are adequately addressed in supplementary material.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is new and well supported by the framework description, the experiment results, and the analysis in the paper. The writing is well organized and clearly written in general. \n\nThis work address embodied ai problem from a relatively new perspective. It demonstrates effectiveness on task performance as well as benefits on reducing annotation effort. This work also provides promising future directions centering around the human-agent dynamics like the form of human feedback, the communication style between human-agent.", "weaknesses": "I don't find any noticeable weakness of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We are looking forward to answering any follow up questions during the discussion period.\n\nQ1, Figure 3 x axis: The x-axis represents a proportion of interactions (so it sums to 100), rather than the exact number of interactions. Each different colored bar represents the proportion of interactions in each round that were given a particular rating. We are using proportions here because the number of interactions per round differs slightly. We will clarify this in the paper.\n\nQ2, demonstration data: To “annotate” the supervised demonstration data with feedback, we assign every action positive feedback, because we assume human demonstrations are gold-standard (i.e., everything a human follower did in the demonstration is correct). We don’t apply reward propagation heuristics to this data, because we assign positive feedback to each action, so in terms of the feedback distribution, demonstration data includes 100% positive feedback while the data collected during human-agent interaction includes examples with negative feedback.\n\nQ3, adaptation and errors: This is a great question. We didn’t find that any particular errors correlated with particular user adaptations. We find that in the direct comparison between the initial and final models ($\\theta_1$ versus $\\theta_{11}$), where user adaptation is reflected in interactions (when both are deployed concurrently in the final round), the initial model still has significantly more errors in all categories except for Errors 5 (which is already very rare) and 6 (which stays static over time anyway). This comparison further shows the error trends reflect genuine improvements in the model, and are only partially influenced by user adaptation (although we indeed see some adaptation, and our experiment measured for it).\n\nQ4, deployment: Thanks for bringing up this confusing wording. What we mean is that for {RewardProp, SimpleReward, NoNegative, SupOnly}, the agent in the first round is exactly the same, as all of these systems are pre-trained on the exact same set of training data, so there’s no difference between them in the first round. But, for {FewerDemo}, the model is pre-trained on a smaller subset of the data. So in practice in the first round, we only deploy two agents: one for {RewardProp, SimpleReward, NoNegative, SupOnly} and one for {FewerDemo}. We will clarify this in the paper. This allowed us to save some interactions (and money) without influencing the experiment.\n\nQ5, feedback ratio: this is an interesting question, and we don’t have a conclusive answer to it. It does seem like the curves converge to a similar ratio across the different systems. We suspect we would need to run this experiment for much longer to confidently say that a consistent ratio emerges. This would likely require re-running with a disjoint pool of workers as well, to rule out the impact of the specific pool of workers. So, we don’t think we are ready to draw strong conclusions, but we do see what the reviewer is pointing to. This raises interesting directions for followup work, which our work enables, such as the impact of learning and interface design decisions on the long-term equilibrium of human user feedback behavior. "}, {"review_id": "J9enQRczys", "reviewer": "Reviewer_Q7Gw", "summary": "The authors propose a method for online continually training an instruction-following agent based on user realtime feedback gathered for a collaborative game CerealBAR. The agent need to follow the human's  instructions and complete the task. The paper utilizes the contextual bandit learning approach, with  immediate rewards computed from user binary feedback. Through the evaluation of thousands of human-agent interactions, they demonstrate a significant improvement in instruction execution accuracy over time. They also conduct experiments with multiple training scenarios and verify the robustness of their approach.", "questions": "Although the model can achieve 15.4% absolute improvement through continual learning over time from continual learning, it is hard to gain insight of how good the an instruction-following agent is. In order to show benefit of off-policy learning in this framework, have we tried some baseline models for comparison, such as offline RL, due to the difficulty of additional online data collection?", "limitations": "The experiments is difficult to reproduce. Training policy with from human realtime feedback can lead to significantly higher cost compared with offline RL or user simulator, which is widely used in dialogue agent design.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors propose and deploy the methods in the collaborative human game. They collected agent-human interaction data will be helpful for future agent design.\n* It is a pilot work to demonstrate the strength of continual learning from human feedback for instruction-follow agent. The authors shows that after training the agent from human-human dataset, the policy can be greatly improved from the online off-policy learning from human binary feedback.\n* The authors conduct comprehensive online experiment with variants of learning choice and human post-interaction evaluation. The variants of learning choice show the robustness of the framework and the human post-interaction evaluation shows empirical analysis on the agent behavior improved after learning from human.", "weaknesses": "* The framework deign is not very clear. The authors do not mention much the design of the policy model, which transform from human instruction and observation into action (Model part in section 4). This paper mainly discusses about how to define user feedback and how to train with continual learning given the feedback.\n* The framework uses binary signal from the user feedback, which is limited to represent the human satisfaction or feedback. It can be difficult to generalize to other tasks with complex instructions and observations.\n* The framework is relatively simple with REINFORCE algorithm. It would be more convincing if the authors can conduct more experiments on other standard off-policy learning algorithm for the robustness of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We are looking forward to answering any follow up questions during the discussion period.\n\nModel design: We provide the design of the policy model in the Appendix. In short, it is a modification of the architecture used by previous work (Suhr et al. 2019). As it is not a main contribution of our work, we have put it in the appendix. Our aim was to take a model that has been shown to work to some extent on this domain, and focus on the learning problem.\n\nBinary feedback: Existing work on learning from human feedback for embodied agents uses this kind of binary, realtime feedback provided by users (e.g., in TAMER and COACH). Binary feedback is simple, and conceptually generalizes to other tasks and feedback scenarios. Effectiveness is an empirical question, of course. However, for example, LLM RLHF uses binary preference signals (albeit unlike our work: given by third party annotators, and are not embodied, not realtime), and which empirically works well for complex reasoning tasks. The simplicity of binary feedback is what allows interleaving it with the interaction. It also doesn’t require complex machinery to interpret it (like natural language feedback). Of course, it carries less information than other forms of feedback (e.g., natural language), even if it requires less of the user. Using such information rich (but more costly to obtain) forms of feedback is an orthogonal and open research question. They come with pluses (more information) and minuses (costly to get, harder to interpret). \n\nRe experimenting with off-policy algorithms: please see the general response and “on using offline RL” below. \n\nInsights on policy quality: We provide comprehensive analysis of policy errors as annotated by workers in Figure 3 (right), which we discuss in the main paper. We see that for nearly all of the error categories, the policy is improving over time; the only error category which remains roughly stable is the efficiency of the policy’s trajectory (roughly 6% of trajectories are marked as inefficient).\n\nOn using offline RL: we agree there are alternative offline RL methods that we could use to swap our REINFORCE-style optimization loop. Integrating them in this way won’t reduce the need to collect feedback data over rounds. These methods are also more complex, while we opted for simplicity in this first study of the problem (there are many open directions for future work). Swapping the whole process with offline RL is less clearly effective. We could swap the supervised initialization using offline RL with the seed data, and that might increase initial performance, or allow us to use less initial data. This would just give a better launching pad for our learning from feedback. Of course, this is an empirical question, which is orthogonal to the questions we study. \n\nOn using a user simulator: While using a simulator could potentially improve model performance as it would generate new data to train on, it fails to capture a fundamental aspect of agents deployed in real interaction with people: the dynamics that arise as real users adapt to the agent through interaction. Manually designing user simulators has been studied in the past and is orthogonal to our contributions; such a manually designed simulator would also be placing assumptions on the kinds of language that people use in interactions with our agents. Building a strong simulator is also a lot of engineering work. Our approach takes advantage of the interaction the system would have with users anyway, and because it derives data from these interactions, it’s always aligned with the data distribution observed in its actual interactions, something that is very difficult to achieve with simulation.\n\nReproducibility: Experiments with real human users are naturally more difficult to reproduce than experiments which rely on fixed / static datasets or simulators; we anticipate that all of the difficulty in reproducibility will arise from recruiting and maintaining a user base. However, we publicly provide code, implementation details, data, etc. Furthermore, conducting these experiments at the scale we did is a key contribution of our paper. Almost no open research that is reported in detail has been done with such deployments over long periods of time, as we do. The complexities are a natural by-product of the core research problem we study, and there is no way around them. Simulators, for example, avoid costs and increase reproducibility, but poorly reflect interactions with real users. For example, simulators are static, whereas real users are dynamic and constantly change their behavior. On a fundamental level, there’s a big difference between the stationary problem simulators provide, and the non stationary problem of a deployed system, as we study. "}, {"review_id": "KhLWqiEqxg", "reviewer": "Reviewer_4ZhD", "summary": "Training of a continuously learning, instruction following agent from feedback provided by users during collaborative interactions. The problem is that humans often give noisy reward and at irregular intervals. The method formulates the learning scenario as a contextual bandit problem and alternates between training and deployment stages. The main issues addressed is the irregular timed reward and credit assignment problem. The method addresses this using heuristics and demonstrated its effectiveness in the CREALBAR environment using human evaluation.", "questions": "- if I understand correctly, since the method formalizes itself as a contextual bandit problem, maximizing the immediate expected reward, Line 77 says \"Feedback signals are given in realtime, and are not timed to coincide with a specific action\" will not be favorable for training since the corresponding action and reward is not even matching up?\n- isn't the assumption of giving a binary reward and optimizing for the contextual bandit objective amounts to gradient descent/ascent on the right/wrong actions (with noise, since humans give reward quite randomly), which is almost equivalent to supervised learning? If this is the case, I feel like the continual learning part is just iteratively collecting more supervised learning data, which performance improvement is quite obvious\n", "limitations": "Limitations and broader impacts are complete and included in the appendix", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is straightforward to follow. The topic is very relevant. The evaluation and improvement throughout the interaction rounds are significant in many aspects. The experiments conducted are very thorough and complete and have convincing results to highlight the continual learning nature of the agent.", "weaknesses": "- The assumption of feedback is matched with an action or close by actions (heuristics presented in the paper) is somewhat restrictive\n- transitions without rewards are discarded and not fully utilised, quite a waste of resources\n- requires expensive labelling process", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We are looking forward to answering any follow up questions during the discussion period.\n\nAssumption of feedback alignment: real time feedback follows patterns of human response, including its delays. So this assumption follows the role of humans in the process and how interactions happen in real time. It’s possible for humans to give feedback to decisions that were made further in the past. Our learning approach does not handle it, and will suffer when such a signal is given. However, this is not something we force our users to avoid, and empirically the approach handles this and other types of noise quite well, showing effective learning. In addition, this correspondence between feedback and action is also seen in prior work (e.g., COACH and TAMER). We agree it does not provide perfect credit assignment (an important direction for future work), but our approach is robust to it. Our heuristics make further assumptions, but our experiments generally show that they are not critical for effective learning. \n\nDiscarded training data: there are two conditions. When we don’t use heuristics, the data is essentially discarded. When applying heuristics, nearly all actions/transitions are assigned feedback (and thus used during training), except those that appear after the last feedback provided by a user in a rollout, as we can’t estimate what feedback the user would have given. This was a key motivation behind the heuristics. However, the heuristics didn’t prove critical, as noted above. It remains an open problem for future work to study if it’s possible to infer a learning signal for actions without feedback. We don’t claim to completely solve the problem of learning from such feedback, but to show an effective approach and its significant potential. \n\nExpense of labeling: except the supervised data used for initialization (a very small amount) and the human evaluation, there is no conventional labeling cost involved in the actual continual learning. The cost of our experiments is paying for people to interact with our system, and user feedback is built into the interaction. It’s just like deploying a system, but we don’t have the ability to deploy a real product for people to interact with, so we create this scenario by paying people to act as users. This approach is advantageous not only against conventional data annotation, but also when compared to most RLHF methods, which rely on post-hoc preferences from third-party annotators (and then you do see high labeling costs). \n\nQuestions:\n\nQ1, alignment between actions and rewards: To clarify this sentence, we mean that we don’t force users to provide feedback for each individual action the agent takes (e.g., by moving the agent only after the user gives feedback for the current action), but we instead allow users to provide feedback at any point in time as they observe the instruction being executed. This creates a more challenging signal than forcing humans to provide feedback after action (which would give perfect alignment), but would make for a very frustrating system use. In practice, alignments between the user-provided feedback and the intended actions are very good, especially after correcting for reaction time delays (as is common when handling human responses). There’s still noise, and the credit assignment problem is not completely solved. But our approach is robust to it, as demonstrated by our experiments that show effective learning over time. \n\nQ2, equivalence to supervised learning: the objective does look a bit like supervised learning (except the IPS coefficient). However, it only appears like a supervised learning objective: the actual supervision isn’t coming from a gold-standard demonstration (i.e. a human demonstration of instruction execution). Instead, the data we train on includes trajectories sampled from our policy (conditioned on user instructions) annotated with feedback provided by users. Additionally, unlike supervised scenarios (and single-agent RL scenarios), this is a non-stationary environment, because humans constantly change their behavior. So the data is not coming from the same constant distribution, as assumed in supervised learning. So in practice while the objective appears similar to a supervised learning objective, the learning problem is far from a supervised learning problem. The objective is also derived from maximizing the value function (i.e., like in REINFORCE), and not by maximizing the data likelihood. But, the mathematical similarity to supervised learning is actually a benefit of our method, as supervised learning is more stable and predictable than many more complex RL approaches.\n\nAnother small note: while user-provided feedback is somewhat noisy, it is not so random that it is impossible to learn from, and analysis of the human-provided feedback shows that it is very high quality. This is also demonstrated by our results, showing very effective learning. \n"}]

Author rebuttal:
We thank the reviewers for their comments and questions! We look forward to continuing the discussion here during the discussion period. Below are responses to general comments.

Experimenting with different learning paradigms: While we use an objective based on the popular policy gradient REINFORCE objective, we use it in a new learning process that is a new contribution of this work (i.e., learning through live interactions with human users who provide feedback in real time), and demonstrate its efficacy. There are many avenues for future work, including trying more complex RL algorithms such as off-policy learning algorithms or PPO. We opted for simplicity here, focusing on sample efficiency, because of the complexity and costs of studies with human users. And, mostly because this kind of problem is relatively understudied, so we decided to start simple. This is a first step, and we hope our work paves the way for more work in this exciting area, which has so many interesting open problems!

Reproducibility: Conducting these learning experiments at scale (including addressing technical challenges involved in successful deployment and training, but also gaining insights from their deployment, e.g., showing the robustness of our approach to system variations,and analyzing user adaptation over time) and their results are key contributions of our paper. While learning from interactions with human users does pose challenges for reproducibility (i.e., from recruiting and managing a user base), these challenges are unavoidable when studying such complex interaction scenarios. For example, relying on manually designed user simulators will place assumptions on how interactions proceed, require significant engineering effort, and will not reflect the dynamics that arise in real human-agent interaction over time. Indeed, we put significant engineering effort.We are releasing everything, including our results, data, and platform (its original version was released as CerealBar in the past, but we modified it to capture feedback and conduct our studies), all under the permissive MIT license. This is another contribution of our paper: a platform that enables conducting such experiments, and will save much of the engineering effort we put in for future research. Our code and data is also attached to the supplementary material.

Model: The model design is not a main contribution of our work, so we put details in the Appendix. Our model is a modification of the neural architecture used by previous work (Suhr et al. 2019); we chose to adapt an existing model that has been shown to work on this domain, and focus our main contributions on the learning problem itself.

Meta-review: First, the final scores of the paper are 5, 5, 7, 8, 8, with an average of 6.6. Reviewer d6RP intended to update the score to 8 but was unable to do so due to the system.

The work designs a contextual bandit learning approach where human users instruct an agent using natural language.

This is a timely work studying human feedback to improve agent models. Reviewers recognize the following strengths of the work
* Important and timely problem to study.
* Thorough evaluation, with strong empirical results.
* The promise of open-sourcing and data sharing.

We thank the authors and reviewers for the engaging discussions. Most of the concerns of the reviewers were addressed in the rebuttal, and 4 reviewers raised their scores. 

For weakness, multiple reviewers pointed out the justification of the 1-step contextual bandit style reward vs. other possible approaches, such as full RL-based methods. The authors opted for simplicity in this work. Although it would be nice to further study the choice of the learning paradigms, the contributions are still meaningful and significant. Please add the further discussions regarding this in the final version.
Decision: Accept (spotlight)