Title: Continual Learning for Instruction Following from Realtime Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: continual learning

interaction

instruction following

user feedback

natural language processing

language grounding

situated interaction

collaboration
Abstract: We propose and deploy an approach to  continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting  user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.

Continual Learning for Instruction Following from Realtime Feedback

Abstract

We propose and deploy an approach to continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.

1 Introduction

The dynamics that arise in situated language interactions between human users and automated agents expose a plethora of language learning signals. A prime example of such a signal is explicit feedback: when users convey their intent via natural language instructions for an agent to follow, they are well positioned to provide feedback, for example, through a stream of binary signals as the agent acts.

Learning from this type of signal has significant potential. It shifts the burden of learning from annotated data to learning through interaction with users, which not only reduces data costs, but also enables continual 2 improvement through interaction with users. This signal also fundamentally differs from gold-standard annotated data: it directly targets the current agent behavior, rather than reflecting optimal human behavior that may be of less relevance to the agent's current policy. This approach stands in contrast with most methods for learning to follow instructions, where the use of demonstration data entails high data costs, and the clear separation between training and deployment passes over opportunities for learning through interaction with users [e.g., 3, 1, 38].

In this paper, we study the problem of continually 2 improving an automated instruction-following agent 3 by learning from user feedback in human-agent interactions. We situate the interaction in a collaborative environment, which provides conditions necessary for this type of learning: users are incentivized to improve the agent's capabilities by providing feedback, and are continuously present as the agent executes their instructions. We use the CEREALBAR scenario [41], where two participants collaborate towards a common goal in a shared world, using natural language instructions to coordinate their actions. During interaction, human users write instructions for an agent, and provide feedback to the agent as it executes them via a stream of binary signals. Figure 1 illustrates

<latexi sh

1_b

64="

m35

RIEcP7

Z

GMSBV

>A

H

DL

F

r

d

k

KT

w

v

8

y9

+

X

O/0

q

J

f

z

o

j

Y

N

U2

<latexi sh

1_b

64="MS7

8

5A

0

>

B+H

c

VDL

N

FJ3U

2

R

C

Z

E

o

y

I

mk

g

9

u

/

z

Q

O

W

d

w

Y

T

f

X

K

<latexi sh

1_b

64="KfEd0/7UzPAR

5Z

G

Vc

LOqY

>

B

H

D

SgM

F

TX

W+

r

y

C

Jm

n

j

N2

k

vu

9

Q

3

8

o

I

w

p

Learning from User Feedback

ρ

= 1

,

2

,

3

, . . .

Rounds

Contextual Bandit

Training

Aggregated

Training Data

User-Agent Interactions

Instruction-

Following Policy

a

∼

π

(

·

|

x, o

;

θ

ρ

)

Raw Feedback

Data

θ

ρ

+1

the setup and our learning process. To the best of our knowledge, our work is the first to demonstrate continual learning from human feedback for natural language instruction following.

Akey challenge is the complexity of the learning signal. Users provide feedback at their discretion and under time pressure as the agent executes their instructions, resulting in an unpredictable, and often noisy, learning signal. While the concurrency of agent actions and user feedback provides a weak alignment, delays in human response and lack of clarity about which action or actions they critique make it challenging to attribute a learning signal to specific actions. Further complicating learning is the feedback loop created by the continually changing agent behavior and constant adaptation by human users. These factors create a scenario that is particularly difficult for complex methods as they often make various assumptions about the learning signal and the interaction dynamics [20, 31]. We opt for a simple approach with minimal assumptions, and formulate learning as a contextual bandit scenario. We alternate between deployment, where the agent interacts with users that provide feedback, and training, where we compute immediate rewards from feedback signals, and weigh examples using the current policy to account for the continually changing agent behavior.

We experiment with our approach through interactions with humans users, where we repeatedly deploy, train, and re-deploy the agent. We study the effects of human adaptation and process design decisions on the efficacy of learning. We design our experiments to tease apart genuine agent improvements from human user adaptation, and observe dramatic improvements in agent behavior beyond what user adaptation explains, 15.4% absolute improvement in instruction execution accuracy in our longer running experiment. Our code and data is released under the MIT license 2023. 4

2 Technical Overview

Interaction Scenario Wesituate user-agent interactions in CEREALBAR [41], a collaborative scenario where two participants, a leader and a follower, collect sets of matching cards together in a shared, 3D environment. The two participants collaborate towards a shared goal, act in the world, and coordinate using natural language. The leader plans what the pair should do, acts according to the plan, and describes the follower's part of the plan using natural language instructions. The follower's role is to follow the instructions. The pair receives a point for each set of cards they successfully collect. Their goal is to maximize their score. We add to the original CEREALBAR setting the ability for the leader to provide binary feedback signals to the follower as they execute instructions. In our study, the agent controls the follower and the leader is always a human user. We use CEREALBAR as it is the only research environment designed to support the real-time collaborative interaction scenario we study: alternatives (e.g., Thor [23]) are not designed to deploy at scale for model-user interaction studies, and would require extensive modifications to support multi-agent collaborative interactions.

Task The agent's task is to map natural language instructions and observations to follower actions. Actions include moving FORWARD or BACKWARD , turning LEFT or RIGHT , and completing instruction execution with STOP . The agent observation is a partial view of the world state from the

first-person perspective of the follower. Formally, 5 given an instruction x and an initial observation o 1 , our goal is to generate a sequence ⟨ ( o 1 , a 1 ) , . . . , ( o m , a m ) ⟩ , where o i an agent observation, a i is the action taken, and a m = STOP .

Inference and Learning We define the follower agent as a policy π ( x, o ; θ ) parameterized by θ that maps instruction x and state observations o to a distribution over actions. 6 During interaction with users, for each user-written instruction x , we sample and execute actions a i ∼ π ( · | x, o i ; θ ) until we sample the instruction completion action. Concurrently, the user may provide positive and negative feedback signals f . Feedback signals are given in realtime, and are not timed to coincide with a specific action. Between any two agent steps, there could be any number of feedback signals.

We optimize parameters through rounds of continual learning (Figure 1). At each round ρ , we deploy the policy with parameters θ ρ to interact with users, and then estimate parameters θ ρ +1 . During deployment, we collect agent instruction executions and user feedback. The execution of an instruction x during interaction creates a trace made of two sequences ( ⟨ ( o i , a i , w a i ) ⟩ i m =1 , ⟨ ( f j , w f j ) ⟩ n j =1 ) , where each action a i and feedback signal f j are paired with wall times w a i and w f j . We compute reward from the feedback to construct training examples ( x, o, a, r ) , where r is a reward. We optimize policy parameters by solving a contextual bandit problem, maximizing immediate expected reward.

Evaluation Our main metric is instruction execution accuracy for user-written instructions. Because we only have access to agent instruction executions, and not ground-truth demonstrations, we cannot compute accuracy automatically. We use crowdsourcing to acquire human judgments of follower accuracy after each round of deployment. We also evaluate trends in user feedback and ratings.

3 Continual Learning

We estimate the policy parameters from user feedback during interaction with users. The process, illustrated in Algorithm 1, progresses in rounds. Each round ρ includes: (a) deploying the agent policy parameterized by θ ρ to interact with users (Section 3.1), (b) computing rewards from user feedback to construct training data D ρ (Section 3.2), and (c) optimizing the policy parameters using all data observed so far ⋃ ρ ρ ′ =0 D ρ ′ to compute θ ρ +1 (Section 3.3). We initialize the process with a policy parameterized by θ 1 estimated on human demonstration data D 0 (Algorithm 1, Line 1).

3.1 Deployment Interactions

During deployment in round ρ , users collaborate with the agent, and delegate to it tasks by typing natural language instructions. For each instruction x , we sample a sequence of actions from the policy a i ∼ π ( · | x, o i ; θ ρ ) (Lines 9-15). Executing an action in the environment results in a new observation o i +1 (Line 11). For each action, we record the wall time w a i as the time when the user starts seeing the follower starting to execute the action (i.e., the action animation starts) (Line 12). We terminate instruction execution when the stop action STOP is sampled (Line 15).

At any point in time during the agent's instruction execution, the user may provide binary (positive or negative) feedback signals, creating a stream ⟨ ( f j , w f j ) ⟩ n j =1 where w f j is the user's wall time when a feedback button is pressed (Line 13). We also allow the user to reboot the follower at any point during an instruction execution (omitted from Algorithm 1). This is meant to terminate very bad executions before they damage the interaction, for example by leading the follower too far astray. Rebooting adds a negative feedback signal to the end of the stream, and removes all queued instructions. 7

The execution of an instruction by the agent and the feedback given by the user are combined together to create a trace made of two sequences ⟨ ( o i , a i , w a i ) ⟩ i m =1 and ⟨ f j , w f j ) ⟩ n j =1 .

Algorithm 1 Continual learning for instruction following from realtime user feedback.

Input Human demonstration data D 0 ; number of interactions per round T ; dataset construction process convert_feedback_to_reward that maps from an instruction and feedback-annotated execution trace to a set of reward-annotated training examples (Section 3.2); learner train_policy that optimizes a contextual bandit policy gradient objective (Section 3.3).

3.2 Dataset Construction

We use all traces collected in round ρ to construct a training dataset D ρ . Each example in D ρ is a tuple ( x, o, a, r ) , where x is an instruction written by the user in round ρ , o is the agent observation when action a was selected during the execution x in the interaction, and r is a numerical reward we compute from the feedback stream. For each instruction written during round ρ we may create no examples if no feedback was given, or multiple examples if computing the rewards from feedback resulted in attributing rewards to multiple actions.

Simple Reward We consider the value of each positive feedback as f = +1 and each negative feedback as f = -1 . We compute the reward r i for an action a i as the sign of sum of feedback signals that occur between it and the next action after correcting for human response delay time [31]:

where w a i is the wall time of action a i , w f j is the wall time of feedback signal f j , and d is a human response delay constant ( d = 0.2 seconds). If the set is empty with no feedback signals within the time window, or if r i = 0, we do not set a reward or create an example for the corresponding action.

Heuristically Propagated Reward As users are not required to follow any specific feedback schedule, the simple reward computation is likely to result in relatively sparse reward assignment, with many actions not receiving any learning signal and thus essentially thrown out as training data. Indeed, in our experiments, we observe that roughly only 63.1% of actions receive feedback. However, human feedback may not be about the most recent action only, but can potentially be attributed to multiple recent actions. For example, the user may not be certain if to discourage a behavior, such as turning and going in a certain direction, until the trajectory is clear after several steps.

We address this by taking inspiration from the use of eligibility traces for credit assignment [42], and create heuristics to propagate the reward to actions that otherwise receive no reward. If an action is assigned no reward, we propagate to it a reward from a later action by assigning to it the reward of the next action that is assigned one, if one exists within the next eight actions. The only exception is that we do not propagate reward from a STOP action that receives a negative reward.

We additionally prevent noisy feedback from being propagated by considering the scenario dynamics. We remove all training examples for actions following an action that both results in a card selection (or de-selection) and receives a negative reward. If an action with positive reward results in an invalid set (i.e., the selected cards do not match), we remove it from the training data and do not propagate its award. These scenario-specific heuristics were developed by identifying common feedback errors in the data. Following propagation, we observe that 82.2% of the actions have reward assigned to them, up from 63.1, without increasing the reward error rate of about 7.0%. 8

3.3 Parameter Optimization

The complexity of human feedback complicates reward attribution. We adopt a simple approach and maximize the immediate reward (i.e., a discount factor of zero). 9 This creates a contextual bandit scenario. Because of the costs of human interaction and our aim to rapidly improve through few interactions, we emphasize simplicity and use a REINFORCE-style policy gradient objective [47]. At the end of each round, we train from scratch using all the data observed so far ⋃ ρ ρ ′ =0 D ρ ′ . We process the initial human demonstration data D 0 to the same form as the reward data we get from user interactions, so we can use it seamlessly with the data we get from the rounds. For each action a and input instruction x and observation o in D 0 , we create an example ( x, o, a, +1) ; we consider all demonstration data as gold-standard, and set the action reward to +1 .

During round ρ , the gradient for an example ( x, o, a, r ) from dataset D ′ ρ , ρ ′ ≤ ρ is:

where c ( · ) computes a clipped inverse propensity scoring (IPS) coefficient of the policy gradient [17], and θ ρ ′ are parameters with which the examples in D ρ ′ were generated. IPS is commonly used for debiasing policies during off-policy learning. We also use IPS to avoid the problem of exploding gradients while training on negative examples ( r < 0 ) [22]. We clip the value of the IPS coefficient to a maximum of 1 to avoid overfitting on positive examples [12].

4 Experimental Setup

Initialization Data Except when specified otherwise, the demonstration training dataset D 0 includes 8,790 instructions from 456 randomly-sampled human-human interactions from Suhr et al. [41]. Pilots studies showed estimating θ 1 using this data provides a good balance of human-human data amount and initial system performance for interaction with human users.

Model We implement the policy π as a neural network with parameters θ . The network design is based on the design of Suhr et al. [41], except that we account for partial observability. Roughly, the instruction x and observation o are embedded independently. We use instruction-conditioned convolutions on the embedded observation to mix features from both inputs, and a modified LINGUNET [6] generates a distribution over actions. Appendix B provides formal model details.

Deployment In each round of deployment, we collect up to a fixed number of interactions per user. We instruct users to use the reboot button sparingly, and only when they anticipate the follower will make a mistake that cannot be easily corrected. We do not enforce any specific feedback patterns. Users are aware that their instructions and feedback will be used to train the agent for future rounds.

Evaluation The main measure of performance is the agent's instruction execution accuracy during interactions with human users. Because we have no human demonstration or annotation for this data, we perform post-hoc manual evaluation for each round. We randomly sample instructions from each round. We only sample instructions the agent completed (i.e., executed the STOP action) to annotate.

2

4

·

10

4

65

70

75

80

85

θ

1

θ

11

Number of

Observed Instructions

Instruction

Execution Accuracy

3

4

5

6

θ

1

θ

11

Game Score

2

4

·

10

4

0

20

40

60

80

100

Number of

Observed Instructions

Proportion of Actions with

Positive and Negative Feedback

0

5

10

15

Positive : Negative

Feedback Ratio

Weassume all instructions rebooted by the user are failures, and adjust down the accuracy as estimated from the annotations according to the ratio of rebooted instructions. Appendix D formally defines the evaluation metric. We show the instruction, an animation of the agent's execution, and a list of cards that were reached by the follower during execution. We ask whether the follower correctly executed the instruction, and whether the follower made any number of errors in a pre-defined list. For performance measurements and statistics, we report the mean and confidence interval, 10 except where noted. When emphasizing the relationship between the amount of training data and each measure, we plot the metric by the number of observed instructions rather than by round index.

Crowdsourcing We use MTurk for both user-agent interactions and post-hoc instruction evaluation. We recruit workers from English-majority locales using a qualification quiz and tutorial. Appendix E contains crowdsourcing management and compensation details. We maintain novice and expert groups of workers. Expert workers have higher compensation, and access to the post-hoc instruction accuracy tasks. To become an expert, workers must show they understand the task and annotation expectations during interactions in the novice pool. We qualify 108 workers, including 65 expert workers. During deployment, we balance the number of interactions between workers by limiting each worker to about eight interactions per agent and round.

5 Results and Analysis

Weconduct two experiments: an 11-round experiment to observe long-term learning and user behavior trends, and a shorter 5-round experiment to compare several learning design decisions.

5.1 Long-Term Experiment

We evaluate over eleven rounds of deployment and training. This experiment uses the heuristically propagated reward, with the goal of obtaining more training examples compared to the simple reward (Section 3.2). In total, we collect 3,368 games and 46,573 instructions, at a cost of $15,944.45 USD.

Agent Accuracy Figure 2 (left) shows instruction execution accuracy (estimated with 800 instructions per round) and number of points across the eleven rounds. Accuracy improves from 66.7 ± 1.5 in the initial round to 82.1 ± 1.1 in the final round. The continual learning process also results in increased

0

20

40

60

80

100

1

11

Round

The follower completed all the

tasks I instructed them

to do.

0

20

40

60

80

100

1

11

Round

The follower only did things that

I asked them to do, and

no more than that.

0

20

40

60

80

100

1

11

Round

The follower executed my

instructions efficiently, using

the minimum number of

steps required.

Strongly

Disagree

Neutral

Agree

Strongly

Disagree

Agree

Error

ρ

= 1

ρ

= 11

Trend

1

Missed card

Missed a card that should have

been selected.

11.1

6.1

0

12

2

Extra card

Selected a card that should not

have been selected.

11.4

4.6

0

12

3

Wrong stop

Stopped in the wrong position.

4.5

2.1

0

5

4

Wrong turn

Turned the wrong way when told

to go left, right, backward, etc.

2.6

1.3

0

3

5

Wrong order

Accomplished the right tasks, but

in the wrong order.

0.6

0.1

0

1

6

Inefficient

The follower

took

unnecessary,

extra, or redundant steps.

5.8

6.3

0

7

7

Other

1.1

0.4

0

2

interaction success: the game score increases from 3.3 ± 0.2 to 5.3 ± 0.2 over time. Performance begins to plateau after the sixth round, which we hypothesize is due to model performance limitations. Appendix F provides additional evaluation on a static, held-out set of human demonstration data.

Influence of User Adaptation on Performance User adaptation over time is a confounding factor, as improvements in agent performance may be due to users adapting what tasks they delegate to the follower agent, or how they phrase their instructions. We conduct an additional deployment with both the initial agent (with parameters θ 1 ) and the last deployed agent ( θ 11 ). The agents are deployed concurrently in a randomized experiment. Users do not know which agent they are collaborating with. We collect 571 interactions, with a total of 7,784 instructions. Points on the righthand side of Figure 2 (left) show the performance of both models during this deployment. Execution accuracy (estimated with 400 instructions) for the initial model θ 1 improves from an average of 66.7 to 70.4 from user adaptation alone, while the final agent achieves an accuracy rate of 83.6 during this side-by-side comparison. This demonstrates that while roughly 20% of the improvement is due to user adaptation, genuine agent improvement through continual learning is extremely effective.

User Perception of Agent Behavior User perception of the agent improves over time. In-game feedback (Figure 2, right) improves: the rate of positive feedback per action increases from 44.9 to 60.3% and the negative feedback rate decreases from 10.1 to 4.7%. The reboot rate per instruction also decreases from 15.2 to 8.7%. After each interaction, we ask the leader for their agreement with several statements. Figure 3 (left) shows the distribution of these post-interaction Likert ratings, which improve significantly over the 11 rounds. For example, the rate of agreement with The follower completed all the tasks I instructed them to do increases from 27.7 to 59.8% of interactions.

Error Analysis Figure 3 (right) shows trends of seven agent error types across rounds in the long-term experiment. Error types are annotated by workers as part of accuracy estimation. All but one error type decreases significantly over time; the rate of inefficient executions remains stable (Error 6). Manual analysis shows that more than a third of remaining missed-card errors (Error 1) could be attributed to poorly-written instructions, and most extra-card errors (Error 2) occur when the incorrect card shares at least one property with the target, or is closer to the starting position than the target.

Language Change Users are likely to change their language over time, as observed in human-human interactions, including in CEREALBAR [11]. In our study, instruction length remains relatively stable over time at an average of 8.6 tokens per utterance. This contrasts with existing findings in reference games, where convention formation facilitates reduction in utterance length over time [24, 9], and shows that interaction dynamics and incentives shape the dynamics of language change. We observe changes in instruction content over time, using heuristics to estimate the number of target cards specified in each instruction. There is a decrease in the rate of instructions that specify no cards to interact with (i.e., cards to select or deselect) from 12.0 in the first round to 7.7% in the last. The rate of instructions specifying a single card increases from 81.5 to 88.7%, while the rate of instructions specifying multiple cards decreases from 6.5 to 3.7%. This is potentially because no-card instructions are relatively inefficient, while multi-card instructions are relatively hard and have a higher risk of cascading errors that require corrections (i.e., deselecting a wrongly selected card). We also find that the rate of instructions containing a reference to an object (e.g., a tree, a pond) decreases over time from 17.9 to 13.5% of instructions. We find that users increasingly shift labor to the agent over time: the average number of steps per set taken by the follower increased from 14.8 to 15.3, while steps taken by the leader decreased from 10.0 to 8.9 steps. These changes illustrate the impact of users developing game strategies and, potentially, a model of the follower's instruction following ability.

5.2 Comparison of Learning Design Choices

We conduct a second deployment experiment to study the impact of the amount of initial demonstration data, negative feedback, and reward propagation heuristics. We also compare the feedback learning signal to supervised demonstration data. We design and deploy five system variations:

REWARDPROP uses the exactly same process as in the long-term experiment (Section 5.1), including reward propagation (Section 3.2).

SIMPLEREWARD measures the effect densifying and de-noising user feedback with heuristics by using only the simple reward (Section 3.2).

NONEGATIVE simulates using positive feedback only. We discard all negative feedback, including reboots, from the data, and then apply the reward propagation heuristics (Section 3.2). 11

FEWERDEMO uses only 144 human-human interactions (2,114 instructions) for initialization, about 25% of the supervised data used in REWARDPROP.

SUPONLY compares training with human demonstrations vs. feedback data given the same number of interactions. SUPONLY does not use data from human-agent interactions, though we deploy it for evaluation. Instead, after each round, we add a fixed number of additional human-human interactions to the training set, equivalent to the amount of data collected with the other variants. Due to the limited amount of demonstration data, we only deploy this approach for three rounds.

We deploy the five variations for five rounds. In each round, all variations are deployed concurrently, each for 200 user interactions. To ensure that each worker interacts with all agents and to reduce ordering bias, we assign each worker a unique, consistent, and random ordering of agents; as they play multiple games, they cycle through agents in this order. Because all variants except FEWERDEMO are pre-trained on the same initial data, in the first round only, we deploy two agents: one agent for REWARDPROP, SIMPLEREWARD, NONEGATIVE, and SUPONLY, and another for FEWERDEMO. After the first round, we deploy five agents, one for each variation. This experiment cost $16,769.46.

Figure 4 shows results for the five agent variants over five rounds. Overall, learning from feedback is robust to the different design choices. Reward propagation (REWARDPROP) has minor benefit relative to the simple reward (SIMPLEREWARD). REWARDPROP shows slightly faster learning, achieving significantly higher accuracy (80.3 ± 1.0) than SIMPLEREWARD (78.1 ± 1.0) in Round 3, which means SIMPLEREWARD gives a poorer user experience early on, though it catches up quickly. 12

We observe that negative feedback is important: compared to REWARDPROP and SIMPLEREWARD, NONEGATIVE learns slower, receives less positive feedback, and more negative feedback and reboots.

1

2

3

4

5

30

40

50

60

70

80

Round

Instruction

Execution Accuracy

REWARDPROP

SIMPLEREWARD

NONEGATIVE

FEWERDEMO

SUPONLY

1

2

3

4

5

10

20

30

40

50

60

70

Round

Feedback

10

15

20

25

30

Reboot Rate

This difference does not show in early rounds, but appearing around round three. It particularly stands out in round four, when it shows an accuracy of 76.8 ± 1.0, compared to 81.0 ± 0.9 of REWARDPROP.

The experiment illustrates that learning can start with a much weaker agent. FEWERDEMO uses roughly a quarter of the demonstration data available to the other variants, and indeed starts with much lower performance, 34.8 ± 1.8 accuracy, compared to 70.6 ± 1.1 for the others. Even this weak model is sufficient for effective learning: it leads to a much steeper learning curve, improving to 64.2 ± 1.5 accuracy after five rounds. However, this comes at cost to the initial user experience: in the first round, only 2% of interactions with FEWERDEMO were rated positively by users.

Finally, we observe the feedback data is roughly equivalent to supervised data as a learning signal, showing similar accuracy and feedback trends. REWARDPROP achieves equivalent performance to SUPONLY with less than half the amount of demonstration data (i.e., as used for initialization). This shows the benefits of training through user-agent interaction: not only do learning signals directly target current agent behavior, but the training process is significantly less expensive, requiring no human-human demonstrations, but instead just user-system interactions.

6 Related Work

Learning for instruction following commonly relies on various levels of supervision, such as goldstandard demonstrations [e.g., 7, 43, 32, 1, 5, 6, 41, 8, 38], or goal annotations [e.g., 3, 33, 40]. We collapse the distinction between learning and deployment, shifting training and evaluation into human-agent interactions, relying only on a small amount of demonstrations data to initialize.

There has been limited work on continual learning for language-related tasks. A major challenge studying this problem is designing a scenario that allows for scalable non-expert feedback and iteratively improving a model and deploying it. CEREALBAR is unique in supporting real-time interaction with situated natural language between agents and human users. Alternatively, continual learning has been studied via synthetic deployment by simulating feedback using supervised datasets [e.g., 34, 29, 12]. We focus on studies with real human instructors, which create dramatically different dynamics and data cost constraints. For example, this line of work usually uses significant amount of supervised data, while we emphasize rapid learning from limited interactions. Kojima et al. [22] studied the problem of continual learning for instruction generation by observing human following behavior, including deployment in an iterative study. Similar to us, they used CEREALBAR because of

its ability for scalable deployment and real-time model-human interaction. Beyond this environment choice, our work differs significantly: we study human feedback and the task of instruction following.

Human feedback through annotation or selection of intended output has been used for semantic parsing [45, 18]. This is related to soliciting partial annotations [2, 44, 48] or post-interaction feedback [30] in dialogue. Recent work has also employed preference-based user judgments for reward model learning to fine-tune language models using RL [RLHF; 26, 25, 49, 39, 35]. In contrast to this existing work, we focus on sequential execution of instructions in embodied interactions, using realtime feedback provided by human users to continually train our language-understanding agent. While work in this line showed improving a learned model once, it did not study iteratively improving and deploying a model multiple times, a core focus of our work.

Continual learning has been studied more extensively outside of a language context, e.g., for robotics. Our work is inspired by TAMER [20, 21, 46] and COACH [31, 4], where humans provide binary feedback to an agent acting in a world. While these methods were developed in the context of an agent learning a single task, we focus on generalization to previously unseen natural language instructions.

7 Discussion

We propose an approach for continually learning to follow instructions through interaction with human users that provide realtime feedback. We demonstrate its effectiveness through multiple rounds of training and deployment, including thousands of interactions. Experiments with various learning design decisions show our approach is robust, and can start from a weak initial agent.

Our work sets the stage for several avenues of further study. First, future work could reconsider a number of our design choices, including the simple contextual bandit objective and the neural architecture. Application of a user simulator [13] could further improve performance; however, this approach fails to capture dynamics that arise in real interactions through user adaptation. An important direction for future work is incorporating additional observational interaction signals (e.g., by observing the instructor behavior) [10, 22]. Such orthogonal signals can accelerate learning. Another promising avenue for research is more expressive feedback mechanisms, such as through natural language and even complete bi-directional dialogue. Although reasoning about such signals significantly complicates both reasoning and learning, they are also much more informative. A potential direction is to use simple feedback signals like ours to bootstrap the language capabilities of the system, and then gradually switch to more complex natural language feedback.

Acknowledgments

This research was supported by ARO W911NF21-1-0106, NSF under grant No. 1750499, and a gift from Open Philanthropy. Suhr was supported by the NSF under grant No. DGE-2139899, and via a Facebook PhD Fellowship. We thank Noriyuki Kojima for discussions and sharing code, Suyi Diao for contributing to the CEREALBAR implementation, and the participating MTurk workers for their work and help in refining the user experience. We also thank Xiang Lorraine Li and Faeze Brahman, as well as the anonymous reviewers, for providing valuable feedback.

References

Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In The IEEE Conference on Computer Vision and Pattern Recognition , pages 3674-3683, 2018.

Yoav Artzi and Luke Zettlemoyer. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 421-432, 2011. URL http://www.aclweb.org/anthology/D11-1039 .

Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association of Computational Linguistics , 1:49-62, 2013. URL http://aclweb.org/anthology/Q13-1005 .

Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L. Littman. Deep reinforcement learning from policy-dependent human feedback. ArXiv , abs/1902.04257, 2018.

Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, and Yoav Artzi. Following high-level navigation instructions on a simulated quadcopter with imitation learning. In Proceedings of the Robotics: Science and Systems Conference , 2018.

Valts Blukis, Dipendra Misra, Ross A. Knepper, and Yoav Artzi. Mapping navigation instructions to continuous control actions with position visitation prediction. In Proceedings of the Conference on Robot Learning , 2018.

David L. Chen and Raymond J. Mooney. Learning to interpret natural language navigation instructions from observations. In Proceedings of the National Conference on Artificial Intelligence , 2011.

Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Herbert H Clark and Deanna Wilkes-Gibbs. Referring as a collaborative process. Cognition , 22 (1):1-39, 1986.

Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Niekum, and W Bradley Knox. The EMPATHIC framework for task learning from implicit human feedback. In Conference on Robot Learning . PMLR, 2020.

Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi. Analysis of language change in collaborative instruction following. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2803-2811, November 2021. doi: 10.18653/v1/2021. findings-emnlp.239. URL https://aclanthology.org/2021.findings-emnlp. 239 .

Ge Gao, Eunsol Choi, and Yoav Artzi. Simulating bandit learning from user feedback for extractive question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 5167-5179, May 2022. doi: 10.18653/v1/2022.acl-long.355. URL https://aclanthology.org/2022.acl-long.355 .

Kallirroi Georgila, James Henderson, and Oliver Lemon. User simulation for spoken dialogue systems: Learning and evaluation. In Proceedings of Interspeech , 2006. doi: 10.21437/ Interspeech.2006-160.

Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. Continual adaptation for efficient machine communication. In Proceedings of the 24th Conference on Computational Natural Language Learning , pages 408-419, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.conll-1.33. URL https: //aclanthology.org/2020.conll-1.33 .

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9, 1997.

Emiel Hoogeboom, Jorn W.T. Peters, Taco S. Cohen, and Max Welling. HexaConv. In International Conference on Learning Representations , 2018. URL https://openreview. net/forum?id=r1vuQG-CW .

D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association , 47(260):663-685, 1952. ISSN 01621459. URL http://www.jstor.org/stable/2280784 .

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a neural semantic parser from user feedback. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 963-973, 2017. doi: 10.18653/v1/ P17-1089. URL http://aclanthology.coli.uni-saarland.de/pdf/P/P17/ P17-1089.pdf .

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations , 2014.

W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: the TAMER framework. In Proceedings of the International Conference on Knowledge Capture , 2009.

W. Bradley Knox, Peter Stone, and Cynthia Breazeal. Training a robot via human feedback: A case study. In Proceedings of the International Conference on Social Robotics , 2013.

Noriyuki Kojima, Alane Suhr, and Yoav Artzi. Continual learning for grounded instruction generation by observing human following behavior. Transactions of the Association for Computational Linguistics , 9:1303-1319, 2021. doi: 10.1162/tacl_a_00428. URL https://aclanthology.org/2021.tacl-1.77 .

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An interactive 3D environment for visual AI. arXiv , 2017.

Robert M. Krauss and Sidney Weinheimer. Changes in reference phrases as a function of frequency of usage in social interaction: a preliminary study. Psychonomic Science , 1:113-114, 1964.

Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 92-105, June 2018. doi: 10.18653/v1/N18-3012. URL https://aclanthology.org/N18-3012 .

Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 1777-1788, July 2018. doi: 10.18653/v1/P18-1165. URL https://aclanthology.org/P18-1165 .

Akshay Krishnamurthy, Alekh Agarwal, and John Langford. PAC reinforcement learning with rich observations. In Advances in Neural Information Processing Systems , 2016.

John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007 , 2007.

Carolin Lawrence and Stefan Riezler. Improving a neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1820-1830, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1169. URL https://aclanthology.org/P18-1169 .

Bing Liu, Gokhan Tür, Dilek Hakkani-Tür, Pararth Shah, and Larry Heck. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2060-2069, June 2018. doi: 10.18653/v1/N18-1187. URL https://aclanthology.org/N18-1187 .

James MacGlashan, Mark K. Ho, Robert Tyler Loftin, Bei Peng, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In Proceedings of the International Conference on Machine Learning , 2017.

Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 1004-1015, 2017. doi: 10.18653/v1/D17-1106. URL http://aclweb.org/anthology/D17-1106 .

Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 2667-2678, 2018. URL https://www.aclweb.org/anthology/D18-1287 .

Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 1464-1474, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1153. URL https://www.aclweb.org/anthology/D17-1153 .

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=8aHzds2uUyB .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv , abs/1707.06347, 2017.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 1715-1725, August 2016. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/P16-1162 .

Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In IEEE Conference on Computer Vision and Pattern Recognition , June 2020.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 30083021, 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper.pdf .

Alane Suhr, Srinivasan Iyer, and Yoav Artzi. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2238-2249, 2018. URL http://aclweb.org/anthology/N18-1203 .

Alane Suhr, Claudia Yan, Charlotte Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. Executing instructions in situated collaborative interactions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2019. doi: 10.18653/v1/D19-1218. URL https://www.aclweb.org/anthology/D19-1218 .

Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks , 9:1054-1054, 1998.

Stephanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence , 2011.

Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the International Joint Conference on Artificial Intelligence , 2015.

Sida I. Wang, Percy Liang, and Christopher D. Manning. Learning language games through interaction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics , pages 2368-2378, 2016. doi: 10.18653/v1/P16-1224. URL https://www. aclweb.org/anthology/P16-1224 .

Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter R Stone. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the Conference on Artificial Intelligence , 2018.

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8, 1992.

Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. An imitation game for learning semantic parsers from user interaction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 6883-6902, November 2020. doi: 10.18653/v1/2020. emnlp-main.559. URL https://aclanthology.org/2020.emnlp-main.559 .

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019. URL https://arxiv.org/abs/1909.08593 .

A Limitations and Broader Impacts

Our study was conducted with a group of 108 workers, all recruited from English-majority locales, due to the complexity of recruiting and training workers given the complexity of the task. The group size limits the variation in language use we observe. Its composition restricts our ability to evaluate generalization to other languages, an important direction for future work. Another question for further study is the dynamics created when completely new users join the community in later stages.

Because of the complexity of our studies we kept our architecture close to previous work on CEREALBAR, and did not study more contemporary architectures or using pre-trained models. We hypothesize using both could lead to better performance, and more consistent improvement trends. This is an important direction for future work.

Our learning setting involves deploying our agent in interaction with human users, which raises time and interaction considerations not present in RL scenarios without a human in-the-loop. This is especially stressed by working in a complex dynamic system where agent observations are coming from a policy-dependent distribution, as users adapt their language and behavior to the previous interactions they have with the agent. Thus, we prioritize sample efficiency when designing our learning algorithm, and opt the simplicity of the contextual bandit optimization problem, which is known to be sample efficient compared to methods maximizing total reward [28, 27]. This also gives us an objective that is mathematically similar to a supervised learning objective, which is a benefit over more complex RL approaches in terms of stability and predictability in learning. Exploring more complex RL algorithms in this setting, e.g., PPO [36], is a compelling direction for future work.

We do not vary the settings of CEREALBAR. Effenberger et al. [11] find that the interaction design and incentives influence the process of language change. Studying the impact of the scenario design decisions would have significantly complicate our experiments, and increase our costs. We decided not to focus on this research question in this work, but treat these parameters as fixed. Although the analysis of Effenberger et al. [11] shows that CEREALBAR creates interesting and complex language dynamics, further study of the impact of interaction design decisions on continual learning is important, and currently under-studied. Our work does not answer these questions, but we hope it will stimulate further research into them.

The data and models we release are not designed to be directly deployed beyond the CEREALBAR scenario. In general, deployment of continually learning systems requires guardrails and monitoring to avoid various undesired outcomes, including acquiring behaviors that may harm users.

B Model

We implement our policy as a neural network based on the design of Suhr et al. [41]. The inputs are an instruction x and observation o , and the output is a distribution over actions. The policy architecture is composed of several modules that combine to a single network.

Embedding Instructions Weembed the instruction x = ⟨ x 1 , . . . , x n ⟩ of length n with a bidirectional recurrent LSTM [15]. This results in a sequence of hidden states ⟨ h 1 , . . . , h n ⟩ . The embedding of x is the final hidden state of the sequence h n .

Embedding Observations Each agent observation o includes information about the observable environment and the instruction execution so far. The follower agent in CEREALBAR has partial observability. We use a representation similar to that of Suhr et al. [41], but without making the simplifying assumption of full observability. The environment state W is a tensor representing the properties of each position in the environment as embedding indices. The properties represented in W also encode information about the follower's trajectory so far, the presence of obstacles in the environment, and the follower's observability. Due to partial observability, each position's representation is derived from its most recent observation; any information that changes about the world may be outdated in W . We embed W into a dense tensor W ′ .

Fusing Embeddings After independently embedding the instruction and observation into h n and W ′ , we compute a joint representation of both inputs using text-conditioned (i.e., via h n ) convolutions over W ′ .

Transforming the Coordinate System Predicting actions requires interactions between representations of multiple positions. W ′ represents the environment using offset coordinates, which do not precisely represent the structure of hexagonal grid in CEREALBAR. We transform W ′ to axial coordinates [16], and translate and rotate the tensor such that the center position represents the agent's current location, and the agent is facing in a consistent direction. These transformations are not parameterized.

LINGUNET We use LINGUNET [6] to predict the policy distribution over actions π ( · | x, o ; θ ) , with slight modifications to the design of Suhr et al. [41]. For all convolutions, we apply hex-based convolutions with kernels that operate only on voxels within a hex diameter of d around the center voxel, for a kernel size of d . We apply instance normalization to the last LINGUNET layer of the input and text-based convolutions. Finally, we do not perform the final transposed convolution. Instead, we directly predict a distribution over the action space given the output of the transposed convolution.

B.1 Inference

We use ensemble-based inference. Given sets of model parameters θ = ⟨ θ 1 , . . . , θ m ⟩ , we construct a policy π over executable actions using voting: 13

Actions are sampled and executed from π ( · | x, o ; θ ) . Executing an action in the environment results in a observation according to the transition function T . We continue to sample actions until the stop action STOP is sampled, or until the leader manually reboots the follower. The STOP action marks the current instruction as complete, which either results in the follower's turn ending, or it receiving the next instruction to follow.

C Implementation Details

We lowercase and tokenize instructions using BPE [37] with a maximum vocabulary size of 4,096 and a minimum wordtype occurrence of 2. 14 We learn size-64 word embeddings from scratch. We encode instructions with a single-layer LSTM RNN [15] with 128 hidden units. We embed each position's properties into vectors of size 16. We use the same LINGUNET hyperparameters as Suhr et al. [41], and did not perform an additional hyperparameter search.

We use an ensemble size of m = 10. We do not train in ensemble, but train ten separate models and apply ensemble-based inference during deployment. When using reward propagation, we use a maximum distance of 8 for propagating to previous actions that received no feedback. For training, we use a batch size of 16 agent steps, a learning rate of 0.001, and ADAM [19] for optimization. We re-initialize model parameters from scratch at the beginning of each round of parameter optimization. We use a held-out subset of the original CEREALBAR training set as a validation set for early stopping, comprising 5% of the original split. After each epoch, we evaluate model performance using SWSD (Appendix D) on the validation set. We use patience for stopping; if ten epochs have passed since the last epoch where the validation SWSD surpassed the previous global maximum, we terminate the training process and choose the model parameters that maximize validation SWSD. We use a single GeForce RTX 2080 Ti for training each model. Training a single model takes about 28.9 hours on average. We run inference on CPU during deployment.

Comparison of Learning Design Choices In our second experiment comparing different learning design choices, we deploy for five rounds. This number of rounds was chosen because after five rounds in the long-term experiment (Section 5.1), learning trends were clear; this choice balances

experiment cost and insight. If we acquire more than 200 interactions per round because of the crowdsourcing process, we select exactly 200 games for training and analysis by preferring earlier games played by each worker. We discard the other games.

D Evaluation

Instruction Execution Accuracy For each deployed agent, we randomly sample instruction execution traces E e ⊆ E c ⊆ E for manual evaluation. E c contains all instructions marked as complete by the agent, and E contains instructions that were either marked as complete or rebooted. 15 Excluding rebooted instructions from this evaluation creates a biased sample, as reboots nearly always reflect incorrect instruction execution, so we re-adjust accuracy estimates based on reboot rates. We assume all rebooted instructions are incorrect executions. The adjusted correctness rate is:

Static Evaluation Data We also evaluate on the development split from Suhr et al. [41], with success weighted by stopping distance (SWSD). SWSD is computed per instruction execution:

where correct ( x, e ) is user judgment of execution e = ⟨ ( o i , a i , w a i ) ⟩ m i =1 for instruction x .

where e ′ is the trace of the agent's execution of an instruction and e ∗ is the human demonstration. e ′ -1 ≡ e ∗ -1 only if e ′ results in the same set of cards selected as in e ∗ . || e ′ -1 -e ∗ -1 || is the hex distance between stopping positions. SWSD is stricter than simple card-state accuracy [41], as it gives only partial credit to instructions where an execution stops in an incorrect position.

E Crowdsourcing and Data Details

This study received an exemption from the Institutional Review Board of the institution where the research was conducted. Worker identities are anonymized in the data we release.

We qualify workers through a tutorial and a short quiz about the game rules. Workers are also required to reside in an English-majority locale and have a HIT approval rate of over 90% with at least 100 approved HITs during their time on MTurk. The base pay for completing the qualification task is $0.50 USD, and qualified workers receive a $2.00 bonus. We qualify 108 workers. Following Suhr et al. [41], we pay workers bonuses per point earned in each game, increasing the compensation per point as the game score increases. On average across all experiments, each game costs $2.91 and workers are paid an average of $21.85 per hour.

We split workers into two pools: expert and novice. Expert workers earn a 50% higher bonus per game than novice workers. Workers are moved to the expert pool after playing at least two games with a score greater than zero, as long as their rate of giving feedback is greater than 75% of instructions. 16 Workers return to the novice pool if they play for two rounds with a feedback rate of less than 75% of instructions. 65 workers achieve and maintain expert status throughout the experiments. Only expert workers are qualified to provide post-hoc instruction execution judgments, where they are paid $0.07 per judgment of instruction execution. Worker IDs are anonymized in the distribution of interaction data. Figure 5 shows the instructions provided to workers in MTurk, and Figure 6 shows the CEREALBAR interface during interaction.

Agreement For about 20% of instruction execution receiving post-hoc judgments, we acquire judgments from three workers; we find that for only 3.6% of these no consensus is achieved among the three workers, which indicates very high overall agreement.

In thisHIT,you'll play a collaborative game with apartner.

·You will be a Leader and will work with an automated Follower(i.e,not anotherworker).

·You should be able to connect quickly toyour partner.If you spend more thana minute or twowaiting fora partner,pleasecontact us.

·If you are confused about the game,please read theinstructions on the gamepage,or visit thispage.You can also email us at[REDACTEDFOR SUBMISSION].

Check theHIT title toseewhetheryou areinthenoviceorexpertgroup.

HITsinthe expertpool have abouta5o%increaseinbonus amountover to thenovicepool.Seetheinstructionspagefor details and thebonus schemefor novice and expert

workers.

Wewill evaluateworkersintheexpertbatchperiodically tomakesurethey arestill adhering to ourexpert standards,and we will evaluate noviceworkers aftereverybatchtoseeif

theycanmoveintotheexpertpool.

·Some things we will takeinto account for the Leader games:

oYou should move around themap with your partner to collect sets.

oYourcommandsshouldbeunambiguous andeasyforyourpartnertofollow.

oYou should understandwhatmakesavalidset(threecardswithdifferentcolors,shapes,andnumbers)

oYou should be actively giving good/bad feedback to your partner(at least a few signals per instruction).

oYou shouldn't use the Reboot button unless theFollower is performingpoorly.

oGivefeedbackfor over75%ofinstructions(feedbackcanincluderebooting thefollower).

Don't attempt toplay more than one game ata time.Our serverwill only accepta single connectionfromeach workerata time.We appreciate yourfull attention whileplaying

thegame!

Leader

Score

Feedback

Follower'sView:

EnO TURn

0

+

Reboot

---[DoNE] turn around and get thered trianglesnext to

In this turn:

themountain

---[CURRENT] then get the orange card

TimeLeft:60

CAMERA

Moves:5

TurnsLeft:639938

HELP

SendCommand...

SEnO

F Additional Results

F.1 User Perception of Agents for Comparison of Learning Design Choices

Figure 7 shows the Likert distribution for the three post-interaction statements users are asked about for the experiments comparing learning design choices, where we concurrently deployed five systems for five rounds.

F.2 Evaluation on Static Data

Evaluation through human-agent interaction is the main focus of our work. However, we also evaluate instruction-following agents against held-out, static data from Suhr et al. [41]. This evaluation does not take into account how the actual data distribution shifts over the agent's lifetime, because of the dynamics between the agent and human users. Figure 8 shows average SWSD for the models deployed in each round. SWSD begins at 39.7 for the initial model, and peaks at 46.4. This improvement is due entirely to adding training data acquired from human-agent interactions.

0

20

40

60

80

100

1

5

REWARDPROP

The follower completed all the tasks

I instructed them to do.

0

20

40

60

80

100

1

5

SIMPLEREWARD

0

20

40

60

80

100

1

5

NONEGATIVE

0

20

40

60

80

100

1

5

FEWERDEMO

0

20

40

60

80

100

1

5

SUPONLY

0

20

40

60

80

100

1

5

REWARDPROP

The follower only did things that I asked them

to do, and no more than that.

0

20

40

60

80

100

1

5

SIMPLEREWARD

0

20

40

60

80

100

1

5

NONEGATIVE

0

20

40

60

80

100

1

5

FEWERDEMO

0

20

40

60

80

100

1

5

SUPONLY

0

20

40

60

80

100

1

5

REWARDPROP

The follower executed my instructions efficiently,

using the minimum number of steps required.

0

20

40

60

80

100

1

5

SIMPLEREWARD

0

20

40

60

80

100

1

5

NONEGATIVE

0

20

40

60

80

100

1

5

FEWERDEMO

0

20

40

60

80

100

1

5

SUPONLY

1

2

3

4

5

·

4

40

42

44

46

Number of Training Instructions

Development Set SWSD

10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraining of a continuously learning, instruction following agent from feedback provided by users during collaborative interactions. The problem is that humans often give noisy reward and at irregular intervals. The method formulates the learning scenario as a contextual bandit problem and alternates between training and deployment stages. The main issues addressed is the irregular timed reward and credit assignment problem. The method addresses this using heuristics and demonstrated its effectiveness in the CREALBAR environment using human evaluation.\n\nSTRENGTHS:\nThe paper is straightforward to follow. The topic is very relevant. The evaluation and improvement throughout the interaction rounds are significant in many aspects. The experiments conducted are very thorough and complete and have convincing results to highlight the continual learning nature of the agent.\n\nWEAKNESSES:\n- The assumption of feedback is matched with an action or close by actions (heuristics presented in the paper) is somewhat restrictive\n- transitions without rewards are discarded and not fully utilised, quite a waste of resources\n- requires expensive labelling process\n\nQUESTIONS:\n- if I understand correctly, since the method formalizes itself as a contextual bandit problem, maximizing the immediate expected reward, Line 77 says \"Feedback signals are given in realtime, and are not timed to coincide with a specific action\" will not be favorable for training since the corresponding action and reward is not even matching up?\n- isn't the assumption of giving a binary reward and optimizing for the contextual bandit objective amounts to gradient descent/ascent on the right/wrong actions (with noise, since humans give reward quite randomly), which is almost equivalent to supervised learning? If this is the case, I feel like the continual learning part is just iteratively collecting more supervised learning data, which performance improvement is quite obvious\n\nLIMITATIONS:\nLimitations and broader impacts are complete and included in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for online continually training an instruction-following agent based on user realtime feedback gathered for a collaborative game CerealBAR. The agent need to follow the human's  instructions and complete the task. The paper utilizes the contextual bandit learning approach, with  immediate rewards computed from user binary feedback. Through the evaluation of thousands of human-agent interactions, they demonstrate a significant improvement in instruction execution accuracy over time. They also conduct experiments with multiple training scenarios and verify the robustness of their approach.\n\nSTRENGTHS:\n* The authors propose and deploy the methods in the collaborative human game. They collected agent-human interaction data will be helpful for future agent design.\n* It is a pilot work to demonstrate the strength of continual learning from human feedback for instruction-follow agent. The authors shows that after training the agent from human-human dataset, the policy can be greatly improved from the online off-policy learning from human binary feedback.\n* The authors conduct comprehensive online experiment with variants of learning choice and human post-interaction evaluation. The variants of learning choice show the robustness of the framework and the human post-interaction evaluation shows empirical analysis on the agent behavior improved after learning from human.\n\nWEAKNESSES:\n* The framework deign is not very clear. The authors do not mention much the design of the policy model, which transform from human instruction and observation into action (Model part in section 4). This paper mainly discusses about how to define user feedback and how to train with continual learning given the feedback.\n* The framework uses binary signal from the user feedback, which is limited to represent the human satisfaction or feedback. It can be difficult to generalize to other tasks with complex instructions and observations.\n* The framework is relatively simple with REINFORCE algorithm. It would be more convincing if the authors can conduct more experiments on other standard off-policy learning algorithm for the robustness of the framework.\n\nQUESTIONS:\nAlthough the model can achieve 15.4% absolute improvement through continual learning over time from continual learning, it is hard to gain insight of how good the an instruction-following agent is. In order to show benefit of off-policy learning in this framework, have we tried some baseline models for comparison, such as offline RL, due to the difficulty of additional online data collection?\n\nLIMITATIONS:\nThe experiments is difficult to reproduce. Training policy with from human realtime feedback can lead to significantly higher cost compared with offline RL or user simulator, which is widely used in dialogue agent design.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work demonstrated a simple yet effective framework for continual learning in instruction following task utilizing human feedback. Using CEREALBAR as testbed, this work demonstrated the framework in abundant details, and show effectiveness through experiment results. This work also conducts various analysis related to human feedback during and post-hoc the interactions. Finally, various decision choices are compared.\n\nSTRENGTHS:\nThe method is new and well supported by the framework description, the experiment results, and the analysis in the paper. The writing is well organized and clearly written in general. \n\nThis work address embodied ai problem from a relatively new perspective. It demonstrates effectiveness on task performance as well as benefits on reducing annotation effort. This work also provides promising future directions centering around the human-agent dynamics like the form of human feedback, the communication style between human-agent.\n\nWEAKNESSES:\nI don't find any noticeable weakness of this paper.\n\nQUESTIONS:\n1. Figure 3: left, the x axis range indicates number of turk workers right? aren't they 108 in total?\n2. Maybe I missed something, but how do you annotated the demonstration data? Also, since reward propagation is one of the main design choice, does the gold data's feedback distribution similar as those annotated by mturkers?\n3. Do you have any clue on how user adaptation contribute to specific error decreasing? (figure 3, right)\n4. line 271: not sure what \"we deploy a single agent for all but FEWERDEMO in the first round\" means. Can you explain more?\n5. (Figure 4, right)There seems to be a golden ratio between positive and negative feedback emerging. What do you think of that?\n\nLIMITATIONS:\nLimitations are adequately addressed in supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis strong work presents a systems contribution in a fully-fledged system for continual learning from language feedback, in the context of situated human-to-robot instruction following tasks. Using the CerealBar environment (roughly inspired the card game SET, with an embodied flair), this work starts by learning an agent's instruction following policy from a set of offline demonstration data. This agent is then deployed against a crowd of *real human users* who work together with the agent to complete more and more task instances in the CerealBar environment, with humans providing (sporadic) binary feedback in real-time.\n\nAfter each learning round (a fixed number of episodes), the agent policy is updated from the real-time reward data, where the authors define a heuristic to assign the real-time feedback to individual (state, action) pairs taken by the robot; this heuristic is derived from empirical \"delay in human response\" data, as well as from the traditional RL literature for assigning credit to actions that aren't explicitly provided feedback. The agent update is a contextual bandit style update, using a variation of the simple REINFORCE policy gradient update (training effectively a 1-step RL policy).\n\nAcross 11 rounds of updates from real-world human data, the proposed system obtains impressive results, showing a constantly growing trend in both individual instruction execution accuracy, as well as total CerealBar score (with the steepest increase in performance happening across the earlier rounds). The work also has a series of systematic ablations (comparing the binary feedback based approach with \"full supervision\" - showing that the proposed bandit approach can almost match the same performance), as well as qualitative Likert scale results from the real crowd users actually interacting with this system.\n\nSTRENGTHS:\nThis is a well-written and extremely well-executed systems contribution - the task of learning to improve situated agents from language and binary feedback is incredibly timely, and every part of the proposed system is implemented carefully and studied thoroughly. The evaluation is strong (both qualitative and quantiative), and definitely show that not only is the agent able to effectively improve with feedback, but it's able to improve *efficientyly* and actually grow its capabilities over time. I am a huge fan of this work.\n\nSeparately, I think the dedication of this paper to open-sourcing all parts of the pipeline, including the multi-round crowdsourced interaction data is admirable, and will be an incredible resource to the community.\n\nWEAKNESSES:\nIn a paper such as this, I can understand it's very hard (costly + time-consuming + introduces a lot of confounds) to evaluate multiple different learning approaches at the full-scale of the real-world evaluation (though I really appreciate the experiments in section 5.2 that does this for a limited number of rounds). That being said, I would love to look at other agent learning paradigms, and justify the use of the 1-step contextual bandit style reward, vs. a multi-step RL approach that automatically learns how to perform credit assignment given the sporadic binary feedback. Especially using some of the more recent tools in the RL toolbox (e.g., learning a value function/advantage function, PPO-style clipped updates, or even off-policy/offline RL methods).\n\nA general worry I have is that much of this paper hinges off the design of the reward/utility function, which starts from the principled binary feedback provided by users, but is then further processed through a series of heuristics that may or may not really capture what's going on (for example, how the current heuristic labels transitions when there isn't an explicit binary reward tied to that timestep). I would love for the authors to address these choices in a bit more detail.\n\nQUESTIONS:\nWhat motivated the choice of using clipped IPS coefficient for the policy gradient update vs. a PPO-style update (with a learned one-step value function estimator for debiasing); given the choice of using REINFORCE policy gradient anyway for the 1-step update, this would've been an easy addition, and possibly more stable?\n\nI'd be curious to see a more detailed breakdown of the types of language instructions (and *when* in an episode a user usually provides feedback) across the different rounds; are there more complex instructions/abstractions learned over time? Do users intervene differently in different rounds? What are the failure modes?\n\nLIMITATIONS:\nThe paper provides a thorough limitations section (in the appendix) that is well thought out and clearly states weaknesses in the current approach. It would be nice to move some of the punchlines to the main body in the final version of the paper though!", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

comments: SUMMARY:
The authors work on the CEREALBAR setting, where two agents (one human and one computer) cooperate using natural language to achieve a shared goal. Specifically, the authors propose a new setting where the human agent can provide binary feedback to the computer agent. In their new setting, the authors follow the contextual bandit setting and use a REINFORCE-like algorithm to train the agent. Overall, experiments show that the agent achieved an improvement by learning from human signals.

STRENGTHS:
1. The authors explore a setting that involves real-time human feedback, which may be under-explored.
2. The experiments involve significant human labour and may be beneficial to the community if the authors share their data.

WEAKNESSES:
1. Limited contribution. The authors simply applied existing algorithms to their settings.
2. Experiments show little insight. Their experiments demonstrate that the agent improves by receiving more human feedback, which is expected.
3. (Minor) Lack of concrete numbers. Results are mostly shown in figures, and it's difficult to see the real performance numbers.

QUESTIONS:
1. Why do you use a contextual bandit rather than the full RL setting? This is a bit unintuitive since you have a sequential decision-making process.
2. In Figure 3, it seems the performance is the highest at around round 5. Is there a reason for this?
3. Do you plan to release the collected human feedback data? I checked the data folder in the supplementary file but didn't see them.
4. What models are you using? The paper mentions neural networks, but could you please give more details?

LIMITATIONS:
Limitations are discussed in the Appendix.
For example, the authors openly disclose that they did not use a more modern architecture for their agent.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-21