SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alane Suhr
University of California, Berkeley* suhr@berkeley.edu
Yoav Artzi
Cornell University
yoav@cs.cornell.edu
We propose and deploy an approach to continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating  $15.4\%$  absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.
The dynamics that arise in situated language interactions between human users and automated agents expose a plethora of language learning signals. A prime example of such a signal is explicit feedback: when users convey their intent via natural language instructions for an agent to follow, they are well positioned to provide feedback, for example, through a stream of binary signals as the agent acts.
Learning from this type of signal has significant potential. It shifts the burden of learning from annotated data to learning through interaction with users, which not only reduces data costs, but also enables continual<sup>2</sup> improvement through interaction with users. This signal also fundamentally differs from gold-standard annotated data: it directly targets the current agent behavior, rather than reflecting optimal human behavior that may be of less relevance to the agent's current policy. This approach stands in contrast with most methods for learning to follow instructions, where the use of demonstration data entails high data costs, and the clear separation between training and deployment passes over opportunities for learning through interaction with users [e.g., 3, 1, 38].
In this paper, we study the problem of continually<sup>2</sup> improving an automated instruction-following agent<sup>3</sup> by learning from user feedback in human-agent interactions. We situate the interaction in a collaborative environment, which provides conditions necessary for this type of learning: users are incentivized to improve the agent's capabilities by providing feedback, and are continuously present as the agent executes their instructions. We use the CerealBar scenario [41], where two participants collaborate towards a common goal in a shared world, using natural language instructions to coordinate their actions. During interaction, human users write instructions for an agent, and provide feedback to the agent as it executes them via a stream of binary signals. Figure 1 illustrates
*Work done while at Cornell University.
2We use the term continual learning to refer to a learning setting where an agent continually improves its task performance [14], in our case following instructions. The term is used at times to describe the domain adaptation challenge of continually learning to handle new tasks. Our work does not address this problem.
3 Throughout the paper, we use agent to refer to an automated instruction-following system.
Figure 1: Illustration of our continual learning process. The process progresses in rounds  $\rho$ , each including deployment and training. In deployment, we sample and execute actions  $a$  from a policy  $\pi$  parameterized by  $\theta_{\rho}$ , conditioned on user-written instruction  $\overline{x}$  and agent observation  $o$ . Concurrently, users provide binary feedback as they observe instruction execution. We convert this feedback to rewards, and estimate parameters  $\theta_{\rho + 1}$  of next round's policy with a contextual bandit objective.
the setup and our learning process. To the best of our knowledge, our work is the first to demonstrate continual learning from human feedback for natural language instruction following.
A key challenge is the complexity of the learning signal. Users provide feedback at their discretion and under time pressure as the agent executes their instructions, resulting in an unpredictable, and often noisy, learning signal. While the concurrency of agent actions and user feedback provides a weak alignment, delays in human response and lack of clarity about which action or actions they critique make it challenging to attribute a learning signal to specific actions. Further complicating learning is the feedback loop created by the continually changing agent behavior and constant adaptation by human users. These factors create a scenario that is particularly difficult for complex methods as they often make various assumptions about the learning signal and the interaction dynamics [20, 31]. We opt for a simple approach with minimal assumptions, and formulate learning as a contextual bandit scenario. We alternate between deployment, where the agent interacts with users that provide feedback, and training, where we compute immediate rewards from feedback signals, and weigh examples using the current policy to account for the continually changing agent behavior.
We experiment with our approach through interactions with humans users, where we repeatedly deploy, train, and re-deploy the agent. We study the effects of human adaptation and process design decisions on the efficacy of learning. We design our experiments to tease apart genuine agent improvements from human user adaptation, and observe dramatic improvements in agent behavior beyond what user adaptation explains,  $15.4\%$  absolute improvement in instruction execution accuracy in our longer running experiment. Our code and data is released under the MIT license 2023. $^{4}$
2 Technical Overview
Interaction Scenario We situate user-agent interactions in CerealBAR [41], a collaborative scenario where two participants, a leader and a follower, collect sets of matching cards together in a shared, 3D environment. The two participants collaborate towards a shared goal, act in the world, and coordinate using natural language. The leader plans what the pair should do, acts according to the plan, and describes the follower's part of the plan using natural language instructions. The follower's role is to follow the instructions. The pair receives a point for each set of cards they successfully collect. Their goal is to maximize their score. We add to the original CerealBAR setting the ability for the leader to provide binary feedback signals to the follower as they execute instructions. In our study, the agent controls the follower and the leader is always a human user. We use CerealBAR as it is the only research environment designed to support the real-time collaborative interaction scenario we study: alternatives (e.g., Thor [23]) are not designed to deploy at scale for model-user interaction studies, and would require extensive modifications to support multi-agent collaborative interactions.
Task The agent's task is to map natural language instructions and observations to follower actions. Actions include moving FORWARD or BACKWARD, turning LEFT or RIGHT, and completing instruction execution with STOP. The agent observation is a partial view of the world state from the
4Our code and data is available here: https://github.com/lil-lab/clif_cb. The original CREALBAR game and data [41] is licensed under MIT License 2022.
first-person perspective of the follower. Formally, given an instruction  $\overline{x}$  and an initial observation  $o_1$ , our goal is to generate a sequence  $\langle (o_1, a_1), \ldots, (o_m, a_m) \rangle$ , where  $o_i$  an agent observation,  $a_i$  is the action taken, and  $a_m = \text{STOP}$ .
Inference and Learning We define the follower agent as a policy  $\pi (\overline{x},o;\theta)$  parameterized by  $\theta$  that maps instruction  $\overline{x}$  and state observations  $o$  to a distribution over actions. During interaction with users, for each user-written instruction  $\overline{x}$ , we sample and execute actions  $a_{i}\sim \pi (\cdot |\overline{x},o_{i};\theta)$  until we sample the instruction completion action. Concurrently, the user may provide positive and negative feedback signals  $f$ . Feedback signals are given in realtime, and are not timed to coincide with a specific action. Between any two agent steps, there could be any number of feedback signals.
We optimize parameters through rounds of continual learning (Figure 1). At each round  $\rho$ , we deploy the policy with parameters  $\theta_{\rho}$  to interact with users, and then estimate parameters  $\theta_{\rho + 1}$ . During deployment, we collect agent instruction executions and user feedback. The execution of an instruction  $\overline{x}$  during interaction creates a trace made of two sequences  $(\langle (o_i, a_i, w_i^a) \rangle_{i=1}^m, \langle (f_j, w_j^f) \rangle_{j=1}^n)$ , where each action  $a_i$  and feedback signal  $f_j$  are paired with wall times  $w_i^a$  and  $w_j^f$ . We compute reward from the feedback to construct training examples  $(\overline{x}, o, a, r)$ , where  $r$  is a reward. We optimize policy parameters by solving a contextual bandit problem, maximizing immediate expected reward.
Evaluation Our main metric is instruction execution accuracy for user-written instructions. Because we only have access to agent instruction executions, and not ground-truth demonstrations, we cannot compute accuracy automatically. We use crowdsourcing to acquire human judgments of follower accuracy after each round of deployment. We also evaluate trends in user feedback and ratings.
3 Continual Learning
We estimate the policy parameters from user feedback during interaction with users. The process, illustrated in Algorithm 1, progresses in rounds. Each round  $\rho$  includes: (a) deploying the agent policy parameterized by  $\theta_{\rho}$  to interact with users (Section 3.1), (b) computing rewards from user feedback to construct training data  $\mathcal{D}_{\rho}$  (Section 3.2), and (c) optimizing the policy parameters using all data observed so far  $\bigcup_{\rho' = 0}^{\rho} \mathcal{D}_{\rho'}$  to compute  $\theta_{\rho + 1}$  (Section 3.3). We initialize the process with a policy parameterized by  $\theta_{1}$  estimated on human demonstration data  $\mathcal{D}_0$  (Algorithm 1, Line 1).
3.1 Deployment Interactions
During deployment in round  $\rho$ , users collaborate with the agent, and delegate to it tasks by typing natural language instructions. For each instruction  $\overline{x}$ , we sample a sequence of actions from the policy  $a_{i} \sim \pi(\cdot \mid \overline{x}, o_{i}; \theta_{\rho})$  (Lines 9-15). Executing an action in the environment results in a new observation  $o_{i+1}$  (Line 11). For each action, we record the wall time  $w_{i}^{a}$  as the time when the user starts seeing the follower starting to execute the action (i.e., the action animation starts) (Line 12). We terminate instruction execution when the stop action STOP is sampled (Line 15).
At any point in time during the agent's instruction execution, the user may provide binary (positive or negative) feedback signals, creating a stream  $\langle (f_j, w_j^f) \rangle_{j=1}^n$  where  $w_j^f$  is the user's wall time when a feedback button is pressed (Line 13). We also allow the user to reboot the follower at any point during an instruction execution (omitted from Algorithm 1). This is meant to terminate very bad executions before they damage the interaction, for example by leading the follower too far astray. Rebooting adds a negative feedback signal to the end of the stream, and removes all queued instructions.<sup>7</sup>
The execution of an instruction by the agent and the feedback given by the user are combined together to create a trace made of two sequences  $\langle (o_i, a_i, w_i^a) \rangle_{i=1}^m$  and  $\langle f_j, w_j^f) \rangle_{j=1}^n$ .
<sup>5</sup>Suhr et al. [41] provide complete formal notation for CerealBar. We use a simplified notation to focus on the follower instruction following task only.
$^6$ Our policy takes as input a single instruction. Considering interaction history, including previous instructions, is an important direction for future work.
7 Cereal Bar allows the leader to queue multiple instructions; the follower only sees the current instruction.
Algorithm 1 Continual learning for instruction following from realtime user feedback.
Input Human demonstration data  $\mathcal{D}_0$  ; number of interactions per round  $T$  ; dataset construction process convert_feedback_to Reward that maps from an instruction and feedback-annotated execution trace to a set of reward-annotated training examples (Section 3.2); learner train_policy that optimizes a contextual bandit policy gradient objective (Section 3.3).   
1:  $\theta_{1}\gets$  train_policy(Do) Initialize policy parameters with human demonstration data.   
2: for round  $\rho = 1,\dots$  do   
3:  $\mathcal{D}_{\rho}\gets \{\}$    
4: for  $T$  interactions do Deploy the current policy for  $T$  interactions.   
5: repeat   
6:  $\overline{x},o_1\gets$  observe instruction from user and current world state   
7:  $\overline{a},\overline{F}\leftarrow \langle \rangle$  Initialize empty action and user feedback sequences.   
8:  $i\gets 1$    
9: repeat Generate actions for instruction.   
10:  $a_i\sim \pi (\cdot |\overline{x},o_i;\theta_\rho)$  Sample action from policy.   
11:  $\begin{array}{r}\left\{ \begin{array}{ll}o_{i + 1}\sim T(o_i,a_i)\\ \overline{f}_i\gets \mathrm{observed eafback} \end{array} \right. \end{array}$  Concurrently: execute action and observe feedback.   
12:  $\overline{a}\gets \overline{a} +\langle (o_i,a_i,\mathrm{walltime})(\rangle)\rangle$    
13:  $\overline{F}\gets \overline{F} +\overline{f}_i$  Collect user feedback given during action execution.   
14:  $i\gets i + 1$    
15: until  $a_{i - 1} = \mathrm{STOP}$    
16:  $\mathcal{D}_{\rho}\gets \mathcal{D}_{\rho}\cup$  convert_feedback_to Reward  $(\overline{x},\overline{a},\overline{F})$    
17: until interaction terminates   
18:  $\theta_{\rho +1}\gets$  train_policy(  $\bigcup_{\rho^{\prime} = 0}^{p}\mathcal{D}_{\rho^{\prime}}$  Train policy for next deployment using all data.
We use all traces collected in round  $\rho$  to construct a training dataset  $\mathcal{D}_{\rho}$ . Each example in  $\mathcal{D}_{\rho}$  is a tuple  $(\overline{x}, o, a, r)$ , where  $\overline{x}$  is an instruction written by the user in round  $\rho$ ,  $o$  is the agent observation when action  $a$  was selected during the execution  $\overline{x}$  in the interaction, and  $r$  is a numerical reward we compute from the feedback stream. For each instruction written during round  $\rho$  we may create no examples if no feedback was given, or multiple examples if computing the rewards from feedback resulted in attributing rewards to multiple actions.
Simple Reward We consider the value of each positive feedback as  $f = +1$  and each negative feedback as  $f = -1$ . We compute the reward  $r_i$  for an action  $a_i$  as the sign of sum of feedback signals that occur between it and the next action after correcting for human response delay time [31]:
$$
r _ {i} = \operatorname {s g n} \sum \left\{f _ {j} \mid w _ {i} ^ {a} <   w _ {j} ^ {f} - d \leq w _ {i + 1} ^ {a} \right\}, \tag {1}
$$
where  $w_{i}^{a}$  is the wall time of action  $a_{i}$ ,  $w_{j}^{f}$  is the wall time of feedback signal  $f_{j}$ , and  $d$  is a human response delay constant ( $d = 0.2$  seconds). If the set is empty with no feedback signals within the time window, or if  $r_{i} = 0$ , we do not set a reward or create an example for the corresponding action.
Heuristically Propagated Reward As users are not required to follow any specific feedback schedule, the simple reward computation is likely to result in relatively sparse reward assignment, with many actions not receiving any learning signal and thus essentially thrown out as training data. Indeed, in our experiments, we observe that roughly only  $63.1\%$  of actions receive feedback. However, human feedback may not be about the most recent action only, but can potentially be attributed to multiple recent actions. For example, the user may not be certain if to discourage a behavior, such as turning and going in a certain direction, until the trajectory is clear after several steps.
We address this by taking inspiration from the use of eligibility traces for credit assignment [42], and create heuristics to propagate the reward to actions that otherwise receive no reward. If an action is assigned no reward, we propagate to it a reward from a later action by assigning to it the reward of the next action that is assigned one, if one exists within the next eight actions. The only exception is that we do not propagate reward from a STOP action that receives a negative reward.
We additionally prevent noisy feedback from being propagated by considering the scenario dynamics. We remove all training examples for actions following an action that both results in a card selection (or de-selection) and receives a negative reward. If an action with positive reward results in an invalid set (i.e., the selected cards do not match), we remove it from the training data and do not propagate its award. These scenario-specific heuristics were developed by identifying common feedback errors in the data. Following propagation, we observe that  $82.2\%$  of the actions have reward assigned to them, up from 63.1, without increasing the reward error rate of about  $7.0\%$ .
3.3 Parameter Optimization
The complexity of human feedback complicates reward attribution. We adopt a simple approach and maximize the immediate reward (i.e., a discount factor of zero). This creates a contextual bandit scenario. Because of the costs of human interaction and our aim to rapidly improve through few interactions, we emphasize simplicity and use a REINFORCE-style policy gradient objective [47]. At the end of each round, we train from scratch using all the data observed so far  $\bigcup_{\rho' = 0}^{\rho} \mathcal{D}_{\rho'}$ . We process the initial human demonstration data  $\mathcal{D}_0$  to the same form as the reward data we get from user interactions, so we can use it seamlessly with the data we get from the rounds. For each action  $a$  and input instruction  $\overline{x}$  and observation  $o$  in  $\mathcal{D}_0$ , we create an example  $(\overline{x}, o, a, +1)$ ; we consider all demonstration data as gold-standard, and set the action reward to  $+1$ .
During round  $\rho$ , the gradient for an example  $(\overline{x}, o, a, r)$  from dataset  $\mathcal{D}_{\rho}'$ ,  $\rho' \leq \rho$  is:
$$
\Delta = c \left(a, \bar {x}, o, \rho^ {\prime}\right) r \nabla \log \pi (a \mid \bar {x}, o; \theta) \tag {2}
$$
$$
c (a, \overline {{x}}, o, \rho^ {\prime}) = \left\{ \begin{array}{l l} \min  \left(1, \frac {\pi   (a   |   \overline {{x}} , o ; \theta)}{\pi   (a   |   \overline {{x}} , o ; \theta_ {\rho^ {\prime}})}\right) & \rho^ {\prime} > 0 \\ 1 & \rho^ {\prime} = 0    , \end{array} \right.
$$
where  $c(\cdot)$  computes a clipped inverse propensity scoring (IPS) coefficient of the policy gradient [17], and  $\theta_{\rho'}$  are parameters with which the examples in  $\mathcal{D}_{\rho'}$  were generated. IPS is commonly used for debiasing policies during off-policy learning. We also use IPS to avoid the problem of exploding gradients while training on negative examples ( $r < 0$ ) [22]. We clip the value of the IPS coefficient to a maximum of 1 to avoid overfitting on positive examples [12].
Initialization Data Except when specified otherwise, the demonstration training dataset  $\mathcal{D}_0$  includes 8,790 instructions from 456 randomly-sampled human-human interactions from Suhr et al. [41]. Pilots studies showed estimating  $\theta_{1}$  using this data provides a good balance of human-human data amount and initial system performance for interaction with human users.
Model We implement the policy  $\pi$  as a neural network with parameters  $\theta$ . The network design is based on the design of Suhr et al. [41], except that we account for partial observability. Roughly, the instruction  $\overline{x}$  and observation  $o$  are embedded independently. We use instruction-conditioned convolutions on the embedded observation to mix features from both inputs, and a modified LINGUNET [6] generates a distribution over actions. Appendix B provides formal model details.
Deployment In each round of deployment, we collect up to a fixed number of interactions per user. We instruct users to use the reboot button sparingly, and only when they anticipate the follower will make a mistake that cannot be easily corrected. We do not enforce any specific feedback patterns. Users are aware that their instructions and feedback will be used to train the agent for future rounds.
Evaluation The main measure of performance is the agent's instruction execution accuracy during interactions with human users. Because we have no human demonstration or annotation for this data, we perform post-hoc manual evaluation for each round. We randomly sample instructions from each round. We only sample instructions the agent completed (i.e., executed the STOP action) to annotate.
${}^{8}$  Error rate determined by manually annotating actions and feedback from 100 randomly-sampled instructions.
<sup>9</sup>We consider reward propagation (Section 3.2) to be part of the mapping of feedback to reward, so not considered as discounting in the learning objective.
Figure 2: Left: Mean estimated instruction execution accuracy and game scores across 11 rounds. The  $x$ -axis shows the number of instructions observed. We mark with  $\times$  the accuracies and scores from the post-hoc study of user adaptation for  $\theta_{1}$  and  $\theta_{11}$ . Dotted lines illustrate the share of the improvement due to user adaptation. Right: Proportion of actions receiving positive and negative feedback from users over 11 rounds, and the ratio of frequency between positive and negative feedback. The  $x$ -axis shows the number of instructions observed. In both plots, dashed lines show round boundaries.
We assume all instructions rebooted by the user are failures, and adjust down the accuracy as estimated from the annotations according to the ratio of rebooted instructions. Appendix D formally defines the evaluation metric. We show the instruction, an animation of the agent's execution, and a list of cards that were reached by the follower during execution. We ask whether the follower correctly executed the instruction, and whether the follower made any number of errors in a pre-defined list. For performance measurements and statistics, we report the mean and confidence interval, $^{10}$  except where noted. When emphasizing the relationship between the amount of training data and each measure, we plot the metric by the number of observed instructions rather than by round index.
Crowdsourcing We use MTurk for both user-agent interactions and post-hoc instruction evaluation. We recruit workers from English-majority locales using a qualification quiz and tutorial. Appendix E contains crowdsourcing management and compensation details. We maintain novice and expert groups of workers. Expert workers have higher compensation, and access to the post-hoc instruction accuracy tasks. To become an expert, workers must show they understand the task and annotation expectations during interactions in the novice pool. We qualify 108 workers, including 65 expert workers. During deployment, we balance the number of interactions between workers by limiting each worker to about eight interactions per agent and round.
We conduct two experiments: an 11-round experiment to observe long-term learning and user behavior trends, and a shorter 5-round experiment to compare several learning design decisions.
5.1 Long-Term Experiment
We evaluate over eleven rounds of deployment and training. This experiment uses the heuristically propagated reward, with the goal of obtaining more training examples compared to the simple reward (Section 3.2). In total, we collect 3,368 games and 46,573 instructions, at a cost of $15,944.45 USD.