SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Agent Accuracy Figure 2 (left) shows instruction execution accuracy (estimated with 800 instructions per round) and number of points across the eleven rounds. Accuracy improves from  $66.7 \pm 1.5$  in the initial round to  $82.1 \pm 1.1$  in the final round. The continual learning process also results in increased
<sup>10</sup>Confidence intervals are computed using bootstrap sampling.  $n = 10,000$  samples for interaction-level metrics, and  $n = 1,000$  samples for instruction- and action-level metrics.
Figure 3: Left: Distribution over interactions of post-hoc user agreement with three statements about agent performance. Right: Instruction execution errors during 11 rounds for seven common categories. Center columns show prevalence of each error type as a proportion of analyzed instructions, for the initial  $(\rho = 1)$  and final  $(\rho = 11)$  agents; the rightmost column shows trend in prevalence over time.
interaction success: the game score increases from  $3.3 \pm 0.2$  to  $5.3 \pm 0.2$  over time. Performance begins to plateau after the sixth round, which we hypothesize is due to model performance limitations. Appendix F provides additional evaluation on a static, held-out set of human demonstration data.
Influence of User Adaptation on Performance User adaptation over time is a confounding factor, as improvements in agent performance may be due to users adapting what tasks they delegate to the follower agent, or how they phrase their instructions. We conduct an additional deployment with both the initial agent (with parameters  $\theta_{1}$ ) and the last deployed agent  $(\theta_{11})$ . The agents are deployed concurrently in a randomized experiment. Users do not know which agent they are collaborating with. We collect 571 interactions, with a total of 7,784 instructions. Points on the righthand side of Figure 2 (left) show the performance of both models during this deployment. Execution accuracy (estimated with 400 instructions) for the initial model  $\theta_{1}$  improves from an average of 66.7 to 70.4 from user adaptation alone, while the final agent achieves an accuracy rate of 83.6 during this side-by-side comparison. This demonstrates that while roughly  $20\%$  of the improvement is due to user adaptation, genuine agent improvement through continual learning is extremely effective.
User Perception of Agent Behavior User perception of the agent improves over time. In-game feedback (Figure 2, right) improves: the rate of positive feedback per action increases from 44.9 to  $60.3\%$  and the negative feedback rate decreases from 10.1 to  $4.7\%$ . The reboot rate per instruction also decreases from 15.2 to  $8.7\%$ . After each interaction, we ask the leader for their agreement with several statements. Figure 3 (left) shows the distribution of these post-interaction Likert ratings, which improve significantly over the 11 rounds. For example, the rate of agreement with The follower completed all the tasks I instructed them to do increases from 27.7 to  $59.8\%$  of interactions.
Error Analysis Figure 3 (right) shows trends of seven agent error types across rounds in the long-term experiment. Error types are annotated by workers as part of accuracy estimation. All but one error type decreases significantly over time; the rate of inefficient executions remains stable (Error 6). Manual analysis shows that more than a third of remaining missed-card errors (Error 1) could be attributed to poorly-written instructions, and most extra-card errors (Error 2) occur when the incorrect card shares at least one property with the target, or is closer to the starting position than the target.
Language Change Users are likely to change their language over time, as observed in human-human interactions, including in CerealBar [11]. In our study, instruction length remains relatively stable over time at an average of 8.6 tokens per utterance. This contrasts with existing findings in reference games, where convention formation facilitates reduction in utterance length over time [24, 9], and shows that interaction dynamics and incentives shape the dynamics of language change. We observe changes in instruction content over time, using heuristics to estimate the number of target cards specified in each instruction. There is a decrease in the rate of instructions that specify no cards to interact with (i.e., cards to select or deselect) from 12.0 in the first round to  $7.7\%$  in the last. The rate of instructions specifying a single card increases from 81.5 to  $88.7\%$ , while the rate of instructions specifying multiple cards decreases from 6.5 to  $3.7\%$ . This is potentially because no-card instructions are relatively inefficient, while multi-card instructions are relatively hard and have a higher risk of cascading errors that require corrections (i.e., deselecting a wrongly selected card). We also find that the rate of instructions containing a reference to an object (e.g., a tree, a pond) decreases over time from 17.9 to  $13.5\%$  of instructions. We find that users increasingly shift labor to the agent over time: the average number of steps per set taken by the follower increased from 14.8 to 15.3, while steps taken by the leader decreased from 10.0 to 8.9 steps. These changes illustrate the impact of users developing game strategies and, potentially, a model of the follower's instruction following ability.
5.2 Comparison of Learning Design Choices
We conduct a second deployment experiment to study the impact of the amount of initial demonstration data, negative feedback, and reward propagation heuristics. We also compare the feedback learning signal to supervised demonstration data. We design and deploy five system variations:
REWARDPROP uses the exactly same process as in the long-term experiment (Section 5.1), including reward propagation (Section 3.2).
SIMPLEREWARD measures the effect densifying and de-noising user feedback with heuristics by using only the simple reward (Section 3.2).
NONEGATIVE simulates using positive feedback only. We discard all negative feedback, including reboots, from the data, and then apply the reward propagation heuristics (Section 3.2).<sup>11</sup>
FEWERDEMO uses only 144 human-human interactions (2,114 instructions) for initialization, about  $25\%$  of the supervised data used in REWARDPROP.
SUPONLY compares training with human demonstrations vs. feedback data given the same number of interactions. SUPONLY does not use data from human-agent interactions, though we deploy it for evaluation. Instead, after each round, we add a fixed number of additional human-human interactions to the training set, equivalent to the amount of data collected with the other variants. Due to the limited amount of demonstration data, we only deploy this approach for three rounds.
We deploy the five variations for five rounds. In each round, all variations are deployed concurrently, each for 200 user interactions. To ensure that each worker interacts with all agents and to reduce ordering bias, we assign each worker a unique, consistent, and random ordering of agents; as they play multiple games, they cycle through agents in this order. Because all variants except FEWERDEMO are pre-trained on the same initial data, in the first round only, we deploy two agents: one agent for REWARDPROP, SIMPLEREWARD, NONNEGATIVE, and SUPONLY, and another for FEWERDEMO. After the first round, we deploy five agents, one for each variation. This experiment cost $16,769.46.
Figure 4 shows results for the five agent variants over five rounds. Overall, learning from feedback is robust to the different design choices. Reward propagation (REWARDPROP) has minor benefit relative to the simple reward (SIMPLEREWARD). REWARDPROP shows slightly faster learning, achieving significantly higher accuracy  $(80.3 \pm 1.0)$  than SIMPLEREWARD  $(78.1 \pm 1.0)$  in Round 3, which means SIMPLEREWARD gives a poorer user experience early on, though it catches up quickly.[12]
We observe that negative feedback is important: compared to REWARDPROP and SIMPLEREWARD, NONNEGATIVE learns slower, receives less positive feedback, and more negative feedback and reboots.
<sup>11</sup>We do not remove the option to provide negative feedback during games in this experiment. Leaders are not informed that negative feedback will be discarded for training. We leave the direct comparison between positive-only and positive-and-negative user interfaces for future work.
Though initial early experiments showed a more striking promise of using heuristics to densify the reward signal, this experiment shows that densification via heuristics is not as influential as initially thought.
Figure 4: Results for five agent variants over five rounds. Left: mean estimated instruction execution accuracy. Right: rate of actions with positive  $(\nearrow)$  and negative  $(\nearrow)$  feedback, and instruction reboot rate  $(\nearrow)$ . Across five rounds, all variants exhibit increased execution accuracy, decreased reboot rates, and increasingly positive (and decreasingly negative) action-level feedback.
This difference does not show in early rounds, but appearing around round three. It particularly stands out in round four, when it shows an accuracy of  $76.8 \pm 1.0$ , compared to  $81.0 \pm 0.9$  of REWARDPROP.
The experiment illustrates that learning can start with a much weaker agent. FEWERDEMO uses roughly a quarter of the demonstration data available to the other variants, and indeed starts with much lower performance,  $34.8 \pm 1.8$  accuracy, compared to  $70.6 \pm 1.1$  for the others. Even this weak model is sufficient for effective learning: it leads to a much steeper learning curve, improving to  $64.2 \pm 1.5$  accuracy after five rounds. However, this comes at cost to the initial user experience: in the first round, only  $2\%$  of interactions with FEWERDEMO were rated positively by users.
Finally, we observe the feedback data is roughly equivalent to supervised data as a learning signal, showing similar accuracy and feedback trends. REWARDPROP achieves equivalent performance to SUPONLY with less than half the amount of demonstration data (i.e., as used for initialization). This shows the benefits of training through user-agent interaction: not only do learning signals directly target current agent behavior, but the training process is significantly less expensive, requiring no human-human demonstrations, but instead just user-system interactions.
Learning for instruction following commonly relies on various levels of supervision, such as gold-standard demonstrations [e.g., 7, 43, 32, 1, 5, 6, 41, 8, 38], or goal annotations [e.g., 3, 33, 40]. We collapse the distinction between learning and deployment, shifting training and evaluation into human-agent interactions, relying only on a small amount of demonstrations data to initialize.
There has been limited work on continual learning for language-related tasks. A major challenge studying this problem is designing a scenario that allows for scalable non-expert feedback and iteratively improving a model and deploying it. CerealBAR is unique in supporting real-time interaction with situated natural language between agents and human users. Alternatively, continual learning has been studied via synthetic deployment by simulating feedback using supervised datasets [e.g., 34, 29, 12]. We focus on studies with real human instructors, which create dramatically different dynamics and data cost constraints. For example, this line of work usually uses significant amount of supervised data, while we emphasize rapid learning from limited interactions. Kojima et al. [22] studied the problem of continual learning for instruction generation by observing human following behavior, including deployment in an iterative study. Similar to us, they used CerealBAR because of
its ability for scalable deployment and real-time model-human interaction. Beyond this environment choice, our work differs significantly: we study human feedback and the task of instruction following.
Human feedback through annotation or selection of intended output has been used for semantic parsing [45, 18]. This is related to soliciting partial annotations [2, 44, 48] or post-interaction feedback [30] in dialogue. Recent work has also employed preference-based user judgments for reward model learning to fine-tune language models using RL [RLHF; 26, 25, 49, 39, 35]. In contrast to this existing work, we focus on sequential execution of instructions in embodied interactions, using real-time feedback provided by human users to continually train our language-understanding agent. While work in this line showed improving a learned model once, it did not study iteratively improving and deploying a model multiple times, a core focus of our work.
Continual learning has been studied more extensively outside of a language context, e.g., for robotics. Our work is inspired by TAMER [20, 21, 46] and COACH [31, 4], where humans provide binary feedback to an agent acting in a world. While these methods were developed in the context of an agent learning a single task, we focus on generalization to previously unseen natural language instructions.
We propose an approach for continually learning to follow instructions through interaction with human users that provide realtime feedback. We demonstrate its effectiveness through multiple rounds of training and deployment, including thousands of interactions. Experiments with various learning design decisions show our approach is robust, and can start from a weak initial agent.
Our work sets the stage for several avenues of further study. First, future work could reconsider a number of our design choices, including the simple contextual bandit objective and the neural architecture. Application of a user simulator [13] could further improve performance; however, this approach fails to capture dynamics that arise in real interactions through user adaptation. An important direction for future work is incorporating additional observational interaction signals (e.g., by observing the instructor behavior) [10, 22]. Such orthogonal signals can accelerate learning. Another promising avenue for research is more expressive feedback mechanisms, such as through natural language and even complete bi-directional dialogue. Although reasoning about such signals significantly complicates both reasoning and learning, they are also much more informative. A potential direction is to use simple feedback signals like ours to bootstrap the language capabilities of the system, and then gradually switch to more complex natural language feedback.
This research was supported by ARO W911NF21-1-0106, NSF under grant No. 1750499, and a gift from Open Philanthropy. Suhr was supported by the NSF under grant No. DGE-2139899, and via a Facebook PhD Fellowship. We thank Noriyuki Kojima for discussions and sharing code, Suyi Dao for contributing to the CerealBar implementation, and the participating MTurk workers for their work and help in refining the user experience. We also thank Xiang Lorraine Li and Faeze Brahman, as well as the anonymous reviewers, for providing valuable feedback.
[42] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 9:1054-1054, 1998.
[43] Stephanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence, 2011.
[44] Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the International Joint Conference on Artificial Intelligence, 2015.
[45] Sida I. Wang, Percy Liang, and Christopher D. Manning. Learning language games through interaction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2368-2378, 2016. doi: 10.18653/v1/P16-1224. URL https://www.aclweb.org/anthology/P16-1224.
[46] Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter R Stone. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the Conference on Artificial Intelligence, 2018.
[47] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 1992.
[48] Ziyu Yao, Yiqi Tang, Wen-tau Yih, Huan Sun, and Yu Su. An imitation game for learning semantic parsers from user interaction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6883-6902, November 2020. doi: 10.18653/v1/2020.emnlp-main.559. URL https://aclanthology.org/2020.emnlp-main.559.
[49] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. URL https://arxiv.org/abs/1909.08593.
A Limitations and Broader Impacts
Our study was conducted with a group of 108 workers, all recruited from English-majority locales, due to the complexity of recruiting and training workers given the complexity of the task. The group size limits the variation in language use we observe. Its composition restricts our ability to evaluate generalization to other languages, an important direction for future work. Another question for further study is the dynamics created when completely new users join the community in later stages.
Because of the complexity of our studies we kept our architecture close to previous work on CERALBAR, and did not study more contemporary architectures or using pre-trained models. We hypothesize using both could lead to better performance, and more consistent improvement trends. This is an important direction for future work.
Our learning setting involves deploying our agent in interaction with human users, which raises time and interaction considerations not present in RL scenarios without a human in-the-loop. This is especially stressed by working in a complex dynamic system where agent observations are coming from a policy-dependent distribution, as users adapt their language and behavior to the previous interactions they have with the agent. Thus, we prioritize sample efficiency when designing our learning algorithm, and opt the simplicity of the contextual bandit optimization problem, which is known to be sample efficient compared to methods maximizing total reward [28, 27]. This also gives us an objective that is mathematically similar to a supervised learning objective, which is a benefit over more complex RL approaches in terms of stability and predictability in learning. Exploring more complex RL algorithms in this setting, e.g., PPO [36], is a compelling direction for future work.
We do not vary the settings of CerealBar. Effenberger et al. [11] find that the interaction design and incentives influence the process of language change. Studying the impact of the scenario design decisions would have significantly complicate our experiments, and increase our costs. We decided not to focus on this research question in this work, but treat these parameters as fixed. Although the analysis of Effenberger et al. [11] shows that CerealBar creates interesting and complex language dynamics, further study of the impact of interaction design decisions on continual learning is important, and currently under-studied. Our work does not answer these questions, but we hope it will stimulate further research into them.
The data and models we release are not designed to be directly deployed beyond the Cerealbar scenario. In general, deployment of continually learning systems requires guardrails and monitoring to avoid various undesired outcomes, including acquiring behaviors that may harm users.
B Model
We implement our policy as a neural network based on the design of Suhr et al. [41]. The inputs are an instruction  $\overline{x}$  and observation  $o$ , and the output is a distribution over actions. The policy architecture is composed of several modules that combine to a single network.
**Embedding Instructions** We embed the instruction  $\overline{x} = \langle x_1, \ldots, x_n \rangle$  of length  $n$  with a bidirectional recurrent LSTM [15]. This results in a sequence of hidden states  $\langle \mathbf{h}_1, \ldots, \mathbf{h}_n \rangle$ . The embedding of  $\overline{x}$  is the final hidden state of the sequence  $\mathbf{h}_n$ .
**Embedding Observations** Each agent observation  $o$  includes information about the observable environment and the instruction execution so far. The follower agent in CerealBar has partial observability. We use a representation similar to that of Suhr et al. [41], but without making the simplifying assumption of full observability. The environment state  $\mathbf{W}$  is a tensor representing the properties of each position in the environment as embedding indices. The properties represented in  $\mathbf{W}$  also encode information about the follower's trajectory so far, the presence of obstacles in the environment, and the follower's observability. Due to partial observability, each position's representation is derived from its most recent observation; any information that changes about the world may be outdated in  $\mathbf{W}$ . We embed  $\mathbf{W}$  into a dense tensor  $\mathbf{W}'$ .
Fusing Embeddings After independently embedding the instruction and observation into  $\mathbf{h}_n$  and  $\mathbf{W}'$ , we compute a joint representation of both inputs using text-conditioned (i.e., via  $\mathbf{h}_n$ ) convolutions over  $\mathbf{W}'$ .
Transforming the Coordinate System Predicting actions requires interactions between representations of multiple positions.  $\mathbf{W}^{\prime}$  represents the environment using offset coordinates, which do not precisely represent the structure of hexagonal grid in CerealBAR. We transform  $\mathbf{W}^{\prime}$  to axial coordinates [16], and translate and rotate the tensor such that the center position represents the agent's current location, and the agent is facing in a consistent direction. These transformations are not parameterized.
LINGUNET We use LINGUNET [6] to predict the policy distribution over actions  $\pi (\cdot \mid \overline{x},o;\theta)$  with slight modifications to the design of Suhr et al. [41]. For all convolutions, we apply hex-based convolutions with kernels that operate only on voxels within a hex diameter of  $d$  around the center voxel, for a kernel size of  $d$ . We apply instance normalization to the last LINGUNET layer of the input and text-based convolutions. Finally, we do not perform the final transposed convolution. Instead, we directly predict a distribution over the action space given the output of the transposed convolution.
B.1 Inference
We use ensemble-based inference. Given sets of model parameters  $\theta = \langle \theta_1, \dots, \theta_m \rangle$ , we construct a policy  $\pi$  over executable actions using voting:
$$
\pi (a \mid \bar {x}, o; \theta) \propto \tag {3}
$$
$$
\exp \left(\sum_ {1 \leq i \leq m} \mathbb {1} _ {a = \arg \max \pi (\cdot | \overline {{x}}, o; \theta_ {i})}\right) .
$$
Actions are sampled and executed from  $\pi (\cdot \mid \overline{x},o;\theta)$ . Executing an action in the environment results in a observation according to the transition function  $\mathcal{T}$ . We continue to sample actions until the stop action STOP is sampled, or until the leader manually reboots the follower. The STOP action marks the current instruction as complete, which either results in the follower's turn ending, or it receiving the next instruction to follow.
We lowercase and tokenize instructions using BPE [37] with a maximum vocabulary size of 4,096 and a minimum wordtype occurrence of 2. $^{14}$  We learn size-64 word embeddings from scratch. We encode instructions with a single-layer LSTM RNN [15] with 128 hidden units. We embed each position's properties into vectors of size 16. We use the same LINGUNET hyperparameters as Suhr et al. [41], and did not perform an additional hyperparameter search.