Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models
Year: 2023

Abstract:
Recent work studies the cognitive capabilities of language models through psychological tests designed for humans. While these studies are helpful for understanding the general capabilities of these models, there is no guarantee that a model possessing sufficient capabilities to pass those tests would actually use those capabilities in performing real-life tasks. In this work, we formulate task-oriented cognitive capabilities, which are human-like cognitive capabilities that language models leverage to perform tasks. These capabilities are (i) the ability to quickly generate good candidate utterances (the search capability) (ii) the ability to predict how a listener interprets those utterances and choose the most appropriate one (the pragmatic capability). We design an evaluation scheme for comparing these capabilities of a language model with those of a human. Applying this scheme to examine various models in a navigation instruction generation problem, we find that their pragmatic capability is severely lacking. This insight leads us to augment them with better models of the listener and obtain a significant boost of 11% in success rate in guiding real humans. Our work advocates for having a principled procedure for aligning language models with humans that involves (i) formulating task-oriented capabilities, (ii) devising a method to quantify their deficiency, and (iii) iteratively improving them.

heading: 1 Introduction
text: To communicate successfully with humans, language models must possess cognitive capabilities similar to those that facilitate human communication. Examining the cognitive capabilities of language models is notoriously challenging because the operations of these models are largely unintelligible to humans. Psychologists faced similar challenges when investigating human cognition, and have devised various behavioral tests
∗The first two authors contribute equally.
to diagnose human cognitive capabilities (Premack and Woodruff, 1978; Wimmer and Perner, 1983; Baron-Cohen et al., 1985; Gopnik and Astington, 1988). Recent work (Sap et al., 2022; Kosinski, 2023; Ullman, 2023) applies these tests to evaluate large language models by inputting the tests to these models as prompts and verifying whether they behave like a normal human would.
While this approach is helpful for understanding the general limitations of language models, it has two potential drawbacks. First, it is applicable to only large language models that can comprehend human-written prompts, entangling linguistic capability with reasoning capability. Second, it shows that a language model can or cannot demonstrate certain mental skills, but does not imply that the model would employ those skills to perform a downstream task. For example, passing false-belief tests does not guarantee that a model will reason about the interpretation of the readers when generating summaries. In general, scoring high on psychological tests may not be sufficient to ensure language models would behave like humans in real-life scenarios.
In this work, we take a different approach to evaluating the cognitive capabilities of language models. We define and evaluate task-oriented cognitive capabilities, which are human-like capabilities that a model actually employs to perform the task it is designed for. Enhancing these capabilities thus warrants improved performance on the task. To identify these capabilities, we build on two lines of work from socio-cognitive science: Bayesian models of cooperative communication (Wang et al., 2020; Goodman and Frank, 2016; Shafto et al., 2014) and studies on how humans implement Bayesian reasoning (Sanborn and Chater, 2016; Sanborn et al., 2010; Vul et al., 2014; Mamassian et al., 2002). We propose a mathematical cognitive model called bounded pragmatic speaker, which can reasonably characterize the rea-
3688
soning processes of both humans and language models. Casting humans and language models in the same way enables us to juxtapose their cognitive capabilities. We mathematically formulate two capabilities that a bounded pragmatic agent must possess in order to generate optimally pragmatic utterances. These conditions correspond to wellknown cognitive capabilities of humans: (i) the ability to efficiently generate relevant utterances (the search capability) (Bloom and Fischler, 1980; Gold et al., 2000; Trosborg, 2010) and (ii) the ability to accurately simulate the listener’s interpretations of their utterances (the pragmatic capability) (Premack and Woodruff, 1978; Gopnik and Astington, 1988; Tomasello, 2019; Call and Tomasello, 2011; Frank and Goodman, 2012). We design a simple procedure to quantitatively evaluate these capabilities of a language model. To evaluate each capability, we compute the task performance gap between the model and an oracle model, which is identical except that the evaluated capability of this model is at human level. Figure 1 illustrates our procedure, which theoretically can be applied to any language model.
We evaluate various language models on a navigation instruction generation problem (Anderson et al., 2018b), where a model generates English instructions to guide real humans in photo-realistic 3D environments.1 Our evaluation reveals an interesting finding: all evaluated agents possess relatively efficient search capability but inadequate pragmatic capability. We improve the pragmatic capability of the evaluated models by enabling them to reason probabilistically about human listeners (Andreas and Klein, 2016; Fried et al., 2018a), employing state-of-the-art instructionfollowing agents (Magalhaes et al., 2019; Shen et al., 2022; Hong et al., 2021) as models of human listeners. We obtain significant improvement in success rate over the original agents, shrinking the gap with human performance on held-out data by 36%. Towards eliminating the remaining gap, we illustrate with empirical evidence a major challenge in developing better listener models. Specifically, when the instruction-following agents are employed as listener models for the instruction-generating agent, they are required to evaluate AI-generated instructions, which may be significantly different from human-generated
1Our human-evaluation dataset and interface are publicly released at https://lingjunzhao.github.io/coop_ instruction.html.
instructions. Hence, a standard supervised-learning training scheme that only exposes these models to human-generated instructions would be inadequate for learning reliable models. We thus call for construction of novel datasets, algorithms, and evaluation methods for developing the pragmatic capability of language models.

heading: 2 Related Work
text: Navigation Instruction Generation. Instruction generation has been commonly studied in navigation settings (Anderson et al., 1991; Byron et al., 2009; Koller et al., 2010; Striegnitz et al., 2011; Goeddel and Olson, 2012; Fried et al., 2018a,b). The Matterport3D simulator and the accompanying datasets (R2R (Anderson et al., 2018b), R4R (Jain et al., 2019), and RxR (Ku et al., 2020)) offer more challenging settings by combining photo-realistic
scenes with long, verbally rich instructions. Recent work on evaluating instruction generation agents (Zhao et al., 2021) reveals the ineffectiveness of standard learning and modeling approaches to this problem. Wang et al. (2021) improve the accuracy and interpretability of instructions in the RxR setting. Kamath et al. (2023) leverage this model to synthesize additional data for training instructionfollowing agents. Our work aim to offer useful principles to further improve these models.
Mathematical Models of Human Communication. Different from communication within agents (Lazaridou et al., 2020; Roman Roman et al., 2020), human communication is a cooperative act (Grice, 1975; Scott-Phillips, 2014; Tomasello, 2019). Pragmatic communication in humans may involve different cognitive capabilities like basic understanding of language and social rules (Trosborg, 2010) and reasoning about the physical world (Bender and Koller, 2020) and human behavior (Enrici et al., 2019; Rubio-Fernandez, 2021). Our work describes similar capabilities but provides a formal mathematical description. Development of mathematical models of human communication have been greatly useful for understanding human behaviors (Ho et al., 2016; Sumers et al., 2022) and building communication agents (Andreas and Klein, 2016; Fried et al., 2018a,b; FAIR, 2022; Lin et al., 2022; Zhu et al., 2021; Bao et al., 2022). Wang et al. (2020) unify these models under an optimal-transport framework. The model we propose in this work is a generalized version capturing the essence of these models.
Evaluating Cognitive Capabilities of Neural Networks. Many benchmarks for evaluating the cognitive capabilities of AI-based agents have been created, focusing on theory-of-mind capabilities (Le et al., 2019; Nematzadeh et al., 2018), grounding (Lachmy et al., 2022; Udagawa and Aizawa, 2019; Haber et al., 2019), or commonsense reasoning (Talmor et al., 2019; Levesque et al., 2012; Zellers et al., 2019; Sap et al., 2019). Large language models have demonstrated exceptional performance on following human instructions and solving complex reasoning tasks (Bubeck et al., 2023; Anil et al., 2023), raising the question of whether their cognitive capabilities are similar or as advanced as those of humans. Mahowald et al. (2023) advocate for separating formal competence (knowledge about linguistic rules and patterns) from their func-
tional competence (knowledge about the world usage in the world) when assessing these models. Our bounded pragmatic speaker framework mathematically formalizes this description, allowing for quantitative evaluation of these competencies. Recent work (Sap et al., 2022; Kosinski, 2023; Ullman, 2023; Hu et al., 2023) examines cognitive capabilities of large language models through tests inspired by human psychological tests. The goal of these studies is to determine the limits of large language models, potentially calibrating the expectation on them. On the other hand, our focus is to devise a method that can be applied to language models of any size and benchmark cognitive capabilities that are relevant for accomplishing a specific task.

heading: 3 Problem Setting
text: We are concerned with instruction generation: learning a speaker agent r that generates language instructions to guide a human listener h to reach states in an environment.
Human Listener. We imagine a human listener h acting in a partially observed environment with states s. The human does not have access to states but only observations oh and takes actions ah. An instruction u ∈ U is a language utterance consisting of words. A trajectory e = (s1, o1, a1, · · · , sT , oT , aT ) is an execution of an instruction. The human can follow instructions to generate trajectories in the environment. For example, in an indoor navigation setting, upon hearing “go the kitchen and stop next to the oven”, a human walks to the specified location. We define Lh(e | u) as the probability that the human generates e upon hearing u.
Speaker Agent. In each task, the speaker agent first imagines an intended trajectory e⋆ = (s1, o r 1, a r 1, · · · , sT , orT , arT ), which specifies a path to get to an intended goal state sT from the human’s current state s1. Because the human’s actions and perception may differ from those of the speaker, they may not be able to comprehend e⋆ even if it is presented to them. Thus, the speaker needs to translate the trajectory into an instruction û that the human can understand and follow. To do so, it implements a language model Sr(u | e), and an inference algorithm Gen(Sr, e) to craft instructions based on the model (e.g., greedy or beam-search decoding). The speaker’s objective is to generate instructions that maximize the expected chance of
the listener reconstructing the intended trajectories
argmax Sr
Ee⋆ [Lh(e⋆ | Gen(Sr, e⋆))] (1)
Evaluation. The speaker is evaluated using a dataset Deval of held-out trajectories. For each trajectory e⋆k ∈ Deval, we generate an instruction ûk = GEN(Sr, e⋆k). The instruction is then presented to a (real) human listener to follow, producing a trajectory ehk ∼ Lh(· | ûk). The performance of the speaker, denoted by ρ(r), is the average similarity, Ψ, between the human-generated and the intended trajectories:
ρ(r) ≜ 1|Deval| ∑
e⋆k∈Deval Ψ(ehk, e ⋆ k) (2)
We will specify the choices for the metric Ψ in the experimental setup section (§6).

heading: 4 Task-Oriented Cognitive Capabilities
text: Human have evolved highly effective cognitive capabilities for communication. How can we endow a speaker agent with similar capabilities and quantify the degree of resemblance between its capabilities and those of a human?
We propose a mathematical framework that reasonably characterizes the human cognitive process for instruction generation (§4.1). We show that this model can also describe the operation of language models, which allows us to compare them with humans on specific cognitive capabilities. We identify two capabilities that are requisite for any agent implementing our framework to generate optimal instructions (§ 4.2), and introduce an evaluation scheme for collating these capabilities (§4.3).

heading: 4.1 A Mathematical Cognitive Model of Instruction Generation
text: To formulate how humans generate instructions, we build on mathematical models of cooperative communication (Wang et al., 2020; Goodman and Frank, 2016; Shafto et al., 2014). We consider a general version where a speaker agent constructs a pragmatic speaker model Sprag(u | e) based on two constituents: a base speaker model Sbase(u | e) and a theory-of-mind (ToM) listener model LToM(e | u). The base speaker represents general knowledge of the agent about the world and the language it speaks. The ToM listener reflects situated knowledge about the listener, simulating how they would behave in the environment given an instruction. A pragmatic speaker aims to
maximize the chance of the listener interpreting its instruction correctly, but it is still influenced by its general knowledge (e.g., social biases, language style). Formally, it is defined as:
Sprag(u | e) ∝ LToM(e | u)Sbase(u | e) (3) To convey an intended trajectory e⋆, this speaker utters an instruction of maximum probability under its model:
ûprag ≜ argmax u∈U Sprag(u | e⋆)
= argmax u∈U
LToM(e ⋆ | u)Sbase(u | e⋆) (4)
Humans as bounded pragmatic speakers. The pragmatic speaker model accounts for human behaviors highly accurately on problems where U is a small discrete space (Frank and Goodman, 2012). However, in problems like instruction generation where U is an unbounded set of linguistic expressions, it is unlikely that humans, which are known to be agents with bounded rationality (Simon, 1957), are able to compute the optimal utterance in Eq 4 exactly. A hypothesis, supported by empirical evidence, is that humans perform approximate inference via Monte-Carlo sampling (Sanborn and Chater, 2016; Sanborn et al., 2010; Vul et al., 2014; Mamassian et al., 2002). Applying this hypothesis to our setting, we derive a more practical model of how human generate instructions, in which they perform the search for the best utterance on only a subspace Usub of U defined by a set of candidates sampled from Sbase
ûbounded-prag ≜ argmax u ∈ Usub ⊂ U LToM(e ⋆ | u) (5) where Usub = {ui ∼ Sbase(· | e⋆) | 1 ≤ i ≤ N}. We call an agent that generates instructions according to Eq 5 a bounded pragmatic speaker (Figure 2). For such a speaker, instruction generation involves two tasks: candidate generation (performed by Sbase) and candidate evaluation (performed by LToM). The former task ensures that the generation of an instruction is efficient, while the latter guarantees the generated instruction conveys the intended meaning to the human listener.

heading: 4.2 Formulating Task-Oriented Cognitive Capabilities
text: What cognitive capabilities enable humans to generate effective instructions? Viewing humans as bounded pragmatic speakers allows us to mathematically characterize those capabilities. Specifi-
cally, we require a bounded pragmatic speaker to be able to output the optimal utterance, i.e. satisfying
ûbounded-prag = u ⋆ ≜ argmax
u Lh(e
⋆ | u) (6)
where Lh is the human listener.
For this equation to hold, the constituent models Sbase and LToM of the bounded pragmatic speaker must meet certain conditions. The condition for Sbase is that the candidate set it generates must contain the optimal instruction, i.e. u⋆ ∈ Usub. This condition requires Sbase to be capable of quickly generating candidates and placing high probability on u⋆ so the instruction can be found by sampling a few candidates. We refer to this capability as the search capability.
Meanwhile, the condition for LToM is that it has to rank u⋆ first among the candidates in Usub. Meeting this condition demands having the capability of constructing a mental emulation of the human listener and simulating the actions of the listener after receiving an instruction. We refer to this capability as the pragmatic capability.
The search and pragmatic capabilities are orthogonal and complementary. An agent with flawless pragmatic capability can evaluate the goodness of instructions given to it, but may not be able to efficiently generate good instructions by itself. In contrast, an agent with effective search capability can quickly bring to attention highly relevant utterances but cannot select the best one to output if its ToM model is erroneous.

heading: 4.3 Evaluating Task-Oriented Cognitive Capabilities
text: We have defined two cognitive capabilities that are requisite for humans in instruction generation. In this section, we will prove that a language model can also be cast as a bounded pragmatic speaker. Hence, we can compare it with a human on the two cognitive capabilities.
Language models as bounded pragmatic speakers. We consider a speaker agent r that learns a language model Sr(u | e) and runs an inference algorithm to compute an output ûinfer = GEN(Sr, e⋆) ≈ argmaxu∈U Sr(u | e⋆). Generative LSTM- or Transformer-based models that implement greedy or beam-search decoding are examples of this agent. We make the following assumption about the inference algorithm.2
Assumption (Better-than-sampling inference algorithm). We assume the inference algorithm is better at finding argmaxu∈U Sr(u | e⋆) than drawing a small number of N samples from Sr. Formally, let γ be the probability of drawing e⋆ and a set of N instructions from Sr such that Sr(ûinfer | e⋆) > maxu∈Usub Sr(u | e⋆), where ûinfer = GEN(Sr, e⋆). We assume that γ is large for a small integer N > 0.
2We empirically verify that this assumption holds for the agents we evaluate with N = 10 and γ ranging from 0.7 to 0.9. We estimate γ by computing the fraction of evaluation examples where the agent’s model ranks ûinfer above N samples drawn from it.
If this assumption holds, then with high probability, the agent r behaves identically to a bounded pragmatic speaker that computes its output as:
û ≜ argmax u∈Ursub Sr(u | e⋆) (7) Ursub ≜ {ûinfer} ∪ {ui ∼ Sr | 1 ≤ i ≤ N} (8) This agent uses Sr as both the base speaker Sbase and ToM listener LToM. Due to our assumption, on most inputs, the agent outputs ûinfer, similar to the original agent. We employ this bounded pragmatic speaker as the proxy for the original agent in comparisons with humans, and also refer to it as r.
Evaluation scheme. To evaluate a cognitive capability (search or pragmatic) of a speaker r, we compute the performance gap between it and an oracle agent that is at human level on the evaluated capability, but is equally good as it is at the other capability. Specifically, we define r⋆search to be an oracle speaker that employs Sr as the ToM model but is given a “gold” candidate set U⋆cand that always contains a human-generated reference instruction u⋆. It selects its output as follows
u⋆search ≜ argmax u∈U⋆cand Sr(u | e⋆) (9)
This agent has similar pragmatic capability as r but human-level search capability. Next, we construct r⋆pragmatic, an oracle that generates candidates using Sr but employs a human Lh to rank the candidates
u⋆pragmatic ≜ argmax u∈Ursub Lh(e ⋆ | u) (10)
with Ursub from Eq 8. The search capability of r⋆pragmatic is as good as r but its pragmatic capability is that of a human.
We calculate the prospective performance gain (PPG) with respect to each capability as follows
PPGsearch(r) ≜ ρ(r⋆search)− ρ(r) (11) PPGpragmatic(r) ≜ ρ(r⋆pragmatic)− ρ(r) (12)
where ρ is the performance on held-out data (Eq 2 of § 3). Each metric computes the potential improvement if the corresponding capability is upgraded to match with that of a human. Comparing the two metrics reveals which of the two capabilities of r is currently more deficient and informs future development direction for the agent. For example, if PPGsearch(r) is large and PPGpragmatic(r) is small, it means that r is scoring the candidate instructions highly accurately but it is bad at finding high-score instructions. In this case, developers
may want to focus on devising a more effective inference algorithm to improve the search capability of r. On the other hand, if r estimates poorly calibrated scores, signified by PPGpragmatic(r) being large, enhancing its inference algorithm is fruitless, but endowing it with a module that simulates the listener’s behavior more accurately would boost its performance.

heading: 5 Improving Pragmatic Capability with
text: Ensemble Instruction-Following Agents
In cases where our evaluation scheme indicates that the pragmatic capability of a language model is deficient, we improve it by installing a better ToM listener model. A common approach to learning this listener model is to use the same dataset used for learning the speaker model (Andreas and Klein, 2016; Fried et al., 2018a,b). We argue that this approach has a potential drawback. A ToM model learned in this way is only exposed to humangenerated input instructions. At deployment time, it would likely experience a covariate shift because as a ToM model, the model is then asked to score instructions generated by a speaker model, not by humans. These instructions may be incorrect, ungrammatical, or may simply have a different style than human-generated instructions. This covariate shift would hamper the model’s judgement. Our preliminary experiments (Appendix §A.6) confirm that using a listener trained on only human-generated inputs as the ToM model hurts rather than improves the performance of various speakers.
We show that this problem can be alleviated by employing ToM models that have calibrated uncertainty on unseen instructions. We obtain calibrated models through ensembling (Lakshminarayanan et al., 2017): we train listener models L̂(k)(e | u), k = 1 . . .K, each on a random 90% subset of the training data with different random initial seeds.
We also leverage access to a simulation of the environment to construct better ToM models. Note that the probability that a ToM model LToM assigns to an instruction can be seen as an expectation of a binary metric: LToM(e⋆ | u) = Ee∼LToM(·|u) [1{e = e⋆}], which does not award credit if e overlaps only partially with e⋆. We propose two augmentations: (i) replace the binary metric with a soft metric Ψ(e, e⋆) that can measure partial similarity between trajectories and (ii) approximate the expectation by executing listeners L̂(k) in the simulated environment to sample trajec-
tories. Our final model selects its instruction as:
ûaugment-ToM ≜ argmax u∈Ursub LToM(e ⋆ | u) (13)
LToM(e ⋆ | u) ∝ 1
KM
K∑
k=1
M∑
j=1
Ψ(ej(L̂ (k),u), e⋆)
Ursub ≜ {ûinfer} ∪ {ui ∼ Sr | 1 ≤ i ≤ N} where e(L,u) denotes a trajectory sampled from a listener model L conditioned on an instruction u, and M is the number of trajectories we sample from each listener. Essentially, the score LToM(e
⋆ | u) of each candidate instruction is the average performance metric of K listeners, each of which attempts to follow the instruction M times.

heading: 6 Experimental Setup
text: Environment and Dataset. We employ Matterport3D (Anderson et al., 2018b), a photo-realistic simulator of the visual perception of a person walking in an indoor environment. At any location, an agent is provided with RGB images capturing the 360-degree panoramic view when looking from that location.
We train and evaluate our models using the Room-to-Room (R2R) language-based navigation dataset. Each data point was collected by asking an English-speaking crowd-worker to write a verbal description of a path in an environment. The dataset is split into a training set (61 environments, 4,675 paths), a seen validation set (environments seen during training, 340 paths), and an unseen validation set (11 environments unseen during training, 783 paths). We train the models using the training set and perform model selection on the unseen validation set. Performance metrics are computed on the seen validation set.
Speaker Models. We evaluate three speaker architectures: (1) a decoder-only GPT-2 pre-trained on text (Radford et al., 2019); (2) an LSTM encoder-decoder (Shen et al., 2022); (3) a Transformer encoder-decoder (Vaswani et al., 2017). Parameters of the latter two models are randomly initialized. Details are in Appendix §A.2.
Human Evaluation. We evaluate each speaker model on 75 paths in the seen validation data split. In the end, we have annotated 1,200 instructions generated by 16 different systems (humans, 3 speaker models, and their ablated and augmented versions). To evaluate a speaker model, we present its generated instructions to a human annotator
and ask them to follow the instructions to navigate in Matterport3D environments. We adapt the PanGEA tool3 to setup a web navigation interface and create a task on Amazon Mechanical Turk (MTurk). We recruit 213 human evaluators in total. More details about the setting are given in Appendix §A.5.
Performance Metrics. The quality of a speaker is determined by the similarity between the intended trajectories and the actual trajectories that the human evaluators generate by following the speaker’s instructions. We compute these similarity metrics: Success rate (SR) averages binary indicators of whether the final location of a humangenerated trajectory is within three meters of the final location of the intended trajectory; SPL (Anderson et al., 2018a) weights the success indicator with the ratio between the intended traveling distance and the actual one; and NDTW and SDTW are metrics based on dynamic time-warping alignment (Magalhaes et al., 2019), capturing the similarity between two point sequences. NDTW computes only a sequence similarity score while SDTW weights the score with the success indicator.

heading: 7 Experiments
text: We investigate the following questions: (a) How well do the speakers perform on our
problem? We find that, despite implementing advanced architectures, these speakers perform poorly compared to human speakers. (b) What causes their performance deficiency? Using our evaluation scheme, we identify that the speakers possess decent search capability but inadequate pragmatic capability. (c) Can we improve the speakers by equipping them with better ToM listeners? We employ ensembles of state-of-the-art instructionfollowing agents as ToM listeners for the speakers, and obtain significant improvements. (d) What are the challenges in bridging the performance gap with human speakers? We show that instruction-following agents trained with only human-generated instructions are not optimal for serving as ToM listener models.
How well do the speakers perform on our problem? As seen in Figure 3, there is a wide margin between the agent speakers and the human
3https://github.com/google-research/pangea
instructors. The best model speaker (EncDecTransformer) lags behind the humans by 21.6 NDTW points. The encoder-decoder architecture with cross-attention of EncDec-Transformer outperforms the decoder-only self-attention architecture of GPT-2 (+11.7 NDTW), indicating that fusing the vision and language features too early in an architecture may be detrimental. On the other hand, EncDec-Transformer leads over EncDec-LSTM by 4.1 points NDTW, suggesting that the Transformer architecture is more effective than LSTM in this problem.
What causes the speakers’ deficiency? Next, we investigate whether the lack of search or pragmatic capability is responsible for the deficiency of the speakers. The prospective performance gains presented in Figure 4 show that it is under-performed pragmatic capability that primarily causes the models to perform poorly. Specifically, while equipping the models with oracle search capability only improves their performance by 9.4% on average, granting them oracle pragmatic capability nearly doubles their performance metrics. In fact, the search capability of the models is already as good as that of the humans we employ, because the models with oracle pragmatic capabil-
ity achieve even slightly higher NDTW scores than the human speakers.
Can we improve the speakers by equipping them with better ToM models? Following the procedure described in Section §5, we train state-of-theart instruction-following agents to serve as ToM listener models for the speakers. Performances of different combinations of speakers and listeners are given in Table 1. We see the largest improvement (+7.9 NDTW) over the best base speaker (EncDecTransformer) by augmenting this speaker with an ensemble of 10 EnvDrop-CLIP listeners as the ToM model. In Figure 5, we show a qualitative example where having a ToM listener enables the speaker to generate a more accurate instruction. More examples are shown in Appendix §A.7.
We observe that ensemble models consistently outperform single models. More results about the effectiveness of ensemble listeners compared to single listeners are given in Appendix §A.6.
What are the challenges in bridging the performance gap with human speakers? Despite the promising improvements, there remains a large gap of 17.9 NDTW points between our best speaker and the human speakers. As suggested by Figure 4, this gap can be closed by developing accurate ToM models. We argue that optimal ToM models cannot be simply obtained by learning optimal instructionfollowing agents, because the latter is learned to execute human-generated instructions while the former is asked to rank model-generated instructions. To illustrate the difference, we measure the agreement between human and model listeners on instructions generated by different speakers. We define the agreement score between a human Lh and a model L̂ as
Agreement(Lh, L̂)
= Averageu∈Deval (NDTW(eh(u), ê(u))) (14)
where eh(u) and ê(u) are the trajectories generated by Lh and L̂ given u, respectively, and Deval denotes the R2R seen validation set.
As seen in Table 2, the listener agents agree more with the humans on human-generated instructions than on model-generated ones. The results imply even an optimal instruction-following agent can fail to improve a base speaker in the presence of an input distribution mismatch. We thus advocate for developing ToM models that are robust or can adapt quickly against covariate shift, and for evaluating
performance of these models on model-generated instructions.

heading: 8 Conclusion
text: This work introduces a framework for analyzing task-oriented cognitive capabilities of instructiongeneration language models. We show that insights from the analysis are helpful in directing development on these models. Our results highlight the necessity of constructing better ToM models for improving these models. We argue that learning accurate ToM listener models is met with novel, distinct challenges. We hope that our findings will motivate the community to focus more on evaluating task-oriented cognitive capabilities and to
create datasets, training methods, and evaluation procedures for enhancing the pragmatic capability of language models.
Limitations
Our work is predicated on hypothetical models of human cognition. These models are still under development by cognitive scientists and need to be validated in more realistic domains. Our method assumes access to a simulation of the environment, which may be costly to construct in some domains.
In general, instruction generation agents pose substantial risk to humans. Previous studies have shown that humans can become overly reliant on AI instructions and commit disastrous mistakes
(Robinette et al., 2016). It is thus important for practitioners to comprehend the constraints of our experimental setting. Our experiments take place in a coarse simulator of real-world indoor environments, which restricts the action and perception of the human listeners. Due to the expensive cost and the large number of agent variants, our human evaluation remains limited in terms of population scale and diversity, and the comprehensiveness of the questionnaires. As each instruction is only evaluated by a single human, we have not investigated the variance of the interpretation of the same instruction among different humans. In addition, human evaluators may “guess” a path even if a part of the instruction is misleading or impossible to follow. Hence, the path-similarity metrics may not reflect faithfully the quality of the instructions. Nevertheless, results shown in Table 4 of §A.5 indicates that instructions generated by our agents are almost as easy to interpret as those generated by humans. But again, these results are still subject to the constraints of our annotator population. To deploy our method, practitioners should carefully re-evaluate its safety and effectiveness in conditions that closely emulate the deployment conditions.

heading: Acknowledgments
text: We thank Jieyu Zhao, Trista Cao and our reviewers for providing helpful feedback to improve the manuscript. We are also grateful to Patrick Shafto and Ted Sumers for great discussions and references to mathematical models of human cognition.

heading: A Appendices
text: A.1 The Room-to-Room dataset
The R2R dataset (Anderson et al., 2018b) was originally created for training instruction-following agents. Each data point was collected by asking a crowd-worker to write a verbal description of a path in an environment. In the end, each path was annotated with three instructions. Each instruction contains 29 words on average. The dataset is split into a training set (61 environments, 4,675 paths), a seen validation set (340 paths) whose paths are sampled in the training environments, and an unseen validation set (11 environments unseen during training, 783 paths). We do not use the unseen test split because it does not provide ground-truth paths of the descriptions. We use the dataset consistent to their MIT License.
A.2 Implementation of Speaker Models
We train the speakers with a standard maximumlikelihood objective using the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of 10−4.
The speaker models take a sequence of visual observations and actions from the trajectory e⋆ as input and output a text instruction u. The model is trained to estimate conditional probability Sθ(u|e⋆). We use grid search to select the model and training hyperparameters, and the best-found
values are listed in Table 3.
Input. The input trajectory e⋆ is a sequence of panoramic views and actions. Each panoramic view at time step t is represented by 36 vectors {ot,i}36i=1, each of which is a visual feature vector extracted from a pre-trained vision model concatenated with orientation features describing the agent’s current gaze direction. The image features of the GPT-2 model are extracted from a ResNet152 model (He et al., 2016), whereas those of the encoder-decoder models are from a CLIP model (Radford et al., 2021). Each ground truth action a⋆t , which moves the agent to an adjacent location, is represented by image features from the gaze direction of the agent when looking towards that adjacent location, and orientation features capturing the direction of the adjacent location relative to the agent’s current gaze direction.
Output. The output of a speaker model is a language instruction describing the input trajectory. At test time, the GPT-2 model employs beam search, and the encoder-decoder models generate instructions via greedy decoding (Shen et al., 2022).
Training Objective. We train the speakers with maximum-likelihood objective:
max θ
∑
(u⋆,e⋆)∈Dtrain
|u⋆|∑
t=1
logSθ(u ⋆ t | e⋆,u⋆<t) (15)
where θ is the speaker model parameters, u⋆t is t-th word of the ground-truth instruction, and u⋆<t is the first t− 1 words of the instruction.
We select the best model based on the unseenvalidation BLEU score (Papineni et al., 2002) of the model-generated instructions with the respect to the ground-truth instructions.
Tools. We use SacreBLEU 2.2.1 to compute BLEU scores. For preprocessing and implementing the speaker models, we use Pytorch 1.7.1, NLTK 3.6.7, SentencePiece 0.1.97, and Huggingface Transformers 4.5.1.
Computation. The GPT-2 model has 124.4 million parameters, and was trained for 24 hours on single NVIDIA GEFORCE RTX 2080 Ti. The EncDec-LSTM model has 7.5 million parameters, taking 24 hours to train on single NVIDIA RTX A6000. The EncDec-Transformer model has 56.6 million parameters, trained on single NVIDIA RTX A6000 for 48 hours.
A.3 Fine-tuning GPT-2 Speaker Model
To represent the trajectory features as a sequence of feature vectors to feed into the GPT-2 model, we first average the view features ōt for each time step:
ōt = 1
36
36∑
i=1
ot,i (16)
We compute the input features e⋆t by concatenating the panoramic view features and ground truth action features:
e⋆t = [ōt; a ⋆ t ] (17)
The sequence of feature vectors e⋆ representing a trajectory is calculated as follows
e⋆ = [tanh(e⋆1W ); · · · ; tanh(e⋆TW )] (18) where W is parameters of a linear layer.
For the instruction u⋆, we perform an embedding look-up of its words. Then, we first prompt the model with e⋆ and then train it to generate u⋆ as a suffix.
A.4 Training Encoder-Decoder Speaker Models
Our EncDec-LSTM model follows the implementation of the speaker in Shen et al. (2022). We implement the EncDec-Transformer model by replacing the LSTM layers of the speaker model described in Tan et al. (2019) with Transformer layers (Vaswani et al., 2017).
A.5 Human Evaluation Interface and Data Collection
We pay the evaluator $5.20 per task which takes about 25 minutes, and the payment is decided by state minimum wage. For each task, we ask the evaluator to follow six instruction-following sessions. One of the six sessions, which appears in all tasks, is a quality-control test featuring an easy-tofollow human-written instruction. We only approve an evaluator if they navigate successfully to the goal destination in this test. Following Zhao et al. (2021), we instruct the judges to not explore the environments unnecessarily and not wander back and forth unless they are lost. We record the trajectories created by the human and use them to compute the performance metrics.
Figure 6 shows the interface for our human evaluation to collect annotations, which we adapted from the PanGEA tool4 consistent with their Apache License v2.0. After a human evaluator finishes following an instruction, we recorded the path they generate and compute similarity metrics with respect to the ground-truth path. After the instructionfollowing sessions, we ask each evaluator to assess the interpretability of the instructions by asking them how easy (or difficult) it was for them to follow the instruction. We provide four rating levels ranging from “1: I couldn’t follow any part of the instruction” to “4: very easy, the instructions gave
4https://github.com/google-research/pangea
accurate and sufficient information for me to follow”. The answer of the evaluators is converted to a score between one and four.
Table 4 shows the human evaluation results of the three speaker models we evaluated.
For the human evaluation survey, participants will be restricted to those fluent in English. There are no other restrictions for this study. Participants
must be at least 18 years old. Before completing the survey, participants will be shown information about the task requirement: You are in a building, and are provided with a short set of instructions to navigate to a target location. Please follow the instructions as closely as possible. Do NOT explore the building unnecessarily and do NOT wander back and forth unless you are lost. Please read ALL of the instructions before you start moving.
We waive consent for this study for several reasons: 1) Minimal risk: The study collects minimal identifying information and there are no known risks for the subjects beyond everyday computer use. 2) Rights and welfare: All participants will be shown all information regarding task requirements before they complete our survey. They must consent to performing the task before they are shown the questions. 3) Practicality: Since the sessions are conducted online on a large scale, it would be infeasible to require all users to send a signed form. 4) Post participation information: We do not think there is any pertinent information that is not already shared with the participants before or during our experiments, so we do not feel it is necessary to provide any additional information after participation. PI information will be shared with the participants to enable them to obtain additional information about the study post completion.
For data anonymization, we removed the only identifying information, Amazon Mechanical Turk ID, after collecting the human annotation data. This information would also be removed for future dataset release. The dataset will be released under MIT license terms, which are compatible with those of the tools used to create it, and will be intended for research usage.
A.6 Single vs. Ensemble Listeners As a preliminary experiment, we compare the effectiveness of a single and an ensemble of 10 VLN⟳ BERT agents when serving as the ToM model of a speaker. Results in Figure 7 show that the ensemble listener is significantly better than the single listener for two different speakers.
A.7 Qualitative Examples In Figure 8, we show additional qualitative examples where having a ToM listener enables the speaker to generate a more accurate instruction.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Limitations.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract, and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 6, Section 7, Section A.1, A.5.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1, Section 6, Section A.1.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section A.1, Section A.5.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section A.1, Section A.5, Section Limitations.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section A.5.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 1, Section A.5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section A.1, Section 6.
C 3 Did you run computational experiments? Section 6.2, Section A.2.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 6.2, Section A.2.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 7.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section A.2.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 6, Section A.5.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section A.5

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 6, Section A.5.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section A.5

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section A.5.

heading: Future Work
text: This information would also be removed for future dataset release. The dataset will be released under MIT license terms, which are compatible with those of the tools used to create it, and will be intended for research usage. A. 6 Single vs. Ensemble Listeners As a preliminary experiment, we compare the effectiveness of a single and an ensemble of 10 VLN⟳ BERT agents when serving as the ToM model of a speaker. Results in Figure 7 show that the ensemble listener is significantly better than the single listener for two different speakers. A. 7 Qualitative Examples In Figure 8, we show additional qualitative examples where having a ToM listener enables the speaker to generate a more accurate instruction. ACL 2023 Responsible NLP Checklist