Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
Authors: Daniel Fried
Nicholas Tomlin
Jennifer Hu
Roma Patel
Aida Nematzadeh
Venue: EMNLP 2023 Findings (2023)
Keywords: grounding
pragmatics
multimodality
survey
TL;DR: We argue that pragmatics in grounded settings is a frontier for NLP. We survey phenomena, existing tasks, and recommend future work.

Abstract: People rely heavily on context to enrich meaning beyond what is literally said, enabling concise but effective communication. To interact successfully and naturally with people, user-facing artificial intelligence systems will require similar skills in pragmatics: relying on various types of context --- from shared linguistic goals and conventions, to the visual and embodied world --- to use language effectively.

We survey existing grounded settings and pragmatic modeling approaches and analyze how the task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. We present recommendations for future grounded task design to naturally elicit pragmatic phenomena, and suggest directions that focus on a broader range of communicative contexts and affordances.

Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
Daniel Fried Nicholas Tomlin Jennifer Hu²
Roma Patel Aida Nematzadeh"
Carnegie Mellon UC Berkeley² MIT Brown University DeepMind
dfried@cs eww.deesccuueeeeee. nicholas www.in@eereeeee..ccooommm wtw.eeeeeercoosss..ooonnneeeeerr mannhu@mit.edu {romapatel, jemmatzadeh@@deepmiiiee
Abstract
has focused on general-purpost models (Tan and Bansal, 2019; Brown al., 2020; Radford et al., 2021; Bommasani et al., 2021, inter alia) that have achieved remarkable performance on a variety of task benchmarks intended to measure literal, se- mantic meaning. However, current LLMs have little, if any, direct access most of the types of context that contribute pragmatic enrichment (see Section 2.1) they condition on and generate text that is only noisy observation of the underly- ing world, interaction context, and communicative effects of language (Andreas, 2022). This lack direct access to the context and effects of language leads to pragmatic failures. Recent work finds that while L.MSs match human performance on some pragmatic tasks, they still struggle with many phc- nomena, e.g. those that rely on social expectation violations (Hu et al., 2022) and theory of mind (Sap et al., 2022). For multimodal LLMs, contemporary work has found pragmatic failures when reasoning about the intended meaning of entities described in text (Rassin et al., 2022) as well as when reasoning about vinio-ling compositions of such entitics (Thrush et al., 2022).
People rely heavily on context to enrich mean- ing beyond what is literally said, enabling con- cise but effective communication. To interact successfully and naturally with people, LLMs and other user-facing NLP systems will require similar skills in pragmaticx: relying on various types of context-frood shared linguistic goals and conventions, the visual and embodied world-ar to use language effectively.
We survey existing grounded settings and prag- matic modeling approaches and analyze how the task goals, environmental contexts, and communicative affordances in each work en- rich linguistic meaning. We present recom- mendations for future grounded task design to naturally elicit pragmatic phenomena, and sug- gest directions that focus broader range of communicative contexts and affordances.
1 Using Language in Context
People use language to achieve goals (Wittgenstein, 1953; Austin, 1975; Clark, 1996; Frank and Good- man, 2012), producing effects on other people and the world. To achieve their goals efficiently, people often only sketch their intended meanings: relying on various types of context to allow their conver- sational partners to enrich meaning beyond what the speaker has literally said. For this reason, lan- guage is highly context-dependent; the meaning of even simple sentence such as "it's nice today" depends on the situation can be an implicit in- vitation, statement contrasting the weather with a previous day, or even ironic if the weather is poor. This broad ability to use language in context to achieve goals is known as pragmatics.
We believe the time is right to focus evalu- ating models tasks (some existing, but mostly to-be-developed) that require contextual communi- cation abilities: ones that elicit pragmatic phenom- ena from people, and benefit from pragmatic abil- itics in systems. These tasks should give proxy for model performance in grounded interactions with real people, while still facilitating comparing methods and benchmarking progress. In this paper, we center our discussion the role of grounded and multimodal context in pragmaties, motivated by the rich contexts of typical language use-our understanding of language is shaped by the envi- ronments that we use it in (Baldwin, 1995; Bloom. 2002: Tomasello, 2005; Vigliocco et al., 2014; Ben- der and Koller, 2020; Bisk et al., 2020).
As large language models (LLMs) and other NLP systems become increasingly integrated into world, they will require similar pragmatic skills use language to interact successfully and ef- ficiently with people in context. Recent work
"qqua contribution.
We first survey how various communicative and

---

environmental context types elicit pragmatic phe- nomena. Using these context types and phenomena, we then survey representative grounded tasks and datasets which have been used both to study prag- matic communication in people, and to build goal- oriented multimodal systems. We present tasks along a spectrum of complexity, ranging from con- strained reference games to goal-oriented embod- ied dialogue. We discuss how choices in grounded environment properties, context types, and communicative affordances shape the pragmatic phenomena that arise in tasks, and provide suggestions for future task and dataset designers. To model these tasks and phenomena, we give an overview of range of computational pragmatic approaches that view communication as goal-directed actions by agents in context.
communicative norms, (2) task goals and collab- orative actions, (3) common knowledge, and (4) discourse context. In addition to common ground. we focus on pragmatic reasoning that also requires multimodal context, such as (5) visual information or (6) embodied interaction. See Appendix A for definitions and examples for each of these types of context; we also point readers to Levinson (1983) and Birner (2012) for more discussion.
task
2.2 Roles of Pragmatics
We adopt a use-oriented view of pragmatics (Clark, 1996), highlighting various ways that pragmatic reasoning may be used in grounded language tasks. Our taxonomy is not intended to be fully exhaus- tive, and we caveat that some categories may par- tially overlap with one another. For a focus on classic linguistic pragmatic phenomena, like deixis and presupposition, we refer readers to existing sur- veys such as Levinson (1983) and Birner (2012).
Finally, we suggest further integrations of com- putational pragmatics with NLP: developing tasks with rich contexts; contextualizing existing tasks; using pragmatic and contextual modeling to allow systems to communicate and interact more suc- cessfully and efficiently with people; and tackling challenges of human evaluation and data sparsity.
Reasoning About Alternatives. Much of linguis- tic meaning comes not just from what we say, but from what we do not say. The utterances that speak- ers choose not to say i.e., the set of alternative ut- terances which are likely in a context, can reveal their intended meanings and mental states (Horn, 1984; Fox and Katzir, 2011; Degen, 2013; Buccola et al., 2022), c.g., some of the apples are red likely conveys that some are not, since the speaker did not say all were red. Many of the following roles of pragmatics also often involve reasoning over alternatives.
2 Pragmatic Phenomena
In linguistics and cognitive science, pragmatics is often defined in contrast to semantics. Broadly speaking, semanties characterises the literal mean- ings of linguistic expressions, whereas pragmat- ies captures the context-dependent components of meaning, which may contain the bulk of ac- tual communication (Clark, 1997; Casasanto and Lupyan, 2015). Pragmatic communication draws upon many different sources of information, rang- ing from environmental factors to inferences about other agents' unspoken information and goals. This makes pragmatics both a critical and challenging component for designing NLP systems that interact with people. In this section, we discuss the types of context in which language can be situated and the non-literal inferences that arise as a result of these contextual pressures.
Understanding Ambiguity. Language is fre- quently ambiguous for many reasons (Piantadosi et al., 2012): ambiguity may be used strategically to achieve communicative efficiency or to remove information that is unhelpful to the task at hand. Moreover, ambiguous instructions often require listeners to reason pragmatically about alternative intentions that speakers might have. For example, when asked to pass the knife in cooking scenario, pragmatic agent might have reason about the context to determine whether to provide a butter, bread, or steak knife. By relying on contextual in- formation to resolve ambiguities in situations such as these, pragmatic interlocutors can communicate more efficiently (Solé and Seoane, 2015; Fortuny and Morominas-Murtre 2013).
2.1 Types of Context
Many aspects of pragmatics involve the mainte- nance of common ground, of contextual in- formation shared between communicative part-
ners (e.g., Lewis, 1969; Clark and Brennan, 1991; Traum, 1994; Stalnaker, 2002; Clark, 2015). Key elements of common ground include (1) social and
Collaborative Planning. Many grounded dia- logue tasks require agents to coordinate to carry

---

out joint activities, e.g., collaboratively agree to goal before executing it. To succeed tasks like these, participants often must reason about each other's possible goals, for example in a collabora- tive building setting, inferring that four planks can be either a command or a description depending on what effect the speaker is trying produce on the listener. In environments with many world states, there are combinatorial number of goals rea- son about and actions take, but participant can usually only communicate with their partner for limited time. Therefore, participants must trade-off between communicating efficiently and acting.
3 Existing Tasks and Environments
In this section, we critically evaluate several well- studied grounded language tasks through the lens of the pragmatic phenomena outlined above. We focus tasks in multimodal domains that make use of natural language data."
3.1 Types of Tasks
Grounded, task-oriented dialogue provides gen- eral setting to study pragmatics. Dialogue tasks provide rich and varied contexts (c.g., different types of common ground, goals, and environments) as well as communicative affordances (e.g., the ability to ask questions, provide information in in- stallments, and adapt to a partner's conventions). These contexts and affordances interact to produce diverse range of pragmatic behavior (Clark, 1996). However, many of these contexts, affordances, and behaviors are also present in more restricted and controlled tasks for which data collection, analysis, modeling, and evaluation are often more tractable. For example, image captioning tasks simplify data collection and modeling by limiting the number of conversational turns to one; instruction interpreta- tion tasks additionally simplify evaluation (so long as it is possible to carry out and validate actions in the world).
Convention Formation and Abstraction. Con- ventions, as characterised by Lewis (1969), are arbitrary but stable solutions to recurring coordina- tion problems that typically form out of the maxims of rational communication (Grice, 1975). For ex- ample, team of workers who communicate with one another daily might initially have lengthy de- scriptions to refer to certain items, but after while, might start to develop a common ground of simpler words to refer to them. These abstractions or con- ventions are hypothesised to emerge as a result of repeated interactions (Garrod and Doherty, 1994). One theory is that conventions form to help resolve ambiguity, yielding more efficient communication at the levels of individuals (Hawkins al., 2017) or populations (Hawkins et al., 2022).
We focus on reference games, image captioning, instruction following, and grounded dialogue tasks that give broad characterization of the diffeerent properties that tasks might have, summarized in Table 1.2 For each task, we specify what type of context is needed, how pragmatic behavior is typically exhibited, and several important elements of task design: partial observability, symmetry, and iterated interaction (see Section 3.2), We present these domains in order of increasing complexity, finding that the most complex grounded dialogue tasks are more likely involve features like partial observability symmetry which induce additional pragmatic phenomena.
Efficiency and Mutual Exclusivity. For many grounded tasks where the goal is to learn a corre- spondence between meanings and utterances, prag- matic reasoning can be used to avoid learning de- generate mappings. For example, on learning that certain label (e.g.. cat) refers to object, an agent might mutual exclusivity to rule out the pos- sibility of another label (c.g., dog) also referring the object (Markman and Wachtel, 1988; Clark, 1988). Models of pragmatic reasoning often induce biases toward mutual exclusivity that can lead to more efficient learning (Wang et al., 2016; McDow- ell and Goodman, 2019). More broadly, pragmatic reasoning may be used to manage the dual pres- sures of informativity and conciseness (Zipf, 1949; Horn, 1984; Blutner, 1998), which are explicitly factored into pragmatic models such as RSA (cf. Section 4.2). As a result, pragmatics may lead to communicative efficiency both during language learning and language use.
'We omit large bodies of work unimodal pragmatics (Degen, 2013; Jeretic et al., 2020; Choi et al., 2021, inter alia) language that might he grounded, but synthetically generated (Johnson al., 2017; Bastings et al., 2018; 8; Zhong al., 2020).
"This is not an exhaustive taxonomy of grounded language learning tasks. For example, VQA (Antol et al., 2015), NLVR2 (Suhr et al., 2019b), the Hateful Memes Challenge (Kirk et al., 2021). )) and Winoground (Thrush et al., 2022) do not fit per- fectly into any of the above categories, although most bear some similarities to image captioning.

---

Task (Durasct)
Types Context
Role of Pragmatics
Pb
Sym.
Iter.
Reference Game
ReferliGome
Reasoning about alternativss, under- standing ambiguity
Visual
x
x
Kazemzudeh et al. (2014)
Colors in Context
Reasoning about alternatives, under- standing ambiguity
Visual
X
V
Monroe et (2017)
Image Captioning
Abstract Scenes
Visual, common knowledge
Reasoning about alternatives
X
X
X
Andreas and Klein (2016)
Conceptual Captions
Visual, common knowledge, joint
Sharma al. (2018):
goals, norms interaction
Efficiency considerations
X
X
X
Alikhani et al. (2020)
Instruction Following SHRDLURN Wang ul. (2016)
Mutual exclusivity, convention forma-
Visual
tion, efficiency considerations
CerealBar
Collaborative planning, understanding ubstractions and conventions
Visual, embodied
X
Suhr et al. (2019a)
Hexagons
Understunding abstractions and ambi- guity, efficiency considerations
Visual, norms of interaction
x
Lachmy et al. (2022)
Grounded Dialogue
Cards Corpus
Visual, embodicd. joint goals, norms of interaction, discourse
Collaborative planning, understanding
Potts (2012)
umbiguity. efficiency
x
OneCommun
Visual, joint goals, of interuc- Udagawa and Aizawa (2019) tion, discourse
Collaborative planning, understanding ambiguity. efficiency
PhotoBook
Visual, common knowledge, joint goals, norms of intterrctiion, discourse
Convention formation, understanding umbiguity. efficiency
Haber et al. (2019)
Table I: Example grounded language learning datasets that involve pragmatic reasoning, organized by task type. The task attributes refer to: partially observable, symmetric, and iterated (multi-turn) interactions. We observe that grounded dialogue and instruction following tasks often involve broader range of pragmatic reasoning hehaviors.
Reference Games. Reference games typically in- volve two players, a listener and a speaker agent, Both players are presented with shared set of referents, e.g., images, objects, abstract illustra- tions, and the speaker is tasked with describing target referent to the listener, who must then guess the target (Clark and Wilkes-Gibe Gibbs, 1986; Gorniak and Roy, 2004; Steels and Belpaeme, 2005; Gol- land et al., 2010; Frank and Goodman, 2012; Ken- nington and Schlangen, 2015). An example refer- ence game is the Colors in Context (Monroe et al., 2017) task, in which players are presented with three color swatches and asked to describe one of them. Even simple phrases like plain blue may have different meanings depending on visual con- text in this task.
how closely their descriptions match these human- written references, which poses challenges given considerable variation in what annotators chose to describe and how they wrote the descriptions (An- derson al., 2016).
Other work, particularly in the computational pragmatics literature, has formulated captioning as a contrastive task (Andreas and Klein, 2016; Vedantam et al., 2017; Cohn-Gordon et al., 2018), where a target image must be described to contrast it from other similar, distractor images. This set- ting can be viewed as a scaled-up reference game involving complex visual inputs, and many such pragmatically- variations on standard im- age captioning have appeared in recent years: Nie et al. (2020) define issue-sensitive image caption- ing, in which models implicitly caption several target images at time, while Alikhani et al. (2020) train coherence-aware captioning models which may vary in the degree of subjectivity or the extent to which inferences about target images are made.
Image Captioning. A broad class of image cap- tioning tasks require producing text to describe an image (Barnard et al., 2003; Farhadi et al., 2010; Mitchell al., 2012; Kulkamn et al., 2013). Most captioning work has only been implicitly goal- oriented: corpora have been constructed by asking annotators to determine and describe the important parts an image (Hodosh et al., 2013; Young et al., 2014; Chen et al., 2015). Systems are evaluated on
Of the task categories we discuss, image cap- tioning has the most immediate real-world appli- cability, especially for accessibility e.g., to pro- vide descriptions that could substitute for images

---

for visually-impaired users on the web (Pont-Tuset et al., 2020). Additionally, practical considerations in this domain often require pragmatic reasoning e.g., Specificil deseribing salient characteristics of an image (e.g., a man versus Barack Obama), being concise, or describing the relevance of the image to document context. We refer the reader to MacLeod et al. (2017) and Kreiss et al. (2021) for further information on this topic.
2012), players collaboratively collect set of cards in a grid world environment by communicating with other players while moving around to pick up cards. Observability is limited to parts of the environment close to the players, requiring them to pool information, and they must collaboratively plan to agree on one of the multiple possible sets of cards they can collect.
The multi-turn nature of dialogue also necessi- tates reasoning about past actions and interactions (perform inference) and likely outcomes in the fu- ture (planning). These are particularly evidenced in collaborative reference tasks such as OneCom- mon (Udagawa and Aizawa, 2019), where players must infer which items they share with their part- ners, aggregating information over the course of dialogue. Finally, repeated interactions dialogue can allow linguistic adaptation. For example, PhotoBook task (Haber et al., 2019)-a collabo- rative reference task where players have repeated conversations about photographs playeers adapt their language over time to match each other, be- coming more efficient over time (c.g., reducing the strange bike with three wheels to strange bike).
Instruction Following. Instruction following tasks require a listener to take instructions from a speaker, predicting trajectories in an envi- ronment (Branavan et al., 2009; Vogel and Ju- rafsky, 2010; Chen and Mooney, 2011; Tellex et al.. 2011; Anderson et al., 2018). Tra- jectories can be gramedr-based actions (e.g.. ORANGE), to specify add an orange block to the left-most brown block in the block-stacking setting of Wang et al. 2016), sequences discrete movements (c.g., be- tween nodes in navigation graph in Chen et al. 2019; Ku et al. 2020), or continuous sequences (e.g., of orientations in Ku et al. 2020).
A speaker must describe target trajectory in a way that allows the listener to correctly carry it out in the presence of (often exponentially many) alternative trajectories (e-g., left versus sharp left). These environments often involve grounddd observations (Anderson et al., 2018; Chen et al., 2019; Ku et al.. 2020), ac- tion hierarchies (Shridhar et al., 2020) or program- matic abstractions (Lachmy et al., 2022) and some parts of the environment may be unobserved to the speaker, the listener, both (see Section 3.2), caus- ing language to be more ambiguous and context- dependent.
3.2 Elements of Task Design
We now outline three especially pragmatically- relevant dimensions to consider when designing tasks and describe how they induce various types of pragmatic phenomena.
Observability. In partially observable tasks, par- ticipants can only see limited portion of the envi- ronment, for example sceing only the parts of the grid closest to them in the Cards task (Potts, 2012). This can make language more context-dependent, in particular creating a dependence on when or where the language was produced. The most com- plex purtially-observee the collaborative dialogue tasks in Table 1, involve participants observing different views of the envi- ronment - requiring them to collaboratively plan to pool their information. Different views can also lead to false agreements where participants believe they have coordinated but actually disagree (Chai et al., 2014; Udagawa and Aizawa, 2019), requiring more explicit pragmatic modeling of the partner's perspective avoid and resolve ambiguity.
Grounded Goal-Oriented Dialogue. We focus on grounded dialogue tasks that involve two-way communication between partners to achieve shared goal (e.g., Chai et al., 2004; Rieser and Lemon, 2008; Das al., 2017; De Vries et al., 2017; Kim et al., 2019; Narayan-Chen al., 2019; Ilinkkh al., 2019).3 These tasks generalize the one-way communication settings above; how- ever, two-way communication provides additional affordances-alloviin players to ask clarification questions, acknowledge understanding, and coordi- nate actions. For example, in the Cards task (Potts,
settings, including all of
Symmetry. Tasks differ in the types of roles per- formed by the communicating agents, which turn shapes the type of language produced and
'Our focus is on task-ooientedd dialogue, given that com- municative goals are less explicit in chit-chat settings (but see Kim et al. (2020) for a recent pragmatic treatment).

---

actions taken. We distinguish between asymmet- ric and symmetric roles. In an asymmetric set- ting - e.g. speaker and listener, or teacher and follower - pragmatics may be helpful for pro- duction and comprehension of language utterances. Symmetric settings ( (Vogel et al., 2013a,b) may be more naturalistic and are often used in coordina- tion tasks, although designing such settings is often more complicated. Asymmetric settings (Monroe al., 2017; 77; Andreas and Klein, 2016) are often the simplest way to introduce pragmatic phenomena, since asymmetry occurs when one agent is missing information.
Theune, 2010; Fried al., 2018a; Zhao al., 2021; Gehrmann et al., 2022). Automated proxies for human listeners are models of how people inter- pret and respond to a system's language, known as user simulation or self-play (Georgila et al., 2006; Rieser and Lemon, 20 Lewis et al., 2017; Kim et al., 2019) in dialogue and communication based evaluation (Newman et al., 2020) in refer- ence games, where speaker generations fed to listener model and evaluated on task success. Au- tomated models can only give rough indicators of how humans might interpret the system's language. For this reason, we stress the importance of making the evaluation model dissimilar from the system and using human evaluations whenever possible.
The nature of interaction(s) between communicating agents affects the language that is produced. In a one-turn interaction, all usable in- formation must be expressed in single utterance, forcing speakers to balance informativity and con- ciseness. In iterated one-sided interactions, the speaker has the opportunity to respond to the lis- tener's actions before planning each new utterance. Finally, in dialogue, agents can freely coordinate and participate in speech acts-they can jointly build common ground, ask clarification questions, and share useful information. These repeated inter- actions between agents require attention to conver- sation history, and may give rise to the formation of conventions (e.g., Hawkins al., 2017).
Interaction.
Communicative efficiency. Beyond task success, secondary criterion for situated agents efficient communication. For example, if the language gen- erated by a speaker, although correct, is difficult to understand, this calls for unnecessary interpre- tation effort from the other agent. To measure whether pragmatic agents enable efficient commu- nication, evaluations can use metrics of commu- nicative cost (Walker et al., 1997) such as time task completion, utterance length and complexity (Effenberger et al., 2021), measures such as lexical entrainment (Clark and Wilkes-Gibbs, 1986; Par- ent and Eskenazi, 2010; Hawkins et al., 2020), and quality ratings (Kojima et al., 2021).
3.3 Evaluating Pragmatic Models
The ultimate goal for user-facing, situated agents is to communicate (1) successfully and (2) efficiently with people. Human evaluations, where agents are paired with people at test-time, are an ideal way to measure this (Walker et al., 1997; Koller et al., 2010; Parent and Eskenazi, 2010; Suhr et al., 201 9a, inter alia), but are not always feasible to carry out since they complicate controlling and replicating experimental setups. Thus, evaluation often resorts either to static, suman-producee corpora or auto- mated model-based evaluations.
4 Modeling Pragmaties
In this section, we discuss frameworks that have been proposed to characterize how listeners can de- rive pragmatic meaning, providing a starting point for modeling the phenomena and tasks above.
4.1 Gricean Maxims
In his seminal proposal, Grice (1975) argues that speakers and listeners are guided by an underly- ing cooperative principle: taking action to jointly achieve communicative goals, and assuming that other agents are acting similarly. Grice divides this principle up into set of maxims. However, at- tempts to directly implement the Gricean maxims computationally (e.g., Hirschberg, 1985a) have had to grapple with substantial underspecification and overlap in Grice's proposal. Later neo-Gricean work in linguistics has streamlined the maxims con- siderably (Horn, 1984; Levinson, 2000) and charac- terizes many pragmatic effects in terms of the trade- off between speaker and listener effort in achieving
Task success. Interpretation tasks are typically amenable to corpora-based evaluation. For ex- ample, listener agents in reference games can be easily evaluated based on the accuracy of refer- ent selection. In contrast, evaluating language generation tasks for speaker agents is more chal- lenging, given that many classical reference-based automated NL.G metrics are unable measure whether or not generated language will be under- stood correctly by human listeners (Krahmer and

---

cooperative goals. These approaches have had few direct computational implementations; however, line of computational work, which we outline in Sections 4.2 and 4.3, derives maxim-like behav- ior through multi-agent modeling rather than by prescriptively implementing the maxims.
cursive reasoning frameworks have been limited to single-tur interactions. However, the multi- turn approaches that we outline Section 4.3 al- low modeling repeated interactions by making the framework simpler along certain axes (e.g., remov- ing higher-order mncory-of-minde
4.3 Multi-Tury Approaches
4.2 Multi-Agent Reasoning
A variety of approaches to multi-turn pragmat- ics have arisen work on task-oriented dia- logue. Many of these treat communication as goal- directed decision -making under uncertainty (Rieser and Lemon, 2011; Young et al., 2013), and can be broadly viewed generalizing the single-tum frameworks of Section 4.2. For generation, variety of dialogue systems explicitly plan utter- ances or speech acts to convey information to their partners (Cohen and Perrault, 1979; Traum, 1994; Walker et al., 2004; Rieser and Lemon, 2009; Kim et al., 2020, inter alia). For interpretation, many systems infer the latent intent or state of the user (Allen and Perrault, 1980; Paek and Horvitz, 2000; Williams and Young, 2007; Schlangen et al., 2009; Young al., 2013, inter alia).
A number of computational frameworks view ut- terance generation and interpretation using multi- agent or game-theoretic lens (Rosenberg and Co- hen, 1964; Cohen and Levesque, 1990; Golland et al., 2010; Jäger, 2012; Franke, 2013). We fo- cus one representative of these, the Rational Speech Acts (RSA) framework (Frank and Good- man, 2012: Goodman and Frank, 2016), has been successfully applied across a range of grounded language settings.
RSA defines a recursive reasoning process where speakers and listeners model each other's goals and interpretations. A rational speaker chooses utterances using an embedded model of how the listener will likely interpret utterances. A rational listener, in turn, reasons counterfactually about a rational speaker generating language in this way reasoning about why the speaker choose an ob- served utterance rather than alternatives-whice can resolve ambiguity in the speaker's utterances.
Planning and inference are classic Al tasks with broad applicability, and most of the works above are closely related to general machinery developed for decentralized POMDPs (Bernstein et al., 2002; Oliehoek and Amato, 2016). However, given com- putational challenges, past work on algorithmic ap- plications of POMDP algorithms to communication have focused on domain-specifee formalisms (the works above) or restricted language settings (Zettle- moyer et al., 2008; Vogel et al., 2013a; Hadfield- Menell et al., 2016; Foerster et al., 2019; Jaques et al., 2019). To enable pragmatic modeling and interaction with people in naturalistic grounded dialogue settings, future work might draw on fur- ther progress that the multi-agent reinforcement learning and planning communities make these underlying algorithmic challenges.
A variety of work has also applied RSA to im- prove performance of NLP systems on a range of tasks involving complex natural language ut- terances, including reference games (Monroe et al.. 2017), instruction following and generation (Fried al., 2018a,b), image captioning (Andreas and et al., 2018), summa-
Klein, 2016; rization (Shen et al., 2019), MT (Cohn-Gordon and Goodman, 2019), and dialogue (Kim et al., 2020; Fried et al., 2021). A number of rational com- munication frameworks also include noteworthy variations on the core RSA setup, include vary- ing the utility function (Zaslavsky et al., 2021), modeling mis-aligned objectives (Asher and Las- carides, 2013), using deeper levels of recursive rea- soning between agents (Wang et al., 2020), and non- linguistic communication (Hadfield-Menell et al., 2017; Jeon et al., 2020; Pu et al., 2020).
Discussion
5.1 Building Pragmatiialll-Inforreee Tasks Pragmatics becomes most essential - and most challenging - in grounded and interactive settings, which have the richest contexts of language use. When people can rely on shared dialogue, environ- mental, and task contexts to convey and decode meanings, pragmatic phenomena emerge that have so far been understudied in NLP, such as conven-
One key limitation of RSA that it models speakers as choosing their utterances from known and fixed set of candidate utterances. second notable limitation that, with a few exceptions (e.g., Khani et al., 2018) 8), applications of full re-

---

tion formation (Section 2.2). While pretraining on
lie convention formation. Can these problems be solved by (multimodal))LLM that condition on the right contexts and are trained on sufficient data, or does there remain a role for explicit pragmatic modeling? What is the best path towards producing human-like pragmatic communicative behavior?
multimodal (Lu al., 2019; Sun et al., 2019; Tan and Bansal, 2019; Radford et al., 2021) and interac- tive (Stiennon et al., 2020; Shuster et al., 2022; Bai et al., 2022) data has driven recent progress in their respective domains, training data and evaluation setups for settings that are both grounded and inter- active remain limited in scope, size, and ecological validity (De Vries al., 2020). This sparsity poses both challenge and an opportunity for pragmatic language use.
While we view these questions as currently un- resolved, we predict that to answer them and to advance grounded NLP, will prove useful to ex- plore pragmatic modeling and contextually condi- tioned, multimodal LLMs in tandem. Reasoning about how context enriches meaning (Section 2.1) allows people say less while still conveying their intentions, and to learn from what could have been said but was not. Similarly, practical benefits of explicit pragmatic modeling include more efficient and accurate communication (Monroc et al., 2017; Khani et al., 2018; Fried et al., 2021) and learn- ing in low data regimes (McDowell and Goodman, 2019; Wang et al., 2016), such as interactive and personalized settings.
We encourage future work to focus on realistic interactive scenarios which contain wide range of pragmatic phenomena. As shown in Table grounded, multi-turn dialogue tasks with partial observability and symmetry often encompass the widest variety of pragmatic language behavior. To date, relatively few tasks have all of these proper- ties, although there are few notable exceptions (e.g.. Potts, 2012: Udagawa and Aizawa, 2019). We argue that such tasks provide useful testbeds for building models of convention formation and collaborative planning, which are currently under- studied in the computational pragmatics literature.
LLMs have already begun to be put to use to tackle the grounded pragmatic tasks we outline in Table 1 and Section 3. For example, Kojima et al. (2021) fine-tune pre-trained GPT-2 model to adapt to people for instruction generation in Cc- realBar. The pragmatic methods in Section 4 are also compatible with LLMs, e.g., Liu et al. (2023) combine RSA with meta-learning to apply GPT models in an image reference game setting; FAIR et al. (2022) use a large BART model (Lewis et al., 2020) in conjunction with a multi-agent planning procedure in the grounded dialogue game of Diplo- macy. As grounded LLM adapters (Alayrac et al.. 2022; Merullo al., 2023; Eichenberg et al., 2022; Koh al., 2023) continue to improve, we expect to see more work applying LLMs as components of pragmatic models for these grounding tasks.
In tandem, we also encourage future work to contextualize existing NL. tasks in order to bring them closer to real-world applicability. For exam- ple, as discussed in Section 3.1, MacLeod et al. (2017) and Kreiss et al. (2021) argue that current image captioning systems fail to meet the needs of visually-impaired users because they do not pro- vide descriptions in the context of the article or of the user's goals. Incorporating and modeling additional context-sute ing (Alikhani al., 2020), interaction history in in- struction following (Kojima et al., 2021: Lin et al., 2022), or emotion (Kim et al., 2021) and person- ality (Wang et al., 2019) in dialogue- may help close the gap between current NLP benchmarks and useful real-world systems.
user intent in caption-
Regardless of the approaches used, we encour- age future work in NLP to evaluate models not just on task success, but also other pragmatic aims: communicative efficiency, understanding of non- literal language. ability to form conventions (Sec- tion 3.3) and consider all approaches that are effec- tive in achieving these goals (e.g., Section 4.2).
5.2 Pragmatic Modeling and LLMs
We predict that focusing on tasks that require prag- matic capabilities will advance the frontier of NLP, and vice versa. Many of the roles of pragmat- ies that outline in Section 2.2 are linguistic manifestations of classical problems in machine learning and AI: inference and model calibration underlie reasoning about alternatives and ambigu- ity; multi-agent search underlies collaborative plan- ning: adaptation and abstraction learning under-
5.3 Challenges and Open Questions
Going forward, accurate evaluation of pragmatics one key challenge in grounded language learning. Some recent benchmarks have been proposed to evaluate pragmatic language behavior in large lan-

---

Our discussion of modeling frameworks for prag- matics in Section 4 focuses on approaches that dis- tinguish between semantics and pragmatics through social reasoning about other agents' beliefs and goals. Due to space limitations, we did not discuss alternate theories proposing that pragmatically en- riched meanings are derived within the grammar of a language, without recourse to probabilistic so- cial reasoning (e.g.. Fox, 2007; Chierchia et al., 2012: Asherov et al., 2021). These theories remain difficult to implement at scale, but we encourage fu- ture work to explore them as candidate hypotheses alongside the frameworks discussed in Section 4. guage models (Sap et al., 2022: Hu al., 2022; Ruis et al., 2022), but evaluation in interactive, multimodal scenarios often requires humans or hu- man proxies. As discussed in Section 3.3, some models may be evaluated in self-play, and future work might draw on improved proxies for human interaction from multi-agent reinforcement learn- ing (Strouse et al., 2021). However, human eval- uation remains the gold standard in most settings, and we encourage future work (1) improve hu- man evaluations, taking lessons from the human computer interaction community and (2) focus on tasks that are useful (Bigham et al., 2010; Stiennon et al.. 2020), fun (Wang et al., 2017: FAIR et al.. 2022), or otherwise intrinsically motivating. Another key challenge is handling data sparsity. Although recent advances in large-scale pretraining have led to few-shot capabilities on many language tasks, interactive and grounded tasks pose addi- tional challenges in sparsity. First, the addition of multimodal context widens the conditioning space. Second, many grounded settings involve domain- specific knowledge; for example, instruction fol- lowing and grounded dialogue settings often have unique action spaces, well as domain-specific ab- stractions or conventions in language use. Finally, many settings require adaptation to individual users, for which there will always be limited data.
also do not discuss tasks involving synthetic or emergent language, but see Lazaridou and Baroni (2020) for survey of the latter.
While we focus mostly on written or typed lan- guage, there is also some computational work that has focused on spoken language pragmatics in grounded settings (Harwath et al., 2016; Sharma et al., 2018) well as work in linguistics the intersection of speech and pragmaties, e.g., focused on prosody (Pierrehumbert and Hirschberg, 1990; Sedivy et al., 1999).
There is also rich body of work on formaliz- ing and modeling discourse context beyond the ap- proaches we cover here, including conversational analysis (Schegloff, 1968; Sacks al., 1974) and discourse coherence and structure (Hobbs, 1979; Grosz and Sidner, 1986; Webber, 1991; Kamp and Reyle, 1993; Grosz et al., 1995; Webber et al.. 2003; Asher and Lascarides, 2003; Barzilay and La- pata, 2008). We refer to Cohen et al. (1990), Clark (1996), Jurafsky and Martin (2014), and Alikhani and Stone (2020) for entry points.
As NLP expands to an ever-wider range of con- texts, we encourage work to include pragmatics as a central component, with the goal of communi- cating successfully, efficiently, and naturally with people in challenging and useful settings.
Limitations
Although we aim to describe a representative sam- ple of tasks in Table 1, our coverage is necessarily incomplete, especially in domains such as image captioning, cnttucctionfollowiing and collabora- tive dialogue, so we refer readers other surveys on these issues (e.g.. Luketina et al., 2019). As noted in Section we focus exclusively on task- oriented grounded domains involving natural lan- guage data. Our survey therefore includes limited discussion of pragmatic phenomena in unimodal text domains such as chitchat dialogue, purely tex- tual task-oriented dialogue, and language classi- fication tasks (although c.f. Section 4.3 and Ap- pendix B), and omits much work on analyzing the abilities of models to perform classic pragmatic tasks such as implicature and presupposition (c.g., Ross and Pavlick, 2019; Jeretic et al., 2020). We
Acknowledgments
We are grateful to Alane Suhr, Justin Chiu, Jessy Lin, Kevin Yang, Ge Gao, Ana Smith, and Her- bert Clark for early discussions that led this survey. We also thank Chris Potts, Alane Suhr, Ari Holtzman, Victor Zhong, Laura Rimell, Chris Dyer, Saujas Vaduguru, Tao Yu, Allen Nie, Dan Klein, Andrew Lampinen, and numerous anony- mous reviewers and ACs for their comments on earlier drafts of our paper. Nicholas Tomlin was supported by the DARPA XAI and LwLL programs and NSF Graduate Research Fellowship. Jennifer Hu was supported by a NSF Graduate Research Fel- lowship and a NSF Doctoral Dissertation Research Improvement Grant.

---

References
Nicholas Asher and Alex Lascarides. 2013. Strategic conversation. Semantics and Pragmatics, 6.
Jean-Baphe -ann-Bptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech. lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning, In NeurlPS, volume 35, pages 233736.
Daniel Asherov, Danny Fox, and Roni Katzir. 2021. On the Irrelevance of contextually given states for the computation of Scalar Implicatures.
John Langshaw Austin. 1975. How to Do Things with Words. Clarendon Press.
Malihe Alikhani, Piyush Sharma, Shengjie Li, Radu Soricut, and Matthew Stone. 2020. Cross-modal co- herence modeling for caption generation. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6525-6535, On- line. Association for Computational Linguistics.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma. Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprimi 2rXiv:2204008866
Malihe Alikhani and Matthew Stone. 2020. Achiev-
ing common ground in multi-modal dialogue. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguisties: Tutorial Abstracts, pages 10-15, Online. Association for Com- putational Linguisties.
Dare A Baldwin. 1995. Understanding the link between joint attention and language. Joint attention: Its origins and role in development, 131:158.
Kobus Barnard, David Forsyth, David M Blei, Michael Jordan, Jaz Kandola, Thomas Hofmann, Tomaso Pog- gio, and John Shawe-Taylor. 2003. Matching words and pictures. Journal of Muchine Learning Research.
James Allen and C Raymond Perrault. 1980. Ana- lyzing intention in utterances. Artificial Intelligence, 15(3):143-178.
Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Compu- tational Linguistics, 34(1):1-34.
Prithviraj Ammanabrolu and Mark O. Riedl. 2021. Situ- ated language learning via interactive narratives. Pat- terns, 2(9).
Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. 2018. Jump to better conclusions: SCAN both left and right. In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 47-55, Brussels, Belgium. As- sociation for Computational Linguistics.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic proposi- tional image caption evaluation. Proceedings of the European Conference on Computer Vision (ECCV), pages 382-398. Springer International Pub- lishing.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understand- ing in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online, Association for Computational Linguistics.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilherstein. 2002. The complexity of decen- tralized control of Markov decision processes. Math- ematics of operations research, 27(4)8819 840.
Jacob Andreas. 2022. Language models as agent mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769-5779. Asso- ciation for Computational Linguistics.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to vi- sual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and tech- nology, pages 333-342.
Jacob Andreas and Dan Klein. 2016. Reasoning about pragmaties with neural listeners and speakers. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1173- 1182, Austin, Texas. Association for Computational Linguistics.
Betty J Birner. 2012. Introduction to Pragmatics, vol- ume 38. John Wiley & Sons.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015, Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nis- nevich, et al. 2020. Experience grounds language. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Nicholas Asher and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press.

---

Paul Bloom. 2002. How children learn the meanings of words. MIT press.
Joyce Y Chai, Pengyu Hong, and Michelle X Zhou. 2004. A probabilistic approach to reference resolu- tion in multimodal user interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces, New York, New York, USA. ACM Press.
Reinhard Blutner. 1998. Lexical Pragmatics. Journal of Semantics, 1522)1155-16.
Dan Bohus and Eric Horvitz. 2010. Facilitating mul- tiparty dialog with gaze, gesture, and speech. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multi- modal Interaction. number Article 5 in ICMI-MLMI '10. pages 1-8, New York, NY, USA. Association for Computing Machinery.
Joyce Y Chai, Lanbo She, Rui Fang, Spencer Ottar- son, Cody Littley, Changsong Liu, and Kenneth Han- son. 2014. Collaborative effort towards common ground in situated human-robot dialogue, In Proceed- ings of the 2014 ACM/IEEF, international conference on Human-robor interaction, HRI '14, pages 33 3-40, New York. NY, USA. Association for Computing Machinery.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint (xXiv2108.0722:.
David L. Chen and Raymond J. Mooncy, 2011. Learn- ing to interpret natural language navigation instruc- tions from observations. In Proceedings of the Twenty- Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011. AAAI Press.
S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Confer- ence on Natural Language Processing of the AFNLP, pages 82-90, Suntee, Singapore. Association for Computational Linguistics.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. 2015. Microsoft COCO cap- tions: Data collection and evaluation server.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are Few-Shot learners. In Neural Information Processing Systems.
Gennaro Chierchia, Danny Fox, and Benjamin Spector. 2012. The Grammatical View of Scalar Implicatures and the Relationship between Semantics and Prag- matics. In Semantics: An International Handbook of Natural Language Meaning, volume 3, pages 2297- 2332, Mouton de Gruyter.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualivation: Making sen- tences stand-alone. Trunsactions of the Association for Computational Linguistics, 9:447-461.
Brian Buccola, Manuel Križ, and Emmanuel Chemla. 2022. Conceptual alternatives. Linguistics and Phi- losophy, 45(2))265-291.
Eve V Clark. 1988. On the logic of contrast. J. Child Lang., 15(2)3317-335
Daniel Casasanto and Gary Lupyan. 2015. All concepts are ad hoc concepts. In E Margolis and S Laurence, editors, The Conceptual Mind: New directions in the study of concepts. MIT Press.
Eve V. Clark. 2015. Common Ground. In The Hand- book of Language Emergence, pages 328-353, John Wiley & Sons, Ltd.
Eve V. Clark and Josie Bernicot. 2008 Repetition as ratification: How parents and children place informa- tion common ground. Journal of Child Language. 35(2):349-371
Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone. 1994. Animated conversation: rule-based generation of fa- cial expression, gesture & spoken intonation for mul- tiple conversational agents. In Proceedings of the s annual conference on Computer graphics and interactive techniques. SIGGRAPH '94, pages 413- 420, New York. NY, USA. Association for Comput- ing Machinery.
Herhert H Clark, 1996. Using Language. Cambridge University Press.
Herbert H Clark. 1997. Dogmas of understanding. Dis- course Processes, 23(3):567-598.
Herbert H. Clark and Susan E. Brennan. 1991. Ground- ing in communication. In Perspectives Socially Shared Cognition, pages 127-149. American Psycho- logical Association.
Joyce Chai, Maya Cakmak, and Candace Sidner. 2019, Teaching robots new tasks through natural interaction. In Interactive Task Learning. The MIT Press.

---

Herbert H Clark and Meredyth Krych. 2004. Speak- ing while monitoring addressees for understanding. J. Mem. Lang., 50(1 (0)68882-81.
Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi. 2021. Analysis of language change in collaborative instruction following. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2803-2811 I., Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Herbert H Clark and Catherine R Marshall. 1981. Def- inite knowledge and mutual knowledge. A. K. Joshi, B. Webber, & I. Sag, editor, Elements of dis- course understanding, pages (3-63.ccm Cambridge Uni- versity Press.
Constantin Eichenberg. Sidney Black, Samuel Wein- bach, Letitia Parcalabescu, and Anette Frank. 2022. MAGMA- - multimodal augmentation of generative models through adapter-based -base finetuning. In Find- ings the Association for Computational Linguistics: EMNLP 2022.
Herbert H Clark and Deanna Wilkes-Gibbe 1986. Referring as a Collaborative Process. Cognition, 22(1):1-39.
Philip R Cohen and Hector Levesque. 1990. Ratio- nal interaction as the basis for communication. In Philip R Cohen, Jerry Morgan, and Martha E Pollack.
Meta Fundamental AI Research Diplomacy Team FAIR. Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, al. 2022. Human- level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378 17666644110
editors, Intentions in Communication.
Philip Cohen, Jerry L Morgan, Morgan Jerry, and Martha E Pollack. 1990. Intentions in Communica-- fion. MIT Press.
Philip R. Cohen and C. Raymond Perrault. 1979. Ele- ments of plan-based theory of speech acts. Cogni- tive Science.
Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi. Peter Young. Cyrus Rashtchian, Julia Hock- emmaier, and David Forsyth. 2010. Every picture tells story: Generating sentences from images. In Proceedings of the European Conference on Com- puter Vision (ECCV), pages 15-29. Springer Berlin Heidelherg.
Reuben Cohn-Gordon and Noah Goodman. 2019. Lost in machine tramslation: A method to reduce meaning
loss.
Reuben Cohn-Gordon, Noah Goodman, and Christo- pher Potts. 2018. Pragmatically informative image captioning with character- level inference. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 2 (Short Pupers), pages 439-443, New Orleans, Louisiana. Association for Computational Linguis- tics.
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Jain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. 2019. Bayesian action decoder for deep multi-agent reinforcement learning. In Proceedings of the International Con- ference on Machine Learning (ICML), pages 1942- 1951.
Jordi Fortuny and Bernat Corominas-Murtra 2013. On the Origin of Ambiguity in Efficient Communica- tion. Journal of Logic, Language, and Information, 22(3):249-261 267. Publisher: Springer.
Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 326-335.
Danny Fox. 2007. Free Choice Disjunction and the Theory of Scalar Implicatures. In Uli Sauerland and Penka Stateva, editors, Presupposition and Implica- ture in Compositional Semantics, pages 71-120. Pal- grave Macmillan.
Harm De Vries, Dzmitry Bahdanau, and Christopher Manning. 2020. Towards ecologically valid re- search on language user interfaces. arXiv preprint arXiv:2007. 14435.
Danny Fox and Roni Katzir. 2011, On the characteri- zation of alternatives. Natural language semantics. 1971)887107 Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. GhessWhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference Computer Vision and Patler Recog- nition (CVPR), pages 5503-5512.
Michael C Frank and Noah D Goodman. 2012. Pre- dieting Pragmatic Reasoning Language Games. Science, 336(6084):998-99)
Michael Franke. 2013. Game theoretic pragmatics. Phi- losophy Compass, 8(3):269-284
Judith Degen. 2013. Alternatives in pragmatic reason- ing. University of Rochester.
Daniel Fried, Jacob Andreas, and Dan Klein, 2018a. Unified pragmatic models for generating and follow- ing instructions. In Proceedings of the 2018 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguisties: Human Lan- guage Technologies, Volume I (Long Papers), pages
Anca D Dragan, Kenton C T Lee, and Siddhartha S Srinivasa. 2013. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Con- ference on Human-Robol Interaction (HRI). IEEE.

---

1951-1963, New Orleans. Louisiana, Association for Computational Linguistics.
Computational Linguisties, pages 1895-1910, Flo- rence, Italy, Association for Computational Linguis- tics.
Daniel Fried, Justin Chiu, and Dan Klein. 2021. Reference-centric models for grounded collaborative dialogue. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 21 30-21477, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Dylan Hadfield-Menell, Anca Dragan, Picter Abbeel, and Stuart Russell. 2016. Cooperative inverse rein- forcement learning.
Dylan Hadfield- -Menell, Hadeield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. 2017. Inverse re- ward design. In Advances in Neural Information Processing Systems (NeurIPS).
Daniel Fried. Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis Louis-Phillppe Morency, Taylor Sang-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018b. Speaker-follower models for -nd-aaauagee navigation. In Proceedings of the 32nd International Conference on Neurul In- formation Processing Systems, pages 3318-3329.
David Harwath, Antonio Torralba, and James Glass. 2016. Unsupervised learning of spoken language with visual context. Advances in Neural Information Processing Systems, 29.
Matthew Hausknecht, Prithviraj Ammanabrolu, Mare- Alexandre Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 7903-7910.
Simon Garrod and Gwyneth Doherty. 1994. Conver-
sation, co-ordination and convention: An empirical investigation of how groups establish linguistic con- ventions. Cognition, 53(3):181-22
Robert D Hawkins, Michael C. Frank, and Noah D Good- man. 2020. Characterizing the dynamics of learn- ing in repeated reference games. Cognitive science, 44(6)=e12845.
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sel- lam. 2022. Repairing the cracked foundation: sur- vey of obstacles in evaluation practices for generated text.
Robert D. Hawkins, Michael Franke, Michael C. Frank, Adele E. Goldberg, Kenny Smith, Thomas L. Grif- fiths, and Noah D. Goodman. 2021. From partners to populations: A hierarchical Bayesian account of coordination and convention. Psychological Review.
Kallirroi Georgila, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: learning and evaluation. In Interspeech, pages 1065- 1068.
Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial de- scriptions. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Process- ing. pages 410-419, Cambridge, MA. Association for Computational Linguistics.
Robert D Hawkins, Michael Franke, Michael C Frank, Adele E Goldberg, Kenny Smith, Thomas L Grif- fiths, and Noah D Goodman. 2022. From partners to populations: A hierarchical bayesian account of coordination and convention. Psychological Review.
Robert XD Hawkins, Michael Frank, and Noah D Goodman. 2017. Fonvention-Formatioon in Iterated Reference Games. In CogSci.
Noah D Goodman and Michael C Frank. 2016. Prag- matic Language Interpretation as Probabilistic Infer- ence. Trends in Cognitive Sciences,
Caitlin Hilliard and Susan Wagner Cook. 2016. Bridg- ing gaps in common ground: Speakers design their gestures for their listeners. Journal of Experimen- tal Psychology: Learning. Memory, and Cognition, 42(1):91-103.
P Gorniak and D Roy. 2004. Grounded semantic com- position for visual scenes.
Herbert Grice. 1975. Logic and Conversation. In Speech Acts, pages 41-58. Brill.
Julia Bell Hirschberg. 1985a. A Theory of Scalar Impli- cature (Natural Languages, Pragmatics, Inference). Ph.D. thesis, University of Pennsylvania, Ann Arbor, United States.
Barbara J Grosz, Aravind K Joshi, and Scott Weinstein. 1995. Centering: framework for modeling the local coherence of discourse. Computational Linguistics, 21 2122)2203-
Julia Bell Hirschberg. 1985b. A Theory of Scalar Impli- cature (Natural Languages, Pragmatics, Inference). PhD Thesis, University of Pennsylvania.
Barbara Grosz and Candace L. Sidner. 1986. Atten- tion, intentions, and the structure of discourse. Com- putional Linguistics, 175-)417
Jerry R Hobbs. 1979. Coherence and coreference. Cog- nitive Science, 3(1):67-90
Janosch Haber. Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook Dataset: Building Common Ground through Visually-Grounded Grounded Dialogue. Proceed- ings of the 57th Annual Meeting of the Association
M Hodosh, P Young, and J Hockenmaier. 2013. Fram- ing image description as ranking task: Data, models and evaluation metrics. Journal of Artificial Intelli- gence Research, 47:853-899.

---

Laurence Horn 1984. Toward new taxonomy for pragmatic inference: Q-based and r- basee implica- ture. Meaning, form, and use in context: Linguistic applications, 11:42.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014, Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pages 787-798.
W. S. Horton and B. Keysar. 1996. When do speak- ers take into account common ground? Cognition. 59(1)911-117
Casey Kennington and David Schlangen. 2015. Simple learning and compositional application of perceptu- ally grounded word meanings for incremental refer- ence resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guisties and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 292-301, Beijing, China. Association for Computational Linguistics.
Julian Hough and David Schlangen. 2017. It's not what you do, it's how you do it: Grounding un- certainty for simple robot. In Proceedings of the 2017 ACM/EEE International Conference on Human-Roba Robot Interaction, HRI '7, pages 274-282, New York, NY, USA. Association for Computing Machinery.
Fereshte Khani. Noah D. Goodman, and Percy Liang, 2018. Planning, inference and pragmatics in sequen- tial language games. Transactions of the Association for Computational Linguistics, 6:543-555.
Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022. A fine- grained comparison of pragmatic language under- standing in humans and language models. arXiv preprint arXiv:2212.06801.
Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra- halli. 2018. A large self-annotated corpus for sar- casm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Nikolai liinkhh, Sina ZarrieB, and David Schlangen. 2019. Meetup! a corpus of joint activity dialogues in visual environment. In 23rd Workshop on the Semantics and Pragmatics of Dialogue.
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020. Will I sound like me? improving persona consistency in dialogues through pragmatic Self- Consciousness.
Gerhard Jäger. 2012. Game theory in semantics and pragmatics. Semantics: An international handbook of natural language meaning, 3:2487-2516.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse. Joel Z Leibo, and Nando De Freitas. 2019. Social influ- ence as intrinsic motivation for multi-agent deep re- inforcement learning. In Proceedings of the Inter- national Conference on Machine Learning (ICML). pages 3040-3049. PMLR.
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2021. Perspective-taking and pragmatics for generat- ing empathetic responses focused on emotion causes. In Proceedings of the 2021 Conference Empiri- cal Methods Natural Language Processing, pages 2227-2240 Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Hong Jun Jeon, Smitha Milli, and Anca D Dragan. 2020. Reward-rational (implicit) choice: A unifying for- malism for reward learning. In Advances in Neural Information Processing Systems (NeurIPS).
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Sak Zhang. Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. CoDraw: Col- laborative drawing as a testbed for grounded goal- driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495-6513 6513, Florence, Italy, Asso- ciation for Computational Linguistics.
Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language infer- ence models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 8690-8705, Online, Association for Computational Linguistics.
Hannah Kirk, Yennie Jun, Paulius Rauba, Gal Wachtel, Ruining Li, Xingjian Bai, Noah Broestl, Martin Doff- Sotta, Aleksandar Shtedritski, and Yuki M Asano. 2021. Memes in the wild: Assessing the generaliz- ability of the hateful memes challenge dataset. In Proceedings of the 5th Workshop Online Abuse and Harms (WOAH 2021). pages 26-35, Online. As- sociation for Computational Linguisties.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding language models to images for multimodal generation. In ICML.
Dan Jurafsky and James H. Martin. 2014. Speech and Language Processing. Pearson.
Noriyuki Kojima, Alane Suhr, and Yoav Artzi. 2021. Continual learning for grounded instruction genera- tion by observing human following behavior. Trans- actions of the Association for Computational Linguis- tics (TACL).
Hans Kamp and Uwe Reyle. 1993. From Discourse Logic, volume 42 of Studies in Linguisties and Philosophy. Springer.

---

Y. Alex Kolchinski and Christopher Potts. 2018. Rep- resenting social media users for sarcasm detection. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 1115-1121, Brussels, Belgium. Association for Com- putational Linguistics.
Stephen C Levinson. 1983. Pragmatics. Cambridge University Press.
David Lewis. 1969. Convention: A Philosophical Study.
Harvard University Press, Cambridge, MA.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeutlemoyer. 2020. BART: Denoising eqquence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Alexander Koller, Konstantina Garoufi, Maria Staudte. and Matthew Crocker. 2012. Enhancing referential success by tracking hearer gaze. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 30-39, Seoul, South Korea. Association for Computational Linguistics.
Alexander Koller. Kristina Striegnitz, Andrew Gar- gett, Donna Byron, Justine Cassell, Robert Dale, Johanna Moore, and Jon Oberlander. 2010. Re- port on the second NLG challenge on generating instructions in virtual environments (GIVE-2). In Proceedings of the 6th International Natural Lan- guage Generation Conference. The Association for Computer Linguistics.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to- end learning of negotiation dialogues. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453 Copenhagen, Denmark. Association for Computa- tional Linguistics.
Emiel Krahmer and Mariet Theune. 2010. Empir- ical Methods Natural Language Generation: Data-oriented Methods and Empirical Evaluation. Springer.
Elissa Li, Sebastian Schuster, and Judith Degen. 2021. Predicting Scalar Inferences From "Or" to "Not Both" Using Neural Sentence Encoders. In Proceedings of the Society for Computation in Linguistics, volume 4.
R. M. Krauss and S. Weinheimer. 1966. Concurrent feedback, confirmation, and the encoding of referents in verhal communication. Journal of Personality and Social Psychology, 4(3):43-334..
Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. 2022. Inferring rewards from language in context. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Elisa Kreiss, Noah D Goodman, and Christopher Potts. 2021. Concadia: Tackling image accessibility with context. arXiv preprint arXiv:2104.08370
Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. 2023. Computational language ac- quisition with theory of mind. ICLR.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. foom-across-rooni Mul- tilingual vision-and- language navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392-4412, Online. Association for Computational Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and- language tasks. Ad- vances in neural information processing systems, 32.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. 2019, A survey of reinforcement learning informed by nat- ural language. In IJCAI.
Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sag- nik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara I. Berg. 2013. Babytalk: understand- ing and generating simple image descriptions. IEFF Transactions on Pattern Analysis and Machine Intel- ligence, 35(12):2891-2900
Haley MacLeod, Cynthia I. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's experiences with computer-generatee captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 5988 5999..
Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, and Reut Tsarfaty. 2022. Draw me a flower: Process- ing and grounding abstraction in natural language. Transactions of the Association for Computational Linguistics (TACL).
Ellen Markman and Gwyn F Wachtel. 1988. Children's 's use of mutual exclusivity to constrain the meanings of words. Cognitive psychology. 2012):211117..
Angeliki Lazaridou and Marco Baroni. 2020. Emergent multi-agent communication in the deep learning era. arXiv e-prints, pages arXiv-2006.
Bill McDowell and Noah Goodman. 2019. Learning from omission. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 619-628, Florence, Italy, Association for Computational Linguistics.
Stephen Levinson. 2000. Presumptive meaning: The theory of generalized conversational implicature. MIT Press.

---

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2023. Linearly mapping from image to text space. In ICLR.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280-291.
Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Ya- maguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg. Tamara Berg, and Hal Daumé, III. 2012. Midge: Generating image descriptions from com- puter vision detections. In Proceedings of the Confer- ence of the Eurepean Chapter of the Association for Computational Linguistics (EACL), pages 747-756, Avignon, France. Association for Computational Lin- guistics.
Janet Pierrchumbert and Julia Bell Hirschberg. 1990. The meaning of intonational contours in the interpre- tation of discourse.
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connect- ing vision and language with localized narratives. In Computer Vision - ECCV 2020, pages 647-664. Springer International Publishing.
Christopher Potts. 2012. Goal-Driven Answers in the Cards Dialogue Corpus. In Proceedings of the 30th West Coast Conference on Formal Linguistics, pages
Will Monroe, Robert X.D. Hawkins, Noah D. Good- man, and Christopher Potts 2017. Colors in context: pragmatic neural model for grounded language understanding. Transactions of the Association
1-20. Cascadilla Proceedings Project.
Computational Linguisties, 5:325-338.
Zahar Prasov and Joyce Y Chai. 2008 What's ' in a gaze? the role of eye-gaze in reference resolution in mul- timodal conversational interfaces. In Proceedings of the 13th international conference Intelligent user interfaces, IUI '08, pages 20-29, New York, NY. USA. Association for Computing Machinery.
Apama S. Nadig and Julie C. Sedivy. 2002. Evidence of perspective-takee taking constraints in children's line reference resolution. Psychological Science, 13(4):329-336.
Anjali Narayan- Nareyan-Chen, Prashant Jayannavar, and Ju- lia Hockenmaier. 2019. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5405-5415, Florence, Italy. Association for Computational Linguistics.
Yewen Pu, Kevin Ellis, Marta Kryven, Josh Tenenbaum, and Armando Solar-Lezama: 2020. Program synthe- with pragmatic communication. Neural Informa- tion Processing Systems, 15:11-22113555
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Benjamin Newman, Reuben Cohno, and Christo- pher Potts. 2020. Communication--asee evaluation for natural language generation. Proceedings of the Society for Computation in Linguistics 2020, pages I 116, New York, New York. Association for Computational Linguistics.
Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. 2022. Dalle-2 is seeing double: flaws in word- to-concept mapping in text2image models. arXiv preprint arXiv:2210.
Allen Nie, Reuben Cohn-Gordon, and Christopher Potts. 2020. Pragmatic issue-sensitive image captioning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1924-1938, Online. Association for Computational Linguisties.
Verena Rieser and Oliver Lemon. 2008. Learning effec- tive multimodal dialogue strategies from Wizard-of- Oz data: Bootstrapping and evaluation. In Proceed- ings of ACL-08: HLT, pages 338-646, Columbus, Ohio. Association for Computational Linguistics.
Frans A Oliehoek and Christopher Amato. 2016. A concise introduction decentralized POMDPs.
Springer.
Verena Rieser and Oliver Lemon. 2009. Natural lan- guage generation as planning under uncertainty for spoken dialogue systems. In Proceedings of the 12th Conference of the Europaan Chapter of the ACL (EACL 2009), pages 183-691, Athens, Greece. Asso- ciation for Computational Linguistics.
Tim Paek and Eric J Horvitz. 2000. Conversation as action under uncertainty. Uncertainty in Artificial Inteliigence.
Gabriel Parent and Maxine Eskenazi. 2010. Lexical entrainment of real users in the let's 's go spoken dialog system. Interspeech.
Verena Rieser and Oliver Lemon. 2011. Reinforcement Learning for Adaptive Dialogue Systems: A Data- driven Methodology for Dialogue Management and Natural Language Generation. Springer Science & Business Media.
Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bow- man, and Tal Linzen. 2021. NOPE: A corpus of naturally-occuriin presuppositions in English. In Proceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 349-366, Online. Association for Computational Linguistics.
Seymour Rosenberg and Bertram D. Cohen. 1964. Speakers' and listeners' processes in word- communication task. Science, 145(3637):2001- 1203.

---

Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNTPP) pages 2230-2240, Hong Kong, China. Association for Computational Linguistics.
Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Cote. Yonatan Bisk, Adam Trischler. and Matthew Hausknecht. 2021. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In International Conference on Learning Representa- lions.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung- Moya Chen, Kushal Arora, Joshua Lane, et al. 2022. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint 22X
Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022. Large language models are not zero-shot communi- cators.
Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of Turn-Taking for conversation.
Candace L Sidner, Christopher Lee, Cory D Kidd, Neal Lesh, and Charles Rich. 2005. Explorations in gagement for humans and robots. Artificial Intelli- gence, (66(1))140-16)) 164.
Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large Ims. arXiv preprint arXiv:2210. 133312
Ricard Solé and Luís F. Seoane. 2015. Ambiguity in language networks, The Linguistic Review, 32(1):5- 35.
Emanuel A Schegloff. 1968. Sequencing in con- versational openings. American Anthropologist. 1066):1075-1055.
Robert Stalnaker. 2002. Common Ground, Linguistics and Philosophy, 2555):001-711
David Schlangen, Timo Baumann, and Michaela Atterer. 2009. Incremental reference resolution: The task, metrics for evaluation, and bayesian filtering model that is sensitive to disfluencies. In Proceedings of the SIGDIAL 2009 Conference, pages 30-37, London, UK. Association for Computational Linguisties.
Robert C. Stalnaker. 1978. Assertion. In Pragmatics, pages 315-332. Brill.
Lue Steels and Tony Belpaeme. 2005. Coordinating per- ceptually grounded categories through language: case study for colour. Behavioral and Brain Sciences, 28(4) 28(4):669-89 -9; discussion 489-529.
Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5387-540) Online. Association for Computational Linguistics.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
John R Searle. 1969. Speech Acts: An Essay in the Phi- losophy of Language. Cambridge University Press.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. 2021. Collaborating with humans without human data. In Advances in Neural Information Processing Systems, volume 34, pages 14502- -550511515 Curran Associates, Inc
Julie C Sedivy, Michael Tanenhaus, Craig G Cham- bers, and Gregory N Carlson. 1999. Achieving in- cremental semantic interpretation through contextual representation. Cognition, 11(2):1991477.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypemymeed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). pages 2556 -5665.5.
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem. Iris Zhang, and Yoav Artzi. 2019a. Executing instructions in situ- ated collaborative interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCCNPP) pages 2119-2130, Hong Kong, China. Association for Computational Linguistics.
Sheng Shen, Daniel Fried, Jacob Andreas, and Dan Klein, 2019. Pragmatically informative text genera- tion.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi. Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10740-10749.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi 2019b. A corpus for reasoning about natural language grounded in pho- tographs. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 6418-6428, Florence, Italy. Association for Computational Linguistics.

---

Chen Sun, Austin Myers, Carl Vondrick, Kevin Mur- phy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learn- ing. In Proceedings of the IEEE/CVF Interationall Conference on Compuler Vision, pages 7464-7473.
Hong Kong, China, Association for Computational Linguistics.
Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning Cross-Modality encoder representations from trans- formers.
Gabriella Vigliocco, Pamela Perniss, and David Vin- son. 2014. Language multimodal phenomenon: implications for language learning, processing and evolution.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth J. Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the Twenty-Fiifh AAAI Conference on Artificial Intelli gence, AAAI 2011, San Francisco, Califormia, USA, August 7-11, 2011. AAAI Press.
Adam Vogel, Max Bodoia, Christopher Potts, and Daniel Jurafsky. 2013a. Emergence of Gricean max- ims from multi-agent decision theory. In Proceed- ings of the Conference of the North American Chap- ter of the Association for Computational Linguistics (NAACL). pages 1072-1081, Atlanta, Georgia. Asso- ciation for Computational Linguistics.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019. Vision-and-Dialog navi- gation. In Proceedings of the Conference on Robot Learning.
Adam Vogel and Dan Jurafsky. 2010. Learning to fol- low navigational directions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 806-814.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. arXiv preprint urX2v:220..0366.
Adam Vogel, Christopher Potts, and Dan Jurafsky- 2013b. Implicatures and nested beliefs in approx- imate decenntaalized-pomee Annual Meeting of the Association for Computational Linguistics (ACI.), pages 74-80.
In Proceedings of the
Michael Tomasello. 2005. Constructing a language: A usage-based theory of language acquisition. Harvard university press.
M A Walker, S Whittaker, A Stent, P Maloor, J Moore, M Johnston, and G Vasireddy, 2004, Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science, 28(5):81
David Traum and Jeff Rickel. 2002. Embodied agents for multi-party dialogue in immersive virtual worlds. In Proceedings of the first international joint confer- ence on Autonomous agents and multiagent systems: part 2, AAMAS '02, pages 766-773, New York, NY, USA. Association for Computing Machinery.
Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. PARADISE: A frame- work for evaluating spoken dialogue agents. In 35th Annual Meeting of the Association for Com- putational Linguistics and 8th Conference of the Eu- ropean Chapter of the Association for Computational Linguisties, pages 27 171-280, Madrid, Spain. Associa- tion for Computational Linguistics.
David R Traum. 1994. A computational theory of grounding in natural language conversation. Techni- cal report, Rochester University Department of Com- puter Science.
Takuma Udagawa and Akiko Aizawa. 2019. A natural language corpus of common grounding under contin- uous and obbteally-bbeervalle context. In Proceed- ings the AAA/ Conference on Artificial Intelligence. volume 33, pages 7120-7127.
Pei Wang, Junqi Wang, Pushpi Paranamana, and Patrick Shafto. 2020. A mathematical theory of cooperative communication. Advances in Neural Information Processing Systems (NeurIPS), 33:17582-17553
Takuma Udagawa, Takato Yamazaki, and Akiko Aizawa. 2020. linguistic analysis of visually grounded di- alogues based on spatial expressions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 750-765, Online. Association for Computational Linguistics.
Sida I. Wang. Samuel Ginn, Percy Liang, and Christo- pher D. Manning. 2017. Naturalizing programming language via interactive learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 929-938, Vancouver, Canada. Association for Computational Linguistics.
Jack Urbanck, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau. Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Ja- son Weston. 2019. Learning to Speak and Act in a Fantasy Text Adventure Game. In Proceedings of the Conference on Empirical Methods Natu- ral Language Processing (EMNLP), pages 673-683,
Sida I. Wang, Percy Liang, and Christopher D. Manning, 2016. Learning language games through interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pages 2368 -2788 Berlin, Germany. Association for Computational Linguistics.

---

Xuewei Wang, Weiyan Shi, Richard Kim. Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Per- suasion for good: Towards personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5635-5649, Florence, Italy, Association for Computational Linguisties.
On the evaluation of Vision Vision-and-Lee and-Language naviga- tion instructions. In Proceedings of the Conference of the European Chapter of the Association for Com- putational Linguistics (EACL).
Victor Zhong, Tim Rocktäschel, and Edward Grefen- stette. 2020. Rtfi:: Generalising to new environment dynamics via reading. In ICLR, pages 1-17. ICLR.
Bonnie Webber, Matthew Stone, Aravind Joshi, and Al- istair Knott. 2003. Anaphora and discourse structure. Computional Linguistics, 2945)554--
George K. Zipf. 1949. Human behavior and the princi- ple of least effort. Addison Wesley Press.
Bonnie Lynn Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107-135.
Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech and Language. 21(2):393-422.
Ludwig Wittgenstein. 1953. Philosophical Investiga- tions. Basil Blackwell.
Si On Yoon and Sarah Brown-Schmidt. 2019. Audi- ence Design in Multiparty Conversation. Cognitive Science, 137774
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2:67-78.
Steve Young, Milica Gašié, Blaise Thomson, and Ja- son D. Williams. 2013. POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5): 160-1179.
Zhou Yu, Dan Bohus, and Eric Horvitz. 2015. Incre- mental coordination: Attention- Cenoori speech pro- duction in a physically situated conversational agent. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 402-406, Prague, Czech Republic. Association for Computational Linguistics.
George Yule. 1996. Prugmatics, 1 edition. Oxford Introduction to Language Study. Oxford University Press.
Noga Zaslavsky, Jennifer Hu, and Roger P. Levy. 2021. A Date-Distootiin View of Human Pragmatic Rea- soning. In Proceedings of the Society for Compu- tation Linguistics 2021, pages 347-348, Online. Association for Computational Linguistics.
Luke Zettlemoyer, Brian Milch, and Leslie Kaelbling. 2008. Multi-agent filtering with infinitely nested beliefs. Advances in Neural Information Processing Systems (NeurIPS), 21.
Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alexander Ku, Jason Baldridge, and Eugene Ie. 2021.

---

A Types of Context
Norms of Interaction. As language is social behavior, speakers and listeners typically abide by a set of norms. For example, Grice (1975) ar- gues that it is generally understood that conver- sational partners act cooperatively and rationally. Grice also proposes set of maxims that govern communication- speakers should be con- cise, informative, and relevant. These norms in turn give rise variety of nonliteral inferences known as conversational implicatures. Suppose, for example, Alice says to Bob: "Carl ate some of the cookies that we baked for the party". Bob likely draws the inference that Carl did not eat all of the cookies, even though the literal meaning of the ut- terance - that Carl ate at least one of the cookies
In this section, we outline the broad types of context that lead to pragmatic enrichment of lan- guage, and point readers to Levinson (2000), Birner (2012), or Yule (1996) for a more comprehensive discussion. In this paper we focus mainly on vi- sual and embodied contexts, for several reasons. First, human communication is typically situated in settings with modalities beyond language, which makes it important capture in order to build NLP models that interact naturally with humans in the world. Indeed, recent work has argued that ground- ing is an essential component of language under- standing (c.g., Bisk et al., 2020; Bender and Koller, 2020). Second, visual and embodied settings intro- duce enough complexity to elicit interesting linguis- tic behaviors and serve challenge for models, while still allowing researchers control experi- mental aspects of the tasks. Finally, there has been a rapid increase in research on multimodal lan- guage learning, which makes studying pragmatics in these models and tasks timely and relevant.
is logically compatible with such scenario. This inference can be explained in the following way: if Alice knows that Carl ate all the cookies, and if she wants be informative, then she would have said "Carl ate of the cookies" instead.
Goals and Joint Actions. In addition to gen- eral norms of interaction, the particular social or task-related goals that elicit a linguistic expres- sion can affect its meaning. The theory of speech acts (Searle, 1969; Austin, 1975) frames utterances (e.g.. "please stand up") as actions several lev- cls: locutionary, the utterance itself; illocutionary, the intention (e.g.. asking the listener to stand up); and perlocutionary, the actual effect that the ac- tion has the world (e.g., the listener stands up). Context can have strong effects on the illocutionary and perlocutionary levels. This is particularly true for formal speech acts which can only take effect under felicity conditions, e.g. making a promise, or performing a marriage, but also occurs in com- monplace situations e.g., asking "Did you get my email?" might be an indirect request to reply, or direct question while debugging an internet con- nection. More generally, interlocutors typically recognize that they are undertaking joint activities together with their partners (Clark, 1996) and try to collaboratively plan and act to coordinate on and realize the relevant goals. These shared goals provide source of context that enriches language.
A.1 Common Ground
To communicate successfully, speakers and lis- teners need to maintain a shared set of informa- tion, taken collectively to be common ground (e.g.. Lewis, 1969; Stalnaker, 1978; Clark and Bren- nan, 1991; Traum, 1994; Stalnaker, 2002: Clark, 2015). A large body of work has demonstrated that humans produce and comprehend language in ways that depend on assumptions about the knowledge of their communicative partners (e.g., Krauss and Weinheimer, 1966; Horton and Keysar, 1996; Nadig and Sedivy, 2002; Clark and Bernicot, 2008; Hilliard and Cook, 2016; Yoon and Brown- Schmidt, 2019; Hawkins et al., 2021). Even in one-shot encounters where there minimal partner- specific knowledge, the success of computational models of pragmatics (Frank and Goodman, 2012: Goodman and Frank, 2016) suggests that humans leverage a rich set of shared assumptions in prag- matic communication from broad expectations that their partners abide by cooperative principles (c.g., Grice, 1975; Horn, 1984) to fine-grained knowledge of the potential utterances, meanings, and utterance-meaning mappings under joint con- sideration. Below, we discuss some key elements common ground that give rise to pragmatically enriched meanings in naturalistic communication.
Common Knowledge. Interpretation is aided by prior information that interlocutors bring in- teraction. For example, suppose Alice asks "What color was the woman's 's scarf?" and Bob answers "green". If Bob is a fashion designer with a keen eye for color palettes, this might implicate that

---

the scarf was a rather prototypical shade of green, and not olive green or chartreuse. On the other hand, if Bob doesn't know many specific color terms, Alice doesn't have grounds to infer that Bob meant to refer to specific subspace of green. The world knowledge and commonsense relationships shared by conversational partners can also give rise to scalar implicatures formed by ad-hoc ordering relationships (Hirschberg, 1985b) and lead to peda- gogic behavior (Chai et al., 2019).
and Marshall, 1981). Reference intepretation can also be affected by the location of the speaker and hearer in the world (Birner, 2012), and can involve physical analogues of implicature (e.g., the black one might be a good description for a dark grey ob- ject if all other visible objects are lighter) (Golland et al., 2010; Udagawa et al., 2020).
Embodied. Facial expressions, gaze, and ges- tures (Cassell et al., 1994; Traum and Rickel, 2002; Sidner et al., 2005; Prasov and Chai, 2008; Bohus and Horvitz, 2010; Koller et al., 2012; Yu et al., 2015) can aid interpretation if they are available, e.g., a speaker first making eye contact with lis- tener, then looking at an intended object. Speak- can issue corrections if they are able to ob- serve a listener carrying out actions (Clark and Krych, 2004; Koller al., 2010; Thomason et al., 2019; Suhr et al., 2019a), and the physical move- ments of the listener can intentionally convey un- certainty (Hough and Schlangen, 2017) and in- tent (Dragan et al., 2013). Physical properties of the environment and tasks (Chai et al., 2019) and the capabilities of the speaker and listener (Chai et al., 2014), also affect the interpretation and gener- ation of commands and requests c.g.. the classic pragmatic example Can you pass the sal1", which typically is an indirect request when spoken to a person, may have a literal interpretation when spo- ken to a robot with a faulty gripper.
Communication is most of- ten not a one-shot utterance, but instead unfolds over time. As a document or a conversation pro- ceeds, the common ground can be updated with new information from the discourse context. At basic level, discourse context includes previously- established information which can be referred to later on, whether explicitly (e.g., a dog bounded into the roon... barked) or implicitly (e.g., a dog bounded into the room... Sam was surprised). Information can also be introduced implicitly, for example through presupposition and accommoda- tion (c.g., Alex stopped smoking presupposes that information
Discourse Context.
Alex smoked). can in some cases (implicature) also be reinforced denied, e.g., Carl ate some of the cookies; in- deed, he ate all of them!.
A.2 Multimodal Context
So far, have discussed aspects of context given by social or linguistic factors. While all of the above types of context also arise in grounded and multimodal settings, the physical context in which communication is situated also plays an additional component in deriving linguistic meaning. As men- tioned above, we focus on visual and embodied contexts in this paper, as these contexts reflect nat- uralistic communication while also allowing for fine-grained experimental control.
B Unimodal Pragmatics
Although we primarily focus on the role of prag- matics in grounded environments, several text-only tasks that emphasize specific pragmatic phenom- ena also exist. For example, IMPPRES (Jeretic et al., 2020) and NOPE (Parrish et al., 2021) are benchmark datasets designed to test whether large language models can reliably predict implicatures and presuppositions, respectively. Similarly, Schus- ter et al. (2020) and Li et al. (2021) evaluate the ability of sentence encoding models to predict the rate at which humans draw scalar implicatures. Other datasets like the Self-Annotated Reddit Cor- pus (SARC) for sarcasm detection (Khodak et al., 2018) may also be viewed as pragmatic in na- ture (Kolchinski and Potts, 2018). While these datasets are limited unimodal text, they have two main advantages over many multimodal tasks: (1) many unimodal pragmatic datasets are naturally- occurring, resulting in larger datasets with more realistic language, and (2) all of these datasets fo-
Visual. Visual context serves to disambiguate and enrich the language of meaning on multiple levels. On a level close to semantics, visual context can disambiguate word senses: e.g., "bank" likely has a different meaning in the caption of a photo of river than in photo of city street. Referring expressions (c.g., the red one) often can only be re- solved in visual context, and deictic expressions, like English here, there, this and that, are frequently used in language to individuate referents in their immediate context, relying on mutual knowledge of what the speaker and listener can see (Clark

---

cus on specific pragmatic phenomena, such pre- supposition. We suggest that future work on mul- timodal pragmatics should take inspiration from these properties and build larger and more targeted datasets.
A separate body of work has investigated situ- ated language understanding through interactive fiction (IF) games (e.g., Ammanabrolu and Riedl, 2021; Hausknecht et al., 2020; Urbanek et al., 2019). IF games offer a framework for investigat- ing goal-driven linguistic behaviors in a dynamic, richly structured world. Players observe natural- language descriptions of the simulated world, take actions via natural language. and receive scores based on their actions. The simulations are also par- tially observable, in that players must reason about the unerlying world state through incomplete tex- tual descriptions of immediate surroundings. In this way, IF games avoid some of the practical issues grounding in visual environments, while still re- quiring actions to situated in rich, dynamic con- texts. Furthermore, Shridhar et al. (2021) demon- strate that commonsense priors learned through IF games can be leveraged for better generalization in visually grounded environments, suggesting that text-only games induce representations that can be adapted multimodal settings.


Reviews:
[{"review_id": "hb7Ns4SEtE", "reviewer": "Reviewer_ED4y", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We’re encouraged that you appreciated our account of pragmatic phenomena and tasks and found the cited work to be comprehensive.\n\nThe primary goal of our paper is to provide an account of pragmatics, targeted toward an NLP audience. As a result, the bulk of our paper does focus on pragmatics. However, while a linguistically-oriented introduction to pragmatics would typically focus on specific types of pragmatic phenomena (e.g., implicature, presupposition, deixis), our work emphasizes the functional roles of pragmatics (sec 2.2) and how they relate to grounded language tasks. As noted in your summary, most of the NLP community is currently focused on machine learning, so we hope this alternative perspective will be useful to practitioners thinking about developing NLP systems, especially in grounded domains.\n\nWe would like to note that, while we agree that our paper does spend most of its length (7 pages) talking about pragmatics, much of this discussion is also highly compatible with LLMs. LLMs have already begun to be put to use to tackle the grounded pragmatic tasks we outline in Table 1 and Section 3. For example, Kojima et al. 2021 fine-tunes a pre-trained GPT-2 model to adapt to people for instruction generation in CerealBar [1]. The pragmatic methods in Section 4 are also quite compatible with LLMs, e.g. Liu et al. 2023 combine RSA with meta-learning to apply GPT models in an image reference game setting [2]; Meta et al. 2022 use a large BART model in conjunction with a multi-agent planning procedure in the grounded dialogue game of Diplomacy [3]. As vision-to-LLM adapters (e.g. LIMBER, Flamingo, Fromage, MAGMA [4-7]) continue to improve, we expect to see more work applying LLMs as components of pragmatic models for these grounding pragmatic tasks. Our discussion section (Section 5) does touch on this LLM+pragmatics synthesis, and we will use the extra page in any future version of the paper to expand this discussion.\n\nWe also provide some discussion of datasets for evaluating unimodal pragmatics (primarily in LLMs) in Appendix B.\n\n[1] Kojima et al. Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior. TACL 2021\n\n[2] Liu et al. Computational Language Acquisition with Theory of Mind. ICLR 2023\n\n[3] Meta et al. Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning. Science 2022\n\n[4] Merullo et al. Linearly Mapping from Image to Text Space. ICLR 2023\n\n[5] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022\n\n[6] Koh et al. Grounding Language Models to Images for Multimodal Inputs and Outputs. ICML 2023\n\n[7] Eichenberg et al. MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning. EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a useful overview of pragmatics phenomena and relevant task settings combined with a comprehensive reference list. The authors thematize the lack of pragmatics in current LLMs, and, in my opinion, rightly suggest to study and further develop (multimodal) LLMs in tandem with pragmatics modelling. ", "reasons_to_reject": "The paper makes a strong claim to take pragmatics serious when developing (multimodal) LLMs. It gives a good account of pragmatics, however, remains vague on how pragmatically informed tasks should be utilized to further develop LLMs. This imbalance it reflected in the room given to the topic of pragmatics  (~ 7 pages) versus the considerations related to pragmatic modelling and LLms (less than one column).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to remind the NLP community (which today is heavily focusing on machine learning) of the fact that language understanding is strongly related to language use in situated interactive scenarios. The authors provide an overview of pragmatics phenomena and list a range of task settings where different aspects of grounded dialogue occur. The authors argue that LLMs should be studied in the light of their (lack of) capabilities to handle pragmatic phenomena. ", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "50xe2VXBgd", "reviewer": "Reviewer_tVhy", "summary": "", "questions": "- Would you say that the difference between typed language and spoken language is irrelevant? Or is the focus on typed language tasks due to methodological considerations?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review! We’re glad that you found the paper valuable to the EMNLP community and appreciated our taxonomy of tasks and datasets, as well as our recommendations for future task design. In response to specific comments:\n\n> existing benchmarks measure literal meaning\n\nWe agree that we presented a simplified distinction between semantics and pragmatics, and that unimodal tasks like question answering can leverage rich pragmatic phenomena. In practice, however, we suggest that many datasets for unimodal tasks like natural language entailment or factoid QA focus primarily on semantic understanding, rather than pragmatics.  \n\n> whether existing methods…simply \"scale up\" to interactive settings\n\nWe agree with the reviewer that this is a very important, and open question, and we do  mention it in our discussion/future work section (lines 660-665). However, since our work is primarily a survey paper, and there’s not (yet) a good answer to this question in the literature, we don’t spend much time on it. We will emphasize this question, and that we view it as still open, in any future version of the paper. \n\n> does not mention the active research fields that look at situated interaction\n\nThank you for pointing this out. While we did not include references to conversational analysis in the main body due to space restrictions, we did mention them in the limitations section (lines 758-768), and we cite some work from HRI briefly in the main body (Dragan et al. 2013, line 1945). We are happy to move some of this supplementary to the main text / expand the citations and descriptions given more space in a future version.\n\n>  implicit claim that there is no qualitative difference between functional tasks and interactive settings\n\nWe didn’t intend to make such a claim, implicitly or explicitly. In contrast, we argue that the richer, interaction-heavy domains like grounded dialogue often elaborate a richer set of pragmatic phenomena (e.g., lines 603-604). We are happy to make this more explicit in the paper, or clarify any particular unclear parts that might have unintentionally made this claim.\n\n> Would you say that the difference between typed language and spoken language is irrelevant? Or is the focus on typed language tasks due to methodological considerations?\n\nWe wouldn’t consider the distinction irrelevant, but we are aware of relatively little computational work that has focused on spoken language pragmatics in grounded settings, with a few exceptions being Harwath et al. 2016 [1] and Sharma et al. 2018. There also some great linguistics work at the intersection of speech and pragmatics, though, e.g., focused on prosody (Pierrehumbert & Hirschberg 1990 [3]) and incrementality (Sedivy et al. 1999 [4]).\n\n[1] Harwath et al. Unsupervised Learning of Spoken Language with Visual Context. NeurIPS 2016\n\n[2] Sharma et al. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. ACL 2018\n\n[3] Pierrehumbert & Hirschberg. The Meaning of Intonational Contours in the Interpretation of Discourse. Intentions in Communication, 1990\n\n[4] Sedivy et al. Achieving incremental semantic interpretation through contextual representation. Cognition, 1999", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- brings a familiar argument from linguistics and psycholinguistis (language primarily is an instrument to be used, in particular contexts and particular settings) to the attention of NLP researchers, who tend to overlook this\n\n- provides a useful systematization of tasks / datasets, connection features of the task to types of relevant context\n\n- provides useful guidance for the design of challenging next tasks\n", "reasons_to_reject": "- does not mention the active research fields that look at situated interaction, outside of narrowly defined NLP (e.g., in HRI, in conversational analysis); only mentions the classics\n\n- underplays the role of interaction in creating the kind of dynamic contexts that mark situated language use; makes no argument for the implicit claim that there is no qualitative difference between functional *tasks* like captioning and (single shot) instruction following on the one hand and interactive settings like \"grounded goal-oriented dialogue\" on the other\n", "questions_to_authors": "- Would you say that the difference between typed language and spoken language is irrelevant? Or is the focus on typed language tasks due to methodological considerations?\n", "paper_topic_and_main_contributions": "This position paper argues that NLP has so far not looked closely enough at language understanding tasks where there is a rich context to which the used language implicitly makes reference. The paper goes into some types of such contexts, and discusses tasks that the authors argue would bring them out and make them available for current methods.\n\nThe paper presents a survey of foundational work that could especially be useful for researchers coming into this field from outside of linguistics (where of course the role of pragmatics has been known at least since Peirce, Bühler, and Morris).\n\nThe motivation given in the introduction rests at least rhetorically a bit on an outdated conception of the division of labour between semantics and pragmatics -- in the claim that existing benchmarks measure literal meaning (line 042), which of course QnA datasets, or any dataset with material that goes beyond a single sentece, do not do, as pragmatic enrichment phenomena are present even in purely linguistic contexts -- but that is a minor point.\n\nAlso, the paper does not address (or even just mention) the question -- not at all obvious to this reviewer -- of whether existing methods based on approximating functions from data of example function applications simply \"scale up\" to interactive settings.\n\nHowever, overall, I think this paper will be very valuable to the EMNLP community, and I recommend acceptance.\n", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- It seems to me that the ReferItGame papers would fit perfectly in this taxonomy, as the setting brilliantly reduces interaction to its minimal form (but not beyond), in that the speaker knows that a listener is present. It also predates much of the work that is cited.\nKazemzadeh, S., Ordonez, V., Matten, M., & Berg, T. L. (2014). ReferItGame : Referring to Objects in Photographs of Natural Scenes. In EMNLP 2014 (pp. 787–798).\n\n- For a more comprehensive argument about the role of interaction (rather than just \"pragmatics\"), could cite:\nDingemanse, M., Liesenfeld, A., Rasenberg, M., Albert, S., Ameka, F.K., Birhane, A., Bolis, D., Cassell, J., Clift, R., Cuffari, E., De Jaegher, H., Novaes, C.D., Enfield, N.J., Fusaroli, R., Gregoromichelaki, E., Hutchins, E., Konvalinka, I., Milton, D., Rączaszek-Leonardi, J., Reddy, V., Rossano, F., Schlangen, D., Seibt, J., Stokoe, E., Suchman, L., Vesper, C., Wheatley, T. and Wiltschko, M. (2023), Beyond Single-Mindedness: A Figure-Ground Reversal for the Cognitive Sciences. Cognitive Science, 47: e13230. https://doi.org/10.1111/cogs.13230\n"}}, {"review_id": "ZnSDHJ390O", "reviewer": "Reviewer_3CNV", "summary": "", "questions": "- Suggestion: be careful about the word \"context\" because in most NLP papers it means textual/lexical context, but the authors mean physical or world context. Please be clear. \n- The paper focuses on pragmatics, but some could argue that there is a lot of overlap with semantics. Can the authors delineate further the semantics-pragmatics interface and how they related to each other?\n- There is a bit of an abrupt jump from the end of section 2 to the start of section 3. Please make the transition smoother. \n- I think it would be helpful to mention the claim that LLMs have emergent behaviors, which may include pragmatic qualities. Can pragmatics follow from textual learning?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We’re glad that you found our paper well-written and important and appreciated our focus on linguistic phenomena. In response to specific comments:\n\n> be careful about the word \"context\"\n\nThis is a good point. We provide some discussion of types of context in Sec 2.1 / Appendix A but will aim to be more precise about the type of context we refer to (typically physical or world context)throughout the paper.\n\n> Can the authors delineate further the semantics-pragmatics interface and how they related to each other?\n\nThe boundary between semantics and pragmatics was also mentioned by tVhy. We agree that the boundary between semantics and pragmatics is quite grey, and if we had to take a stance, our position largely holds with Birner 2012: semantics largely concerns truth-conditional, or context-independent, meaning of an utterance; while pragmatics largely concerns context-dependent meaning, beyond truth-conditional. The truth-conditional & context-independent definitions of the boundary do not produce the same distinction on all phenomena (e.g. anaphoric pronouns; conventional implicature), but as we are most interested in this paper in grounded phenomena which are both heavily-contextual and deal with non truth-conditional meaning, we chose to not delineate the boundary, as it seemed less relevant to our intended audience. \n\n> Can pragmatics follow from textual learning?\n\nThis was also mentioned by reviewer tVhy. Our response: “We agree with the reviewer that this is a very important, and open question, and we do mention it in our discussion/future work section (lines 660-665). However, since our work is primarily a survey paper, and there’s not (yet) a good answer to this question in the literature, we don’t spend much time on it. We will emphasize this question, and that we view it as still open, in any future version of the paper.\" \n\nThank you also for the suggested reference! It looks very interesting and relevant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic is beyond timely. Taking a long, hard look at pragmatics is long overdue. \n- The paper is well-written, easy to follow with plenty of references and citations to back up any claims. \n- The focus is actually on an important aspect of language, not just on models. ", "reasons_to_reject": "- There's no new proposed model or evaluation, which is traditional at EMLNP, but I don't think that should be a reason to reject. ", "questions_to_authors": "- Suggestion: be careful about the word \"context\" because in most NLP papers it means textual/lexical context, but the authors mean physical or world context. Please be clear. \n- The paper focuses on pragmatics, but some could argue that there is a lot of overlap with semantics. Can the authors delineate further the semantics-pragmatics interface and how they related to each other?\n- There is a bit of an abrupt jump from the end of section 2 to the start of section 3. Please make the transition smoother. \n- I think it would be helpful to mention the claim that LLMs have emergent behaviors, which may include pragmatic qualities. Can pragmatics follow from textual learning?", "paper_topic_and_main_contributions": "This paper reviews how pragmatics are under-studied, yet important in how they affect language grounding. Though not the typical paper that introduces a new model or dataset, this paper is quite comprehensive in its depth and breadth of the relevant literature. ", "excitement": 5, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The authors were thorough, but a very recent paper that I came across I think is relevant here. Though the author doesn't mention pragmatics, there are some similar arguments that I think are relevant to the discussion:\n\nhttps://arxiv.org/abs/2307.04518"}}]

Author rebuttal:

Meta-review: Reviewers tVhy and 3CNV provide a positive assessment of validity and are quite excited about the paper, whereas Reviewer ED4y gives low scores on both dimensions.

Synthesizing the reviews, the following major strengths or weaknesses were mentioned:

Strengths

- provides a useful overview of pragmatics (R1, R3) for a community that may overlook relevant key properties of language (R2, R3), provinces guidance for relevant challenging tasks (R2)

Weaknesses

- remains vague on how pragmatics could actually come to play in developing LLMs (R1)
perceived relatively narrow focus in the way linguistic interaction is framed and conceived (R2)

Reconciling the disparate scores given by reviewers:
The weakness mentioned by R1 (ED4y) – i.e., remaining vague on how pragmatics can come to play in developing LLMs – appears to be largely related to excitement rather than soundness, as it does not undermine the validity of the conclusions drawn by the paper or the arguments made in it. Thus, I believe their low soundness score (1) is not well justified by the review and can be discounted.
On the other hand, the three excitement scores (2,4,5), while variable, are supported as subjective assessments by the three reviewers.
Decision: Accept-Findings