Introduction
Our success as a social species depends on our ability to understand, and be understood by, different communicative partners across different contexts. Theory of mind—the ability to represent and reason about others' mental states—is considered to be the key mechanism that supports such context-sensitivity in our everyday social interactions. Being able to reason about what others see, want, and think allows us to make more accurate predictions about their future behavior in different contexts and adjust our own behaviors accordingly BIBREF0 . Over the past two decades, however, there has been sustained debate over the extent to which adults actually make of use theory of mind in communication.
On one hand, accounts of language use in the tradition of BIBREF1 and BIBREF2 , BIBREF3 implicitly assume a fundamental and pervasive role for theory of mind mechanisms. The meaning of an utterance is established against a backdrop of inference, intention, and common ground: knowledge that is taken to be shared by both parties BIBREF4 , BIBREF5 . This view of adults as natural mind-readers is consistent with extensive evidence from the psycholinguistics literature: for instance, we spontaneously calibrate our referential expressions to our intended audiences BIBREF6 and make use of partner-specific history BIBREF7 , BIBREF8 . Yet in other cases the evidence appears to be more consistent with a more egocentric or “reflexively mind-blind” view of language processing BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . Under this view, although adults have the ability to deploy theory of mind, it is effortful and costly to do so. Thus people may initially anchor on their own perspective and only adjust to account for other perspectives when a problem arises and when sufficient cognitive resources are available.
Much of this debate has centered around the influential director-matcher paradigm, a variant of classic reference games BIBREF13 where a confederate speaker gives participants instructions about how to move objects around a grid. By introducing an asymmetry in visual access—certain cells of the grid are covered such that participants can see objects that the speaker cannot (e.g. Fig. 1 )— BIBREF14 designed a task to expose cases where participants (listeners) either succeed or fail to take into account what the speaker sees. In particular, BIBREF14 argued that if listeners were reliably using theory of mind, they would only consider mutually visible objects as possible referents. For instance, on one trial a roll of Scotch tape was mutually visible and a cassette tape was hidden from the speaker's view. When the confederate speaker produced an ambiguous utterance, “tape,” participants should still interpret it as a reference to the mutually visible object even if it fits the hidden object better; the idea is that a speaker who cannot see an object wouldn't possibly be referring to it.
While the visual asymmetries constructed by BIBREF14 may provide the starkest test of this hypothesis, variations on this basic paradigm have manipulated other dimensions of non-visual knowledge asymmetry, including those based on spoken information BIBREF15 , BIBREF16 , spatial cues BIBREF17 , BIBREF18 , private pre-training on object labels BIBREF19 , cultural background BIBREF20 , and other task-relevant information BIBREF21 , BIBREF22 . Questions about speaker perspective-taking during production have similarly been explored by reversing the direction of the asymmetry so the speaker has private knowledge that the listener does not and examining whether this private information leaks into their utterances BIBREF23 , BIBREF24 , BIBREF25 , BIBREF26 , BIBREF27 , BIBREF28 , BIBREF29 . Numerous rounds of reinterpretation and methodological criticism have puzzled over seemingly contradictory findings in this sprawling body of work: some studies find strong evidence consistent with an egocentric view—listeners initially consider and even attempt to move such objects—while others find that information from the speaker's perspective is integrated from the very earliest stages of processing BIBREF30 , BIBREF31 .
Recent computational models have begun to unify this literature under a probabilistic framework. For instance, some models assume that listeners BIBREF32 and speakers BIBREF33 simultaneously integrate their own perspective with that of their partner, leading to behavior that lies between purely egocentric and purely guided by common ground. These constraint-based models BIBREF34 , BIBREF35 introduce a probabilistic weighting parameter between the two domains of reference and show that an intermediate weighting explains the gradient of communicative behavior better than a purely egocentric or purely perspective-adopting model. Yet these constraint-based models leave open a key puzzle for rational models of language use: why do people use the proportion they do in a given context? In other words, while different factors influencing the weighting have been proposed, no formal mechanism yet explains why incorporating egocentric knowledge would be adaptive when full common ground is available.
We argue in this paper for a resource rational account of perspective-taking in communication BIBREF36 , BIBREF37 . In a communicative interaction with another agent, the participants share the goal of successfully being understood while minimizing joint effort BIBREF38 , BIBREF4 . If theory of mind use is indeed effortful and cognitively demanding to some degree BIBREF39 , BIBREF40 , BIBREF41 , then the question for a rational agent is when and how to best allocate its cognitive resources to achieve its goals. This sets up a natural division of labor between the speaker and listener in how the effort should be shared, which in principle admits many solutions. Rather than being guided by rigid heuristics, individuals may rationally and adaptively calibrate their perspective-taking based on expectations about their partner's likely behavior. Critically, these expectations may themselves be derived from a targeted use of theory of mind.
Here, we explore one particular source of expectations derived from Gricean expectations of informativity, which have been largely neglected by prior work in the perspective-taking literature BIBREF42 . Just as making sense of an agent's physical behaviors requires a broad, accurate mental model of how the agent's visual access, beliefs, and intentions translate into motor plans BIBREF43 , BIBREF44 , making sense of an agent's linguistic behaviors depends on an accurate model of what a speaker would say, or what a listener would understand, in different situations BIBREF45 , BIBREF46 , BIBREF47 , BIBREF48 , BIBREF49 . From this perspective, theory of mind use not only incorporates people’s mental models of a partner’s knowledge or visual access but also their inferences about how their partner would behave in a communicative context. To instantiate this account, we elaborate the family of probabilistic weighting models by proposing that theory of mind use under knowledge asymmetries not only involves integrating a partner's knowledge but also recursive reasoning about how they will likely produce or interpret utterances in particular communicative contexts BIBREF50 .
The Gricean notion of cooperativity BIBREF3 , BIBREF4 refers to the idea that speakers try to avoid saying things that are confusing or unnecessarily complicated given the current context, and that listeners expect this. For instance, imagine trying to help someone spot your dog at a busy dog park. It may be literally correct to call it a “dog,” but as a cooperative speaker you would understand that the listener would have trouble disambiguating the referent from many other dogs. Likewise, the listener would reasonably expect you to say something more informative than “dog” in this context. You may therefore prefer to use a more specific or informative expressions, like “the little terrier with the blue collar.” BIBREF7 , BIBREF51 . Critically, you might do so even when you happen to see only one dog at the moment, but know there are likely to be other dogs from the listener's point of view. In the presence of uncertainty about their partner's visual context, a cooperative speaker may tend toward additional specificity.
Now, what level of specificity is pragmatically appropriate in the particular director-matcher task used by BIBREF52 ? This task requires the speaker to generate a description such that a listener can identify the correct object among distractors, even though several cells are hidden from the speaker's view (e.g. Fig. 2 , bottom). It is thus highly salient to the speaker that there are hidden objects she cannot see but her partner can. Gricean reasoning, as realized by recent formal models BIBREF46 , BIBREF47 , BIBREF49 , predicts that a speaker in this context will compensate for her uncertainty about the listener's visual context by increasing the informativity of her utterance beyond what she would produce in a completely shared context. (See Appendix A for a formal model of pragmatic reasoning in this situation and a mathematical derivation of the informativity prediction.). The director-matcher task used by BIBREF52 is therefore not only challenging for the listener; it also requires a sophisticated use of theory of mind, vis a vis pragmatic reasoning, on the part of the speaker, to understand that the listener may expect her to increase the informativity of her utterance. While extensive prior work has examined how speakers adjust their utterances, or not, depending on their own private information, it remains untested how they pragmatically compensate for their lack of access to the listener's private information by flexibly modifying their informativity.
In the following experiments, we ask whether people, as speakers, show such sensitivity to their own uncertainty about their partner's visual access. Furthermore, we suggest that such sensitivity (and the listener's expectations about this sensitivity) can help us understand why listeners in prior work (e.g., in the Director-Matcher task) made frequent errors. A listener's rational reliance on the speaker's informativity, which allows them to efficiently neglect the speaker's visual access under cognitive load, may backfire and lead to errors when paired with a confederate speaker who violates Gricean expectations. First, we directly test our model's prediction by manipulating the presence and absence of occlusions in a simple, interactive, natural-language reference game. Second, we conduct a replication of BIBREF52 with an additional unscripted condition to evaluate whether the scripted referring expressions used by confederate speakers in prior work accord with what a real speaker would say in the same interactive context BIBREF54 , BIBREF55 , BIBREF56 . If confederate speakers were using scripts that were uncooperative and underinformative compared to what speakers naturally say, this previously unrecognized violation of Gricean expectations may have implications for the rational basis of listener errors. Our main goal here is to directly establish the adaptive pragmatic behavior of speakers. It is important to note that our broader claim about the source of listener errors emerges from establishing the plausibility of a resource-rational basis for perspective-neglect, showing that speakers are adaptive (Exp.1) and listeners indeed make more errors when speakers violate their expectations (Exp.2); causally manipulating listener expectations is beyond the scope of the current work. We return to the broader implications and predictions of this account in the discussion.
Experiment 1: Speaker behavior under uncertainty
How does an unscripted speaker change her communicative behavior when there is uncertainty about exactly what her partner can see? To address this question empirically, we randomly assigned participants to the roles of speaker and listener and paired them over the web to play an interactive communication task BIBREF57 .
Methods
We recruited 102 pairs of participants from Amazon Mechanical Turk and randomly assigned speaker and listener roles. After we removed 7 games that disconnected part-way through and 12 additional games according to our pre-registered exclusion criteria (due to being non-native English speakers, reporting confusion about the instructions, or clearly violating the instructions), we were left with a sample of 83 full games.
On each trial, both players were presented with a $3\times 3$ grid containing objects. One target object was privately highlighted for the speaker, who freely typed a message into a chat box in order to get the listener to click the intended referent. The objects varied along three discrete features (shape, texture, and color), each of which took four discrete values (64 possible objects). See Appendix Fig. 7 for a screenshot of the interface.
There were four types of trials, forming a within-pair $2 \times 2$ factorial design. We manipulated the presence or absence of occlusions and the closeness of shared distractors to the target (see Fig. 2 ). On `shared' trials, all objects were seen by both participants, but on `hidden' trials, two cells of the grid were covered with occluders (curtains) such that only the listener could see the contents of the cell. On `far' trials, the target is the only object with a particular shape; on `close' trials, there is also a shared distractor with the target's shape, differing only in color or texture.
In order to make it clear to the speaker that there could really be objects behind the occluders without providing a statistical cue to their identity or quantity on any particular trial, we randomized the total number of distractors in the grid on each trial (between 2 and 4) as well as the number of those distractors covered by curtains (1 or 2). If there were only two distractors, we did not allow both of them to be covered: there was always at least one visible distractor. Each trial type appeared 6 times for a total of 24 trials, and the sequence of trials was pseudo-randomized such that no trial type appeared more than twice in each block of eight trials. Participants were instructed to use visual properties of the objects rather than spatial locations in the grid.
Finally, we collected mouse-tracking data analogous to the eye-tracking common in referential paradigms. We asked the matcher to wait until the director sent a message; when the message was received, the matcher clicked a small circle in the center of the grid to show the objects and proceed with the trial. We recorded at 100Hz from the matcher's mouse in the decision window after this click, until the point where they clicked and started to drag one of the objects. While we did not intend to analyze these data for Exp. 1, we anticipated using it in our second experiment below and wanted to use the same procedure across experiments for consistency.
We recruited 200 pairs of participants from Amazon Mechanical Turk. 58 pairs were unable to complete the game due to a server outage. Following our preregistered exclusion criteria, we removed 24 games who reported confusion, violated our instructions, or made multiple errors on filler items, as well as 2 additional games containing non-native English speakers. This left 116 pairs in our final sample.
The materials and procedure were chosen to be as faithful as possible to those reported in BIBREF52 while allowing for interaction over the web. Directors used a chat box to communicate where to move a privately cued target object in a $4 \times 4$ grid (see Fig. 1 ). The listener then attempted to click and drag the intended object. In each of 8 objects sets, mostly containing filler objects, one target belonged to a `critical pair' of objects, such as a visible cassette tape and a hidden roll of tape that could both plausibly be called `the tape.'
We displayed instructions to the director as a series of arrows pointing from some object to a neighboring unoccupied cell. Trials were blocked into eight sets of objects, with four instructions each. As in BIBREF52 , we collected baseline performance by replacing the hidden alternative (e.g. a roll of tape) with a filler object that did not fit the critical instruction (e.g. a battery) in half of the critical pairs. The assignment of items to conditions was randomized across participants, and the order of conditions was randomized under the constraint that the same condition would not be used on more than two consecutive items. All object sets, object placements, and corresponding instruction sets were fixed across participants. In case of a listener error, the object was placed back in its original position; both participants were given feedback and asked to try again.
We used a between-subject design to compare the scripted labels used by confederate directors in prior work against what participants naturally say in the same role. For participants assigned to the director role in the `scripted' condition, a pre-scripted message using the precise wording from BIBREF52 automatically appeared in their chat box on half of trials (the 8 critical trials as well as nearly half of the fillers). Hence, the scripted condition served as a direct replication. To maintain an interactive environment, the director could freely produce referring expressions on the remainder of filler trials. In the `unscripted' condition, directors were unrestricted and free to send whatever messages they deemed appropriate on all trials. In addition to analyzing messages sent through the chat box and errors made by matchers (listeners), we collected mouse-tracking data in analogy to the eye-tracking common in these paradigms.
Behavioral results
Our primary measure of speaker behavior is the length (in words) of naturally produced referring expressions sent through the chat box. We tested differences in speaker behavior across conditions using a mixed-effect regression of context and occlusion on the number of words produced, with maximal random effect structure containing intercept, slopes, and interaction. First, as a baseline, we examined the simple effect of close vs. far contexts in trials with no occlusions. We found that speakers used significantly more words on average when there was a distractor in context that shared the same shape as the target ( $b = 0.56, t = 5.1, p < 0.001$ ; see Fig. 3 A). This replicates the findings of prior studies in experimental pragmatics BIBREF7 , BIBREF58 . Next, we turn to the simple effect of occlusion in far contexts (which are most similar to the displays used in the director-matcher task which we adopt in Exp. 2 BIBREF52 ). Speakers used 1.25 additional words on average when they knew their partner could potentially see additional objects ( $t = 7.5, p < 0.001$ ). Finally, we found a significant interaction ( $b = -0.49, t = 3.8, p <0.001$ ) where the effect of occlusion was larger in far contexts, likely indicating a ceiling on the level of informativity required to individuate objects in our simple stimulus space.
What are these additional words used for? As a secondary analysis, we annotated each utterance based on which of the three object features were mentioned (shape, texture, color). Because speakers nearly always mentioned shape (e.g. `star', `triangle') as the head noun of their referring expression regardless of context ( $\sim 99\%$ of trials), differences in utterance length across conditions must be due to differentially mentioning the other two features (color and texture). To test this observation, we ran separate mixed-effect logistic regressions for color and texture predicting mention from context; due to convergence issues, the maximum random effect structure supported by our data contains only speaker-level intercepts and slopes for the occlusion effect. We found simple effects of occlusion in far contexts for both features ( $b = 1.33, z = 2.9, p = 0.004$ for color; $b = 4.8, z = 6.4, p < 0.001$ for texture, see Fig. 3 B). In other words, in displays like the left column of Fig. 2 where the target was the only `star', speakers were somewhat more likely to produce the star's color—and much more likely to produce its texture—when there were occlusions present, even though shape alone is sufficient to disambiguate the target from visible distractors in both cases. Finally, we note that listener errors were rare: 88% of listeners made only one or fewer errors (out of 24 trials), and there was no significant difference in error rates across the four conditions ( $\chi ^2(3) = 1.23, p = 0.74$ ). We test the connections between context-sensitive speaker behavior and listener error rates more explicitly in Exp. 2.
Model comparison
While our behavioral results provide qualitative support for a Gricean account over an egocentric account, formalizing these two accounts in computational models allows a stronger test of our hypothesis by generating graded quantitative predictions. We formalized both accounts in the probabilistic Rational Speech Act (RSA) framework BIBREF47 , BIBREF46 , BIBREF49 , BIBREF59 , BIBREF48 , which has successfully captured a variety of other pragmatic phenomena. In this framework, speakers are decision-theoretic agents attempting to (soft-)maximize a utility function balancing parsimony (i.e., a preference for shorter, simpler utterances) with informativeness (i.e., the likelihood of an imagined listener agent having the intended interpretation). The only difference between the two accounts in the RSA framework is how the asymmetry in visual access is handled: the `occlusion-blind' speaker simply assumes that the listener sees the same objects as she herself sees, while the `occlusion-sensitive' speaker represents uncertainty over her partner's visual context. In particular, she assumes a probability distribution over the possible objects that might be hidden behind the occlusions and attempts to be informative on average. The two models have the same four free parameters: a speaker optimality parameter controlling the soft-max temperature, and three parameters controlling the costs of producing the features of shape, color, and texture (see Appendix B for details).
We conducted a Bayesian data analysis to infer these parameters conditioning on our empirical data, and computed a Bayes Factor to compare the models. We found extremely strong support for the occlusion-sensitive model relative to the occlusion-blind model ( $BF = 2.2 \times 10^{209}$ ; see Appendix Fig. 8 for likelihoods). To examine the pattern of behavior of each model, we computed the posterior predictive on the expected number of features mentioned in each trial type of our design. While the occlusion-blind speaker model successfully captured the simple effect of close vs. far contexts, it failed to account for behavior in the presence of occlusions. The occlusion-sensitive model, on the other hand, accurately accounted for the full pattern of results (see Fig 4 ). Finally, we examined parameter posteriors for the occlusion-sensitive model (see Appendix Fig. 9 ): the inferred production cost for texture was significantly higher than that for the other features, reflecting the asymmetry in production of texture relative to color.
Experiment 2: Comparing confederates to natural speakers
Experiment 1 directly tested the hypothesis that speakers increase their specificity in contexts with asymmetry in visual access. We found that speakers are not only context-sensitive in choosing referring expressions that distinguish target from distractors in a shared context, but are occlusion-sensitive, adaptively compensating for uncertainty. Critically, this resulted in systematic differences in behavior across the occlusion conditions that are difficult to explain under an egocentric theory: in the presence of occlusions, speakers were spontaneously willing to spend additional time and keystrokes to give further information beyond what they produce in the corresponding unoccluded contexts, even though that information is equally redundant given the visible objects in their display.
These results validate our prediction that speakers appropriately increase their level of specificity in contexts containing occlusions. In Experiment 2, we recruited pairs of participants for an online, interactive version of the original director-matcher task BIBREF52 which used occluded contexts to demonstrate limits on visual perspective-taking for the listener. Given the results of Exp. 1, we predicted that participants in the director role (i.e. speakers) would naturally provide more informative referring expressions than the confederate directors used in prior work. This would suggest that the confederate directors in prior work were pragmatically infelicitous, violating listeners' expectations. This violation of listeners' cooperative expectations may have led to detrimental consequences for listener performance.
Results
Our scripted condition successfully replicated the results of BIBREF52 with even stronger effects: listeners incorrectly moved the hidden object on approximately 50% of critical trials. However, on unscripted trials, the listener error rate dropped by more than half, $p_1 = 0.51, p_2 = 0.20, \chi ^2(1) = 43, p < 0.001$ (Fig. 5 A). While we found substantial heterogeneity in error rates across object sets (just 3 of the 8 object sets accounted for the vast majority of remaining unscripted errors; see Appendix Fig. 10 ), listeners in the unscripted condition made fewer errors for nearly every critical item. In a maximal logistic model with fixed effect of condition, random intercepts for each dyad, and random slopes and intercepts for each object set, we found a significant difference in error rates across conditions ( $z = 2.6, p = 0.008$ ).
Even if participants in the unscripted condition make fewer actual errors, they may still be considering the hidden object just as often on trials where they go on to make correct responses. As a proxy for the eye-tracking analyses reported by BIBREF52 , we conducted a mouse-tracking analysis. We computed the mean (logged) amount of time spent hovering over the hidden distractor and found a significant interaction between condition and the contents of the hidden cell ( $t = 3.59, p <0.001$ ; Fig. 5 B) in a mixed-effects regression using dyad-level and object-level random intercepts and slopes for the difference from baseline. Listeners in the scripted condition spent more time hovering over the hidden cell when it contained a confusable distractor relative to baseline, again replicating BIBREF52 . In the unscripted condition there was no difference from baseline.
Next, we test whether these improvements in listener performance in the unscripted condition are accompanied by more informative speaker behavior than the scripted utterances allowed. The simplest measure of speaker informativity is the raw number of words used in referring expressions. Compared to the scripted referring expressions, speakers in the unscripted condition used significantly more words to refer to critical objects ( $b = 0.54, t = 2.6, p=0.019$ in a mixed-effects regression on difference scores using a fixed intercept and random intercepts for object and dyads). However, this is a coarse measure: for example, the shorter “Pyrex glass” may be more specific than “large measuring glass” despite using fewer words. For a more direct measure, we extracted the referring expressions generated by speakers in all critical trials and standardized spelling and grammar, yielding 122 unique labels after including scripted utterances.
We then recruited an independent sample of 20 judges on Amazon Mechanical Turk to rate how well each label fit the target and hidden distractor objects on a slider from “strongly disagree” (meaning the label “doesn't match the object at all”) to “strongly agree” (meaning the label “matches the object perfectly”). They were shown objects in the context of the full grid (with no occlusions) such that they could feasibly judge spatial or relative references like “bottom block.” We excluded 4 judges for guessing with response times $< 1s$ . Inter-rater reliability was relatively high, with intra-class correlation coefficient of $0.54\, (95\% CI = [0.47, 0.61])$ . We computed the informativity of an utterance (the tape) as the difference in how well it was judged to apply to the target (the cassette tape) relative to the distractor object (the roll of tape).
Our primary measure of interest is the difference in informativity across scripted and unscripted utterances. We found that speakers in the unscripted condition systematically produced more informative utterances than the scripted utterances ( $d = 0.5$ , 95% bootstrapped CI = $[0.27, 0.77], p < .001$ ; see Appendix C for details). Scripted labels fit the hidden distractor just as well or better than the target, but unscripted labels fit the target better and the hidden distractor much worse (see Fig. 6 A). In other words, the scripted labels used in BIBREF52 were less informative than expressions speakers would normally produce to refer to the same object in this context.
These results strongly suggest that the speaker's informativity influences listener accuracy. In support of this hypothesis, we found a strong negative correlation between informativity and error rates across items and conditions: listeners make fewer errors when utterances are a better fit for the target relative to the distractor ( $\rho = -0.81$ , bootstrapped 95% CI $= [-0.9, -0.7]$ ; Fig. 6 B). This result suggests that listener behavior is driven by an expectation of speaker informativity: listeners interpret utterances proportionally to how well they fit objects in context.
General Discussion
Are human adults expert mind-readers, or fundamentally egocentric? The longstanding debate over the role of theory of mind in communication has largely centered around whether listeners (or speakers) with private information consider their partner's perspective BIBREF30 , BIBREF16 . Our work presents a more nuanced picture of how a speaker and a listener use theory of mind to modulate their pragmatic expectations. The Gricean cooperative principle emphasizes a natural division of labor in how the joint effort of being cooperative is shared BIBREF4 , BIBREF60 . It can be asymmetric when one partner is expected to, and able to, take on more complex reasoning than the other, in the form of visual perspective-taking, pragmatic inference, or avoiding further exchanges of clarification and repair. One such case is when the speaker has uncertainty over what the listener can see, as in the director-matcher task. Our Rational Speech Act (RSA) formalization of cooperative reasoning in this context predicts that speakers (directors) naturally increase the informativity of their referring expressions to hedge against the increased risk of misunderstanding; Exp. 1 presents direct evidence in support of this hypothesis.
Importantly, when the director (speaker) is expected to be appropriately informative, communication can be successful even when the matcher (listener) does not reciprocate the effort. If visual perspective-taking is effortful and cognitively demanding BIBREF39 , the matcher will actually minimize joint effort by not taking the director's visual perspective. This suggests a less egocentric explanation of when and why listeners neglect the speaker's visual perspective; they do so when they expect the speaker to disambiguate referents sufficiently. While adaptive in most natural communicative contexts, such neglect might backfire and lead to errors when the speaker (inexplicably) violates this expectation. From this point of view, the “failure” of listener theory of mind in these tasks is not really a failure; instead, it suggests that both speakers and listeners may use theory of mind to know when (and how much) they should expect others to be cooperative and informative, and subsequently allocate their resources accordingly BIBREF36 . Exp. 2 is consistent with this hypothesis; when directors used underinformative scripted instructions (taken from prior work), listeners made significantly more errors than when speakers were allowed to provide referring expressions at their natural level of informativity, and speaker informativeness strongly modulated listener error rates.
Our work adds to the growing literature on the debate over the role of pragmatics in the director-matcher task. A recent study questions the communicative nature of the task itself by showing that selective attention alone is sufficient for successful performance on this task, and that listeners become suspicious of the director's visual access when the director shows unexpectedly high levels of specificity in their referring expressions BIBREF61 . Our results further sbolster the argument that pragmatic reasoning about appropriate levels of informativity is an integral aspect of theory of mind use in the director-matcher task (and communication more generally). Note however that in BIBREF61 , participants became suspicious, while in our study participants overtrusted the speaker to be informative; a more detailed look at differences between experimental paradigms, as well as further experimental work, is necessary to better understand why participants had different expectations about the speaker. Prior work also suggests that although speakers tend to be over-informative in their referring expressions BIBREF62 a number of situational factors (e.g., perceptual saliency of referents) can modulate this tendency. Our work hints at an additional principle that guides speaker informativity: speakers maintain uncertainty about the listener's visual context and their ability to disambiguate the referent in that context.
Additionally, while our model builds on probabilistic models weighting different perspectives BIBREF32 , BIBREF33 , we leave the formal integration of resource-rational recursive reasoning mechanisms with perspective-weighting mechanisms for future work. While BIBREF33 focused on cases where the speaker has private information unknown to the listener, our model focuses on the reverse case: how speakers behave when they know that the listener has additional private information BIBREF52 . Furthermore, whether the allocation of resources, and ensuing perspective neglect, is a fixed strategy or one that adjusts dynamically remains an open question: given sufficient evidence of an unusually underinformative partner, listeners may realize that vigilance about which objects are occluded yields a more effective strategy for the immediate interaction. An important direction for future work is to directly explore listener adaptability in adjusting their use of visual perspective-taking as a function of Gricean expectations for a given partner BIBREF63 , BIBREF64 .
In sum, our findings suggest that language use is well-adapted to contexts of uncertainty and knowledge asymmetry. The pragmatic use of theory of mind to establish division of labor is also critical for other forms of social cooperation, including pedagogy BIBREF65 and team-based problem solving BIBREF66 , BIBREF67 . Enriching our notion of theory of mind use to encompass these pragmatic expectations, not only expectations about what our partner knows or desires, may shed new light on the flexibility of social interaction more broadly.
Acknowledgements
This manuscript is based in part on work presented at the 38th Annual Conference of the Cognitive Science Society. The first author is supported by a NSF Graduate Research Fellowship and a Stanford Graduate Fellowship. A pilot of expt. 2 was originally conducted under the supervision of Michael Frank, with early input from Desmond Ong. We’re grateful to Boaz Keysar for providing select materials for our replication. This work was supported by ONR grants N00014-13-1-0788 and N00014-13- 1-0287, and a James S. McDonnell Foundation Scholar Award to NDG.
Author contributions
R.X.D.H. and N.D.G. initially formulated project. R.X.D.H. performed experiments, analyzed data, and performed computational modeling. All authors planned experiments, interpreted result, and wrote the paper. Unless otherwise mentioned, all analyses and materials were preregistered at https://osf.io/qwkmp/. Code and materials for reproducing the experiment as well as all data and analysis scripts are open and available at https://github.com/hawkrobe/pragmatics_of_perspective_taking.
Appendix A: Derivation of qualitative model predictions
Our experiments are motivated by the Gricean observation that speakers should attempt to be more informative when there is an asymmetry in visual access, such that their partner sees something they do not. In this appendix, we formalize this scenario in a computational model of communication as recursive social reasoning and prove that the predicted increase in informativity qualitatively holds under fairly unrestrictive conditions.
Following recent advances in the Rational Speech Act (RSA) framework, we define a speaker as a decision-theoretic agent who must choose a referring expression $u$ to refer to a target object $o$ in a context $C$ by (soft)-maximizing a utility function $U$ : $S(u | o, C) \propto \exp \lbrace \alpha U(u; o, C)\rbrace $
Definition The basic utility used in RSA models captures the informativeness of each utterance to an imagined literal listener agent $L$ who is attempting to select the target object from alternatives in context: $U_{basic}(u; o, C) = \log L(o | u, C)$
This information-theoretic expression measures how certain the listener becomes about the intended object after hearing the utterance. The literal listener is assumed to update their beliefs about the target object according to Bayesian inference, conditioning on the literal meaning of the utterance being true of it: $L(o | u, C) \propto \mathcal {L}(o,u) P(o)$
where normalization takes place over objects $o \in C$ and $\mathcal {L}$ represents the lexical semantics of $u$ . If $u$ is true of $o$ then $\mathcal {L}(o,u) = 1$ ; otherwise, $\mathcal {L}(o,u) = 0$ .
This basic setup assumes that the speaker reasons about a listener sharing the same context $C$ in common ground. How should it be extended to handle asymmetries in visual access between the speaker and listener, where the speaker has uncertainty over the possible distractors behind the occlusions? In the RSA framework, speaker uncertainty is represented straightforwardly by a prior over the state of the world: for example, BIBREF48 examined a case where the speaker has limited perceptual access to the objects they are describing. For the director-matcher task, we construct this prior by positing a space of alternative objects $\mathcal {O}$ , introducing uncertainty $P(o_h)$ over which object $o_h \in \mathcal {O}$ , if any, is hidden behind an occlusion, and marginalizing over these alternatives when reasoning about the listener.
Definition This gives us a utility for conditions of asymmetries in visual access: $U_{asym}(u; o, C) =\sum _{o_h \in \mathcal {O}} P(o_h) \log L(o | u, C \cup o_h)$
where $C$ denotes the set of objects in context that the speaker perceives.
We define “specificity” extensionally, in the sense that if $u_0$ is more specific than $u_1$ , then the objects for which $u_0$ is true is a subset of the objects for which $u_1$ is true:
Definition Utterance $u_0$ is said to be more specific than $u_1$ iff $\mathcal {L}(u_0, o_h) \le \mathcal {L}(u_1, o_h)\ \forall o_h \in \mathcal {O}$ and there exists a subset of objects $\mathcal {O}^* \subset \mathcal {O}$ such that $\sum _{o^* \in \mathcal {O}^*} P(o^*) > 0$ and $\mathcal {L}(u_0, o^*) < \mathcal {L}(u_1, o^*)$ for $o* \in \mathcal {O}^*$ .
We now show that the recursive reasoning model predicts that speakers should prefer more informative utterances in contexts with occlusions. In other words, that the asymmetry utility leads to a preference for more specific referring expressions than the basic utility.
Theorem If $u_0$ is more specific than $u_1$ , then the following holds for any target $o^t$ and shared context $C$ : $ \frac{S_{asym}(u_0 | o^t, C)}{S_{asym}(u_1| o^t, C)} > \frac{S_{basic}(u_0 | o^t, C)}{S_{basic}(u_1 | o^t, C)} $
Since $S(u_0|o^t, C)/S(u_1|o^t, C) = \exp (\alpha \cdot (U(u_0; o^t, C) - U(u_1;o^t,C)))$ it is sufficient to show $ U_{asym}(u_0 ; o, C) - U_{asym}(u_1; o, C) > U_{basic}(u_0 ; o, C) - U_{basic}(u_1 ; o, C) $
We first break apart the sum on the left-hand side:
$$U_{asym}(u_0 | o^t, C) - U_{asym}(u_1 | o^t, C) &=& \displaystyle \sum _{o_h \in \mathcal {O}} p(o_h)\left[\log L(o | u_0, C\cup o_h) - \log L(o|u_1, C \cup o_h)\right] \\ & = & \displaystyle \sum _{o^*\in \mathcal {O}^*} p(o^*) \log \frac{L(o^t|u_0, C\cup o^*)}{L(o^t|u_1, C\cup o^*)} \\ & & + \displaystyle \sum _{o_h\in \mathcal {O}\setminus \mathcal {O}^*} p(o_h) \log \frac{L(o^t|u_0, C\cup o_h)}{L(o^t|u_1, C\cup o_h)} $$   (Eq. 9)
By the definition of “more specific” and because we defined $o^*\in \mathcal {O^*}$ to be precisely the subset of objects for which $\mathcal {L}(u_0, o^*) < \mathcal {L}(u_1, o^*)$ , for objects $o_h$ in the complementary set $\mathcal {O} \setminus \mathcal {O^*}$ we have $\mathcal {L}(u_0, o_h) = \mathcal {L}(u_1, o_h)$ . Therefore, for , $L(o^t | u_i, C \cup o_h) = L(o^t | u_i, C)$ , giving us $\log \frac{L(o^t | u_0, C)}{L(o^t|u_1, C)}\sum _{o_h\in \mathcal {O}\setminus \mathcal {O}^*}p(o_h)$
For the ratio in 9 , we can substitute the definition of the listener $L$ and simplify: $ \begin{array}{rcl} \displaystyle \frac{L(o^t|u_0, C\cup o^*)}{L(o^t|u_1, C\cup o^*)} & = & \displaystyle \frac{\mathcal {L}(o^t, u_0) [\sum _{o\in C \cup o^*}\mathcal {L}(o,u_1)]}{\mathcal {L}(o^t, u_1) [\sum _{o\in C \cup o^*}\mathcal {L}(o,u_0)]} \\[.5cm] & = & \displaystyle \frac{\mathcal {L}(o^t, u_0) [\sum _{o\in C}\mathcal {L}(o,u_1) + \mathcal {L}(o^*, u_1)]}{\mathcal {L}(o^t, u_1) [\sum _{o\in C}\mathcal {L}(o,u_0) + \mathcal {L}(o^*, u_0)]} \\[.5cm] & < & \displaystyle \frac{\mathcal {L}(o^t, u_0) [\sum _{o\in C}\mathcal {L}(o,u_1)]}{\mathcal {L}(o^t, u_1) [\sum _{o\in C}\mathcal {L}(o,u_0)]} \\[.5cm] & = & \displaystyle \frac{L(o^t|u_0, C)}{L(o^t|u_1, C)} \end{array} $
Thus, $ \begin{array}{rcl} U_{asym}(u_0 | o^t, C) - U_{asym}(u_1 | o^t, C) & < & \log \frac{L(o^t | u_0, C)}{L(o^t|u_1, C)}\left(\displaystyle \sum _{o^*\in \mathcal {O}^*}p(o^*) + \displaystyle \sum _{o_h\in \mathcal {O}\setminus \mathcal {O}^*}p(o_h)\right) \\ &=& \log L(o^t | u_0, C) - \log L(o^t | u_1, C) \\ &=& U_{basic}(u_0 | o^t, C) - U_{basic}(u_1 | o^t, C) \end{array} $
Note that this proof also holds when an utterance-level cost term $\textrm {cost}(u)$ penalizing longer or more effortful utterances is incorporated into the utilities $ \begin{array}{lcl} U_{asym}(u; o, C_s) & = & \sum _{o_h \in \mathcal {O}} \log L_0(o | u, C_s \cup o_h)P(o_h) - \textrm {cost}(u) \\ U_{basic}(u; o, C) & = & \log L(o | u, C) - \textrm {cost}(u) \end{array} $
since the same constant appears on both sides of inequality. In principle, it can also be extended to real-valued meanings $\mathcal {L}$ , though additional assumptions must be made.
Appendix B: Quantitative model fit for Exp. 1
In addition to the qualitative predictions derived in the previous section, our speaker model makes direct quantitative predictions about Exp. 1 data. Here, we describe the details of a Bayesian Data Analysis evaluating this model on the empirical data, and comparing it to an occlusion-blind model which does not reason about possible hidden objects.
Because there were no differences observed in production based on the particular levels of target features (e.g. whether the target was blue or red), we collapse across these details and only feed the model which features of each distractor differed from the target on each trial. After this simplification, there were only 4 possible contexts: far contexts, where the distractors differed in every dimension, and three varieties of close contexts, where the critical distractor differed in only shape, shape and color, or shape and texture. In addition, we included in the model information about whether each trial had cells occluded or not.
The space of utterances used in our speaker model is derived from our feature annotations: for each trial, the speaker model selected among 7 utterances referring to each combination of features: only mentioning the target's shape, only mentioning the target's color, mentioning the shape and the color, and so on. For the set of alternative objects $\mathcal {O}$ , we used the full 64-object stimulus space used in our experiment design, and we placed a uniform prior over these objects such that the occlusion-sensitive speaker assumed they were equally likely to be hidden.
Our model has four free parameters which we infer from the data using Bayesian inference. The speaker optimality parameter, $\alpha $ , is a soft-max temperature such that at $\alpha = 1$ , the speaker produces utterances directly proportional to their utility, and as $\alpha \rightarrow \infty $ the speaker maximizes. In addition, to account for the differential production of the three features (see Fig. 2B), we assume separate production costs for each feature: a texture cost $c_t$ , a color cost $c_c$ , and a shape cost $c_s$ . We use (uninformative) uniform priors for all parameters: $ \begin{array}{rcl} \alpha & \sim & \textrm {Unif}(0,50) \\ c_t, c_c, c_s & \sim & \textrm {Unif}(0,10) \end{array} $
We compute speaker predictions for a particular parameter setting using (nested) enumeration and infer the posterior over parameters using MCMC. We discard 5000 burn-in samples and then take 5000 samples from the posterior with a lag of 2. Our posterior predictives are computed from these posteriors by taking the expected number of features produced by the speaker marginalizing over parameters and possible non-critical distractors in context (this captures the statistics of our experimental contexts, where there was always a distractor sharing the same color or texture but a different shape as the target). Finally, to precisely compute the Bayes Factor, we enumerated over a discrete grid of parameter values in the prior. We implemented our models and conducted inference in the probabilistic programming language WebPPL (Goodman & Stuhlmuller, 2014). All code necessary to reproduce our model results are available at the project github: https://github.com/hawkrobe/pragmatics_of_perspective_taking.
Appendix C: Multi-stage bootstrap procedure for Expt. 2
The statistical dependency structure of our ratings was more complex than standard mixed-effect model packages are designed to handle and the summary statistic we needed for our test was a simple difference score across conditions, so we instead implemented a simple multi-stage, non-parametric bootstrap scheme to appropriately account for different sources of variance. In particular, we needed to control for effects of judge, item, and speaker.
First, to control for the repeated measurements of each judge rating the informativity of all labels, we resampled our set of sixteen judge ids with replacement. For each label, we then computed informativity as the difference between the target and distractor fits within every judge's ratings, and took the mean across our bootstrapped sample of judges. Next, we controlled for item effects by resampling our eight item ids with replacement. Finally, we resampled speakers from pairs within each condition (scripted vs. unscripted), and looked up the mean informativity of each utterance they produced for each of the resampled set of items. Now, we can take the mean within each condition and compute the difference across conditions, which is our desired test statistic. We repeated this multi-stage resampling procedure 1000 times to get the bootstrapped distribution of our test statistic that we reported in the main text. Individual errors bars in Fig. 4 are derived from the same procedure but without taking difference scores.

Question: Did participants behave unexpectedly?
Answer: No

Question: Was this experiment done in a lab?
Answer: No