How visual saliency affects referent accessibility

How visual saliency affects referent accessibility Jorrig Vogels (j.vogels@uvt.nl) Emiel Krahmer (e.j.krahmer@uvt.nl) Alfons Maes (maes@uvt.nl) Tilburg centre for Cognition and Communication (TiCC), Tilburg University PO Box 90153, 5000 LE Tilburg, The Netherlands Abstract As referents are more accessible in discourse, they can be referred to with more attenuated expressions, such as pronouns. Accessibility is known to be affected by the referent’s saliency in the linguistic context, but much less is known about the effect of saliency in the visual context. In this paper, we investigate whether a referent’s visual saliency affects the choice of referring expression in a discourse context. The results of a story completion experiment show that visually salient referents induce more attenuated expressions, but only when they are linguistically non-salient. Linguistically salient referents receive more reduced references when they are visually non-salient. We argue that visual saliency affects accessibility when the impact of linguistic factors is moderated. In addition, when the story does not match people’s expectations, processing difficulties might result in the use of less costly expressions. Keywords: saliency; accessibility; visual context; referring expressions; language production. Introduction In discourse, the same entity can be referred to with different types of expressions, ranging from elaborate descriptions involving full noun phrases and modifiers (e.g. the blonde girl with the big earrings) to short, low- informative elements such as pronouns (e.g. she). It has been argued that the referring expression a speaker chooses reflects the cognitive status of the referent (Ariel, 1990; Gundel, Hedberg, & Zacharski, 1993). That is, it is believed that speakers make assumptions about the degree of activation of the referent in the memory of their addressees. The more accessible an entity is to the addressee, the less information a referring expression needs to contain to be correctly understood. In addition, production processes are constrained by speaker-internal factors, such as cognitive load, that may affect referent accessibility in the speaker’s memory (e.g. Arnold & Griffin, 2007). An important source of a referent’s accessibility is its saliency. The role of saliency in the choice of referring expressions within a discourse has mainly been investigated in relation to the preceding linguistic context. For example, subjects and topics (i.e. what a sentence is about) are considered to be salient entities in a discourse (e.g. Arnold, 1998; Givon, 1983; Gordon, Grosz, & Gilliom, 1993; Grosz, Joshi, & Weinstein, 1995). Hence, a referent that is the subject or the topic of a preceding utterance is more accessible than other possible referents. Therefore, it is more likely to be referred to with an attenuated expression, such as a pronoun, while a less accessible referent needs a more elaborate description, such as a full noun phrase. Other factors that affect a referent’s saliency in the linguistic context include recency, givenness, thematic roles and syntactic position (e.g. Clark & Sengul, 1979; Gundel et al., 1993; Stevenson, Crawley, & Kleinman, 1994). It is much less clear whether saliency in the visual context also plays a role in the accessibility of referents in a discourse. In everyday life, the things we talk about are often not only available to us through previous linguistic mention, but in many cases they are also physically present. In this paper, we investigate whether the accessibility of a referent is influenced by its salience in the visual context. Language production studies that have taken into account the visual context suggest that speakers use non-linguistic information in planning their utterances. For instance, the syntactic structure of visual scene descriptions is affected by where people look in the scene (e.g. Gleitman et al., 2007). Visual information is also used to choose a referring expression. For example, people use disambiguating expressions to refer to visually ambiguous referents (e.g. Brown-Schmidt & Tanenhaus, 2006; Ferreira, Slevc, & Rogers, 2005), and more reduced expressions when referents are visually in focus (Beun & Cremers, 1998). When a referent is also accessible from the linguistic context, it might be the case that effects of visual information are overruled by linguistic information. In a story completion experiment, Arnold and Griffin (2007) found that participants used fewer pronouns to refer to the target character when a competitor referent was mentioned in the linguistic context. Whether the competitor was also visually present in the target scene did not make a difference, suggesting that the linguistic presence of the competitor affected accessibility, but not its visual presence. In contrast, Fukumura, Van Gompel and Pickering (2010) found in a similar experiment that visual context did influence the choice of referring expression: Participants used fewer pronouns to refer to the target referent when a competitor was visually present than when it was not visually present. However, the effect of the visual context was larger when the competitor was also linguistically present than when it was not mentioned at all. This suggests that accessibility is affected most by linguistic context, but that the influence of visual context becomes more apparent when the linguistic context is less compelling. Fukumura et al. (2010) argued that the visual presence of the competitor in their experiment reduced the salience of the referent, which led to a decrease in accessibility.