Rational over-specification in visually-situated comprehension and production

Contrary to the Gricean maxims of quantity (Grice,  Cole, Morgan (eds) Syntax and semantics: speech acts, vol III, pp 41–58, Academic Press, New York, 1975), it has been repeatedly shown that speakers often include redundant information in their utterances (over-specifications). Previous research on referential communication has long debated whether this redundancy is the result of speaker-internal or addressee-oriented processes, while it is also unclear whether referential redundancy hinders or facilitates comprehension. We present an information-theoretic explanation for the use of over-specification in visually-situated communication, which quantifies the amount of uncertainty regarding the referent as entropy (Shannon in Bell Syst Tech J 5:10, 10.1002/j.1538-7305.1948.tb01338.x, 1948). Examining both the comprehension and production of over-specifications, we present evidence that (a) listeners’ processing is facilitated by the use of redundancy as well as by a greater reduction of uncertainty early on in the utterance, and (b) that at least for some speakers, listeners’ processing concerns influence their encoding of over-specifications: Speakers were more likely to use redundant adjectives when these adjectives reduced entropy to a higher degree than adjectives necessary for target identification.

Related papers:

title: Do speakers and listeners observe the Gricean Maxim of Quantity
abstract: The Gricean Maxim of Quantity is believed to govern linguistic performance. Speakers are assumed to provide as much information as required for referent identification and no more, and listeners are believed to expect unambiguous but concise descriptions. In three experiments we examined the extent to which naive participants are sensitive to the Maxim of Quantity. The first was a production experiment which demonstrated that speakers over-describe almost one-third of the time. The second experiment showed that listeners do not judge over-descriptions to be any worse than concise expressions. The third experiment used the Visual World Paradigm to assess listeners’ moment-by-moment interpretations of over-described utterances. This last experiment revealed that over-descriptions trigger eye movements that can be interpreted as indicating confusion. The results provide support for the use of a simple heuristic such as Minimal Attachment or Argument Saturation to create an initial parse. We conclude that people are only moderately Gricean.

title: Overspecification facilitates object identification
abstract: Referential expressions often contain more information than what is strictly necessary for unique identification of the entity the expression refers to. If this extra information slows down the identification process, then this would suggest a violation of one of the Gricean maxims in the category quantity. A perception experiment was conducted in which overspecification was defined exophorically as the elements of referential expressions that denote non-uniquely identifying information of entities in a particular task environment. In the experiment, identifying information either related to object characteristics (size, color, shape) or location (on vertical or horizontal axis). The results of the experiment provide information about the effect of overspecification on the identification time. Overspecified expressions lead to faster identification when they allowed the reader (i) to fully complete a mental image of the entity and (ii) to delimit search behavior to a specific part of the task context. Extra information about vertical location (top, bottom) proved more efficient than extra information about horizontal location (left, right).

title: ERP indices of situated reference in visual contexts
abstract: Violations of the maxims of Quantity occur when utterances provide more (over-specified) or less (under-specified) information than strictly required for referent identification. While behavioural data suggest that under-specified expressions lead to comprehension difficulty and communicative failure, there is no consensus as to whether over-specified expressions are also detrimental to comprehension. In this study we shed light on this debate, providing neurophysiological evidence supporting the view that extra information facilitates comprehension. We further present novel evidence that referential failure due to underspecification is qualitatively different from explicit cases of referential failure, when no matching referential candidate is available in the context.

title: Uncertainty About the Rest of the Sentence
abstract: A word-by-word human sentence processing complexity metric is presented. This metric formalizes the intuition that comprehenders have more trouble on words contributing larger amounts of information about the syntactic structure of the sentence as a whole. The formalization is in terms of the conditional entropy of grammatical continuations, given the words that have been heard so far. To calculate the predictions of this metric, Wilson and Carroll's (1954) original entropy reduction idea is extended to infinite languages. This is demonstrated with a mildly context-sensitive language that includes relative clauses formed on a variety of grammatical relations across the Accessibility Hierarchy of Keenan and Comrie (1977). Predictions are derived that correlate significantly with repetition accuracy results obtained in a sentence-memory experiment (Keenan & Hawkins, 1987).

title: Redundancy and reduction: Speakers manage syntactic information density
abstract: A principle of efficient language production based on information theoretic considerations is proposed: Uniform Information Density predicts that language production is affected by a preference to distribute information uniformly across the linguistic signal. This prediction is tested against data from syntactic reduction. A single multilevel logit model analysis of naturally distributed data from a corpus of spontaneous speech is used to assess the effect of information density on complementizer that-mentioning, while simultaneously evaluating the predictions of several influential alternative accounts: availability, ambiguity avoidance, and dependency processing accounts. Information density emerges as an important predictor of speakers' preferences during production. As information is defined in terms of probabilities, it follows that production is probability-sensitive, in that speakers' preferences are affected by the contextual probability of syntactic structures. The merits of a corpus-based approach to the study of language production are discussed as well.

title: Overspecification in written instruction
abstract: A much debated question with respect to referential expressions is under what conditions speakers and writers produce overspecified expressions, i.e. expressions that contain more information than what is necessary for unique identification. In an earlier study we found that overspecifications facilitate the identification process for the understander, in particular if they contain complete object information and/or complete location information. In the present study, a production experiment is discussed in which the importance (high, low) of the instructed referential task was manipulated. In the high-importance condition more overspecifications were produced than in the low-importance condition, and this was particularly the case for overspecifications that facilitate identification for the understander.

title: Tracking the time course of multidimensional stimulus discrimination: Analyses of viewing patterns and processing times during “same”-“different“ decisions
abstract: We investigated the time course of conjunctive “same”-“different“ judgements for visually presented object pairs by means of combined reaction time and on-line eye movement measurements. The analyses of viewing patterns, viewing times, and reaction times showed that participants engaged in a parallel self-terminating search for differences. In addition, the results obtained for objects differing in only one dimension suggest that processing times may depend on the relative codability of the stimulus dimensions. The results are reviewed in a broader framework in view of higher-order processes. We propose that overspecifications of colour, often found in object descriptions, may have an “early” visual rather than a “late” linguistic origin. In a parallel assessment of the detection materials, participants overspecified the objects' colour substantially more often than their size. We argue that referential overspecifications of colour are largely attributable to mechanisms of visual discrimination.

title: Social interaction and the development of definite descriptions
abstract: The present study examines whether a deficit in the production of definite descriptions can be compensated for and successfully resolved in social interaction. We first discuss the developmental reasons for contextually inadequate descriptions and also the mechanisms by which social interaction can reduce processing demands. Then we report on a cross-sectional study using 50 Dutch subjects aged 3, 6, and 9, compared to adults, in which we analyzed the interaction between speaker and addressee in a referential communication task. Our results reveal that all the referential ambiguities in the 6 and 9 year olds, and in the adults, were successfully resolved when the addressee repeated the preceding description in question format, where the initial description (plus the succeeding specification) had not been explicit enough to identify the target referent. In the youngest age group, the 3 year olds, the analogous percentage of resolutions was 89%. Moreover, we found that the length of interaction required depends on age-related differences in the explicitness of the initial descriptions and the (induced) repairs. The main results raise several questions that are discussed under three headings: the role of feedback in speaker-addressee interaction, the possibility of a procedural explanation for production deficits and their compensation, and the possible learning mechanisms underlying developmental changes in referential descriptions.

title: The Effect of Scene Variation on the Redundant Use of Color in Definite Reference
abstract: This study investigates to what extent the amount of variation in a visual scene causes speakers to mention the attribute color in their definite target descriptions, focusing on scenes in which this attribute is not needed for identification of the target. The results of our three experiments show that speakers are more likely to redundantly include a color attribute when the scene variation is high as compared with when this variation is low (even if this leads to overspecified descriptions). We argue that these findings are problematic for existing algorithms that aim to automatically generate psychologically realistic target descriptions, such as the Incremental Algorithm, as these algorithms make use of a fixed preference order per domain and do not take visual scene variation into account.

title: How Distractor Objects Trigger Referential Overspecification: Testing the Effects of Visual Clutter and Distractor Distance
abstract: In two experiments, we investigate to what extent various visual saliency cues in realistic visual scenes cause speakers to overspecify their definite object descriptions with a redundant color attribute. The results of the first experiment demonstrate that speakers are more likely to redundantly mention color when visual clutter is present in a scene as compared to when this is not the case. In the second experiment, we found that distractor type and distractor color affect redundant color use: Speakers are most likely to overspecify if there is at least one distractor object present that has the same type, but a different color than the target referent. Reliable effects of distractor distance were not found. Taken together, our results suggest that certain visual saliency cues guide speakers in determining which objects in a visual scene are relevant distractors, and which not. We argue that this is problematic for algorithms that aim to generate human-like descriptions of objects (such as the Incremental Algorithm), since these generally select properties that help to distinguish a target from all objects that are present in a scene.

title: Reference Management in Instructive Discourse
abstract: This article investigates the effect of 2 language-in-use factors on the introduction and maintenance of referents in instructive discourse. These factors, implemented as conditions in an instructive production task, were the assumed visual identity for the reader of the objects or referents to be referred to in the instructions (visually same vs. visually different) and the assumed goal of the reader (reading to do vs. reading to learn). The results show that both factors have a strong impact on the writers' referential behavior. Visually same referents are introduced and reintroduced fairly systematically by means of perceptually overspecified NPs. Visually different referents are introduced systematically by extra propositional identification speech acts and they are reintroduced more often by attenuated anaphoric expressions. Apart from that, writers show a number of referential strategies which fit in with the assumed reader's goal. Writers in the reading to learn condition use more overspecified expressions than writers in the reading to do condition. The results contribute to a more comprehensive understanding of the factors that determine the form of referential expressions in discourse. Referential expressions which are more informative than necessary for identification purposes do not only occur when the activation level of discourse referents is low, but also when the writer anticipates specific conditions in which the information will be used by the reader.

title: Incremental speech production and referential overspecification
abstract: Speech is often produced incrementally: speakers start to articulate an utterance before knowing exactly what they are going to say. The time devoted to articulating first parts of an utterance is simultaneously used to process further information which is to be incorporated into that utterance. Empirical evidence is presented which supports the assumption that referential noun phrases are indeed often produced incrementally. This assumption explains several phenomena: first, speakers' production of overspecified ('redundant') utterances; second, irregularities concerning the ordering ofprenominal adjectives; and third, the way acoustic stress is assigned in referential noun phrases. The basic problem that a speaker faces in what is usually called 'referential communication' is that he has to refer unambiguously to a target object in the context of other objects in such a way that a listener is able to single it out from all relevant alternatives. In order to characterize a target object unambiguously in the context of others, the speaker must determine some appropriate set of distinguishing features, that is, those features that distinguish the target from all relevant alternatives. Consider the simple example depicted in Figure 1. Suppose a referential domain consists of three objects: a black and white triangle and a white circle. Suppose the white circle is the target object on which the speaker wants to focus the listener's attention. In this case the distinguishing feature is the object class. The information 'circle' is sufficient for the listener to know which object is meant by the speaker. In contrast, color would be nondistinguishing information in this case, since color does not help the listener to differentiate the circle from the white triangle. Research on referential communication has primarily been oriented toward developmental questions (for overviews, see Dickson 1981; Glucksberg et al. 1975; Shatz 1978). For most researchers, success or failure in the child's referential-communication tasks has served as an Linguistics 27 (1989), 89-110 0024-3949/89/0027-0089 $2.00 © Mouton de Gruyter, Amsterdam

title: How Redundant Are Redundant Color Adjectives? An Efficiency-Based Analysis of Color Overspecification
abstract: Color adjectives tend to be used redundantly in referential communication. I propose that redundant color adjectives (RCAs) are often intended to exploit a color contrast in the visual context and hence facilitate object identification, despite not being necessary to establish unique reference. Two language-production experiments investigated two types of factors that may affect the use of RCAs: factors related to the efficiency of color in the visual context and factors related to the semantic category of the noun. The results of Experiment 1 confirmed that people produce RCAs when color may facilitate object recognition; e.g., they do so more often in polychrome displays than in monochrome displays, and more often in English (pre-nominal position) than in Spanish (post-nominal position). RCAs are also used when color is a central property of the object category; e.g., people referred to the color of clothes more often than to the color of geometrical figures (Experiment 1), and they overspecified atypical colors more often than variable and stereotypical colors (Experiment 2). These results are relevant for pragmatic models of referential communication based on Gricean pragmatics and informativeness. An alternative analysis is proposed, which focuses on the efficiency and pertinence of color in a given referential situation.

title: Overspecification of color, pattern, and size: salience, absoluteness, and consistency
abstract: The rates of overspecification of color, pattern, and size are compared, to investigate how salience and absoluteness contribute to the production of overspecification. Color and pattern are absolute and salient attributes, whereas size is relative and less salient. Additionally, a tendency toward consistent responses is assessed. Using a within-participants design, we find similar rates of color and pattern overspecification, which are both higher than the rate of size overspecification. Using a between-participants design, however, we find similar rates of pattern and size overspecification, which are both lower than the rate of color overspecification. This indicates that although many speakers are more likely to include color than pattern (probably because color is more salient), they may also treat pattern like color due to a tendency toward consistency. We find no increase in size overspecification when the salience of size is increased, suggesting that speakers are more likely to include absolute than relative attributes. However, we do find an increase in size overspecification when mentioning the attributes is triggered, which again shows that speakers tend to refer in a consistent manner, and that there are circumstances in which even size overspecification is frequently produced.

title: Visual determinants of preferred adjective order
abstract: In referential communication, speakers refer to a target object among a set of context objects. The NPs they produce are characterized by a canonical order of prenominal adjectives: The dimensions that are easiest to detect (e.g., absolute dimensions) are commonly placed closer to the noun than other dimensions (e.g., relative dimensions). This stands in stark contrast to the assumption that language production is an incremental process. According to this incremental-procedural view, the dimensions that are easiest to detect should be named first. In the present paper, an alternative account of the canonical order effect is presented, suggesting that the prenominal adjective ordering rules are a result of the perceptual analysis processes underlying the evaluation of distinctive target features. Analyses of speakers’ eye movements during referential communication (Experiment 1) and analyses of utterance formats produced under time pressure (Experiment 2) provide evidence for the suggested perceptual classification account.

title: Reference Production as Search: The Impact of Domain Size on the Production of Distinguishing Descriptions.
abstract: When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic findings in visual search; however, unlike models of reference production, visual search models also predict that search can become very efficient under certain conditions, something that reference production models do not consider. This paper investigates the predictions of these models empirically. In two experiments, we show that the time taken to plan a referring expression-as reflected by speech onset latencies-is influenced by distractor set size and by the number of properties required, but this crucially depends on the discriminability of the properties under consideration. We discuss the implications for current models of reference production and recent work on the role of salience in visual search.

title: EEG can Track the Time Course of Successful Reference Resolution in Small Visual Worlds
abstract: Previous research has shown that language comprehenders resolve reference quickly and incrementally, but not much is known about the neural processes and representations that are involved. Studies of visual short-term memory suggest that access to the representation of an item from a previously seen display is associated with a negative evoked potential at posterior electrodes contralateral to the spatial location of that item in the display. In this paper we demonstrate that resolving the reference of a noun phrase in a recently seen visual display is associated with an event-related potential that is analogous to this effect. Our design was adapted from the visual world paradigm: in each trial, participants saw a display containing three simple objects, followed by a question about the objects, such as Was the pink fish next to a boat?, presented word by word. Questions differed in whether the color adjective allowed the reader to identify the referent of the noun phrase or not (i.e., whether one or more objects of the named color were present). Consistent with our hypothesis, we observed that reference resolution by the adjective was associated with a negative evoked potential at posterior electrodes contralateral to spatial location of the referent, starting approximately 333 ms after the onset of the adjective. The fact that the laterality of the effect depended upon the location of the referent within the display suggests that reference resolution in visual domains involves, at some level, a modality-specific representation. In addition, the effect gives us an estimate of the time course of processing from perception of the written word to the point at which its meaning is brought into correspondence with the referential domain.

title: Pragmatic Versus Form-Based Accounts of Referential Contrast: Evidence for Effects of Informativity Expectations
abstract: Characterizing the relationship between form-based linguistic knowledge and representation of context has long been of importance in the study of on-line language processing. Recent experimental research has shown evidence of very rapid effects of referential context in resolving local indeterminacies on-line. However, there has been no consensus regarding the nature of these context effects. The current paper summarizes recent work covering a range of phenomena for which referential contrast has been shown to influence on-line processing, including prenominal and postnominal modification, focus operators, and intonational focus. The results of the body of work suggest that referential context effects are not limited to situations in which the linguistic form of the utterance directly specifies the point of contact with context. Rather, context effects of a pragmatic, Gricean nature appear to be possible, suggesting the relationship between linguistic form and context in rapid on-line processing can be of a very indirect nature.

title: Ordering adjectives in referential communication
abstract: We contrasted two hypotheses concerning how speakers determine adjective order during referential communication. The discriminatory efficiency hypotheses claims that speakers place the most discriminating adjective early to facilitate referent identification. By contrast, the availability-based ordering hypothesis assumes that speakers produce most available adjectives early to ease production. Experiment 1 showed that speakers use more pattern-before-color modifier orders (than the reversed) when pattern, not color, distinguished the referent from alternatives, providing support for the discriminatory efficiency hypothesis. Participants also overspecified color more often than pattern, and they generally favored color-before-pattern orders, in support of the availability-based ordering hypothesis. Experiments 2 and 3 replicated both effects in a dialogue setting, where speakers’ adjective ordering was also primed by their partner’s ordering, using conjoined and non-conjoined constructions. We propose a novel model ( PASS ) that explains how discriminability and availability simultaneously influence adjective selection and ordering via competition in the speaker’s message representation.

title: Incremental interpretation at verbs: restricting the domain of subsequent reference
abstract: Participants' eye movements were recorded as they inspected a semi-realistic visual scene showing a boy, a cake, and various distractor objects. Whilst viewing this scene, they heard sentences such as 'the boy will move the cake' or 'the boy will eat the cake'. The cake was the only edible object portrayed in the scene. In each of two experiments, the onset of saccadic eye movements to the target object (the cake) was significantly later in the move condition than in the eat condition; saccades to the target were launched after the onset of the spoken word cake in the move condition, but before its onset in the eat condition. The results suggest that information at the verb can be used to restrict the domain within the context to which subsequent reference will be made by the (as yet unencountered) post-verbal grammatical object. The data support a hypothesis in which sentence processing is driven by the predictive relationships between verbs, their syntactic arguments, and the real-world contexts in which they occur.

title: The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye-movements in depicted events
abstract: Studies monitoring eye-movements in scenes containing entities have provided robust evidence for incremental reference resolution processes. This paper addresses the less studied question of whether depicted event scenes can affect processes of incremental thematic role-assignment. In Experiments 1 and 2, participants inspected agent-action-patient events while listening to German verb-second sentences with initial structural and role ambiguity. The experiments investigated the time course with which listeners could resolve this ambiguity by relating the verb to the depicted events. Such verb-mediated visual event information allowed early disambiguation on-line, as evidenced by anticipatory eye-movements to the appropriate agent/patient role filler. We replicated this finding while investigating the effects of intonation. Experiment 3 demonstrated that when the verb was sentence-final and thus did not establish early reference to the depicted events, linguistic cues alone enabled disambiguation before people encountered the verb. Our results reveal the on-line influence of depicted events on incremental thematic role-assignment and disambiguation of local structural and role ambiguity. In consequence, our findings require a notion of reference that includes actions and events in addition to entities (e.g. Semantics and Cognition, 1983), and argue for a theory of on-line sentence comprehension that exploits a rich inventory of semantic categories.

title: Uncertainty Reduction as a Measure of Cognitive Processing Effort
abstract: The amount of cognitive effort required to process a word has been argued to depend on the word's effect on the uncertainty about the incoming sentence, as quantified by the entropy over sentence probabilities. The current paper tests this hypothesis more thoroughly than has been done before by using recurrent neural networks for entropy-reduction estimation. A comparison between these estimates and word-reading times shows that entropy reduction is positively related to processing effort, confirming the entropy-reduction hypothesis. This effect is independent from the effect of surprisal.

title: Complexity Metrics in an Incremental Right-Corner Parser
abstract: Hierarchical HMM (HHMM) parsers make promising cognitive models: while they use a bounded model of working memory and pursue incremental hypotheses in parallel, they still achieve parsing accuracies competitive with chart-based techniques. This paper aims to validate that a right-corner HHMM parser is also able to produce complexity metrics, which quantify a reader's incremental difficulty in understanding a sentence. Besides defining standard metrics in the HHMM framework, a new metric, embedding difference, is also proposed, which tests the hypothesis that HHMM store elements represents syntactic working memory. Results show that HHMM surprisal outperforms all other evaluated metrics in predicting reading times, and that embedding difference makes a significant, independent contribution.

title: Uncertainty and Expectation in Sentence Processing: Evidence From Subcategorization Distributions
abstract: There is now considerable evidence that human sentence processing is expectation based: As people read a sentence, they use their statistical experience with their language to generate predictions about upcoming syntactic structure. This study examines how sentence processing is affected by readers' uncertainty about those expectations. In a self-paced reading study, we use lexical subcategorization distributions to factorially manipulate both the strength of expectations and the uncertainty about them. We compare two types of uncertainty: uncertainty about the verb's complement, reflecting the next prediction step; and uncertainty about the full sentence, reflecting an unbounded number of prediction steps. We find that uncertainty about the full structure, but not about the next step, was a significant predictor of processing difficulty: Greater reduction in uncertainty was correlated with increased reading times (RTs). We additionally replicated previously observed effects of expectation violation (surprisal), orthogonal to the effect of uncertainty. This suggests that both surprisal and uncertainty affect human RTs. We discuss the consequences for theories of sentence comprehension.

title: Eye’ll Help You Out! How the Gaze Cue Reduces the Cognitive Load Required for Reference Processing
abstract: Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cognitive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual‐world eye‐tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.

title: The Index of Cognitive Activity as a Measure of Cognitive Processing Load in Dual Task Settings
abstract: Increases in pupil size have long been used as an indicator of cognitive load. Recently, the Index of Cognitive Activity (ICA), a novel pupillometric measure has received increased attention. The ICA measures the frequency of rapid pupil dilations, and is an interesting complementary measure to overall pupil size because it disentangles the pupil response to cognitive activity from effects of light input. As such, it has been evaluated as a useful measure of processing load in dual task settings coordinating language comprehension and driving. However, the cognitive underpinnings of pupillometry, and any differences between rapid small dilations as measured by the ICA and overall effects on pupil size are still poorly understood. Earlier work has observed that the ICA and overall pupil size may not always behave in the same way, reporting an increase in overall pupil size but decrease in ICA in a dual task setting. To further investigate this, we systematically tested two new dual-task combinations, combining both language comprehension and simulated driving with a memory task. Our findings confirm that more difficult linguistic processing is reflected in a larger ICA. More importantly, however, the dual task settings did not result in an increase in the ICA as compared to the single task, and, consistent with earlier findings, showed a significant decrease with a more difficult secondary task. This contrasts with our findings for pupil size, which showed an increase with greater secondary task difficulty in both tasks. Our results are compatible with the idea that although both pupillometry measures are indicators of cognitive load, they reflect different cognitive and neuronal processes in dual task situations.

title: Pupillometry reveals processing load during spoken language comprehension
abstract: This study investigated processing effort by measuring peoples’ pupil diameter as they listened to sentences containing a temporary syntactic ambiguity. In the first experiment, we manipulated prosody. The results showed that when prosodic structure conflicted with syntactic structure, pupil diameter reliably increased. In the second experiment, we manipulated both prosody and visual context. The results showed that when visual context was consistent with the correct interpretation, prosody had very little effect on processing effort. However, when visual context was inconsistent with the correct interpretation, prosody had a large effect on processing effort. The interaction between visual context and prosody shows that visual context has an effect on online processing and that it can modulate the influence of linguistic sources of information, such as prosody. Pupillometry is a sensitive measure of processing effort during spoken language comprehension.

title: Early effects of word surprisal on pupil size during reading
abstract: Early effects of word surprisal on pupil size during reading Stefan L. Frank (s.frank@ucl.ac.uk) Department of Cognitive, Perceptual and Brain Sciences University College London 26 Bedford Way, London WC1H 0AP, United Kingdom Robin L. Thompson (robin.thompson@ucl.ac.uk) Deafness, Cognition and Language Research Centre Department of Cognitive, Perceptual and Brain Sciences University College London 49 Gordon Square, London WC1H 0PD, United Kingdom  This study investigated the relation between word surprisal and pupil dilation during reading. Participants’ eye movements and pupil size were recorded while they read single sentences. Surprisal values for each word in the sentence stimuli were estimated by both a recurrent neural network and a phrase- structure grammar. Higher surprisal corresponded to longer word-reading time, and this effect was stronger when surprisal values were estimated by the neural network. In addition, there was an early, positive effect of surprisal on pupil size, from about 250 ms before word fixation until 100 ms after fixation. This early effect, which was only significant for the network- based surprisal estimates, is suggestive of a preparation-based account of surprisal.  Reading; Eye tracking; Pupillometry; Sentence comprehension; Surprisal; Recurrent neural network; Phrase- structure grammar Introduction Language comprehension is mostly incremental: When lis- tening to or reading a sentence, each word is immediately integrated with information from the sentence so far (e.g., Just, Carpenter, & Woolley, 1982). It has been argued that the amount of cognitive effort required to process a given word can be quantified by its surprisal (Hale, 2001; Levy, 2008), an information-theoretic measure of the extent to which the word’s occurrence was unexpected. Formally, if w 1...t de- notes the sentence’s first t words, the surprisal of the fol- lowing word is: surprisal(w t+1 ) = − log P(w t+1 |w 1...t ). These values can be estimated by any language model that assigns probabilities to word sequences. The relationship between surprisal and cognitive load (i.e., relative difficulty in processing) has indeed been observed in reading studies: Words with higher surprisal values take longer to read, which accounts for several phenomena in sen- tence comprehension, such as garden-path effects (Brouwer, Fitz, & Hoeks, 2010) and anti-locality effects (Levy, 2008). More generally, reading times at each word in sentences or texts have been shown to correlate with surprisal (e.g., Boston, Hale, Patil, Kliegl, & Vasishth, 2008; Demberg & Keller, 2008; Fernandez Monsalve, Frank, & Vigliocco, 2012; Frank & Bod, 2011; Smith & Levy, 2008). Here, we investigate an alternative empirical index of cog- nitive load; one that can be measured continuously and with precise time-resolution: pupil size. By analyzing how and when effects of word surprisal appear in pupillometry data, we are able to use a physiological measure to investigate the fine-grained time course of sentence-comprehension pro- cesses. A large number of studies, using a variety of tasks, have looked at the relationship between cognitive load and pupil dilation (for a recent overview, see Laeng, Sirois, & Gre- deb¨ack, 2012). Although these studies differ in how cogni- tive load is operationalized, increased cognitive load is invari- ably found to result in larger pupil size. In a non-linguistic context, Preuschoff, ’t Hart, and Einh¨auser (2011) showed that pupil size (and therefore, presumably, cognitive load) in- creases when a stimulus is less expected. They had partici- pants perform a simple gambling task and found that expe- riencing surprise causes pupil dilation: Pupil size correlated not with the gambling outcome itself but with its unexpected- ness. Whether unexpectedness of words in sentences also results in pupil dilation is still an open question. In fact, there has been only very little pupillometry research in psycholinguis- tics. Engelhardt, Ferreira, and Patsenko (2010) found that a mismatch between syntactic and prosodic structure of audito- rily presented sentences results in larger pupil size compared to a condition in which the two structures matched. In another sentence-listening study, Piquado, Isaacowitz, and Wingfield (2010) found a pupil response to both syntactic complexity and sentence length. To the best of our knowledge, there ex- ists only two published studies in which pupillometry is ap- plied during sentence reading: Raisig, Hagendorf, and Van der Meer (2012) presented participants with written descrip- tions of simple events in everyday activities and found in- creased pupil dilation when the order of presentation was incongruent with the actual temporal order of the described activities. Just and Carpenter (1993) compared object- and subject-relative clauses and found increased reading times and pupil dilation on the object-relatives, which have long been known to be more difficult to process (Hakes, Evans, & Brannon, 1976). Moreover, the occurrence of a semantically implausible word resulted in increased pupil size compared to a plausible-word condition. Here, we did not compare particular sentence pairs but, in- stead, investigate the general relation between word surprisal

title: The role of prosody in the interpretation of structural ambiguities: A study of anticipatory eye movements
abstract: An eye-tracking experiment examined whether prosodic cues can affect the interpretation of grammatical functions in the absence of clear morphological information. German listeners were presented with scenes depicting three potential referents while hearing temporarily ambiguous SVO and OVS sentences. While case marking on the first noun phrase (NP) was ambiguous, clear case marking on the second NP disambiguated sentences towards SVO or OVS. Listeners interpreted case-ambiguous NP1s more often as Subject, and thus expected an Object as upcoming argument, only when sentence beginnings carried an SVO-type intonation. This was revealed by more anticipatory eye movements to suitable Patients (Objects) than Agents (Subjects) in the visual scenes. No such preference was found when sentence beginnings had an OVS-type intonation. Prosodic cues were integrated rapidly enough to affect listeners' interpretation of grammatical function before disambiguating case information was available. We conclude that in addition to manipulating attachment ambiguities, prosody can influence the interpretation of constituent order ambiguities.

title: Random effects structure for confirmatory hypothesis testing: Keep it maximal.
abstract: Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the standards that have been in place for many decades. Through theoretical arguments and Monte Carlo simulation, we show that LMEMs generalize best when they include the maximal random effects structure justified by the design. The generalization performance of LMEMs including data-driven random effects structures strongly depends upon modeling criteria and sample size, yielding reasonable results on moderately-sized samples when conservative criteria are used, but with little or no power advantage over maximal models. Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data from populations where subjects and/or items vary in their sensitivity to experimental manipulations always generalize worse than separate F1 and F2 tests, and in many cases, even worse than F1 alone. Maximal LMEMs should be the 'gold standard' for confirmatory hypothesis testing in psycholinguistics and beyond.

title: Saccadic overhead: Information-processing time with and without saccades
abstract: Information-processing time was compared for serial and spatially distributed visual presentations with performance measures that permit the separation of total time into its during-display and post-display components. For all subjects, there was a significant saccadic overhead, that is, less time was required with the serial format, which allowed data access without eye movements. However, the magnitude of the overhead decreased as task complexity increased. All subjects were able to exercise some control over the distribution of total processing time, trading off short during-display times with longer post-display times and vice versa.