Retrieving structures from memory causes difficulty during incremental processing

Cory Shain1, Marten van Schijndel1, Richard Futrell2, Edward Gibson2, and William Schuler1 1OSU,2MIT Any incremental model of sentence processing where an abstract meaning representation is built up word by word must involve storage and retrieval of information about previously encountered material from some memory store. The retrieval operations have been hypothesized to be associated with increased processing time [4, 8, 12], and this prediction has been borne out in experiments using constructed stimuli [4, 5, 1, 11]. However, memory-based latency effects have been null or even negative in broad-coverage reading time experiments using naturally-occurring text data that included baseline controls for n-gram and PCFG surprisal [2, 10]. There are potential confounds associated with each type of stimulus. The effects observed in constructed stimuli could be due to (1) information theoretic phenomena (e.g., surprisal) that such experiments rarely control for, (2) limited syntactic domains (e.g., relative clauses), or (3) lack of discourse context, rather than due to difficulty retrieving information from working memory. On the other hand, the lack of positive latency effects in studies using naturally-occurring input could be (1) because of the small number of subjects – ten – in the Dundee corpus [7] used by e.g. [2, 10] or (2) because naturally-occurring newswire texts might contain too low a proportion of memory-intensive constructions to reveal a generalized memory effect. The present work seeks to address these concerns using the new Natural Stories corpus [3], a ‘constructed-natural’ self-paced reading corpus that occupies an intermediary position between constructed stimuli on the one hand and naturally-occurring stimuli on the other. The corpus consists of ten fluent, context-rich narrative texts that are constructed in order to contain a high proportion of memory-intensive constructions, with self-paced reading data collected from 181 subjects, yielding 848,207 reading events. These data were partitioned into exploratory (1/3) and confirmatory (2/3) datasets, and the exploratory corpus was used for empirical selection from a number of plausible implementations of two broad models of human sentence processing: the Dependency Locality Theory (DLT) [4] and left-corner parsing [6, 8]. The DLT predicts processing difficulty as a function of dependency length (defined as the number of intervening discourse referents), while left-corner models predict processing difficulty as a function of parser operations that involve memory retrieval. Using linear mixed-effects models and a strong baseline that includes controls for both n-gram and PCFG surprisal [9], we find highly significant inhibitory effects for both DLT integration cost and a left-corner parsing operation that corresponds to the termination of a phrasal constituent (Table 1). The DLT effect improves considerably with certain independently-motivated modifications to its implementation (see [9] for details). The DLT and left-corner predictors make independent contributions to model fit, with both predictors improving significantly over the baseline and over each other individually. To our knowledge, this is the first strong evidence of memory effects in broad-coverage sentence processing, and our results suggest the cognitive reality of important constructs from syntactic theory such as dependency (DLT) and constituency (left-corner parsing). The existence of separate effects for the DLT, which encodes semantic information such as asymmetries in referential status, and left-corner models, which are sensitive only to syntactic tree configurations, might suggest the existence of separate effects for computing dependencies on the one hand and retrieving syntactic structures from memory on the other, a question we leave to future research. We also note that all effect sizes are quite small (≤ 4 ms), a fact that might shed light on the aforementioned discrepancy between experiments based on naturally-occurring vs. constructed stimuli. Our results suggest that memory effects do indeed exist in broad-coverage sentence processing, but are so small that they may not be observable without difficult stimuli and/or large corpora that include data from many subjects.