Evidence of semantic processing difficulty in naturalistic reading

Cory Shain1, Richard Futrell2, Marten van Schijndel3, Edward Gibson2, William Schuler1, and Evelina Fedorenko2; 1Ohio State, 2MIT, 3Johns Hopkins Language is a powerful vehicle for conveying our thoughts to others and inferring thoughts from their utterances. Much research in sentence processing has investigated factors that affect the relative difficulty of processing each incoming word during language comprehension, including in rich naturalistic materials [4, 8, 6, 24, 27, 23]. However, in spite of the fact that language is used to convey and infer meanings, prior research has tended to focus on lexical and/or structural determinants of comprehension difficulty. This focus has plausibly been due to the fact that lexical and syntactic properties can be accurately estimated in an automatic fashion from corpora [11] or using high-accuracy automatic incremental parsers [22, 26]. Comparable incremental semantic parsers are currently lacking. However, recent work in machine learning has found that distributed representations of word meanings — based on patterns of lexical co-occurrence — contain a substantial amount of semantic information [16, 14], and predict human responses in a range of psycholinguistic tasks [19, 2, 21, 5]. To examine the effects of semantic relationships among words on comprehension difficulty, we estimated a novel measure — incremental semantic relatedness — for three naturalistic reading time corpora: Dundee [12], UCL [7], and Natural Stories [9]. In particular, we embedded all three corpora using GloVe vectors [20] pretrained on the 840B word Common Crawl dataset, then computed the mean vector distance between the current word and all content words preceding it in the sentence. This provides a measure of a word’s semantic relatedness to the words that precede it without requiring the construction of carefully normed stimuli, permitting us to evaluate semantic relatedness as a predictor of comprehension difficulty in a broad-coverage setting. Hypothesis testing was done with ablative likelihood ratio testing of linear mixed effects models, controlling for word length in characters, position in the sentence, 5-gram surprisal as computed by KenLM [11] trained on Gigaword 3 [10], and PCFG surprisal as computed by the [26] parser trained on the WSJ corpus [15] re-annotated into Generalized Categorial Grammar [18]. 1 We found a significant positive effect of mean cosine distance on reading time duration in each corpus. In summary, in line with previous work that has shown that the semantic relationship between a target word and its context affects comprehension in constructed stimuli presented in isolation [13, 17], we provide strong broad-coverage evidence of this factor, over and above linear (5-gram) and syntactic (PCFG) models of linguistic expectation. Our results are consistent with at least two (perhaps complementary) interpretations. Semantically related context might facilitate processing of the target word through spreading activation [1]. Or vector distances might approximate the surprisal values of a semantic component of the human language model, thus yielding a rough estimate of semantic surprisal. Future advances in incremental semantic parsing may permit more precise exploration of these possibilities.