Cognition 128 (2013) 302–319

Contents lists available at SciVerse ScienceDirect

Cognition
journal homepage: www.elsevier.com/locate/COGNIT

The effect of word predictability on reading time
is logarithmic q
Nathaniel J. Smith a,⇑, Roger Levy bab

Department of Cognitive Science, University of California, San Diego, United States
Department of Linguistics, University of California, San Diego, United Statesa r t i c l e

i n f oArticle history:
Received 10 September 2012
Revised 4 February 2013
Accepted 16 February 2013

Keywords:
Psycholinguistics
Expectation
Probabilistic models of cognition
Information theory
Readinga b s t r a c t
It is well known that real-time human language processing is highly incremental and context-driven, and that the strength of a comprehender’s expectation for each word encountered is a key determinant of the difﬁculty of integrating that word into the preceding
context. In reading, this differential difﬁculty is largely manifested in the amount of time
taken to read each word. While numerous studies over the past thirty years have shown
expectation-based effects on reading times driven by lexical, syntactic, semantic, pragmatic, and other information sources, there has been little progress in establishing thequantitative relationship between expectation (or prediction) and reading times. Here,by combining a state-of-the-art computational language model, two large behavioral
data-sets, and non-parametric statistical techniques, we establish for the ﬁrst time thequantitative form of this relationship, ﬁnding that it is logarithmic over six orders of magnitude in estimated predictability. This result is problematic for a number of establishedmodels of eye movement control in reading, but lends partial support to an optimal perceptual discrimination account of word recognition. We also present a novel model inwhich language processing is highly incremental well below the level of the individual
word, and show that it predicts both the shape and time-course of this effect. At a moregeneral level, this result provides challenges for both anticipatory processing and semantic
integration accounts of lexical predictability effects. And ﬁnally, this result provides evidence that comprehenders are highly sensitive to relative differences in predictability –even for differences between highly unpredictable words – and thus helps bring theoreticalunity to our understanding of the role of prediction at multiple levels of linguistic structure
in real-time language comprehension.
Ó 2013 The Authors. Published by Elsevier B.V. All rights reserved.

1. Introduction
Making probabilistic predictions about the future is anecessary component of essentially every task that theqThis is an open-access article distributed under the terms of theCreative Commons Attribution License, which permits unrestricted use,
distribution, and reproduction in any medium, provided the original
author and source are credited.
⇑ Corresponding author. Address: School of Informatics, University of
Edinburgh, Informatics Forum 3.29, 10 Crichton Street, Edinburgh EH89AB, United Kingdom.
E-mail address: nathaniel.smith@ed.ac.uk (N.J. Smith).

brain performs, to the point that it has been proposed asa fundamental principle underlying its operation (Bar,2009). One example of this is in language comprehension:
As you read this text, you are unconsciously anticipating
upcoming words based on the constantly-evolving context.
For example, the sentence.
(1)

My brother came inside to. . .

may well continue any number of ways, but native English
speakers are in general agreement—and you will likely
immediately recognize—that the sentence

0010-0277/$ - see front matter Ó 2013 The Authors. Published by Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.cognition.2013.02.013

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319(2)

The children went outside to. . .

is almost certain to continue with the word play.
Although play is perfectly reasonable as a continuationof (1), in (2) it is read more quickly on average (Ehrlich& Rayner, 1981; Kliegl, Nuthmann, & Engbert, 2006;
McDonald & Shillcock, 2003a; Rayner & Well, 1996). A
wide range of studies have shown that such effects ofpredictability or expectation for speciﬁc words affectnot only reading times but also neural responses(DeLong, Urbach, & Kutas, 2005; Kutas & Hillyard,
1984; Kutas & Federmeier, 2011; Van Berkum, Brown,
Zwitserlood, Kooijman, & Hagoort, 2005; Wicha, Bates,
Moreno, & Kutas, 2003) and interpretation of temporarily
ambiguous input (Altmann & Kamide, 1999; Dahan &Tanenhaus, 2004; Kamide, Altmann, & Haywood, 2003;
Knoeferle, Crocker, Scheepers, & Pickering, 2005; Tanenhaus,
Spivey-Knowlton, Eberhard, & Sedivy, 1995).
A second, related strand of research has shown that
incremental processing difﬁculty is also affected by expectations for more abstract levels of linguistic content, including
the predictability of different syntactic (Demberg & Keller,
2008; Ferreira et al., 1986; McRae, Spivey-Knowlton, &Tanenhaus, 1998), semantic (Federmeier & Kutas, 1999),
and pragmatic (Ni, Crain, & Shankweiler, 1996) structures.
However, the relationship between the effects of expectations for speciﬁc-words and expectations for more abstract
structures remains poorly understood. The most widespread method for assessing expectations for speciﬁcwords is the cloze task (Taylor, 1953), in which native
speakers are asked to write continuations of an incomplete
sentence; in the examples above, play is the ﬁrst word inover 90% of continuations of (2) but almost never appears
as the ﬁrst word of continuations of (1). However, the cloze
task makes it quite difﬁcult to precisely measure predictabilities <5–10%, and it is commonly assumed that differences in lexical expectation between items in this range
do not produce behavioral effects. This contrasts withstudies involving more abstract levels of linguistic structure, where expectation-based effects are observed eventhough the speciﬁc word instantiating the structure may
rarely or never be produced in a cloze task. To take one recent example, Levy, Fedorenko, Breen, & Gibson, (2012)
showed that the word who and the immediately following
region of the sentence were read more quickly in sentences
like (3) than in sentences like (4):(3) After the show, a performer who had really
impressed the audience bowed.
(4) After the show, a performer bowed who had really
impressed the audience.

The word who never occurs in practice as a cloze continuation in either context (unpublished data), and so this result would conventionally be interpreted as arising fromsyntactic expectations (Hale, 2001; Ilkin & Sturt, 2011;
Levy, 2008; Levy et al., 2012; Lau, Stroud, Plesch, & Phillips,
2006; Staub & Clifton, 2006): In (4) who is introduced by agrammatical construction (relative-clause extraposition)that corpus data indicate is both lower frequency and less303

likely given the grammatical context than the constructionin (3) (ordinary postmodiﬁcation by a relative clause), eventhough both are infrequent and unlikely in absolute terms.
Probability theory, however, tells us that such differences in syntactic expectation should also produce differences in lexical expectation, even if these latter
differences are too small to measure via the cloze task.
We can quantify the predictability of a word w in context
C as its conditional probability of occurrence in that context, P(wjC). Similarly, we write the predictability of a syntactic construction S as P(SjC). In these particular contexts,
w = who can only occur if S = relative clause. The laws ofconditional probability then let us decompose the lexical
predictability as the product of two terms (Demberg & Keller, 2008; Fossum & Levy, 2012; Roark, Bachrach, Cardenas,& Pallier, 2009):

PðwhojCÞ ¼ Pðrel: clausejCÞ  Pðwhojrel: clause; CÞThe ﬁrst term is the syntactic predictability, and the second measures the likelihood that this relative clause will
begin with the word who (as opposed to, say, that). The latter is presumably roughly constant between these twocontexts, which means that while the precise lexical predictabilities in (3) and (4) are too small to measure directly,
the ratio between them should be similar to the ratio between their syntactic predictabilities.
Motivated by such considerations, Hale has suggested
that syntactic and other types of abstract expectations
may affect processing difﬁculty purely by modulating lexical predictability, which under the surprisal theory of
incremental language processing is measured as log-probability (Hale, 2001; Levy, 2008). Within surprisal theory,
lexical predictability forms a ‘‘causal bottleneck’’ throughwhich the many different kinds of more abstract expectation discussed above must act. But as the above example
shows, an essential requirement for this theory is that
small absolute differences in expectation for low-predictability words must be capable of producing relatively large
effects on processing difﬁculty, and it is not known
whether this is the case. In fact, almost nothing is known
about the quantitative form of the relationship between
word predictability and the measurable correlates of processing difﬁculty such as reading time. This is in striking
contrast to the study of isolated word recognition, where
it has been known since the 1950s that recognition timevaries almost exactly as a logarithmic function of frequency1 (Howes & Solomon, 1951), and the need to explain
this pattern has motivated a wide range of theories (Adelman, Brown, & Quesada, 2006; Baayen, 2010a; Murray &Forster, 2004; Morrison, Hirsh, & Duggan, 2003; Norris,
2006). But the few extant published studies (Kliegl et al.,2006; Rayner & Well, 1996) that have investigated thequantitive relationship between word predictability andprocessing time have yielded only limited insights, particularly regarding the shape of this relationship for highly
unpredictable words, partly because of the cloze method’s
1Note that in psycholinguistics, the term frequency refers speciﬁcally to aword’s unconditional probability of occurrence without regard to context,
P(w), making it quite distinct from context-dependent predictability.

304

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

Proposed relationships between
predictability and reading time

Reading time

Linear(guessing)

Logarithmic(optimal visual
discrimination,highly incremental
processing)Super−logarithmic(could explain UID effects)

Reciprocal(hypothesized by
Narayanan & Jurafsky, 2004)

Probability (log scale)Fig. 1. Several hypothesized forms for the predictability effect, plotted inlog space. (Of course many other forms are also possible a priori; here weshow only those previously mentioned in the literature.) A linear effect ispredicted by a simple guessing model. A logarithmic effect is predicted byboth an optimal visual discrimination account (Norris, 2006) and anincremental processing account (see text). A super-logarithmic effect ispredicted by the audience design theory of uniform information density
effects (Levy & Jaeger, 2007).

limitations at measuring small differences in absolute
predictability.
Here we overcome these limitations by combining twolarge behavioral datasets of word-by-word reading times
with probabilistic language models from computational
linguistics (Chen & Goodman, 1998; Kneser & Ney, 1995;
Manning & Schütze, 1999) and nonparametric statisticalanalysis. These methods allow us to establish for the ﬁrst
time the functional form of the predictability/reading-time
relationship, and we do so over six orders of magnitude inprobability, from near-obligatory to one-in-a-million
events. (Due to the Zipﬁan distribution of language weencounter instances of the latter class of events relatively
often, even though each individual such event is extremely
rare.) We ﬁrst describe a number of potential functional
forms which have been hypothesized in the literature(see also Fig. 1)—as well as giving a new theoretical motivation for a previously-hypothesized functional form—andthen proceed to our empirical analysis.
2. Theories relating word predictability and readingtime
2.1. Simple guessing (prediction: linear)
The simplest possible curve that might relate predictability and processing time is a straight line; indeed, several modern models of eye movement control in reading
apply a logarithmic transformation to frequency, but enter
predictability linearly (Engbert, Nuthmann, Richter, &Kliegl, 2005; Reichle, Pollatsek, Fisher, & Rayner, 1998).

While we are not aware of any published justiﬁcation forthis practice, it does arise naturally from a simple and intuitive theory: Suppose that before reading each word, comprehenders make a guess at its identity by sampling fromthe distribution P(wjC) (a probability matching strategy).
If their guess is correct, then they continue undisturbed
and can read the word in some baseline amount of timetbaseline. Otherwise, they must spend some ﬁxed additionalamount of time recovering from their error—call this time
tincorrect. If comprehenders’ estimates of this probability areaccurate, then they will guess correctly on P(wjC) proportion of trials and incorrectly on (1  P(wjC)) proportion.
Thus the average reading time will be (tbaseline)+ (1  P(wjC))tincorrect—i.e., reported reading time will varylinearly with the word’s probability.
Other authors have proposed a reciprocal (Narayanan &Jurafsky, 2004) or logarithmic (Hale, 2001; Levy, 2008)
relationship, but based more on principles of elegance thanany particular mechanism. There are, however, several
more detailed reasons we might expect some speciﬁc
non-linear relationship.
2.2. Analogy with frequency (prediction: none)
At ﬁrst glance, we might expect a logarithmic relationship for predictability because of an argument by analogy:
Predictability is conceptually similar to frequency, and frequency has a logarithmic effect. However, the predictability
and frequency of any particular word vary on very different
time-scales: a word’s predictability may be radically different every single time it is encountered, because it isencountered in different contexts, but that same word’scontext-independent frequency will remain effectively
constant over a timescale of months at least. This presents
an obstacle to wholesale importing of theories designed to
explain the shape of the frequency effect. To illustrate this,consider Forster’s serial-search model (Forster, 1976;
Murray & Forster, 2004), perhaps the most explicit extant
proposal for why frequency has a logarithmic effect. In essence, this model assumes a frequency-sorted lexicon accessed by serial search; therefore accessing the 100th
most frequent word takes twice as long as accessing the50th most frequent word. Then, by Zipf’s law, these rankfrequencies turn out to be approximately equal to the log
numerical frequencies. The fact that frequencies are relatively stable over time makes a frequency-sorted lexiconplausible. Predictabilities, though, are dependent on context, and so change radically from one word to the next. A
serial search model of predictability effects would thus require us to accept a lexicon that is completely reordered before processing each word. This is difﬁcult, given that such areordering would necessarily involve examining every
lexical item, which would seem to remove the need for asecond search step.
Similarly, connectionist models of the word frequency
effect explain it as arising from connection weights which
are determined solely by training, and do not vary between
contexts during the testing phase (Plaut, McClelland,
Seidenberg, & Patterson, 1996; Seidenberg & McClelland,
1989; Zevin & Seidenberg, 2002; Zorzi, Houghton, &Butterworth, 1998). Other theories have attributed this ef-

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

fect to, for example, the age at which the word was ﬁrst
learned (Morrison & Ellis, 1995; Morrison et al., 2003), or
differences in the ease of memory retrieval as determinedby the number of previous exposures to the word, modulated by recency (Morton, 1969) or the diversity of previous contexts (Adelman et al., 2006). What all of these
theories have in common is that they attribute the classic
word frequency effect to some property of the word itself,while predictability is intrinsically a property of the interaction between a word token and its current context. We
are currently unaware of any theoretical mechanism which
would link the effects of long-term frequencies and local
predictabilities, or cause them to behave in similar quantitative fashions—and the absence of such a mechanism
makes it difﬁcult to accord any weight to the argument
by analogy.
2.3. Optimal visual discrimination (prediction: logarithmic)
Norris, in the tradition of previous work using theSequential Probability Ratio Test as a model of humanchoice reaction time (Carpenter & Williams, 1995; Gold &Shadlen, 2007; Laming, 1968; Stone, 1960), has proposeda model of word recognition as optimal visual discrimination: The Bayesian Reader (Norris, 2006; Norris, 2009). In
this theory, the comprehender receives samples of noisy
visual information at a ﬁxed rate from their perceptual system, and their goal is to identify a word to some ﬁxed degree of certainty as quickly as possible. According toBayesian principles, the proper way to do this is to initiallyset one’s belief in the word’s identity to match its priorprobability of occurrence—i.e., its predictability. Then, as
each input sample arrives, this belief is updated by multiplying in the likelihood of the new sample and renormalizing. When the belief reaches some threshold, the word is
declared to be identiﬁed. When transformed to log-probability space, this multiplication becomes addition, and thus
the belief update process becomes a random walk in which
each sample on average causes the log posterior probability of the correct word to increase with a near-constantstep size. The expected number of steps—and thus the expected time—before reaching threshold for the word’s true
identity is thus linear in the log of the word’s prior probability, so that in this model, the amount of time required to
identify a word is proportional to its log-predictability(Carpenter & Williams, 1995).
2.4. Highly incremental processing (prediction: logarithmic)
Here we introduce a second theory which predicts alogarithmic relationship between word probability andcomprehension time. In fact, we show that if processingis ‘‘incremental enough’’, then such a relationship will arise
almost inevitably.
We start from the observation that, so far as we candetermine, stimuli which have higher predictability areprocessed more efﬁciently in every task and species where
this has been studied (e.g., Carpenter & Williams, 1995;
Froehlich, Herbranson, Loper, Wood, & Shimp, 2004; Janssen & Shadlen, 2005; Pang, Merkel, Egeth, & Olton, 1992).
We take this as evidence that there are domain-indepen-305dent mechanisms by which predictability modulates the
efﬁciency of cognitive processing, and assume that therelationship between word predictability and reading timearises because of these mechanisms affecting linguistic
processing, rather than some mechanism speciﬁc to language comprehension (Smith & Levy, 2008). (Of course,
the processes which produce expectancies for particular
words are themselves highly sensitive to linguistic structure and usage; here we’re speaking only of the mechanisms that link expectancies to reading time.) This
assumption alone does not suggest any speciﬁc quantitative relationship between predictability and reading time.
But if this is an instance of a more general predictability effect in which processing time for any stimulus is sensitive
to that stimulus’s predictability, P(stimulusjcontext), thenwe need to ask what constitutes a linguistic ‘stimulus’.
Words are a privileged unit of linguistic representation,and psycholinguistic theories therefore tend to assume
that cognitive mechanisms for manipulating linguistic representations will be sensitive to the properties of individual words. But are words the units by which real-time
language comprehension proceeds?
Our theory’s second assumption is that they are not:that instead, language comprehension proceeds incrementally, by which we mean that processing a word involves
processing a sequence of sub-word fragments (Tanenhaus
et al., 1995). In auditory comprehension, this is well established; speech unfolds over time, and if a speaker says candy then the listener’s language processor will be quite
happy to start working on the initial /kæn-/ before they
hear /-di/ (Tanenhaus et al., 1995). In reading, incrementality at the sub-word level is less well established, but might
involve this same process after phonological recoding(Frost, 1998), or alternatively might involve multiple visualfeatures (Morton, 1969) which arrive with different latencies and are processed in sequence. (Note that this does not
require that the order in which features arrive matches theleft-to-right order of letters within the word.)
Putting these two assumptions together, we have thatthe processing time for each fragment depends on the predictability of that fragment. Furthermore, we assume thatprediction for each fragment takes into account the previous fragments, and that the total time required to process
two sequences of fragments is the sum of the time required
to process the ﬁrst sequence plus the time required to process the second. These conditions are trivially satisﬁed if
fragments are processed in a strictly serial manner, but thisis not a requirement; they are also satisﬁed by, e.g., models
in which the processing for adjacent fragments overlaps intime, but uses a limited pool of shared computational resources so that higher degrees of parallelism result inslower overall processing. This is analogous to the observation that while there may be overlap in the processing ofadjacent words (i.e., spillover effects), nonetheless reading
multiple words takes longer than reading a single word.
In this model, the predictability of words per se has nodirect effect on their reading time; unpredictable words
take longer to process only because they contain unpredictable fragments. This would seem to make it difﬁcult
to test the theory, because experimentally we can only
measure word predictability and word reading time, not306

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

fragment predictability and fragment reading time. But
fortunately, effects at the fragment level turn out to produce characteristic patterns at the word level. This follows
from another difference between predictability and frequency. With frequency, there is no necessary relationshipbetween fragment frequency and word frequency; e.g., aword can be rare without containing any rare syllables.
True word frequency effects must therefore arise fromwhole-word processing. By contrast, the axioms of probability dictate that the context-conditional predictabilityof a word is the product of the context-conditional predictability of its parts,

Pð=kæ=jcontextÞ ¼ Pð=kæn-=jcontextÞ
 Pð=-di=jcontext; =kæn-=Þ:More formally, take a word with conditional probability
pword that is composed of k fragments with conditional
probabilities p1, . . . , pk respectively. (E.g., if candy isprocessed as /kæn-/, /-di/ then k = 2; processing it as /k-/,/-æ-/,/-n-/, /-d-/, /-i/ gives k = 5. Processing it truly continuously gives k = 1.) And, let f(x) be the function that gives
the processing time for a fragment that has probability x(we assume that there is a single f for all fragment types).
Then looking only at the portion of processing time which
is dependent on predictability, we have two equalities:

pword ¼ p1      pk
total processing time ¼ f ðp1 Þ þ    þ f ðpk Þ:
These equations let us simulate the total processing time
that
would
resultfromdifferent
choicesofpword, k, f(x), and p1, . . . , pk. Fig. 2 shows the result of such
simulations, and reveals a regularity: As k increases, thetotal processing time becomes a better and better approximation to a logarithmic function of word predictability pword:

total processing time  h log pword(Here h is an arbitrary scaling parameter). Ultimately, thispattern is caused by the fact that in the equations above,

2.5. Uniform information density (prediction: superlogarithmic)
The uniform information density (UID) effect is thatspeakers seem to use various strategies to lengthen or
shorten parts of their utterances so that the average predictability (as measured in bits) per unit time ends up
being roughly constant (Aylett & Turk, 2004; Genzel &Charniak, 2002; Jaeger, 2010; Piantadosi, Tily, & Gibson,2011). Levy and Jaeger (2007) proposed that one possible
source of this pattern is as an audience design strategy. If
comprehension difﬁculty induced by low-predictability
words grows more quickly than the logarithm, then outof all ways of distributing a ﬁxed amount of information
across an utterance, the one which adheres most closely
to the UID principle is also the one which will produce
the lowest total comprehension difﬁculty. Intuitively, thisoccurs because for a super-logarithmic difﬁculty curve,
Fig. 1, peaks in unpredictability produce a disproportionate
amount of difﬁculty, which cannot be balanced out by an

(b)
k=1

k = 100

k = 10k=1

Whole−word probability (log scale)

Whole−word total processing time

Whole−word total processing time(a)probabilities multiply, while times add. If we set
f(x) = hlog x, then the above approximation becomes exact for all k, because logarithms convert products intosums. This makes the logarithm the unique ﬁxed point of
this process, which other choices of f(x) converge to as k increases. (A proof is given in Appendix A.)
We do not, of course, suggest that the brain is actually
literally calculating any limits; presumably some speciﬁc
f(x) and k apply for each word that is read. What the above
analysis means, though, is that so long as k is large—that is,processing is ‘highly incremental’—then we will be near
the limit, and the details of choice of f(x) or the distributionof probability within the word will have only minimal effect on the observed whole-word reading time. A logarithmic reading time curve arises inevitably—and, perhaps,
epiphenomenally—from using a coarse whole-word measure to examine a collection of ﬁne-grained sub-word processes, each of which are sensitive to predictability.

k = 10k = 100

Whole−word probability (log scale)Fig. 2. These graphs show the whole-word processing times resulting from different variants of the incremental processing model. We consider: A linear
effect at the fragment level (a, f(x) = x) versus a reciprocal effect (b, f(x) = 1/x), for different values of k. For k > 1, we also consider two different possibilities
pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
for how probability is distributed among the fragments: Either uniformly (pi ¼ k pword , solid lines) or with later fragments more predictable than earlierðkþ1iÞ2
fragments (pi ¼ pk
with pk chosen so that p1    pk = pword, dashed lines). In all cases, more highly incremental processing (larger k) produces alogarithmic effect at the word level (f(p1)+    +f(pk)  log pword).

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

adjacent trough of similar size. So the logic of the proposedmechanism is: if producers attempt to make their utterances easy to comprehend, and if predictability has asuper-logarithmic effect on comprehension time, then producers should adhere to the UID principle. The logic ofempirical inference then inverts this: producers in many
circumstances do adhere to the UID principle, and there
must be some reason for this, so we should be predisposed
to expect a super-logarithmic relationship between predictability and reading time; and, if one is found, thatwould provide further support for this account of UID
effects.
3. Materials and methods
Accurately assessing the shape of the word predictability effect requires a large number of data points distributed
evenly over a wide range of predictability values. The availability of such data has previously been restricted by the
difﬁculty and expense of gathering cloze data, and its analysis limited by the use of factorial designs. Three aspects of
our approach allow us to overcome these challenges. First,
instead of relying on cloze, we estimate word probabilities
using a state-of-the-art computational language modeltrained on a large corpus. While undoubtedly more errorful than good cloze norming, this allows us to estimate predictability for relatively unexpected words and over verylarge stimulus sets, which compensates for the increasein noise. Other psycholinguistic studies have used suchcomputational methods (Boston, Hale, Kliegl, Patil, & Vasishth, 2008; Demberg & Keller, 2008; Roark et al., 2009);
the primary difference is that our language model is chosen to give best-effort broad-coverage word probability
estimates, not to proxy for any particular psycholinguistic
theory (Frank & Bod, 2011) or to give high-quality estimates for speciﬁc grammatical structures (Levy, 2008).
Second, we avoid the factorial approach in favor of a
spline-based regression technique designed for measuring
non-linear curve shapes (Wood, 2006). (For a previous
application of this technique to psycholinguistic data, see
Baayen, 2010b.) This also enables us to control for confounds (e.g., word frequency) post hoc, which allows us
to analyze large stimuli sets using relatively natural texts
rather than carefully normed sentences. Finally, the useof regression allows us to directly ask how the probabilityof word n affects reading time for word n + 1, after controlling for the probability of word n + 1. This reduces confounding by controlling for word-to-word correlations infrequency, predictability, etc. More importantly, it gives
us a new and powerful way to measure spill-over effects(Mitchell, 1984; Rayner, 1998), letting us better capture
predictability’s full effect while additionally giving insight
into its time-course.
3.1. Eye-tracking
First pass gaze durations (Rayner, 1998) were extracted
from the English portion of the Dundee corpus (Kennedy,
Hill, & Pynte, 2003), which records eye movements of 10
native speakers each reading 51,502 words of British

307

newspaper text. Previous work (Demberg & Keller, 2008;
Frank & Bod, 2011; Kennedy, Pynte, Murray, & Paul, inpress) has reported predictability effects in this corpus,
but did not examine curve shape.
3.2. Self-paced reading
Moving-window self-paced reading times (Just, Carpenter, & Woolley, 1982) were measured for 35 UCSD undergraduate native speakers each reading short (292–902
word) passages drawn from the Brown corpus of American
English (2860–4999 total words per participant, mean
3912). In this paradigm, the participant must press a button to reveal each word in turn, and the time elapsed between button presses is recorded. Three participants withcomprehension–question performance at chance were
excluded.
3.3. Probability estimation
Interpolated modiﬁed Kneser–Ney trigram word probabilities (Chen & Goodman, 1998; Kneser & Ney, 1995) were
estimated from the British National Corpus (BNC Consortium, 2001) using SRILM v1.5.7 (Stolcke, 2002), and combined with a conditional bigram cache (Goodman, 2001).
Self-paced reading analyses were adjusted for British/
American spelling differences using VARCON (Atkinson,2004). Our primary consideration in selecting this model
was to maximize what Frank and Bod (2011) term ‘linguistic accuracy’, i.e., the model’s ability to accurately predictwords in corpora (perplexity), without regard to behavioral
data. We certainly do not claim that this model is an appropriate theory of how the human comprehension system
goes about making predictions. But, to the extent that
our model and the brain are both attempting to achieve
linguistic accuracy, they should arrive at numerically similar estimates (see also Fossum & Levy, 2012), and the analyses we present here depend only on our estimatedprobabilities acting as an accurate statistical proxy forthe true subjective probabilities.
One possible source of inaccuracy is that in practice, ourmodel relies primarily on local context for estimating predictabilities; in this respect it is similar to the transitional
probabilities used in previous research (Demberg & Keller,
2008; Frisson, Rayner, & Pickering, 2005; McDonald &Shillcock, 2003a; McDonald & Shillcock, 2003b), though
we use a larger local context and a substantially moresophisticated estimation procedure. In addition, the bigram cache portion of our model reaches beyond local context to create increased expectancies for repeatedmentions of words and short phrases across the entiretyof each stimulus text. Nonetheless, there remain a varietyof long-distance linguistic dependencies induced by syntax, semantics, etc., which this model captures only imperfectly. This is in contrast to humans, who are generally
sensitive to such long-distance dependencies. While thissensitivity is of great importance to theories about human
expectancy generation, it does not affect our analyses here
unless such dependencies have a large and systematic effect on the numerical magnitude of expectation for a large
proportion of the words in our stimuli, which seems unli-

308

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

kely. Extensive experience in computational linguistics
conﬁrms that local-context models empirically outperform
syntax-based models when it comes to achieving high linguistic accuracy in unrestricted-domain texts like ours.
This suggests that while distant context can have large effects on the predictability of some words, in practice it usually does not; on average, local context is the most reliablecue to word predictability. Our estimates, therefore, whilesometimes noisy, should serve as an accurate statisticalproxy overall. Most importantly, there is no clear reason
why this choice of language model would bias our results
regarding curve shape in any particular direction.
3.4. Curve estimation
We used mgcv v1.6–2 (Wood, 2004; Wood, 2006) to predict reading times using penalized cubic spline functions (20d.f.) of word log-probability. As controls, we entered a spline
function of position in text, a two-dimensional tensor splineinteraction between orthographic word length and log-frequency, and factors indicating participant identity and(eye-tracking only) whether the previous word had been ﬁxated. (This was motivated by preliminary analyses which revealed weak or non-existent interactions between
predictability and either frequency or word length, but a
substantial interaction between frequency and word length,
which is consistent with previous ﬁndings (Kliegl et al.,2006; Pollatsek, Juhasz, Reichle, Machacek, & Rayner,2008).) To capture spillover (Mitchell, 1984; Rayner,1998), log-frequency/word-length interaction and probability terms were included for each word in an M-word window up to and including the current word, where M waschosen empirically to capture the effect present in each dataset (eye-tracking: M = 2; self-paced reading: M = 4). All
words were analyzed except those at the beginning or endof a line, or for which some word in the window did not appear in the British National Corpus, occurred adjacent topunctuation, or contained numerals. Eye-tracking analyses
excluded unﬁxated words; self-paced reading analyses excluded outliers (reading times <80 ms, >1500 ms, or >4 sd
aboveparticipant-speciﬁcmeans).
Eye-tracking:N = 166,522; self-paced reading: N = 51,552. Fitting was by(penalized) least squares; conﬁdence intervals were estimated by bootstrapping both participants and cases within
participants, using the mgcv ﬁtter’s weights parameter toavoid replicating data across folds in its internal cross-validation routine. (This method takes subject random effects
into account; for a discussion of item random effects in theseanalyses, see Appendix B.) All reported results were robust
to choice of spline basis, use of least squares estimation versus maximum likelihood estimation with the assumption ofheavy-tailed (gamma-distributed) error, and the use of larger spillover windows (increased M); see Appendix C for further validation of penalized spline regression in this setting.
4. Results
Fig. 3 shows how the probability of a word w affects thereading time for w and the words immediately succeedingit (the spillover region). Our two data-sets show marked

differences in time-course. For eye-tracking, the effect begins immediately and extends onto the next word, but is
not seen on words further downstream. For self-paced
reading, the effect does not begin until the succeeding
word, and lasts through the third succeeding word. Nonetheless, if we sum these curves to ﬁnd the total slowdown
due to a particular unpredictable word (Fig. 4), then weﬁnd nearly identical effect sizes. This suggests that thesetasks involve similar processing, though this processing isdifferently distributed through time with respect to saccades and button-presses respectively.
Crucially, Fig. 4 shows clearly that the relationship between word predictability and reading time is, in fact, logarithmic across at least six orders of magnitude inprobability. (Lower probability items occur, but not often
enough to reliably estimate curve shape without a larger
data set; see online supplementary information for graphswith the full x-axis.)
Fig. 4 contains little visual evidence for super-logarithmicity. To check this more formally, we re-ran theabove model ﬁts, but now entering linear and quadratic
functions of log-probability instead of an arbitrary spline(but keeping the same controls). A positive b coefﬁcienton the quadratic term would indicate a super-logarithmic
curve. We found no support for any quadratic component,
positive or otherwise (eye-tracking: total b = 0.05, 95%
CI = (0.45, 0.37), one-tailed p = 0.59; self-paced reading:total b = 0.04, 95% CI = (0.90, 1.07), one-tailed p = 0.46;
statistics via bootstrap).
For the Dundee corpus, there were sufﬁcient data to ﬁtparticipant-speciﬁc models; results from these analyses
are shown in Fig. 5. Nine out of ten participants showed
clear effects of log-probability, all of which are overall linear in shape. The individual-participant data for the Brown
self-paced reading corpus were not plentiful enough to
conduct participant-speciﬁc analyses.

5. Discussion
The predictability effect on word comprehension incontext takes a regular logarithmic form over at least sixorders of magnitude in estimated predictability. This ﬁnding has both practical and theoretical consequences.
Practically speaking, predictability is potentially affected by nearly any manipulation one can make to linguistic structure. It is therefore a potential confound in mostpsycholinguistic studies, and knowing the quantitative
form of this confound allows us to better control it. This
non-linearity is very severe—Fig. 6. When word predictability is included as a covariate in regression analyses itshould be log transformed; in factorial designs where average predictability is matched between conditions, it should
be log-predictability rather than raw predictability that ismatched. Since the uncertainty in the estimate of a word’s
log-predictability for any given context will grow as theword’s predictability decreases, this also implies that inpractice it is very difﬁcult to assert with conﬁdence fromcloze norms that two different sets of word/context pairs
are truly ‘‘equally’’ unpredictable in the sense that mattersfor real-time comprehension behavior. For example, a309

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319
Effect of P(wordn |context) on first−pass gaze duration measured at...

(a)wordn + 1

30
20
10
0

Amount of slowdown (ms)

40

wordn

10−6 10−5 10−4 10−3 10−2 10−1

10−6 10−5 10−4 10−3 10−2 10−1 1P(wordn |context)
Effect of P(wordn |context) on reading time measured at...

(b)wordn + 1

wordn + 2

word n + 3

30
20
10
0

Amount of slowdown (ms)

40

wordn

10−6 10−5 10−4 10−310 −2 10−1

10−6 10−5 10−4 10−3 10−2 10−1 10−6 10−5 10−4 10−3 10−2 10−1 10−6 10−5 10−4 10−3 10−2 10−1 1P(wordn |context)
Fig. 3. The effect of the probability of word n on reading time measured at word n and on successive words (the spill-over region). Curves are penalized
splines with point-wise 95% conﬁdence intervals. To correct for inter-subject variability, we measure the effect of probability against the notional baselineof a perfectly predictable word; zero on this graph does not indicate an instantaneous overall reading time. Conﬁdence intervals do not include theuncertainty induced by measurement error in probability estimation. Lower panels show the proportion of data available at each level of probability. (a)First-pass gaze durations. (b) Self-paced reading times.

word whose true probability is 102 will act more like aword whose true probability is 1 than like one whose trueprobability is 106—yet these will most likely be measured
as having 0%, 100%, and 0% cloze, respectively. Our results
suggest that there is no such thing as an unexpected word;
there are only words which are more or less expected.
5.1. Anticipation versus integration
Our results bear on the theoretical debate about
whether predictability effects in general arise from anticipatory pre-activation of speciﬁc words, or from post hoc
effects that arise while integrating the word into somekind of larger semantic context. The integration difﬁculty
account (Brown & Hagoort, 1993; Foss, 1982; Hagoort,
Baggio, & Willems, 2009; Hess, Foss, & Carroll, 1995; Traxler & Foss, 2000) holds that predictability itself does not affect comprehension difﬁculty, but rather that words whichhave high predictability scores are also those which aresomehow more related to the prior context, and words
which are more related to the prior context are also easier
to integrate semantically. For example, processing theword play in Examples (1) and (2) presumably requires
us to construct some representation of two different scenarios: one involving my brother playing inside, and another involving children playing outside. If the latter
scenario is easier to construct, then we expect play to beread more quickly in (2) than in (1). Crucially, under thisaccount, predictability effects do not arise until after thecomprehension system encounters the actual word; there
may appear to be effects of predictability, but they do not
result from any cognitive process of prediction. On theother hand, the anticipatory processing account holds thatpredictability effects do arise from some kind of processing
which is predictive in the sense that it is dependent on theidentity of the upcoming word, but occurs before this word

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

020

Total amount of slowdown (ms)

40
20
0

Total amount of slowdown (ms)

40

60

80

(b)

60

(a)

80

310

10

−6

10

−5

10

−4

10

−3

10

−2

10

−1

1

P (word |context)

10

−6

10

−5

10

−4

10

−3

10

−2

10

−1

1

P (word |context)

Fig. 4. By summing the curves in Fig. 3, we can estimate the total slowdown caused by an unpredictable word, regardless of where in the spillover region
this slowdown occurs. (a) First-pass gaze durations. (b) Self-paced reading times. Lower panels show the proportion of data available at each level ofprobability.

identity is known (DeLong et al., 2005; Van Berkum et al.,2005). Our results provide challenges for both of theseaccounts.
It seems plausible that predictability will, in general, be
correlated with semantic integration difﬁculty, so perhaps
the apparent effects of predictability in empirical studies
are actually a result of this confounding. But, is this correlation tight enough to explain our results? Intuitively, we
expect these measures to be similar in some cases, but todiverge in others. For instance, producers avoid saying
things which are too obvious from context, and so statements of obvious facts presumably have a simultaneously
low integration difﬁculty and a low predictability; similarly, syntactic alternatives with similar semantic content
presumably produce similar degrees of integration difﬁculty, but may have wildly different predictabilities. Our
results do not rule out an integration difﬁculty account,but given the precise and law-like relationship we found,the challenge for such accounts becomes to explain why
integration difﬁculty should vary in a quantitatively exact
way with the logarithm of predictability.
The anticipatory processing account avoids this difﬁculty, because it is obvious why predictive processing
would be sensitive to predictability per se; if you want tostart processing words in some manner before you actuallyencounter them, then a word’s probability of occurrence
given the available information, P(wjC), may be a useful
guide to decide which words should receive such processing, and to what degree. (Compare this to the situation
after you have encountered the word, at which point it
would seem mostly irrelevant how predictable it used tobe when you had less information available.) And it hasother appealing properties. It is independently motivated:
there is ample independent evidence that the comprehension system anticipates upcoming material in at least somesituations (Altmann & Kamide, 1999; DeLong et al., 2005;
Kamide et al., 2003; Kamide, Scheepers, & Altmann,
2003; Knoeferle et al., 2005; Van Berkum et al., 2005; Wicha et al., 2003). And, it provides an obvious reason why
predictability differences would produce differences inreading time (as higher predictability words will receive
more anticipatory processing, and thus require less post
hoc processing).
However, the most straightforward instantiation of the
anticipatory processing idea is the ‘simple guessing’ model
we formalized above, which predicted a linear effect ofpredictability. Our results clearly rule this out. More generally, and for the same reason, these results are incompatible with any theory which assumes both that (a)predictability effects on reading time arise from processing
which precedes the actual appearance of the word, and (b)the comprehension system can only apply this processing
to a small number of words at any given moment (relative
to the size of the lexicon). When such a model encounters aword with probability <105, it will almost never haveformed any expectation regarding it—yet the observed effect is just as strong in this region of word log-probability
as anywhere else. The reading time difference between
words with probability 106 and words with probability
105 is just as large as the difference between words withprobability 102 and those with probability 101. Thus, wemust reject either (a) or (b). Integration difﬁculty accounts

311

sb

sc

sd

se

sf

sg

sh

si

sj

30

60

sa

0

Total amount of slowdown (ms)N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

10

−4

10

−2

P(word |context)

Fig. 5. To visualize inter-individual variation, we break down the Dundee corpus summed slowdown data (Fig. 4a), analyzing each participant separately.
Participant codes from the corpus are shown in the upper right of each panel. Dashed lines represent bootstrapped point-wise 95% conﬁdence intervals. The
variation in ‘wiggliness’ of the main curves results in part from noise and numerical instability in mgcv’s GCV-based penalization selection (Wood, 2004)
allowing over/under-ﬁtting in some cases. Even so, 9 out of 10 participants show effects of log-probability with an overall linear trend, while no effect was
found for participant ‘sg’.

reject (a). If we wish to preserve an anticipatory processing
account of these data, we must instead reject (b), and build
theories in which expectancies do not take the form of simple guesses; instead, the comprehension system must beable to simultaneously pre-activate large portions of its
lexicon in a quantitatively graded fashion (Smith & Levy,
2008).
Yet another possibility would be for anticipatory processing to be directed not at words, but at word fragments,
which would make this account consistent with the incremental processing theory we propose here (which takes as
granted that there is some mechanism linking predictability and processing time, and focuses on explaining theresulting curve shape), while potentially reducing the degree to which parallel pre-activation is necessary (as there
are e.g. far fewer potential upcoming phonemes than there
are potential upcoming words).

gest that overall audience interpretation time is entirely
unaffected by the uniformity or non-uniformity of information density. However, this need not be taken to ruleout the possibility that other forms of audience design
might motivate a UID principle. For example, if speech isconsistently produced more quickly than it can be comprehended then it will eventually become incomprehensible,
which gives producers an incentive to slow down on difﬁcult content and let comprehenders catch up. In the case ofpredictability-related difﬁculty, producers who follow thisstrategy will end up following the UID principle, though
under this revised account more local variation in information density would be acceptable. The original theory predicts that information density should be optimized on thetime scale of individual processing fragments; here, what
would matter is uniformity on a time-scale only ﬁnegrained enough to avoid overloading comprehenders’working memory.

5.2. Consequences for UID
Our results lend no support to the audience-design account of uniform information density effects proposed by
Levy and Jaeger (2007), which required a super-linear relationship between log probability (surprisal) and processing
difﬁculty. We ﬁnd no evidence for deviation from a purelogarithmic curve, which under their analysis would sug-

5.3. The Bayesian Reader versus the incremental processing
account
There are two theories which predict the precise logarithmic effect we found: The Bayesian Reader (Norris,
2006; Norris, 2009) and the incremental processing account. Both ﬁnd support in Fig. 4, but they make different

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

80

(b)

60
40
0

20

Total amount of slowdown (ms)

40
20
0

Total amount of slowdown (ms)

60

(a)

80

312

0

0.5

P(word |context)

1

0

0.5

1

P(word |context)

Fig. 6. The same curves shown in Fig. 4, but here plotted against raw predictability to better show the severity of the non-linearity. (a) First-pass gaze
durations. (b) Self-paced reading times. Lower panels show the proportion of data available at each level of probability. (While these accurately indicate thatthe majority of our data is concentrated in the <0.1 range, the scale here is somewhat misleading; both analyses contain >10,000 data points withconditional probability >0.1.)

predictions about the time-course of these effects, Fig. 3. In
the Bayesian Reader model as originally formulated, predictability affects how much visual information the eyeneeds to gather from each word. This makes a clear prediction in the case of self-paced reading, where only one word
is displayed at a time: Since you cannot gather perceptual
input from a word that is no longer visible, the model expects a word’s predictability to affect viewing time for thatword only, with no spillover effect. Fig. 3b, however, showsthe exact opposite pattern: There is little or no effect on theword itself, with a large spillover effect. We can perhaps
overcome this difﬁculty at the cost of some theoretical elegance, by postulating that the noise bottleneck occurs notin visual perception per se, but at some later moment
where word identity must be communicated between
two internal processing stages connected by a noisy channel. Further analysis would be needed to determine
whether such a mechanism could produce slowdowns distributed over such a wide temporal span (2–3 words).
The incremental processing account, by contrast, isbased on the assumption that predictability affects not
perception, but the speed of cognitive processing generally.
Therefore, under this account we expect to see predictability effects at every moment that lexically associated processing occurs. Since spillover effects occur consistently
in the psycholinguistic literature, this model makes the
opposite prediction from the Bayesian Reader—that predictability effects should not be restricted to the period
when the word is actually visible, which is what we ﬁnd.
While this prediction is not particularly surprising, it does
make this the only extant model which can directly explain

our full pattern of results. And this model, if correct, raises
a number of new questions. Most obviously, what is theform of the true underlying function f(x) relating predictability and processing time? Is it an arbitrary and idiosyncratic function that, say, varies between individuals, or isthere some regularity to it, and if so, what? Answering thiswould require some other methodology, as per-word reading times are too coarse-grained a measurement to yieldmuch insight. Of even greater theoretical interest is thequestion of the value of k, the grain-size of incremental
processing; larger k would correspond to the processor
operating on ﬁner-grained or perhaps even truly continuous chunks of input (McClelland & Elman, 1986; Spivey,2007). Although these new results do not give direct
knowledge of k, consider that most possible functions are
not logarithms, and that Fig. 2 indicates that for some possible functions a rather large k (10) is required to produce a near-logarithmic curve shape like the ones weobserve. Such a high degree of incrementality goes beyond
what has already been established in the visual world paradigm (Tanenhaus et al., 1995) for the incremental processing of the speech signal, and certainly beyond what
has otherwise been observed in reading. These results
and model together, then, may provide an initial, tantalizing glimpse of a more ﬁne-grained linguistic processor
than has so far been exposed to experimental view. Other
methods which allow more detailed measures of thetime-course of processing, such as EEG/MEG, mouse-tracking (Spivey, Grosjean, & Knoblich, 2005), or hazard function analysis of eye movements (Feng, 2009) may yieldfurther insights in this regard.

313

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

5.4. Surprisal as a causal bottleneck
Finally, these results conﬁrm that it is plausible that all
reading time predictability effects are mediated by lexical
predictability, in accordance with the causal bottleneck
hypothesis of surprisal theory. Since the seminal work of
Shannon on quantifying the bit rate of English (Shannon,
1951), information-theoretically informed work on language has recognized that all types of hierarchical predictive information present in language—syntactic, semantic,
pragmatic, and so forth—must inevitably bottom out in
predictions about what speciﬁc word will occur in a given
context, and that when measured in bits, expectations at
each successive level combine naturally in a simple additive fashion. This is illustrated by our example from theintroduction, where the total bits carried by the word
who is the sum of the bits associated with the fact of a relative clause’s appearance in context C and the bits associated with the fact that the particular word introducing
the relative clause is who:

log PðwhojCÞ ¼ log Pðrel: clausejCÞ
þ log Pðwhojrel: clause; CÞOur present results reveal that the bit is also the correct
unit for measuring the processing time needed in generalduring incremental language comprehension by a native
speaker; a logarithmic effect of lexical predictability both
implies and subsumes logarithmic effects of transitional
probability, syntactic predictability, semantic predictability, etc., allowing us to explain these apparently disparate
effects as arising via a single uniﬁed mechanism. With contemporary probabilistic models of language structure wecan measure the bits carried by a wide variety of abstract
linguistic structures; the way is thus paved for their contributions to the time required for incremental language
comprehension to be investigated and quantiﬁed using
this common currency.

then as k goes to inﬁnity, the total processing time will become a closer and closer approximation to a logarithm ofpword. Here we formalize and prove this claim.
First, we need a way to talk about the pi values as k increases. Let’s assume we have a sequence of increasingly
long vectors (p1, . . . , pk)k, for k = 1, 2, . . .. We’ll write theith entry in the kth vector as pi,k, with 1 6 i 6 k. As per
above, we require that for all k, p1,k    pk,k = pword. We
do not, though, require that there be any necessary relationship between pi,k and pi+1,k; our result holds even if
each new vector is totally different than the one before.
What we do need is a condition to avoid degenerate behavior. What we want to avoid is a sequence like this one,
where all the uncertainty remains restricted to a single
fragment:

pi;k ¼

This work was supported in part by NSF Grant 0953870
and NIH Grant HD065829 to R.L., NIH Grant T32-DC000041
to the Center for Research in Language, NIH Grant T32MH20002 to the Institute for Neural Computation, funding
from the Army Research Laboratories Cognition & Neuroergonomics Collaborative Technology Alliance, and the European Commission through the EU Cognitive Systems
Project Xperience (FP7-ICT-270273). We thank: H. Lu and
N. Katz for assistance in collecting self-paced reading data,
and G. Altmann, V. Ferreira, T.F. Jaeger, R. Kliegl, M. Kutas,
E.H. Levy, E. Mukamel, K. Rayner, E.D. Reichle, and threeanonymous reviewers for comments on the manuscript.
Appendix A
In the main body, we claimed that if these two equalities hold:pword ¼ p1      pk
total processing time ¼ f ðp1 Þ þ    þ f ðpk Þ

pword

i¼1

1

i–1

Therefore, we require that as k increases, the probability ofthe most-surprising fragment should go to 1:

lim min pi;k ¼ 1

k!1

iThis forces the word’s overall uncertainty to be distributed
over an increasing number of fragments.
Next we consider f(x). For convenience, we assume thatf(1) = 0. If it does not, then we can just subtract off a constant c, and this would not alter the shape of f(x). This will
also affect the total processing time graph by shifting it up
or down by k  c units, but this only affects the baseline
reading time, not the shape of the curve. To avoid degenerate behavior, we also assume that the derivative of f(x) is
deﬁned at x = 1. This derivative will play an important rolein our story, so we give it a name: f0 (1) = h.
Now given these conditions, we wish to show that

lim

k!1

Acknowledgments



!
k
X
f ðpi;k Þ ¼ h log pword
i¼1

It will be easier to work with this after a change of vari~word ¼ log pword ; p~i;k ¼ log pi;k , and gð~
ables. Let p
xÞ ¼ f ðe~x Þ.
Plugging these transformations into our assumptions
above, we have as given that~word ¼
p

k
X
~i;k
p
i¼1

~i;k Þ ¼ 0
lim ðminp

k!1i

gð0Þ ¼ f ð1Þ ¼ 0
g 0 ð0Þ ¼ f 0 ð1Þ  e0 ¼ f 0 ð1Þ ¼ hand wish to show thatk
X~i;k Þ
lim
gðp

k!1

!
~word :
¼ h  p

i¼1

P~word , our result would hold if gð~~i;k ¼ p
xÞNote that sincepwere the linear function h  ~x. The idea of our proof is to
use the derivative to approximate gð~xÞ as a linear function
near 0. This turns out to be enough since as k increases, the~i;k values approach 0.
p314

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

More formally, recall from the deﬁnition of the derivative thath ¼ g 0 ð0Þ ¼ lim
h!0

Now, for k sufﬁciently large, we are guaranteed thatpi,k 2 (  d, 0]. And so for such a k, we can infer thatPk~i¼1 gðpi;k Þ is bounded bygð0 þ hÞ  gð0ÞgðhÞ
¼ lim
h!0hhðh  Þ

Therefore, for any  > 0 there exists a d > 0 such that for allh 2 (d, 0),kX~i;k ¼ ðh  Þ  p
~word
p
i¼1

~word Þ  ð  p~word Þ:¼ ðh  p



gðhÞ


<

ðhÞ
 h


That is, we can always pick k so as to guarantee thatPk~~
i¼1 gðpi;k Þ is as close as we like to h  pword , which iswhat we wanted to prove.

which implies thatgðhÞ ðhÞ < 
h
gðhÞ
h<
< h þ hðh  Þh < gðhÞ < ðh þ Þh<

Appendix B

That is, for h 2 (d, 0), g(h) is approximately linear withslope h ± .

A discussion of item effects in our analyses: In nullhypothesis signiﬁcance testing within psycholinguistics,
it is widely recognized that it is essential to take into account idiosyncratic differences among individuals andexperimental items relevant to the dependent measure—(a) Effect of P(word |context) on first−pass gaze duration measured at...
n

10

20

30

wordn + 1

0

Amount of slowdown (ms)

40

wordn

−6

10

−4

10

−3

10

−2

10

−1

10

P (wordn |context)

−6

−5

−4

−3

−2

−1

10 10 10 10 10 101

Effect of P(wordn |context) on reading time measured at...
wordn

wordn + 1

word n + 2

word n + 3

0Amount of slowdown (ms)
10
20
30

40

(b)

−5

10

10−6 10−5 10−4 10−3 10−2 10−1 10−6 10−5 10−4 10−3 10−2 10−1 10−6 10−5 10−4 10−3 10−2 10−1 10−6 10−5 10−4 10−3 10−2 10−1 1

P ( wordn | context)
Fig. B1. ‘‘By item’’ analysis of per-token mean reading times aggregated across participants, showing the effect of the predictability of wordn on wordn andsucceeding words. 95% conﬁdence intervals calculated by bootstrapping over cases. (a) Eye-tracking. (b) Self-paced reading. Lower panels show the
proportion of data available at each level of probability.

315

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

both overall proclivities and sensitivities to psychologicalvariables—because they break the conditional independence assumptions implicit in non-hierarchical (‘‘ﬂat’’)regression models. This issue is what motivates the useof by-participant and by-item ‘‘random effects’’ in repeated-measures ANOVA and mixed-effects regression
models (Baayen, Davidson, & Bates, 2008; Clark, 1973;
see also Barr, Levy, Scheepers, & Tily (2013) for speciﬁc discussion on this issue). The analyses above use a hierarchical bootstrap procedure (ﬁrst on participant clusters, thenon observations within each participant) that takes into account the by-subjects clustering structure in our data, but
it does not take into account the clustering structure deriving from the fact that our data involve multiple measurements taken from a given [word, context] pairing—what
would be called item random effects. Upon this view, thetheoretically critical effect of word probability is ‘‘between
items’’ rather than ‘‘within items’’, since a given[word, context] pair always has the same conditional probability in our language model, so that a reasonable way tomodel item random effects would be to assume that theunderlying hypothetical ‘‘average’’ reading time for a given[word, context] pair across the potential participant population is offset from that predicted by the other components by a factor bw drawn from some distribution withzero mean. This is known as an ITEM RANDOM INTERCEPT in themixed-effects models literature (Baayen et al., 2008). It is
worth carefully considering how the results we obtained
here might be affected by our omission of item random
intercepts if they are present with non-negligible variancein the underlying generative process:1. Omitting them could induce overconﬁdence in theparametric form of the probability/time relationship,
artiﬁcially narrowing our conﬁdence intervals.
2. Omitting them could induce the non-parametric modelto overﬁt arbitrary, small deviations from the true
underlying probability/reading-time function, since closely matching observed mean reading times for speciﬁc[word, context] pairs would produce spuriously highcross-validation scores and lead to the selection of too
small a penalization term.
The former could lead to unjustiﬁed overconﬁdence in
the inferred effect size and shape; the effect of the latter
would, if anything, make us less likely to obtain a cleanly
linear effect shape. Since we nevertheless did obtain anearly perfectly linear effect shape in both datasets, it isthe ﬁrst of these two possibilities, (1), that is of primaryconcern.
We ﬁrst demonstrate that when the effect of word
log-probability is assumed to be linear, its effect is highly
signiﬁcant even when crossed subject and item random
effects are taken into account. For both the Brown and
Dundee datasets we ﬁt parametric linear mixed-effects
models (Baayen et al., 2008) to reading times. As ﬁxed
effects, we entered all predictors used in the main analysis
except for participant identity, and with linear effects
substituting for all splines. Our random effects structure
included (i) a random intercept for word token, and (ii)random subject slopes for all word probability measuresentered as ﬁxed effects, with all correlations allowed(a ‘‘maximal’’ random-effects structure in the sense of Barr

−5

10

−4

10

−3

10

−2

10

P (word |context)

−1

10

1

20

40

60
−6

10

0

Total amount of slowdown (ms)

40
20
0

Total amount of slowdown (ms)

60

80

(b)
80

(a)

−6

10

−5

10

−4

10

−3

10

−2

10

−1

10

1

P (word |context)

Fig. B2. ‘‘By item’’ analysis of per-token mean reading times aggregated across participants, showing the total reading time slowdown attributable to word
predictability. (The sum of the curves in Fig. B1). 95% conﬁdence intervals calculated by bootstrapping over cases. (a) Eye-tracking. (b) Self-paced reading.
Lower panels show the proportion of data available at each level of probability.

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

1

10−6 10−5 10−4 10−3 10−2 10−1

180

Total amount of slowdown (ms)

−20

40
20
0
10−6 10−5 10−4 10−3 10−2 10−1

1

10−6 10−5 10−4 10−3 10−2 10−1

1

P (word |context)

Total amount of slowdown (ms)

80

80
60
40
20
0
−20

Total amount of slowdown (ms)P (word |context)

60

(c)

P (word |context)

60

80
60
40
20
0
10−6 10−5 10−4 10−3 10−2 10−1

P (word |context)

80
60
40
20
0
−20

Total amount of slowdown (ms)Penalized spline regression as implemented by mgcv
(Wood, 2004, 2006) is a powerful and principled techniquefor estimating unknown non-linear relations. In order to ﬁtnearly-arbitrary smooth curves, it uses a high-dimensional
spline basis; in order to avoid the over-ﬁtting that otherwise plagues such high-dimensional models, it combines
the standard maximum likelihood criterion with a curvature penalty term that biases the regression towards less‘wiggly’ curves. Critically, the relative weight placed on
the likelihood term (which attempts to follow the data)
versus the penalty term (which attempts to make the line
smoother and closer to a straight line) is determined bycross-validation. In theory, therefore, this method’s ﬁtted

401

Appendix C

20

10−6 10−5 10−4 10−3 10−2 10−1

the eye-tracking effect. If true, this may be an artifact of
our excluding unﬁxated words from the eye-tracking
analysis.

0

1

P (word |context)

−20

40
20
0
10−6 10−5 10−4 10−3 10−2 10−1(b)Total amount of slowdown (ms)80

Total amount of slowdown (ms)

60

(a)

−20

et al., 2013). Models were ﬁt with (unrestricted) maximum
likelihood estimation using the lme4 package (Bates,
Maechler, & Bolker, 2011). Consistent with the ﬁndings ofour non-parametric analyses, these analyses found thatthe linear effects of word log-probability were highly
signiﬁcant in all cases except for that of current-word
log-probability in the Brown self-paced reading dataset(Dundee: current-word jtj = 2.88, one-back jtj = 5.59;
Brown: current-word jtj = 1.35, one-back jtj = 4.83,
two-back jtj = 3.68, three-back jtj = 3.13).
In addition, we conducted a ‘‘by-items’’ analysis of eachdataset, computing mean reading time for each word token(aggregating across subjects) and then ﬁtting our nonparametric model to each dataset. (For the eye-tracking
dataset this meant discarding the predictor of whether
the previous word was ﬁxated.) Results are shown in
Fig. B1 and B2; once again we recovered effects on reading
time that were linear in word log-probability. The maindifference from our earlier results is that in Fig. B2, theself-paced reading effect appears somewhat stronger than

−20

316

P (word |context)

Fig. C1. The effect of penalization in controlling over-ﬁtting. (a) Our original, penalized model (a repeat of Fig. 4). (b) The same model as in a, but ﬁt withraw probability entered instead of log probability, then plotted in log-space. (c) The same model as in a, but ﬁt without penalization. Upper panels show
ﬁrst-pass gaze durations; lower panels show self-paced reading times. That the lower panels show more wiggliness than the upper ones is presumably due
to the relative sizes of the two data sets; in the absence of penalization, the smaller data set allows more overﬁtting than the larger. Dashed lines denote
point-wise 95% conﬁdence intervals. Lower panels show the proportion of data available at each level of probability.

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

curves should be biased towards smoothness only to theextent that this helps it better match the true curve
describing the underlying phenomenon.
But since our key empirical ﬁnding is that such a ﬁt produces a straight line, it seems prudent to verify that this is
not an artifact introduced by penalization. We therefore repeated our analysis, but using two different methods to remove this potential bias.
First, we ran the same model ﬁts, but entering raw
probability in place of log-probability; in this case we predict that the splines should attempt to form a steep logarithmic curve (since we believe that is the true
underlying relationship), while the penalization pushes towards a linear relationship (Fig. 1). As expected, mgcv’s
algorithm chose to apply very small penalization weights(ranging from 65 to 64,000 times smaller than the corresponding weights chosen in the original analyses), which
in turn allowed the resulting spline ﬁt to form an highlynonlinear, approximately logarithmic curve with substantial local variation around this underlying trend (Fig. C1b;note that while the ﬁt was performed using raw probability, we plot the result against log probability to facilitatecomparison with other ﬁts). Second, we ﬁt our originalmodel, but with penalization simply disabled (i.e., using
standard least-squares); this produced similar results(Fig. C1c). The three models thus agree that the underlying
relationship is approximately logarithmic.
Finally, we would like to conﬁrm that the local non-linear deviations from this trend that we see in models (b)and (c) are the result of over-ﬁtting rather than a true effect. We veriﬁed this by performing 1000-fold cross-validation on all three models, and found that in both data
sets, the original penalized model (Fig. C1a) achieved thehighest log-likelihood on held-out data. (Similar results,
not shown, were obtained when performing cross-validation of the penalized model versus the other models on
the ‘‘by item’’ data set described above.) Thus we conclude
that, to the limits of our data, the underlying relationshipbetween word probability and processing time is in fact
logarithmic.

Appendix D. Supplementary material
Supplementary data associated with this article can befound, in the online version, at http://dx.doi.org/10.1016/
j.cognition.2013.02.013.

References
Adelman, J. S., Brown, G. D. A., & Quesada, J. F. (2006). Contextual
diversity, not word frequency, determines word-naming and lexicaldecision times. Psychological Science, 17(9), 814–823. http://dx.doi.org/10.1111/j.1467-9280.2006.01787.x.
Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation atverbs: Restricting the domain of subsequent reference. Cognition,
73(3), 247–264. http://dx.doi.org/10.1016/S0010-0277(99)00059-1.
Atkinson, K. (2004). The VARCON database, version 4.1. <http://
wordlist.sourceforge.net/>.
Aylett, M., & Turk, A. (2004). The smooth signal redundancy hypothesis: A
functional explanation for relationships between redundancy,
prosodic prominence, and duration in spontaneous speech.
Language & Speech, 47(1), 31–56.

317

Baayen, R. H. (2010a). Demythologizing the word frequency effect: A
discriminative learning perspective. The Mental Lexicon, 5(3),436–461. http://dx.doi.org/10.1075/ml.5.3.10baa.
Baayen, R. H. (2010b). The directed compound graph of English: An
exploration of lexical connectivity and its processing consequences.
In S. Olsen(Ed.). New impulses in word-formation (Vol. 17,
pp. 383–402). Hamburg: Buske.
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects
modeling with crossed random effects for subjects and items. Journalof Memory and Language, 59(4), 390–412. http://dx.doi.org/10.1016/
j.jml.2007.12.005.
Bar, M. (2009). Predictions: A universal principle in the operation of thehuman brain. Philosophical Transactions of the Royal Society B:
Biological Sciences, 364(1521), 1181–1182. http://dx.doi.org/10.1098/
rstb.2008.0321.
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects
structure for conﬁrmatory hypothesis testing: Keep it maximal.
Journal of Memory and Language, 68(3), 255–278. http://dx.doi.org/
10.1016/j.jml.2012.11.001.
Bates, D., Maechler, M., & Bolker, B. (2011). lme4: Linear mixed-effects
models using S4 classes [Computer software]. (R package version
0.999375-42).
BNC Consortium. (2001). The British National Corpus (version 2, BNC World)[Text corpus]. <http://www.natcorp.ox.ac.uk/>.
Boston, M. F., Hale, J., Kliegl, R., Patil, U., & Vasishth, S. (2008). Parsing
costs as predictors of reading difﬁculty: An evaluation using the
Potsdam Sentence Corpus. Journal of Eye Movement Research, 2, 1–12.
Brown, C., & Hagoort, P. (1993). The processing nature of the N400:
Evidence from masked priming. Journal of Cognitive Neuroscience, 5(1),34–44.
Carpenter, R. H. S., & Williams, M. L. L. (1995). Neural computation of log
likelihood in control of saccadic eye movements. Nature, 377(6544),
59–62. http://dx.doi.org/10.1038/377059a0.
Chen, S. F., & Goodman, J. (1998). An empirical study of smoothing
techniques for language modeling (Technical Report No. TR-10-98).
Cambridge, MA: Computer Science Group, Harvard University. <ftp://
ftp.deas.harvard.edu/techreports/tr-10-98.ps.gz>.
Clark, H. H. (1973). The language-as-ﬁxed-effect fallacy: A critique of
language statistics in psychological research. Journal of Verbal
Learning and Verbal Behavior, 12, 335–359.
Dahan, D., & Tanenhaus, M. K. (2004). Continuous mapping from sound tomeaning in spoken-language comprehension: Immediate effects ofverb-based thematic constraints. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 30(2), 498–513.
DeLong, K. A., Urbach, T. P., & Kutas, M. (2005). Probabilistic word preactivation during language comprehension inferred from electricalbrain activity. Nature Neuroscience, 8, 1117–1121.
Demberg, V., & Keller, F. (2008). Data from eye-tracking corpora as
evidence for theories of syntactic processing complexity. Cognition,
109(2), 193–210.
Ehrlich, S. F., & Rayner, K. (1981). Contextual effects on word perception
and eye movements during reading. Journal of Verbal Learning and
Verbal Behavior, 20(6), 641–655.
Engbert, R., Nuthmann, A., Richter, E. M., & Kliegl, R. (2005). SWIFT: A
dynamical model of saccade generation in reading. Psychological
Review, 112(4), 777–813.
Federmeier, K. D., & Kutas, M. (1999). A rose by any other name: Longterm memory structure and sentence processing. Journal of Memory
and Language, 41, 469–495.
Feng, G. (2009). Time course and hazard function: A distributional
analysis of ﬁxation duration in reading. Journal of Eye Movement
Research, 3(2), 1–22.
Ferreira, F., & Clifton, C. Jr., (1986). The independence of syntactic
processing. Journal of Memory and Language, 25, 348–368.
Forster, K. (1976). Accessing the mental lexicon. In R. J. Wales & E. C. T.
Walker (Eds.), New approaches to language mechanisms (pp. 257–287).
Amsterdam: North-Holland.
Foss, D. J. (1982). A discourse on semantic priming. Cognitive Psychology,
14(4), 590–607. http://dx.doi.org/10.1016/0010-0285(82)90020-2.
Fossum, V., & Levy, R. (2012). Sequential vs. hierarchical syntactic models
of human incremental sentence processing. In Proceedings of the 3rdworkshop on cognitive modeling and computational linguistics (CMCL2012) (pp. 61–69). Montrèal, Canada: Association for Computational
Linguistics. <http://www.aclweb.org/anthology/W12-1706>.
Frank, S. L., & Bod, R. (2011). Insensitivity of the human sentenceprocessing system to hierarchical structure. Psychological Science,22(6), 829–834. http://dx.doi.org/10.1177/0956797611409589.
Frisson, S., Rayner, K., & Pickering, M. J. (2005). Effects of contextual
predictability and transitional probability on eye movements during

318

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319

reading. Journal of Experimental Psychology: Learning, Memory, andCognition, 31(5), 862–877.
Froehlich, A. L., Herbranson, W. T., Loper, J. D., Wood, D. M., & Shimp, C. P.
(2004). Anticipating by pigeons depends on local statisticalinformation in a serial response time task. Journal of Experimental
Psychology: General, 133(1), 31–45.
Frost, R. (1998). Toward a strong phonological theory of visual word
recognition: True issues and false trails. Psychological Bulletin, 123(1),71–99 (WOS:000071350100006, doi:10.1037//0033-2909.123.1.71).
Genzel, D., & Charniak, E. (2002). Entropy rate constancy in text. In
Proceedings of ACL.
Gold, J. I., & Shadlen, M. N. (2007). The neural basis of decision making.
Annual Reviews of Neuroscience, 30(1), 535–574. http://dx.doi.org/
10.1146/annurev.neuro.29.051605.113038.
Goodman, J. T. (2001). A bit of progress in language modeling, extended
version (Technical Report No. MSR-TR-2001-72). Redmond, WA:
Microsoft Research.
Hagoort, P., Baggio, G., & Willems, R. M. (2009). Semantic uniﬁcation. In
M. S. Gazzaniga (Ed.). The new cognitive neurosciences (4th ed.) (Vol.
4., pp. 819–836). Boston, MA: MIT Press.
Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model.
In Proceedings of NAACL-2001 (pp. 159–166). Stroudsburg, PA:
Association for Computational Linguistics.
Hess, D. J., Foss, D. J., & Carroll, P. (1995). Effects of global and local context
on lexical processing during language comprehension. Journal of
Experimental Psychology: General, 124(1), 62–82. http://dx.doi.org/
10.1037/0096-3445.124.1.62.
Howes, D. H., & Solomon, R. L. (1951). Visual duration threshold as afunction of word-probability. Journal of Experimental Psychology, 41,
401–410.
Ilkin, Z., & Sturt, P. (2011). Active prediction of syntactic information
during sentence processing. Dialogue & Discourse, 2(1), 35–58.
Jaeger, T. F. (2010). Redundancy and reduction: Speakers manage
syntactic information density. Cognitive Psychology, 61(1), 23–62.
http://dx.doi.org/10.1016/j.cogpsych.2010.02.002.
Janssen, P., & Shadlen, M. N. (2005). A representation of the hazard rate ofelapsed time in macaque area LIP. Nature Neuroscience, 8(2), 234–241.
http://dx.doi.org/10.1038/nn1386.
Just, M. A., Carpenter, P. A., & Woolley, J. D. (1982). Paradigms andprocesses in reading comprehension. Journal of Experimental
Psychology: General, 111(2), 228–238.
Kamide, Y., Altmann, G. T. M., & Haywood, S. L. (2003). The time-course ofprediction in incremental sentence processing: Evidence fromanticipatory eye movements. Journal of Memory and Language, 49(1),133–156. http://dx.doi.org/10.1016/S0749-596X(03)00023-8.
Kamide, Y., Scheepers, C., & Altmann, G. T. M. (2003). Integration ofsyntactic and semantic information in predictive processing: Crosslinguistic evidence from German and English. Journal of
Psycholinguistic Research, 32(1), 37–55. http://dx.doi.org/10.1023/
A:1021933015362.
Kennedy, A., Hill, R., & Pynte, J. (2003). The Dundee corpus. In Proceedingsof the 12th European conference on eye movement.
Kennedy, A., Pynte, J., Murray, W. S., & Paul, S.-A. (in press). Frequency andpredictability effects in the Dundee corpus: An eye movement
analysis. The Quarterly Journal of Experimental Psychology. http://dx.doi.org/10.1080/17470218.2012.676054.
Kliegl, R., Nuthmann, A., & Engbert, R. (2006). Tracking the mind during
reading: The inﬂuence of past, present, and future words on ﬁxation
durations. Journal of Experimental Psychology: General, 135(1), 12–35.
Kneser, R., & Ney, H. (1995). Improved backing-off for M-gram language
modeling. In Proc. ICASSP (pp. 181–184).
Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J. (2005). The
inﬂuence of the immediate visual context on incremental thematicrole-assignment: Evidence from eye-movements in depicted events.
Cognition,95(1),
95–127.
http://dx.doi.org/10.1016/
j.cognition.2004.03.002.
Kutas, M., & Federmeier, K. D. (2011). Thirty years and counting: Finding
meaning in the N400 component of the event-related brain potential(ERP). Annual Review of Psychology, 62(1), 621–647. http://dx.doi.org/
10.1146/annurev.psych.093008.131123.
Kutas, M., & Hillyard, S. A. (1984). Brain potentials reﬂect word
expectancy and semantic association during reading. Nature, 307,
161–163.
Laming, D. R. J. (1968). Information theory of choice-reaction times. London:
Academic Press.
Lau, E., Stroud, C., Plesch, S., & Phillips, C. (2006). The role of structural
prediction in rapid syntactic analysis. Brain & Language, 98, 74–88.
Levy, R. (2008). Expectation-based syntactic comprehension. Cognition,
106, 1126–1177.

Levy, R., Fedorenko, E., Breen, M., & Gibson, T. (2012). The processing ofextraposed structures in English. Cognition, 122(1), 12–36. http://dx.doi.org/10.1016/j.cognition.2011.07.012.
Levy, R., & Jaeger, T. F. (2007). Speakers optimize information density
through syntactic reduction. Advances in Neural Information Processing
Systems, 19, 849.
Manning, C. D., & Schütze, H. (1999). Foundations of statistical natural
language processing. MIT Press.
McClelland, J. L., & Elman, J. L. (1986). The TRACE model of speech
perception. Cognitive Psychology, 18(1), 1–86. http://dx.doi.org/
10.1016/0010-0285(86)90015-0.
McDonald, S. A., & Shillcock, R. C. (2003a). Eye movements reveal theonline computation of lexical probabilities during reading.
Psychological Science, 14(6), 648–652.
McDonald, S. A., & Shillcock, R. C. (2003b). Low-level predictive inference
in reading: The inﬂuence of transitional probabilities on eye
movements. Vision Research, 43, 1735–1751.
McRae, K., Spivey-Knowlton, M. J., & Tanenhaus, M. K. (1998). Modelingthe inﬂuence of thematic ﬁt (and other constraints) in on-line
sentence comprehension. Journal of Memory and Language, 38(3),283–312.
Mitchell, D. C. (1984). An evaluation of subject-paced reading tasks
and other methods for investigating immediate processes inreading. In D. E. Kieras & M. A. Just (Eds.), New methods inreading comprehension research. Hillsdale, New Jersey: Lawrence
Erlbaum Associates.
Morrison, C. M., & Ellis, A. W. (1995). Roles of word frequency and age ofacquisition in word naming and lexical decision. Journal of
Experimental Psychology: Learning, Memory, and Cognition, 21(1),116–133. http://dx.doi.org/10.1037/0278-7393.21.1.116.
Morrison, C. M., Hirsh, K. W., & Duggan, G. B. (2003). Age of acquisition,
ageing, and verb production: Normative and experimental data. The
Quarterly Journal of Experimental Psychology. A – Human Experimental
Psychology,
56(4),
705–730.
http://dx.doi.org/10.1080/
02724980244000594.
Morton, J. (1969). Interaction of information in word recognition.
Psychological Review, 76(2), 165–178.
Murray, W. S., & Forster, K. I. (2004). Serial mechanisms in lexical access:The rank hypothesis. Psychological Science, 111(3), 721–756.
Narayanan, S., & Jurafsky, D. (2004). A Bayesian model of human sentence
processing. <http://www.icsi.berkeley.edu/snarayan/newcog.pdf>.
Ni, W., Crain, S., & Shankweiler, D. (1996). Sidestepping garden paths:
Assessing the contributions of syntax, semantics and plausibility inresolving ambiguities. Language & Cognitive Processes, 11(3),283–334.
Norris, D. (2006). The Bayesian reader: Explaining word recognition as an
optimal Bayesian decision process. Psychological Review, 113(2),327–357.
Norris, D. (2009). Putting it all together: A uniﬁed account of word
recognition and reaction-time distributions. Psychological Review,
116(1), 207–219.
Pang, K., Merkel, F., Egeth, H., & Olton, D. S. (1992). Expectancy andstimulus frequency: A comparative analysis in rats and humans.
Perception & Psychophysics, 51(6), 607–615.
Piantadosi, S. T., Tily, H., & Gibson, E. (2011). Word lengths are optimized
for efﬁcient communication. Proceedings of the National Academy of
Sciences, 108(8), 3526–3529.
Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996).
Understanding normal and impaired word reading: Computational
principles in quasi-regular domains. Psychological Review, 103(1),56–115. http://dx.doi.org/10.1037/0033-295X.103.1.56.
Pollatsek, A., Juhasz, B. J., Reichle, E. D., Machacek, D., & Rayner, K. (2008).
Immediate and delayed effects of word frequency and word length oneye movements in reading: A reversed delayed effect of word length.
Journal of Experimental Psychology: Human Perception & Performance,
34(3), 726–750.
Rayner, K. (1998). Eye movements in reading and information processing:
20 years of research. Psychological Bulletin, 124(3), 372–422.
Rayner, K., & Well, A. D. (1996). Effects of contextual constraint on eye
movements in reading: A further examination. Psychonomic Bulletin &Review, 3(4), 504–509.
Reichle, E. D., Pollatsek, A., Fisher, D. L., & Rayner, K. (1998). Toward amodel of eye movement control in reading. Psychological Review,105(1), 125–157.
Roark, B., Bachrach, A., Cardenas, C., & Pallier, C. (2009). Deriving lexicaland syntactic expectation-based measures for psycholinguistic
modeling via incremental top-down parsing. In Proceedings of the2009 conference on empirical methods in natural language processing(pp. 324–333).

N.J. Smith, R. Levy / Cognition 128 (2013) 302–319
Seidenberg, M. S., & McClelland, J. L. (1989). A distributed, developmental
model of word recognition and naming. Psychological Review, 96(4),523–568.
Shannon, C. E. (1951). Prediction and entropy of printed English. Bell
Systems Technical Journal, 30, 50–64.
Smith, N. J., & Levy, R. (2008). Optimal processing times in reading: A
formal model and empirical investigation. In B. C. Love, K. McRae, & V.
M. Sloutsky (Eds.), Proceedings of the thirtieth annual conference of the
Cognitive Science Society (pp. 595–600). Austin, TX: Cognitive Science
Society.
Spivey, M. J. (2007). The continuity of mind. New York: Oxford University
Press.
Spivey, M. J., Grosjean, M., & Knoblich, G. (2005). Continuous attraction
towards phonological competitors. Proceedings of the National
Academy of Sciences, 102, 10393–10398.
Staub, A., & Clifton, C. (2006). Syntactic prediction in language
comprehension: Evidence from either . . . or. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 32(2), 425–436. http://dx.doi.org/10.1037/0278-7393.32.2.425.
Stolcke, A. (2002). SRILM—An extensible language modeling toolkit. In
Proc. intl. conf. on spoken language processing (Vol. 2, pp. 901–904).
Denver.
Stone, M. (1960). Models for choice-reaction time. Psychometrika, 25,
251–260.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C.
(1995). Integration of visual and linguistic information in spoken
language comprehension. Science, 268, 1632–1634.

319

Taylor, W. L. (1953). ‘‘Cloze procedure’’: A new tool for measuring
readability. Journalism Quarterly, 30, 415–433.
Traxler, M. J., & Foss, D. J. (2000). Effects of sentence constraint on priming
in natural language comprehension. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 26(5), 1266–1282.
http://dx.doi.org/10.1037/0278-7393.26.5.1266.
Van Berkum, J. J. A., Brown, C. M., Zwitserlood, P., Kooijman, V., &Hagoort, P. (2005). Anticipating upcoming words in discourse:
Evidence from ERPs and reading times. Learning and Memory,31(3), 443–467.
Wicha, N. Y. Y., Bates, E. A., Moreno, E. M., & Kutas, M. (2003). Potato not
pope: Human brain potentials to gender expectation and agreementin spanish spoken sentences. Neuroscience Letters, 346(3), 165–168.
http://dx.doi.org/10.1016/S0304-3940(03)00599-8.
Wood, S. N. (2004). Stable and efﬁcient multiple smoothing parameter
estimation for generalized additive models. Journal of the American
Statistical Association, 99, 673–686.
Wood, S. N. (2006). Generalized additive models: An introduction with R.
Boca Raton: Chapman and Hall.
Zevin, J. D., & Seidenberg, M. S. (2002). Age of acquisition effects in word
reading and other tasks. Journal of Memory and Language, 47(1), 1–29.
http://dx.doi.org/10.1006/jmla.2001.2834.
Zorzi, M., Houghton, G., & Butterworth, B. (1998). Two routes or one inreading aloud? A connectionist dual-process model. Journal of
Experimental Psychology: Human Perception and Performance, 24(4),1131–1161. http://dx.doi.org/10.1037/0096-1523.24.4.1131.