The neural dynamics of word recognition and integration
Authors: Jon Gauthier
Roger P. Levy
Venue: EMNLP 2023 Main (2023)
Keywords: processing
word recognition
speech comprehension
EEG
neuroscience
cognitive neuroscience
cognitive science
psychology
time series
TL;DR: We design multi-level models to study the process of online auditory word recognition and its manifestations in the EEG neural responses to words.

Abstract: Listeners recognize and integrate words in rapid and noisy everyday speech by combining expectations about upcoming content with incremental sensory evidence. We present a computational model of word recognition which formalizes this perceptual process in Bayesian decision theory. We fit this model to explain scalp EEG signals recorded as subjects passively listened to a fictional story, revealing both the dynamics of the online auditory word recognition process and the neural correlates of the recognition and integration of words.

The model reveals distinct neural processing of words depending on whether or not they can be quickly recognized. While all words trigger a neural response characteristic of probabilistic integration — voltage modulations predicted by a word's surprisal in context — these modulations are amplified for words which require more than roughly 150 ms of input to be recognized. We observe no difference in the latency of these neural responses according to words' recognition times. Our results support a two-part model of speech comprehension, combining an eager and rapid process of word recognition with a temporally independent process of word integration.
However, we also developed alternative models of the scalp EEG signal not incorporating word recognition dynamics which showed similar performance improvements. We discuss potential future modeling steps which may help to separate these hypotheses.

The neural dynamics of auditory word recognition and integration
Jon Gauthier and Roger Levy
Department of Brain and Cognitive Sciences Massachusetts Institute of Technology jon@gauthiers, net, rplevyemit.edi
Abstract
comprehension process is the N400 ERP, a centro- parietally distributed negative voltage modulation measured at the scalp by clectroencephaalogrmm (EEG) which peaks around 400 ms after the on- set of word. This negative component is am- plified for words which are semantically incom- patible with their sentence discourse context (Kutas and Hillyard, 1984; Brown and Hagoort, 1993; Kutas and Federmeier, 2011; Heilbron et al., 2022). This effect has been taken as evidence that comprehenders actively predict features up- coming words (DeLong et al., 2005; Kuperberg and Jaeger, 2016; Kuperberg et al., 2020). On one popular account, predictions about upcoming content are used to pre-activate linguistic repre- sentations likely to be used when that content ar- rives. The N400 reflects the integration of a rec- ognized word with its context, and this integration is facilitated just when the computational paths taken by the integration process align with those already pre-activated by the listener (Kutas and Federmeier, 2011; Federmeier, 2007).
Listeners recognize and integrate words in rapid and noisy everyday speech by combin- ing expectations about upcoming content with incremental sensory evidence. We present a computational model of word recognition which formalizes this perceptual process in Bayesian decision theory. We fit this model to explain scalp ERG signals recorded as subjects passively listened fictional story, revealing both the dynamics of the online auditory word recognition process the neural correlates of the recognition and integration of words.
The model reveals distinct neural processing words depending on whether or not they can be quickly recognized. While all words trig- ger a neural response characteristic of proba- bilistic integration - voltage modulations pre- dicted by word's surprisal in context these modulations are amplified for words which re- quire more than roughly 150 ms of input be recognized. We observe no difference in the latency of these neural responses according to words' recognition times. Our results are sistent with a two-part model of speech com- prehension, combining an eager and rapid pro- cess of word recognition with temporally in- dependent process of word integration. How- ever, we also developed alternative models of the scalp EEG signal not incorporating word recognition dynamics which showed similar performance improvements. We discuss po- tential future modeling steps which may help separate these hypotheses.
Despite the extensive research on the N400 and its computational interpretation, its relationship with the upstream process of word recognition is still not well understood. Some authors have ar- gued that integration processes should be tempo- rally yoked to word recognition: that is, compre- henders should continue gathering acoustic evi- dence as to the identity of a word until they are suf- ficiently confident to proceed with subsequent in- tegration processes "Marslen-Wilsom 1987). It is also possible, however, that integration processes are insensitive to the progress of word recognition: that integration is temporally regular semantic operation which begins regardless of the listener's confidence about the word being spoken (Hagoort, 2008; Federmeier and Laszlo, 2009).
Psycholinguistic studies at the neural and be- havioral levels have detailed how listeners ac- tively predict upcoming content at many levels of linguistic representation (Kuperberg and Jacger, 2016), and use these predictions to drive their behavior far before the relevant linguistic input is complete (Allopenna al., 1998). One well- studied neural correlate of this prediction-driven
Experimental studies have attempted to assess the link between these two processes, modeling the timing of word recognition through an offline
Code to reproduce our analyses is available at wtt.uu..co//aan aasaan///oooeee hwt.://ooorrrooiie--e

---

behavioral paradigm known as gating (Grosjean, 1980): by presenting incrementally longer clips of speech to subjects and asking them to predict what word is being spoken, authors estimate the time point at which there sufficient information to identify word from its acoustic form. Sev- eral EEG studies have asked whether the N400 re- sponse varies with respect to this estimate of word recognition time, but have arrived contradic- tory answers to this question (van den Brink al., 2006; O Rourke and Holcomb, 2002).
Meuning
Bounds
Recognition threshold (oq. 3)
(0, 1)
Evidence temperature (eq. 2)
[0, oc)
Scatter point (eq.
(0.
dp Prior seatter point (eq.
(0,1)
kg Word wy 's recognition point (eq. 3)
{0, [0,1.... 1....... ....).,//.l
Word toy 's's recognition time (eq. 4)
0,,x)
Table I: Cognitive model parameters and outputs.
the context, following a clock which is insensitive to the specific state of word recognition.
In this paper, we introduce a computational model which targets these dynamics of word recognition, and their manifestation in neural EEG signals recorded during naturalistic listening. The model allows us to connect trial-level variation in word recognition times to aspects of the neural re- sponse to words. We the model address two cross-cutting questions:
It is necessary to moderate these conclusions, however:
also develop alternative models of the neural correlates of word integration which im- prove beyond the performance of our baselines, without incorporating facts about the dynamics of word recognition. We discuss in Section 4 how more elaborate neural linking theories will be nec- essary to better separate these very different cog- nitive pictures of the process of word recognition and its neural correlates.
Onset:
Are words integrated only after they successfully recognized, or is the timing of integration insensitive to the state of word recognition?
Model
Response properties: Does the shape of the neural response words differ based on their recogmition times? so, this could indicate distinct inferential mechanisms deployed for words depending on their case recognition.
Our model consists of two interdependent parts: a cognitive model of the dynamics of word recogni- tion, and a neural model that estimates how these dynamics drive the EEG response to words.
1.1 Cognitive model
We jointly optimize the cognitive and neural parameters of this model explain EEG data recorded as subjects listened to naturalistic En- glish speech. Model comparison results suggest that semantic integration processes are not tem- porally yoked to the status of word recognition: the neural traces of word integration have just the same temporal structure, regardless of when words are successfully recognized. However, the neural correlates of word integration qualitatively differ based on the status of word recognition: words not yet recognized by the onset of word integration ex- hibit significantly different neural responses.
We first design a cognitive model of the dynamics of word recognition in context, capturing how a listener forms incremental beliefs about the word they are hearing wi as a function of the linguistic context C and some partial acoustic evidence Lgk- We formalize this as Bayesian posterior (Norris and McQueen, 2008):
P(w; C, & P(w; C)) P(Igk wx) (1)
which factorizes into a prior expectation of the word wi in context (first term) and likelihood of the partial evidence of phonemes Tsk (sec- ond term). This model thus asserts that the con- text C' and the acoustic input I are conditionally independent given w, We parameterize the prior P(w; C) P(w; wes) using left-to-right neural network language model. The likelihood is noisy-channel phoneme recognition model:
These results suggest a two-part model of word recognition and integration. First, the success of our word recognition model in predicting the neu- ral response to words suggests that there exists rapid lexical interpretation process which inte- grates prior expectations and acoustic evidence in order to pre-activate specific lexical items mem- ory. Second, an independent integration process composes these memory contents with a model of
P(TsH w.) x II P(L, wir)
(2)
15j5k

---

where per-phoneme confusion probabilities are drawn from prior phoneme recognition studies (Weber and Smits, 2003) and reweighted by a tem- perature parameter X.
Model name
Onset
Response properties
Baseline Shift
0
unitary linear response unitary linear response independent linear re- sponses for early, mid-, and late-recognized words independent linear responses for low-, mid-, and high- surprisal words
Fi
(cq. 4)
Variable
0
We evaluate this posterior for every word with each incremental phoneme, from k = 0 (no in- put) to A |ax]| (conditioning on all of the word's phonemes). We define hypothetical cognitive event of word recognition which is time-locked to the phoneme Kę where this posterior first exceeds a confidence threshold y:
Prior-variable
Table 2: Neural linking models with different commit- ments about the temporal onset of word features (rela- tive to word onset) and the flexibility of the parameters linking word features to neural response.
k min {ki P(w; C,Ick) } O5k5/all
(3)
We define word's recognition time T; to be frac- tion of the span of the K kr-ith phoneme. In the special case where k 0 and the word is confi- dently identified prior to acoustic input, we take T be fraction Op of its first phoneme's duration (visualized in Figure la):
rate, and their neural responses likely highly convolved as consequence (Crosse et al., 2016). We define a feature time series X, c Rdxx containing de features of the objective auditory stimulus, such as acoustic and spectral features, resampled to match the T samples of the neural time series. We also define word-level feature matrix X₂ E Rdwxn for the The words in the stimulus. Crucially, X, contains estimates of each word's s surprisal (negative (og-probability) in con- text. Prior studies suggest that surprisal indexes the peak amplitude of the naturalistic N400 (Frank et al., 2015; Gillis et al., 2021; Heilbron et al., 2022).
Fi (k;) + a (k;) Opp dur,
i i
(4)
where (ks; and dkr;(k:) are the onset time (relative word onset) and duration of the k-th phoneme word i, and a, Up are free parameters fitted jointly with the rest of the model.
1.2 Neural model
We assume that X, causes a neural response in- dependent of word recognition dynamics, while the neural response to features X. may vary as function of recognition dynamics. These two fea- ture matrices will be merged together to yield the design matrix X Equation
We next define a set of candidate linking models which describe how the dynamics of the cognitive model (specifically, word recognition times T;) af- fect observed neural responses. These models are all variants of temporal receptive field model (TRF; Lalor et al., 2009; Crosse al., 2016), which predicts scalp EEG data over S sensors and T samples, Y E SXT, as a convolved set of lin- ear responses to lagged features the stimulus:
We enumerate several possible classes neu- ral models which describe different ways that a word's 's recognition time may affect the neural response. Each model class constitutes differ- ent answer to our framing questions of onset and response properties (Table 2 and Figure 1b), by specifying different featurizations of word-level properties X, in the TRF design matrix X:
Ys ΣΣ 4=0
S.SAA ( Est (5)
where Ty is the maximum expected lag (in sec- onds) between the onset of a feature f and its cor- relates in the neural signal; and the inner sum is accumulated in steps of the relevant neural sam- pling rate. This deconvolutional model estimates characteristic linear response linking each feature of the stimulus the neural data over time. The model allows us to effectively uncover the neural response to individual stimulus features in natu- ralistic data, where stimuli (words) arrive at a fast
Unitary response aligned to word onset (baseline model): All words exhibit uni- tary linear neural response to recognition and integration, time-locked to the word's onset in the stimulus. This baseline model, which does not incorporate the cognitive dynamics of recognition in any way, is what has been assumed by prior naturalistic modeling work.
This model asserts that each word's features

---

words in each quantile.
a%
42%
This model thus asserts that it is possible to group words by their recognition dynamics such that they have a characteristic neural response within-group, but differ freely be- tween groups.
0.00
after word onset
0.25
0.00
sec after word onset 025
(a) Computation recognition time 7; for recognition point after phoneme & (left) or recognition prior to input, 4g- = (right) for a spoken word fish /ftf/. See eq. 4. Shift: Unitary response, aligned recognition time
4.
Variable response by word surprisal, aligned word onset (prior-variable modely: This model is identical model, except that words are divided into quantiles based on their surprisal in context rather than their recognition time.
M1
the above variable
M3
0.08
0.1
02
Variable: Variable respanse, aligned to word onset
Recognition time early W>) (wa)
This model instantiates the hypothesis that the shape of the neural response to words varies based on listeners' expectations, but only those driven by the preceding linguis- tic context. On this reading, words are pre- activated according to their prior probability, rather than their rapidly changing posterior probability under some acoustic input."
Baseline: Unitary response, aligned to word onset
0.0
0.3
0.4
0.5
Time since word onset (s)
(b) Candidate neural model logic linking three words' reong- nition times T, to neural modulations by surprisal.
Figure I: Sketches of model logic.
For a set of recognition time predictions Ti we estimate within-subject TRFs under each of these linking models, yielding per-subject pa- rameters θj describing the combined neural re- sponse objective stimulus features and word- level features. This estimation procedure allows for within-subject variation in the shape of the neural response.
Xui trigger a neural response beginning at the onset of word i, and that this neural response can be captured by single characteristic sponse to all words.
2. Unitary response aligned to recognition time (shift model): All words exhibit a unitary lin- ear neural response to recognition and inte- gration, time-locked to the word's recogni- tion time T,
Methods and dataset
We jointly infer? across-subject parameters of the cognitive model (Table 1) and within-subject pa- rameters of the neural model in order to minimize regularized L2 loss on EEG data, estimated by 4-fold cross-validation. We then compare the fit models on held-out test data, containing 25% of the neural time series data for each subject. For each comparison of models mi, 122, we compute the Pearson correlation coefficient between the predicted and observed neural response for each subject at each EEG sensor s. We then use paired t-tests to ask whether the within-subject difference
This model asserts that each word's features Xui trigger neural response beginning at т seconds after word onset, and that this neural response can be captured by a single charac- teristic response all words.
3. Variable response by recognition time, aligned to word onset (variable model): Words exhibit differential neural response to recognition and integration based their recognition time. The temporal onset of these integration processes is insensitive to the progress of word recognition.
'This reading compatible with pre-activation theories (e.g. Brothers and Kuperberg. 2021). At their present level of specificity, it unelear whether this focus prior prob- ability a substantive commitment, simply a choice of modeling expediency.
We account for variable responses by defin- ing a quantile split Q -> on the in- ferred recognition times t; We then estimate distinct TRF parameters for the features of
We conduct tree-structured Parzen estimator random search (Bergstra et al., 201 1) with Optuna (Akiba et al., 2019).

---

in pooled across sensors significantly differs be- tween my and m2:
Recognition quantile
Count
400
Early
g
Ys m1.5
1 5
Y Mmzis)
Ys
(6)
200
Late
Mid
8.0
0.1
0.2 0.3
0.4
Recognition time (sec after word onset)
Dataset We analyze EEG data recorded as 19 subjects listened to Hemingway's The Old Man and the Sea, published in Heilbron et al. (2022). The 19 subjects each listened the first hour of the recorded story while maintaining fixation. We analyze sensors distributed across the centro- parietal scalp: one midline sensor and two lateral sensors per hemisphere at central and posterior po- sitions. The EEG data were acquired using a 128- channel ActiveT'wo system at a rate of 512 Hz, and down-sampled offline to 128 Hz and re-referenced the mastoid channels. We follow the authors' preprocessing method, which includes band-pass filtering the EEG signal between 0.5 and Hz, vi- sual annotation of bad channels, and removal of eyeblink components via independent component analysis. 3 The dataset also includes force-aligned annotations for the onsets and durations of both words and phonemes in these time series.
Figure 2: Distribution of inferred recognition times (relative to word onset) for all words, predicted by the optimal cognitive model parameters. Salmon ver- tical lines indicate tertile partition of words by their recognition time; light yellow regions indicate the me- dian duration of phonemes at each integer position within word. An example stimulus word, occasional, is aligned with phoneme duration regions above the graph.
who estimated patterns of confusion in phoneme recognition within English consonants and vow- els by asking subjects transcribe spoken sylla- bles. Their raw data consists of count matrices Wey the for consonants and vowels, respectively, where each cell ([ij] denotes the number of times an experimental subject transcribed phoneme j as phoneme i, summing over different phonologi- cal contexts (syllable-initaal or -final) and differ- ent levels of acoustic noise in the stimulus pre- sentation. We concatenate this confusion data into single matrix, imputing a count of 1 for unob- served confusion pairs, and normalize each col- umn to yield the required conditional probability distributions.
We generate predictor time series X₃ aligned with this EEG time series (Appendix B), ranging from stimulus features (features of the speech en- velope and spectrogram) to sublexical cognitive features (surprisal and entropy over phonemes). By including these control features in our mod- els, we can better understand whether or not there is cognitive and neural response to words dis- tinct from responses to their constituent properties (see Section 4.2 for further discussion). We gener- ate in addition set of word-level feature vectors Xy 33x*= consisting of an onset feature and
Results
We first evaluate the baseline model relative to a TRF model which incorporates no word-level fea- tures X, except for a word onset feature, and find that this model significantly improves held-out prediction performance (! = 4.91, p = 0.000113). The model recovers negative response to word surprisal centered around 400 ms post word on- set (Figure 6), which aligns with recent EEG stud- ies of naturalistic language comprehension in both listening (Heilbron et al., 2022; Gillis al., 2021; Donhauser and Baillet, 2020) and reading (Frank et al., 2015).
1. word surprisal in context, computed with GPT Neo 2.7B (Black et al., 2021),* and
2. word unigram nog-frequency from SUB- TLEXus 2 (Brysbaert and New, 2009).
Likelihood estimation Our cognitive model quires an estimate of the confusability between English phonemes (Equation 2). We draw on the experimental data of Weber and Smits (2003),
"See Appendix further details on our choice band- pass filter width.
We next separately infer optimal model parame- ters for the shift and variable models, and evaluate their error on held-out test data. We find that the variable model significantly exceeds the bascline model (t = 5.15,p 6.70 x 10 5), while the shift
'Preliminary experiments using our buseline model showed that surprisal estimates from GPT Neo 2.7B best ex- plained held-out EEG signals, compared among other sizes of GPT Neo and OpenAl GPT-2 models (Radford et al., 2019; Brown et al., 2020).

---

1e-3
0.25
0.00
TRF coefficient (a.u.)
-0.25
1.
1,0
0,9
0.8
007
0.6
0.5
16-3
-0.50
-0.75
Recognition time quantile
-1.00
Early
-1.25
Mid
-1.50
Late
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Epoch time (sec after word onset)
Figure 3: Modulation of scalp voltage at centro-parictal site by surprisal for words with early (< 64 ms, blue),
middle (< 159 ms, orange), or late (> 159 ms, green) recognition times. Lines denote inferred coefficients of word surprisal in averaged over subjects for the sensor highlighted in the inset. Error regions denote s.e.m. (n 19). Inset: spatial distribution of surprisal modulations averaged for each recognition time quantile within vertical gray regions, where less saturated colors denote more negative response. The surprisal response peaks <-400 ms post onset, amplified for late-recognized words (green).
model does not (t 2.23. - 0.039). This sug- gests that neural responses to words are not simply temporally yoked to their recognition times.
which differ as a function of recognition time:
Figure 3 shows word surprisal modulations estimated at centro-parietal site for the three recognition time quantiles. Words recog- nized late (159 ms or later post word on- set) show exaggerated modulation due word surprisal. The peak negative amplitude of this response is significantly more nega- tive than the peak negative response to early words (fig. 3, green line peak minus blue line peak in the shaded region; within-subject paired t 5.233, p 5.71 x 10-5). This modulation is spatially distributed similarly to the modulation for carly-recognize words (compare the green inset scalp distribution that of the blue and orange scalps).
We next investigate the parameters of the opti- mal variable model. Figure 2 shows the distribu- tion of predicted word recognition times т; under the optimal variable model on stimulus data from the held-out test set, charted relative to the onset a word. Our model predicts that one third of words are recognized prior to 64 ms post word on- set, another third are recognized between 64 ms and 159 ms, and long tail are recognized after 159 ms post word onset. This entails that at least third of words recognized prior any mean- ingful processing of acoustic input. This predic- tion aligns with prior work in multiple neuroimag- ing modalities, which suggests that listeners pre- activate features of lexical items far prior to their acoustic onset in the stimulus (Wang et al., 2018; Goldstein et al., 2022).
There no significant difference in the la- tency of the N400 response for words recog- nized early vs. late. The time at which the surprisal modulation peaks negatively does not significantly differ between early and late words (fig. 3, green line peak time minus blue line peak time; within-subjest paired 2.17, 0.0440).
These inferred recognition times maximize the likelihood of the neural data under the linking vari- able model parameters O Figure 3 shows the vari- able model's parameters describing a neural re- sponse word surprisal for each of three recog- nition time quantiles, time locked to word onset. We see two notable trends in the N400 response
These model comparisons and analyses of opti- mal parameters yield answers our original ques- tions about the dynamics of word recognition and integration:
direct comparison of the variable model and shift model performance also favors the variahle model (t 5.49. = 3.21 x 10 19.

---

Response properties: Neural modulations due to surprisal are exaggerated for words recog- nized late after their acoustic onset.
Future work will need leverage other types of neural data to distinguish these models. We dis- cuss this further Section 4 and the Limitations section.
Onset: The variable model, which asserted integration processes are initiated relative to words' onscts rather than their recognition times, demonstrated a better fit to the data. The opti- mal parameters under the variable model further showed that while word recognition times seem to affect the amplitude of neural modulations due to surprisal, they do not affect their latency.
Discussion
This paper presented a cognitive model of word recognition which yielded predictions about the recognition time of words in context 7₁. A sec- ond neural linking model, the variable model, es- timated the neural response words recognized at early, intermediate, and late times according to the cognitive model's predictions. This latter model significantly improved in held-out generalization performance over a baseline model which did not allow for differences in the neural signal func- tion of a word's 's recognition time. We also found, however, that neural model which estimated dis- tinct shapes of the neural response words based on their surprisal - not their recognition times- also improved beyond our baseline, and was indis- tinguishable from the variable model. More elabo- rate neural linking theories describing how words' features drive the neural response will be neces- sary to distinguish these models (see e.g. the en- coding model of Goldstein et al., 2022).
3.1 Prior-variable model
We compute a surprisal-basee quantile split over words in the training dataset. The first third of low-surprisal words had a surprisal lower than I 1.33 bits, while the last third of high-surprisal words had a surprisal greater than 3.71 bits.
We next estimate the vrior-variable neural model parameters, which describe independent neural responses to words in low-, mid-, and high- surprisal quantiles. This model also significantly exceeds the baseline model (L 7.78, = 3.64 x 10 : see Appendix C for inferred model param- eters). Figure 4 shows a comparison of the way the prior- variable model and the variable model sorted words into different quantiles. While the two models rarely made predictions at the oppo- site extremes (labeling a low-surprisal word as late-recog,ized, high-surprisal word early- recognized; bottom left and upper right black cor- ners in fig. 4a), there were many disagreements in- volving sorting words into neighboring time bins (off-diagonal in fig. 4a). Figures 4b and 4c show some meaningful cases in which the models dis- agree to be due to differences in the relevant phonological neighborhood early in the onset of a word. Figure 4c shows the recognition model's posterior belief over words (eq. 1) given the incre- mental phonetic input at the top of the graph. The left panel of Figure 4c shows how the word disgust is recognized relatively late due to large num- ber of contextually probable phonological neigh- bors (such as dismay and despair); the right panel shows how the word knelt is recognizable rela- tively early, since most of the contextually prob- able completions (took, had) are likely to be ruled out after the presentation of second phone.
Our positive findings are consistent with a two- part model of auditory word recognition and in- tegration, along the lines suggested by van den Brink et al. (2006) and Hagoort (2008, $3c). In this model, listeners continuously combine their expectations with evidence from sensory input in order load possible lexical interpretations of the current acoustic input into a memory buffer. Our model's prediction of word's recognition time т; measures the time at which this buffer resolves in a clear lexical inference.
A second integration process reads out the con- tents of this buffer and merges them with repre- sentations of the linguistic context. Our latency results show that the timing of this process is independent a listener's current confidence their lexical interpretations, instead time-locked to word onset. This integration process thus ex- hibits two distinct modes depending on the lis- tener's buffer contents: one standard, in which the buffer is clearly resolved, and one exceptional, in which the buffer contents are still ambiguous, and additional inferential or recovery processes must be deployed in order to proceed with integration. Future work could spell out this distinction mech-
The variable model's generalization perfor- mance not significantly different than that of this variall model (t = -0.422, = 0.678).

---

Context
Prior-only
Rec. time
prediction
prediction
...he looked at it in disgust
Mid
Late
the old man was now definitely and fiually
Mid
Late
2500
drew his knife across one of the strips
Mid
Late
Low
2676
768
100
cheeks. The blotches
High
Surprisal quantile Mid
2000
THe knelt
High
T
Mid Early
1500
702
1885
1029
Examples of disagreements in word labeling between the prior- only model and recognition model.
1000
$:0g
High
180
970
2438
500
disbitt
did
disper
Early
Late
dismer
tok
Recognition time quantile
desgest
ne
0.2
0.4
1.0
0.0
0.2
de
08
10
Evidence for
Evidence word
(a) Confusion matrix comparing partitions of words by the prior-variable model (hased on word surprisal; verti- (c) Example posterior predictive distributions for words recognized cal axis) and the optimal word recognition model (based late due to a dense neighhorhood (left); and early due to sparse on recognition time; horizontal axis).
neighborhood (right).
Figure 4: Differing predictions of the word recognition model and the prior-variable courpisal--aseen model.
anistically in order to explain how buffers in the "exceptional" state elicit these distinct neural re- sponses.
chances of success the expected case.?
Alternatively, listeners may use the results of the word recognition process to schedule upcom- ing attempts at word integration. After recogniz- ing each word w;, listeners may form expecta- tion about the likely onset time of word wi+1, us- ing knowledge about the form of w; and the speech rate. Listeners could instantiate a clock based on this prediction, counting down to a time some fixed distance from the expected onset of w at which semantic integration would be most likely to succeed on average. Such theory could ex- plain how word recognition and integration least approximately optimal given limited cogni- tive resources (Simon, 1955; Lieder and Griffiths, 2020): they are designed to successfully process linguistic inputs in expectation, under the archi- tectural constraint of a fixed integration clock.
4.1 What determines integration timing?
Our findings on the stable timing of the natural- istic N400 align with some prior claims in the ex- perimental ERP literature (Federmeier and Laszlo, 2009, $5). These results strengthen the notion that, even in rapid naturalistic environments, the timing of the early semantic integration of word meanings is driven not by when words are recog- nized, but rather by the tick of external clock.
at
this integration process is not sensitive to the status of word recognition, then what drives its dynamics? Federmeier and Laszlo (2009) ar- gue that this regularly timed integration process is exnguage-external, functioning to bind early rep- resentations of word meaning with existing cog- nitive representations of the context via temporal synchrony (see also Kutas and Federmeier, 201 However, other language-internal internal mechanisms are also compatible with the data. Listeners may adapt to low-level features of the stimulus, such as their counterpart's speech rate or prosodic cues, manip- ulating the timing of integration to maximize the
4.2 Words as privileged units of processing Our results suggest that words exist at privi- leged level of representation and prediction dur- ing speech processing. This is not necessary property of language processing: it is possible that word-level processing effects (neural or be- havioral responses to word-level surprisal) could emerge as an epiphenomenon of lower-level pre-
"This is a claim about the wüthin-subject consistency of N400 timing, despite substantial between-subject variability, for example, by age and language experience (Fodermeier and Laszlo, 2009).
'See Verschueren et al. (2022, Figure 6 and Table 4) for evidence against this point, demonstrating that controlled variation in stimulus speech rate does not affect the latency of the N400 response.

---

diction and integration of sublexical units, e.g., graphemes or phonemes. Smith and Levy (2013, $2.4) illustrate how a "highly incremental" model which is designed to predict and integrate sublex- ical units (grapheme- or phoneme-based predic- tion) but which is measured at higher levels (in word-level reading times or word-level neural re- sponses) could yield apparent contrasts that are suggestive of word-level prediction and integra- tion. On this argument, neural responses to word- level surprisal are not alone decisive evidence for word-level prediction and integration (versus the prediction and integration sub-lexical units).
nition dynamics which also exceeded baseline models describing the N400 integration response. More substantial linking hypotheses bridging be- tween the cognitive state the word recognition model and the neural signal will be necessary to separate these distinct models.
Limitations
There are several important methodological limi- tations to the analyses in this paper.
We assume for the sake of modeling expedi- ency that all listeners experience the same word recognition dynamics in response to linguistic stimulus. Individual differences contextual ex- pectations, attention, and language knowledge cer- tainly modulate this process, and these differences should be accounted for in an elaborated model.
Our results add critical orthogonal piece of ev- idence in favor of word-level integration: we char- acterized integration architecture whose timing is locked to the appearance of word units in the stimulus. While the present results cannot iden- tify the precise control mechanism at play here (section 4, the mere fact that words are the tar- get of this timing process indicates an architecture strongly biased toward word-level processing.
We also assume relatively low-dimensional neural response to words, principally asserting that the contextual surprisal of a word drives the neural response. This contrasts with other re- cent brain mapping evaluations which find that high-dimensional word representations also ex- plain brain activation during language compre- hension (Goldstein et al., 2022; Caucheteux and King, 2022; Schrimpf al., 2021). A more elaborate neural linking model integrating higher- dimensional word representations would likely al- low us to capture much more granular detail at the cognitive level, describing how mental representa- tions of words are retrieved and integrated real time. Such detail may also allow us to separate the two models (the variable and prior-variable mod- els) which were not empirically distinguished by the results of this paper.
4.3 Prospects for cognitive modeling
The cognitive model of word recognition intro- duced in this paper is extension of Shortlist B (Norris and McQueen, 2008), race architecture specifying the dynamics of single-word recogni- tion within sentence contexts. We used neural net- work language models to scale this model to de- scribe naturalistic speech comprehension. While we focus here on explaining the neural response to words, future work could test the predictions of this model in behavioral measures of the dynamics of word recognition, such as lexical decision tasks (Tucker al., 2018; Ten Bosch et al., 2022).
Acknowledgments
5 Conclusion
We thank Aixiu An, Jacob Andreas, Canaan Breiss, Trevor Brothers, Tyler Brooke Wilson, Samer Nour Eddine, Evelina Fedorenko, Micha Heilbron, Shailee Jain, Peng Qian, Cory Shain, Jakub Szewezyk, and Josh Tenenbaum for com- ments on earlier versions of this paper. We thank Micha Heilbron, Marlies Gillis, and Tamar Regev for invaluable advice on EEG data analysis, and for sharing analysis code and data. JG grate- fully acknowledges support from the Open Philan- thropy Project and RPL gratefully acknowledges support from Newton Brain Science Research Seed Award.
This paper presented model of the cognitive and neural dynamics of word recognition and inte- gration. The model recovered the classic N400 integration response, while also detecting dis- tinct treatment of words based on how and when they are recognized: words not recognized until more than 150 ms after their acoustic onset ex- hibit significantly amplified neural modulations by surprisal. Despite this processing difference, we found distinction in the latency of integration depending word's recognition time.
However, we developed an alternative model of the neural signal not incorporating word recog-

---

References
Katherine A. DeLong, Thomas P. Urbach, and Marta Kutas. 2005. Probabilistic word pre-activation dur- ing language comprehension inferred from electri- cal brain activity, Nature Neuroscience, 8(8):1117- 1121.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Op- tuna: A next-generation hyperparameter optimiza- tion framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Peter W. Donhauser and Sylvain Baillet. 2020, Two Distinet Neural Timescales for Predictive Speech Processing. Neuron, 105(2)33553333.e)
Paul Allopenna, James Magnuson, and Michael
Tanenhaus. 1998. Tracking the time course of spo- ken word recognition using eye movements: Evi-
Kara D Federmeier. 2007. Thinking ahead: The role and roots prediction in language comprehension. Psychophysiology, 44(4):491
dence for continuous mapping models. Journal of memory and language, 1894):419-439.
Kara D Federmeier and Marta Kutas. 1999. A rose by any other name: Long-term memory structure and sentence processing. Journal of memory and Lan- guage, 41 41(4):469-495.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. Advances in neural information pro- cessing systems, 24.
Kara Federmeier and Sarah Laszlo. 2009. Time for meaning: Electrophysiology provides insights into the dynamics of representation and processing se- mantic memory. Psychology of learning and mori- vation, 51:1-44.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Stefan L. Frank, Leun J. Otten, Giulia Galli, and Gabriella Vigliocco. 2015. The ERP response to the amount of information conveyed by words in sen- tences. Brain and Language, 140:1 1-:1--11.
Trevor Brothers and Gina R Kuperberg. 2021. Word predictability effects are linear, not logarithmie: Im- plications for probabilistic models of sentence com- prehension. Journal of Memory and Language, 116:104174.
Marlies Gillis, Jonas Vanthornhout, Jonathan Z. Si- mon, Tom Francart, and Christian Brodbeck. 2021. Neural Markers of Speech Comprehension: Measur- ing EEG Tracking of Linguistic Speech Representa- tions, Controlling the Speech Acoustics. Journal of Neuroscience,
Trevor Brothers, Emily Morgan, Anthony Yacovone, and Gina Kuperberg. 2023. Multiple predictions during language comprehension: Friends, foes, or indifferent companions? Cognition, 241:105602.
Colin Brown and Peter Hagoort. 1993. The Processing Nature of the N400: Evidence from Masked Prim- ing. Journal of Cognitive Neuroscience, 5(1):34-44
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mari- ano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Co- hen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha De- vore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Ma- tias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson. 2022. Shared computational principles for language processing in humans and deep language models. Nature Neursciencce, 25(3):369-333
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Mare Brysbaert and Boris New. 2009. Moving be- yond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41 (147777779900
François Grosjean. 1980. Spoken word recognition processes and the gating paradigm. Perception & Psychophysics, 2874):667
Peter Hagoort. 2008 The fractionation of spoken lan- guage understanding by measuring electrical and magnetic brain signals. Philosophical Transac- tions of the Royal Society B: Biological Sciences, 363(1493); 10555:0055-1069
Charlotte Caucheteux and Jean-Rémi King- 2022. Brains and algorithms partially converge in natu- ral language processing. Communications biology, 5(1):134.
Michael Crosse, Giovanni M. Di Liberto, Adam Bed- nar, and Edmund C. Lalor. 2016. The multivariate temporal response function (mtrf) toolbox: A mat- lab toolbox for relating neural signals to continuous stimuli. Frontiers in Human Neuroscience, 10.
Micha Heilbron, Kristijan Armeni, Jan-Mathijs Schof- felen, Peter Hagoort, and Floris P De Lange. 2022. A hierarchy of linguistic predictions during natural language comprehension. Proceedings of the Nu- tional Academy of Sciences, 119(32)ee2011668

---

Gina R Kuperberg, Trevor Brothers, and Edward W Wlotko. 2020. A tale of two positivities and the n400: Distinct neural signatures are evoked by con- firmed and violated predictions at different levels of representation. Journal of Cognitive Neuroscience, 52(1))12-335
Darren Tanner, Kara Morgan-Shone and Steven Luck. 2015. How inappropriate nigh-pass filters can pro- duce artifactual effects and incorrect conclusions in erp studies of language and cognition. Psychophys- iology, 52(8):997-1009
Louis Ten Bosch, Lou Boves, and Mirjam Ernestus. 2022. Diana, a process-oriented model of human au- ditory word recognition. Brain Sciences, 12(5)
Gina R. Kuperberg and T. Florian Jaeger. 2016. What do we mean by prediction in language comprehen- sion? Language, Cognition and Neuroscience, 31(1):32-59.
Benjamin V. Tucker, Dan Brenner, D.
Kyle Daniel- son, Matthew C. Kelley, Filip Nenadié, and Michelle Sims. 2018. The massive auditory lexical deci- sion (mald) database. Behavior Research Methods, 51:1187 1204.
Marta Kutas and Kara D. Federmeier. 2011. Thirty Years and Counting: Finding Meaning in the N400 Component of the Event-Related Brain Potential (ERP). Annual Review of Psychology. 62(1):621- 647.
Daniëlle van den Brink, Colin M. Brown, and Peter Hagoort. 2006. The cascaded nature of lexical se- lection and integration in auditory sentence process- ing. Journal of Experimental Psychology: Learn- ing, Memory, and Cognition, 32(2):364 3722.
Marta Kutas and Steven A. Hillyard. 1984. Brain po- tentials during reading reflect word expectancy and semantic association. Nature, 307(5947):161-163.
Edmund C. Lalor, Alan J. Power, Richard B. Reilly, and John Foxe. 2009, Resolving precise tempo- ral processing properties of the auditory system us- ing continuous stimuli. Journal of Neurophysiology,
Eline Verschueren, Marlies Gillis, Lien Decruy, Jonas Vanthornhout, and Tom Francart. 2022. Speech un- derstanding oppositely affects acoustic and linguis- tic neural tracking in a speech rate manipulation paradigm. Journal of Neuroscience, 42(39):7442- 7453.
102(1):349-355 PMID: 19439675.
Falk Lieder and Thomas L Griffiths. 2020. Resource- rational analysis: Understanding human cognition the optimal use of limited computational sources. Behavioral and brain sciences, 43:e1.
Lin Wang. Gina Kuperherg. and Ole Jensen. 2018. Specific lexico-semantic predictions associated with unique spatial and temporal patterns of neural activity. Elife, 7:e39061.
William D Marslen- Wilson. 1987. Functional paral- lelism in spoken word-recognition. Cognition, 25(1- 2):71-102.
Andrea Weber and Roel Smits. 2003. Consonant And Vowel Confusion Patterns By American English Lis- teners. page
Norris and McQueen. 2008 Shorllist B: Bayesian model of continuous speech recognition. Psychological review.
A Relation to pre-activation accounts
Timothy В O'Rourke and Phillip Holcomb. 2002. Electrophysiological evidence for the efficiency of spoken word processing. Biological psychology, 60(2-3):121-11)
Our theoretical account discussed in Section 4 is partly compatible with pre-activation accounts of prediction in language comprehension, which likewise suggest that listeners eagerly pre-activate features at multiple levels of linguistic represen- tation, according to both contextual expectations and partial sensory input (see e.g. Federmeier (2007); Federmeier and Laszlo (2009); Kutas and Federmeier (201 I); Kuperberg and Jaeger (2016) for reviews). Our cognitive model of word recog- nition provides a mechanism for the temporal dy- namics of this pre-activation process. This mech- anism is an aggressively incremental process, de- pending on a probabilistic inference which repeat- edly integrates novel acoustic evidence with exist- ing expectations drawn from the context.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Iya Sutskever, al. 2019. Lan- guage models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Eivelina Fedorenko. 202 I. The neural architecture of language: Integrative modeling converges on predictive processing. Pro- ceedings of the Nutional Academy of Sciences, 118(45):e2 105646111
Herbert A Simon. 1955. A behavioral model of rational choice. The quarterly journal of economics, pages 99-118.
Pre-activation accounts suggest that what is pre-activated are abstract semantic features rather than specific lexical items (Federmeier and Kutas, 1999; Kuperberg and Jaeger, 2016). The present
Nathaniel J Smith and Roger Levy. 2013. The effect of word predictability on reading time logarithmic. Cognition, 128(3)3002-31.

---

model is stated at the computational level and is thus not directly comparable in this respect. Fu- ture modeling work can instantiate specific repre- sentational alternatives within this predictive word recognition model and explore how their predic- tions might settle these questions.
words which are exactly phonologically compat- ible with an observed prefix. See Heilbron al. (2022) for further details on the model specifica- tion.
C Inferred neural response under the prior-variable model
B Model featurization
Figure shows the inferred neural response to words of different surprisal quantiles under the prior- model described in Section 3.1. We amplified negative peak in high-surprisal words, similar to that in Figure 3 for late- recognized words.
We use subset of the sublexical features from Heilbron et al. (2022) in our TRF models (named as X, in Section 1.2). These features are shared across all models tested in our main and baseline analysis:
onset features for each phoneme in the audio stimulus;
D Baseline estimates of the neural response to surprisal
phoneme-onset aligned features:
Figure 6 shows the baseline model's estimated re- sponse to word's surprisal. The model recov- ers the standard broad negative response centered around 400 ms post word onset, which aligns with recent EEG studies of naturalistic language prehension in both listening (Heilbron al., 2022: Gillis al., 2021; Donhauser and Baillet, 2020) and reading (Frank et al., 2015).
acoustic control features, averaged within the span of phoneme: average variance in the broadband envelope, and spectral power measures averaged within eight bins spaced evenly on a log-mel scale
the entropy over a next-phoneme distri- bution P(p; wi,<;) and the surprisal of the ground-truth phoneme, using the hi- erarchical predictive model of Heilbron al. (2022) (see below).
Figure 7 shows estimates of the neural response to phoneme surprisal from both the baseline model and the optimal variable model. All models tested in this paper included this phoneme surprisal pre- dictor; the main results of the paper thus tar- get neural activity above and beyond what plained by phoneme-level responses. See Sec- tion 4.2 for further discussion.
B.1 Phoneme probability estimator
The phoneme model of Heilbron et al. (2022), whose surprisal and entropy measures we use as control predictors, combines word-level lan- guage model prior and cohort-based likelihood. For some prior phoneme sequence Ply... Pt..1..Pt-1 and some incoming phoneme Pt in a linguistic context C, we define
E Choice of band-pass filter
critical preprocessing step in our data analy- sis to band-pass filter the raw EEG signal, re- taining signals within a frequency window of 0.5- 8 Hz. This choice of filter parameters is similar to that of other recent studies of naturalistic language comprehension which use temporal receptive field models (see e.g. Gillis et al., 2021; Heilbron et al., 2022). A reviewer points out, however, that this filter window is substantially narrower than that of classic controlled studies of the evoked N400 based on trial-averaging ERP analyses (e.g. Ku- tas and Hillyard, 1984; Brown and Hagoort, 1993; Brothers et al., 2023). This choice of narrow filter parameters for our temporal receptive field analy- sis has several motivations:
P(pel Pi..... Pt 1.C) α Σ P(w C.PI,...,P4 P(pel w)
WCV
Σ P(w| 1(w E Coh(pi... ..........1111... P. WEV
where V is vocabulary of all possible word forms, and Coh(pi...... Pt) denotes the cohort of phoneme sequence Pi...., Pt i.e. all the words which share the given prefix of phonemes.
This model thus effectively renormalizes lan- guage model's word-level prior P(w C) among
We wish to focus on evoked responses time- locked to events (e.g. onsets of words and

---

1e-3
0.50
0.25
TRF coefficient (a.u.)
0.00
-0.25
-11
-1.0
-0.9
-0.8
-0.7
-0.6
-0.5
16-3
0..50
-0.75
-1.00
Surprisal quantile
-1.25
Low
Mid
-1.50
High
0.0
0.1
0.2
0.3
0.4 0.5
0.6 0.7
Epoch time (sec after word onset)
Figure 5: Modulation of scalp voltage at centro-parictal site by surprisal for words with low (< 1.33 bits, blue), mid (< 3.71 bits, orange), or high (> 3.71 bits, green) surprisals. Lines denote inferred coefficients word surprisal averaged over subjects for the sensor highlighted in the inset. Error regions denote s.e.m. (n 19). Inset: spatial distribution of surprisal modulations averaged for each surprisal quantile within vertical gray regions, where less saturated colors denote more negative response. This is replication of Figure with the parameters of the prior-variable model.
1e-3
16-3
0.50
Baseline model estimate Variable model estimate
0.0
coefiicient (a...
TRIP coufficient (a.b.)
0.2
0.25
0.6
0.50
-0.75
-
-1.00
-12
-1.25
0.0
0.1
0.2
0.3
0.4
0.5
0.5
0.7
Epoch time (sec after word onset)
Epoch time (sec after phoneme onset)
Figure 6: Modulation of scalp voltage by word sur- prisal in the baseline model central posterior sen- sor, highlighted in inset figure. Error regions denote s.c.m. (n = 19). Inset: spatial distribution sur- prisal modulations averaged within vertical gray gion, where less saturated colors denote more negative response.
Figure 7: Modulation of scalp voltage the same tral parietal sensor used in Figure 3 by phoneme sur- prisal, estimated the baseline model and the optimal variable model. Error regions denote s.e.m. (n 19).
pacity to explain drift in the signal.
However, it is possible that this choice of fil- ter parameters could introduce artifacts in the fil- tered signal which affect the outcomes of our N400-focused analysis. In particular, Tanner et al. (2015) point out that aggressive high-pass filters (~ 0.5 Hz and above) can conflate evoked N400 responses with later ERPs such as the P600, and yield inflated estimates N400 amplitude.
phonemes, and changes in cognitive state due to those stimuli) with rates around this fre- quency range. Including a wider spectrum adds variance to the signal which we cannot explain using our features of interest,
2. A high low-cut (more aggressive high-pass filter) allows us to account for signal drift; while this is handled through baselining and detrending in classic ERP analyses, temporal receptive field models have no equivalent ca-
E.1 Stability of the baseline model
We thus conducted a post-hoc stability analysis to better understand the sensitivity of this paradigm

---

Low cut
High cut
Result
We thus re-fit the temporal receptive field pa- rameters the optimal variable model described in this paper on EEG data preprocessed with a low-cut of 0.3 Hz, the lowest frequency cut at which the baseline model clearly establishes that an evoked surprisal response is readable in the sig- nal. Figure 8 shows the estimated neural modula- tion by word surprisal in these preprocessed data. of the inflated N400 response and an amplification of a P600 response.
0.5 Hz
8 Hz
f t= 4.91, P = O. 0.000113
0.3 Hz
8 Hz
[ (= 3.26, p = 0.00435
0.1 Hz
20 Hz
t = 1.95, P 0.0666
0.1 Hz
8 Hz
== 1.84, p 0.0826
Table 3: Post-hoc stability checks on the baseline model comparison with respect to the low- and high- cut of the band pass filter.
We found that this variable model displayed the same qualitative patterns in neural parameters. Quantitatively, we found a similar effect size of in- flated N400 amplitude (fig. 8 green line peak mi- nus blue line peak in the shaded region; within- subject paired
to our choice of band-pass filter parameters. We first repeated our initial model comparison on EEG data preprocessed with different band-pass filter parameters. This model comparison evalu- ates the improvement in predictive performance of temporal receptive field model which in- corporates control acoustic- phonntic features and word-level features (word surprisal and frequency) above model which does not include these word- level features. (This is the same model comparison described in the beginning of Section 3.) Table 3 shows the results of this evaluation.
-5.0, 8.71 10-5).
These supplementary analyses suggest that main findings are stable different parameteriza- tions of high-pass filter in EEG preprocessing.
F Reproducibility information
We jointly estimated the parameters of the cogni- tive model together with the hyperparameters and parameters of the neural linking model using mul- tivariate tree-structured Parzen estimator random search (Bergstra al., 2011) with Optuna (Akiba et al., 2019). For subjects = 1,..., S,, and held-out EEG time series data for subject i at sensor 8 Yi.s we maximized the value V:
We find that the predictive power of these word- level features diminishes as we decrease the low- cut frequency: beneath 0.3 Hz, this model compar- ison no longer shows a significant improvement in prediction due to word-level features. We do not take this result to invalidate the claim that word surprisal yields an evoked EEG response in nat- uralistic comprehension, since this has been sup- ported in other studies of naturalistic comprehen- sion with classic trial-averaging methods (Frank al., 2015).
1,..., N, sensors
1 N
V
Σ
max (E(1.....))
(YiiorPas)
(8)
However, it is important to check whether the central finding of this paper - which rests on an inflated N400 amplitude in response to some types of words - is sensitive to these parameter changes. In the next section, we reproduce our main qualitative findings for those preprocessing parameters which yield a clear positive baseline outcome of the evoked N400 response to surprisal.
which is the average across subjects of the maxi- mal Pearson correlation of predicted and observed EEG response among sensors. Table 4 shows the precise bounds for each parameter and hyper- parameter in this search procedure. We evaluated 20 trials (random settings of parameters) for the baseline model (which only incorporated the L.2 coefficient), and 500 trials for all other models. The model results presented in this paper (in vi- sualizations and statistical tests) correspond to the outcome of each grid search. Table shows the total count of free parameters under optimization. These counts do not include the parameters of the language model used to com- pute word surprisal, or the word recognition model
E.2 Stability of our main findings
The argument of Tanner et al. (2015) would pre- dict that the inflated N400 amplitude we observe in response to late-recognized words could be ex- plained away as an artifact of the high-pass fil- ter, which could confound the N400 with later evoked response (such as the P600). If this finding were purely artifactual, then if we were to relax this high-pass filter, we should see an attenuation
*Our latency foundings also held null (green line peak time minus blue line peak time; within-subject paired t = 2.17, = 0.043).

---

1e-3
0.25
0.00
TRF coefficient (a.u.)
-0.25
-1.
-1.0
-0.9
-0.8
-0.7
-0.6
-0.5
16-3
-0.50
00.75
-1.00
Recognition time quantile
-1.25
Early
-1.50
Mid
Late
-177
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Epoch time (sec after word onset)
Figure 8: Modulation of scalp voltage centro-parietal site by surprisal for words of different recognition time quantiles (according to the variable model of Figure 3), estimated on EEG data band-pass filtered with a low-cut of 0.3 Hz.
PHraer)Parameter
Bounds
Notes
Regression L2 coefficient
[10?, 10")
Logarithmic space. Bounds manually selected and restricted based on early runs of each model in order reduce total runtime
Y (recognition threshold)
(0, 1)
(evidence temperature)
[0.1,3]
a (scatter point)
[]
O'p (prior scatter point)
[0,
Table Specifications for parameter and hyperparameter bounds random search, details on meaning these parameters, see Table 1.
likelihood parameters, since these were kept fixed during optimization.
All temporal receptive field models were fit with a receptive field ranging from 0 ms to 750 ms post word onset.
We implemented all training and inference with GPU operations in PyTorch. Due the large memory requirements of the EEG time series data and the lagged regression computations, we de- ployed each model on two NVIDIA A100 GPUs. Each of the model fits completed in two days or fewer.

---

Model class
Parameter count
Decomposition
Baseline
138,226
138,225 TRF parameters 1 hyperparameter
Shift
147,446
147,440 TRE parameters + cognitive parameters + 1 hy-
perparameter
Variable
230,381
230,375 TRF parameters + 5 cognitive parameters + 1 hy- perparameter
Prrillvariable
230,375
230,374 TRF parameters 1 hyperparameter
Table 5: Number of free parameters in all fitted models.


Reviews:
[{"review_id": "qOjJCIMDEz", "reviewer": "Reviewer_oXeZ", "summary": "", "questions": "Given the robust existing modeling work on single-word recognition, I would have appreciated more explicit discussion of what is original here about the non-contextual elements of the model, and what is shared with existing frameworks. (On my understanding, the current proposal is akin to a continuous feed-forward architecture along the lines of RACE, but lacks feedback connections a-la TRACE and differs from COHORT in that word recognition is not constrained to begin after the first syllable). I'm not sure if my understanding is correct, but it seems like there is a missed opportunity to engage with some of these debates.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review.\n\nRegarding the band pass filter: this is an important point. We are not able to perform this robustness analysis (which entails re-running our entire pipeline) within this short rebuttal period, but will do this prior to the camera ready and report these results in the paper, preferentially over the current methods with a high low-cut.\n\nRegarding the space of recognition models: you are broadly correct. This is a feed forward race architecture operating on the level of a single word, akin to Shortlist B [1]. It is explicitly Bayesian, unlike TRACE, and is similar to a cohort model, except the likelihood function allows for substantial mismatches between acoustic input and a perceived word. We will add this detail and references in the final paper. We are excited to more thoroughly explore and compare the space of computational recognition models (in predicting neural data!) in future work.\n\nSince this review setup lacks a general commentary, we are adding this to each rebuttal: We would like to notify the reviewers of a new result. Upon further inspection of the prior-variable model (detailed in section 3.1) we discovered evaluation settings in which this model also significantly exceeds the generalization performance of the baseline model, and has a performance which does not differ significantly from that of the Variable model highlighted in the paper. (The significance tests behind these model comparisons are the same as detailed in the paper). This means that our results are no longer decisively in favor of the Variable model, and new data or new designs will be necessary in order to separate the two models that exceed baseline. In the camera ready we will present these updated comparisons and modulate our conclusions accordingly.\n\n[1] Norris, D., & McQueen, J.M. (2008). Shortlist B: a Bayesian model of continuous speech recognition. Psychological review, 115 2, 357-95 .", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Tight connection between theoretical foundation, modeling work, and neural signals\n- Model comparison provides clear theoretical deliverable in terms the relationship between word recognition and contextual integration\n", "reasons_to_reject": "The EEG data are highly band-limited to 0.5-8 Hz. This is an emerging norm in TRF analyses such as those presented here, but different substantially from the band used for \"standard\" ERP analysis (e.g. 0.1 - 20 Hz); Tanner et al. 2015 Psychophysiology in particular point out that both phase and amplitude distortions on the evoked response that can be caused by such a high low-cut. I for one would appreciate a robustness analysis to better understand how sensitive the model fitting results are to the choice of filter here.", "questions_to_authors": "Given the robust existing modeling work on single-word recognition, I would have appreciated more explicit discussion of what is original here about the non-contextual elements of the model, and what is shared with existing frameworks. (On my understanding, the current proposal is akin to a continuous feed-forward architecture along the lines of RACE, but lacks feedback connections a-la TRACE and differs from COHORT in that word recognition is not constrained to begin after the first syllable). I'm not sure if my understanding is correct, but it seems like there is a missed opportunity to engage with some of these debates.\n", "paper_topic_and_main_contributions": "This paper presents a bayesian model of context-sensitive word recognition and evaluates variants of this model which differ in how word-recognition and contextual factors modulate ERP signals such as the N400. Overall this is an exciting integration of psycholinguistic theories of word recognition, computational modeling, and highly time-resolved neural signals. The results suggest that word recognition and contextual integration may begin even before word-recognition processes are completed.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ARVATBZwMY", "reviewer": "Reviewer_FAWT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review.\n\nRegarding audience: our work is part of a growing trend combining neuroscientific data with modern NLP models for psycholinguistic insights [1,2,3,4,6,7]. We believe these methods and data sources can be useful for addressing core architectural questions in language processing, and that our paper helps demonstrate this utility. We will work to clarify and extend the neural model presentation in the final version to be more friendly to readers unfamiliar with neural data.\n\nRegarding predictions and interpretation: one half of our multi-level model is stated at the cognitive level, describing the perceptual state of the listener at any point in stimulus-time. It makes critical predictions about the specific point in each word at which a listener can successfully identify it. These predictions can be compared with other empirical measures of word recognition (e.g. [5]), which we plan to address in future work. For the moment, we see these cognitive-level predictions pay off in the way they help us better explain EEG neural responses to words, under the Variable linking model. This success is indirect evidence for the reality of these cognitive predictions.\n\nRegarding what is essential to the model: it's true that this model is one of many possible cognitive models. We chose this model for its simple and elegant computational-level explanation of the recognition phenomenon. Future work should more thoroughly explore the space of possible models; as this paper stands, we believe that our combination of a structured cognitive model with high-dimensional neural analysis is already an important methodological contribution.\n\n[1] Gauthier, J., & Levy, R. (2019). Linking artificial and human neural representations of language. Conference on Empirical Methods in Natural Language Processing.\n[2] Toneva, M., & Wehbe, L. (2019). Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). NeurIPS.\n[3] Jain, S., & Huth, A.G. (2018). Incorporating Context into Language Encoding Models for fMRI. bioRxiv.\n[4] LeBel, A., Jain, S., & Huth, A.G. (2021). Voxelwise Encoding Models Show That Cerebellar Language Representations Are Highly Conceptual. The Journal of Neuroscience, 41, 10341 - 10355.\n[5] Tucker, B.V., Brenner, D., Danielson, D.K., Kelley, M.C., Nenadić, F., & Sims, M. (2018). The Massive Auditory Lexical Decision (MALD) database. Behavior Research Methods, 51, 1187-1204.\n[6] Wehbe, L., Murphy, B., Talukdar, P.P., Fyshe, A., Ramdas, A., & Mitchell, T.M. (2014). Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses. PLoS ONE, 9.\n[7] Fyshe, A., Talukdar, P.P., Murphy, B., & Mitchell, T.M. (2014). Interpretable Semantic Vectors from a Joint Model of Brain- and Text- Based Meaning. Proceedings of the conference. Association for Computational Linguistics. Meeting, 2014, 489-499 .\n\nSince this review setup lacks a general commentary, we are adding this to each rebuttal: We would like to notify the reviewers of a new result. Upon further inspection of the prior-variable model (detailed in section 3.1) we discovered evaluation settings in which this model also significantly exceeds the generalization performance of the baseline model, and has a performance which does not differ significantly from that of the Variable model highlighted in the paper. (The significance tests behind these model comparisons are the same as detailed in the paper). This means that our results are no longer decisively in favor of the Variable model, and new data or new designs will be necessary in order to separate the two models that exceed baseline. In the camera ready we will present these updated comparisons and modulate our conclusions accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and, as far as I am able to evaluate, the methods are sound. This modeling/statistical analysis sheds light on questions about the neural and temporal dynamics of spoken word recognition.", "reasons_to_reject": "I'm not sure if this fits well with the primary topics of the conference and as I note above I'm a little stuck about how to situation or interpret this work. It seems to me to be a model-comparison of fits to EEG data, but that the primary contribution is a method for uncovering statistical patterns present in existing EEG data, rather than a cognitive model which explains what's going on in the listeners mind. In particular, I'm wondering if the model makes any novel, testable predictions? Or if the fit of the model to the EEG data is being assessed purely on the basis of consistency. How much of the model's performance is parameter-dependent, and which assumptions made at the model architecture level -- i.e. framing the task in terms of Bayesian decision theory -- are necessary for explanation or are something that while formally elegant are incidental to the output.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a set of model comparisons linking linguistic input to neural EEG response data recorded during natural listening / audio comprehension. The paper is well-written and the motivation for understanding the temporal and process-level dynamics of spoken-word / speech recognition are well presented.\n\nI will note first those on a procedural note that the paper does slightly exceed the 8 page limit for the conference, with the limitations section going onto the 9th page.\n\nOn the content side, the brunt of the work is of interest primarily to a cognitive neuroscience (or neuroscience of language) audience rather than those attending EMNLP, even in the psycholinguistics track. I found a some of the neuro-details to be outside my area of expertise, for instance the details of the TRF model. While I am by no means a neuroscientist, this makes me wonder how approachable the topic will be to the general audience at the conference. Furthermore, my comments about the paper should be taken with a grain of salt (given my lack of neuroscience data modeling experience). \n\nThat said I'm a little stuck about how to situation or interpret this work. It seems to me to be a model-comparison of fits to EEG data, but that the primary contribution is a method for uncovering statistical patterns present in existing EEG data, rather than a cognitive model which explains what's going on in the listeners mind. In particular, I'm wondering if the model makes any novel, testable predictions? Or if the fit of the model to the EEG data is being assessed purely on the basis of consistency. How much of the model's performance is parameter-dependent, and which assumptions made at the model architecture level -- i.e. framing the task in terms of Bayesian decision theory -- are necessary for explanation or are something that while formally elegant are incidental to the output.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NsqLkKulLN", "reviewer": "Reviewer_twWy", "summary": "", "questions": "Simply looking forward to getting access to data and script after acceptance ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review! We are excited to share our analysis pipeline and invite others to remix and build on the novel modeling work we have done here.\n\nSince this review setup lacks a general commentary, we are adding this to each rebuttal: We would like to notify the reviewers of a new result. Upon further inspection of the prior-variable model (detailed in section 3.1) we discovered evaluation settings in which this model also significantly exceeds the generalization performance of the baseline model, and has a performance which does not differ significantly from that of the Variable model highlighted in the paper. (The significance tests behind these model comparisons are the same as detailed in the paper). This means that our results are no longer decisively in favor of the Variable model, and new data or new designs will be necessary in order to separate the two models that exceed baseline. In the camera ready we will present these updated comparisons and modulate our conclusions accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well structured, well written and it contributes to\nan important research focus.\nResults and discussion on results make it a valuable contribution.", "reasons_to_reject": "No reason at all.", "questions_to_authors": "Simply looking forward to getting access to data and script after acceptance ", "paper_topic_and_main_contributions": "The paper proposes a computational model of auditory word processing based on a Bayesian decision model, which estimates EEG data, i.e. Event-Related Potentials representing the human brain neural activation responses during speech input.\nInterestingly, the approach well replicates EEG collected data, and represents an “in-vitro” way to disentangle some still open questions in the human serial recognition process of auditory speech and parallel interpretation process.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The reference list is not long, however it does contain all the most relevant contributions accrued in a couple of decades on the human processing of speech input.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper can be accepted as it is."}, "extra_text": {}}]

Author rebuttal:

Meta-review: > Overview

This paper presents a cognitive model of word integration in context, which makes predictions about word recognition times. It then proposes two linking functions (termed *shift model* and *variable model*) of how their predicted word recognition times affect observed neural responses, which they compare to a baseline (which does not use word recognition time information). They find that the variable model significantly outperforms the baseline.

> Meta Review

The three reviewers and I agree that this paper’s analysis sheds light on the neural and temporal dynamics of spoken word recognition. They also agree that the paper’s methods are sound (modulo a robustness analysis the authors promised to address for camera ready). There is some disagreement, however, regarding how to interpret the paper’s theoretical contributions. While two of the reviewers appreciate the proposed cognitive model for word recognition in context, reviewer FAWT questions which kind of testable predictions it makes. Personally, I share some of reviewer FAWT’s scepticism about the proposed model (longer comment below). Regardless of these issues, however, I found this paper interesting and believe it might motivate more future work on the topic. 

> Further Comments


I personally believe that: (i) this paper's topic does fit EMNLP; (ii) this paper's analysis makes a significant contribution in investigating the temporal dynamics of spoken word recognition; and (iii) this paper's contributions are not immediately clear from its text.

Regarding (iii), I had questions similar to reviewer FAWT's, not being able to clearly situate this paper's contributions. At a first read of this paper, I believed that this paper’s main contribution was the two proposed linking functions (described in section 1.2) and the comparison of model fits to EEG data. After re-reading all reviews and author responses in detail, however, I now believe that the paper also claims its Bayesian cognitive model (in section 1.1) as a key contribution. If this is the case, I would expect (as noted by reviewer oXeZ) a longer discussion comparing this model with prior work modelling single-word and word-in-context recognition. Ideally, I would expect to even see an empirical comparison between how predictive of neural responses are the *word recognition times* output by this cognitive model vs prior works’. The proposed cognitive model (in particular, eqs. 1 and 2 here) also seems very similar to, e.g., the word recognition model from Smith and Levy (2010)—which was developed for analysing reading times—and it would be nice to see a discussion comparing them. Further, the *variable model* could be more carefully motivated in my opinion. In particular, I don't see why the word integration process would rely on clusters based on a word's required recognition time, instead of, e.g., varying smoothly as a function of the word's recognition time.

Finally, I believe that reviewer FAWT’s question about how much of the model's performance is parameter-dependent becomes particularly relevant given the new results (in the authors’ response) that the proposed *variable model* (which predicts neural signals using clusters of word recognition times) is not significantly better than the *prior-variable model* (which predicts neural signals using clusters of surprisal values).

Smith and Levy (2010). Fixation durations in first-pass reading reflect uncertainty about word identity. 

> Potential Typos

I believe there might be a typo in eq. 5, as I don't see how word-level features would be incorporated into this equation.
Decision: Accept-Main