# Modeling evidence for non-competitive lexical selection in serial cloze

Cassandra Jacobs, Loïc Grobol, Ryan Hubbard, Kara Federmeier

<figure><img src="image_1.png" /></figure>

To cite this version:

Cassandra Jacobs, Loïc Grobol, Ryan Hubbard, Kara Federmeier. Modeling evidence for non-competitive lexical selection in serial cloze. International Workshop on Language Production 2024, Jun 2024, Marseille, France. . hal-05304518

HAL Id: hal-05304518

https://hal.science/hal-05304518v1

Submitted on 8 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Modeling evidence for non-competitive lexical selection in serial cloze

Cassandra L. Jacobs¹, Loïc Grobol², Ryan J. Hubbard³,⁴, & Kara D. Federmeier⁴

¹University at Buffalo, SUNY; ² Université Paris Nanterre; ³ University at Albany, SUNY; ⁴ University of Illinois at Urbana-Champaign

<figure><img src="image_3.png" /></figure>

## Introduction

* Competition during lexical selection may shape the speed and fluency of production.

* Fluency in a non-competitive production system should be insensitive to the specific activation of other responses

* Appears to hold for speeded, sentence-final cloze (Staub et al., 2015)

* "Wrap up" sentence (Just & Carpenter, 1980; Kuperberg et al., 2011)

* Does it hold for serial cloze tasks?

* Task can be seen as explicitly competitive from a task demand perspective (i.e., producing "the best continuation of the sentence" requires comparing different potential continuations)

* No serial cloze tasks with RT data due to "survey" format (e.g. Luke & Christianson, 2018)

* Goals:

* Present first large-scale serial cloze dataset with naming latencies (Jacobs, Hubbard, & Federmeier, 2022)

* Novel method for identifying semantic competitors

* Analyze relationship between reaction times and cloze probabilities

## Serial cloze dataset

* 109,225 completions of sentence fragments from CLOC dataset (Federmeier et al., 2007)

* 282 sentences (1/2 strongly and ½ weakly constraining)

* ~40 participants per sentence

* Implemented in jsPsych

* Participants incrementally guessed next word from each sentence a sentence fragment

* Latency to submit a response – here, the time to complete the entire word

* Includes deliberation time

* Manually standardized participants' responses to normalize case and spelling

* Cloze probability of each response

* Proportion of participants who produced that string for that preamble

## Quantifying semantic proximity

* Bayesian implementation of Gaussian mixture models (BGMMs) over RoBERTa (Liu et al., 2019) embeddings

* No specification of semantic features

* Weight concentration prior allows the model to:

* Select as many clusters as it needs

* Allocate as many data points as needed to those clusters

* Words that occur in the same cluster are assumed to be competitors

* Quantitatively and qualitatively coherent

* Manuscript with more details: https://osf.io/n4avt

<figure><img src="image_4.png" /></figure>

Gaussian mixture models recover structure of covariant data points in a continuous vector space (synthetic data)

<figure><img src="image_5.png" /><figcaption>### Semantic facilitation from neighbors for rarer responses</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>### Semantic competition from neighbors for common responses</figcaption></figure>

<figure><img src="image_7.png" /></figure>

<table><thead><tr><th>71%</th><th>9%</th><th>4%</th></tr></thead><tbody><tr><td>raven</td><td>magpie</td><td>hoodie</td></tr><tr><td>bird</td><td>cawcaw</td><td>hooded</td></tr><tr><td>crow</td><td>macaw</td><td></td></tr><tr><td>blackbird</td><td>jackdaw</td><td></td></tr><tr><td>...</td><td>...</td><td></td></tr></tbody></table>

Top 3 cloze response clusters for fragment: "A black European crow is called a ___"

## Analysis of reaction times

* The log odds of a cloze response (Balatsou et al., 2022, Cognition) strongly predicts reaction time for serial cloze responses

* We add to our modeling of RT

* A cluster probability (the proportion of cloze responses for the cluster to which a given response belongs)

* Whether the cluster probability is the same as the cloze probability (if cluster > cloze competitors present)

<table><thead><tr><th colspan="4">LME of log scaled naming latency</th></tr><tr><th>Fixed Effect</th><th>Estimate</th><th>SE</th><th>p</th></tr></thead><tbody><tr><td>Intercept</td><td>-0.22</td><td>0.01</td><td>&lt; .001</td></tr><tr><td>Scaled log cloze prob</td><td>-0.16</td><td>0.01</td><td>&lt; .001</td></tr><tr><td>Scaled log cluster prob</td><td>-0.11</td><td>0.01</td><td>&lt; .001</td></tr><tr><td>Competitor (Centered)</td><td>0.04</td><td>0.006</td><td>&lt; .001</td></tr><tr><td>Cloze prob *Cluster prob</td><td>-0.04</td><td>0.003</td><td>&lt; .001</td></tr><tr><td>Cloze * Competitor</td><td>-0.01</td><td>0.003</td><td>&lt; .01</td></tr><tr><td>Cloze*Cluster*Competitor</td><td>-0.02</td><td>0.003</td><td>&lt; .001</td></tr></tbody></table>

Formula: RT (log transformed and scaled) ~ (1 | Preamble) + (1 | Response) + Log Cloze Probability * Log Cluster Probability * Competitors

## Results and Conclusions

* Both sentence-final (Staub et al., 2015) and serial cloze production tasks show non-competitive language production mechanisms

* However, competitors seem to hurt production for high-probability responses

* Generalizing previous findings and experimental paradigms while providing large-scale support

* MOAR CONCLUSIONS

* Future work

## References

* Staub, A., Grant, M., Astheimer, L., & Cohen, A. (2015). The influence of cloze probability and item constraint on cloze task response time. Journal of Memory and Language.

* Just, M. A., & Carpenter, P. A. (1980). A theory of reading: from eye fixations to comprehension. Psychological review

* Kuperberg, G. R., Brothers, T., & Wlotko, E. W. (2020). A tale of two positivities and the N400: Distinct neural signatures are evoked by confirmed and violated predictions at different levels of representation. Journal of Cognitive Neuroscience

* Luke, S. G., & Christianson, K. (2018). The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods

* Jacobs, C. L., Hubbard, R. J., & Federmeier, K. D. (2022, February). Masked language models directly encode linguistic uncertainty. In Proceedings of the Society for Computation in Linguistics 2022

* Federmeier, K. D. et al. (2007). Multiple effects of sentential constraint on word processing. Brain Research

* Liu, Y. et al. (2019). RoBERTa: A robustly optimized BERT pretraining approach

* Balatsou, E., Fischer-Baum, S., & Oppenheim, G. M. (2022). The psychological reality of picture name agreement. Cognition

<figure><img src="image_8.png" /></figure>