# Semi-automatic selection of semantic substitutes in sentence comprehension stimuli

Cassandra L. Jacobs, Loïc Grobol, Alvin Tsang, Ryan J. Hubbard

► To cite this version:

Cassandra L. Jacobs, Loïc Grobol, Alvin Tsang, Ryan J. Hubbard. Semi-automatic selection of semantic substitutes in sentence comprehension stimuli. Architectures and Mechanisms for Language Processing, Sep 2025, Praha, Czech Republic. . [https://hal.science/hal-05304514](https://hal.science/hal-05304514) [hal-05304514]

HAL Id: hal-05304514

https://hal.science/hal-05304514v1

Submitted on 8 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Semi-automatic selection of semantic substitutes in sentence comprehension stimuli

<figure><img src="image_2.png" /></figure>

Cassandra L. Jacobs¹, Loïc Grobol², Alvin Tsang¹, & Ryan J. Hubbard³

University at Buffalo, SUNY; Université Paris Nanterre; University at Albany, SUNY

<figure><img src="image_3.png" /></figure>

## Research Questions

* How easily can we automatically select or validate within- versus between-category membership to understand semantic processing in sentence processing tasks?

* Present challenges:

* Category membership depends on the context

* Want to side-step human error in large-scale stimulus generation in studies of semantic processing

## Clustering Methodology

**Goal:** Cluster participants' sentence-final cloze responses to English and French preambles to great new stimuli

* Bayesian Gaussian mixture models produce optimal cluster assignments from data

* Prior work (Jacobs et al., 2025) was limited to English

* Learned clusters roughly approximate (syntactic/semantic) categories

### Processing steps

1. Encode sentence and response using a Transformer language model

2. Compute embedding of the response in context

3. Estimate clusters of responses

4. After estimation, assign cluster label to each response

<figure><img src="image_4.png" /><figcaption>### Study 1: Validating cluster assignment against Federmeier and Kutas (1999)</figcaption></figure>

* Clusters estimated over a large-scale English cloze corpus (Peelle et al., 2020) for 132 preambles

* "They wanted the hotel to seem like a tropical resort. So, along the avenue they planted rows of _______"

* Expected (E): palms

* Within-category (WC): pines

* Between-category (BC): tulips

* Fully correct means palms matches pines and neither matches tulips

<figure><img src="image_5.png" /><figcaption>### Study 2: Subjective cluster evaluation over Peelle et al. (2020) and Brunel et al. (2025)</figcaption></figure>

* Language-specific model computed clusters for each dataset

* Random sample of 50 sentences per language

* Expected: Top answer, top cluster

* Within-category: Second answer, top cluster

* Between category: Top answer, second cluster

* English and French clusters show high proportion of usable stimuli with similar structure to Federmeier and Kutas

* E/WC rated same-category 80% vs. E/BC at 32%

<table><caption>Table 1A. English sentences</caption><thead><tr><th rowspan="2">Sentence</th><th colspan="2">Within- Between-</th></tr><tr><th>Expected</th><th>Category</th><th>Category</th></tr></thead><tbody><tr><td>Cara went to the pet store to buy her new</td><td>cat</td><td>dog</td><td>parrot</td></tr><tr><td>From the mountaintop, Wyatt could see the whole</td><td>town</td><td>city</td><td>valley</td></tr><tr><td>Phil restarted the stalled lawn mower with a strong</td><td>pull</td><td>tug</td><td>hum</td></tr><tr><td>The baker said those lemons were especially</td><td>sour</td><td>tart</td><td>good</td></tr></tbody></table>

<table><caption>Table 1B. French sentences</caption><thead><tr><th rowspan="2">Sentence</th><th colspan="2">Within- Between-</th></tr><tr><th>Expected</th><th>Category</th><th>Category</th></tr></thead><tbody><tr><td>Le lapin s'est caché dans son L'origami est un art du</td><td>terrier<br>pliage</td><td>trou<br>design</td><td>chapeau<br>papier</td></tr><tr><td>Il est le chef de son gang, c'est le</td><td>boss</td><td>leader</td><td>meilleur</td></tr><tr><td>Le thé avait refroidi, il est devenu</td><td>froid</td><td>tiède</td><td>imbuvable</td></tr></tbody></table>

## Conclusions

* Quantitative method successfully re-constructs contextual within- and between-category differences (Study 1)

* Qualitative assessment of simple heuristics based on clusters shows good (but imperfect) performance (Study 2)

* Clustering provides additional means to study semantic prediction in sentence processing

* More work needed to validate cluster membership automatically – human evaluation is critical

## References

* Brunel, J., Dujardin, E., Delord, S., & Mathey, S. (2025). Emotional valence, cloze probability, and entropy: Completion norms for 403 French sentences. *Behavior Research Methods*, 57, 81.

* Federmeier, K. D., & Kutas, M. (1999). A rose by any other name: Long-term memory structure and sentence processing. *Journal of Memory and Language*, 41, 469-495.

* Jacobs, C. L., Hubbard, R. J., Grobol, L., & Federmeier, K. D. (2025). Uncovering patterns of semantic predictability in sentence processing. *Journal of Memory and Language*, 144, 1046-1053.

* Peelle, J. E., Miller, R. L., Rogers, C. S., Spehar, B., Sommers, M. S., & Van Engen, K. J. (2020). Completion norms for 3085 English sentence contexts. *Behavior Research Methods*, 52, 1795-1799.