<figure><img src="image_1.png" /></figure>

HAL
open science

# Spontaneous Speech Variables for Evaluating LLMs
## Cognitive Plausibility

Sheng-Fu Wang, Laurent Prevot, Jou-An Chi, Ri-Sheng Huang, Shu-Kai Hsieh

<figure><img src="image_2.png" /></figure>

► To cite this version:

Sheng-Fu Wang, Laurent Prevot, Jou-An Chi, Ri-Sheng Huang, Shu-Kai Hsieh. Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility. 2026. hal-05266194

HAL Id: hal-05266194

https://hal.science/hal-05266194v1

Preprint submitted on 17 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2505.16277v1 [cs.CL] 22 May 2025

# Spontaneous Speech Variables for Evaluating LLMs' Cognitive Plausibility

**Sheng-Fu Wang**

**Laurent Prévot**

Academia Sinica

CNRS & MEAE

Institute of Linguistics

CEFC

Taipei, Taiwan

Taipei, Taiwan

sftwang@gate.sinica.edu.tw

laurent.prevot@cnrs.fr

**Jou-An Chi**

**Ri-Sheng Huang**

**Shu-Kai Hsieh**

Graduate Institute of Linguistics

Department of CSIE

Graduate Institute of Linguistics

National Taiwan University

National Taiwan University

National Taiwan University

Taipei, Taiwan

Taipei, Taiwan

Taipei, Taiwan

r11142005@ntu.edu.tw

r13922102@csie.ntu.edu.tw

shukaihsieh@ntu.edu.tw

## Abstract

The achievements of *Large Language Models* in Natural Language Processing, especially for high-resource languages, call for a better understanding of their characteristics from a cognitive perspective. Researchers have attempted to evaluate artificial models by testing their ability to predict behavioral (e.g., eye-tracking fixations) and physiological (e.g., brain responses) variables during language processing (e.g., reading/listening). In this paper, we propose using spontaneous speech corpora to derive production variables (speech reductions, prosodic prominences) and applying them in a similar fashion. More precisely, we extract. We then test models trained with a standard procedure on different pretraining datasets (written, spoken, and mixed genres) for their ability to predict these two variables. Our results show that, after some fine-tuning, the models can predict these production variables well above baselines. We also observe that spoken genre training data provides more accurate predictions than written genres. These results contribute to the broader effort of using high-quality speech corpora as benchmarks for LLMs.

## 1 Introduction

The success of *Large Language Models* in Natural Language Processing, especially for high-resource languages, calls for a better understanding of their characteristics from a cognitive perspective. Yet, the difference in scale between the data size required to train LLMs and the amount of linguistic exposure needed for humans to achieve language proficiency calls for a better understanding of these models from a cognitive viewpoint. Researchers have attempted to evaluate artificial models by test-

ing their ability to predict behavioral (e.g., eye-tracking fixations) and (neuro-)physiological (e.g., brain responses) variables during language processing (e.g., reading/listening). In this paper, we propose using spontaneous speech corpora to derive production variables and applying them in a similar fashion. More precisely, we extract *speech reduction* and *prosodic prominences* from three corpora in different languages (English, French, and Mandarin). Speech reductions shortens words duration by dropping sounds or merging syllables. Prosodic prominence makes certain syllables or words stand out through prosodic marking.

The BABYLM initiative (Choshen et al., 2024; Hu et al., 2024; Warstadt et al., 2023b,a) has created an impulse for creating and evaluating cognitively plausible LLMs, both for the machine learning purpose of designing an efficient model with a small amount of training data (10-100M tokens) and for the cognitive science purpose of better understanding the similarities and differences between artificial and human language learning. An observation about the initiative to date is that the datasets used are all in English. Although this is a natural starting point, it represents a significant limitation. Expanding the scope to include more languages is not only about better representing linguistic communities or potential model users; it is also about achieving comparable, contrastive results across different languages, which should offer valuable insights into both the learning models and the underlying learning processes.

While gathering a 100M token conversational dataset based on real spoken conversational data might be challenging, a 10M token dataset is accessible for languages like English, French, Mandarin,
and some other languages studied from a corpus
linguistics perspective. Conversational speech is
the genre within which humans acquire their basic
language skills. It is a genre quite distant from
the usual written or web content on which LMs are
trained, increasing the risk of biases in the LMs pro-
duced. Moreover, it has been argued that this genre
is highly relevant to language emergence (Levin-
son, 2020; Christiansen and Chater, 2022). How
could a purely interactional dataset, including both
child-directed and general conversation transcripts,
be compared to written genres or more balanced
mixtures (Feng et al., 2024)?

In this context, current evaluation metrics, such
as the ones used in BABYLM, while a good start-
ing point, appear biased in two ways: they tend
to favor canonical written forms and to prioritize
syntactic, semantic, and commonsense pragmat-
ics. However, language and communicative com-
petence include many other dimensions. Although
the initiative clearly emphasizes the importance
of using speech transcripts—both child-directed
and everyday conversations—as training data, to
our knowledge, none of the evaluation metrics em-
ployed explicitly address the specificities of spon-
taneous speech.

To summarize, we argue that datasets and eval-
uation metrics are just as crucial as models for
understanding the computational learning of lan-
guage structure. More precisely, researchers have
attempted to evaluate artificial models by testing
their ability to predict behavioral (e.g., eye-tracking
fixations) (Hollenstein et al., 2021) and physiolog-
ical (e.g., brain responses) (Bingel et al., 2016;
Hollenstein et al., 2018; Pasquiou et al., 2022)
variables during language processing (e.g., read-
ing/listening). We propose using spontaneous
speech corpora to derive production variables and
applying them in a similar fashion. More precisely,
we extract *speech reduction* and *prosodic promi-
nences* from three corpora in different languages
(English, French, and Taiwan Mandarin). The three
languages were chosen because of the availability
of high-quality speech corpora (from which these
variables can be reliably extracted), the availabil-
ity of enough spontaneous speech transcripts (to
train the model), and to ensure the inclusion of lan-
guages from different typological families. We then
test models trained with a standard procedure on
different pretraining datasets (written, spoken, and
mixed genres) to assess their ability to predict these
two variables. Our results show that the models can,

after some fine-tuning, predict these production
variables well above baselines. We also observe
that spoken genre training data provides more accu-
rate predictions than written genres. These results
open the possibility of using high-quality speech
corpora as benchmarks for LLMs.

## 2 Related Work

Since the emergence of large language models,
there has been strong interest in the computational
linguistics community in understanding why they
are so successful. Warstadt et al. (2020b) explored
the conditions (e.g., the amount of training data)
under which ROBERTA develops and leverages
linguistic features, such as part of speech (POS)
and morphology, as opposed to relying on simpler
surface-level features like position-based or length-
based features. More recently, several studies have
probed LLMs to better characterize their perfor-
mance across various domains, particularly with
regard to their linguistic competence versus com-
monsense reasoning. These studies have also exam-
ined the relationship between model performance
and the amount of training data required for differ-
ent tasks. In particular, Zhang et al. (2021) used
training sets of varying sizes—1M, 10M, 100M,
and 1B tokens—to show that syntactic and seman-
tic competence become robust within the 10M-
100M range, whereas larger datasets are needed
to achieve strong results in pragmatic and common-
sense reasoning tasks.

More broadly, there have been proposals for eval-
uating the performance of LLMs on diverse lin-
guistic tasks. Warstadt et al. (2019b) leveraged
a substantial body of generative syntax-semantics
literature to develop benchmarks based on accept-
ability judgments, coming from the linguistic liter-
ature, like the COLA benchmark, which was fur-
ther extended by exploiting more sources and data
augmentation methods in BLIMP (Warstadt et al.,
2020a). In addition to these binary decision tasks,
Zhang et al. (2021) combined three other types of
evaluation metrics: *classifier probing* (following
(Ettinger et al., 2016; Adi et al., 2017)), which
includes tasks from POS tagging to coreference
resolution; *information-theoretic probing* based on
the minimum description length (MDL) principle;
and *fine-tuning on higher-level tasks* such as those
in the SUPERGLUE benchmark.

Most of the benchmarks have been proposed for
English. However, BLIMP Warstadt et al. (2019a)
has inspired a series of language-specific benchmarks, such as CLIMP for Mandarin Chinese (Xiang et al., 2021), as well as benchmarks for other languages like Japanese (Someya and Oseki, 2023), Dutch (Suijkerbuijk et al.), and Russian (Taktsheva et al., 2024). These are important additions to the evaluation landscape. While these benchmarks represent important extensions to the general evaluation framework, they all rely on syntax-semantics structures derived from introspection and textbook data, as will be discussed in the next section. In parallel with these efforts, monolingual language models have been developed using large amounts of data (Chang et al., 2024), as well as experiments involving varied data quantities (Micheli et al., 2020).

In another line of research, several studies have tested the ability of large language models (LLMs) to perform tasks inspired by cognitive science, particularly in the domains of semantics and pragmatics (Ettinger, 2020; Binz and Schulz, 2023).

Our approach of using actual speech data to extract production-based metrics can be related to studies that use behavioral or neurophysiological data linked with linguistic datasets. Specifically, there has been significant work focusing on textual datasets combined with eye-tracking (Hollenstein et al., 2021) or neurophysiological (Bingel et al., 2016; Hollenstein et al., 2018) measures. Additionally, datasets from passive listening tasks, linked to fMRI, have been released for various languages (e.g., French, Mandarin, and English) (Li et al., 2022). These datasets have been used, for instance, to study the impact of training parameters on a language model's ability to predict neurophysiological data (Pasquiou et al., 2022). Focusing on spontaneous speech, (Rauchbauer et al., 2019; Hmamouche et al., 2024) examined the predictability of fMRI-derived signals from conversational variables, including lexical information.

In terms of specialized language models, (Cabiddu et al., 2025) developed LMs based on child-directed speech transcripts and evaluated them on word-sense disambiguation tasks. They concluded that word acquisition trajectories could be better captured by multimodal models that incorporate acoustic features, among other aspects. Regarding tokenizers, Beinborn and Pinter (2023) proposed an evaluation paradigm focusing on the cognitive plausibility of subword tokenization. They compared BPE, WordPiece, and UnigramLM and revealed a lower “cognitive correlation” for the latter.

Furthermore, in the most recent BABYLM edition, (Martinez et al., 2023) introduced an interesting learning curriculum that constrained vocabulary in the early stages to simulate more cognitively plausible learning curves. Although this approach did not yield consistent overall results, marginal gains were observed in selected tasks.

Lastly, a recent audio-based question-answering benchmark for spoken language models incorporates spontaneous speech phenomena such as filler words, disfluencies, false starts, and corrections into different “speech style” conditions for testing the robustness of these models (Cui et al., 2025). Notably, the injection of speech phenomena into the benchmark is text-based and LLM-generated instead of being collected from natural speech.

## 3 A proposal for a new source of metrics

Most of the work mentioned above is grounded in text-based and/or handcrafted paradigms, potentially coupled with behavioral or physiological lab measures. In contrast, we propose using actual spontaneous conversational transcripts to build complementary benchmarks that test not only syntactic-semantic dimensions but also real-world language use. These metrics will remain fundamentally linguistic in nature rather than focusing on task-specific or end-to-end evaluation.

Language is acquired, especially in its early stages, within spontaneous, conversational environments. While conversational language shares grammatical structures with other genres, its unique characteristics suggest that simply listing syntactic “errors” or semantic incongruities does not fully capture linguistic competence. Moreover, in a conversational context, what may be considered a production error from a formal grammatical perspective is often perfectly acceptable and successfully achieves its communicative purpose. Therefore, we aim to develop a complementary approach that provides a broader set of metrics to evaluate language models from both cognitive and communicative perspectives along with existing benchmarks.

Specifically, we propose using spontaneous speech corpora, as they offer insights into human language processing through various observable production phenomena. Our approach is a kind of classifier probing (Ettinger et al., 2016; Adi et al., 2017; Warstadt et al., 2019b), but rather than focusing on metalinguistic tasks (e.g., predicting syntactic categories), we aim to predict phenomena
that serve as partial indicators of language processing. We propose two preliminary metrics: *speech reductions* and *prosodic prominences*, which are grounded in spontaneous speech production, and each have been the subject of extensive research.

## 3.1 Speech reductions

Speech reductions have been studied across a range of linguistic levels, especially when considering the issue of signal information density. In spontaneous speech, some chunks of speech are produced in a reduced manner, both in terms of duration and articulatory amplitude. The location of these reductions is not random. For example, studies have suggested that speakers tend to smooth the information density of their speech signal over time, with reductions serving as a mechanism to achieve this smoothing effect (Aylett and Turk, 2004).

The relationship between information density and speech reduction has led to research developments on this topic with various approaches. These approaches may differ in the probabilistic measures used to predict reductions, such as lexical frequency, contextual probability, and informativity (Aylett and Turk, 2004; Gahl, 2008; Cohen Priva, 2012; Seyfarth, 2014). They also differ in terms of the linguistic level at which reductions occur, whether at the phoneme-, syllable-, word-level, or in terms of overall speech rate. Many of these studies include and compare different types of probabilistic measurements (e.g., lexical frequency and contextual probability) within a single study (e.g., (Seyfarth, 2014; Cohen Priva and Jaeger, 2018)) and some of them also compare probabilistic measurements calculated at different linguistic levels (e.g., segment- and syllable-levels in Van Son et al. (1999), segment- and word-level measurements in Van Son and Pols (2003), syllable- and word-level measurements in Wang (2022)). Inclusion and comparison of reductions or phonetic variability across various linguistic levels in the same study have also been done (e.g., for individual segments and prefixes as a whole in Pluymaekers et al. (2005); morphemes and words in Tang and Bennett (2018)), albeit less frequently.

These studies show that phonetic reduction can be predicted to varying degrees based on the statistical distribution of linguistic units, and the prediction has been repeatedly found with different types of measurements at various levels of linguistic units. This motivates the development of a reduction-labeling task for evaluating language

models.

## 3.2 Prosodic Prominences

Prosodic prominence refers to the emphasis placed on certain units, often demarcated at the level of words or syllables, within a spoken utterance. This emphasis can be measured through (and perceived based on) acoustic cues such as movements in fundamental frequencies, duration, intensity, and segmental properties such as the formant structure of vowels. Recent work by Wolf et al. (2023) has shown a significant degree of redundancy between the representations encoded from tokens alone and those derived from acoustic-prosodic information. Acoustic-prosodic features such as word-level energy, fundamental frequency, duration, pause, and composite measurements derived using a wavelet-based algorithm (Suni et al., 2017) were used to quantify this redundancy. Their findings suggest that prosodic information can be predicted, to some extent, from the word itself and its surrounding context. There are also studies showing that language models (e.g., BERT) can predict prosodically-prominent tokens based from speech transcripts (Kakouros and O'Mahony, 2023; Talman et al., 2019). These findings suggest that part of prosodic prominence can indeed be predicted by a language model, even though predicting based on text alone should never be sufficient (as also noted by Wolf et al. (2023)).

# 4 Method

## 4.1 Pre-training LLMs

We pretrained multiple RoBERTa models (an architecture choice motivated in part by a more direct comparison of our topline choice, XLM-RoBERTa, which has a more manageable size for fine-tuning) using three corpora in different languages: English, French, and Taiwan Mandarin. For each language, the pretraining data is categorized into three distinct types: conversational, written, and a mix of these two genres. For English, the spoken data was drawn from BNC and Switchboard, and the written data was drawn from Simple Wikipedia (both extracted from the BABYLM 100M dataset). The mixed data is a 9M-token subset of the BABYLM 10M training data, which contains CHILDES and Gutenberg collections in addition to the three aforementioned sources. The 1M-token dev sets were built from the corresponding validation sets in the BABYLM data.
For French, the conversational data consists of 10 million tokens from conversational datasets specifically ORFEO (Benzitoun et al., 2016) and CHILDES-FR (MacWhinney, 2014; Rose and MacWhinney, 2014), and the conversational data contains an additional 10 million tokens sourced from Wikipedia.

For Taiwan Mandarin, the 10M-token conversational data was derived from the NCCU Spoken Corpus of Taiwan Mandarin (Chui and Lai, 2008), Taiwan Corpus of Child Mandarin (TCCM)<fnref n="1" /> (Chang et al., 2011), Open Subtitles (Lison and Tiedemann, 2016), and the transcript of Taiwan Legislative Yuan <fnref n="2" />. The written data was from a subset of Traditional Chinese Wikipedia <fnref n="3" />. For French and Mandarin, the mixed data was built from a 50-50 mix of the conversational and Wiki data, and the prepared datasets for all three types underwent a 90-10 split to build the training and development sets.

We employed the SentencePiece tokenizer (Kudo, 2018a) with the Unigram model (Kudo, 2018b). SentencePiece allows tokenization without requiring predefined word boundaries, making it particularly suitable for languages with non-segmented scripts, such as Taiwan Mandarin. Additionally, we applied a minimum token frequency threshold of 2. The vocabulary size was set to 10,000 tokens, which was determined empirically based on preliminary experiments. Each model was initialized with XLM-RoBERTa as the pre-trained base model and trained for 100 epochs using a batch size of 32. The English, French, and Mandarin models were pretrained with a learning rate of 1e-4, 1e-4, and 2e-4 respectively, with early stopping if validation loss failed to decrease in 10 epochs.

## 4.2 Benchmarks

For these experiments, we used three sources to build benchmarks: the Buckeye Corpus for English<fnref n="4" /> (Pitt et al., 2005), the Corpus of Interactional Data (CID) for French<fnref n="5" /> (Blache et al., 2017), and The Sinica Mandarin Conversational Dialogue Corpus (Sinica MCDC8)<fnref n="6" /> for Mandarin (Tseng, 2013). CID is an 8-hour corpus of 1-hour conversations

between friends (16 speakers). It features fiercely spontaneous conversational speech. Buckeye is a corpus with 38.1 hours of spontaneous speech (40 speakers) recorded in an interview format. MCDC8 features 8 one-hour natural conversations between recruited participants (16 speakers) who did not know their interlocutor in advance. The main reason for the choice of these corpora is the high quality of their speech transcript alignment, down to the syllable or even the segment level.

### 4.2.1 Speech Reduction

There are several methods to determine whether a portion of speech is reduced. Following approaches in the literature, we first derived ratios of every word token's actual duration and its expected duration. For the French and Mandarin benchmarks, we used annotations of syllable boundaries in the corpus and developed a model that predicts syllable duration based on the segment it contains, similar to Wang (2022). A model is trained on one-half of the corpus and then applied to estimate the expected token duration in the remaining half of the corpus. For the English benchmark, we followed the literature (Bell et al., 2009; Gahl et al., 2012; Seyfarth, 2014) and made use of the available segmental duration in the corpus and calculated a word' expected duration. The corpus was similarly divided in halves with expected duration in one half calculated with segment duration from the other.

In both cases, we then converted the ratios into binary labels by applying a threshold of 0.5 (i.e., a reduction of at least 50%) for English and French, and a threshold of 0.6 (reduction of at least 40%) for Mandarin. The threshold was adjusted for Mandarin to keep the proportion of reduced tokens more similar to other two languages, as Mandarin's syllable-timed nature make substantial reduction a rare event. These thresholds resulted in 13.58%, 17.54%, and 11.09% of the tokens in English, French, and Mandarin being labeled as reduced. These labels were then encoded in BIO format.

### 4.2.2 Prosodic Prominences

To detect prosodically prominent tokens we used Suni et al.'s (2017) method based on wavelet method that combines acoustic features (fundamental frequency, energy, duration) to determine prominence at the token level. One of the reasons for this tool choice is that it has already been used in the LM literature (Wolf et al., 2023) to

<footnote n="1">¹https://lope.linguistics.ntu.edu.tw/tccm/</footnote>

<footnote n="2">²https://www.parliamentarytv.org.tw/</footnote>

<footnote n="3">³https://huggingface.co/datasets/zetavg/zh-tw-wikipedia/</footnote>

<footnote n="4">⁴https://buckeyecorpus.osu.edu/</footnote>

<footnote n="5">⁵https://hdl.handle.net/11403/sldr000720</footnote>

<footnote n="6">⁶https://www.aclclp.org.tw/use Mat.php#mcdc</footnote>
quantify the amount of redundancy between textual and prosodic levels. We applied the default configuration of this tool and set a threshold score of 1.25 (See figure 8 in Appendix A for details of the score distribution), which yielded 14%, 13.79%, and 12.77% of prosodic prominent tokens in English, French, and Mandarin.

## 4.3 Fine-tuning Experiments

The experiments evaluated different pre-trained models for our set of tasks.<fnref n="7" /> More precisely, we fine-tuned the pretrained models separately on a token classification task to predict which tokens were labeled (reduced / prominent) and which were not. A simple cross-validation was conducted across groups of speakers to maximize diversity across the folds. The models were fine-tuned for 10 epochs, with the validation fold being used for early stopping (patience = 5). For each model-task combination, the fine-tuning experiment was run for 10 iterations, i.e., five learning rates (2e-5, 4e-5, 6e-5, 8e-5, 1e-4) × two batch sizes (32, 16), with the combination that resulted in the highest averaged f1 score across 8 folds being used in the analysis.

Finally, we include two conditions using fine-tuning XLM-ROBERTA BASE and LARGE. This condition is not fair with the other models, since it is pretrained on much larger data set. It was originally included both as a topline and to assess the feasability of the task. For more efficient fine-tuning, these models underwent vocabulary pruning (Yang et al., 2022) on the training texts for our three languages.

# 5 Results

The main results are presented in Figure 1. (see more detailed results in Appendix B). For reduction, the results indicate that models comfortably outperform the label distribution baselines (dotted lines), especially for English and French. Between models, in all three languages, the Wikipedia model is outperformed by at least the conversational model. Furthermore, the topline RoBERTa models do not exhibit obvious advantages over the smaller model. It is also worth noting that models perform much worse in Mandarin than in the other two languages.

As for prominence, the performance is more comparable across the three languages. There is

also a similar advantage for conversational and mixed models over the Wikipedia model. Contrary to the reduction task, the topline ROBERTA models exhibit an advantage over the smaller models.

## 5.1 Perplexity and benchmarking

As language models' capacity is often evaluated on their perplexity on a given corpus, we calculated per-token perplexity (handling multi-token words with the masking method in (Kauf and Ivanova, 2023)) for each language-model combination on each fold in the benchmarks, shown in Figure 2. For the smaller models, three languages exhibit distinct patterns when it comes to the difference between models: For English, the perplexity scores were very similar, while for French, the conversational model outperforms the other models in having the lowest perplexity, followed by the mixed model and the Wiki model. For Mandarin, the pattern is reversed with the Wiki model having the lowest perplexity. In other words, for English and especially Mandarin, there is a divergence of perplexity scores between the benchmark and the fine-tuning experiment.

We also examined the correlation between token-level surprisal and gold labels, an analysis similar to the aforementioned studies that examine whether surprisal/perplexity predicts behavioral measurements such as reading time. The results are shown in Figure 3. Based on observations from phonetic research, perplexity's correlation is expected to be negative for both reduction and prominence (Aylett and Turk, 2004; Seyfarth, 2014). For prominence, the correlation is more positive for conversational and mixed models as observed in the fine-tuning experiments, thereby suggesting that models' fine-tuning performance on categorical prominence prediction is partially influenced by the behavior of the base models.

For reduction, the connection is less established: The expected negative correlation between perplexity and reduction is only borne out for conversational and mixed models in English, even though the relative difference between Wiki and conversational/mixed models in French is similar. Notably, the Wiki model is the only small model that showed the expected correlation in Mandarin. As these patterns are not perfect matches of the fine-tuning results, it suggests that at least for reduction, the fine-tuning task is not as trivial as something that can be predicted from base models' behavior.

<footnote n="7">⁷Notebooks for pretraining LMs and performing the experiment can be accessed at ***.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 1: F-scores on the benchmarks as a function of model (x), task and language (prominence). Model comparisons are based on Bayesian regression analyses where MODEL is the fixed predictor and FOLD is a random intercept. The models were run with weak (uniform) priors using the *brms* package in R, with post hoc hypothesis testing focused on comparing three small models (Wiki, conversational, mixed). Stars in the figure indicate a one-sided hypothesis with a posterior probability above 95%. Dotted lines correspond to random baselines.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 2: Perplexity on the benchmarks</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 3: The correlation between perplexity and labels.</figcaption></figure>

These observations, summarized in terms of the winner of the comparison between the conversational and Wiki models in Table 1, paint a more nuanced picture of the relationship between fine-tuning performance, perplexity, and how perplexity predicts reduction and prominence. They also add to the previous discussions on how and whether better language models in perplexity do not guarantee better cognitive modeling (Kuribayashi et al., 2021).

## 5.2 Error Analysis

To better understand model errors in predicting prominence and reduction, we calculated the most over-predicted and under-predicted words for each language based on the difference between the models' predictions and gold label rates, as shown in Appendix C. The variances are very low across models, showing that models with different pre-training data have similar tendencies in over- and under-predictions.
<table><caption>Table 1: Winners of the comparison between the conversational and Wiki models in different criteria: higher F1 in fine-tuning (FT); lower perplexity (ppl), lower perplexity-label correlation (ppl-label cor.) for reduction and higher correlation for prominence.</caption><thead><tr><th>lge</th><th>task</th><th>FT</th><th>ppl</th><th>ppl-label cor.</th></tr></thead><tbody><tr><td>Eng.</td><td>both</td><td>conv</td><td>n.s.</td><td>conv</td></tr><tr><td>Fre.</td><td>both</td><td>conv</td><td>conv</td><td>conv</td></tr><tr><td>Man.</td><td>reduc</td><td>conv</td><td>wiki</td><td>wiki</td></tr><tr><td>Man.</td><td>prom</td><td>conv</td><td>wiki</td><td>conv</td></tr></tbody></table>

Across languages, the most over-predicted words for reduction are function words and discourse markers. As for prominence, over-predicted words include discourse markers and content words directly linked to the topic of the discourse. Under-predicted reductions are mostly content words and less commonly reduced expressions. For prominence, under-predicted words also tend to be content words. We refer the readers to Appendix C for examples of these words.

A general explanation may be that models have learned that some words are intrinsically more likely to be reduced (Bodur et al., 2023) or to be prominent, and therefore biased towards the more common class. The tendency is likely related to lexical frequency and distribution of words within the discourse structure. It is worth noting that even though over- and under-predictions are observed, the overall correlation between frequency and label assignments did not differ greatly between model predictions and gold labels (Figures 10 and 11 in the Appendix D).

# 6 Potential shortcomings and Limitations

**Information-centric nature.** One potential limitation is that the models may only capture the information-theoretic contribution to our tasks, i.e., the well-noted relationship between information-theoretic notions such as information density, entropy, and predictability and phonetic realizations. However, the prediction of these phenomena cannot be reduced to information-theoretic explanations alone, and our analyses do show some divergences between fine-tuning experiments and direct correlations between perplexity and benchmark labels. As each metric introduces its own set of subtleties related to language processing, our goal is to evaluate LLMs in terms of their ability to grasp these subtleties.

**Text-only.** The proposed phenomena for

probing the models are inherently related to speech processing, which goes beyond what can be achieved with a text-only approach. Beyond the acoustic modality, the visual channel also plays a role. Our goal in proposing these metrics is not to achieve state-of-the-art performance in predicting these phenomena. Rather, we aim to treat them as 'traces' of human language processing visible at the surface level, and to test which models are better at predicting these traces from text-only input.

**Surface level shortcuts.** A concern related to the previous point is the risk that models may rely on surface-level elements as shortcuts to predict the target variables. Addressing such a concern is inherently challenging, as speech phenomena are deeply intertwined with observable surface patterns. Nonetheless, we believe it is still worth pursuing this line of investigation, which can be extended particularly through controlled evaluation sets, similar to the approach in McCoy et al. (2019), who compared the models with some heuristics to isolate the influence of surface-level cues. The types of cross-lingual phenomena in our proposed benchmarks also invite future work that develops principled methodologies in evaluating the 'cognitive' process inside a language model.

**Triviality of the main result.** From a machine-learning perspective, it might be seen as a trivial result that models trained on data similar to test sets perform better than models trained on other types of data. First of all, it is worth emphasizing that pretraining datasets and benchmarks in our experiments are completely independent as they do not come from the same raw corpora. Also, the pre-training datasets and corpora for building benchmarks have been curated by different teams and transcribed following different conventions. Nevertheless, we cannot deny that the conversational datasets are by all aspects (sentence length distribution, lexical frequencies, etc) more similar to benchmarks than Wikipedia datasets are.

As trivial as it seems, it may be one of our main points: to produce models more closely related to human cognition, one should use data sets made of spontaneous speech (and not generic textual / web content). Furthermore, the additional analyses on perplexity and how perplexity directly correlates with labels in benchmarks show that the advantage of conversational models in these conversational benchmarks is not guaranteed in all potential ways of 'evaluating' these models.
# 7 Conclusion

In this paper, we propose advancing the benchmarking of language models to include spontaneous speech phenomena and to extend beyond English. Motivated by the BABYLM initiative, we evaluated models trained on developmentally plausible amounts of data and showed how models trained in different genres of texts perform differently.

In the future, it will be crucial to explore more nuanced variations in training data, such as balancing conversational speech, child-directed speech, and simple texts as well as developing complementary evaluation metrics.

From a broader perspective, we hope to show that benchmarks like BLIMP that require a significant amount of expert and naive human input to build, can be complemented, as we have explored in this paper, with benchmarks derived from existing high-quality linguistic corpora, without additional human efforts.

# Acknowledgments

We would like to thank Shu-Chuan Tseng and Pierre Magistry for discussions in relation to this paper. This study was supported by Taiwan's National Science and Technology Council (NSTC-112-2410-H-001-098-MY2).

# References

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In *Proceedings of ICLR Conference Track*, Toulon, France.

Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. *Language and speech*, 47(1):31–56.

Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In *The 2023 Conference on Empirical Methods in Natural Language Processing*.

Alan Bell, Jason M Brenier, Michelle Gregory, Cynthia Girand, and Dan Jurafsky. 2009. Predictability effects on durations of content and function words in conversational English. *Journal of Memory and Language*, 60(1):92–111.

Christophe Benzitoun, Jeanne-Marie Debaisieux, and Henri-José Deulofeu. 2016. Le projet orféo: un corpus d'étude pour le français contemporain. *Corpus*, (15).

Joachim Bingel, Maria Barrett, and Anders Søgaard. 2016. Extracting token-level signals of syntactic processing from fmri-with an application to pos induction. In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 747–755.

Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. *Proceedings of the National Academy of Sciences*, 120(6):e2218523120.

Philippe Blache, Roxane Bertrand, Gaëlle Ferré, Berthille Pallaud, Laurent Prévot, and Stéphane Rauzy. 2017. The corpus of interactional data: A large multimodal annotated resource. *Handbook of linguistic annotation*, pages 1323–1356.

Kübra Bodur, Corinne Fredouille, Stéphane Rauzy, and Christine Meunier. 2023. Exploring the nuances of reduction in conversational speech: Lexicalized and non lexicalized reductions. *PsyArXiv*.

Francesco Cabiddu, Mitja Nikolaus, and Abdellah Fourtassi. 2025. Comparing children and large language models in word sense disambiguation: Insights and challenges. *Language Development Research*, 5(1).

Lih-Huei Chang, Zhong-Ru Chang, Yan-Chang Ke, and Su-Hui Xiao. 2011. Taiwan child language corpus (tccm). National Science Council Research Project, NSC96-2420-H-002-030. Available at http://lope.linguistics.ntu.edu.tw/tccm/.

Tyler A Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K Bergen. 2024. Goldfish: Monolingual language models for 350 languages. *arXiv preprint arXiv:2408.10441*.

Leshem Choshen, Ryan Cotterell, Michael Y Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. [call for papers] the 2nd babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. *arXiv preprint arXiv:2404.06214*.

Morten H Christiansen and Nick Chater. 2022. *The language game: How improvisation created language and changed the world*. Random House.

Kawai Chui and Huei-ling Lai. 2008. The nccu corpus of spoken chinese: Mandarin, hakka, and southern min. *Taiwan Journal of Linguistics*, 6(2).

Uriel Cohen Priva. 2012. *Sign and signal: Deriving linguistic generalizations from information utility*. Ph.D. thesis, Stanford University.

Uriel Cohen Priva and T Florian Jaeger. 2018. The interdependence of frequency, predictability, and informativity in the segmental domain. *Linguistics Vanguard*, 4(s2).

Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. 2025. Voxeval: Benchmarking the knowledge understanding capabilities of end-to-end spoken language models. *arXiv preprint arXiv:2501.04962*.
Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. *Transactions of the Association for Computational Linguistics*.

Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In *Proceedings of the 1st workshop on evaluating vector-space representations for nlp*, pages 134–139.

Steven Feng, Noah Goodman, and Michael C Frank. 2024. Is child-directed speech effective training data for language models? In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 22055–22071.

Susanne Gahl. 2008. Time and thyme are not homo-phones: The effect of lemma frequency on word du-rations in spontaneous speech. *Language*, 84(3):474–496.

Susanne Gahl, Yao Yao, and Keith Johnson. 2012. Why reduce? phonological neighborhood density and pho-netic reduction in spontaneous speech. *Journal of memory and language*, 66(4):789–806.

Youssef Hmamouche, Magalie Ochs, Laurent Prévot, and Thierry Chaminade. 2024. Interpretable predic-tion of brain activity during conversations from multi-modal behavioral signals. *Plos one*, 19(3):e0284342.

Nora Hollenstein, Emmanuele Chersoni, Cassandra L Jacobs, Yohei Oseki, Laurent Prévot, and Enrico San-tus. 2021. Cmcl 2021 shared task on eye-tracking prediction. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 72–78.

Nora Hollenstein, Jonathan Rotsztejn, Marius Troen-dle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading. *Scientific data*, 5(1):1–13.

Michael Y Hu, Aaron Mueller, Candace Ross, Ad-ina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the sec-ond babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. *arXiv preprint arXiv:2412.05149*.

Sofoklis Kakouros and Johannah O'Mahony. 2023. What does bert learn about prosody? In *20th International Congress of Phonetic Sciences ICPhS*. International Phonetics Association.

Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pages 925–935.

T Kudo. 2018a. Sentencepiece: A simple and lan-guage independent subword tokenizer and detok-enizer for neural text processing. *arXiv preprint arXiv:1808.06226*.

Taku Kudo. 2018b. Subword regularization: Improving neural network translation models with multiple sub-word candidates. *arXiv preprint arXiv:1804.10959*.

Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing*, pages 5203–5217. Association for Computational Linguistics.

Stephen C Levinson. 2020. On the human "interaction engine". In *Roots of human sociality*, pages 39–69. Routledge.

Jixing Li, Shohini Bhattasali, Shulin Zhang, Berta Franzluebbers, Wen-Ming Luh, R Nathan Spreng, Jonathan R Brennan, Yiming Yang, Christophe Pallier, and John Hale. 2022. Le petit prince multilingual naturalistic fmri corpus. *Scientific data*, 9(1):530.

Pierre Lison and Jörg Tiedemann. 2016. Opensub-titles2016: Extracting large parallel corpora from movie and tv subtitles.

Brian MacWhinney. 2014. *The CHILDES project: Tools for analyzing talk, Volume 1: Transcription format and programs*. Psychology Press.

Richard Diehl Martinez, Hope McGovern, Zebulon Goriely, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn. 2023. Climb-curriculum learning for infant-inspired model building. In *Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning*, pages 112–127.

Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 3428–3448, Florence, Italy. Association for Computational Linguistics.

Vincent Micheli, Martin d'Hoffschmidt, and François Fleuret. 2020. On the importance of pre-training data volume for compact language models. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 7853–7858, Online. Association for Computational Linguistics.

Alexandre Pasquiou, Yair Lakretz, John T Hale, Bertrand Thirion, and Christophe Pallier. 2022. Neural language models are not born equal to fit brain data, but training helps. In *International Conference on Machine Learning*, pages 17499–17516. PMLR.
Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005. The buck-eye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. *Speech Communication*, 45(1):89–95.

Mark Pluymaekers, Mirjam Ernestus, and R Harald Baayen. 2005. Lexical frequency and acoustic reduction in spoken dutch. *The Journal of the Acoustical Society of America*, 118(4):2561–2569.

Birgit Rauchbauer, Bruno Nazarian, Morgane Bourhis, Magalie Ochs, Laurent Prévot, and Thierry Chaminade. 2019. Brain activity during reciprocal social interaction investigated using conversational robots as control condition. *Philosophical Transactions of the Royal Society B*, 374(1771):20180033.

Yvan Rose and Brian MacWhinney. 2014. The phon-bank project: Data and software-assisted methods for the study of phonology and phonological development.

Scott Seyfarth. 2014. Word informativity influences acoustic duration: Effects of contextual predictability on lexical representation. *Cognition*, 133(1):140–155.

Taiga Someya and Yohei Oseki. 2023. Jblimp: Japanese benchmark of linguistic minimal pairs. In *Findings of the Association for Computational Linguistics: EACL 2023*, pages 1581–1594.

Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L Frank. Blimp-nl.

Antti Suni, Juraj Šimko, Daniel Aalto, and Martti Vainio. 2017. Hierarchical representation and estimation of prosody using continuous wavelet transform. *Computer Speech & Language*, 45:123–136.

Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, and Ekaterina Artemova. 2024. Rublimp: Russian benchmark of linguistic minimal pairs. *arXiv preprint arXiv:2406.19232*.

Aarne Talman, Antti Suni, Hande Celikkanat, Sofoklis Kakouros, Jörg Tiedemann, and Martti Vainio. 2019. Predicting prosodic prominence from text with pre-trained contextualized word representations. In *Proceedings of the 22nd Nordic Conference on Computational Linguistics*, pages 281–290.

Kevin Tang and Ryan Bennett. 2018. Contextual predictability influences word and morpheme duration in a morphologically complex language (kaqchikel mayan). *The Journal of the Acoustical Society of America*, 144(2):997–1017.

Shu-Chuan Tseng. 2013. Lexical coverage in taiwan mandarin conversation. In *International Journal of Computational Linguistics & Chinese Language Processing, Volume 18, Number 1, March 2013*.

RJMH Van Son and Louis CW Pols. 2003. How efficient is speech. In *Proceedings of the institute of phonetic sciences*, volume 25, pages 171–184.

RJMH Van Son, Louis CW Pols, et al. 1999. Effects of stress and lexical structure on speech efficiency. In *EUROSPEECH*.

Sheng-Fu Wang. 2022. The interaction between predictability and pre-boundary lengthening on syllable duration in taiwan southern min. *Phonetica*, 79(4):315–352.

Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023a. Call for papers–the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. *arXiv preprint arXiv:2301.11796*.

Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. 2023b. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In *Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning*, pages 1–34.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng Fu Wang, and Samuel R. Bowman. 2019a. Blimp: The benchmark of linguistic minimal pairs for english. *Transactions of the Association for Computational Linguistics*.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020a. Blimp: The benchmark of linguistic minimal pairs for english. *Transactions of the Association for Computational Linguistics*, 8:377–392.

Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019b. Neural network acceptability judgments. *Transactions of the Association for Computational Linguistics*, 7:625–641.

Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 217–235, Online. Association for Computational Linguistics.

Lukas Wolf, Tiago Pimentel, Evelina Fedorenko, Ryan Cotterell, Alex Warstadt, Ethan Wilcox, and Tamar Regev. 2023. Quantifying the redundancy between prosody and text. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 9765–9784, Singapore. Association for Computational Linguistics.

Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In *Proceedings*
of the 16th Conference of the European Chapter of
the Association for Computational Linguistics: Main
Volume, pages 2784–2790, Online. Association for
Computational Linguistics.

Ziqing Yang, Yiming Cui, and Zhigang Chen. 2022.
TextPruner: A model pruning toolkit for pre-trained
language models. In Proceedings of the 60th An-
nual Meeting of the Association for Computational
Linguistics: System Demonstrations, pages 35–43,
Dublin, Ireland. Association for Computational Lin-
guistics.

Yian Zhang, Alex Warstadt, Xiaocheng Li, and
Samuel R. Bowman. 2021. When do you need bil-
lions of words of pretraining data? In Proceedings
of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing
(Volume 1: Long Papers), pages 1112–1125, Online.
Association for Computational Linguistics.
A Distribution of benchmark labels

<figure><img src="image_7.png" /><figcaption>Figure 4: Distribution reduction ratios as calculated in the English Dataset and the threshold selected.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 7: Distribution of prominence scores as calculated in the English Dataset and the threshold selected.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 5: Distribution reduction ratios as calculated in the French Dataset and the threshold selected.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: Distribution of prominence scores as calculated in the French Dataset and the threshold selected.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 6: Distribution of reduction ratios as calculated in the Mandarin Dataset and the threshold selected.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 9: Distribution of prominence scores as calculated in the Mandarin Dataset and the threshold selected.</figcaption></figure>
## B Complete scores

<table><caption>Table 2: Full results on the proposed speech-based benchmarks</caption><thead><tr><th>Language</th><th>Task</th><th>Model</th><th>F1</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><td rowspan="10">English</td><td rowspan="5">reduction</td><td>wiki</td><td>.433 (.020)</td><td>.451 (.038)</td><td>.422 (.052)</td></tr><tr><td>conv</td><td>.442 (.014)</td><td>.465 (.023)</td><td>.426 (.041)</td></tr><tr><td>mixed</td><td>.436 (.019)</td><td>.448 (.025)</td><td>.429 (.048)</td></tr><tr><td>xlm-r-base</td><td>.452 (.013)</td><td>.458 (.029)</td><td>.451 (.041)</td></tr><tr><td>xlm-r-large</td><td>.448 (.013)</td><td>.433 (.033)</td><td>.473 (.055)</td></tr><tr><td rowspan="5">prominence</td><td>wiki</td><td>.378 (.028)</td><td>.407 (.035)</td><td>.356 (.044)</td></tr><tr><td>conv</td><td>.394 (.027)</td><td>.428 (.027)</td><td>.366 (.041)</td></tr><tr><td>mixed</td><td>.390 (.025)</td><td>.414 (.040)</td><td>.372 (.035)</td></tr><tr><td>xlm-r-base</td><td>.414 (.018)</td><td>.463 (.046)</td><td>.378 (.032)</td></tr><tr><td>xlm-r-large</td><td>.422 (.024)</td><td>.447 (.039)</td><td>.405 (.051)</td></tr><tr><td rowspan="10">French</td><td rowspan="5">reduction</td><td>wiki</td><td>.441 (.034)</td><td>.472 (.051)</td><td>.415 (.031)</td></tr><tr><td>conv</td><td>.455 (.023)</td><td>.488 (.036)</td><td>.429 (.038)</td></tr><tr><td>mixed</td><td>.456 (.024)</td><td>.498 (.038)</td><td>.424 (.035)</td></tr><tr><td>xlm-r-base</td><td>.460 (.027)</td><td>.491 (.047)</td><td>.437 (.042)</td></tr><tr><td>xlm-r-large</td><td>.454 (.028)</td><td>.473 (.036)</td><td>.438 (.030)</td></tr><tr><td rowspan="5">prominence</td><td>wiki</td><td>.416 (.015)</td><td>.415 (.039)</td><td>.422 (.032)</td></tr><tr><td>conv</td><td>.443 (.017)</td><td>.453 (.035)</td><td>.436 (.032)</td></tr><tr><td>mixed</td><td>.445 (.031)</td><td>.456 (.030)</td><td>.435 (.045)</td></tr><tr><td>xlm-r-base</td><td>.474 (.030)</td><td>.478 (.037)</td><td>.472 (.038)</td></tr><tr><td>xlm-r-large</td><td>.464 (.020)</td><td>.463 (.036)</td><td>.468 (.029)</td></tr><tr><td rowspan="10">Mandarin</td><td rowspan="5">reduction</td><td>wiki</td><td>.169 (.016)</td><td>.154 (.023)</td><td>.195 (.038)</td></tr><tr><td>conv</td><td>.241 (.024)</td><td>.248 (.044)</td><td>.239 (.033)</td></tr><tr><td>mixed</td><td>.230 (.029)</td><td>.251 (.051)</td><td>.220 (.042)</td></tr><tr><td>xlm-r-base</td><td>.192 (.031)</td><td>.179 (.036)</td><td>.215 (.045)</td></tr><tr><td>xlm-r-large</td><td>.180 (.026)</td><td>.174 (.034)</td><td>.194 (.043)</td></tr><tr><td rowspan="5">prominence</td><td>wiki</td><td>.365 (.019)</td><td>.345 (.023)</td><td>.390 (.033)</td></tr><tr><td>conv</td><td>.401 (.008)</td><td>.398 (.018)</td><td>.405 (.025)</td></tr><tr><td>mixed</td><td>.395 (.008)</td><td>.379 (.019)</td><td>.413 (.015)</td></tr><tr><td>xlm-r-base</td><td>.444 (.020)</td><td>.447 (.029)</td><td>.444 (.037)</td></tr><tr><td>xlm-r-large</td><td>.426 (.031)</td><td>.429 (.037)</td><td>.425 (.036)</td></tr></tbody></table>
## C Word-level error analysis

For each language and task, we measured the most over-predicted and the most under-predicted words by rate difference, which is the difference between the model prediction's true rate and the ground truth's true rate. Top five and bottom five words ranked by rate difference are listed below. Since a single word may be split into multiple tokens, we determine a word's prediction based on majority voting among its tokens. In the case of a tie, the word is classified as "true" (reduced/prominent). We only include words that appear at least 50 times.

<table><caption>Table 3: The most over-predicted and under-predicted words for each task. For each pretraining scheme (wiki, conversational, mixed, ROBERTA-base and ROBERTA-large), we selected the best-performing model, and calculated every word's rate difference. Only the top-5 and bottom-5 words in rate difference are listed, along with the mean across the 5 models (and the standard deviation in parentheses).</caption><thead><tr><th>Language</th><th>Task</th><th>Top-5</th><th>Rate difference</th><th>Bottom-5</th><th>Rate difference</th></tr></thead><tbody><tr><td rowspan="10">English</td><td rowspan="5">reduction</td><td>youre</td><td>.284 (.021)</td><td>u<sup>1</sup></td><td>-.255 (.033)</td></tr><tr><td>dont</td><td>.219 (.033)</td><td>everybody</td><td>-.232 (.070)</td></tr><tr><td>theyre</td><td>.209 (.063)</td><td>only</td><td>-.232 (.111)</td></tr><tr><td>a</td><td>.179 (.008)</td><td>ill</td><td>-.226 (.057)</td></tr><tr><td>at</td><td>.170 (.022)</td><td>many</td><td>-.209 (.055)</td></tr><tr><td rowspan="5">prominence</td><td>basically</td><td>.215 (.082)</td><td>vote</td><td>-.225 (.023)</td></tr><tr><td>obviously</td><td>.165 (.145)</td><td>grade</td><td>-.144 (.047)</td></tr><tr><td>okay</td><td>.134 (.022)</td><td>wanted</td><td>-.126 (.036)</td></tr><tr><td>yeah</td><td>.121 (.048)</td><td>ten</td><td>-.125 (.044)</td></tr><tr><td>um</td><td>.110 (.027)</td><td>times</td><td>-.125 (.024)</td></tr><tr><td rowspan="10">French</td><td rowspan="5">reduction</td><td>jesuis</td><td>.302 (.036)</td><td>ont</td><td>-.183 (.047)</td></tr><tr><td>jesais</td><td>.236 (.012)</td><td>vais</td><td>-.180 (.071)</td></tr><tr><td>peutêtre</td><td>.205 (.026)</td><td>vois</td><td>-.160 (.022)</td></tr><tr><td>ai</td><td>.160 (.041)</td><td>oh</td><td>-.154 (.021)</td></tr><tr><td>de</td><td>.147 (.031)</td><td>te</td><td>-.149 (.036)</td></tr><tr><td rowspan="5">prominence</td><td>mhm</td><td>.352 (.179)</td><td>etc</td><td>-.167 (.108)</td></tr><tr><td>problème</td><td>.185 (.074)</td><td>école</td><td>-.144 (.107)</td></tr><tr><td>accord</td><td>.178 (.055)</td><td>soit</td><td>-.117 (.048)</td></tr><tr><td>gamin</td><td>.161 (.051)</td><td>oh</td><td>-.115 (.069)</td></tr><tr><td>moment</td><td>.134 (.039)</td><td>pense</td><td>-.106 (.037)</td></tr><tr><td rowspan="10">Mandarin</td><td rowspan="5">reduction</td><td>de0</td><td>.201 (.191)</td><td>xue2-xiao4</td><td>-.241 (.094)</td></tr><tr><td>ke3-shi4</td><td>.192 (.059)</td><td>zhe4-ge0</td><td>-.089 (.015)</td></tr><tr><td>yin1-wei1</td><td>.148 (.044)</td><td>bian4-cheng2</td><td>-.084 (.040)</td></tr><tr><td>le0</td><td>.143 (.072)</td><td>da4-jia1</td><td>-.079 (.040)</td></tr><tr><td>suo3-yi3</td><td>.142 (.053)</td><td>bu4-neng2</td><td>-.070 (.048)</td></tr><tr><td rowspan="5">prominence</td><td>dang1-bing1</td><td>.272 (.044)</td><td>san1</td><td>-.136 (.049)</td></tr><tr><td>xue2-xiao4</td><td>.183 (.104)</td><td>jiao4</td><td>-.131 (.027)</td></tr><tr><td>wei4-shen2-me0</td><td>.119 (.057)</td><td>na4-me0</td><td>-.091 (.047)</td></tr><tr><td>ru2-guo3</td><td>.112 (.102)</td><td>dong1-xi1</td><td>-.079 (.019)</td></tr><tr><td>da4-jia</td><td>.105 (.049)</td><td>che1</td><td>-.076 (.011)</td></tr></tbody></table>

## D Token frequency vs. gold and predicted labels

<footnote n="1">¹ u as in the initialism “OSU (Ohio State University)”, which appears frequently in the Buckeye corpus.</footnote>
<figure><img src="image_13.png" /><figcaption>Figure 10: Predicted and gold reduction label rates as a function of log frequency (x axis, z-scored) and model type × language (panels)</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 11: Predicted and gold prominence label rates as a function of log frequency (x axis, z-scored) and model type × language (panels)</figcaption></figure>