<figure><img src="image_1.png" /></figure>

HAL
open science

# Evaluating Data-Efficient LLMs on a Benchmark of Disfluency Minimal Pairs

Sheng-Fu Wang, Laurent Prevot

► To cite this version:

Sheng-Fu Wang, Laurent Prevot. Evaluating Data-Efficient LLMs on a Benchmark of Disfluency Minimal Pairs. 12th edition of the Disfluency in Spontaneous Speech Workshop, Sep 2025, Lisbon, Portugal. hal-05266207

HAL Id: hal-05266207

https://hal.science/hal-05266207v1

Submitted on 18 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluating Data-Efficient LLMs on a Benchmark of Disfluency Minimal Pairs

Sheng-Fu Wang<fnref n="1" />, Laurent Prévot²,³

¹Institute of Linguistics, Academia Sinica, Taiwan

²Aix Marseille Université & CNRS, LPL, France

³CNRS & MEAE, CEFC, Taiwan

sftwang@gate.sinica.edu.tw, laurent.prevot@cnrs.fr

## Abstract

Zero-shot benchmarks based on minimal pairs have become an essential part of the toolkit for evaluating large language models' linguistic capacities. Most of these tasks focus on syntactic, semantic, and morphological phenomena and are built from expert-crafted or semi-automatically generated sentences. Motivated by the crucial role of spontaneous speech in language processing, we experimented with creating a benchmark that leverages spontaneous speech corpora in three languages (English, French, and Mandarin). Crucially, the benchmark tests LLMs on disfluencies, a ubiquitous and essential feature of spontaneous speech. Our findings show that models pretrained on conversational data exhibit a clear advantage in handling disfluencies compared to those trained on written encyclopedic text. Furthermore, cross-linguistic LLMs trained on much larger datasets did not exhibit strong advantages in our proposed benchmark, highlighting the potential of disfluency-based tasks as a challenging problem for language models.

**Index Terms:** disfluency, large language models, filler, silent pause, spontaneous speech

## 1. Introduction

The success of LLMs in solving a wide range of language-related tasks requires linguists to better understand them and their performance. Most of the effort in this domain has, however, been focused either on evaluation based on canonical standard linguistic data or on end-to-end high-level tasks in which language is buried within a range of other dimensions.

One of the main goals of the present study is to supplement existing efforts in linguistic benchmarks by building them from spontaneous speech corpora. The focus on spontaneous speech comes from its central role in daily language use, even more so than the written form of language that dominates both pre-training and evaluation in most NLP applications. Compared with written data, spontaneous speech transcripts like Example (1) exhibit more syntactic and lexical variability, making the utterances in them drastically different from the sentences that typically make up linguistic benchmarks. Furthermore, spontaneous speech contains an abundance of fillers (e.g., 'um' in English, 'euh' in French) and repeated words or segments.

(1) (from Buckeye) yeah she's got # she's got * a lot of kids
* * there's um # three boys and...<fnref n="1" />

It has long been noted that these occurrences are far from simply irregular disruptions in speech, as their occurrences follow certain statistical and linguistic regularities, such as utterance length, positions within utterances, and the co-occurrence

effects of disfluencies [1]. Furthermore, disfluencies such as fillers have also been shown to be a window into speech planning [2]. In other words, the production and perception of disfluencies are likely to be indispensable parts of linguistic competence, just like the commonly targeted competences in syntax, semantics, and morphology.

In this study, we hope to demonstrate the promising approach of building benchmarks that target disfluencies from spontaneous speech in three languages (English, French, and Mandarin). This paper reports the construction of a benchmark for evaluating large language models' capacity to model disfluencies in spontaneous speech. The goals of this study are as follows. First, we aim to demonstrate how the richness of spontaneous speech phenomena can be extracted from annotated corpora to build resources for large language models. Second, we call for benchmarks in more diverse languages. Third, we show the benefits of including conversational data in data-efficient pretraining of large language models.

## 2. Related Works

Building databases or benchmarks that test computational models' capacity for language has been a crucial way that linguists contribute to the field of natural language processing and computational linguistics. Significant early contributions include SNLI [3] and CoLA [4]: The former tests models' ability to identify inference relationships, while the latter focuses on syntactic judgments sourced from linguistic literature in English. These tasks are classification tasks where models need to be trained to perform them. As pretrained models become the starting point of practical applications in natural language processing and computational linguistics, experiments involving these benchmarks are increasingly done by 'fine-tuning' these language models on these tasks.

The advancement of large language models also motivates the construction of benchmarks that naturally support zero-shot evaluations, i.e., tasks that can be performed without fine-tuning. In the linguistics-related literature, the most prominent task involves presenting a language model with two sequences: one grammatical or acceptable, and the other with a minimal difference that renders the sequence ungrammatical or less acceptable. Evaluation of LLMs with these minimal-pair benchmarks is straightforward in a zero-shot setting: the success of a model is measured by how often it assigns a higher probability to the 'good example' within a minimal pair [5, 6]. BLiMP [7] is a representative instance of such a benchmark in English, with a collection of minimal pairs that target 67 types of contrasts in syntactic, semantic, and morphological acceptability. Similar datasets in other languages have also been built [8, 9].

As for disfluencies and LLMs, in addition to a notable study

<footnote n="1">¹Symbol # and * correspond respectively to short pauses and noises.</footnote>
that directly shows how disfluencies tend to occur close to tokens with high perplexity (calculated with GPT-2, [10]), there are also some recent developments in using LLMs to generate disfluent sentences in a data augmentation setting. [11] experimented with this approach for eventually feeding back into solving the problem of disfluency detection, while [12] used LLM-generated disfluencies to strengthen training data for more robust dialogue systems. It has also been shown that prompts with disfluencies and repairs may affect LLMs' performance in QA tasks, especially repairs based on speech errors [13]. Relatedly, [14] experimented with using fine-tuning LLMs to insert disfluencies in speech and showed that text-to-speech outcomes are rated as more spontaneous when disfluencies are thus inserted. One other line of research that connects LLMs and disfluencies concerns the roles of disfluencies in Alzheimer patients' speech. Notably, [15] report that by encoding pauses into the transcripts, the fine-tuning performance of LLMs such as BERT can be significantly improved.

Finally, [16] tested LLMs' ability to predict reduced and prominent tokens in spontaneous speech based on transcripts alone in fine-tuning experiments in English, French, and Mandarin. They showed the promise of models trained with 10M tokens, especially with conversational data.

# 3. Methodology

## 3.1. Benchmarks

### 3.1.1. Data

For English, we used the Buckeye Corpus [17] with 38.1 hours of spontaneous speech (40 speakers) recorded in an interview format. For French, SUMM-RE [18] is a 1M-word meeting corpus, with a section of about 20% of the corpus provided with manually checked fine-grained transcriptions. For Mandarin, we used the Sinica Mandarin Conversational Dialogue Corpus (Sinica MCDC8)<fnref n="2" /> [19]. The main reason for the choice of these corpora is the high quality of their transcriptions, in which disfluencies, pauses, and repetitions are accurately rendered.

### 3.1.2. Minimal Pair Creation

Following the paradigm of zero-shot evaluations using minimal pairs, for the first filler-related task (*moved filler*), we first identified utterances or speaker turns that contained the following tokens for each language: 'euh' for French [20], 'um' or 'uh' for English [2], and 'uhn', 'en/un', 'nage' (that) for Mandarin [19]. These utterances made up the 'positive examples' in the minimal pair. We then created the 'negative examples' by randomly moving the fillers around within the sequence. Thus, this task aimed to test models' ability to prefer sequences where the fillers are in their naturally occurring positions.

For a second filler-related task (*filler-pause swap*), we identified utterances and turns with both fillers and silent pauses. To create the *negative example*, one of the filler-silent pause pairs had their positions swapped. This tests the hypothesis that fillers and silent pauses assume different functions and distributions in spontaneous speech [21].

The *moved pause* task was similar to the *moved filler* task, where utterances with silent pauses were chosen and the pause was randomly moved to another position in the utterance. Finally, for the *moved repetition* task, we first identified utterances with repeated words next to each other. We then removed the repetition and randomly selected another

<table><caption>Table 1: *example of minimal pairs*</caption><thead><tr><th colspan="2">Filler</th></tr></thead><tbody><tr><td>Original</td><td>I exactly, I'll figure something out # so <strong>uh</strong> right on</td></tr><tr><td>Moved Filler</td><td>I exactly, I'll figure something <strong>uh</strong> out # so right on</td></tr><tr><td>Filler-pause swap</td><td>I exactly, I'll figure something # out <strong>uh</strong> so right on # personally</td></tr><tr><td>Shuffled</td><td>figure exactly something # I so # right <strong>uh</strong> out I'll on</td></tr><tr><th colspan="2">Silent Pause</th></tr><tr><td>Original</td><td>um he he was old cat # my older cat</td></tr><tr><td>Moved Pause</td><td>um he he was # old cat my older cat</td></tr><tr><td>Shuffled</td><td>he um cat # cat he my older was old,</td></tr><tr><th colspan="2">Repetition</th></tr><tr><td>Original</td><td>so <strong>we we</strong> have tons of like lakes and stuff but it's just not the same</td></tr><tr><td>Moved repetition</td><td>so we have tons of like lakes <strong>and and</strong> stuff but it's just not the same</td></tr><tr><td>Shuffled</td><td>have so just tons it's the of and <strong>we</strong> same but not stuff like <strong>we</strong> lakes</td></tr></tbody></table>

<table><caption>Table 2: *number of minimal pairs (En/Fr/Zh)*</caption><thead><tr><th colspan="2">filler</th><th>pause</th><th>repetition</th></tr></thead><tbody><tr><td><strong>moved</strong></td><td><strong>swapped</strong></td><td><strong>moved</strong></td><td><strong>moved</strong></td></tr><tr><td>500/492/206</td><td>149/68/56</td><td>500/500/500</td><td>500/500/222</td></tr></tbody></table>

word in the utterance to be repeated. Examples of these manipulations are shown in Table 1.

As randomization is involved in the generation of bad examples, we generated ten sets of minimal pairs for each task in each language. Finally, we had to decide on a pause symbol both for pretraining and benchmark data, and we settled on using commas. The number of pairs is summarized in Table 2.

## 3.2. Zero-shot experiments

### 3.2.1. Models

For each language, we tested five models, including three RoBERTa models pretrained from scratch on three types of data: conversational, written, and a mixture of these two. Motivated by the BabyLM initiative [22], these models were trained on datasets with moderate sizes (10M words for English and French, and 5M characters for Mandarin). We also included XLM-ROBERTA-BASE and XLM-ROBERTA-LARGE<fnref n="3" /> [23] in the experiments to serve as potential topline performers in the minimal pair tasks. We do not consider it very fruitful to discuss whether such sizes qualify our model as a *Large* Language Model. However, we remind the reader that the ongoing race for ever-larger models targets general reasoning abilities rather than language-specific ones, which can be attained with much smaller amounts of pretraining data [24]. Moreover, there are simply not enough data available to pre-train on spontaneous speech transcripts with more than 10M tokens.

For pretraining English RoBERTa models, we drew on the training data provided in the 2024 BabyLM challenge [22]. As the training data itself was a mix of spoken (Switchboard

<footnote n="2">²https://www.aclclp.org.tw/use Mat.php#mcdc</footnote>

<footnote n="3">³https://huggingface.co/FacebookAI/xlm-roberta-base & https://huggingface.co/FacebookAI/xlm-roberta-large</footnote>
[25], BNC [26] dialogues, and CHILDES) and written data (Gutenberg and Simple Wikipedia), we directly took the 10M version for our 'mixed' model pretraining. The Switchboard and BNC dialogues were used to pretrain the spoken model, and the Simple Wikipedia part was used to pretrain the written model. For pretraining French models, we used a conversational dataset built from a mix of (ORFEO [27])<fnref n="4" /> and CHILDES-FR<fnref n="5" /> [28, 29]) and a 10M-token dataset from Wikipedia. The 'mixed' model was pretrained by using fifty percent of the aforementioned two datasets respectively. For Taiwan Mandarin, the conversational data was derived from the NCCU Spoken Corpus of Taiwan Mandarin [30], the Taiwan Corpus of Child Mandarin (TCCM) [31], the media interview sections of the Academia Sinica Balanced Corpus [32], and automatic transcripts from three sources: a subset of meeting recordings from Taiwan's Legislative Yuan<fnref n="6" />, the CALLFRIEND Mandarin Chinese-Taiwan Dialect corpus [33], and 116 episodes of 'PTS Theme Night SHOW', a show featuring interviews and discussions<fnref n="7" />. The written data was from a subset of Traditional Chinese Wikipedia<fnref n="8" />. A mixed dataset was similarly created to pretrain the third model.

For more consistent formatting between conversational and written data, we inserted line breaks after all major punctuation marks (periods, exclamation marks, and question marks), while also changing colons to commas. For the Mandarin data, we standardized the orthography of one of the common particles, 'eh'.

We used *SentencePiece* tokenizers trained on the corresponding data with the Unigram model [34]. Vocabulary size was set to 10K. Models were trained with similar hyperparameters for languages with 'small' data (i.e., 5–10 MB) in the Goldfish LM project [35]: 4 layers, 8 attention heads, and a hidden layer size of 512. Batch size was set to 64, with a learning rate of 5e-4 for English and French, and 1e-3 for Mandarin.

### 3.2.2. Experiments

The main experiment focuses on the benchmark of disfluencies in three languages. To evaluate the capacity of the same models on a more traditional type of zero-shot linguistic benchmark, we also ran the English models on the filtered and supplement versions of BLiMP [7] used in the 2024 BabyLM Challenge [22], the Mandarin models on CLiMP [36], and the French models on the newly proposed multilingual BLiMP [8]. These tasks target syntactic, semantic, and morphological phenomena, and can serve as points of comparison for our proposed benchmark.

We used the minicons [37] package in Python for estimating the mean log probability of each sequence in a minimal pair. For English and French, within-word left-to-right masking [38] was used. A model's prediction was deemed correct when a higher probability was assigned to the positive example.

## 4. Results

### 4.1. Fillers, silent pauses, and repetitions

Across three languages, the *Conv* and *Mixed* models significantly outperform the *Wiki* model in the filler-movement task

as shown in Figure 1⁹. In other words, the conversational models are better at 'preferring' the sequences with fillers in their original positions. Similar trends are observed for the task filler-pause swapping task.

<figure><img src="image_3.png" /><figcaption>Figure 1: Accuracy (y axis) in filler-related tasks as a function of model type (x axis), language (columns), and task type (rows).</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 2: Accuracy (y axis) in filler-related tasks: filler type (color), model (x axis), language (columns), and task (rows).</figcaption></figure>

As the English and Mandarin tasks involve more than one filler, we show the item-specific results in Figure 2. For English, while the trends are largely similar between 'um' and 'uh', we do observe some trends, such as the conversational model having a particular advantage in 'um' movements, and the XLM-RoBERTa models being challenged with detecting the switch between 'um' and silent pauses. For Mandarin, the most striking observation is that while 'um/un' movements were easier for the models than 'nage' (that), the pattern is flipped with filler-pause swaps, which likely reflects how 'um/un' are more similar in function (and distribution) to a silent pause.

As shown in Figure 3, the *Conv* models in English and Mandarin significantly outperform their *Wiki* counterparts in the pause movement task, but not in French. English and Mandarin also differ in whether the mixed model outperforms the spoken model. The *Conv* models also exhibit a consistent advantage over the *Wiki* models across languages in repetition-related tasks, as shown in Figure 4. With the exception of French,

<footnote n="4">⁴https://hdl.handle.net/11403/cefc-orfeo</footnote>

<footnote n="5">⁵https://phon.talkbank.org/access/French/</footnote>

<footnote n="6">⁶https://www.parliamentarytv.org.tw/</footnote>

<footnote n="7">⁷https://www.youtube.com/ playlist?list=PLSzfF9jXmOMOrWt5brkz9FudYKJGbQuYG</footnote>

<footnote n="8">⁸https://huggingface.co/datasets/zetavg/zh-tw-wikipedia/</footnote>

<footnote n="9">⁹Model comparisons are based on Bayesian regression analyses with weak (uniform) priors using the *brms* package in R, with MODEL as the fixed predictor and iteration (i.e., different randomizations) as a random intercept. Post hoc hypothesis tests focused on comparing the three small models (*Wiki*, *Conv*, *Mixed*) in the movement and swapping tasks.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 3: Accuracy (y axis) in the pause task: model type (x axis), language (columns), and tasks (rows).</figcaption></figure>

the *Conv* models also outperform the mixed models, suggesting strong advantage of conversational data in this task.

<figure><img src="image_6.png" /><figcaption>Figure 4: Accuracy (y axis) in repetition task: model type (x axis), language (columns), and tasks (rows).</figcaption></figure>

## 4.2. BLiMP tasks

Results on BLiMP-style syntactic and semantic tasks, as shown in Figure 5, are mixed: English and Mandarin results showed a significant advantage for the *Wiki* model, which in Mandarin was in turn outperformed by the *Mixed* model. On the other hand, the French results show a trend of advantages for the *Conv* model. Overall, XLM-RoBERTa consistently outperforms the smaller models by a margin of 10% or more, which is not observed in our disfluency tasks.

<figure><img src="image_7.png" /><figcaption>Figure 5: Accuracy (y axis) in BLiMP-style tasks as a function of model type (x axis), language (columns), and tasks (rows).</figcaption></figure>

# 5. Discussion & Conclusion

This paper demonstrates how corpora of spontaneous speech featuring disfluencies can be utilized for building benchmarks for zero-shot LLMs evaluations. As the benchmark is built from transcripts and the evaluated models are text-based, it might be

argued that this benchmark simply provides another genre compared to ones built either constructed or collected from text corpora. However, we believe that spontaneous speech's role in daily language use, along with the disfluencies that we used as a target, makes it a uniquely interesting domain to challenge LLMs with, even if the transcripts are just idealizations of much richer patterns in spontaneous speech. The fact that larger XLM-RoBERTa models, trained on typical text-based data, did not have apparent advantages over our smaller models despite having much more training data and larger size, likely suggests that this type of dataset can indeed be challenging for LLMs.

One of the most consistent results is that models trained on spoken-language transcripts generally performed better than models trained on written data (that is, Wikipedia). While this result appears trivial if viewed from a train-task data congruence standpoint (models perform better if the domains of training and testing data are more aligned), we would like to argue that our results still show precisely whether and how this outcome is borne out. In addition, by pretraining models with amounts of data that are more realistic from the point of view of language acquisition [22], we are more likely to arrive at meaningful conclusions about human language, acquisition, and cognition (compared to discussing results from models trained in more words than a human being can realistically be exposed to). While we focused on encoder-based models to facilitate comparison with standard ROBERTA, we would like to compare the outcomes with decoder-based models pretrained on the same datasets in future work.

From the perspective of research on disfluencies, the different levels of success that these LLMs reach across tasks may also shed light on the extent to which patterns of disfluencies are predictable from 'texts', again suggesting that the occurrences of disfluencies indeed follow some linguistic regularities. Broadly speaking, the findings can also be placed into the context of how phonetic events such as reduction and prominence can be predicted from textual information in the predictability/surprisal literature [39, 40].

To conclude, we believe that the current landscape of linguistic benchmarks can benefit from the inclusion of more phenomena that tap into a wider range of linguistic competence, as well as the use of a wider set of languages and a more reasonable setting in terms of data size. Disfluencies and spontaneous speech, combined with testing models trained with different modalities (crucially including conversational data), serve as an adequate and promising start for this line of inquiry.

# 6. Acknowledgments

We would like to thank Hsieh Shu-Kai, Tseng Shu-Chuan and Huang Ri-Sheng for comments and discussions related to the paper. This work benefited from ANR-Funded project SUMMER (ANR-20-CE23-0017) and from Institut Convergence ILCB (ANR-16-CONV-0002), managed by the ANR and A*MIDEX, as well as Taiwan's National Science and Technology Council (NSTC-112-2410-H-001-098-MY2) and the Institute of Linguistics, Academia Sinica (LING-114-DLR-01).

# 7. References

[1] E. E. Shriberg, "Preliminaries to a theory of speech disfluencies," Ph.D. dissertation, Citeseer, 1994.
[2] H. H. Clark and J. E. F. Tree, "Using uh and um in spontaneous speaking," *Cognition*, vol. 84, no. 1, pp. 73-111, 2002.
[3] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, "A
large annotated corpus for learning natural language inference,” in *EMNLP*, 2015, pp. 632–642.

[4] A. Warstadt, A. Singh, and S. R. Bowman, “Neural network acceptability judgments,” *Transactions of the Association for Computational Linguistics*, vol. 7, pp. 625–641, 2019.

[5] T. Linzen, E. Dupoux, and Y. Goldberg, “Assessing the ability of lstms to learn syntax-sensitive dependencies,” *TACL*, vol. 4, no. 1, pp. 521–535, 2016.

[6] R. Marvin and T. Linzen, “Targeted syntactic evaluation of language models,” in *Proceedings of EMNLP*, 2018, pp. 1192–1202.

[7] A. Warstadt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S.-F. Wang, and S. R. Bowman, “BLiMP: The benchmark of linguistic minimal pairs for English,” *TACL*, vol. 8, pp. 377–392, 2020.

[8] J. Jumelet, L. Weissweiler, and A. Bisazza, “Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs,” *arXiv preprint arXiv:2504.02768*, 2025.

[9] B. Xiang, C. Yang, Y. Li, A. Warstadt, and K. Kann, “CLiMP: A benchmark for Chinese language model evaluation,” in *Proceedings of the 16th EACL: Main Volume*, P. Merlo, J. Tiedemann, and R. Tsarfaty, Eds., Apr. 2021, pp. 2784–2790.

[10] P. Sen, “Speech disfluencies occur at higher perplexities,” in *Proceedings of the Workshop on the Cognitive Aspects of the Lexicon*, 2020, pp. 92–97.

[11] Z. Cheng, J. Guo, H. Sun, and Y. Zhang, “Boosting disfluency detection with large language model as disfluency generator,” in *ICME*, 2024, pp. 1–6.

[12] B. Marie, “Disfluency generation for more robust dialogue systems,” in *Findings of the Association for Computational Linguistics: ACL 2023*, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds., Toronto, Canada, Jul. 2023, pp. 11479–11488.

[13] M. Rohanian, F. Nooralahzadeh, O. Rohanian, D. Clifton, and M. Krauthammer, “Disfluent cues for enhanced speech understanding in large language models,” in *Findings of the Association for Computational Linguistics: EMNLP 2023*, H. Bouamor, J. Pino, and K. Bali, Eds., Singapore, Dec. 2023, pp. 3676–3684.

[14] S. Z. Hassan, P. Lison, and P. Halvorsen, “Enhancing naturalness in llm-generated utterances through disfluency insertion,” *arXiv preprint arXiv:2412.12710*, 2024.

[15] J. Yuan, Y. Bian, X. Cai, J. Huang, Z. Ye, and K. Church, “Disfluencies and fine-tuning pre-trained language models for detection of alzheimer's disease.” in *Interspeech*, 2020, pp. 2162–6.

[16] S.-F. Wang, L. Prevot, J. an Chi, R.-S. Huang, and S.-K. Hsieh, “Spontaneous speech variables for evaluating llms cognitive plausibility,” in *CMCL*, 2025. [Online]. Available: https://arxiv.org/abs/2505.16277

[17] M. A. Pitt, K. Johnson, E. Hume, S. Kiesling, and W. Raymond, “The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability,” *Speech Communication*, vol. 45, no. 1, pp. 89–95, 2005.

[18] J. Hunter, H. Yamasaki, O. Granier, J. Louradour, R. Bertrand, K. Thompson, and L. Prévot, “Meeting: A corpus of french meeting-style conversations,” in *Actes de JEP-TALN-RECITAL 2024. 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1: articles longs et prises de position*. ATALA & AFPC, 2024, pp. 508–529.

[19] S.-C. Tseng, “Lexical coverage in taiwan mandarin conversation,” in *IJCLCLP, Volume 18, Number 1, March 2013*, 2013.

[20] B. Pallaud, R. Bertrand, L. Prevot, P. Blache, and S. Rauzy, “Sus-
pensive and disfluent self interruptions in french language interac-
tions,” *Fluency and Disfluency across Languages and Language
Varieties. Corpora and Language in Use–Proceedings*, vol. 4, pp.
109–138, 2019.

[21] R. Rose and M. Watanabe, “A crosslinguistic corpus study of silent and filled pauses: When do speakers use filled pauses to fill pauses,” in *Proceedings of 19th ICPhS*, 2019.

[22] L. Choshen, R. Cotterell, M. Y. Hu, T. Linzen, A. Mueller, C. Ross, A. Warstadt, E. Wilcox, A. Williams, and C. Zhuang, “[call for papers] the 2nd babylm challenge: Sample-efficient pre-training on a developmentally plausible corpus,” *arXiv preprint arXiv:2404.06214*, 2024.

[23] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2020, pp. 8440–8451.

[24] A. Warstadt, A. Mueller, L. Choshen, E. Wilcox, C. Zhuang, J. Ciro, R. Mosquera, B. Paranjabe, A. Williams, T. Linzen et al., “Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora,” in *Proceedings of the BabyLM Challenge at the 27th CoNLL*, 2023, pp. 1–34.

[25] J. J. Godfrey, E. C. Holliman, and J. M. McDaniel, “Switchboard: Telephone speech corpus for research and development,” 1992. [Online]. Available: https://catalog.ldc.upenn.edu/LDC97S62

[26] B. Consortium et al., “British national corpus,” *Oxford Text Archive Core Collection*, 2007.

[27] C. Benzitoun, J.-M. Debaisieux, and H.-J. Deulofeu, “Le projet orféo: un corpus d’étude pour le français contemporain,” *Corpus*, no. 15, 2016.

[28] B. MacWhinney, *The CHILDES project: Tools for analyzing talk, Volume 1: Transcription format and programs*. Psychology Press, 2014.

[29] Y. Rose and B. MacWhinney, “The phonbank project: Data and software-assisted methods for the study of phonology and phonological development,” in *The Oxford Handbook of Corpus Phonology*. Oxford University Press, 05 2014.

[30] K. Chui and H.-l. Lai, “The NCCU corpus of spoken Chinese: Mandarin, Hakka, and Southern Min,” *Taiwan Journal of Linguistics*, vol. 6, no. 2, 2008.

[31] L.-H. Chang, Z.-R. Chang, Y.-C. Ke, and S.-H. Xiao, “Taiwan child language corpus (tccm),” National Science Council Research Project, NSC96-2420-H-002-030, 2011, available at http://lope.linguistics.ntu.edu.tw/tccm/.

[32] K.-J. Chen, C.-R. Huang, L.-P. Chang, and H.-L. Hsu, “Sinica corpus: Design methodology for balanced corpora,” in *Language, Information and Computation: Selected Papers from the 11th PACIFIC: 20-22 December 1996, Seoul*. Waseda University, 1996, pp. 167–176.

[33] A. Canavan, G. Zipperlen, and J. Bartlett, “CALLFRIEND Mandarin Chinese-Taiwan Dialect Second Edition LDC2020S06,” Web Download, Philadelphia, 2020.

[34] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” *arXiv preprint arXiv:1804.10959*, 2018.

[35] T. A. Chang, C. Arnett, Z. Tu, and B. K. Bergen, “Goldfish: Monolingual language models for 350 languages,” *arXiv preprint arXiv:2408.10441*, 2024.

[36] B. Xiang, C. Yang, Y. Li, A. Warstadt, and K. Kann, “CLiMP: A benchmark for Chinese language model evaluation,” in *Proceedings of the 16th EACL: Main Volume*, P. Merlo, J. Tiedemann, and R. Tsarfaty, Eds., Apr. 2021, pp. 2784–2790.

[37] K. Misra, “minicons: Enabling flexible behavioral and represen-
tational analyses of transformer language models,” *arXiv preprint
arXiv:2203.13112*, 2022.

[38] C. Kauf and A. Ivanova, “A better way to do masked language model scoring,” in *Proceedings of the 61st Annual Meeting of the ACL (Volume 2: Short Papers)*, 2023, pp. 925–935.

[39] L. Wolf, T. Pimentel, E. Fedorenko, R. Cotterell, A. Warstadt, E. Wilcox, and T. Regev, “Quantifying the redundancy between prosody and text,” in *Proceedings of the 2023 EMNLP*, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 9765–9784.

[40] S. Seyfarth, “Word informativity influences acoustic duration: Effects of contextual predictability on lexical representation,” *Cognition*, vol. 133, no. 1, pp. 140–155, 2014.