## 4.5 Additional experiments

In this subsection, we report three additional experiments that we carried out for the purpose of gaining deeper insight about our corpora.

First, we divided each corpus into two according to the proficiency level of the target vocabulary items (B1 or B2). In such a scenario, the web-crawled corpus, which consists of authentic texts not specifically conceived for language learners, increases in significance of difference with the reference one; in particular, in relation to level B2, for which it exhibits 12 significantly different features (as opposed to 5 when the entire corpus is considered).

In contrast, sensitivity in relation to proficiency levels is noticeable among the LLM corpora. Features associated with textual complexity, such as the total number of words, the number of letters per word and the number of first-person pronouns per sentence, demonstrate significantly differing values compared to when the corpus is taken in its entirety. In particular, the Gemini corpus shows the highest modification of values based on proficiency level. Interestingly, there are even cases where LLM corpora show higher sensitivity to the level at hand than the reference corpus<fnref n="14" />. As the different CEFR levels are also associated with different domains, the most frequent words in the LLM corpora now reflect the domain at hand (derivatives of "science" being common for the scientific domain and words such as "crop" and "soil" for agronomy).

As a second experiment, the complete set of textual characteristics was used within a global distance metric in order to determine which corpus is globally closest to the reference one. For this purpose, min-max normalisation was applied, and the Euclidean distance between corpora was calculated. The one-shot Gemini corpus ranked first (2.96), followed by the web-crawled one (3.8) and the three zero-shot LLM corpora, which in turn demonstrated relatively similar values<fnref n="15" />.

Finally, as regards the one-shot scenario, the features of the corresponding corpus were compared to those of its zero-shot counterpart, revealing that their majority<fnref n="16" /> come closer to the reference, reducing the significance in divergence in 10 out of all 13 cases. The one-shot generation method also leads to significantly increased global closeness to the reference.

## 5 Discussion

Given the high value ranges and SD pertaining to the web-crawled corpus, as well as the tendency between its and LLM-generated texts to diverge from the baseline in opposite directions, the two types of corpora can work together effectively within an educational framework to provide a variety of contexts for ESP vocabulary. The web-crawling method is computationally efficient and tends to provide texts that are naturally close to the reference examples. In turn, the benefits of LLM generation include sensitivity to the CEFR level at hand and high malleability, including the potential to derive a large number of examples from a single one within a one-shot setting.

Table 2 shows the example sentences for the verb "to avoid" in all discussed corpora. In accordance

<footnote n="14">14. For instance, when solely B1 examples are considered, the reference corpus unintuitively has higher values for the percentages of non-stem words and words outside the Dale-Chall frequency list compared to when the entire corpus is considered.</footnote>

<footnote n="15">15. Mistral : 4.03 ; Vicuna : 4.06 ; Gemini : 4.57</footnote>

<footnote n="16">16. excluding "percentage of hapax legomena", "average concreteness" and several standard deviation values</footnote>
with the results stated in Section 4, the web-crawled example exhibits high complexity : it is longest and contains three proper nouns and a direct quotation. The Vicuna and Gemini examples show similarity with the reference if one considers the number of verbs and the joint number of adjectives and adverbs <fnref n="17" />. Mistral adds variety with its use of first-person language, which however results in the example's reduced formality and in-domain quality. In turn, Gemini's example is narrowly associated with the scientific domain. Much akin to the reference, the "Gemini : one-shot" example features additional vocabulary items that are relevant for the same learner audience ("essential", "biases", "skew"), yet it does not demonstrate any perceivable copying of the reference content.

<table><caption>TABLE 2 – Examples for the vocabulary item "avoid" (domain "science"; level "B1")</caption><tr><td><strong>Reference</strong></td><td>There is still time to reverse the warming trend and <strong>avoid</strong> global environmental and economic catastrophe.</td></tr><tr><td><strong>Web-Crawled</strong></td><td>"There are few institutional structures to achieve co-operation globally on the sort of scales now essential to <strong>avoid</strong> very serious consequences," warns lead author Dr Brian Walker of Australia's CSIRO.</td></tr><tr><td><strong>Mistral</strong></td><td>I try to <strong>avoid</strong> using my phone while I study because it can be a great distraction.</td></tr><tr><td><strong>Vicuna</strong></td><td>To <strong>avoid</strong> overfishing, it is essential to manage fisheries sustainably and establish marine protected areas.</td></tr><tr><td><strong>Gemini</strong></td><td>To <strong>avoid</strong> contamination, the scientist carefully wore gloves and a lab coat while conducting the experiment.</td></tr><tr><td><strong>Gemini : one-shot</strong></td><td>In scientific experiments, it is essential to <strong>avoid</strong> biases that could skew the results.</td></tr></table>

## 6 Conclusion and Future Directions

This work discussed context corpora derived via two discrete NLP-based methods (web-crawling and generation by LLMs), which can be used to aid university students in the acquisition of ESP vocabulary. The stylistic characteristics of the generated contexts were compared to a professionally crafted baseline of context examples through quantitative evaluation based on readability features. The comparison revealed higher similarity among different LLM-generated corpora than between them and corpora of a different nature. The "Gemini : one-shot" corpus was discovered to be globally closest to the reference, thereby showing that generation can be refined through prompt engineering. Future experiments including few-shot use of only several high-quality examples independent of a vocabulary list can help mitigate the current limitation of reliance on a reference corpus.

Associated future plans include an evaluation of the pedagogic characteristics of contexts as opposed to their readability-based qualities. In more practical terms, pre- and post-tests of students' performance will be conducted in relation to the introduction of a large inventory of NLP-derived examples in the implied university courses.

<footnote n="17">17. respectively, 3 and 3 for the reference, 4 and 4 for Vicuna and 3 and 1 for Gemini</footnote>
## References

AKTER S. N., YU Z., MUHAMED A., OU T., BÄUERLE A., CABRERA A. A., DHOLAKIA K., XIONG C. & NEUBIG G. (2023). An in-depth look at Gemini's language abilities. arXiv : 2312.11444.

ANIL R., BORGEAUD S., WU Y., ALAYRAC J.-B., YU J., SORICUT R., SCHALKWYK J., DAI A. M., HAUTH A., MILLICAN K., SILVER D., PETROV S., JOHNSON M., ANTONOGLOU I., SCHRITTWIESER J., GLAESE A., CHEN J., PITLER E. & VINYALS O. (2023). Gemini : A family of highly capable multimodal models. arXiv : 2312.11805.

BRYSBAERT M., WARRINER A. B. & KUPERMAN V. (2014). Concreteness ratings for 40 thousand generally known English word lemmas. *Behavior Research Methods, **46*(3), 904–911. DOI: 10.3758/s13428-013-0403-5.

CECH R. & KUBAT M. (2018). Morphological richness of text. In M. FIDLER & V. CVRCEK, Éds., *Taming the Corpus*, p. 63–77. Springer. DOI: 10.1007/978-3-319-98017-1_4.

COLLINS-THOMPSON K. (2014). Computational assessment of text readability : A survey of current and future research. *International Journal of Applied Linguistics*, **165**(2), 97–135.

DEUTSCH T., JASBI M. & SHIEBER S. (2020). Linguistic features for readability assessment. In *Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications* : Association for Computational Linguistics. DOI: 10.18653/v1/2020.bea-1.1.

DUBAY W. H. (2007). *The Classic Readability Studies*. Rapport interne, ERIC Clearinghouse. DOI: 10.1109/TPC.2008.2007872.

FERRAND L. (2007). *Psychologie cognitive de la lecture*. Bruxelles : De Boeck.

FRANÇOIS T. (2011). La lisibilité computationnelle : un renouveau pour la lisibilité du français langue première et seconde? *International Journal of Applied Linguistics (ITL)*, **160**, 75–99.

GODWIN-JONES R. (2018). Evolving views on vocabulary development. *Language Learning & Technology*, **22**(3), 1–19.

HEILMAN M., ZHAO L., PINO J. & ESKENAZI M. (2008). Retrieval of reading materials for vocabulary and reading practice. In J. TETREAULT, J. BURSTEIN & R. DE FELICE, Éds., *Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications*, p. 80–88, Columbus, Ohio : Association for Computational Linguistics.

HUCKIN T. & COADY J. (1999). Incidental vocabulary acquisition in a second language : A review. *Studies in second language acquisition*, **21**(2), 181–193.

HUSSIN A., CHAN Y. F. & ZUBAIDAH A. (2010). Scientific structural changes within texts of adapted reading materials. *English Language Teaching*, **3**. DOI: 10.5539/elt.v3n4p216.

JIANG A. Q., SABLAYROLLES A., MENSCH A., BAMFORD C., CHAPLOT D. S., CASAS D. D. L. & EL SAYED W. (2023). Mistral 7B. arXiv : 2310.06825.

JIN T. & LU X. (2018). A data-driven approach to text adaptation in teaching material preparation : Design, implementation, and teacher professional development. *TESOL Quarterly*, **52**, 457–467. DOI: 10.1002/tesq.434.

KYLE K. (2019). Measuring lexical richness. In S. WEBB, Éd., *The Routledge Handbook of Vocabulary Studies*, p. 454–476. Routledge. DOI: 10.4324/9780429291586.

LAM K.-Y., CHENG V. C. W. & YEONG Z. K. (2023). Applying large language models for enhancing contract drafting. In *Proceedings of the Third International Workshop on Artificial Intelligence and Intelligent Assistance for Legal Professionals in the Digital Workplace*, p. 70–80.
LITMAN D. J. (2016). Natural language processing for enhancing teaching and learning. In AAAI Conference on Artificial Intelligence.

MEHTA I. (2022). ShareGPT lets you easily share your ChatGPT conversations. Blog post.

MEURERS D. (2021). *Natural Language Processing and Language Learning*, In *The Encyclopedia of Applied Linguistics*, p. 1–15. John Wiley Sons, Ltd. DOI: https://doi.org/10.1002/9781405198431.wbeal0858.pub2.

MEURERS D., ZIAI R., AMARAL L., BOYD A., DIMITROV A., METCALF V. & OTT N. (2010). Enhancing authentic web pages for language learners. In J. TETREAULT, J. BURSTEIN & C. LEACOCK, Éds., *Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications*, p. 10–18, Los Angeles, California : Association for Computational Linguistics.

RAMOS R. & DARIO F. (2015). Incidental vocabulary learning in second language acquisition : A literature review. *Profile Issues in Teachers Professional Development, 17*(1), 157–166.

SHAIKH S., YAYILGAN S. Y., KLIMOVA B. & PIKHART M. (2023). Assessing the usability of ChatGPT for formal english language learning. *European Journal of Investigative Health Psychology and Education, 13*, 1937–1960. DOI: 10.3390/ejihpe13090140.

SHAPIRO S. S. & WILK M. B. (1965). An analysis of variance test for normality (complete samples). *Biometrika*, **52**(3/4), 591–611. DOI: 10.2307/2333709.

URE J. (1971). Lexical density and register differentiation. *Applications of linguistics*, **23**(7), 443–452.

VAJJALA S. (2022). Trends, limitations and open challenges in automatic readability assessment research. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, p. 5366–5377.

WILKENS R., ALFTER D., WANG X., PINTARD A., TACK A., YANCEY K. P. & FRANÇOIS T. (2022). FABRA : French aggregator-based readability assessment toolkit. In N. CALZOLARI, F. BÉCHET, P. BLACHE, K. CHOUKRI, C. CIERI, T. DECLERCK, S. GOGGI, H. ISAHARA, B. MAEGAARD, J. MARIANI, H. MAZO, J. ODIJK & S. PIPERIDIS, Éds., *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, p. 1217–1233, Marseille, France : European Language Resources Association.

WILKENS R., WATRIN P., CARDON R., PINTARD A., GRIBOMONT I. & FRANÇOIS T. (2024). Exploring hybrid approaches to readability : experiments on the complementarity between linguistic features and transformers. In *Findings of the Association for Computational Linguistics : EACL 2024*, p. 2316–2331.

WILSON R. (2004). Computers and the internet : Together a great tool for esl/efl learners.

YOON S.-Y., LEE C. M., HOUGHTON P., LOPEZ M., SAKANO J., LOUKINA A., KROVETZ B., LU C. & MADNANI N. (2017). Analyzing item generation with natural language processing tools for the toeic ® listening test : Analyzing item generation with nlp tools. *ETS Research Report Series, 2017*. DOI: 10.1002/ets2.12183.

YOUNG J. C. & SHISHIDO M. (2023a). Evaluation of the potential usage of ChatGPT for providing easier reading materials for ESL students. In *Proceedings of EdMedia and Innovate Learning*, p. 155–162 : Association for the Advancement of Computing in Education (AACE).

YOUNG J. C. & SHISHIDO M. (2023b). Investigating OpenAI's ChatGPT potentials in generating chatbot's dialogue for English as a foreign language learning. *International Journal of Advanced Computer Science and Applications, 14*(6), West Yorkshire. DOI: 10.14569/IJACSA.2023.0140607.

YULE G. U. (1944). *The Statistical Study of Literary Vocabulary*. Cambridge University Press.
Appendix 1 : List of Crawled Websites

https://climate.ec.europa.eu/climate-change_en

https://climate.ec.europa.eu/eu-action_en

https://climate.ec.europa.eu/index_en

https://climate.nasa.gov/

https://engineeringdiscoveries.com/

https://newatlas.com

https://sciencedemonstrations.fas.harvard.edu/

https://sustainability.stanford.edu/

https://world-nuclear.org

https://www.advancedsciencenews.com

https://www.computerworld.com/

https://www.eurekalert.org/

https://www.green.earth/

https://www.iea.org/

https://www.ipcc.ch

https://www.livescience.com/

https://www.nationalgeographic.org/society/

https://www.nature.com/

https://www.ncbi.nlm.nih.gov/

https://www.networkworld.com/

https://www.newscientist.com/

https://www.npr.org/sections/science/

https://www.pcworld.com

https://www.pewresearch.org/topic/internet-technology/

https://www.pewresearch.org/topic/science/

https://www.popularmechanics.com/

https://www.science.org/

https://www.sciencealert.com/

https://www.sciencedaily.com/

https://www.scienceopen.com/

https://www.scientificamerican.com

https://www.triplepundit.com/

https://www.un.org/en/

https://www.un.org/en/climatechange

https://www.usgs.gov/programs/earthquake-hazards/

https://www.wwf.org.uk
# Appendix 2 : Prompts used for LLM Generation

## Zero-shot setting for Mistral, Vicuna, and Gemini :

Here is a sentence a at CEFR level {level} showing how you use the
{pos if verb/noun/adverb/adjective; else 'word' or 'expression'} "{item}"
({domain}) :

## One-shot setting for Gemini :

Please provide an example (between {lower} a and {upper} words at CEFR level {level} showing how you use the {pos if verb/noun/adverb/adjective; else 'word' or 'expression'} "{item}" ({domain}) :

Example : {reference_example}

<footnote>a. The reason for 'sentence' to be used rather than 'example', even though some of the reference examples consist of more than a single sentence, is that using 'example' tends to result in the rendition of extensive explanations instead of or in addition to an example of use. This problem does not persist with the one-shot setting, for which therefore the word 'example' is used instead.</footnote>

<footnote>a. 'Lower' and 'upper' denote a range of example lengths, which differs for the different CEFR levels (8 to 43 words for B1 and 20 to 87 words for B2). The ranges are defined as +/- 1.5 standard deviations from the average value per level. This value as well as the addition of information about length itself was decided upon following a process of trial and error based on the behaviour of 20 sample examples in comparison to the reference's counterparts.</footnote>
Appendix 3 : Features Used in Corpus Comparison

<table><tr><td>Length-Based</td><td>total number of examples in the sample<br/>total number of words in the sample<br/>average/min/max/SD number of words per sentence<br/>average/min/max/SD number of syllables per sentence<br/>average/min/max/SD number of letters per word<br/>average/min/max/SD number of syllables per word</td></tr><tr><td>Morphosyntactic</td><td>average/min/max/SD number of noun phrases per sentence<br/>average/min/max/SD number of non-stem words per s-ce<br/>percentage of sentences ending in question mark<br/>percentage of sentences ending in exclamation mark<br/>average/min/max/SD number of punctuation signs per s-ce<br/>morphological richness</td></tr><tr><td>Lexico-Semantic</td><td>average/min/max/SD number of verbs per sentence<br/>average/min/max/SD number of adj. and adv. per s-ce<br/>average/min/max/SD number of 1st-person pronouns per s-ce<br/>average/min/max/SD number of proper nouns per sentence<br/>percentage of words not present in the Dale-Chall list<br/>percentage of hapax legomena<br/>type-to-token ratio (word-based)<br/>type-to-token ratio (lemma-based)<br/>average concreteness<br/>10 most frequent words (excluding stop words)<br/>10 most frequent words (including stop words)</td></tr><tr><td>Discourse-Related</td><td>average/min/max/SD number of pronouns per sentence<br/>average/min/max/SD cosine distance between sentences<br/>average/min/max/SD % of anaphora-denoting words per sentence</td></tr></table>
Appendix 4 : Detailed Results

Entire Sample

<table><thead><tr><th>Feature</th><th>Reference</th><th>Web-Crawled</th><th>Mistral</th><th>Vicuna</th><th>Gemini</th><th>Gemini : one-shot</th></tr></thead><tbody><tr><td>Total # examples in sample</td><td>244</td><td>244</td><td>244</td><td>244</td><td>244</td><td>244</td></tr><tr><td>Total # words in sample</td><td>9823</td><td>7269</td><td>4615</td><td>4852</td><td>4160</td><td>10366</td></tr><tr><td>Avg. # words / s-ce</td><td>13.59</td><td>14.93***</td><td>9.34**</td><td>9.6**</td><td>8.51***</td><td>12.26***</td></tr><tr><td>Min. # words / s-ce</td><td>3</td><td>7</td><td>4</td><td>8</td><td>8</td><td>5</td></tr><tr><td>Max. # words / s-ce</td><td>58</td><td>69</td><td>36</td><td>30</td><td>36</td><td>44</td></tr><tr><td>SD # words / s-ce</td><td>8.59</td><td>11.42</td><td>4.84</td><td>5.04</td><td>4.61</td><td>5.77</td></tr><tr><td>Avg. # syllables / s-ce</td><td>21.52</td><td>24.47***</td><td>15.33</td><td>15.93</td><td>14.65*</td><td>20.47</td></tr><tr><td>Min. # syllables / s-ce</td><td>2</td><td>12</td><td>4</td><td>10</td><td>14</td><td>8</td></tr><tr><td>Max. # syllables / s-ce</td><td>93</td><td>120</td><td>63</td><td>68</td><td>57</td><td>84</td></tr><tr><td>SD # syllables / s-ce</td><td>14.82</td><td>20.5</td><td>8.77</td><td>9.51</td><td>8.61</td><td>10.97</td></tr><tr><td>Avg. # letters / word</td><td>5.29</td><td>5.38</td><td>5.3</td><td>5.43</td><td>5.6*</td><td>5.53</td></tr><tr><td>Min. # letters / word</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Max. # letters / word</td><td>21</td><td>23</td><td>15</td><td>17</td><td>17</td><td>20</td></tr><tr><td>SD # letters / word</td><td>2.89</td><td>3.01</td><td>2.94</td><td>3.02</td><td>3.07</td><td>3.01</td></tr><tr><td>Avg. # syllables / word</td><td>1.58</td><td>1.64*</td><td>1.64***</td><td>1.66***</td><td>1.72***</td><td>1.67***</td></tr><tr><td>Min. # syllables / word</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td></tr><tr><td>Max. # syllables / word</td><td>7</td><td>8</td><td>6</td><td>6</td><td>6</td><td>6</td></tr><tr><td>SD # syllables / word</td><td>0.91</td><td>0.99</td><td>0.94</td><td>0.96</td><td>1</td><td>0.96</td></tr></tbody></table>
<table><tbody><tr><td>Avg. # noun phrases / s-ce</td><td>5.87</td><td>8.16***</td><td>5.56</td><td>5.53</td><td>4.92***</td><td>5.39*</td></tr><tr><td>Min. # noun phrases / s-ce</td><td>0</td><td>2</td><td>1</td><td>2</td><td>2</td><td>1</td></tr><tr><td>Max. # noun phrases / s-ce</td><td>18</td><td>19</td><td>10</td><td>10</td><td>11</td><td>14</td></tr><tr><td>SD # noun phrases / s-ce</td><td>2.71</td><td>3.4</td><td>1.59</td><td>1.68</td><td>1.49</td><td>1.9</td></tr><tr><td>Avg. % non-stem words / s-ce</td><td>33.56</td><td>31.56***</td><td>35.14***</td><td>35.59***</td><td>36.36***</td><td>36***</td></tr><tr><td>Min. % non-stem words / s-ce</td><td>0</td><td>0</td><td>11.76</td><td>11.11</td><td>9.1</td><td>6.25</td></tr><tr><td>Max. % non-stem words / s-ce</td><td>70</td><td>56.25</td><td>65</td><td>70</td><td>64.29</td><td>83.33</td></tr><tr><td>SD % non-stem words / s-ce</td><td>11.17</td><td>9.18</td><td>9.82</td><td>10.07</td><td>9.95</td><td>11.15</td></tr><tr><td>% s-ces ending in “?”</td><td>0.63</td><td>1.62</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>% s-ces ending in “!”</td><td>0.42</td><td>0</td><td>0.4</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Avg. # punct. signs / s-ce</td><td>1.56</td><td>2.7</td><td>0.77***</td><td>0.99***</td><td>0.75***</td><td>1.25*</td></tr><tr><td>Min. # punct. signs / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # punct. signs / s-ce</td><td>6</td><td>7</td><td>4</td><td>3</td><td>4</td><td>6</td></tr><tr><td>SD # punct. signs / s-ce</td><td>0.55</td><td>0.77</td><td>0.32</td><td>0.35</td><td>0.32</td><td>0.48</td></tr><tr><td>Morphological richness</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td></tr><tr><td>Avg. # verbs / s-ce</td><td>2.45</td><td>3.83</td><td>2.54***</td><td>2.44***</td><td>2.27***</td><td>2.53</td></tr><tr><td>Min. # verbs / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # verbs / s-ce</td><td>8</td><td>11</td><td>7</td><td>7</td><td>5</td><td>8</td></tr><tr><td>SD # verbs / s-ce</td><td>1.53</td><td>1.84</td><td>1.08</td><td>1.26</td><td>1.06</td><td>1.14</td></tr><tr><td>Avg. # adj. and adv. / s-ce</td><td>2.96</td><td>4.13***</td><td>2.21***</td><td>2.31***</td><td>2.29***</td><td>2.52**</td></tr><tr><td>Min. # adj. and adv. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>
<table><tbody><tr><td>Max. # adj. and adv. / s-ce</td><td>10</td><td>14</td><td>10</td><td>7</td><td>8</td><td>9</td></tr><tr><td>SD # adj. and adv. / s-ce</td><td>1.97</td><td>2.62</td><td>1.48</td><td>1.46</td><td>1.41</td><td>1.56</td></tr><tr><td>Avg. # 1st-person pron. / s-ce</td><td>0.1</td><td>0.12</td><td>0.39***</td><td>0.11</td><td>0.06**</td><td>0.07*</td></tr><tr><td>Min. # 1st-person pron. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # 1st-person pron. / s-ce</td><td>2</td><td>4</td><td>4</td><td>2</td><td>3</td><td>4</td></tr><tr><td>SD # 1st-person pron. / s-ce</td><td>0.38</td><td>0.41</td><td>0.75</td><td>0.38</td><td>0.3</td><td>0.36</td></tr><tr><td>Avg. # proper nouns / s-ce</td><td>0.9</td><td>1.46</td><td>0.06***</td><td>0.32***</td><td>0.1***</td><td>0.27***</td></tr><tr><td>Min. # proper nouns / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # proper nouns / s-ce</td><td>14</td><td>12</td><td>3</td><td>15</td><td>7</td><td>10</td></tr><tr><td>SD # proper nouns / s-ce</td><td>1.84</td><td>2.44</td><td>0.31</td><td>1.56</td><td>0.54</td><td>0.94</td></tr><tr><td>% words not in Dale-Chall list</td><td>45.21</td><td>45.38</td><td>42.25</td><td>45.22</td><td>48.34</td><td>47.41</td></tr><tr><td>% hapax legomena</td><td>16.13</td><td>22.56</td><td>16.25</td><td>18.18</td><td>19.75</td><td></td></tr><tr><td>Type-to-token ratio (words)</td><td>0.26</td><td>0.33</td><td>0.27</td><td>0.29</td><td>0.3</td><td>14.14</td></tr><tr><td>Type-to-token ratio (lemmas)</td><td>0.25</td><td>0.31</td><td>0.25</td><td>0.27</td><td>0.29</td><td>0.23</td></tr><tr><td>Average concreteness</td><td>2.46</td><td>2.36</td><td>2.44</td><td>2.44</td><td>2.42</td><td>2.41</td></tr><tr><td>10 most frequent words (excl. stop words)</td><td>water, change, species, world, people, new, plants, global, could</td><td>climate, would, could, said, people, water, also, international, may, must, new</td><td>crop, soil, crops, farmers, scientists, agriculture, water, yields, new, order</td><td>soil, crop, farmers, agriculture, crops, yields, practices, water, agricultural, climate</td><td>crop, soil, new, scientists, farmers, crops, practices, yields, scientist, sustainable</td><td>water, soil, farmers, crops, crop, plant, species, food, practices, yields</td></tr><tr><td>10 most frequent words (incl. stop words)</td><td>the, of, and, to, in, a, is, that, are, for</td><td>the, of, and, to, in, a, that, for, be, is</td><td>the, to, of, and, in, a, that, I, for, can</td><td>to, the, of, and, in, a, that, is, can, as the, of, to, a, and, in, for, crop, soil</td><td>the, of, to, a, and, in, for, crop, soil</td><td>the, of, to, and, in, a, for, is, that, are</td></tr><tr><td>Avg pron. / s-ce</td><td>0.88</td><td>1.27</td><td>1.06**</td><td>0.8**</td><td>0.5***</td><td>0.73*</td></tr><tr><td>Min. pron. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. pron. / s-ce</td><td>5</td><td>6</td><td>5</td><td>6</td><td>4</td><td>4</td></tr></tbody></table>
<table><tr><td>SD pron. / s-ce</td><td>1.03</td><td>1.33</td><td>1.1</td><td>0.92</td><td>0.78</td><td><strong>0.93</strong></td></tr><tr><td>Avg. % anaphora</td><td>20.49</td><td>10.78</td><td>10.47</td><td>10.43</td><td>13.42***</td><td><strong>24.59</strong></td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Min. % anaphora</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td><strong>0</strong></td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Max. % anaphora</td><td>42.86</td><td>30</td><td>37.5</td><td>30.77</td><td>30.77</td><td><strong>31.25</strong></td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD % anaphora</td><td>6.55</td><td>5.64</td><td>6.97</td><td>6.53</td><td>6.93</td><td><strong>6.8</strong></td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Avg. cos. distance</td><td>0.14</td><td>0.1***</td><td>0.18***</td><td>0.17***</td><td>0.18***</td><td><strong>0.15*</strong></td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Min. cos. distance</td><td>-0.25</td><td>-0.25</td><td>-0.25</td><td>-0.32</td><td>-0.26</td><td><strong>-0.27</strong></td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Max. cos. distance</td><td>0.72</td><td>0.78</td><td>0.85</td><td>0.9</td><td>0.87</td><td><strong>0.89</strong></td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD cos. distance</td><td>0.13</td><td>0.12</td><td>0.17</td><td>0.16</td><td>0.16</td><td><strong>0.15</strong></td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></table>

Comparison between the investigated corpora based on a sample of textual features. Features in *italics* have been tested for statistical significance, and the extent of the significance is marked with *, ** and *** from lowest to highest. The one-shot Gemini corpus is marked with **bold** to denote its highest global similarity to the reference corpus.
Per Level : B1 (domain "Science")

<table><thead><tr><th>Feature</th><th>Reference</th><th>Web-Crawled</th><th>Mistral</th><th>Vicuna</th><th>Gemini</th><th>Combined LLM</th></tr></thead><tbody><tr><td>Total # examples in sample</td><td>132</td><td>132</td><td>132</td><td>132</td><td>132</td><td>132</td></tr><tr><td>Total # words in sample</td><td>3402</td><td>3987</td><td>2358</td><td>2421</td><td>1976</td><td>2824</td></tr><tr><td>Avg. # words / s-ce</td><td>11.01</td><td>15.1***</td><td>8.8*</td><td>9.03</td><td>7.46***</td><td>9***</td></tr><tr><td>Min. # words / s-ce</td><td>3</td><td>7</td><td>9</td><td>8</td><td>8</td><td>5</td></tr><tr><td>Max. # words / s-ce</td><td>42</td><td>69</td><td>31</td><td>20</td><td>33</td><td>38</td></tr><tr><td>SD # words / s-ce</td><td>7.52</td><td>12.03</td><td>4.44</td><td>5.27</td><td>3.51</td><td>5.33</td></tr><tr><td>Avg. # syllables / s-ce</td><td>17.81</td><td>24.62***</td><td>14.66</td><td>14.93</td><td>12.72*</td><td>14.71***</td></tr><tr><td>Min. # syllables / s-ce</td><td>2</td><td>12</td><td>10</td><td>11</td><td>14</td><td>8</td></tr><tr><td>Max. # syllables / s-ce</td><td>83</td><td>120</td><td>63</td><td>56</td><td>48</td><td>65</td></tr><tr><td>SD # syllables / s-ce</td><td>13.93</td><td>20.85</td><td>8.37</td><td>9.65</td><td>6.39</td><td>9.78</td></tr><tr><td>Avg. # letters / word</td><td>3.56</td><td>5.35</td><td>5.27</td><td>5.35</td><td>5.45</td><td>5.35</td></tr><tr><td>Min. # letters / word</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Max. # letters / word</td><td>20</td><td>19</td><td>15</td><td>16</td><td>17</td><td>20</td></tr><tr><td>SD # letters / word</td><td>2.95</td><td>3</td><td>3.01</td><td>2.99</td><td>3.05</td><td>3</td></tr><tr><td>Avg. # syllables / word</td><td>1.62</td><td>1.63</td><td>1.67</td><td>1.65</td><td>1.71*</td><td>1.64*</td></tr><tr><td>Min. # syllables / word</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td></tr><tr><td>Max. # syllables / word</td><td>7</td><td>8</td><td>6</td><td>6</td><td>6</td><td>6</td></tr><tr><td>SD # syllables / word</td><td>0.95</td><td>0.97</td><td>0.96</td><td>0.95</td><td>1</td><td>0.95</td></tr></tbody></table>
<table>
  <tbody>
    <tr>
      <td>Avg. # noun phrases / s-ce</td>
      <td>5.51</td>
      <td>8.01***</td>
      <td>5.26</td>
      <td>5.3</td>
      <td>4.38***</td>
      <td>5.14**</td>
    </tr>
    <tr>
      <td>Min. # noun phrases / s-ce</td>
      <td>0</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <td>Max. # noun phrases / s-ce</td>
      <td>14</td>
      <td>19</td>
      <td>9</td>
      <td>10</td>
      <td>8</td>
      <td>11</td>
    </tr>
    <tr>
      <td>SD # noun phrases / s-ce</td>
      <td>2.49</td>
      <td>3.5</td>
      <td>1.38</td>
      <td>1.74</td>
      <td>1.19</td>
      <td>1.87</td>
    </tr>
    <tr>
      <td>Avg. % non-stem words / s-ce</td>
      <td>34.3</td>
      <td>31.01</td>
      <td>34.37***</td>
      <td>34.41***</td>
      <td>34.72***</td>
      <td>34.1</td>
    </tr>
    <tr>
      <td>Min. % non-stem words / s-ce</td>
      <td>0</td>
      <td>0</td>
      <td>11.76</td>
      <td>11.11</td>
      <td>9.09</td>
      <td>8.33</td>
    </tr>
    <tr>
      <td>Max. % non-stem words / s-ce</td>
      <td>61.54</td>
      <td>56.25</td>
      <td>61.54</td>
      <td>70</td>
      <td>57.14</td>
      <td>55</td>
    </tr>
    <tr>
      <td>SD % non-stem words / s-ce</td>
      <td>10.61</td>
      <td>9.63</td>
      <td>10.1</td>
      <td>10.31</td>
      <td>9.35</td>
      <td>9.95</td>
    </tr>
    <tr>
      <td>% s-ces ending in “?”</td>
      <td>0</td>
      <td>2.22</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <td>% s-ces ending in “!”</td>
      <td>1.14</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <td>Avg. # punct. signs / s-ce</td>
      <td>1.33</td>
      <td>2.71***</td>
      <td>0.63***</td>
      <td>0.88**</td>
      <td>0.46***</td>
      <td>0.91***</td>
    </tr>
    <tr>
      <td>Min. # punct. signs / s-ce</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <td>Max. # punct. signs / s-ce</td>
      <td>6</td>
      <td>7</td>
      <td>3</td>
      <td>3</td>
      <td>3</td>
      <td>6</td>
    </tr>
    <tr>
      <td>SD # punct. signs / s-ce</td>
      <td>0.54</td>
      <td>0.79</td>
      <td>0.28</td>
      <td>0.33</td>
      <td>0.24</td>
      <td>0.4</td>
    </tr>
    <tr>
      <td>Morphological richness</td>
      <td>0.02</td>
      <td>0.02</td>
      <td>0.02</td>
      <td>0.02</td>
      <td>0.02</td>
      <td>0.02</td>
    </tr>
    <tr>
      <td>Avg. # verbs / s-ce</td>
      <td>2.17</td>
      <td>3.84***</td>
      <td>2.41</td>
      <td>2.35</td>
      <td>2.03***</td>
      <td>2.21</td>
    </tr>
    <tr>
      <td>Min. # verbs / s-ce</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <td>Max. # verbs / s-ce</td>
      <td>7</td>
      <td>11</td>
      <td>7</td>
      <td>7</td>
      <td>4</td>
      <td>7</td>
    </tr>
    <tr>
      <td>SD # verbs / s-ce</td>
      <td>1.39</td>
      <td>1.95</td>
      <td>1.03</td>
      <td>1.25</td>
      <td>0.86</td>
      <td>1.23</td>
    </tr>
    <tr>
      <td>Avg. # adj. and adv. / s-ce</td>
      <td>2.78</td>
      <td>4.09</td>
      <td>2.04***</td>
      <td>2.13***</td>
      <td>1.97***</td>
      <td>2.07***</td>
    </tr>
    <tr>
      <td>Min. # adj. and adv. / s-ce</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
<table><tbody><tr><td>Max. # adj. and adv. / s-ce</td><td>9</td><td>14</td><td>10</td><td>7</td><td>6</td><td>6</td></tr><tr><td>SD # adj. and adv. / s-ce</td><td>1.81</td><td>2.67</td><td>1.45</td><td>1.43</td><td>1.22</td><td>1.32</td></tr><tr><td>Avg. # 1st-person pron. / s-ce</td><td>0.12</td><td>0.16</td><td>0.55***</td><td>0.14</td><td>0.1</td><td>0.08</td></tr><tr><td>Min. # 1st-person pron. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # 1st-person pron. / s-ce</td><td>2</td><td>4</td><td>4</td><td>2</td><td>3</td><td>2</td></tr><tr><td>SD # 1st-person pron. / s-ce</td><td>0.42</td><td>0.51</td><td>0.88</td><td>0.41</td><td>0.39</td><td>0.38</td></tr><tr><td>Avg. # proper nouns / s-ce</td><td>0.71</td><td>1.64*</td><td>0.1***</td><td>0.36***</td><td>0.11***</td><td>0.2***</td></tr><tr><td>Min. # proper nouns / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # proper nouns / s-ce</td><td>7</td><td>12</td><td>3</td><td>15</td><td>2</td><td>5</td></tr><tr><td>SD # proper nouns / s-ce</td><td>1.34</td><td>2.72</td><td>0.38</td><td>1.45</td><td>0.36</td><td>0.69</td></tr><tr><td>% words not in Dale-Chall list</td><td>46.06</td><td>45.78</td><td>41.65</td><td>44.4</td><td>46.51</td><td>45.56</td></tr><tr><td>% hapax legomena</td><td>24.06</td><td>29.82</td><td>23.3</td><td>26.57</td><td>28.11</td><td>26.84</td></tr><tr><td>Type-to-token ratio (words)</td><td>0.36</td><td>0.4</td><td>0.35</td><td>0.38</td><td>0.39</td><td>0.38</td></tr><tr><td>Type-to-token ratio (lemmas)</td><td>0.34</td><td>0.38</td><td>0.33</td><td>0.35</td><td>0.37</td><td>0.36</td></tr><tr><td>Average concreteness</td><td>2.47</td><td>2.39</td><td>2.38</td><td>2.41</td><td>2.37</td><td>2.41</td></tr><tr><td>10 most frequent words (excl. stop words)</td><td>climate, change, water, people, areas, earthquake, greenhouse, earth, world, atmosphere, the, of, and, in, to, a, is, are, as, can</td><td>would, could, time, may, said, water, help, new, using, must</td><td>scientists, science, study, new, experiment, behavior, scientific, research, climate, understand</td><td>scientists, scientific, research, study, field, climate, science, discovery, certain, understand</td><td>scientists, new, scientist, experiment, earth, research, data, chemical, scientific, study</td><td>water, experiment, scientists, chemical, new, used, species, different, each, causing</td></tr><tr><td>10 most frequent words (incl. stop words)</td><td>Avg. pron. / s-ce</td><td>0.67</td><td>1.27**</td><td>1.25***</td><td>0.77</td><td>0.47*</td><td>0.65*</td></tr><tr><td>Min. pron. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. pron. / s-ce</td><td>5</td><td>6</td><td>5</td><td>4</td><td>3</td><td>4</td></tr></tbody></table>
<table><tr><td>SD pron. / s-ce</td><td>0.99</td><td>1.39</td><td>1.11</td><td>0.91</td><td>0.69</td><td>0.91</td></tr><tr><td>Avg. % anaphora</td><td>12.95</td><td>10.63</td><td>11.7*</td><td>11.41</td><td>15.9***</td><td>16.53*</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Min. % anaphora</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Max. % anaphora</td><td>27.27</td><td>30</td><td>37.5</td><td>30.77</td><td>30.77</td><td>31.25</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD % anaphora</td><td>6.41</td><td>5.73</td><td>6.82</td><td>6.76</td><td>7.05</td><td>6.59</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Avg. cos. distance</td><td>0.13</td><td>0.09***</td><td>0.14***</td><td>0.13</td><td>0.16***</td><td>0.11***</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Min. cos. distance</td><td>-0.22</td><td>-0.25</td><td>-0.25</td><td>-0.25</td><td>-0.26</td><td>-0.27</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Max. cos. distance</td><td>0.71</td><td>0.64</td><td>0.84</td><td>0.8</td><td>0.87</td><td>0.78</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD cos. distance</td><td>0.14</td><td>0.11</td><td>0.14</td><td>0.13</td><td>0.14</td><td>0.11</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></table>
Per Level : B2 (domain "Agronomy")

<table><thead><tr><th>Feature</th><th>Reference</th><th>Web-Crawled</th><th>Mistral</th><th>Vicuna</th><th>Gemini</th><th>Gemini : one-shot</th></tr></thead><tbody><tr><td>Total # examples in sample</td><td>112</td><td>112</td><td>112</td><td>112</td><td>112</td><td>112</td></tr><tr><td>Total # words in sample</td><td>6421</td><td>3282</td><td>2257</td><td>2431</td><td>2184</td><td>7542</td></tr><tr><td>Avg. # words / s-ce</td><td>15.51</td><td>14.72***</td><td>9.99</td><td>10.26</td><td>9.75</td><td>14.21***</td></tr><tr><td>Min. # words / s-ce</td><td>3</td><td>11</td><td>4</td><td>9</td><td>11</td><td>7</td></tr><tr><td>Max. # words / s-ce</td><td>58</td><td>64</td><td>36</td><td>30</td><td>36</td><td>44</td></tr><tr><td>SD # words / s-ce</td><td>9.09</td><td>10.69</td><td>5.02</td><td>4.64</td><td>4.57</td><td>5.85</td></tr><tr><td>Avg. # syllables / s-ce</td><td>24.29</td><td>24.29***</td><td>16.12</td><td>17.06</td><td>16.94</td><td>23.91</td></tr><tr><td>Min. # syllables / s-ce</td><td>3</td><td>16</td><td>4</td><td>10</td><td>19</td><td>9</td></tr><tr><td>Max. # syllables / s-ce</td><td>93</td><td>106</td><td>57</td><td>68</td><td>57</td><td>84</td></tr><tr><td>SD # syllables / s-ce</td><td>15.28</td><td>20.16</td><td>9.02</td><td>9.15</td><td>8.72</td><td>11.22</td></tr><tr><td>Avg. # letters / word</td><td>5.26</td><td>5.4</td><td>5.33</td><td>5.5*</td><td>5.75***</td><td>5.6***</td></tr><tr><td>Min. # letters / word</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Max. # letters / word</td><td>21</td><td>23</td><td>15</td><td>17</td><td>17</td><td>20</td></tr><tr><td>SD # letters / word</td><td>2.85</td><td>3.01</td><td>2.86</td><td>3.05</td><td>3.09</td><td>3.03</td></tr><tr><td>Avg. # syllables / word</td><td>1.57</td><td>1.65**</td><td>1.61*</td><td>1.66***</td><td>1.74***</td><td>1.68***</td></tr><tr><td>Min. # syllables / word</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td></tr><tr><td>Max. # syllables / word</td><td>7</td><td>8</td><td>5</td><td>5</td><td>6</td><td>6</td></tr><tr><td>SD # syllables / word</td><td>0.89</td><td>1.02</td><td>0.91</td><td>0.97</td><td>1</td><td>0.97</td></tr></tbody></table>
<table><tbody><tr><td>Avg. # noun phrases / s-ce</td><td>6.09</td><td>8.34***</td><td>5.91</td><td>5.78</td><td>5.57</td><td>5.49</td></tr><tr><td>Min. # noun phrases / s-ce</td><td>0</td><td>3</td><td>1</td><td>2</td><td>2</td><td>1</td></tr><tr><td>Max. # noun phrases / s-ce</td><td>18</td><td>17</td><td>10</td><td>9</td><td>11</td><td>14</td></tr><tr><td>SD # noun phrases / s-ce</td><td>2.81</td><td>3.28</td><td>1.74</td><td>1.58</td><td>1.55</td><td>1.91</td></tr><tr><td>Avg. % non-stem words / s-ce</td><td>31.64</td><td>32.22***</td><td>35.95***</td><td>36.77***</td><td>37.83***</td><td>36.71***</td></tr><tr><td>Min. % non-stem words / s-ce</td><td>0</td><td>14.71</td><td>12.5</td><td>13.04</td><td>15.79</td><td>6.25</td></tr><tr><td>Max. % non-stem words / s-ce</td><td>70</td><td>56.25</td><td>65</td><td>61.11</td><td>64.29</td><td>83.33</td></tr><tr><td>SD % non-stem words / s-ce</td><td>11.41</td><td>8.61</td><td>9.45</td><td>9.71</td><td>10.41</td><td>11.51</td></tr><tr><td>% s-ces ending in “?”</td><td>1</td><td>0.89</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>% s-ces ending in “!”</td><td>0</td><td>0</td><td>0.87</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Avg. # punct. signs / s-ce</td><td>1.7</td><td>2.69***</td><td>0.93***</td><td>1.1***</td><td>1.1***</td><td>1.4</td></tr><tr><td>Min. # punct. signs / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # punct. signs / s-ce</td><td>6</td><td>5</td><td>4</td><td>3</td><td>4</td><td>6</td></tr><tr><td>SD # punct. signs / s-ce</td><td>0.56</td><td>0.76</td><td>0.36</td><td>0.38</td><td>0.39</td><td>0.51</td></tr><tr><td>Morphological richness</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td><td>0.02</td></tr><tr><td>Avg. # verbs / s-ce</td><td>2.61</td><td>3.81***</td><td>2.7***</td><td>2.54***</td><td>2.56***</td><td>2.67*</td></tr><tr><td>Min. # verbs / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # verbs / s-ce</td><td>8</td><td>10</td><td>6</td><td>6</td><td>5</td><td>8</td></tr><tr><td>SD # verbs / s-ce</td><td>1.58</td><td>1.69</td><td>1.13</td><td>1.27</td><td>1.2</td><td>1.5</td></tr><tr><td>Avg. # adj. and adv. / s-ce</td><td>3.07</td><td>4.17***</td><td>2.42***</td><td>2.5***</td><td>2.67***</td><td>2.71</td></tr><tr><td>Min. # adj. and adv. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>
<table><tbody><tr><td>Max. # adj. and adv. / s-ce</td><td>10</td><td>14</td><td>8</td><td>6</td><td>8</td><td>9</td></tr><tr><td>SD # adj. and adv. / s-ce</td><td>2.04</td><td>2.58</td><td>1.5</td><td>1.48</td><td>1.53</td><td>1.62</td></tr><tr><td>Avg. # 1st-person pron. / s-ce</td><td>0.09</td><td>0.06*</td><td>0.2</td><td>0.08*</td><td>0.02***</td><td>0.06*</td></tr><tr><td>Min. # 1st-person pron. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # 1st-person pron. / s-ce</td><td>2</td><td>1</td><td>2</td><td>2</td><td>1</td><td>4</td></tr><tr><td>SD # 1st-person pron. / s-ce</td><td>0.35</td><td>0.24</td><td>0.5</td><td>0.35</td><td>0.13</td><td>0.35</td></tr><tr><td>Avg. # proper nouns / s-ce</td><td>1.05</td><td>1.24**</td><td>0.02***</td><td>0.28***</td><td>0.11***</td><td>0.3***</td></tr><tr><td>Min. # proper nouns / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. # proper nouns / s-ce</td><td>14</td><td>10</td><td>2</td><td>15</td><td>7</td><td>10</td></tr><tr><td>SD # proper nouns / s-ce</td><td>2.07</td><td>2.04</td><td>0.19</td><td>1.67</td><td>0.7</td><td>1.03</td></tr><tr><td>% words not in Dale-Chall list</td><td>44.76</td><td>44.91</td><td>42.89</td><td>46.03</td><td>50</td><td>48.11</td></tr><tr><td>% hapax legomena</td><td>20.15</td><td>27.51</td><td>18.91</td><td>21.35</td><td>21.89</td><td>15.77</td></tr><tr><td>Type-to-token ratio (words)</td><td>0.31</td><td>0.39</td><td>0.3</td><td>0.32</td><td>0.33</td><td>0.27</td></tr><tr><td>Type-to-token ratio (lemmas)</td><td>0.29</td><td>0.37</td><td>0.29</td><td>0.31</td><td>0.31</td><td>0.25</td></tr><tr><td>Average concreteness</td><td>2.45</td><td>2.36</td><td>2.51</td><td>2.48</td><td>2.47</td><td>2.41</td></tr><tr><td>10 most common words (excl. stop words)</td><td>water, species, climate, change, could, plants, world, many, help, new</td><td>could, people, said, would, also, international, humanitarian, countries, must, called</td><td>crop, soil, crops, farmers, agriculture, yields, water, farmer, yield, conditions</td><td>crop, soil, farmers, agriculture, crops, yields, practices, agricultural, sustainable, water</td><td>crop, soil, farmers, crops, practices, yields, sustainable, agricultural, agronomist, farmer</td><td>water, soil, farmers, crop, crops, yields, plant, farming, practices, food</td></tr><tr><td>10 most frequent words (incl. stop words)</td><td>the, of, to, and, in, a, is, that, are, for</td><td>the, and, of, to, in, that, a, for, it, with</td><td>the, to, and, of, in, crop, soil, crops, a, can</td><td>to, the, and, in, of, crop, soil, can, that, are</td><td>the, to, of, and, in, a, for, crop, soil, farmers</td><td>the, end, to, of, a, in, for, that, their, can</td></tr><tr><td>Avg pron. / s-ce</td><td>1</td><td>1.27***</td><td>0.84***</td><td>0.84***</td><td>0.54***</td><td>0.77</td></tr><tr><td>Min. pron. / s-ce</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Max. pron. / s-ce</td><td>4</td><td>5</td><td>5</td><td>5</td><td>4</td><td>4</td></tr></tbody></table>
<table><tr><td>SD pron. / s-ce</td><td>1.03</td><td>1.27</td><td>1.06</td><td>0.93</td><td>0.88</td><td>0.94</td></tr><tr><td>Avg. % anaphora</td><td>10.5</td><td>10.97</td><td>9.01***</td><td>9.27***</td><td>10.51</td><td>10.75</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Min. % anaphora</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Max. % anaphora</td><td>42.86</td><td>27.27</td><td>30.77</td><td>27.27</td><td>25</td><td>30</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD % anaphora</td><td>6.6</td><td>5.53</td><td>6.91</td><td>5.96</td><td>5.53</td><td>6.58</td></tr><tr><td>words / s-ce</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Avg. cos. distance</td><td>0.16</td><td>0.12***</td><td>0.38***</td><td>0.32***</td><td>0.35***</td><td>0.24***</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Min. cos. distance</td><td>-0.21</td><td>-0.22</td><td>-0.05</td><td>-0.22</td><td>-0.06</td><td>-0.17</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Max. cos. distance</td><td>0.72</td><td>0.78</td><td>0.85</td><td>0.89</td><td>0.87</td><td>0.89</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>SD cos. distance</td><td>0.13</td><td>0.13</td><td>0.13</td><td>0.17</td><td>0.14</td><td>0.16</td></tr><tr><td>btwn s-ces</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></table>
# La reconnaissance automatique des relations de cohérence RST en français

Martial Pastor¹ Erik Bran Marino² Nelleke Oostdijk¹

(1) Centre for Language Studies, Radboud University, 6525 HT Nimègue, Pays-Bas

(2) CIDEHUS, Universidade de Évora, 7004-516 Évora, Portugal

{martial.pastor|nelleke.oostdijk}@ru.nl, erik.marino@uevora.pt

## RÉSUMÉ

Les parseurs de discours ont suscité un intérêt considérable dans les récentes applications de traitement automatique du langage naturel. Cette approche dépasse les limites traditionnelles de la phrase et peut s'étendre pour englober l'identification de relation de discours. Il existe plusieurs parseurs spécialisés dans le traitement automatique du discours, mais ces derniers ont été principalement évalués sur des corpus anglais. Par conséquent, il n'est pas évident de bien cerner les éléments linguistiques importants sur lesquels les parseurs se basent pour classifier les relations de discours en dehors de l'anglais. Cet article évalue les performances du parseur DMRST sur le corpus RST-DT traduit en français. Nous constatons que les performances de classification des relations de discours en français sont comparables à celles obtenues pour d'autres langues. En analysant les succès et échecs de la classification des relations, nous soulignons l'impact des marqueurs de discours et des structures syntaxiques sur la précision du parseur.

## ABSTRACT

RST relation label classification for French.

Discourse parsers have attracted considerable interest in recent natural language processing applications. This approach goes beyond the conventional scope of sentences and can extend to encompass discourse relation identification. There are several parsers specialized in Discourse Parsing, but these have primarily been evaluated on English corpora. Consequently, it is not straightforward to capture the important linguistic elements upon which parsers rely to classify discourse relations outside of English. This article evaluates the performance of the DMRST parser on the RST-DT corpus translated into French. We find that parser performance for the classification of coherence relations in French is comparable to those obtained for other languages. By analyzing the successes and failures of relation classification, we highlight the impact of discourse markers and syntactic structures on the parser's accuracy.

**MOTS-CLÉS** : parseurs de discours, rhetorical structure theory (RST), relations de discours, traitement automatique du discours, marqueurs de discours, structures syntaxiques.

**KEYWORDS**: discourse parsers, rhetorical structure theory (RST), discourse relations, discourse parsing, discourse markers, syntactic structures.

<figure><img src="image_150.png" /></figure>
# 1 Introduction

Les parseurs de discours ont suscité un intérêt considérable dans les récentes applications de traitement automatique du langage naturel (Chernyavskiy et al., 2024; Braud et al., 2023). Cette tâche consiste à identifier les relations qu'entretiennent des unités de textes à l'intérieur d'un texte plus large. Cette approche dépasse les limites traditionnelles de la phrase et peut s'étendre pour englober l'identification des Relations de Cohérence au niveau du discours. L'un des formalismes les plus populaires pour représenter les Relations de Cohérence est la Rhetorical Structure Theory (RST; Mann & Thompson, 1988), qui a encouragé la création de jeux de données maintenant utilisés pour l'analyse automatique du discours. Cette dernière tâche est complexe et les parseurs de discours n'ont pas atteint le même niveau de succès que pour d'autres tâches au niveau de la phrase.

Il existe plusieurs parseurs spécialisés dans les structures de type RST (Nguyen et al., 2021, Guz & Carenini, 2020), mais ces derniers ont été principalement entraînés sur le corpus anglais RST-DT (Carlson et al., 2003). Par conséquent, en ce qui concerne les questions d'explicabilité des parseurs orientés deep learning, il n'est pas évident de bien cerner les éléments linguistiques importants sur lesquels les parseurs se basent pour classifier les relations de discours en dehors de l'anglais. Toutefois, des corpus RST sont disponibles pour d'autres langues que l'anglais, ce qui a donné lieu au développement de parseurs multilingues. Cela dit, à notre connaissance, l'analyse des performances des parseurs et les questions d'explicabilité restent limitées à des analyses effectuées sur de l'anglais. Par exemple, certaines études montrent que l'efficacité des parseurs dépend de la présence de marqueurs de discours <fnref n="1" /> (voir exemple (1) <fnref n="2" />) qui rendrait la classification plus facile pour les relations de discours signalées par ces derniers (Pitler et al., 2008).

(1) « [If I sell now,] $\frac{\text{pred} : \text{condition}}{\text{gold} : \text{condition}} \rightarrow$ [I'll take a big loss.] » wsj_2386

Par ailleurs, selon les analyses effectuées sur des parseurs plus récents (Liu et al., 2023), il semble que certaines structures syntaxiques, comme des *modifications nominales* (2), jouent un rôle déterminant dans la capacité des parseurs à identifier de manière plus efficace les relations de discours.

(2) « [Negotiable, bank-backed business credit instruments] $\leftarrow\frac{\text{pred} : \text{elaboration}}{\text{gold} : \text{elaboration}}$ [typically financing an import order.] » wsj_0602

Cet article propose une évaluation puis une analyse des réussites et des échecs du parseur DMRST sur le corpus RST-DT traduit en français. Nous commençons par reproduire les expériences avec le parseur multilingue DMRST, que nous évaluons ensuite sur le corpus test du RST-DT. Nous constatons initialement que les performances en termes de classification des relations de cohérence en français sont comparables à celles obtenues pour d'autres langues. Ensuite, nous procédons à une analyse qualitative des cas de succès et d'échecs dans la classification des relations, en mettant en lumière des cas de figure où les relations sont signalées soit par des marqueurs de discours, soit

<footnote n="1">1. La définition à laquelle nous faisons référence dans cet article concerne les *Discourse Markers* en anglais, qui sont des connecteurs, généralement des conjonctions, entre propositions indépendantes.</footnote>

<footnote n="2">2. "pred" ici correspond à l'étiquette prédite par le parseur DMRST présenté dans la section 2, et "gold" correspond à l'étiquette annotée dans le jeu de données RST-DT. La direction de la flèche pointe vers le noyau. La différenciation entre le noyau et le satellite repose sur le fait qu'une unité textuelle agissant comme satellite peut être retirée sans altérer la cohérence du discours, tandis que le retrait d'une unité textuelle agissant comme noyau rendrait le texte incohérent (Mann & Thompson, 1988).</footnote>
par des structures syntaxiques. Les résultats révèlent que certains signaux syntaxiques facilitent la classification, tandis que certains marqueurs de discours contribuent à une confusion accrue entre différents types de relations.

## 2 Méthodologie

**La Rhetorical Structure Theory.** La RST est un modèle d'analyse textuelle des relations de cohérence. Avec celui-ci, les relations entre les segments de texte sont annotées avec différentes classes de relations de cohérence telles que l'élaboration, le contraste, le causal, le temporel, etc. (voir Table 2 pour une liste complète des relations utilisées dans cet article). Les segments de texte d'un arbre RST sont des "unités discursives élémentaires" (EDUs), qui sont des ensembles contigus de tokens approximativement similaires à des propositions indépendantes. Les relations se font non seulement entre les segments de texte mais aussi entre des groupes de segments de texte, ce qui signifie que la représentation finale de RST d'un texte complet (livre, chapitre, article, commentaire, etc.) est un arbre hiérarchique de segments de texte connectés par des relations de cohérence comme sur la Figure 1.

<figure><img src="image_151.png" /><figcaption>FIGURE 1 – Analyse RST traduite en français issue du document wsj_609 dans le corpus RST-DT qui décrit le modèle d'analyse textuelle des relations de cohérence.</figcaption></figure>

**Le corpus RST Discourse Treebank.** Le corpus RST-DT (Carlson et al., 2003) a été largement utilisé pour l'analyse RST en anglais et est devenu un choix standard pour évaluer les parseurs RST. Il est reconnu pour ses structures arborescentes hiérarchiques et a été initialement annoté avec 76 relations de cohérence. Les relations examinées ici proviennent de l'ensemble de test RST-DT, qui contient un total de 38 documents, comprenant environ 21 600 tokens. En ce qui concerne les étiquettes de relations, nous utilisons actuellement l'ensemble harmonisé de 18 étiquettes (Braud et al., 2017).

**Le parseur multilingue DMRST.** Le parseur DMRST développé par Liu et al., 2021 est basé sur XLM-ROBERTA-BASE (Conneau et al., 2020) et est un système multilingue TOP-DOWN qui gère simultanément la segmentation des EDUs et l'analyse des arbres RST. Sa pertinence pour notre étude réside dans ses performances à l'état de l'art en matière de classification des relations de cohérence.
Les auteurs ont fourni un accès à un modèle optimisé pour l'inférence disponible en ligne<fnref n="3" />. Ce modèle particulier a été entraîné sur une collection multilingue de corpus RST, offrant une prise en charge native pour six langues : l'anglais, le portugais, l'espagnol, l'allemand, le néerlandais et le basque. Il est important de noter que, bien que le parseur puisse prédire la structure arborescente et les relations directement à partir de texte brut, notre étude choisit d'utiliser la segmentation des unités discursives (EDU) de référence. Dans notre configuration expérimentale, nous utilisons à la fois le texte brut de l'ensemble de test RST-DT d'origine (traduit en français) et la segmentation des EDUs de référence.

**Traduction du corpus RST-DT en français avec ChatGPT.** Afin d'évaluer les performances du système DMRST en français, nous avons traduit l'ensemble test du corpus RST-DT en français en utilisant ChatGPT. La traduction avait pour objectif de rester aussi fidèle que possible au texte original : chaque unité discursive des 38 documents (soit 2308 EDUs) a été traduite individuellement pour préserver la structure RST de chaque texte et éviter la génération de paraphrases. De plus, ChatGPT prend en compte le contexte de chaque EDU dans sa traduction ; bien que les unités discursives soient traduites individuellement, les accords grammaticaux, de sujets-verbes ou encore de références sont préservés. Par exemple, la deuxième unité ici accorde correctement au pluriel : [sans les frais de rachat] [qui sont courants pour les rentes.].

La version de ChatGPT utilisée est GPT-4-Turbo. Le prompt qui a été utilisé pour obtenir les traductions est le suivant :

message : {

role.system : "Vous êtes traducteur expert de l'anglais vers le français et un linguiste spécialisé dans le domaine de la Rhetorical Structure Theory (RST)."

role.user : "Traduire le texte ci-dessous en Francais segment par segment, tout en veillant à conserver le contexte global du texte dans son ensemble.

Texte à traduire :

segment 1

segment 2

...

Afficher le résultat avec chaque segment traduit sur une ligne distincte."

}

FIGURE 2 – Prompt utilisé pour traduire le corpus de test RST-DT en français. Ce prompt a été utilisé 38 fois pour les 38 documents de ce corpus.

Encore une fois, il était nécessaire de contraindre le système à traduire EDU par EDU afin de préserver les structures textuelles annotées en relations de cohérence du corpus RST-DT. La question qui se pose dès lors est la suivante : le texte produit est-il bien du français ? Afin d'analyser les biais introduits par une telle contrainte, nous avons comparé, pour un ensemble de 10 documents du corpus de test RST-DT choisis au hasard, une traduction automatique des documents traduits en entier avec la traduction EDU par EDU. Ces 10 mêmes documents ont ensuite été relus et évalués par 3 locuteurs natifs français.

<footnote n="3">3. https://github.com/seq-to-mind/DMRST_Parse</footnote>
En ce qui concerne la comparaison entre le texte traduit par segments et le texte traduit en entier, étant donné que les traducteurs automatiques traduisent souvent phrase par phrase, les traductions étaient similaires, à l'exception de légères variations dans le vocabulaire utilisé et la génération de paraphrases où l'ordre de quelques groupes prépositionnels a été inversé. Nous avons ensuite comparé ces deux traductions automatiques pour chacuns des 10 documents avec les métriques BLEU, ROUGE, TER et WER et avons obtenu les scores moyens suivants : 0.65, 0.82, 20.0 et 0.35.

Quant à l'évaluation manuelle des 3 locuteurs, les 10 documents traduits en français ont été jugés intelligibles et corrects dans leur ensemble. Toutefois les participants ont noté un manque de fluidité, une absence totale d'idiomaticité et la présence, rare mais non négligeable, de calques syntaxiques venus de l'anglais produisant des imprécisions grammaticales, comme « an equity position in Leaseway » traduit par « une position en capitaux dans Leaseway ».

Les résultats de ces analyses nous permettent de conclure que, bien que la contrainte de traduction EDU par EDU n'introduise pas de biais trop importants par rapport à la traduction du texte entier (comme le montrent les scores obtenus), la traduction du français obtenue n'est pas la plus naturelle qui soit. Toutefois, comme nous le verrons dans la partie 4, les éléments linguistiques analysés restent pertinents et correspondent à des usages naturels en français.

## 3 Expérimentations et résultats

**Reconnaissance des relations de cohérence RST en français.** Les 38 textes du corpus RST-DT traduits en français avec ChatGPT ont été parsés avec le système multilingue DMRST. Nous présentons ci-dessous les résultats évalués avec la métrique RST-Parseval (Marcu, 2000) pour la classification de relation.

<table><caption>TABLE 1 – Résultats globaux évalués avec la métrique RST-Parseval (Marcu, 2000) pour la classification de relation en français et en anglais.</caption><thead><tr><th></th><th>Précision</th><th>Rappel</th><th>F1-score</th></tr></thead><tbody><tr><td>Français</td><td>0.54</td><td>0.57</td><td><strong>0.54</strong></td></tr><tr><td>Anglais</td><td>0.65</td><td>0.67</td><td>0.65</td></tr></tbody></table>

Sans surprise, les résultats sont moins bons que ceux obtenus pour l'anglais, mais sont comparables aux résultats obtenus sur d'autres corpus RST-DT dans différentes langues (Avec un F1-score de 0.62 en portugais, 0.63 en espagnol, 0.47 en allemand, 0.52 en néerlandais et 0.48 en basque, conformément aux résultats signalés par Liu et al., 2021). Comme nous pouvons le voir avec la matrice de confusion sur la Figure 3 ci-dessous, nous remarquons également que la confusion entre les différentes classes de relations est similaire à ce que nous obtenons pour l'anglais concernant les classes ELABORATION et JOINT, cela étant dû à un déséquilibre des classes dans le jeu de données RST-DT.
<table><caption>TABLE 2 – Fréquence absolue et relative des étiquettes de relation en or dans RST-DT pour un total de 2306 relations.</caption><thead><tr><th>Relation</th><th>rel. frq. (%)</th><th>abs. frq. (N)</th></tr></thead><tbody><tr><td>elaboration</td><td>34.43</td><td>794</td></tr><tr><td>attribution</td><td>14.87</td><td>343</td></tr><tr><td>joint</td><td>9.19</td><td>212</td></tr><tr><td>contrast</td><td>6.37</td><td>147</td></tr><tr><td>same-unit</td><td>5.51</td><td>127</td></tr><tr><td>explanation</td><td>4.77</td><td>110</td></tr><tr><td>enablement</td><td>3.82</td><td>88</td></tr><tr><td>cause</td><td>3.56</td><td>82</td></tr><tr><td>evaluation</td><td>3.47</td><td>80</td></tr><tr><td>temporal</td><td>3.17</td><td>73</td></tr><tr><td>background</td><td>2.95</td><td>68</td></tr><tr><td>topic-comment</td><td>1.69</td><td>39</td></tr><tr><td>summary</td><td>1.39</td><td>32</td></tr><tr><td>comparison</td><td>1.26</td><td>29</td></tr><tr><td>condition</td><td>1.21</td><td>28</td></tr><tr><td>manner-means</td><td>1.17</td><td>27</td></tr><tr><td>topic-change</td><td>0.78</td><td>18</td></tr><tr><td>textualorganization</td><td>0.39</td><td>9</td></tr></tbody><tfoot><tr><td>TOTAL</td><td>100.0</td><td>2306</td></tr></tfoot></table>

<figure><img src="image_152.png" /><figcaption>FIGURE 3 – Matrice de confusion de 18 étiquettes harmonisées pour le français.</figcaption></figure>

<figure><img src="image_153.png" /><figcaption>FIGURE 4 – Matrice de confusion pour l’anglais.</figcaption></figure>

## 4 Discussion et analyse des cas d’échecs / succès

Dans cette section, nous procédons à une analyse qualitative d'un sous-ensemble de relations. Les cas sélectionnés pour l'annotation visent à illustrer en quoi la présence de marqueurs de discours ou de certaines structures syntaxiques influence les erreurs de classification du parseur DMRST. Les cas analysés ci-dessous proviennent d'un ensemble de 301 relations. Ces 301 cas ont été sélectionnés à partir du RST Signaling corpus (Das & Taboada, 2018), où chacune des relations du RST-DT
a été annotée en signaux de relations de cohérence pour de l'anglais. Nous avons extrait les cas correspondant aux catégories décrites ci-dessous et les avons réannotés pour vérifier si ces signaux étaient également applicables au français. À la suite de cette procédure, 21 relations ont été exclues. L'accord inter-annotateur, mesuré par le coefficient Kappa de Cohen, est de 0,83 pour cette phase d'annotations.

**Présences de marqueurs de discours.** Dans certaines situations, les marqueurs de discours jouent un rôle décisif et contribuent efficacement à prédire la relation. Par exemple, la présence du marqueur de discours *si*, signalant une relation de **CONDITION**, permet une reconnaissance précise dans 73% des cas pour 41 relations annotées. Cependant, nous constatons que les marqueurs de discours posent souvent des problèmes en raison de leur présence dans de nombreuses relations. Par exemple, nous observons que le marqueur de discours *et* est largement utilisé dans la relation **JOINT**, mais il est également présent dans de nombreuses autres relations telles que la relation **TEMPORAL**. Par conséquent, seulement 17% (sur 26 annotations) des relations **TEMPORAL** signalées par ce marqueur sont correctement prédites. Nous observons une tendance similaire en ce qui concerne la confusion entre les relations **TEMPORAL** et les relations **BACKGROUND**, souvent signalées par des marqueurs tels que *après que*, *depuis que que*, *tant que*, *pendant que*, *alors que* ou d'autres formes + **QUE** comme dans l'exemple (3).

$$ (3) \quad \text{[dit-il]} \leftarrow \frac{\text{pred : background}}{\text{gold : temporal}} \text{[alors que le boucher s'éloigne ».] wsj\_1146} $$

La relation **BACKGROUND** se produit entre une unité de texte principal (nucleus) et un satellite qui est nécessaire à la compréhension de la première unité. Bien que cette relation n'implique pas d'aspect temporel, elle est souvent signalée par des marqueurs de discours tels que *depuis que*, *tant que* ou tout simplement par un *après*, comme le montre l'exemple 4, qui sont également utilisés pour signaler la relation **TEMPORAL**.

$$ (4) \quad \text{[Mme Crump a déclaré que le portefeuille de son club d'investissement Ashwood avait perdu environ un tiers de sa valeur]} \leftarrow \frac{\text{pred : temporal}}{\text{gold : background}} \text{[après le crash du Black Monday] wsj\_2386} $$

**Présences de structures syntaxique.** D'un autre côté, on remarque que les relations signalées par des structures syntaxiques ont plus de chances d'être systématiquement prédites. Nous avons annoté ici les structures telles que les *constructions syntaxiques parallèles* (voir exemple (5)) pour la relation **JOINT** et les *propositions relatives* pour la relation **ELABORATION**). Nous notons que 79% des relations annotées avec des *constructions syntaxiques parallèles*, soit 72 cas annotés, sont correctement prédites. De même, nous notons un taux de réussite de 85% pour les 141 cas annotés des *propositions relatives*.

$$ (5) \quad \text{[Parlez-nous de la retenue en matière de dépenses]} \leftarrow \frac{\text{pred : temporal}}{\text{gold : background}} \text{[Parlez-nous des scandales du HUD] » wsj\_0623} $$

**Ambiguïté et Spécificité.** L'ambiguïté et la spécificité jouent un rôle crucial dans les performances différenciées du parseur en fonction de la présence de marqueurs de discours ou de structures syntaxiques. Les marqueurs de discours ont tendance à être plus ambigus, car ils peuvent avoir une forme plus ou moins lexicalisée, comme *alors que* ou *après*, qui peuvent être utilisés dans différents contextes. Cependant, cela ne s'applique pas aux marqueurs tels que *si*, qui sont très spécifiques à la relation **CONDITION**. D'autre part, on observe que les structures syntaxiques sont beaucoup plus spécifiques à certaines relations et, par conséquent, elles induisent moins de confusion pour le parseur.
# 5 Conclusion

Actuellement, il est important de noter que nous dépendons d'un seul point de contrôle de modèle pour l'expérimentation, ce qui introduit le potentiel d'erreurs influencées par des certaines variations lors de l'entraînement. De plus, nous tenons à souligner que le corpus est limité aux données issues de la presse, et explorer des données provenant de différents genres serait susceptible de fournir des perspectives supplémentaires. Par ailleurs, l'annotation manuelle effectuée ici s'est concentrée sur des cas particuliers inspirés par des analyses menées sur de l'anglais. La création d'un corpus annoté avec des signaux de relations de cohérence, à l'instar de ce qu'ont fait Das & Taboada, 2018 pour l'anglais, nous permettrait d'avoir une vue plus complète des analyses des cas d'échecs et de succès.

En conclusion, cet article a évalué les performances du parseur DMRST sur le corpus RST-DT traduit en français. Nous avons constaté que les performances de ce parseur pour la classification des relations de cohérence en français sont comparables à celles obtenues pour d'autres langues. En analysant les succès et les échecs de la classification des relations, nous avons mis en évidence l'impact des marqueurs de discours et des structures syntaxiques sur la précision du parseur. Cela est attribuable à la spécificité des structures syntaxiques, qui sont étroitement liées à des relations individuelles et ne sont pas aussi ambiguës que les marqueurs de discours.

# Remerciements

Le travail présenté dans cet article a été réalisé dans le cadre du projet HYBRIDS, un réseau doctoral Marie Skłodowska-Curie financé par l'Union européenne sous le numéro de subvention 101073351 et par le UK Research and Innovation (UKRI) Horizon Funding Guarantee.

# Références

BRAUD C., COAVOUX M. & SØGAARD A. (2017). Cross-lingual RST discourse parsing. In *Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics : Volume 1, Long Papers*, p. 292–304, Valencia, Spain : Association for Computational Linguistics.

BRAUD C., LIU Y. J., METHENITI E., MULLER P., RIVIÈRE L., RUTHERFORD A. & ZELDES A. (2023). The DISRPT 2023 shared task on elementary discourse unit segmentation, connective detection, and relation classification. In C. BRAUD, Y. J. LIU, E. METHENITI, P. MULLER, L. RIVIÈRE, A. RUTHERFORD & A. ZELDES, Éds., *Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023)*, p. 1–21, Toronto, Canada : The Association for Computational Linguistics. DOI: 10.18653/v1/2023.disrpt-1.1.

CARLSON L., MARCU D. & OKUROWSKI M. E. (2003). *Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory*, In J. VAN KUPPEVELT & R. W. SMITH, Éds., *Current and New Directions in Discourse and Dialogue*, p. 85–112. Springer Netherlands : Dordrecht.
DOI: 10.1007/978-94-010-0019-2_5.

CHERNYAVSKIY A., ILVOVSKY D. & NAKOV P. (2024). Unleashing the power of discourse-enhanced transformers for propaganda detection. In *Proceedings of the 18th Conference of the*
*European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)*, p. 1452–1462.

CONNEAU A., KHANDELWAL K., GOYAL N., CHAUDHARY V., WENZEK G., GUZMÁN F., GRAVE E., OTT M., ZETTLEMOYER L. & STOYANOV V. (2020). Unsupervised cross-lingual representation learning at scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, p. 8440–8451, Online : Association for Computational Linguistics. DOI : 10.18653/v1/2020.acl-main.747.

DAS D. & TABOADA M. (2018). Signalling of coherence relations in discourse, beyond discourse markers. *Discourse Processes*, **55**(8), 743–770. DOI: 10.1080/0163853X.2017.1379327.

GUZ G. & CARENINI G. (2020). Coreference for discourse parsing : A neural approach. In C. BRAUD, C. HARDMEIER, J. J. LI, A. LOUIS & M. STRUBE, Éds., *Proceedings of the First Workshop on Computational Approaches to Discourse*, p. 160–167, Online : Association for Computational Linguistics. DOI : 10.18653/v1/2020.codi-1.17.

LIU Y. J., AOYAMA T. & ZELDES A. (2023). What's hard in English RST parsing? predictive models for error analysis. In *Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue*, p. 31–42, Prague, Czechia : Association for Computational Linguistics.

LIU Z., SHI K. & CHEN N. (2021). DMRST : A joint framework for document-level multilingual RST discourse segmentation and parsing. In *Proceedings of the 2nd Workshop on Computational Approaches to Discourse*, p. 154–164, Punta Cana, Dominican Republic and Online : Association for Computational Linguistics. DOI : 10.18653/v1/2021.codi-main.15.

MANN W. C. & THOMPSON S. A. (1988). Rhetorical structure theory : Toward a functional theory of text organization. *Text-interdisciplinary Journal for the Study of Discourse*, **8**(3), 243–281. DOI : /doi.org/10.1515/text.1.1988.8.3.243.

MARCU D. (2000). The Rhetorical Parsing of Unrestricted Texts : A Surface-based Approach. *Computational Linguistics*, **26**(3), 395–448. DOI: 10.1162/089120100561755.

NGUYEN T.-T., NGUYEN X.-P., JOTY S. & LI X. (2021). RST parsing from scratch. In K. TOU-TANOVA, A. RUMSHISKY, L. ZETTLEMOYER, D. HAKKANI-TUR, I. BELTAGY, S. BETHARD, R. COTTERELL, T. CHAKRABORTY & Y. ZHOU, Éds., *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies*, p. 1613–1625, Online : Association for Computational Linguistics. DOI : 10.18653/v1/2021.naacl-main.128.

PITLER E., RAGHUPATHY M., MEHTA H., NENKOVA A., LEE A. & JOSHI A. (2008). Easily identifiable discourse relations. In *Coling 2008 : Companion volume : Posters*, p. 87–90, Manchester, UK : Coling 2008 Organizing Committee.
# SUMM-RE: A corpus of French meeting-style conversations

Julie Hunter¹ Hiroyoshi Yamasaki² Océane Granier²

Jérôme Louradour¹ Roxane Bertrand² Kate Thompson¹ Laurent Prévot²

(1) LINAGORA Labs, Toulouse, France

{jhunter, jlouradour, cthompson}@linagora.com

(2) Aix Marseille Université & CNRS, LPL, Aix-en-Provence, France

{hiroyoshi.yamasaki, oceane.granier, roxane.bertrand, laurent.prévot}@univ-amu.fr

## RÉSUMÉ

Nous présentons le corpus SUMM-RE, un ensemble de données d'environ 95 heures de conversations spontanées de type réunion en français. Le corpus est conçu pour servir de base à des tâches en aval telles que le résumé de réunions. Dans son état actuel, il offre 25 heures de transcriptions corrigées manuellement et alignées sur le signal audio, ce qui en fait une ressource précieuse pour l'évaluation des systèmes d'ASR et de reconnaissance du locuteur. Il comprend également des transcriptions automatiques et des alignements de l'ensemble du corpus qui peuvent être utilisés pour des tâches de NLP en aval. L'objectif de cet article est de décrire la conception, la production et l'annotation du corpus jusqu'à l'étape de transcription, ainsi que de fournir une description quantitative du corpus permettant de comprendre ses principales caractéristiques linguistiques.

## ABSTRACT

We present the SUMM-RE corpus, a dataset of roughly 95 hours of spontaneous meeting-style conversations in French. The corpus is designed to serve as a foundation for downstream tasks such as meeting summarization. In its current state, it offers 25 hours of manually corrected transcripts that are aligned with the audio signal, making it a valuable resource for evaluating ASR and speaker recognition systems. It also includes automatic transcripts and alignments of the whole corpus which can be used for downstream NLP tasks. The aim of this paper is to describe the conception, production and annotation of the corpus up to the transcription level as well as to provide statistics that shed light on the main linguistic features of the corpus.

**MOTS-CLÉS** : Corpus, Réunions, Conversation spontanée, Français, Dialogue, Transcription.

**KEYWORDS** : Corpus, Meetings, Spontaneous conversation, French, Dialogue, Transcription.

## 1 Introduction

Spontaneous, multiparty conversation poses particular problems for both speech and natural language processing tasks. Disfluent speech, non-standard or incorrect grammar, idiosyncratic speaker styles, and overlapping speech and interruptions — not to mention transcription errors introduced when transcripts are automatically generated — complicate the task of understanding conversation transcripts, especially for NLP models that have been trained largely on text (Rennard et al., 2023).

<figure><img src="image_154.png" /></figure>
These linguistic features together with complicated acoustic conditions, shared microphones, different accents, and rapid speech aggravate the problem for speech processing tasks such as transcription and speaker identification (Yamasaki et al., 2023).

Progress on these topics is hindered by a lack of data especially for specific tasks such as meeting summarization. Few people are comfortable sharing recordings of their discussions and even if they are, preparing the data for training poses a significant hurdle. Data scarcity is even more serious when we look to languages other than English.

The principal goal of the SUMM-RE project is to develop models for conversation and meeting summarization with a particular focus on French. A critical step in accomplishing this goal has been to collect and prepare a corpus of roughly 100 hours of meeting-style conversations in French. The aim of this paper is to present this corpus and to detail its conception and production.

## 2 Related corpora

AMI (Augmented Multi-party Interaction; Carletta et al., 2005) and ICSI (International Computer Science Institute; Janin et al., 2003; McCowan et al., 2005) are the most well-known meeting corpora (and until recently, were the only meeting corpora at all). AMI contains 137 scenario-driven meetings that last from 15 to 45 minutes each, for a total of around 65 hours of conversations. In each meeting, four participants play roles in a fictitious electronics company and participate in a sequence of four meetings. The roles and scenarios are well developed and always the same, which facilitates the task of getting four strangers to carry out structured discussions on topics for which they have little background knowledge and also avoids privacy concerns triggered by real meanings. On the other hand, while the language remains spontaneous, the heavy corpus design engenders conversational styles and interactions that are arguably much cleaner than real-life meetings and also leads to a homogeneity of content and vocabulary. SUMM-RE by contrast, is designed to encourage more fluid discussion on a range of topics; the focus is on trying to elicit particular types of discursive interactions that are characteristic of meetings without insisting that the participants play employee-like roles.

ICSI consists of natural, weekly meetings that last about one hour each for a total of roughly 72 hours of recordings. On average, the meetings involve six participants that can include undergraduates, graduate students, and professors who meet to discuss technical topics related to natural language processing, computational linguistics and even the ICSI corpus itself. As ICSI contains real meetings between people who were actively collaborating on projects at the time of recording, the interactions are more natural than those in AMI. At the same time, they draw on technical vocabulary and specialized subjects as well as a considerable amount of shared knowledge between participants, which can complicate the interpretation of the content for models that do not have this knowledge. While such a scenario is undeniably realistic, the SUMM-RE corpus is designed to strike a balance between naturalness and feasibility for automatic summarization. As such, it includes light guidance of the meeting structure and limits the impact of shared background knowledge by bringing in participants who in many cases did not know each other before participating in our corpus.

While both AMI and ICSI are entirely in English, there have been recent efforts to expand to other languages. VCSum (Versatile Chinese Meeting Summarization Dataset Wu et al., 2023) is a collection of transcripts and videos of roundtable meetings in Chinese found on the internet. A total of 239 meetings were selected for a total of over 230 hours of recording time. The corpus is called "versatile"
because it contains a variety of annotations that can be relevant for different summarization tasks. The ELITR corpus (Nedoluzhko et al., 2022) contains transcripts for 113 technical project meetings in English but also for 53 meetings in Czech, for a total of over 160 hours of content. Like ICSI, the meetings in ELITR are natural, leading to many of the same advantages and drawbacks. Unlike ICSI, AMI, VCSum and SUMM-RE however, the ELITR audio files have not been released and parts of the meetings are censored for privacy.

There are also a variety of smaller, conversational corpora in French (for a recent list, see Hunter et al., 2023). CID (The Corpus of Interactional Data Blache et al., 2017), which contains eight one-hour dialogues between friends, has notable similarities with SUMM-RE in that a major effort was involved in adding different levels of annotation, including dialogue-central information that can be exploited by downstream NLP models. With only eight hours of recording, however, it remains very small and is not focused specifically on meetings.

# 3 Corpus Design

To bypass the concern of sharing personal information, the SUMM-RE corpus does not contain real meetings. The conversations were nevertheless designed to imitate certain important features of meeting-style conversation, developing situations in which participants have to make decisions or plan out a list of action items or report on things they have done. They were also designed to have some continuity with past meetings, as real meetings often do : almost every one of the 96 individual experiments in the corpus is made up of a series of three 20 minute meetings that develop a given topic.<fnref n="1" /> In general,<fnref n="2" /> the participants were asked to plan an event during the third meeting, while the second meeting focused on deciding what kind of event to plan, and the first meeting involved participants going around the table to discuss their experience or opinions about certain topics. Each one-hour experiment contains the same set of participants—usually four but sometimes fewer—throughout.

In some cases, meeting participants knew each other before the experiment. This condition adds an arguably realistic element to the interactions, as many meetings are held by people who have worked together before. However, because our meeting scenarios were artificial, we also feared that it would encourage playful interactions full of jokes and laughter. While such interactions are certainly possible in professional meetings, many meetings involve a higher level of seriousness and personal distance. In an effort to vary the interactions and imitate different levels of professionalism, we aimed to balance the number of groups in which all participants knew each other, some participants knew each other or no participants knew each other.

To encourage participants to speak naturally and spontaneously while also pushing them to stick to a meeting-like agenda, we had to strike a balance between role playing and freedom for the participants to talk about their own experiences. To encourage freedom, we defined a set of topics that we assumed most participants would be comfortable discussing, including a) internet and technology-related topics like social networks and the societal influence of companies like Google and Amazon, b) films and TV series, c) fundraising, d) music, e) environment-related topics such as global warming and renewable energy. Within these larger topics, participants were allowed to choose a subtopic that

<footnote n="1">1. Due to technical problems, we had to remove five meetings from the final corpus.</footnote>

<footnote n="2">2. In the early pilot studies, we tried different types of organization but ultimately decided upon the one described here.</footnote>

<footnote n="3">3. We abandoned this topic, which is discussed in 4.4% of the overall data, because participants struggled to develop it.</footnote>
interested them.

To add structure to the conversations, we proposed a series of subtopics or responsibilities that each participant could choose to lead as well as a series of points, questions, or tasks (depending on the format of the meeting) that the participants might want to pursue. For each subtopic, the participants received an individual card or a collective document listing the points so that they had a “cheat sheet” if they struggled to develop their contribution to the conversation.

Here, for example, is the set of questions (translated from French to English here) given to a participant who chose to discuss Amazon during a reporting meeting:

> You have chosen to be responsible for leading the discussion about Amazon. You will need to be able to talk about this subject for 3-4 minutes. To help guide you, below is a list of questions to which you might respond (but you are free to choose other questions if you find them more suitable):

— What is Amazon? How do we interact with Amazon in our daily lives

— What are the positive and negative sides of Amazon?

— What do you think about their approach to package delivery, the Prime video platform, employee conditions, etc?

— How do you think that Amazon will evolve in the future? Will they become more influential? Will other actors replace them?

— ...

> You can draw from your personal experience to respond to these questions or provide concrete examples.

Finally, for each 20 minute conversation, the group was asked to choose a moderator among them. In addition to managing the discussion for a 20 minute conversation (also following suggested guidelines), each moderator was assigned the task of taking notes and of making an oral summary at the end of the meetings based on these notes. To preserve the continuity of the conversation, the summary was recorded immediately after the conversation on the same record. The summary file was later extracted in post-processing.

# 4 Data acquisition

The original plan was to record the entire SUMM-RE corpus in the H2C2⁴. Unfortunately, corpus collection began in 2020 and ran through 2021 when the Covid pandemic was still a threat, so in-person meetings were not always an option. We thus adopted two different strategies for corpus collection: in person recordings and Zoom.

In all cases, the recordings shared certain characteristics. For instance, each one hour experiment involved the same set of participants. In general, there were four participants but sometimes, we were only able to find three and on very rare occasions, two. Each participant had their own microphone for recording, as explained below. Finally, each experiment was led by one of the co-authors of this paper, who would begin by giving instructions to the participants and making sure they understood the task, but would leave the room during recording to avoid interaction with the participants.

Most in-person recordings took place in the H2C2, though there are a few exceptions in which it was

<footnote n="4">4. https://plateformeh2c2.fr/</footnote>
easier to go into peoples' homes. Participants for these recordings were generally recruited through our platform, calls for participants in social media or announcements made in university courses. In total, 248 out of 283 meetings were recorded in person (see Table 3 for more detail).

The H2C2 studio is composed of two rooms :

- an experimental room where participants interact during the recordings

- an observation and recording room where the person in charge of managing the experiment goes during the experiments and can observe participants through a one-way mirror.

Each participant was equipped with an individual headset with microphone (AKG 520). An additional microphone was used to capture the streams of all speakers at once. All microphones were recorded with a Zoom H8 handy recorder.

While recording conditions in the H2C2 were near ideal, we encountered two problems that impacted the quality of the final recordings. First, although the room was spacious enough to leave a fair amount of distance between speakers and each headset microphone was adjusted to the voice of the person wearing it, sometimes a participant's voice would vary between the test conditions and the final recording conditions. A speaker might end up using a higher pitch during recording due to elevated emotion, for example. In such cases, individual microphones would often end up capturing the voices of multiple speakers. Our efforts to isolate the contributions of the main speaker in such cases are described in Section 5. The second complication resulted from efforts to navigate health and safety regulations enforced during the Covid pandemic. To reduce contact between speakers, we first tried installing plexiglass barriers so that participants could see each other's mouths, but this led to an echo in the recordings. Ultimately, we asked the participants to wear masks. This worked in most cases, but idiosyncratic approaches to mask wearing still led to suboptimal recording in some cases.

35 out of 283 of the meetings were recorded through Zoom. For these, participants were recruited through the crowd sourcing platform Prolific.<fnref n="5" /> Each person was required to use their own microphone to take part in the experiment in an effort to limit background noise. While Zoom facilitated the task of speaker identification and minimized the impact of phenomena like overlapping speech, dependence on personal equipment and internet connections led to poor recording quality in some cases. See 7.2 for more information on the effect of communication modality on participants' behavior.

# 5 Annotation and post-processing

The SUMM-RE corpus is partitioned into *train*, *dev* and *test* data sets with proportions at roughly 75%, 12.5% and 12.5%, respectively. Files were assigned one of three classes randomly with the constraint that the ratio of Zoom experiments to in-person experiments as well as relative proportions of scenarios were kept roughly constant. After this automatic process, minor adjustments were made to ensure that certain files that had been manually corrected ended up in the *dev* set. This procedure resulted in 210 files in the *train* set, 36 files in the *dev* set and 37 files in the *test* set.

The entirety of the SUMM-RE corpus has been automatically transcribed and the entirety of the *dev* set, roughly 12 hours, has been manually corrected and annotated. Correction of the *test* set is underway and will soon be complete. Manual correction is performed in two phases. First, the transcripts are manually corrected for transcription errors. Corrections at this stage are made with the software Praat (Boersma & Van Heuven, 2001). In places where Whisper misses a substantial

<footnote n="5">5. https://www.prolific.com/</footnote>
pause (i.e. on the order of hundreds of milliseconds) a pause marker # is added. If the IPU boundaries differ significantly from the true boundary, this is also manually corrected. All files are then manually verified for any obvious errors made during the first correction as well as minor adjustments such as adding non-linguistic annotations for non-speech sounds like laughs and coughs to improve tier boundary accuracy. On the dev set, corrections took one month and were carried out by one of the co-authors who was paid for the task and who is a native speaker of French. Manual verification was performed by another co-author (non-native speaker).

While the details of our automatic pipeline are described in Yamasaki et al. (2023), we give a brief overview here. At a high-level, the pipeline can be divided into two parts : the detection of inter-pausal units (IPUs)—segments of audio in which the principal speaker is speaking—and the transcription of the words that were uttered in the IPU along with their start and end times. IPU detection was necessary because, as explained in Section 4, individual microphones often captured the voices of multiple speakers. For this task, we employed out-of-the-box IPU annotation provided by the SPPAS package (Bigi & Priego-Valverde, 2019; Bigi, 2015) combined with an approach which relies on speaker diarization using the Pyannote package (Bredin et al., 2020; Bredin & Laurent, 2021). This second step was necessary because the voice intensity processed by SPPAS and the voice quality processed by Pyannote contain complementary information. Once the IPUs were identified, we created new audio files in which background voices were replaced with silences and then passed the resulting audio files to Whisper (Radford et al., 2022), OpenAI's speech to text model.<fnref n="6" />

The quality of the annotation and alignment predicted by our pipeline was extensively evaluated in Yamasaki et al. (2023) on a 3.3 hour subset of the dev set. Table 1 shows the word error rate (WER) for this subset, broken down by deletions (Del), insertions (Ins) and substitutions (Sub). It also includes T-δ, an average (in milliseconds) of the absolute difference in start and end times for each word and an F1 score inspired by Bain et al. (2023). See Yamasaki et al. (2023) for details.

<table><caption>TABLE 1 – Brief summary of automatic word level evaluation of the SUMM-RE corpus including F1 score for annotation correctness, time-δ for alignment error (in milliseconds) and word error rate (WER) and corresponding deletion, insertion and substitution scores.</caption><thead><tr><th>Pipeline</th><th>F1</th><th>T-δ</th><th>WER</th><th>Del</th><th>Ins</th><th>Sub</th></tr></thead><tbody><tr><td>Our Reference</td><td><strong>0.81</strong></td><td>108</td><td><strong>18.8</strong></td><td>8.1</td><td>5.8</td><td>4.8</td></tr></tbody></table>

A final point is that in order to prevent identification of individual participants we developed a script to remove participant names from both audio files and transcripts and replace the corresponding audio intervals with a beep. This was done by :

- identifying possible candidates for mentions of individual names by calculating the Levenshtein distance of each token to participants' first and second names

- manually filtering false positives

- replacing the resulting list of names by anon and the corresponding WAV interval with a single beep

This anonymization scheme assumes the correctness of the transcription, which is not an issue for the

<footnote n="6">6. There are several variants of Whisper models such as Klein (2023); Bain et al. (2023). We chose Louradour (2023) after a detailed comparison of their performance as detailed in Yamasaki et al. (2023). We opted for the large v2 model as a starting point but added two custom features. The first employed prompting to encourage Whisper to transcribe disfluencies, yielding a more faithful transcript. The second involved introducing precise word alignment through techniques developed in Louradour (2023) and the Julius forced aligner (Lee et al., 2001). Further details of both the IPU detection algorithm and the Whisper transcription process can be found in Yamasaki et al. (2023).</footnote>
test and dev splits as they are manually corrected, but is for the train set which is not. In cases where an individual's name was not correctly transcribed it may not be anonymized. However, we noticed that it is highly rare for participants to use full names or family names to refer to each other. As first names are not particularly identifiable we believe this anonymization scheme is sufficient.

# 6 Basic corpus statistics

The SUMM-RE corpus includes a total of 207 unique participants. Due to the recruitment procedure described in Section 4, our data is biased with respect to age, gender and occupation. In particular, the majority of participants were students with a mean age of 28.7, there are nearly three times more female than male participants, and a large portion of participants were monolingual French speakers who grew up in France. See 2 in the Appendix for further metadata on participants.

Our data set consists of 96 sessions with 3 conversations for almost every session, yielding a total of 283 conversations of roughly 20 minutes each. <fnref n="7" /> Of these, 8 sessions (22 files) are pilot experiments and the remaining 88 sessions (261 files) are non-pilot experiments. As stated above (Section 4) some meetings (35/283 files) were recorded on Zoom due to Covid restrictions.

# 7 Linguistic statistics

Dialogue corpora vary greatly depending on a wide range of contextual factors. To better understand the nature of a dialogue corpus, it is crucial to look at its linguistic, and in particular its interactional, properties. In this section, we focus on some core metrics that are interesting proxies to characterize the corpus. Namely, we look at (i) distribution of Inter-Pausal Unit (IPU) durations (as a proxy to sentence length in the written realm); (ii) automatically extracted backchannels; (iii) amount of overlapping speech; (iv) filled pause count; and (v) speaker dominance. Together these metrics provide an insight about the degree of spontaneity and interaction in the corpus.

## 7.1 By metric

**IPU distribution** Figure 1 shows the distribution of IPU duration and number of IPUs for dev, train and test. As the splits were random, both IPU duration and count are similar across splits, though there are slightly more IPUs shorter than 1 second for train than for dev and test. This may be due to erroneous IPUs (e.g., from someone else laughing) removed during correction.

Independent t-tests indicated that while the number of IPUs in meetings from *train* and *dev* varied significantly ($p \le 0.01$), *test* was consistent with *train*, meaning that there should not be an issue when using *test* for evaluation. There was no significant difference for pilot and experiment recordings either ($p = 0.17$), which justifies combining them into a single data set. We also performed t-tests to compare our different scenarios (a-e) and meeting styles (Section 3). For scenarios, 6 pairs reached significance: a-c $p \le 0.001$, a-d $p \le 0.05$, a-e $p \le 0.01$, b-c $p \le 10^{-4}$, c-d $p \le 10^{-4}$, c-e $p \le 10^{-5}$. For tasks, reporting differed significantly from both decision ($p \le 10^{-4}$) and planning ($p \le 10^{-5}$), which is to be expected given that reporting meetings were designed to be less interactive.

<footnote n="7">7. 005b_PBP, 005b_PBR, 012a_EBR, 017b_EBD, 087c_EEP were rejected due to technical issues.</footnote>
<figure><img src="image_155.png" /><figcaption>FIGURE 1 – Left : kernel density estimate of IPU duration by splits, Right : number of IPUs by splits</figcaption></figure>

**Token distribution.** Figure 2 shows the distribution of token duration and speech rate. As expected, there is only a small difference between splits. Average number of tokens per file was 1187 words for *train*, 1107 words for *dev* and 1218 words for *test*. Independent t-test revealed no significant differences between splits (train-dev : *p* = 0.11, dev-test : *p* = 0.08, train-test : *p* = 0.55). Pilot vs experiment comparison was not significant (*p* = 0.99). Comparison by scenarios gave 4 significant pairs : a-c *p* ≤ 0.01, b-c *p* ≤ 0.05, c-d *p* ≤ 0.01 and c-e *p* ≤ 0.01. Comparison by task showed that reporting again differed from both decision (*p* ≤ 0.001) and planning (*p* ≤ 0.05).

<figure><img src="image_156.png" /><figcaption>FIGURE 2 – Left : kernel density estimate of token duration by splits, Right : speech rate by splits</figcaption></figure>

**Backchannels.** Figure 3 (Left) shows the number of backchannels found per minute. For the purpose of this paper we used a rather simplistic definition of backchannel in which we only considered single word utterances that matched "oui", "ouais", "hm", "mh", "non", "ok", "ah", "ben", "bien", "eh", "euh", "voilà". This decision was made for the purpose of simplicity but has a disadvantage of underestimating the actual number.

The number of backchannels did not differ between *train* and *test* (*p* = 0.62) but was slightly lower for *dev* than *train* (*p* ≤ 0.01) and *test* (*p* ≤ 0.01). Comparison between pilots and experiments did not reach significance (*p* = 0.37). Seven scenario pairs reached significance : a-b *p* ≤ 0.01, a-c *p* ≤ 0.01, a-d *p* ≤ 0.01, b-c *p* ≤ 0.001, c-d *p* ≤ 10⁻⁴, c-e *p* ≤ 0.001. The reporting
task again differed from decision ($p \le 10^{-5}$) and planning ($p \le 10^{-5}$). Finally, in person meetings had more backchannels than Zoom meetings ($p \le 0.001$). See Appendix B.3 for the figures.

<figure><img src="image_157.png" /><figcaption>FIGURE 3 – Left : Backchannel counts by sets, Right : Number of speakers at the same time</figcaption></figure>

**Overlap.** Figure 3 (Right) shows the relative duration of intervals in which $N \in 0, 1, 2, 3, 4$ speakers were speaking at the same time. This was calculated by 1) segmenting the entire conversation into small segments of equal duration (here 0.02 seconds), 2) counting how many speakers are speaking in each segment, 3) aggregating by number of simultaneous speakers, 4) normalizing by the total duration. As expected, in the majority of cases, there is either one speaker or no speaker; overlaps with more than three participants are extremely rare, in line with previous findings (Çetin & Shriberg, 2006). This suggests that the turn-taking system is robust in multi-party interaction, though overlaps still constitute an important factor (around 10% of speaking time) to consider in downstream processing. We observe that overlaps are even more rare in zoom meetings. See Appendix B.4 for full results.

**Filled pause.** To assess the degree of conversational fluidity, we considered the number of filled pauses (e.g. “euh”, marked as *fp* in the final annotation). The results showed no significant difference for data set, split or task, although there were more filled silences for Zoom meetings than in-person meetings ($p \le 10^{-5}$). Full results can be found in Appendix B.5.

**Dominant speaker.** We also looked at whether certain speakers spoke substantially more than others. Results (Appendix B.6) show that on average, there is a dominant speaker and that they tend to speak around 50% of total speaking time followed by the second most active speaker at around 30%. The decrease in percentage is fairly linear.<fnref n="8" /> There were no obvious trends across different conditions.

## 7.2 By condition

**In-person vs. Zoom.** Figure 4 shows that the number of IPUs ($p \le 10^{-5}$) and the number of tokens per minute ($p \le 10^{-5}$) were lower for Zoom than in-person meetings. The number of backchannels per minute was also lower for Zoom ($p \le 0.001$), while the number of filled pauses was significantly larger ($p \le 10^{-5}$). Overall, these results indicate that Zoom meetings are less interactive than in person meetings, which is to be expected given the social distance and the fact that participants often cut their microphones when they did not have the floor (as people tend to in real-life online meetings).

<footnote n="8">8. Some conversations have only three participants meaning the fourth participant's speaking time is set to zero. Thus the data about the fourth participant should be interpreted with care.</footnote>
<figure><img src="image_158.png" /><figcaption>FIGURE 4 – Comparisons of in-person and Zoom meetings from left to right : IPU count, token count, number of backchannels, number of filled pauses</figcaption></figure>

**Differences between different scenarios.** As noted in Section 3, scenario (c) was too difficult and had to be abandoned. This is consistent with the fact that IPU counts, number of tokens per minute, filled pauses and backchannel counts all differed significantly between scenario (c) and the others (see above and Appendix B for figures and details). Scenario (a) also stood out. IPU counts were smaller for (a) (technology) than (e) (environment, *p* ≤ 0.01) and (d) (music, *p* ≤ 0.05). Backchannel counts per minute were different for : a-d (*p* ≤ 0.05), a-e (*p* ≤ 0.01), a-c (*p* ≤ 0.001).

Overlaps paint a slightly different picture. While scenario (c) has more pauses and less speech, consistent with the above, scenario (a) has the smallest amount of silence and highest amount of one person talking. This might suggest that scenario (a) is a “more serious” topic, for which people have a tendency to speak longer and in a less chat-like manner (see Appendix B for figures).

**Differences between different tasks.** Comparison by task revealed that reporting meetings have fewer IPUs than planning ($p \le 10^{-5}$) and decision ($p \le 10^{-4}$), fewer tokens per minute than planning ($p \le 0.05$) and decision ($p \le 0.001$), and fewer backchannels than planning/decision ($p \le 0.0001$). Reporting also had highest portion where only one person is speaking (see Appendix B for figures). This is to be expected due to the fact that reporting meetings were designed in a round-table style in order to encourage monologue. There was no difference in the number of filled pauses across tasks.

# 8 Conclusion

We have presented the SUMM-RE corpus, a new dataset containing 96 hours of spontaneous, multiparty meeting-style conversations in French. The corpus is the only French corpus of its kind, and one of the only large-scale meeting corpora in a language other than English. While the meetings are based on loose role-playing, they remain natural and are designed to elicit basic discursive interactions that we can expect from real meetings. We have offered preliminary analyses of the data to illustrate the level of spontaneity and interactivity in the corpus and have shown how this can vary depending on the type of meeting involved, the subject and—although this was not a part of the initial aim of the corpus—whether the conversation was recorded in-person or on Zoom. The dataset is available on Hugging Face at https://huggingface.co/datasets/linagora/SUMM-RE.

# 9 Acknowledgements

We gratefully acknowledge support from the ANR grant SUMM-RE (ANR-20-CE23-0017).
Références

BAIN M., HUH J., HAN T. & ZISSERMAN A. (2023). Whisperx : Time-accurate speech transcription of long-form audio. *arXiv preprint arXiv :2303.00747*.

BIGI B. (2015). Sppas-multi-lingual approaches to the automatic annotation of speech. *The Phonetician*. *Journal of the International Society of Phonetic Sciences*, **111**(ISSN : 0741-6164), 54–69.

BIGI B. & PRIEGO-VALVERDE B. (2019). Search for inter-pausal units : application to cheese! corpus. In *9th Language & Technology Conference : Human Language Technologies as a Challenge for Computer Science and Linguistics*, p. 289–293.

BLACHE P., BERTRAND R., FERRÉ G., PALLAUD B., PRÉVOT L. & RAUZY S. (2017). The corpus of interactional data : A large multimodal annotated resource. *Handbook of linguistic annotation*, p. 1323–1356.

BOERSMA P. & VAN HEUVEN V. (2001). Speak and unspeak with praat. *Glot International*, **5**(9/10), 341–347.

BREDIN H. & LAURENT A. (2021). End-to-end speaker segmentation for overlap-aware resegmentation. In *Interspeech*.

BREDIN H., YIN R., CORIA J. M., GELLY G., KORSHUNOV P., LAVECHIN M., FUSTES D., TITEUX H., BOUAZIZ W. & GILL M.-P. (2020). pyannote.audio : neural building blocks for speaker diarization. In *IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)*.

CARLETTA J., ASHBY S., BOURBAN S., FLYNN M., GUILLEMOT M., HAIN T., KADLEC J., KARAISKOS V., KRAAIJ W., KRONENTHAL M. et al. (2005). The AMI meeting corpus : A pre-announcement. In *International workshop on machine learning for multimodal interaction*, p. 28–39 : Springer.

ÇETIN O. & SHRIBERG E. (2006). Analysis of overlaps in meetings by dialog factors, hot spots, speakers, and collection site : Insights for automatic speech recognition. In *Ninth international conference on spoken language processing*.

HUNTER J., LOURADOUR J., RENNARD V., HARRANDO I., SHANG G. & LORRÉ J.-P. (2023). The claire french dialogue dataset. *arXiv preprint arXiv :2311.16840*.

JANIN A., BARON D., EDWARDS J., ELLIS D., GELBART D., MORGAN N., PESKIN B., PFAU T., SHRIBERG E., STOLCKE A. et al. (2003). The ICSI meeting corpus. In *2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003 (ICASSP'03)*, volume 1, p. I–I : IEEE.

KLEIN G. (2023). Faster whisper transcription with ctranslate2. *GitHub repository*.

LEE A., KAWAHARA T., SHIKANO K. et al. (2001). Julius-an open source real-time large vocabulary recognition engine. In *INTERSPEECH*, p. 1691–1694.

LOURADOUR J. (2023). whisper-timestamped. *GitHub repository*.

MCCOWAN I., CARLETTA J., KRAAIJ W., ASHBY S., BOURBAN S., FLYNN M., GUILLEMOT M., HAIN T., KADLEC J., KARAISKOS V., KRONENTHAL M., LATHOUD G., LINCOLN M., LISOWSKA MASSON A., POST W., REIDSMA D. & WELLNER P. (2005). The AMI meeting corpus. *International Conference on Methods and Techniques in Behavioral Research*.

NEDOLUZHKO A., SINGH M., HLEDÍKOVÁ M., GHOSAL T. & BOJAR O. (2022). ELITR Minuting Corpus : A novel dataset for automatic minuting from multi-party meetings in English and Czech. In *Proceedings of the 13th International Conference on Language Resources and Evaluation (LREC-2022)*, Marseille, France : European Language Resources Association (ELRA). In print.
RADFORD A., KIM J. W., XU T., BROCKMAN G., MCLEAVEY C. & SUTSKEVER I. (2022).
Robust speech recognition via large-scale weak supervision. *arXiv preprint arXiv:2212.04356*.

RENNARD V., SHANG G., HUNTER J. & VAZIRGIANNIS M. (2023). Abstractive meeting summarization: A survey. *Transactions of the Association for Computational Linguistics*, **11**, 861–884.

WU H., ZHAN M., TAN H., HOU Z., LIANG D. & SONG L. (2023). VCSUM: A versatile Chinese meeting summarization dataset. In A. ROGERS, J. BOYD-GRABER & N. OKAZAKI, Éds., *Findings of the Association for Computational Linguistics : ACL 2023*, p. 6065–6079, Toronto, Canada : Association for Computational Linguistics. DOI: 10.18653/v1/2023.findings-acl.377.

YAMASAKI H., LOURADOUR J., HUNTER J. & PRÉVOT L. (2023). Transcribing and aligning conversational speech: A hybrid pipeline applied to french conversations. In *2023 IEEE Automatic Speech Recognition and Understanding*.
## A Basic data set information

<table><caption>TABLE 2 – Participants metadata summary</caption><tr><td>Age</td><td></td><td>28.7 ± 13.4 years</td></tr><tr><td rowspan="3">Gender</td><td>M</td><td>N = 56</td></tr><tr><td>F</td><td>N = 146</td></tr><tr><td>Other</td><td>N = 5</td></tr><tr><td rowspan="2">Country</td><td>France</td><td>N = 158</td></tr><tr><td>Other</td><td>N = 49</td></tr><tr><td rowspan="2">Occupation</td><td>Student</td><td>N = 120</td></tr><tr><td>Other</td><td>N = 87</td></tr><tr><td>Languages spoken</td><td></td><td>1.6 ± 0.9</td></tr><tr><td>Total</td><td></td><td>207</td></tr></table>

<table><caption>TABLE 3 – Session metadata summary</caption><tr><td rowspan="2">Pilot</td><td>pilot</td><td>N = 22</td></tr><tr><td>experiment</td><td>N = 261</td></tr><tr><td rowspan="4">Location</td><td>zoom</td><td>N = 35</td></tr><tr><td>H2C2</td><td>N = 212</td></tr><tr><td>Home</td><td>N = 18</td></tr><tr><td>LPL</td><td>N = 18</td></tr><tr><td rowspan="3">Task</td><td>Reporting</td><td>N = 95</td></tr><tr><td>Decision</td><td>N = 94</td></tr><tr><td>Planning</td><td>N = 94</td></tr><tr><td rowspan="5">Scenario</td><td>A</td><td>N = 84</td></tr><tr><td>B</td><td>N = 86</td></tr><tr><td>C</td><td>N = 12</td></tr><tr><td>D</td><td>N = 36</td></tr><tr><td>E</td><td>N = 65</td></tr><tr><td rowspan="2">Video</td><td>yes</td><td>N = 241</td></tr><tr><td>no</td><td>N = 42</td></tr><tr><td>duration</td><td></td><td>19min 42 ± 3min 8</td></tr></table>

<figure><img src="image_159.png" /><figcaption>FIGURE 5 – distribution of age by gender</figcaption></figure>
<figure><img src="image_160.png" /></figure>

FIGURE 6 – Left : number of languages spoken by the participant, Right : Highest degree obtained by the participant

<figure><img src="image_161.png" /></figure>

FIGURE 7 – Occupations of the participants

<figure><img src="image_162.png" /></figure>

FIGURE 8 – Countries participants are from (apart from France)
# B Additional linguistic statistics

## B.1 IPU

### B.1.1 Distribution

<figure><img src="image_163.png" /><figcaption>FIGURE 9 – Left : IPU duration distribution by pilot vs experiment, Right : IPU duration distribution by scenario</figcaption></figure>

<figure><img src="image_164.png" /><figcaption>FIGURE 10 – Left : IPU duration distribution by task, Right : IPU duration distribution by place</figcaption></figure>
B.1.2 Count

<figure><img src="image_165.png" /><figcaption>FIGURE 11 – Left : IPU count by pilot vs experiment, Right : IPU count by scenario</figcaption></figure>

<figure><img src="image_166.png" /><figcaption>FIGURE 12 – Left : IPU count by task, Right : IPU count by place</figcaption></figure>
## B.2 Token

### B.2.1 Distribution

<figure><img src="image_167.png" /><figcaption>FIGURE 13 – Left : Token duration distribution by pilot vs experiment, Right : Token duration distribution by scenario</figcaption></figure>

<figure><img src="image_168.png" /><figcaption>FIGURE 14 – Left : Token duration distribution by task, Right : Token duration distribution by place</figcaption></figure>
B.2.2 Count

<figure><img src="image_169.png" /><figcaption>FIGURE 15 – Left : Token count by pilot vs experiment, Right : Token count by scenario</figcaption></figure>

<figure><img src="image_170.png" /><figcaption>FIGURE 16 – Left : Token count by task, Right : Token count by place</figcaption></figure>
B.3 Backchannel

<figure><img src="image_171.png" /><figcaption>FIGURE 17 – Left : Backchannel count by pilot vs experiment, Right : Backchannel by scenario</figcaption></figure>

<figure><img src="image_172.png" /><figcaption>FIGURE 18 – Left : Backchannel count by task, Right : Backchannel by place</figcaption></figure>
B.4 Overlap

<figure><img src="image_173.png" /><figcaption>FIGURE 19 – Left : Overlaps by pilot vs experiment, Right : Overlaps by scenario</figcaption></figure>

<figure><img src="image_174.png" /><figcaption>FIGURE 20 – Left : Overlaps by task, Right : Overlaps by place</figcaption></figure>
B.5 Filled pause

<figure><img src="image_175.png" /><figcaption>FIGURE 21 – Left : Filled Pause by pilot vs experiment, Right : Filled Pause by scenario</figcaption></figure>

<figure><img src="image_176.png" /><figcaption>FIGURE 22 – Left : Filled Pause by task, Right : Filled Pause by place</figcaption></figure>
## B.6 Dominant speaker

<figure><img src="image_177.png" /><figcaption>FIGURE 23 – Left : Speaker dominance by pilot vs experiment, Right : Speaker dominance by scenario</figcaption></figure>

<figure><img src="image_178.png" /><figcaption>FIGURE 24 – Left : Speaker dominance by task, Right : Speaker dominance by place</figcaption></figure>
# Claire: Large Language Models for Spontaneous French Dialogue

Jérôme Louradour¹ Julie Hunter¹ Ismaël Harrando¹
Guokan Shang² Virgile Rennard¹ Jean-Pierre Lorré¹

(1) LINAGORA, Paris and Toulouse

(2) MBZUAI, Paris

{jlouradour, jhunter, iharrando, vrennard, jplorre}@linagora.com,
guokan.shang@mbzuai.ac.ae

## RÉSUMÉ

Nous présentons la famille de modèles Claire, une collection de modèles de langage conçus pour améliorer les tâches nécessitant la compréhension des conversations parlées, tel que le résumé de réunions. Nos modèles résultent de la poursuite du pré-entraînement de deux modèles de base exclusivement sur des transcriptions de conversations et des pièces de théâtre. Aussi nous nous concentrons sur les données en français afin de contrebalancer l'accent mis sur l'anglais dans la plupart des corpus d'apprentissage. Cet article décrit le corpus utilisé, l'entraînement des modèles ainsi que leur évaluation. Les modèles, les données et le code qui en résultent sont publiés sous licences ouvertes, et partagés sur Hugging Face et GitHub.

## ABSTRACT

We present the Claire family of language models, a collection of foundation models designed to serve as the basis for further fine-tuning on downstream tasks, such as meeting summarization, that require understanding of spoken conversation. Our models result from continuing the pretraining of two foundation models exclusively on conversation transcripts and theater plays. We focus on French dialogue data in an effort to offset the English-heavy focus of much training corpora. This paper describes the data sets and their preparation, as well as model training and evaluation. The resulting models, data and code are released under open licenses and shared on Hugging Face and GitHub.

**MOTS-CLÉS** : Modèles de langue, Français, Dialogue, Parole spontanée, Pré-entraînement.

**KEYWORDS**: Language models, French, Dialogue, Spontaneous speech, Continual pretraining.

# 1 Introduction

A lot of information is shared through spoken conversation – in meetings, medical appointments, assistance calls, lectures, to give just a few examples. Due to the spontaneous nature of such interactions, the kind of language we employ in these contexts can differ considerably from that found in text documents used to train large language models (LLMs). We hesitate, repeat ourselves, revise our wording in mid-sentence, leading to utterances that are ungrammatical by written standards. We might employ slightly different vocabulary and even syntactic constructions.

(1) A : Ok, c'est quoi le plus, le souvenir, le premier souvenir, le plus clair que tu as en tête, de nous deux ?
B : Le souvenir ?
A : Ton premier souvenir de nous en fait.
B : Le premier souvenir de nous le plus clair que j'ai, euh, je pense que c'est en Algérie. Je crois que

<figure><img src="image_179.png" /></figure>
c'est en Algérie. Et je sais pas, si tu te souviens. On était sur la place. Et, euh, et on courait.

In (1), it takes multiple lines and two turn changes for the speakers to establish that the question under discussion is simply, “Quel est ton premier souvenir de nous ?”. A reformulates their question multiple times in the first line, leading to constructions such as “le souvenir le premier souvenir” that we would not expect to find in a grammar book. There are filled pauses with expressions like “euh”, which do not appear in written text. And we have focus constructions such as “Le premier souvenir de nous le plus clair que j’ai, je pense que c’est...” that are characteristic of spoken language. A further contrast with most written documents is that conversation involves multiple speakers.

One might hypothesize that a foundation model that has been trained to be sensitive to the kinds of interactions that we see in spontaneous conversations might serve as a better base for downstream fine-tuning on language generation and understanding tasks that focus on natural conversation. A model for meeting summarization or transcript querying, for example, would need to be able to navigate these types of interactions. It might be interesting to create chatbots that could produce more natural and informal language to improve the feel of interaction.

In this paper, we describe the first step that we took to addressing our hypothesis : the creation of foundation models trained on transcripts of spontaneous conversation and theater plays. Given what would inevitably be a small dataset, we decided to start with foundation models and continue pretraining—that is, to continue training with an autoregressive language modeling objective—rather than training dialogue models from scratch. We were curious to see how the additional French training impacted the quality of French responses and also how conversational we could get our models to be without specific chat-style training. For the latter reason, we chose pure foundation models that had not received specific instruction or chat training. The result was two models : Claire-7B, based on Falcon-7B,<fnref n="1" /> and Claire-Mistral-7B, based on Mistral-7B.

We also decided to focus on French. A drawback of many current LLMs is that most of their training data is in English, which is a double cause for concern. Not only might these models struggle to produce high-quality responses in other languages, but they will also have a poor representation of non-anglophone cultures. Consider this example of Mistral-7B <fnref n="2" /> completion :

<table><tr><td><i>prompt</i> :</td><td>Pour faire un boeuf bourguignon il faut</td></tr></table>

Mistral completion : commencer par un bon vin.

To make a beef bourguignon you must start with a good wine.

– Julia Child

What we see is that the first line of the response is in French but then the model quickly switches to English—a phenomenon that we observed frequently with Mistral (see Section 5). What’s more, the response cites Julia Child—a chef made famous in the States for introducing French cuisine to an American audience, but hardly known in France. Ideally, a French speaker who turns to an LLM to ask a question about French culture (such as French cuisine !) would be able to receive a response in French from a French source.

The resulting Claire models are quite conversational in French and even have a touch of humor. This is an example output from Claire <fnref n="3" /> :

<footnote n="1">1. https://huggingface.co/tiiuae/falcon-7b</footnote>

<footnote n="2">2. https://huggingface.co/mistralai/Mistral-7B-v0.1</footnote>

<footnote n="3">3. Because all of the models that we discuss are 7B size, we drop the “7B” mention for the rest of the paper.</footnote>
<table><tr><td><strong><em>prompt :</em></strong></td><td>— Dis-moi, où est-ce que je peux trouver des cèpes ?</td></tr><tr><td><strong><em>Claire completion :</em></strong></td><td>— En général, on les trouve un peu avant la Toussaint.<br>— Et comment on les reconnaît, ces champignons ?<br>— Ils sont très différents.<br>— Comment tu peux le savoir ?<br>— Parce que, en fait, je suis champignon.<br>— Oh, dis donc.<br>— Oui, c'est moi qui t'ai parlé.<br>— Tu es vraiment un champignon ?<br>— Mais oui.<br>— Tu es un champignon et tu es un homme ?<br>— Je suis un homme et je suis un champignon.</td></tr></table>

We emphasize that our models are foundation models trained to produce extended sequences of dialogue in one go, not to engage in back and forth chat with a human, and that it is expected for them to imitate spoken dialogue features, including disfluencies.

In what follows, we present our dataset and data preparation approach as well as an overview of our model training and evaluation approach.

## 2 Data

While the majority of data used to train high-profile LLMs comes from English (LLaMA2, for example, uses 89.7% English data Touvron et al. (2023)), the large-scale datasets tend to include some data from other languages, especially web-crawled data. ROOTS (Responsible Open-science Open-collaboration Text Sources Laurençon et al., 2023), which BigScience assembled for training BLOOM (Scao et al., 2023), contains 1.6TB of data from 59 languages, including French.<fnref n="4" /> RefinedWeb (Penedo et al., 2023), OSCAR (Suárez et al., 2019), and RedPajama (Computer, 2023) are filtered and refined versions of CommonCrawl data dumps<fnref n="5" /> prepared specifically for LLM training. These datasets do not concentrate on dialogue data as we did for our models, however.

On the dialogue side, there have been recent efforts to assemble collections of dialogue datasets that can be used to train conversational AI agents, though these focus on English. DialogStudio (Zhang et al., 2023) regroups some well-known English spoken dialogue corpora, such as AMI (McCowan et al., 2005), ICSI (Janin et al., 2003) and MediaSum (Zhu et al., 2021). It also contains a variety of short, written dialogues collected through crowdsourcing. Other similar but smaller-scale collections include InstructDial (Gupta et al., 2022) and ParlAI (Miller et al., 2017).

Our dataset (see Hunter et al. (2023) and https://huggingface.co/datasets/OpenLLM-France/Claire-Dialogue-French-0.1) is broken down in Table 1. It benefited considerably from other open-data initiatives such as Ortolang<fnref n="6" /> (Plate-forme d'outils et de ressources linguistiques pour un traitement optimisé de la langue française) and Orféo (Outils et Ressources sur le Français Ecrit et Oral)<fnref n="7" />. Projects such as the CEFC<fnref n="8" /> (Corpus d'Etude pour le Français Contemporain) and Parole Publique

<footnote n="4">4. https://huggingface.co/bigscience-data</footnote>

<footnote n="5">5. https://commoncrawl.org/</footnote>

<footnote n="6">6. https://www.ortolang.fr/fr/accueil/</footnote>

<footnote n="7">7. Orféo platform : http://ortolang107.inist.fr/</footnote>

<footnote n="8">8. https://repository.ortolang.fr/api/content/cefc-orfeo/11/documentation/site-orfeo/home/index.html</footnote>
(Nicolas et al., 2002) also propose both text and oral corpora, and often offer standardized formats.

<table><caption>TABLE 1 – A breakdown of sub-corpora in our training dataset and their sources. The sub-corpora are organized into categories and listed in order of size starting from the largest, at 135 million words, to the smallest, at 86 thousand words. Citations for the original datasets are in Appendix C.</caption><thead><tr><th>Conversation type</th><th>M Words</th><th>Samp. Weight</th><th>Constituent datasets</th></tr></thead><tbody><tr><td>Parliamentary Proceedings</td><td>135</td><td>35 %</td><td>Assemblée Nationale</td></tr><tr><td>Theater</td><td>16</td><td>18 %</td><td>Théâtre Classique, Théâtre Gratuit</td></tr><tr><td>Interviews</td><td>6.4</td><td>29 %</td><td>TCOF, CFPP, CFPB, ACSYNT, PFC, Valibel</td></tr><tr><td>Free conversation</td><td>2.2</td><td>10 %</td><td>CRFP, OFROM, CID, CLAPI, Rhapsodie, ParisStories, PFC, C-ORAL-ROM, ESLO</td></tr><tr><td>Meetings</td><td>1.2</td><td>5 %</td><td>SUMM-RE, LinTO, ORFEO</td></tr><tr><td>Debates</td><td>0.402</td><td>&lt; 2 %</td><td>FreD, ESLO</td></tr><tr><td>Assistance</td><td>0.159</td><td>&lt; 1 %</td><td>ORFEO, UBS, OTG, ESLO</td></tr><tr><td>Presentations</td><td>0.086</td><td>&lt; 0.5 %</td><td>Valibel, LinTO, ESLO</td></tr><tr><td>Total</td><td>160</td><td>100 %</td><td></td></tr></tbody></table>

As reflected in Table 1, we grouped sub-corpora of our dataset into categories to reflect the types of interactions that we might expect them to contain. The categories are based on descriptions provided by those who distributed the corpora, where possible, and some manual verification by authors of this paper. While this approach is far from fool-proof, we hypothesized that having a rough categorization would not only help ensure that our model had seen a diverse distribution of interactions but would also offer the possibility of selecting portions of our dataset that might be more pertinent than others for a specific downstream task. Note that dividing by category led us to split up certain corpora, such as ESLO, ORFEO, LinTO, and PFC.

For parliamentary proceedings, we expect the conversations to be slightly more formal and to include multiple participants. Theater plays imitate multiparty spoken conversation to some extent, but have their own polished, theatrical style. In interviews, we expect more question/answer sequences, whereas free conversations are generally unguided discussions. The meeting category includes both real and simulated meetings and are expected to contain more structured interactions, though arguably less structured than those found in debates, where turns are controlled to a large extent by a moderator and questions are posed in a formal manner. Finally, assistance interactions involve one person asking for information from another in a professional context while presentations involve longer monological sequences, frequently accompanied by question/answer sequences.

The second column of Table 1 reflects the number of words in the corpora for each category before data augmentation, and the third column shows the sampling weights that we chose for each category in order to balance the different types of interactions seen by our models. Each sampling weight represents the chance of taking a random sequence from the corresponding augmented and tokenized sub-corpus to constitute a training batch. The final weights were determined by assigning custom penalties to texts from parliamentary proceedings and from theater plays. These datasets make up around 95% of the diarized transcripts/scripts that we were able to collect; penalizing them allowed us to increase the impact of less formal conversational data which, while more representative of what we were searching for, was also harder to find in the form of transcribed and diarized transcripts.
3 Data preparation and augmentation

As our datasets came from a variety of sources, we had to contend with diverse data formats, and preparing a high-quality dataset for training often required dataset-specific solutions. Some corpora came with punctuation, while others did not. Different corpora adopted different annotation conventions for background noises, laughter, and other sounds. We attempted to standardize these annotations using a few tags like [NOISE], as in (2), so that they could be easily found later if desired. The [PII] tag, which stands for “Personally Identifying Information,” marks anonymized content.

(2) [speaker001:] C'est Madame [PII] qui m'envoie.
[speaker002:] Oui... [NOISE] Que veut-elle ?
[speaker001:] Savoir où en est son contrat !

For training, we opted to remove the [NOISE] labels and to replace the [PII] tags with random names,⁹ yielding results like (3) :

(3) [speaker001:] C'est Madame Ronald qui m'envoie.

Because part of our aim was to train a model to represent the interactive turn taking characteristic of conversation, it was necessary that all transcripts contain speaker labels. Recovering these labels was not always straightforward and sometimes required going into separate files to recombine this information with the transcript. We also standardized the style of the speaker label so that it could be easily identified for data augmentation, as explained below. In the end, we opted for a format in which speaker names were encased in square brackets and followed by a colon before the closing bracket as shown in example (2).

With our conventions in place for marking non-verbal sounds, anonymized content and speaker labels, we were able to exploit the square brackets to augment our data. When dealing with machine learning models, we always have a choice : we can normalize the data so that the model is insensitive to things it should not be sensitive to, or we can augment the data to increase model robustness, without adding a preprocessing, normalization brick. We chose the latter option.

First, we generated multiple variants for speaker labels by either enclosing the label in brackets, as described above and shown in (4-a) and (4-b), or by marking speaker changes with dashes as in (4-c).

(4) a. [Intervenant:] C'est Madame Ronald qui m'envoie.
b. [Michelle Tate:] C'est Madame Ronald qui m'envoie.
c. — C'est Madame Ronald qui m'envoie.

The bracketed variants contained either “Intervenant” or a proper name. The former was chosen as a replacement for “speaker” after preliminary tests with Falcon suggested that this label helped the model return responses in French. The option of using a dash (-) in conversations with only two speakers also came from experiments with Falcon and Mistral in which both models produced dialogues formatted in this way. For augmentation, we used the “Intervenant” labels to add an anonymous alternative for corpora that used names and we used randomly generated proper names to add a named alternative for datasets that were anonymized. The choice between using a first and last name or only a first name throughout a transcript added another dimension for augmentation.

Next, we played with changing case and removing punctuation. Ultimately, we wanted to design our models to be robust to different transcript formats, keeping in mind that these transcripts, many of which will be produced by ASR systems, may need to handle transcripts without case or punctuation.

⁹ While [PII] tags can indicate other types of censored content, such as addresses, our observations suggested that they usually indicate proper names and the majority of our datasets do not distinguish different types of censored content. We judged that the benefit of including this data outweighed the risk of replacing an address with a proper name during data augmentation.
Combining these three options for data augmentation—speaker labels, case and punctuation—left us with up to nine different formats that we could pull from for training. (In some cases, as when a dataset did not contain punctuation or case, we did not have all nine options of course.)

A final step in the data preparation pipeline involved cutting individual documents from our data set into smaller chunks that would fit into the context windows of Falcon and Mistral, which are limited to 2048 and 4096 tokens, respectively. To do this, we split tokenized documents so that each training sample begins with the start of a speech turn.

# 4 Model training

Because many of the original corpora in our training set were shared for research purposes only, we published Claire-Falcon and Claire-Mistral under a non-commercial license (CC BY-SA-NC 4.0). Simultaneously, however, we released two models under an Apache 2.0 license, Claire-Apache and Claire-Mistral-Apache, trained only on the datasets from our corpus that allow commercial use.

To train our Claire models, we used LoRA (Hu et al., 2021) with bfloat16 precision. LoRA is a lightweight technique that greatly reduces the number of parameters to be trained, making training more efficient. Because training with LoRA enforces the change in model parameters to be of low rank, an additional advantage of this method is that it greatly reduces the chances of catastrophic forgetting, in which a pretrained model forgets what it learned from its previous training. While LoRA is often associated with fine-tuning, we note that we applied it to all layers of the original models and used it to continue unsupervised training with an auto-regressive objective, and so our task was closer to pretraining than traditional fine-tuning. Our hyperparameter configuration was : LoRA with rank $r = 16$ and $\alpha = 32$, AdamW optimizer with a learning rate of $1e^{-4}$, batch size of 128, dropout ratio of 0.05, weight decay regularization factor of 0.01, and gradient clipping of norm 1.

Training was carried out on the Jean Zay supercomputer run by GENCI (Grand Equipement National De Calcul Intensif) and installed at IDRIS, the national computer center put in place for the CNRS (Centre national de la recherche scientifique). To take advantage of Jean Zay's multi-GPU nodes, we used Fully Sharded Data Parallel (FSDP) (Xu et al., 2020; Zhao et al., 2023), which shards a model's parameters, gradients and optimizer states across different workers instead of making a full copy of these states on each GPU, greatly facilitating multi-GPU training. We trained with 8 A100 GPUs$^{10}$, each with 80GB of memory and processed tokens at a rate between 7 and 8 million tokens per hour.

Figure 1 shows the convergence curves for the different variants of the model. The monitored loss is cross-entropy, which is in the case of multi-class classification (with discrete targets that are text tokens here) equivalent to the average Negative Log-Likelihood. Note that perplexity of the model is then the exponentiated loss. Convergence curves show how fast that loss decreases on several hold-out validation subsets. The noisy background curve in light blue is the cost on the training samples estimated in real time during training.

The first scale indicates the number of sequences that the model has seen up to a certain point, where a sequence can be a whole conversation or a proper part of a conversation when context size limitations forced to split the conversation. The second scale shows the number of training (subword) tokens, including padding tokens that are ignored in the loss function : this total number of tokens simply equals the number of sequences multiplied by the training context size (2048 for Falcon, 4096 for

<footnote n="10">10. only Claire-Falcon was trained on a single GPU, with a slightly different batch size (132 instead of 128).</footnote>
<figure><img src="image_180.png" /><figcaption>FIGURE 1 – Convergence curves for the different models.</figcaption></figure>

Mistral). The last scale measures training GPU hours. With around 50H GPU hours, training a Claire model consumes a total energy around 25 kWh on the Jean Zay supercomputer, equivalent to around 1.5kg C02 eq following the estimation of (Luccioni et al., 2022).

The thick grey line is the online validation curve that we used for early stopping; that is, training is automatically stopped when the online validation loss does not decrease for some time. The colored lines show the loss curves for our data by subcategories. To get these results, we produced model checkpoints every hour and then tested the different types of data at the various checkpoints. The legend in the figure on the right sorts the data sets by level of learning difficulty. We see that the theater pieces were the most difficult to learn, probably because they were quite heterogeneous in nature, ranging from old French up through modern plays. After that come conversations, then meetings, speeches, debates, interviews, the National Assembly, which is our biggest dataset, and then call-center type dialogues, which seem to be the easiest.

Figure 1 shows that overfitting appears earlier and more prominently for the Apache models, which is explained by the fact that they did not see all of the data presented in Section 2. It might also seem to suggest that that Claire-Mistral (bottom left) learns more efficiently than Claire-Falcon (top left) and converges to a lower loss. However, the evaluation results presented in Section 5 reveal the opposite, showing that average token perplexity is not relevant to compare models with different tokenization and different context sizes.
# 5 Human evaluation

Evaluation of causal foundation models—which have only been trained with an autoregressive training objective and not fine-tuned to follow instructions or perform a specific task—is a nebulous affair. The only gold data against which model output can be compared is data held out from the training set for validation, which is just a diverse collection of (punctuated) sequences of words. For this reason, foundation models are often evaluated for perplexity, or how well they learn to match a probability distribution over word sequences with the actual probability distribution of the validation set.

This does not mean that foundation models should not be evaluated using standard benchmarks, of course: such evaluation can facilitate the comparison between a model's behavior before and after fine-tuning and can be used to shed light on how different training corpora can impact the model. It's just that the output of these evaluations should be taken with a grain of salt.

In our case, we saw that our Claire models responded to their continued training and wanted to get a clearer idea of how. In particular, we wanted to see to what extent the dialogue data encouraged a style of language that appears interactive, conversational and spontaneous. Standard benchmarks, however, tend to focus on knowledge and reasoning-based tasks, including general knowledge tasks (e.g., SQuADv2; Rajpurkar et al., 2016), natural language inference (e.g., HellaSwag; Zellers et al., 2019), coding (e.g., MBPP; Austin et al., 2021), maths (GSM8K; Cobbe et al., 2021), and occasionally a combination of multiple domains (e.g., MMLU from Hendrycks et al. (2021) and BIG-bench from bench authors (2023)). Not even MT-Bench (Zheng et al., 2023), which assesses chat interactions, focuses on a model's ability to generate spoken-style conversational interactions.

There are also limited options for benchmarks in French. One approach is to translate the existing resources into other languages, as is done by Bactrian-X (Li et al., 2023), Taco (Upadhayay & Behzadan, 2023) and MT-Bench-Fr<fnref n="11" />, though it has been noted that the translation process can compromise the quality of the dataset, and thus of the evaluation. Another approach is to use the high-quality but limited evaluation resources available for a specific target language as Bawden et al. (2024) do in their evaluation of Bloom (BigScience Workshop, 2022) using hand-picked French datasets. These evaluation suites still do not target dialogue dynamics however.

To evaluate our Claire models, we developed our own approach to human evaluation that targets conversational abilities, evaluating model output along three dimensions: Interaction, Fluency and Relevance. For Interaction, our main interest was to determine not whether the model produced coherent language but rather if it succeeded in acting like it was engaging in conversation. We looked for evidence of turn taking, direct addressing of the “other speaker(s)”, and expressions used to smooth conversation such as “well yeah”. (5) shows an example question from the Interaction dimension.

(5) En cas de dialogue avec plusieurs échanges, semble-t-il que les interlocuteurs cherchent à engager une discussion (en se tutoyant/vouvoyant directement, répondant aux questions, utilisant des expressions conversationnelles comme “oui, c'est vrai”, etc.) ?

Fluency questions focused on such factors as whether the model consistently output French, whether the French was acceptable for oral conversation, and more generally, whether the model output seemed human. For Relevance, evaluators were asked to judge whether the response actually addressed the prompt, whether the response stayed more or less on topic, whether it was logically coherent.

We carried out two evaluation campaigns. In both, each evaluator was asked to review a set of five surveys, where each survey included one prompt together with the generated responses from the

<footnote n="11">11. https://huggingface.co/datasets/bofenghuang/mt-bench-french</footnote>
four models that we wished to compare : Falcon, Mistral, Claire-Falcon and Claire-Mistral. For each generated output, evaluators were asked 13 questions (covering the three dimensions) and additionally asked to rank the four generated outputs. We created 8 distinct groups of surveys, leading to 40 unique surveys for each campaign. Each survey was reviewed by two people.

For each campaign, we made a list of 10 prompts and varied the form of the speaker label along two dimensions. In the first campaign, we used either the [Intervenant:] label or no speaker label at all, keeping a 50/50 distribution in our surveys. The idea here was two-fold: use monologue-style prompts to test whether the Claire models would output dialogue even without dialogue-specific prompting and also minimize the bias towards our models, which were trained on [Intervenant:] labels. In the second campaign, all prompts included speaker labels. Here we opted for a 50/50 distribution between labels with brackets and those without. The former category included [Intervenant:] labels but also proper names in brackets; the latter included turns marked with dashes and proper names without brackets. The idea again was to make sure that half of the prompt variants were in a style that had been seen by the Claire models while the other half remained neutral for the base models. We also randomly selected two generated responses per prompt variant (per model) so that each of the 10 prompts was seen in four different ways (2 speaker-label styles x 2 samples) throughout the surveys.

In the second survey, we varied the nature of the prompt. In the first round, our ten prompts (a mixture of interrogative and declarative prompts) were what we call “culture prompts”. These, like *Dis-moi, où est-ce que je peux trouver des cèpes?*, were designed to discuss concepts that were assumed to be better represented in French training data than in English in order to target the models’ cultural training as well. In the second round, we kept five culture prompts that we felt were the most natural from the first round and added five “casual” prompts in which we asked the model generic questions such as “What will you cook for us today?” or “What are your favorite French novels?”. This shows Claire-Falcon’s response to the casual prompt :

<table><tr><td><i>prompt :</i></td><td>[Dominique:] Dis moi, quels sont tes hobbies ou centres d'intérêt ?</td></tr><tr><td><i>Claire completion :</i></td><td>[Camille:] J'adore la lecture. Ça me permet de m'évader.<br/>[Dominique:] Tu peux me parler de tes lectures ?<br/>[Camille:] Je n'ai pas trop de centre d'intérêt. Je lis tout, de la science-fiction à la poésie, des thrillers aux livres historiques. Je me laisse pas enfermer dans une seule catégorie. Ça m'arrive de lire un bouquin et après de passer à un autre qui m'intéresse tout autant.<br/>[Dominique:] Ok, et tu as une bibliothèque ?<br/>[Camille:] Non. Je lis en ligne.</td></tr></table>

Our two campaigns of evaluation revealed certain interesting tendencies (see Appendix B for more details). First, while Mistral had a clear tendency to output English or switch to code or other data formats when prompted in French, our Claire-Mistral model did not, suggesting that a small amount of additional French training was sufficient to correct the problem.

Claire-Falcon was consistently rated above Claire-Mistral, while Mistral was consistently rated below the other three models. Note that this ranking of Mistral runs contrary to what one might expect from the perplexity results in Figure 1 in the Appendix, underscoring our warning from Section 4 that perplexity does not permit a direct comparison of the models.

In both rounds of evaluation, the question that correlated the most with a model's overall preference rank was “Does the output seem human?”, which underscores the difficulty of evaluating foundation models on a task like spontaneous dialogue generation. In second place, positive responses to whether
the model generated excessive disfluencies were negatively correlated with model preference, even though such disfluencies are common in spoken language and ultimately quite “human”.

The comparison of Claire-Falcon and Falcon was more complex. In the first round, Claire-Falcon was the clear winner, but this advantage disappeared in the second round. In the end, we traced the difference in results to the nature of the prompts: the results for Claire-Falcon and Falcon on culture prompts were consistent in the two rounds of evaluation. However, Falcon outperforms Claire-Falcon on casual prompts, which were introduced only in round two, explaining why Claire-Falcon was roughly tied with Falcon in the second round. These results show how much LLM performance can be influenced by subtle, and often seemingly unexplainable, differences in prompts.

Because of the incredible cost of human annotation, we decided to carry out our evaluation experiments using GPT-3.5 and GPT-4 as evaluators, following the *LLM-as-a-Judge* evaluation paradigm. Unfortunately, testing on December 14 and 15, 2023, we found that both models, and especially GPT-3.5, were unable to differentiate the performance of the models we were evaluating, especially on subjective questions, adding to the already existing body of evidence that LLMs are unable to match or replace human evaluators when it comes to assessing human preference (Koo et al., 2023).

# 6 Conclusion

We have presented the Claire family of language models, a set of models designed to focus on dialogue dynamics with the aim of improving performance on downstream tasks involving spoken dialogue understanding. This work complements that of Pelloin et al. (2022), though our datasets are largely manually transcribed and include more spontaneous spoken language (as opposed to language from TV or radio). Our models were produced by continuing the training of the Falcon and Mistral 7B foundation models on conversation transcripts and theater plays. A key feature of our models, apart from the dialogue component, is their focus on French, contributing to a growing trend to emphasize multi-linguality in language models (Faysse et al., 2024; Groeneveld et al., 2024; Scao et al., 2023). Our two principal models are released under a CC BY-SA-NC 4.0 license<fnref n="12" /> while the other two, trained on a subset of our data that allows commercial use, have an Apache 2.0 license¹³. The GitHub repository with code used to train our Claire models is also available publicly¹⁴. We are currently working on extending the approach presented in this paper to train a bilingual English-French dialogue model with the objective of fine-tuning the resulting model for meeting summarization in order to test the impact of our dialogue pretraining on downstream tasks involving dialogues.

# 7 Acknowledgements

This work was granted access to the HPC resources of IDRIS under the GENCI Grant 2023-AD011014561. It was supported by the ANR projects SUMM-RE (ANR-20-CE23-0017) and LLM4ALL (ANR-23-IAS1-0008-02) and the European Union project Cortex2 (101070192).

<footnote n="12">12. Claire-Falcon model is available at https://huggingface.co/OpenLLM-France/Claire-7B-0.1 and Claire-Mistral at https://huggingface.co/OpenLLM-France/Claire-Mistral-7B-0.1
13. Claire-Falcon with Apache license is available at https://huggingface.co/OpenLLM-France/Claire-7B-Apache-0.1 and Claire-Mistral with Apache license at https://huggingface.co/OpenLLM-France/Claire-Mistral-7B-Apache-0.1
14. https://github.com/OpenLLM-France/Lit-Claire</footnote>
Références

ANTOINE J.-Y., GOULIAN J., VILLANEAU J. & LE TALLEC M. (2009). Word order phenomena in spoken french : a study on four corpora of task-oriented dialogue and its consequences on language processing. In *Proc. Corpus Linguitics*.

ANTOINE J.-Y., LETELLIER-ZARSHENAS S., NICOLAS P. & SCHADLE I. (2002). Corpus OTG et ECOLE MASSY : vers la constitution d'un collection de corpus francophones de dialogue oral diffusés librement. *Actes TALN 2002*.

ASSEMBLÉE NATIONALE (2023). https://www.assemblee-nationale.fr/.

ATILF (2020). TCOF : Traitement de corpus oraux en français. ORTOLANG (Open Resources and TOols for LANGuage) -www.ortolang.fr.

AUSTIN J., ODENA A., NYE M., BOSMA M., MICHALEWSKI H., DOHAN D., JIANG E., CAI C., TERRY M., LE Q. *et al.* (2021). Program synthesis with large language models. *arXiv preprint arXiv:2108.07732*.

AVANZI M. (2012). *L'interface prosodie/syntaxe en français : dislocations, incises et asyndètes*. Gramm-R. P.I.E. Peter Lang.

AVANZI M., BÉGUELIN M.-J., CORMINBOEUF G., FEDERICA D. & JOHNSEN L.-A. (2012–2023). Corpus OFROM – corpus oral de français de suisse romande. Université de Neuchâtel.

AVANZI M., SIMON A.-C., GOLDMAN J.-P. & AUCHLIN A. (2010). C-PROM. un corpus de français parlé annoté pour l'étude des proéminences. *Actes des 23èmes journées d'étude sur la parole*.

BALDAUF-QUILLIATRE H., COLÓN DE CARVAJAL I., ETIENNE C., JOUIN-CHARDON E., TESTON-BONNARD S. & TRAVERSO V. (2016). CLAPI, une base de données multimodale pour la parole en interaction : apports et dilemmes. *Corpus, 15*, 165–194. DOI: 10.4000/corpus.2991, HAL: halshs-01316283.

BAWDEN R., BOURFOUNE H., CABOT B., CASSEREAU N., CORNETTE P., NAGUIB M., NÉVÉOL A. & YVON F. (2024). Les modèles Bloom pour le traitement automatique de la langue française. working paper or preprint.

BENCH AUTHORS B. (2023). Beyond the imitation game : Quantifying and extrapolating the capabilities of language models. *Transactions on Machine Learning Research*.

BERTRAND R., BLACHE P., ESPESSER R., FERRÉ G., MEUNIER C., PRIEGO-VALVERDE B. & RAUZY S. (2008). Le CID - corpus of interactional data - annotation et exploitation multimodale de parole conversationnelle. *Revue TAL : traitement automatique des langues, 49*(3), pp.105–134.

BIGSCIENCE WORKSHOP (2022). BLOOM (revision 4ab0472). DOI: 10.57967/hf/0003.

BLACHE P., BERTRAND R., BRUNO E., BIGI B., ESPESSER R., FERRÉ G., GUARDIOLA M., HIRST D., TAN N., CELA E., MARTIN J.-C., RAUZY S., MOREL M.-A., MURISASCO E. & NESTERENKO I. (2010). Multimodal annotation of conversational data. In N. XUE & M. POESIO, Éds., *Proceedings of the Fourth Linguistic Annotation Workshop*, p. 186–191, Uppsala, Sweden : Association for Computational Linguistics.

BRANCA-ROSOFF S., FLEURY S., LEFEUVRE F. & PIRES M. (2013). Corpus de français parlé parisien des années 2000 (CFPP).

CARRUTHERS J. (2008). Annotating an oral corpus using the text encoding initiative. methodology, problems, solutions. *Journal of French Language Studies, 18*(1), 103–119. DOI: 10.1017/S0959269507003183.
CLESTHIA (2018). Cfpp2000. ORTOLANG (Open Resources and TOols for LANGuage) -www.ortolang.fr.

COBBE K., KOSARAJU V., BAVARIAN M., CHEN M., JUN H., KAISER L., PLAPPERT M., TWOREK J., HILTON J., NAKANO R., HESSE C. & SCHULMAN J. (2021). Training verifiers to solve math word problems. *arXiv preprint arXiv:2110.14168*.

COGNITION, LANGUE, LANGAGES, ERGONOMIE (CLLE) (2013). ACSYNT. ORTOLANG (Open Resources and TOols for LANGuage) -www.ortolang.fr.

COMPUTER T. (2023). Redpajama : an open dataset for training large language models.

CRESTI E., DO NASCIMENTO F. B., SANDOVAL A. M., VERONIS J., MARTIN P. & CHOUKRI K. (2004). The C-ORAL-ROM CORPUS. a multilingual resource of spontaneous speech for Romance languages. In M. T. LINO, M. F. XAVIER, F. FERREIRA, R. COSTA & R. SILVA, Éds., *Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)*, Lisbon, Portugal : European Language Resources Association (ELRA).

DEBAISIEUX J.-M., BENZITOUN C. & DEULOFEU H.-J. (2016). Le projet ORFEO : Un corpus d'études pour le français contemporain. *Corpus, 15*, 91–114. DOI : 10.4000/corpus.2936, HAL : hal-01449600.

DISTER A., FRANCARD M., HAMBYE P. & SIMON A. C. (2007). Du corpus à la banque de données. du son, des textes et des métadonnées. I’ évolution de banque de données textuelles orales VALIBEL (1989-2006). *Cahiers de Linguistique, 33*(2), 113–129.

DISTER A. & LABEAU E. (2017). Le corpus de français parlé à bruxelles : origines, hypothèses, développements et prédicitions. *Cahiers AFLS, 21*(1).

DURAND J., LAKS B. & LYCHE C. (2009). Le projet PFC (Phonologie du Français Contemporain) : une source de données primaires structurées. *Phonologie, variation et accents du français*, p. 19–61.

ESHKOL-TARAVELLA I., BAUDE O., MAUREL D., HRIBA L., DUGUA C. & TELLIER I. (2011). Un grand corpus oral disponible : le corpus d'orléans 1968-2012 [a large available oral corpus : Orleans corpus 1968-2012]. *Traitement Automatique des Langues, 52*(3), 17–46.

FAYSSE M., FERNANDES P., GUERREIRO N., LOISON A., ALVES D., CORRO C., BOIZARD N., ALVES J., REI R., MARTINS P. et al. (2024). Croissantllm : A truly bilingual french-english language model. *arXiv preprint arXiv:2402.00786*.

FISCHER F., BÖRNER I., GÖBEL M., HECHTL A., KITTEL C., MILLING C. & TRILCKE P. (2019). Programmable Corpora : Introducing DraCor, an Infrastructure for the Research on European Drama. In *Proceedings of DH2019 : "Complexities", Utrecht, July 9–12, 2019* : Utrecht University. DOI : 10.5281/zenodo.4284002.

GRAVELLIER L., HUNTER J., MULLER P., PELLEGRINI T. & FERRANÉ I. (2021). Weakly supervised discourse segmentation for multiparty oral conversations. In M.-F. MOENS, X. HUANG, L. SPECIA & S. W.-T. YIH, Éds., *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, p. 1381–1392, Online and Punta Cana, Dominican Republic : Association for Computational Linguistics. DOI : 10.18653/v1/2021.emnlp-main.104.

GROENEVELD D., BELTAGY I., WALSH P., BHAGIA A., KINNEY R., TAFJORD O., JHA A. H., IVISON H., MAGNUSSON I., WANG Y. et al. (2024). Olmo : Accelerating the science of language models. *arXiv preprint arXiv:2402.00838*.

GUPTA P., JIAO C., YEH Y.-T., MEHRI S., ESKENAZI M. & BIGHAM J. (2022). InstructDial : Improving zero and few-shot generalization in dialogue through instruction tuning. In Y. GOLDBERG, Z. KOZAREVA & Y. ZHANG, Éds., *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, p. 505–525, Abu Dhabi, United Arab Emirates : Association for Computational Linguistics. DOI : 10.18653/v1/2022.emnlp-main.33.
HENDRYCKS D., BURNS C., BASART S., ZOU A., MAZEIKA M., SONG D. & STEINHARDT J. (2021). Measuring massive multitask language understanding. *Proceedings of the International Conference on Learning Representations (ICLR)*.

HU E. J., SHEN Y., WALLIS P., ALLEN-ZHU Z., LI Y., WANG S. & CHEN W. (2021). Lora : Low-rank adaptation of large language models. CoRR, abs/2106.09685.

HUNTER J., LOURADOUR J., RENNARD V., HARRANDO I., SHANG G. & LORRÉ J.-P. (2023). The claire french dialogue dataset. arXiv preprint arXiv:2311.16840.

JANIN A., BARON D., EDWARDS J., ELLIS D., GELBART D., MORGAN N., PESKIN B., PFAU T., SHRIBERG E., STOLCKE A. et al. (2003). The ICSI meeting corpus. In *2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003 (ICASSP'03)*, volume 1, p. I–I : IEEE.

KAHANE S., CARON B., STRICKLAND E. & GERDES K. (2021). Annotation guidelines of UD and SUD treebanks for spoken corpora : A proposal. In D. DAKOTA, K. EVANG & S. KÜBLER, Éds., *Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021)*, p. 35–47, Sofia, Bulgaria : Association for Computational Linguistics.

KOO R., LEE M., RAHEJA V., PARK J. I., KIM Z. M. & KANG D. (2023). Benchmarking cognitive biases in large language models as evaluators.

LACHERET A. (2003). *La prosodie des circonstants en français parlé*. Volume 85 de (Collection Linguistique). Peeters. ISBN : 90-429-1414-9 (Peeters Leuven). - 2-87723-771-0 (Peeters France)., HAL : halshs-00349268.

LAURENÇON H., SAULNIER L., WANG T., AKIKI C., DEL MORAL A. V., SCAO T. L., WERRA L. V., MOU C., PONFERRADA E. G., NGUYEN H., FROHBERG J., ŠAŠKO M., LHOEST Q., MCMILLAN-MAJOR A., DUPONT G., BIDERMAN S., ROGERS A., ALLAL L. B., TONI F. D., PISTILLI G., NGUYEN O., NIKPOOR S., MASOUD M., COLOMBO P., DE LA ROSA J., VILLEGAS P., THRUSH T., LONGPRE S., NAGEL S., WEBER L., MUÑOZ M., ZHU J., STRIEN D. V., ALYAFEAI Z., ALMUBARAK K., VU M. C., GONZALEZ-DIOS I., SOROA A., LO K., DEY M., SUAREZ P. O., GOKASLAN A., BOSE S., ADELANI D., PHAN L., TRAN H., YU I., PAI S., CHIM J., LEPERCQ V., ILIC S., MITCHELL M., LUCCIONI S. A. & JERNITE Y. (2023). The BigScience ROOTS Corpus : A 1.6TB Composite Multilingual Dataset.

LI H., KOTO F., WU M., AJI A. F. & BALDWIN T. (2023). Bactrian-x : A multilingual replicable instruction-following model with low-rank adaptation.

LPL (2021). Transcriptions du corpus CID. ORTOLANG (Open Resources and TOols for LAN-Guage) –www.ortolang.fr.

LUCCIONI A. S., VIGUIER S. & LIGOZAT A.-L. (2022). Estimating the carbon footprint of bloom, a 176b parameter language model.

MCCOWAN I., CARLETTA J., KRAAIJ W., ASHBY S., BOURBAN S., FLYNN M., GUILLEMOT M., HAIN T., KADLEC J., KARAISKOS V., KRONENTHAL M., LATHOUD G., LINCOLN M., LISOWSKA MASSON A., POST W., REIDSMA D. & WELLNER P. (2005). The AMI meeting corpus. *International Conference on Methods and Techniques in Behavioral Research*.

MERTENS P. (1987). L'intonation du français. *De la description linguistique à la reconnaissance automatique. unpublished doctoral dissertation, Catholic University of Leuven, Belgium*.

MILLER A., FENG W., BATRA D., BORDES A., FISCH A., LU J., PARIKH D. & WESTON J. (2017). ParlAI : A dialog research software platform. In L. SPECIA, M. POST & M. PAUL, Éds., *Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing : System Demonstrations*, p. 79–84, Copenhagen, Denmark : Association for Computational Linguistics. DOI : 10.18653/v1/D17-2014.
MILLING C., FISCHER F. & (EDS.) M. G. (2021). French Drama Corpus (FreDraCor) : A TEI P5 Version of Paul Fièvre's "Théâtre Classique" Corpus. https://github.com/dracor-org/fredracor.

MODYCO & RUG (2017). PFC - phonologie du français contemporain. ORTOLANG (Open Resources and TOols for LANGuage) -www.ortolang.fr.

NICOLAS P., LETELLIER-ZARSHENAS S., SCHADLE I., ANTOINE J. & CAELEN J. (2002). Towards a large corpus of spoken dialogue in french that will be freely available : the "parole publique" project and its first realisations. In *Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002, May 29-31, 2002, Las Palmas, Canary Islands, Spain* : European Language Resources Association.

PELLOIN V., DARY F., HERVÉ N., FAVRE B., CAMELIN N., LAURENT A. & BESACIER L. (2022). Asr-generated text for language model pre-training applied to speech tasks. *arXiv preprint arXiv:2207.01893*.

PENEDO G., MALARTIC Q., HESSLOW D., COJOCARU R., CAPPELLI A., ALOBEIDLI H., PANNIER B., ALMAZROUEI E. & LAUNAY J. (2023). The RefinedWeb dataset for Falcon LLM : outperforming curated corpora with web data, and web data only. *arXiv preprint arXiv:2306.01116*.

RAJPURKAR P., ZHANG J., LOPYREV K. & LIANG P. (2016). SQuAD : 100,000+ Questions for Machine Comprehension of Text. *arXiv e-prints*, p. arXiv:1606.05250.

RENNARD V., SHANG G., GRARI D., HUNTER J. & VAZIRGIANNIS M. (2023). FREDSum : A dialogue summarization corpus for french political debates. In *Findings of Empirical Methods in Natural Language Processing (EMNLP)*.

RHAPSODIE (2015). https://rhapsodie.modyco.fr/propriete-intellectuelle/.

SCAO T. L., FAN A., AKIKI C., PAVLICK E., ILIĆ S., HESSLOW D., CASTAGNÉ R., LUCCIONI A. S., YVON F., GALLÉ M., TOW J., RUSH A. M., BIDERMAN S., WEBSON A., AMMANAMANCHI P. S., WANG T., SAGOT B., MUENNIGHOFF N., DEL MORAL A. V., RUWASE O., BAWDEN R., BEKMAN S., MCMILLAN-MAJOR A., BELTAGY I., NGUYEN H., SAULNIER L., TAN S., SUAREZ P. O., SANH V., LAURENÇON H., JERNITE Y., LAUNAY J., MITCHELL M., RAFFEL C., ... 357 AUTHORS ... & WOLF T. (2023). Bloom : A 176b-parameter open-access multilingual language model.

SUÁREZ P. J. O., SAGOT B. & ROMARY L. (2019). Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In *7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)* : Leibniz-Institut für Deutsche Sprache.

THÉÂTRE CLASSIQUE (2022). http://www.theatre-classique.fr/.

THÉÂTRE GRATUIT (2023). https://theatregratuit.com/.

TOUVRON H., MARTIN L., STONE K., ALBERT P., ALMAHAIRI A., BABAEI Y., BASHLYKOV N., BATRA S., BHARGAVA P., BHOSALE S. et al. (2023). Llama 2 : Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*.

TUTIN A. & GROSSMAN F. (2014). L'écrit scientifique : du lexique au discours. *Autour de Scientext*, p. 27-44.

UPADHAYAY B. & BEHZADAN V. (2023). Taco : Enhancing cross-lingual transfer for low-resource languages in llms through translation-assisted chain-of-thought processes. *arXiv preprint arXiv:2311.10797*.

XU Y., LEE H., CHEN D., CHOI H., HECHTMAN B. & WANG S. (2020). Automatic cross-replica sharding of weight update in data-parallel training.
YAMASAKI H., LOURADOUR J., HUNTER J. & PRÉVOT L. (2023). Transcribing and aligning conversational speech: A hybrid pipeline applied to french conversations. In *Workshop on Automatic Speech Recognition and Understanding (ASRU)*.

ZELLERS R., HOLTZMAN A., BISK Y., FARHADI A. & CHOI Y. (2019). Hellaswag: Can a machine really finish your sentence? In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*.

ZHANG J., QIAN K., LIU Z., HEINECKE S., MENG R., LIU Y., YU Z., SAVARESE S. & XIONG C. (2023). DialogStudio: Towards richest and most diverse unified dataset collection for conversational AI. *arXiv preprint arXiv:2307.10172*.

ZHAO Y., GU A., VARMA R., LUO L., HUANG C.-C., XU M., WRIGHT L., SHOJANAZERI H., OTT M., SHLEIFER S., DESMAISON A., BALIOGLU C., DAMANIA P., NGUYEN B., CHAUHAN G., HAO Y., MATHEWS A. & LI S. (2023). Pytorch fsdp: Experiences on scaling fully sharded data parallel.

ZHENG L., CHIANG W.-L., SHENG Y., ZHUANG S., WU Z., ZHUANG Y., LIN Z., LI Z., LI D., XING E., ZHANG H., GONZALEZ J. E. & STOICA I. (2023). Judging LLM-as-a-judge with MT-bench and chatbot arena. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*.

ZHU C., LIU Y., MEI J. & ZENG M. (2021). Mediasum: A large-scale media interview dataset for dialogue summarization. *CoRR*, abs/2103.06410.

ÉQUIPE DELIC (2004). Autour du corpus de référence du français parlé. *Recherches sur le français parlé*, **18**, 265.
# A Survey questions from the second evaluation campaign

Here is a list of the survey questions corresponding to the second round of evaluation. Most of the questions have a yes/no format, with a third option proposed if the question does not apply or to quantify some characteristic (e.g. for 2.d, the options are "Plusieurs erreurs"/"Quelques erreurs"/"Aucune erreur").

## 1. Interaction

(a) En présence de "tours" délimités par des tirets ("-") ou des étiquettes telles que "[Nom 1:]" ou "Prénom:", ces marqueurs sont-ils positionnés de manière intuitive et, pour les étiquettes, dans un ordre plausible ? (Ignorez les erreurs mineures de format, e.g., [Intervenant 2) :])

(b) En excluant les étiquettes de locuteurs, la réponse semble-t-elle davantage être extraite d'une conversation ou d'un document écrit ? (La cohérence de la réponse n'est pas primordiale ici; l'accent est mis sur le style généré par le modèle.)

(c) En cas de dialogue avec plusieurs échanges, semble-t-il que les interlocuteurs cherchent à engager une discussion (en se tuyant/vousvoyant directement, répondant aux questions, utilisant des expressions conversationnelles comme 'oui, c'est vrai', etc.) ?

## 2. Fluidité

(a) La réponse bascule-t-elle vers l'anglais ou un autre type de langage ?

(b) La réponse présente-t-elle des répétitions non motivées ? (Par exemple, le modèle semble-t-il répéter de manière robotique les mêmes expressions, sans justification ? Les répétitions naturelles de la conversation sont considérées comme justifiées.)

(c) Le niveau de disfluences (telles que "euh", "hm", "quoi") vous semble-t-il naturel par rapport à une conversation orale ? (s'il y en a pas, répondez "oui")

(d) En dehors des disfluences et des répétitions robotiques, la partie restante de la réponse est-elle formulée dans un français parlé correct ? (Veuillez noter que les constructions typiques du français parlé, telles que "y a pas" ou "ton père, il est où ?", sont acceptables.)

(e) Dans l'ensemble, avez-vous l'impression que la conversation semble humaine (par opposition à "générée par l'IA") ?

## 3. Pertinence

(a) La suite de la conversation, répond-elle de manière spécifique au début de la conversation proposé, même si elle est incorrecte sur le plan factuel ? (Cela s'oppose à une suite qui pourrait être tout aussi valable pour un début de conversation bien différent.)

(b) Y a-t-il des changements de sujet ou de ton qui attirent particulièrement l'attention ? (Ignorez les changements légers de sujet courants dans la conversation spontanée; un passage à un sujet très différent avant d'avoir traité le premier pourrait être considéré comme frappant.)

(c) Le niveau de disfluences (telles que "euh", "hm", "quoi") vous semble-t-il naturel par rapport à une conversation orale ? La conversation semble-t-elle stagnante sur un sujet (plutôt que de progresser comme elle le devrait) ?

(d) En dehors des disfluences et des répétitions robotiques, la partie restante de la réponse est-elle formulée dans un français parlé correct ? La conversation contient-elle des stagnante ?

(e) Le modèle semble-t-il inventer des choses de manière exagérée ? (On exclut les détails à vérifier dans une encyclopédie; l'accent est mis sur les détails qui semblent inventés et perturbent la cohérence de la conversation.)
## B Evaluation results

As explained in Section 5 and illustrated in Figure 2, Mistral (red) had a clear tendency to switch to English, code or another data format when prompted in French. This tendency disappeared for the Claire-Mistral model (orange). The results in Figure 2 are from our second evaluation campaign and indicate a score out of 80, as 80 was the number of surveys conducted.

<figure><img src="image_181.png" /><figcaption>FIGURE 2 – The tendency of each model to stick to French when prompted in French</figcaption></figure>

Results from both campaigns showed a clear overall preference for Claire-Falcon over Claire-Mistral, with Mistral coming it at fourth place. See Figure 3.

<figure><img src="image_182.png" /><figcaption>FIGURE 3 – Overall rankings of the four models in the second evaluation campaign</figcaption></figure>

Figure 4 shows judgments regarding whether the model seemed human. These results were the most strongly correlated with overall preference. In this evaluation, Claire-Falcon (blue) was very slightly preferred over Falcon (green).

Finally, if we focus on “culture”-style prompts, as shown in Figure 5, we see that Claire-Falcon is clearly preferred over Falcon, a tendency that was observed in the first round of evaluations as well. Falcon takes the lead for “casual” prompts, however.
<figure><img src="image_183.png" /><figcaption>FIGURE 4 – Judgments concerning the number of times the response of a model was judged as human</figcaption></figure>

<figure><img src="image_184.png" /><figcaption>FIGURE 5 – Results showing that Claire-Falcon is clearly preferred for “culture” prompts. (A result reflected also in the first evaluation, which contained only culture prompts.)</figcaption></figure>

C Sources for the original datasets used to train the principal Claire models

ACSYNT : Cognition, Langue, Langages, Ergonomie (CLLE) (2013)

Assemblée Nationale : Assemblée nationale (2023)

Orféo-CEFC : Debaisieux et al. (2016); Carruthers (2008); Tutin & Grossman (2014)

— C-ORAL-ROM : Cresti et al. (2004)

— CRFP : Équipe Delic (2004)

— FLEURON : no citation found

— Valibel : Dister et al. (2007)

Orféo : (other)

— CFPB : Dister & Labeau (2017)

— Réunions De Travail : (Work Meetings)

CFPP : Branca-Rosoff et al. (2013); CLESTHIA (2018)

CID : Bertrand et al. (2008); Blache et al. (2010); LPL (2021)
**CLAPI** : Baldauf-Quilliatre *et al.* (2016)

**ESLO** : Eshkol-taravella *et al.* (2011)

**FREDSum** : Rennard *et al.* (2023)

**LinTO** : Gravellier *et al.* (2021)

**OFROM** : Avanzi *et al.* (2012 2023)

**Parole Publique** :

— **Accueil UBS** : Nicolas *et al.* (2002); Antoine *et al.* (2009)

— **OTG** : Nicolas *et al.* (2002); Antoine *et al.* (2002)

**Paris Stories** : Kahane *et al.* (2021)

**PFC** : Durand *et al.* (2009); MoDyCo & RUG (2017)

**Rhapsodie** : Rhapsodie (2015); Branca-Rosoff *et al.* (2013); Avanzi (2012); Lacheret (2003); Mertens (1987); Avanzi *et al.* (2010); Eshkol-taravella *et al.* (2011); Durand *et al.* (2009)

**SUMM-RE** : Yamasaki *et al.* (2023)

**TCOF** : ATILF (2020)

**Théâtre Classique** : Théâtre classique (2022); Milling *et al.* (2021); Fischer *et al.* (2019)

**Théâtre Gratuit** : Théâtre Gratuit (2023)
# Modéliser la facilité d'écoute en FLE : vaut-il mieux lire la transcription ou écouter le signal vocal ?

Minami Ozawa¹, ² Rodrigo Wilkens¹ Kaori Sugiyama² Thomas François¹

(1) Université catholique de Louvain, IL&C, CENTAL, Louvain-la-Neuve, Belgique

(2) Université Seinan Gakuin, Fukuoka, Japon

minami.ozawa@uclouvain.be, rodrigo.wilkens@uclouvain.be,

sugiyama@seinan-gakuin.jp, thomas.francois@uclouvain.be

## RÉSUMÉ

Le principal objectif de cette étude est de proposer un modèle capable de prédire automatiquement le niveau de facilité d'écoute de documents audios en français. Les données d'entrainement sont constituées d'enregistrements audios accompagnés de leurs transcriptions et sont issues de manuels de FLE dont le niveau est évalué sur l'échelle du Cadre européen commun de référence (CECR). Nous comparons trois approches différentes : machines à vecteurs de support (SVM) combinant des variables de lisibilité et de fluidité, wav2vec et CamemBERT. Pour identifier le meilleur modèle, nous évaluons l'impact des caractéristiques linguistiques et prosodiques ainsi que du style de parole (dialogue ou monologue) sur les performances. Nos expériences montrent que les variables de fluidité améliorent la précision du modèle et que cette précision est différente par style de parole. Enfin, les performances de tous les modèles varient selon les niveaux du CECR.

## ABSTRACT

### Modelling listenability for FFL : is it better to read the transcript or listen to the speech signal ?

Our main goal is to design a model able to automatically predict the level of listenability of audio documents in French as a foreign language (FFL). The training data consists of audio recordings accompanied by their transcriptions and are extracted from FFL textbooks whose level is assigned accordingly to the Common European Framework of Reference (CEFR) scale. We compare three different approaches : support vector machines combining readability and fluency variables, wav2vec, and CamemBERT. To identify the best model, we evaluate the impact of linguistic and fluency features as well as of speech style (dialogue or monologue) on performance. Our experiments show that fluency variables improve model accuracy, and that this accuracy differs by speech style. Finally, the performance of all models varies according to CEFR scales.

MOTS-CLÉS : facilité d'écoute, lisibilité, FLE, wav2vec.

KEYWORDS: listenability, readability, FFL, wav2vec.

## 1 Introduction

La compréhension orale est une activité langagière dynamique activement impliquée dans l'interprétation. Le développement de cette compétence est directement lié à l'acquisition de la compétence langagière en général : si la compréhension orale ne progresse pas, cela peut nuire à la bonne communication dans des situations authentiques (Kumai, 1992). Pour soutenir son développement

<figure><img src="image_185.png" /></figure>
chez les apprenants de langue étrangère, il est courant de faire écouter divers documents audios dans le cadre d'exercices de compréhension à l'audition. L'efficacité de documents audio-visuels authentiques est bien établie (Yoshimi, 2019), mais leur utilisation en contexte de classe reste difficile, car il est ardu de juger de l'adéquation de ce type de documents audios au niveau de compétence linguistique des apprenants. Une solution pour faire face à cette difficulté pratique pourrait consister à se reposer sur des modèles d'intelligence artificielle capables d'évaluer automatiquement le niveau de difficulté de documents oraux, sur le modèle de ce qui est fait en évaluation automatisé de la lisibilité des documents écrits. Si les travaux en lisibilité se sont imposés comme une des thématiques du TAL (François, 2011; Vajjala, 2021), ceux sur la facilité d'écoute (ou *listenability*, en anglais) sont nettement moins nombreux, en particulier en français.

Les objectifs de cette étude sont doubles. Tout d'abord, le principal objectif est de développer un modèle capable de prédire automatiquement le niveau de facilité d'écoute de documents audios en français. À notre connaissance, un tel modèle n'existant actuellement pas en français, il s'agit déjà d'une contribution significative. Dans cette optique, deux questions émergent. D'une part, quelle est la meilleure façon d'encoder les caractéristiques stylistiques des documents oraux ? Pour y répondre, nous comparerons l'apport de variables linguistiques définies par des experts et combinées à l'aide de machines à vecteur de support (SVM) à l'utilisation de l'architecture transformer et CamemBERT (Martin et al., 2020) qui représente le contenu des documents audios à l'aide de plongements de mots. D'autre part, quel est l'apport de variables prosodiques ? En effet, la grande majorité des travaux en facilité d'écoute se limitent à prédire la difficulté sur la base de la transcription et de variables calculées sur le texte, sans prendre en compte les caractéristiques spécifiques à l'oralité. Nous comparerons donc ces deux approches au sein de nos modèles, encodant les caractéristiques prosodiques soit à l'aide de variables incluses dans les SVM, soit en tirant profit de l'architecture wav2vec (Baevski et al., 2020). Le second objectif de cet article vise à évaluer l'impact du style de parole sur les performances des modèles précités. Nous disposons d'une variété de documents oraux que nous regroupons simplement selon deux styles de parole : le dialogue et le monologue. Nous examinons si les performances des modèles diffèrent en fonction de la présence ou de l'absence d'un interlocuteur dans le discours.

Cet article propose d'abord une vue d'ensemble de la notion centrale de la facilité d'écoute dans la section 2, suivie de la description de la méthodologie de recherche dans la section 3. Il présente et compare ensuite les résultats des modèles SVM, de wav2vec et de CamemBERT dans la section 4, avant de conclure.

## 2 Le domaine de la facilité d'écoute

Plusieurs définitions de la facilité d'écoute ont été proposées jusqu'à présent (Harwood, 1950; Harwood & Cartier, 1952; Cartier, 1952; Rubin, 2012). En tenant compte de ces définitions, dans cet article, le terme *facilité d'écoute* est défini comme la facilité ou la difficulté qu'un auditeur donné – avec son expérience spécifique – éprouve à comprendre un discours oral dans une situation de communication particulière, laquelle est fonction de l'effet des caractéristiques stylistiques de ce discours (ex. lexicale, syntaxique, prosodique, etc.) sur les processus cognitifs de l'auditeur.

En tant que domaine de recherche, la facilité d'écoute s'est principalement développée en langue anglaise. Dans les années 1940, ce champ a été initié par l'application des modèles issus des études de lisibilité à l'analyse des transcriptions de la langue orale (Flesch, 1943; Chall & Dial, 1948). Par
la suite, les travaux continuent à évaluer la facilité d'écoute sur la base de transcriptions et avec des formules de lisibilité développées pour l'écrit (Cartier, 1955; O'Keefe, 1971).

Toutefois, peu à peu, des formules spécifiques à l'oral apparaissent. Des recherches (Rogers, 1962; Fang, 1967; Kiyokawa, 1990) conçoivent des modèles directement sur des données orales, mais ils ne prennent toujours en compte que des variables liées au contenu (longueur des phrases, des mots, proportion de mots polylexicaux, etc.) et les observent sur des transcriptions. Au niveau de la facilité d'écoute dans le contexte spécifique des apprenants, il y a une recherche qui utilisent la régression multiple pour combiner une série de variables de contenu afin de prédire la difficulté subjective de compréhension orale des phrases anglaises, évaluée par 90 apprenants japonais de l'anglais (Ueda et al., 2013). Le coefficient de corrélation multiple R de l'équation atteint seulement 0,54.

C'est avec les travaux de Kotani et al. (2014) que, en complément des variables de contenu, des caractéristiques phonologiques sont enfin considérées, à savoir le débit de parole, le taux d'élision<fnref n="1" />, de réduction<fnref n="2" />, de contraction<fnref n="3" />, de liaison<fnref n="4" /> et de déduction<fnref n="5" /> dans une phrase. La présence de coefficients de régression négatifs pour le taux de contraction et de déduction suggère que des variations phonologiques peuvent accroître la facilité d'écoute pour les apprenants. Parallèlement, il y a une recherche qui examine dans quelle mesure le débit de parole, la complexité linguistique et l'explicitation (le degré d'expression explicite des idées) du texte influent sur la compréhension orale en L2 (Révész & Brunfaut, 2013). Les auteurs utilisent des analyses de Rasch et de régression pour estimer la difficulté de 18 tâches et sa relation avec les caractéristiques du texte. Les résultats démontrent que des indices de complexité lexicale et discursive expliquent une part significative de la difficulté des tâches de compréhension orale.

Enfin, plus récemment, les arbres de décision (Kotani & Yoshimi, 2017) ou les machines à vecteurs de support (SVM) (Yoshimi & Kotani, 2020) ont été utilisés. Néanmoins, toutes les recherches susmentionnées ont été menées en anglais. En ce qui concerne la langue française, les investigations sont très restreintes aux travaux de Ruggia (2019, 2020, 2021). De plus, un outil est aussi disponible pour l'évaluation automatique des textes oraux, DeepFLE<fnref n="6" />, mais il reste encore limité à l'analyse des transcriptions et n'intègre pas de variables prosodiques. Enfin, l'utilisation de représentations latentes (ex. plongements de mots) et de wav2vec plutôt que des variables ne semble pas avoir encore été envisagé dans ce domaine.

## 3 Méthodologie

Dans cet article, nous cherchons à évaluer l'apport de variables linguistiques et prosodiques à un modèle de facilité d'écoute pour le français, mais posons également la question de savoir si les représentations latentes du contenu ne sont pas préférables à des variables définies par des experts, comme cela a déjà été établi en lisibilité (Martinc et al., 2021; Yancey et al., 2021). Cette section commence par décrire les données utilisées pour entraîner nos modèles, puis décrit les différentes variables « expert » considérées, avant de se clôturer avec une présentation des différents modèles évalués.

<footnote n="1">1. L'élision est l'élimination des phonèmes.</footnote>

<footnote n="2">2. La réduction est l'affaiblissement du son en transformant une voyelle en schwa.</footnote>

<footnote n="3">3. La contraction est la combinaison de deux mots.</footnote>

<footnote n="4">4. La liaison consiste à relier le son final d'un mot au son initial du mot suivant.</footnote>

<footnote n="5">5. La déduction est l'élimination des sons entre les mots.</footnote>

<footnote n="6">6. http://deeptext.unice.fr/FLE/.</footnote>
## 3.1 Données

Les données sont constituées d'enregistrements audios accompagnés de leurs transcriptions que nous avons extraits de 25 manuels de FLE. Chaque enregistrement s'est vu attribué un niveau de compétence sur l'échelle du Cadre européen commun de référence (CECR), à savoir A1, A2, B1, B2 ou C (C1 et C2 sont regroupés pour notre étude). Le niveau attribué à un enregistrement donné est tout simplement celui du manuel dont il a été tiré.

Plusieurs prétraitements ont été effectués. Tout d'abord, le contenu d'une seule piste audio est en principe traité comme une seule donnée. Toutefois, si une piste contient, par exemple, plusieurs dialogues dans différents contextes, l'audio (et la transcription) est alors divisé manuellement. Ensuite, nous avons éliminé les éléments de soutien pédagogique (tels que les exercices de grammaire et les listes de mots), ainsi que les données provenant des premières et dernières unités de chaque manuel (qui sont trop proches du niveau précédent ou suivant). Dans notre corpus, les enregistrements comprennent des conversations quotidiennes, des annonces au public, des interviews médias et des monologues. Toutes ces données ont été classées manuellement en deux grands styles de parole : les dialogues et les monologues. Enfin, les données dont le contenu informatif est extrêmement élevé ou faible dans une certaine catégorie de niveau/style de parole peuvent réduire l'homogénéité lors de l'examen des caractéristiques de ce niveau/style de parole. Pour cette raison, les cinq données présentant des valeurs aberrantes extrêmes en termes de syllabes par minute sans les pauses ou/et de mots par minute sans les pauses ont été exclues. Ces cinq données se composaient d'un document B2, d'un document C1 et de trois documents C2. Les documents B1, C1 et un des C2 ne comportaient qu'une ou deux phrases. La petite taille des données par rapport au niveau/style pourrait être considérée comme une indication d'hétérogénéité. Les deux autres documents C2 étaient des données avec beaucoup de bruit et des phrases en langue étrangère non mentionnés dans les transcriptions. Ils ont été traitées comme des pauses pour des raisons de commodité et, par conséquent, les valeurs aberrantes ont montré que ces données étaient hétérogènes et ne convenaient pas comme données pour l'analyse. Le nombre total final de paires de données (audio et transcription) est de 1 323 documents (Table 1). Le nombre de mots ainsi que la durée d'enregistrement pour les niveaux C1 et C2 sont largement supérieurs à ceux des autres niveaux. Cependant, les variables que l'on utilise sont normalisées lorsque c'est nécessaire afin d'éviter les biais.

## 3.2 Variables

Dans cette étude, la difficulté de compréhension (comprenant nos cinq niveaux issus du CECR comme modalités) a été utilisée comme variable dépendante tandis que 68 variables capturant les caractéristiques prosodiques et le contenu textuel sont utilisées comme variables indépendantes. Les variables indépendantes sont divisées en deux catégories principales : les variables de fluidité (7 variables) et les variables de lisibilité (61 variables). Dans cet article, les variables liées à la prosodie se limitent à celles relatives au débit de parole, à la durée de parole et aux pauses.

La seule variable prosodique qui a été identifiée dans la littérature comme étant corrélée à la facilité d'écoute est le nombre de mots par minute (Harwood, 1955; Kotani et al., 2014; Kotani & Yoshimi, 2017). Cependant, étant donné le nombre réduit de recherches sur cette question, nous avons décidé d'enrichir la liste des variables prises en compte en nous référant au concept de fluidité, qui est souvent utilisé dans les recherches sur la production orale. La fluidité se réfère à la capacité d'utiliser la langue en temps réel, de mettre l'accent sur le sens et d'utiliser les systèmes lexicaux (Skehan
<table><caption>TABLE 1 – Description du jeu de données, qui précise le nombre de documents, de mots par document et la durée en minute, de façon globale et par style de parole.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Nb de données (%)</th><th colspan="2">Nb de mots (%)</th><th colspan="2">Durée d'enregistrement (minutes) (%)</th></tr><tr><th>Dialogue</th><th>Monologue</th><th>Dialogue</th><th>Monologue</th><th>Dialogue</th><th>Monologue</th></tr></thead><tbody><tr><td>A1</td><td>290 (100 %)</td><td></td><td>18 950 (100 %)</td><td></td><td>160 (100 %)</td><td></td></tr><tr><td></td><td>205 (71 %)</td><td>85 (29 %)</td><td>14 576 (77 %)</td><td>4 374 (23 %)</td><td>120 (75 %)</td><td>40 (25 %)</td></tr><tr><td>A2</td><td>309 (100 %)</td><td></td><td>33 755 (100 %)</td><td></td><td>230 (100 %)</td><td></td></tr><tr><td></td><td>180 (58 %)</td><td>129 (42 %)</td><td>25 665 (76 %)</td><td>8 090 (24 %)</td><td>180 (78 %)</td><td>50 (22 %)</td></tr><tr><td>B1</td><td>240 (100 %)</td><td></td><td>32 818 (100 %)</td><td></td><td>190 (100 %)</td><td></td></tr><tr><td></td><td>146 (61 %)</td><td>94 (39 %)</td><td>25 289 (77 %)</td><td>7 529 (23 %)</td><td>150 (79 %)</td><td>40 (21 %)</td></tr><tr><td>B2</td><td>241 (100 %)</td><td></td><td>51 920 (100 %)</td><td></td><td>290 (100 %)</td><td></td></tr><tr><td></td><td>131 (54 %)</td><td>110 (46 %)</td><td>30 825 (59 %)</td><td>21 095 (41 %)</td><td>170 (59 %)</td><td>120 (41 %)</td></tr><tr><td>C1/C2</td><td>243 (100 %)</td><td></td><td>127 396 (100 %)</td><td></td><td>710 (100 %)</td><td></td></tr><tr><td></td><td>121 (50 %)</td><td>122 (50 %)</td><td>95 934 (75 %)</td><td>31 462 (25 %)</td><td>520 (73 %)</td><td>190 (27 %)</td></tr><tr><td>Total</td><td>1 323 (100 %)</td><td></td><td>264 839 (100 %)</td><td></td><td>1 580 (100 %)</td><td></td></tr><tr><td></td><td>783 (59 %)</td><td>540 (41 %)</td><td>192 289 (73 %)</td><td>72 550 (27 %)</td><td>1 140 (72 %)</td><td>440 (28 %)</td></tr></tbody></table>

& Foster, 1999). Étant donné que la compréhension orale, tout comme la production orale, est une aptitude qui requiert de telles compétences, il est raisonnable d'appliquer ce concept. Nous nous référons à la méthode de calcul utilisée dans les recherches précédentes (Cucchiarini et al., 2002; Ginther et al., 2010; Préfontaine, 2010; Peltonen, 2017; Segalowitz et al., 2017) pour définir les variables utilisées dans cet article, à savoir : (1) le nombre de syllabes par minute, (2) le nombre de syllabes par minute sans les pauses, (3) le nombre de mots par minute, (4) le nombre de mots par minute sans les pauses, (5) la durée totale, (6) la durée moyenne des pauses, et (7) le nombre de pauses par minute.

Pour les variables de lisibilité, 61 variables de FABRA (Wilkens et al., 2022) sont utilisées. FABRA est un outil en ligne permettant de calculer un large éventail de variables prédictives de lisibilité pour le français. Le domaine de la facilité d'écoute s'est développé par l'application des formules de lisibilité à l'analyse des transcriptions, et l'existence signalée de variables valides pour la facilité d'écoute ne peut être ignorée. Cet article se focalise sur les variables de lisibilité qui ont été considérées comme pertinentes dans les études précédentes sur la facilité d'écoute. Ainsi, bien que FABRA puisse fournir des informations sur 509 variables liées à la lisibilité, certaines d'entre elles sont moins pertinentes dans le contexte de la facilité d'écoute et ne font pas partie de notre analyse. Les variables liées au nombre de ponctuations et de guillemets dans FABRA, par exemple, ne sont pas directement pertinentes pour cette analyse. Bien que certaines variables relatives aux erreurs dans FABRA existent, il n'est pas non plus nécessaire de les considérer, étant donné que l'étude ne se concentre pas sur la production des apprenants. En outre, cet article se concentre sur les variables liées aux mots de contenu, nécessaires pour comprendre le sens du discours, plutôt qu'aux mots fonctionnels. De plus, certaines variables de FABRA sont relativement redondantes, mesurant le même phénomène à l'aide de variantes. Dans ce cas, seule l'une d'entre elles est prise en considération (par exemple, la proportion de mots A1 et A2 selon la ressource FLELex (François et al., 2014)<fnref n="7" /> est fort redondante. Dès lors, seule la proportion de mots A1 dans le texte selon la ressource a été prise en considération).

<footnote n="7">7. FLELex est un lexique pour le FLE qui donne les fréquences normalisées des lemmes à chaque niveau du CECRL (http://cental.uclouvain.be/flelex/).</footnote>
Au terme de cette première sélection de variables, 61 variables de lisibilité ont été retenues parmi les variables de FABRA. Cette procédure de sélection est conforme à la première étape de sélection des variables, qui consiste à construire un meilleur ensemble de caractéristiques « ad hoc » à partir de la connaissance du domaine de la facilité d'écoute, selon Guyon & Elisseeff (2003). En résumé, les variables comprennent (1) 2 variables basées sur la longueur (ex. nombre de syllabes par mot), (2) 17 variables lexicales (ex. fréquence moyenne des lemmes dans FLELex pour les noms, les noms propres, les verbes, les adjectifs ou les adverbes, (3) 41 variables syntaxiques basées sur le formalisme UD et l'analyseur Stanza (Qi et al., 2020) (ex. proportion d'indicatifs présent, mots identifiés comme adjectifs, etc.) et (4) le score de la formule de lisibilité de Kandel et Moles (Kandel & Moles, 1958)⁸⁹.

## 3.3 Modèles

Nous comparons trois approches différentes pour entraîner notre modèle de facilité d'écoute. La première s'ancre dans les travaux de lisibilité computationnelle, combinant les 68 variables décrites à la section 3.2 à l'aide de SVM. Elle vise à reproduire une méthodologie fiable, qui a fait ses preuves en lisibilité et est relativement facile à entraîner. La seconde approche consiste simplement à affiner une architecture Camembert (Martin et al., 2020).

Plus original, l'emploi du modèle wav2vec 2.0 (Baevski et al., 2020) en facilité d'écoute vise à capturer au mieux la richesse du signal vocal. Le modèle traite la forme d'onde brute du signal vocal avec un réseau neuronal convolutionnel multicouches pour obtenir des représentations audio latentes. Ces représentations sont ensuite introduites dans un quantificateur et un transformateur. Le quantificateur choisit une unité vocale pour la représentation audio latente à partir d'un inventaire d'unités apprises. Environ la moitié des représentations audio sont masquées avant d'être introduites dans le transformateur. Le transformateur ajoute des informations provenant de l'ensemble de la séquence audio. Enfin, la sortie du transformateur est utilisée pour résoudre une tâche contrastive. Cette tâche exige que le modèle identifie les unités de parole quantifiées correctes pour les positions masquées. Pour la version française, nous utilisons le modèle *facebook/wav2vec2-large-xlsr-53-french*<fnref n="10" />, qui est un modèle wav2vec 2.0 entraîné sur des données audio non annotées de 12 langues provenant du benchmark Common Voice.<fnref n="11" />

Nous avons également ajouté une tête de classification à wav2vec. Cette tête est composée d'un pooling de moyennes 1D sur le dernier état caché de wav2vec. Ensuite, les informations regroupées passent par un MLP composé de deux couches linéaires denses (la première avec une activation tanh et la seconde avec une activation softmax) chacune précédée d'une couche de Dropout avec des probabilités de 0,5 et 0,1.

Afin de tester nos deux hypothèses principales : l'effet des variables prosodiques et du style de parole, plusieurs modèles ont été entraînés sur la base des 3 architectures ci-dessus. En ce qui concerne l'effet des variables prosodiques, trois jeux de variables différents ont été utilisés pour les SVMs : un jeu ne comprenant que des variables de fluidité (SVM-F), un second jeu ne comprenant que des variables de lisibilité (SVM-L) et un dernier jeu incluant à la fois les variables de fluidité et celles de

<footnote n="8">8. Voir https://central.uclouvain.be/fabra/docs.html pour le détail des variables.</footnote>

<footnote n="9">9. Les détails de toutes les variables utilisées dans cette étude sont donnés en annexe.</footnote>

<footnote n="10">10. https://huggingface.co/facebook/wav2vec2-large-xlsr-53-french</footnote>

<footnote n="11">11. Pour plus d'informations, voir https://ai.meta.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/.</footnote>