TCFLE-8: a Corpus of Learner Written Productions for French as a Foreign Language and its Application to Automated Essay Scoring
Authors: Rodrigo Wilkens
Alice Pintard
David Alfter
Vincent Folny
Thomas François
Venue: EMNLP 2023 Main (2023)
Keywords: Learner corpus
AES
French
Learner written essays
French certification exam
TL;DR: A new French corpus composed of 6.5k essays from a certification exam, its description and some experiments with AES.

Abstract: Automated Essay Scoring (AES) aims to automatically assess the quality of essays. Automation enables large-scale assessment, improvements in consistency, reliability, and standardization. Those characteristics are of particular relevance in the context of language certification exams. However, a major bottleneck in the development of AES systems is the availability of corpora, which, unfortunately, are scarce, especially for languages other than English. In this paper, we aim to foster the development of AES for French by providing the TCFLE-8 corpus, a corpus of 6.5k essays collected in the context of the \textit{Test de Connaissance du Français} (TCF - French Knowledge Test) certification exam. We report the strict quality procedure that led to the scoring of each essay by at least two raters according to the CEFR levels and to the creation of a balanced corpus. In addition, we describe how linguistic properties of the essays relate to the learners' proficiency in TCFLE-8. We also advance the state-of-the-art performance for the AES task in French by experimenting with two strong baselines (i.e. RoBERTa and feature-based). Finally, we discuss the challenges of AES using TCFLE-8.

TCFLE-8: a Corpus of Learner Written Productions for French as a Foreign Language and its Application to Automated Essay Scoring
Rodrigo Wilkens", Alice Pintard', David Alfteri, Vincent Folny', Thomas François"
"Cental, IL&C, UCLouvain, 'University of Gothenburg, FFance Education international
Abstract
Semantic Analysis (e.g.. Landauer et al. (1997)). NI.-extracted features combined with multiple re- gression (c.g.. Burstein et al. (1998)) or Bayesian text classification (e.g., Rudner and Liang (2002)). As noted by Dikli (2006), a small amount of essays (less than 1000) could be enough for training such systems in some contexts. However, even collect- ing such small corpus was difficult, the essays need to be manually rated, and essays reliable as- sessment a notoriously difficult task for humans (Wolfe et al., 2016).
Automated Essay Scoring (AES) aims to auto- matically assess the quality of essays. Automa- tion enables large-scale assessment, improve- ments consistency, reliahility, and standard- ization. Those characteristics of particular relevance in the context of language certifica- tion exams. However, major bottleneck the development of AES systems is the availabil- ity of corpora, which, unfortunately, are scarce, especially for languages other than English. In this paper, we aim to foster the development of AES for French by providing the TCFLE-8 cor- pus, corpus of 6.5k essays collected the con- text of the Test de Connaissance du Françuis (TCF French Knowledge Test) certification exam. We report the strict quality procedure that led to the scoring of each essay by at least two raters according the levels of the Com- mon European Framework of Reference for Languages (CEFR) and to the creation of a bal- anced corpus. In addition, we describe how linguistic properties of the essays relate to the learners' proficiency in TCFLE-8. We also ad- vance the state-of-the-art performance for the AES task in French by experimenting with two strong baselines (i.e., RoBERTa and feature- based). Finally, we discuss the challenges of AES using TCFLE-8.
Recent advances in AES have been made possi- ble by Deep Learning (DL) approaches and large language models (Ramesh and Sanampudi, 2022). Prominent studies used embeddings (Alikaniotis et al., 2016), recurrent neural network (Taghipour and Ng. 2016), attention (Dong et al., 2017 20.7). and BERT-based architcctures (Mayfield and Black, 2020). These approaches have also led to growing need for large corpora.
Consequently, AES teams have turned their at- tention to learner corpus research, a branch of cor- pus linguistics providing large-scale, computerized, naturalistic learner production. Pioncering works such as the International Corpus of Learner En- glish (ICLE) (Granger, 1993) and the European Science Foundation L.2 Database (Perdue, 1993) demonstrated the potential of such learner data col- lections for Second Language Acquisition (SLA) research, but it is only recently that more learner corpora fitted for AES, i.e., large enough and annotated with different proficiency levels, were developed for various languages (Yannakoudakis et al., 2011: Blanchard al., 2013; Geertzen al., 2014; Wisniewski et al., 2013; Mendes et al., 2016; Sakoda and Hosoi, 2018).
1 Introduction
Automated Essay Scoring (AES) aims to develop algorithms that can assess the quality of essays sim- ilarly to humans. The field may be traced back to the seminal work of Page (1966). Since then, several publications have been studying AES.? In the late 1990's, several functional AES systems were already available, either relying on Latent
Unfortunately, there is such large corpus for French, making the situation for French AES far from encouraging. The first systems thus relied on unsupervised approaches: Lemaire and Dessus (2001) used Latent Semantic Analysis to compare
'TCFLE-8 is available sammee--ducatiionnneeeaan ftr.aioooon.../uuuss
https://www.
'For comprehensive reviews see Ramesh and Sanampudi (2022); Lagakis and Demetriadis (2021): Klebanov and Mad- nani (2021): Uto (2021); Klebanov and Madnani (2020); Ke and Ng (2019); Shermis et al. (2013).

---

native language (LI) of student essays with text- book passages, whereas AUTO-EVAL (Zaghouani, 2002) automatically captured several L1 essay fea- tures, which are heuristically combined. More re- cently, Parslow (2015a) trained a Naive Bayes clas- sifier on a very small corpus of 200 essays writ- ten in foreign language (FL.. Finally, Ranković et al. (2020) were the first to fine-tune BERT for FL French AES on more data, but they did not release it and only a single LI is represented. Therefore, in order to support the development of AES solutions for French, the need for a large and reliable corpus of written French essays be- comes apparent. In this paper, we make two main contributions. First, we provide the community with the TCFLE-8 corpus², composed of 6,569 learner essays, with 8 different languages of ha- bitual use, scores, from least 2 raters, for the 6 levels of the Common European Framework of Reference for Languages (CEFR) (Council of Eu- rope, 2001), and automatically annotated with +5k features. These essays were collected in the context of the official French Knowledge Test (TCF) exam, one of the main certification exams for French. Sec- ond, we provide solid baselines for future research in AES using TCFLE-S. This is the largest AES modeling study that has been done for French. This paper is organized as follows. Section presents the main characteristics and uniqueness of corpora used for AES. We then present the corpus compilation process, presenting the official certifi- cation exam which our corpus is based on and the essay selection process (Section 3). Next, in Sec- tion 4, we present the TCFLE-8 corpus, discussing its size, metadata and annotation. An exploration of TCFLE-8 for AES systems is presented in Section 5. Finally, final remarks are presented in Section 6. TCFLE-8 is freely available for rescarch purposes.
2015; Granger et al., 2013) point out the preva- lence of English as target language for more than half of the corpora, the rest focusing on German (Siemen et al., 2006; Gut, 2012; Belz, 2004), Span- ish (Lozano, 2009; Cestero Mancera et al., 2002), French (Granger, 2003; Granfeldt et al., 2006), Ital- ian (Di Nuovo et al., 2022) and others (Atwell and Alfaifi, 2014; Wang et al., 2015; Martin et al., 2012). In terms of usual or native language of the learners (LI), the majority of learner corpora are mono-L1. Multi-Li corpora are however favored today because they allow to study the influence of various LIs on the target language and they offer a wider degree of generalization.
Among this large body of written learner cor- pora, will focus on two types relevant for this work: corpora used for AES and learner corpora targeting French foreign language.
2.1 Corpora for AES
Corpora built for AES can focus of specific di- mensions, such as the organizational skill of essay writing (e.g., ICLE (Granger, 1993)) or the persua- sive nature of the essay (e.g., Argument Annotated Essays (Stab and Gurevych, 2014)), but they usu- ally address the global level of learner production on proficiency scale. One of the most commonly used scales for foreign languages is the Common European Framework of Reference for Languages (CEFR) (Council of Europe, 2001), describing six levels of proficiency from A1 (beginner) to C2 (ad- vanced). As the mapping between each essay and its proficiency level is critical in AES, it is best use essays written in the context of official L2 cer- tifications, as they benefit from strict rating proce- dures, usually with at least two professional raters grading each production. We distinguish these cor- pora, which we call candidate corpora, from other learner corpora containing productions collected in language classes on web forums.
2 Related Work
Developing French corpus for AES means tak- ing part in the field of learner corpus research, which, since its emergence in the late 1980s, gave rise to more than 200 learner corpora around the world. Reviews on learner corpora (Gilquin,
2.1.1 Candidate corpora
In addition to having more reliable proficiency rat- ings, candidate corpora also contain more varied learner profiles in terms of L1, age and background. The largest candidate corpus is the Cambridge Learner Corpus (CLC) (Nicholls, 2003) with more than 50 million words from 200,000 written pro- ductions and 138 different Lls. It was compiled from English exams of Cambridge Assessment En- glish and two subparts of this corpus are available for research. First, the OpenCLC (Lexical Com-
Test de connaissance du français (French knowledge test); FLE stands for frunçuis langue éirungère (French as a foreign language) and 8 refers to the eight different languages habitual use included in the corpus.
"See the Learner Corpora around the World (https:/ https:///uclouvann...m wwe/ee//eeaarchaeee
tutes/ ilc/cecl /eee//eeeeerrccc cecerneer-corooroe porp around- the-worle d. html) for corpora that have been descrihed in scientific publications.

---

puting Limited, 2017) is composed of more than 10,000 texts from candidates of 7 different Lls. The second available subpart of the CLC, First Certificate of English (CLC-FCE), contains 1,238 texts aligned with the CEFR (Yannakoudakis et al., 2011; Vajjala and Rama, 2018). Another corpus targeting English was released by Educational Test- ing Service, the ETS corpus of non-native written English or TOFFL11 1 (Blanchard et al., 2013). Ini- tially compiled to perform LI detection tasks, this corpus was later used in AES to explore both tra- ditional machine learning (Rupp et al., 2019) and deep learning models (Nadeem et al., 2019). It contains 12,100 English essays written by TOEFL candidates of 11 non-English native languages. The essays are presented with their prompt and profi- ciency level given by ETS now-medium-hine Collaborations between certified testing organ- isations and research groups also developed for other European languages, resulting into three re- cent candidate corpora. The MERLIN corpus (Wis- niewski et al., 2013) contains 2,290 written produc- tions from standardized tests targeting German, Ital- ian (TELC institute) and Czech (UJOP Institute). Its design allowed for cross-lingual AES experi- ments (Vajjala, 2018; Arhiliuc et al., 2020; Bestgen, 2020; Caines and Buttery, 2020). The COPLE2 corpus (Mendes et al., 2016), containing 966 essays written in Portuguese (ICLP and CAPLE institutes), and the ASK corpus (Tenfjord et al., 2006b), with 1,936 texts written by candidates to the Norwegian Language Test, have also both been used for AES (del Río et al., 2016; Berggren et al., 2019; Carlsen, 2012). Table 6 in Appendix A provides an addi- tional detailed description of existing candidate corpora.
traceable to the CEFR scale, and the prompts are level-specific (Geertzen et al., 2013).
AES experiments were also conducted for Span- ish on CEDEL2, a learner corpus of more than million words from 4,399 learners of 11 different LIs (Lozano, 2009), for Swedish on the Swel.. cor- pus containing approximately 600 texts (Volodina et al., 2016) and for Japanese on the I-JAS corpus of texts written by 1000 learners of 12 different native languages (Sakoda and Hosoi, 2018). These exper- iments involve traditional machine learning work with features (del Río et al., 2016; Pilán and Volod- ina, 2018; Lee and Hasebe, 2020) or deep learning (Lilja, 201 8; Ruan, 2020; Hirao et al., 2020a).
2.2 Learner corpora targeting French
To the best of our knowledge, there are no can- didate corpora for French. Most learner corpora targeting French were compiled to study interlan- guage (Selinker, 1972). They were collected from language courses at university, so the levels repre- sented are mainly intermediate and advanced. The French Interlanguage Database (Granger, 2003) contains 450,000 words. Other corpora designed for interlanguage investigation include the Learner Corpus French (Yanderbauwhede, 2012), con- taining 500,000 words, and the Chy-FLE/Hellas- FLE (Valetopoulos and Zajac, 2012), containing 150,000 words. The Corpus Interlangue (Gaillat and Roa, 2020), written/spoken and bilingual corpus, contains texts and interviews from 115 students. The Corpus Ecrit de Français Langue Etrangère (Granfeldt et al., 2006) approaches learn- ers interlanguage in the language development se- quences. It is the only corpus representative of all proficiency levels for French, and it contains 100,000 words. It has been used for AES to find the features most correlated with CEFR levels (Parslow, 2015a). Finally, the French part of the Word Reference Corpus (Berdicevskis, 2020) con- stitutes the largest learner corpus for French with 4 million words from forum posts on the Word Ref- erence website. has been used to study contact- induced simplification, but despite its considerable size, it was not used for AES, because it is noisy and text levels have not been evaluated. More in- formation learner corpora targeting French is presented in Table in Appendix A.
2.1.2 Corpora from language classes
Some corpora compiled from learner productions in language classes also prove to be suitable for AES tasks. For example, EFCAMDAT (Educa- tion Cambbridde Open Language Database) (Geertzen et al., 2014) has been used in AES to investigate features related to the CEFR scale (Arnold et al., 2018), to classify based on errors (Ballier and Gaillat, 2016) or neural AES models (Kerz et al., 2021). To the best of our knowledge, this is the largest L2 corpus used in AES that does not come from certification exams. EFCAMDAT contains 83 million words from more than mil- lion essays written by learners of Education First's online English school. These essays span 16 levels
'Interlanguage describes the unique linguistic organisation developed by a foreign language learner, which presents some features of previously acquired language and may overgener- alize 1.2 patterns.

---

3 Corpus compilation
related criteria. Until now, language test providers used both analytical and holistic scales (Hamp- Lyons, 1995). There is no clear consensus on the superiority of one type of scale in terms of reliabil- ity and efficiency (Ono et al., 2019).
3.1 Data collection
TCFLE-8 being a candidate corpus for French, it has been collected by one of the agencies carrying out official certification in L2 French: France Edu- cation International (FET). FEI is French agency under the supervision of the Ministry of National Education and Youth. With a workforce of over 250 employees and a network of more than 1,000 experts, FEI acts in various fields of cooperation in education and training and contributes to the promotion of the French language and the French- speaking world. FEI offers wide range of certifi- cations in French aligned with the six CEFR levels: initial diploma in French language (DILF), diploma in French language studies (DELF), diploma in advanced French language studies (DALF) and French knowledge test (TCF). Around 650,000 can- didates take one of these examination on an annual base in more than 180 countries.
Language competence is multidimensional (Bachman, 1990; Bachman and Palmer, 2010; Oller and Hinofotis, 1980; Vollmer and Sang, 1983) and is measurable skill (Vollmer and Caroll, 1983). Measuring writing skill implies consider- ing various facets: candidate proficiency, rater le- niency/harshness and difficulty of the task. To this aim, "Many-facet Rasch measurement (MFRM) is psychometric approach that establishes coher- ent framework for drawing reliable, valid, and fair inferences from rater-mediated assessments, thus answering the problem of fallible human ratings" (Eckes, 2009). Therefore, we applied MFRM the FEI dataset of TCF exams in order to identify and avoid the fallible human ratings in the data set.
As its name implies, TCFLE-8 is based on the TCF, linear test aligned with the six CEFR lev- els. The TCF is used mainly for academic studies, migration purposes and citizenship. Its written component, made up of three independent lasks, is taken annually by 120,000 candidates, 60% of which sit their exam on computer.
3.2 Data cleaning
The original data collected by FEI had to be cleaned at various levels. First, outlier identification con- sisted in removing candidates' responses that did not achieve the A1 level, were copies of the prompt, too short/long, or off-topic. Next, we leverage the Rasch information in the dataset to detect texts for which human raters might have failed to provide a reliable judgment. We compared FEI raters' orig- inal scores and the scores adjusted by the Rasch method, using standardized residuals. After an empirical evaluation, we removed all essays with standardized residual value greater than 4.7 In addition, we also dropped essays with low confi- dence assessment (c.g., candidates that are on the borderline between levels). To accomplish this, we removed all cases where both raters disagree with each other and with the candidate's final score, and we also removed the cases where there is a distance of three CEFR levels between the lowest and the highest ratings. After this process, we set the essay score as the candidate's CEFR level when at least one of the raters assigned that level to the essay, Alternatively, if both raters agreed with the essay's 's score, we duly assigned this level to the essay. Any essay that does not fit any of these two criteria has been removed.
The correction is performed by professional raters. FEI has pool of about 100 raters, recruited occupational profiles (experienced teachers, pre- vious experience for rating with French). Appli- cant raters take ing competence test for writing and attend a two- day training. At the end of this procedure, the recruitment is confirmed or not. To ensure reliabil- ity in the long term, reliability indices of raters are assessed annually, and decision is made regarding whether to retain them in the pool. In addition, to ensure reliability at the candidate level, FEI adopts double rating approach. In case of discrepancy, third rater is called to independently rate the productions. The final level of the candidate is established based on the frequency of the CEFR levels given to the three candidate's productions.
rat-
To identify the CEFR level, the raters use adapted CEFR descriptors and scales. The descrip- tors (and the rating) are holistic, although each descriptor is aiming at linguistic (organisational), pragmatic and sociolinguistic dimensions and their
'In our empirical evaluation, we explored four standard- ized residue values (2, 3 and 4). observing that around 5.6% of the corpus has a standardized residue of 2, 0.9% has a value of 3 and 0.4% a value of 4.
"I has to be mentioned that the rating of the set of the 3 tasks is done by the same rater, thus not being independent.

---

After outlier removal, the next step was to get a representative sample from the of TCF essays available. For a fair representation, the level of the text is an obvious variable to control. In addition, we controlled for the language of habitual use", aiming for a representation of the most frequent languages. As the top five were all European ones and the 6th was Kabyle, Afro-Asiatic language, we also included Chinese and Japanese to get better representation of various typological fami- lies of languages. Thus, we launched a random sampling controlling for the 6 CEFR levels and the candidate's language of habitual use. To ap- ply this algorithm, we set up an objective function that approximates the CEFR scores distribution by language. In order to reflect the distribution in the whole dataset, we divided the current language into frequency bands: very frequent (English and Ara- bic), frequent (Spanish, Kabyle, Portuguese and Russian) and infrequent (Chinese and Japanese) languages. For a description of the resulting corpus see Section 4.
2020) for (pseuddo-anonyymizaionn. With this tool, we larget entities: names, address (i.e., coun- try, city, building, territory and place), date (i.e., day of week, month, year and day), e-mail, or- ganization, amount and phone. After the pseudo- anonymization, we assessed its quality. 10 During this process. we noticed some consistent flaws in the tool that were corrected in the corpus to im- prove its quality. The observed issues consisted of an overanonymization of words at sentence begin- nings when predicated by name, and an omission to replace email addresses.
The TCFLE-8 corpus
At the end of the compilation process, the final TCFLE-8 corpus comprises 6,569 essays (581,333 words). Some figures about the corpus size by CEFR levels are shown in Tables 1, 2, 3 and 4. It is expected that beginner-level essays tend be shorter than the other ones (Frase et al., 1998). Moreover, the extreme levels (A1 and C2) are less represented the corpus. This might be caused by two factors: (1) few A1-level learners seek an official language exam since this level is rarely sufficient for official purposes (e.g., employment and visa requirements), and (2) reaching the C2 level in foreign language is extremely difficult.
3.3 (Pseudo- sreeddo-janonymizatto
Candidates sometimes include personal informa- tion in their essays. While this does not pose prob- lem for the assessment, it can expose candidates when the texts become public. This exposure gen- erally tackled with anonymization methods (e.g..
Wisniewski et al. (2013); Mendes et al. (2016);
#essays
% essays
avg #wrd (stdev)
A1
689
10.49
69.78 (34.02)
Tenfjord et al. (2006a); Gablasova et al. (2019); Rakhilina et al. (2016)) or sseudo-anonymizztion (e.g.. (Glaznieks et al.. 2020; Preradovic et al.. 2015; Rosen et al., 2020; Dirdal et al., 2022)) in the literature on learner corpora. Typically, these processes capture names (e.g., Gablasova al. (2019); Preradovic al. (2015); Rosen al. (2020); Rakhilina et al. (2016)), but sometimes they also capture other information, such as loca- tion and date (e.g., Glaznieks al. (2020); Tenfjord et al. (2006a); Wisniewski et al. (2013) ). geo-data (e.g., Volodina et al. (2019)) and language-specific substitutions (e.g., Wisniewski et al. (2013)). In our work, we decided to provide anonymous and pseudo-anonymous versions of TCFLE-8. The lat- ter is intended provide more natural text, but pseudo-anonymizztion may introduce grammatical errors (c.g., wrong contractions).
A2
1375
20.93
91.69 (44.80)
B1
1466
22.32
119.11 (49.89)
B2
1427
21.72
133.61 (44.92)
CI
1127
17.16
133.92 (45.91)
C2
485
7.38
138.92 45)
Total
6569
100.0
119.67 (50.31)
Table I: Description of the TCFLE-8 corpus by CEFR level: number of essays, percentage, and mean and standard deviation of word number per essay.
Table indicates the number of essays distin- guishing the gender. It is interesting to note that about 58% of the sample is composed of women and this proportion is not the same at at each level. Table 3 shows the essays by three tasks in the TCF exam, where there is general difference between one task and the others. Finally, Table 4 picture the amount of essays in the different languages of habitual use.
This work uses the MAPA tool (Gianola et al.,
'The language of habitual use is the language the candidate indicates as the one they usually use. "https: tla..aa..a..eeeee//ee/eeeeeeeebt wtw.ttla...oooom/aa cOm/MAPA- Cttp///PA-EU-PPAA EU-Project,
The evaluation scores are presented in Section B.

---

(CEFR) Level
Men
Women
Lang.
A1
A2
B1
B2
C1
C2
A1
394
295
JPN
8
135
171
170
48
2
A2
574
801
CHI
34
165
244
189
45
4
B1
596
870
SPA
124
187
175
182
178
58
B2
537
890
ARA
135
160
163
153
160
135
C1
441
686
POR
102
187
182
191
172
38
C2
198
287
ENG
125
163
167
165
169
128
Total
2740
3829
RUS
103
198
183
196
180
29
KAB
58
180
181
181
175
91
Table 2: Number of essays according to the gender, per
(CEFR) level
Table 4: Number of essays according to the language of habitual use (language code follows ISO639-2)
(CEFR) Level
Task
Task
Task
A1
225
223
241
The Quadratic Weighted Kappa (QWK) between the candidate's level and the CEFR-level of the es- say is 0.98. It is expected that this value should be high, but not equal to 1, due to the cases where candidates cannot maintain consistent level of es- say quality. In addition, the scores assigned by the FEI raters in the double rating procedure also available. They have QWK of 0.71 (correlation of 0.93) with each other and 0.84 (correlation of 0.85) with the CEFR-level of the essay.
A2
426
413
536
B1
447
475
544
B2
485
485
457
CI
358
431
338
C2
156
173
156
Total
2097
2200
2272
Table Number of essays according to task number
Comparing TCFLE-8 to existing corpora (see Ta- ble 6 in Appendix A), is the largest French learner corpus suitable for AES both in size and LI rep- resentation -, the third largest candidate corpus to our knowledge and its annotation layers provide the richest information (see Section 4.2). It also covers all CEFR levels.
Finally, the prompt and its position in the se- quence of three TCF tasks are also reported. This information, which relates the exam, contextu- alizes the essay's 's input and the grading sequence (described in Section 3.1). Regarding the prompt position, it is balanced TFCFL-8 (32% essays for the first task, 33% for the second, and 35% for the third). Table shows the distribution of the three tasks across the six CEFR levels.!
4.1 Metadata
As a complement to the text of the essays and their CEFR scores, assigned according the procedure described in Section 3.2, the TCFLE-8 corpus pro- vides information about the candidate who wrote the essay and the essay prompt.
4.2 Essay annotation
In addition to the above metada, TCFLE-8 also includes linguistic annotation layer aimed to de- scribe the Iearners' proficiency. This annotation was automatically performed using the FABRA toolkit (Wilkens et al., 2022). It allows comput- ing the distribution of over 400 linguistic variables grouped by family of related variables (e.g., lcxical diversity, and lexical frequency). These distribu- tions are aggregated using 18 statistical descriptors, which results more than 5k annotations per es- say. In addition, we extended the existing FABRA features by including others related to SLA.. In par-
Regarding the candidates' information, their gen- der and language of habitual use are provided. The candidates communicate this information when they register for the TCF exam. Overall, the corpus contains slightly more women than men (58% vs 41%), see Table 2. As for the language habitual use, the corpus covers languages, as described in Section 3.2: English, Arabic, Spanish, Russian, Portuguese, Kabyle, Chinese, and Japanese, respec- tively. with 917, 906, 904, 889, 872, 866, 681, and 534 essays (see Table 4.
Il "ee calculated the correlation Spearman for continous variables or Point-biscrial for binary variables - between the CEFR score and the metadata described above to identify possible biases in the scores; all correlations were between 0.038 and 0.09. This analysis confirms that the sampling process did not induce unexpected biases.
The CEFR level achieved by each candidate con- sidering the three written productions also re- ported. This is the official CEFR level assigned to the candidate for the written part of the TCF exam.

---

0.6
Swedish (Pilán and Volodina, 2018), for Japanese (Hirao al., 2020b) and for French Parslow (2015b). As regards lexicon, diversity measures (e.g. type/token ration) correlate with proficiency in English (Lu, 2012; Vajjala, 2018), whereas so- phistication measures based on word frequencies have yielded similar results in number of studies: more proficient writers use, average, fewer fre- quent words (Laufer and Nation, 1995; Attali and Burstein, 2006; Crossley and McNamara, 2012; Guo et al., 2013). Finally, the most discriminat- ing feature in TCFLE-8, namely the error-rate, is also one of the most correlated features to profi- ciency levels in the CLC-FCE and TOEFLI] (Yan- nakoudakis et al., 2011; Vajjala, 2018). In addi- tion, while this analysis confirms existing research findings in AES, it also points out that TCFLE-8 may be helpful for new SLA studies. Indeed, we also provided several features explored in other acquisition-relater fields, such as the OLD20 (mea- suring orthographic similarity) (Coltheart et al., 1977; Yarkoni et al., 2008), which are significant in our corpus but had not been linked to L2 writing proficiency so far, to the best of our knowledge.
0.4
0.2
0.0
0.2
0.4
-0.6
-0.8
Errors
Gradled lexicons.
diversity.
Lex. frequency
Ortho, neightors
Morphology
Likellhood
Word length
sophisticate
Figure 1: Box-plot of feature correlations by family.
ticular, we included the error annotation provided by Language tool which includes, among others, the identification of agreement, casing, grammar, typography, punctuation, and typos. We also in- cluded pedagogical annotation based on the work of Pintard and François (2020) for extending the CEFR level-related vocabulary. should be noted that, as the ascudo-anonimzztion process may al- ter text properties, we have chosen to perform this feature extraction on the original essays.
AES for French
In order to better characterize how linguistic properties present in TCFLE-8 are associated with the learners' proficiency, we computed Spearman's correlations between each of above feature (gath- ered by the families in Wilkens et al. (2022)) and the essays' CEFR level. In the process, we dropped features with correlations lower than 0.4 or values higher than 0.05. Next, as many features are variants of each other, we calculated the correlation matrix within each family identify redundant features (i.e., an absolute correlation above 0.90). Finally, for each set of similar features, we consid- ered only the one most correlated with the CEFR level. After this procedure, we kept 119 correlated features. In Figure 1, we show their distribution by family of variable.!? 13
In this section, we analyze the applicability of the TCFLE-8 corpus for training AES systems. For this purpose, we explore two approaches: deep learning, since most AES systems relied on neural networks (Ramesh and Sanampudi, 2022), and feature- based machine leaning.
We split the anonymized corpus with 80% for training, 10% for validation, and 10% for testing. stratifying by score and language. In addition, to explore the impact of model initialization, we per- formed repetitions of the training process; each one, we adjusted the test set so that it does not overlap with the others. We performed a hy- perparameter exploration using the accuracy the validation set.
For the deep learning model, we used Camem- BERT (Martin et al., 2020), a RoBERTa-based model for French. As for the hyperparameters we use a learning rate of 5e-5 and an early stop of 5. For machine learning, we use the XGBoost 15 and
Our analysis of the selected features highlighted linguistic properties of essays already reported in studies investigating foreign language writing. For example, measures of word length have been known to be good predictors of proficiency level for English (Ferris, 1994; Grant and Ginther, 2000), for
''The hyperparameters search explored Ie-4, 5e-5, le-5, 5e-6 and le-6 as learning rate, 1, 3, and 10 as early stop patience, searching up to 40 epuchs.
12 Ptttps: htpppppppppppoo/ https///pppppoo/poopppppee org/project// language-Lood tool-python/
"The hyperparameters used for XGBoost and the values explored are ghtree as booster, alternatively exploring ghtree, gblincar and dart, 0.3 as subsample, from 0.3 and 0.6, 3 as
"Table 9 in the appendices presents the list of all correlated features and their correlation values.

---

logistic regression as feature-based baseline. These were trained using the 119 features extracted using the method described Section 4.2. The evaluation of these models is shown in Table 5.
three models, XGBoost suffers from catastrophic failure, achieving even lower performance than the Logistic model. This general weak performance is not entirely surprising. as texts at this level tend to explore language idiosyncrasies, to be precise, to have very coherent and organized structure, etc. In contrast, the beginner levels (i.e., Al and A2), for which transformers and XGBoost models had close results, is characterized by texts with simple vocabulary and grammatical structures." 19
In order characterize human level performance on the task, we report standard AES evaluation metries for the human raters (column "raters" in Table 5), namely accuracy, adjacent accuracy, F1- score, and QWK." !7 Those metrics were calculated by a direct comparison between the ratings of one of the two evaluators and the reference CEFR levels for each essay. Those results show that the task of identifying the CEFR level of an essay is hard, even for humans. However, the adjacent accuracy of 0.99 clearly shows that the identification gap is typically up to one level. In the same direction, the QWK points out that once the ordinality existing between CEFR levels is considered, the agreement among raters is remarkably strong. As expected, none of our models achieved results competitive with human performance." 18
As TCFLE-8 is a new corpus for the French lan- guage, we cannot fairly compare our results with previous works, due to the considerable difference in corpus size. In the French AES literature, we identified only two papers focusing on I.2 profi- ciency identification. First, Parslow (2015a), who used a corpus of 200 essays to train Naive Bayes classifier and reported FI-scores ranging from 0.51 to 0.74 for the levels AI to B2. Second, Ranković et al. (2020) used CamemBERT intermediate layers as features to predict level in a corpus of 100 essays and reported MSF ranging from 0.35 to 0.55.
Among the AES models explored, the bransformer-based CamemBERT achieved the best values. Despite this performance, it can be seen that there is still room for improvement when comparing the results with the evaluation by experts (column raters). Considering that the transformers model performs in a range between raters and XGBoost, it is interesting to remark that the transformers model is closer to the raters' performance when we consider the ordinality relation between levels (i.e., QWK and AccuracyAdjacena Adjacent). Focusing on the ability of mod- els to discriminate specific levels, the fine-tuned version of CaR emerged as model of better performance. Morcover, the logistic model is clearly a weak baseline. Interestingly, at the C2 level, which was the most challenging for all
6 Final Remarks
In this work, we presented TCFLE-8, corpus of 6,569 candidates' essays written during the French knowledge test (TCF), with different languages of habitual use. This paper described the data gathering by France Education International (FEI), data cleaning, anonymization, and annotation per- formed to compile this corpus, which is the largest French corpus targeting French as foreign lan- guage for AES. This corpus, along with its meta- data (i.e., essays, metadata and annotation) is avail- able to the community. We also described the learn- ers' proficiency the corpus using numerous lin- guistic variables related to SLA. This description confirms that these linguistic features could capture developmental patterns in TCFLE-8 in a similar fashion to other learner corpora.
max depth, from 3, and 9, 0 as max delta step, exploring O, 5 and 10, as min child weight, from 1, and 10, 0.1 as eta, exploring 0.01, 0, 0.3, 0.5 and 0.7, as gamma, from 0, 1, 10, lossguide as grow policy, exploring lossguide and depthwise, multi softmax as objective function and 50 estimatons.
Exploring TCFLE-8 for ABS, we applied differ- ent machine learning algorithms. CamemBERT ap- pears to be more accurate and XGBoost, feature- based model, achieved similar results at beginner level. This raises a question about what features should explored for better describing the interme- diate and advanced levels. Interestingly, part of this answer may come from the transformer model
"For the logistic regression, we explored the following hyperparameters: penalty from 12 none, C. from I, 10, 100. max interaction from 100 or 300, and multi class process from multinomial or one-vs-rest. After this short exploration. set the solver 1bígs, 12 as the penalty, the C and the max interaction 100 and 300, and the class processing one-vs-rest.
"FFor a transparent presentation of our results, Appendix shows the confusion matrices of the transformer-based model. 18 "Note that the scores in the "raters" column are inflated, because the rating assigned hy each rater contributes to the final CEFR score of each essay.
19 We compared the results of training the models on the anonymized corpus with the corresponding models trained on the original corpus (hefore anonymization) and no statistical difference was identified.

---

CamemBERT
XGBoost
Logistic
Raters
QWK
0.88 (0.01)
0.79 (0.02)
0.69 (0.02)
0.93 (0.01)
Accuracy
0.57 (0.01)
0.46 (0.01)
0.37 (0.01)
0.76 (0.01)
Accuracy Adjacemt
0.98 (0.01)
0.92 (0.02)
0.80 (0.01)
0.99 (0.01)
Fl weighted
0.56 (0.01)
0.46 (0.02)
0.36 (0.02)
0.76 (0.01)
AI
0.63 (0.01)
0.59 (0.04)
0.54 (0.06)
0.76 (0.02)
A2Fl
0.57 (0.04)
0.53 (0.01)
0.40 (0.05)
0.76 (0.03)
B B1
0.56 (0.04)
0.45 (0.05)
0.32 (0.02)
0.75 (0.01)
B2F1
0.56 (0.04)
0.43 (0.03)
0.34 (0.03)
0.76 (0.02)
ClFl
0.56 (0.04)
0.42 (0.03)
0.30 (0.05)
0.77 (0.02)
C2F1
0.48 (0.09)
0.19 (0.07)
0.31 (0.02)
0.80 (0.04)
Table 5: Average and standard deviation of the evaluation score using of using the TFCFL-8 for AES and the performance human raters
itself (c.g. through probing approach (Tenney al., 2019)).
is corpus designed for supporting the research in French foreign language, including AES, In this work, the focus is on the corpus compilation. Despite the initial tests performed here, our goal does not include an exhaustive verification the corpus' applications nor an evaluation of various AES approaches. Finally, we do not intent nor recommend using TCFLE-8 for fully-automated evaluation environment but to improve writing as- sessment in French foreign language.
Finally, TCFLE-8 was portrayed in this paper as a corpus for French AES but its properties allow for different applications NLP, SLA and educational studies. It may be a valuable corpus for pedagog- ical material development, whether they be dic- tionaries (Longman, 2002), activities focusing on common learner difficulties and errors (Kaszubski, 1998; Reppen, 2010), computer-asse cosputer-sssisted language learning software (Granger, 2003) L2 writing aids (Link et al., 2014). Activities of data-driven learning language class (Friginal, 2018) could also take advantage of this corpus. With different languages of habitual use, this corpus could also be beneficial for cross- linguusti studies such as trans- fer mechanisms and L1 influence L2 production (Golden et al., 2017; Werner et al., 2020), and for automatic native language identification (Tetreault al., 2013). Another possible application for this new corpus the one of errors detection and correc- tion (Dahlmeier et al., 2013), that we are currently investigating as future work on TCFLE-8.
A Acknowledgements
This research has been funded by the Fonds de Recherche Scientifique de Belgique (F.R.S.-FNRS) under the grant MIS/PGY F.4518211 and by a re- search convention with France Éducation Interna- tional. Computational resources have been pro- vided by the Consortium des Equipements de Cal- cul Intensif (CÉCI), funded by the Fonds de la Recherche Scientifique de Belgique (F.R.S.-FRS) under Grant No. 2.5020. and by the Walloon Region
7 Limitations
References
Several normalization steps were applied in order to develop a coherent corpus for AES, aiming to compile a high quality corpus illustrating the pro- ficiency levels with learner productions on which professional raters would agree. As a consequence, some potentially interesting cases were removed. This concerns for example texts on the borderline between two levels. Although they interesting cases as they could support studies on understand- ing of the level boundarics, we opted for a corpus that represents the texts of each level. TCFLE-8
Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural net- works. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- 1: Long Papers), pages 715-725.
Cristina Arhiliue, Jelena Mitrović, and Michael Gran- itzer. 2020. Language Proficiency Scoring. In Pro- ceedings the /2th Language Resources and Evalua- tion Conference, pages 5624 5630, Marseille, France. European Language Resources Association.
Taylor Arnold, Nicolas Ballier, Thomas Gaillat, and Paula Lisson. 2018. Predicting CEFRL levels in

---

learner English on the basis of metries and full texts, arXiv:1806.111 1109 [cs]. ArXiv: 1809.11099.
Cecilie Carlsen. 2012. Proficiency Level-a Fuzzy Vari- able in Computer Learner Corpora. Applied Linguis- rics, 33(2):161-18;
Yigal Attali and Jill C. Burstein. 2006. Automated essay scoring with e-rater@ v. 2. The Journal of Technology, Learning and Assessment, 4(3).
Ana María Cestero Mancera, Inmaculada Pe- nadés Martínez, Ana Blanco Canales, Laura Ca- margo Fernnndez, and José Francisco Simón Granda. 2002. Corpus para el análisis de errores de aprendices de E/LE (CORANE). In Aclas del XII Congreso Internacional de ASELE: tecnologias de la información y de las comunicaciones en la enseñanza de la E/LE, 2002, ISBN 14-9705--12-6 págs. 527-534, pages 527-534. edUPV, Editorial Universitat Politècnica de València.
Eric Atwell and A. Alfaifi. 2014, Arabic learner corpus (ALC) v2: a new written and spoken corpus of Arabic learners. In Learner Corpus Studies in Asia and the World (LCSAW).
Lyle Bachman and Adrian Palmer. 2010. Language assessment in practice: developing language assess- ments and justifying their use in the real world. Ox- ford applied linguistics. Oxford Univ. Press, Oxford.
Max Coltheart, Eileen Davelaar, Jon Torfi Jonasson, and Derek Besner. 1977. Access to the Internal Lexicon. In Artention and Performance VI. Routledge. Num Pages: 21.
00000.
Lyle F Bachman. 1990. Fundamental considerations in language testing. Oxford University Press. 00000.
Council of Europe. 2001. Common European Frame- work of Reference for Lan- guages: Learning. Teach- ing, Assexsment. Press Syndicate the University of Cambridge.
Nicolas Ballier and Thomas Gaillat. 2016. Classifica- tion d' apprenants francophones de I' l'anglais sur la base des metriques complexite lexicale et syntax- ique. In TEP-TALN-RECTIE 2016, volume 9 of ELTAL, pages 1-14, Paris, France.
Scott A. Crossley and Danielle S. McNamara. 2012. Predicting second language writing proficiency: the roles of cohesion and linguistic sophistication. Jour- nal of Research in Reading, 35-5):15--33.
Julie A. Belz. 2004. Learner corpus analysis and the de- velopment of foreign language proficiency. System, 32(4):577-591.
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mci Wu. 2013. Building Large Annotated Corpus of Learner Ennliih: The NUS Corpus of Learner linglish. In Proceedings of the Eighth Workshop Innovative Use of NLP for Building Educational Applications, pages 22-31, Atlanta, Georgia. Association for Com- putational Linguistics.
Aleksandrs Berdicevskis. 2020. Foreigner-directed speech is simpler than native-directed: Evidence from social media. In Proceedings of the Fourth Workshop on Natural Language Processing and Com- putational Social Science, pages 163-172, Online. Association for Computational Linguistics.
Iria del Río, Sandra Antunes, Amália Mendes, and Maarten Janssen. 2016. Towards error annotation in learner corpus of Portuguese. In 5th NLP4CALL and 1st NLP4LA workshop in Sixth Swedish Lan- guage Technology Conference (SLTC), volume 130, pages 8-17. Linköping University Electronic Press.
Stig Johan Berggren, Taraka Rama, and Lilja Øvrelid. 2019. Regression or classification? Automated Es- say Scoring for Norwegian. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 92-102.
Elisa Di Nuovo, Manuela Sanguinetti, Alessandro Mazzei, Elisa Corino, and Cristina Bosco. 2022. VALICO-UD: Treebanking an Italian Learner Corpus in Universal Dependencies. ICCoL. Italian Journal of Commutational Linguistics, 8(1).
Yves Bestgen. 2020. Reproducing Monolingual, Mul- tilingual and Cross Cross-Lingual CEFR Predictions. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5595-5602, Marseille, France. European Language Resources Association.
Semire Dikli. 2006. An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment, 5(1).
Daniel Blanchard, Joel Tetreault, Derrick Hig- gins, Avife Cahill, and Martin Chodorow, 2013. TOEFLII: A Corpus of Non-Native English. ETS Research Report Series, 151
H. Dirdal, I. K. Hasund, E.-M. Drange, E. T. Vold, and E. M. Berg. 2022. Design and construction of the tracking written learner language (trawl) corpus: A longitudinal and multilingual young learner corpus. Nordic Journal of Language Teaching and Learning, 10(2):115-133
Jill Burstein, Karen Kukich, Susanne Wolff, Chi Lu, and Martin Chodorow. 1998. Computer analysis of essays. In NCME Symposium on automated Scoring.
Andrew Caines and Paula Buttery. 2020. REPROLANG 2020: Automatic Proficiency Scoring Czech, En glish, German, Italian, and Spanish Learner Essays. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5614-5623, Marseille, France. European Language Resources Association.
Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention- based recurrent convolutional neural network for au- tomatic essay scoring. In Proceedings of the 21 conference on computational natural language learn- ing (CoNLL 2017), pages 153-162.

---

T. Eckes. 2009. Quantitative Data Analysis for Lan- guage Assessment Volume 1: Fundamental Tech- niques, edition. Routledge.
Anne Golden, Scott Jarvis, and Kari Tenfjord. 2017. Crosslinguistic Influence and Distinctive Patterns of Language Learning: Findings and Insights from a Learner Corpus. Multilingual Matters.
Dana R. Ferris. 1994. Lexical and syntactic features of esl writing by students at different levels of 12 proficiency, TESOL Quarterly, 28(2)414-420.
Jonas Granfeldt, Pierre Nugues, Emil Persson, Jonas Thulin, Malin Agren, and Suzanne Schlyter. 2006. CEFLE and Direkt Profil: A new computer learner corpus in French L.2 and a system for grammatical profiling. In Proceedings of the Fifth International Conference on Language Resources and Evaluation. pages 365-570. ELRA.
Lawrence T. Frase, Joseph Faletti, April Ginther, and Leslie Grant. 1998. Computer Analysis of the Toefl Test of Written English. ETS Research Report Series, 1998(2);-26.
Sylviane Granger. 1993. The International Corpus of Learner English. In The European English Messen- ger, page 34.
Eric Friginal. 2018. Corpus Linguistics for English Teachers: Tools, Online Resources, and Classroom Activities, 1st edition edition. Routledge, New York, NY.
Sylviane Granger. 2003. Error-tagged Learner Corpora and CALL; A Promising Synergy. CALICO Journal, (033):4654480
Gablasova, Brezina, and T. McEnery. 2019. The trinity lancaster corpus: Development, description and application. International Journal of Learner Corpus Research, 122):126-158.
Sylviane Granger, Gaëtanelle Gilquin, and Fanny Me- unier. 2013. Twenty Years of Learner Corpus Re- search: Looking back, Moving ahead.
Thomas. Gaillat and C. Roa. 2020. The corpus in- terlangue project: Storing language learner data in a huma-num nakala database for automatic online retrieval. The CLARIN Bazaar 2020. Book of Ab- stracfs. CLARIN2020, Virtual Event.
Leslie Grant and April Ginther. 2000. Using computer- tagged linguistic features to describe 12 writing dif- ferences. Journal of Second Language Writing, 9(2):123-145.
Liang Guo, Scott A. Crossley, and Danielle S. McNa- mara. 2013. Predicting human judgments of essay quality both integrated and independent second language writing samples: A comparison study. As- sessing Writing, 18(3):218-22 238.
Jeroen Geertzen, Theodora Alexopoulou, and Anna Ko- rhonen. 2014. Automatic Linguistic Annotation of Large Scale L2 Databases: The EF- CaFCbibrr Open Language Database (EFCamDat). In Selected Pro- ceedings of the 2012 Second Language Research Fo- rum: Building Bridges berween Disciplines, pages 240-254, Somerville, MA, USA. Cascadilla Proceed- ings Project.
Ulrike Gut. 2012. The LeaP corpus : A multilingual corpus of spoken learner German and learner En glish. Multilingual corpora and multilingual corpus analysis, 14:3-23.
Jeroen Geertzen, Theodora Alexopoulou, Anna Korho- nen, al. 2013. Automatic linguistic annotation of large scale 12 databases: The ef-cambridge open lan- guage database (efcamdat). In Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project, pages 240-254. Citeseer.
Liz Hamp-Lyons. 1995. Rating Nonnative Writing: The Trouble with Holistic Scoring. TESOL Quarterly. 29(4):759.
Reo Hirao, Mio Arai, Hiroki Shimanaka, Satoru Kat- sumata, and Mamoru Komachi. 2020a. Automated Essay Scoring System for Nonnative Japanese Leamn ers. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1250-1257, Mar- seille, France. European Language Resources Asso- ciation.
Lucie Gianola, Eriks Ajausks. Victoria Arranz, Ona de Gibert, and Maite Melero. 2020. Automatic removal of identifying information in official eu languages for public administrations: The mapa project. In 33rd International Conference on Legal Knowledge and Information Syxtems (JURIX 2020): proceedings, Dec 2020, Brno, Prague, Czech Republic, volume 334, pages 223 226. IOS Press.
Reo Hirao, Mio Arai, Hiroki Shimanaka, Satoru Kat- sumata, and Mamoru Komachi. 2020b. Automated essay scoring system for nonnative japanese learners. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1250-1257.
Gaëtanelle Gilquin. 2015. From design to collection of learner corpora. The Cambridge Handbook of Learner Corpus Research, pages 9-34.
Przemyslaw Kaszubski. 1998. Learner corpora: The cross-roads of linguistic norm. TALC98 Proceediggs pages 24-27.
Aivars Glaznieks, Jennifer-Carmen Frey, Maria Stopfner, Lorenzo Zanasi, and Lionel Nicolas. 2020. LLEONIDE-Longiudee Learner Corpus in Italiano, Deutsch and English 1.0.
Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey the state the art. In IJCAI. volume 19, pages 6300-6308.

---

Elma Kerz, Daniel Wiechmann, Yu Qiao, Emma Tseng, and Marcus Ströbel. 2021. Automated Classification of Written Proficiency Levels the CEFR-Scale through Complexity Contours and RNNs. In Pro- ceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 199-209.
Cristóbal Lozano. 2009. CEDEL2: Corpus escrito del español L2. In Carmen M. Bretones Callejas, José Francisco Fernández Sánchez, José Ramón lháñez lbáñez, María Elena García Sánchez, M Enri- queta Cortés de los Ríos, Sagrario Salaberri Ramiro, M' Soledad Cruz Martínez, Nobel Perdá Honeyman, and Blasina Cantizano Márquez, editors, Applied Lin- guisties Now: Understanding Language and Mind La Lingüásstic Aplicada Hoy: Comprendiendo el Lenguaje y la Mente, pages 197-212. Universidad de Almería, Almería, Spain.
Beata Beigman Klebanov and Nitin Madnani. 2020. Automated evaluation of writing-50 years and count- ing. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7796-7810.
Xiaofei Lu. 2012. The Relationship of Lexical Rich- ness the Quality of ESL Learners' Oral Narra- tives. The Modern Language Journal, 96(2): 208. https:/ / /nl ine ibrary. www.w.com/dwe....ooo cwm/ /oi/ 2011 1.01 21232_1.X
Beata Beigman Klebanov and Nitin Madnani. 2021. Automated Essay Scoring. Synthesis Lectures on Human Language Technologies, 1435).1-314.
Paraskevas Lagakis and Stavros Demetriadis. 2021. Au- tomated essay scoring: A review of the field. In 2027 International Conference on Computer, Information and Telecommunication Systems (CITS), pages 1-4 66 IEEE.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Ville- monte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. Camembert: a tasty french language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Thomas K Landauer. Darrell Laham, Bob Rehder, and Missy E Schreiner. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans. In Proceedings of the 19th annual meeting of the Cognitive Science Society, pages 412-417.
Maisa Martin, Riikka Alanen, Ari Huhta, Paula Kalaja, Katja Mäntylä, Mirja Tarnanen, and Äsa Palviainen. 2012. CEFLING: Combining Second Language Ac- quisition and Testing Approaches to Writing. Studies in Writing, 25.
Elijah Mayfield and Alan W Black. 2020. Should you fine-tune bert for automated essay scoring? In Pro- ceedings of the Fifleenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 151-162.
Batia Laufer and Paul Nation. 1995, Vocabulary Size and Use: Lexical Richness in L2 Written Production. Applied Linguistics, 1673):307-322.
Jae-Ho Lee and Yoichiro Hasebe. 2020. Quantitative Analysis of JFL. I Learners' Writing Abilities and the Development of a Computational System to Estimate Writing Proficiency. Learner Corpus Studies in Asia and the World, 5: 1200
Amália Mendes, Sandra Antunes, Maarten Janssen, and Anabela Gonçalves. 2016. The COPLE2 corpus: learner corpus for Portuguese. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC" 16), pages 3207- 3214, Portorož, Slovenia. European Language Re- sources Association (ELRA).
Benoit Lemaire and Philippe Dessus. 2001. A Sys- tem to Assess the Semantic Content of Student Es- says. Journal of Educational Computing Research, 24(3):305-33 320.
Farah Nadeem, Huy Nguyen, Yang Liu, and Mari Ostendorf. 2019. Automated Essay Scoring with Discourse Aware Neural Models. In Proceedings of the Fourleenth Workshop on Innovative Use of NL. for Building Educational Applications, pages 484- 493, Florence, Italy. Association for Computational Linguistics.
Lexical Computing Limited. 2017.
Open- CLC (vl). https://www. www.skketttceeeeeee eu/ ttttorrpsstttgllleeeee htgp:/ttggggiiieee id-1.
Distributed by Lexical Computing Limited on behalf of Cambridge University Press and Cambridge English Language Assessment.
Diane Nicholls. 2003. The Cambridge Learner Cor- pus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 conference, volume 16, pages 572-581.
Mathias Lilja. 2018 Automatic Exsay Scoring of Swedish Essays using Neural Networks. Ph.D. thesis, Uppsala University.
Stephanic Link, Ahmet Dursun, Kadir Karakaya, and Volker Hegelheimer. 2014. Towards Better ESL Prac- tices for Implementing Automated Writing Evalua- tion. Calico Journal, 31(3).
John W. Oller, Jr and F. B. Hinofotis. 1980. Two mu- tually exclusive hypotheses about second language ability factor analytic studies of a variety of lan- guage subtests.
Longman. 2002. Longman Essential Activator. Pearson ESL, Harlow.
Masumi Ono, Hiroyuki Yamanishi, and Yuko Hijikata. 2019. Holistic and Analytic Assessments of the

---

TOEFL iBT@ Integrated Writing Task. JLIA Jour- nal, 22(0))65-88. 88.
Alexandr Rosen, Jirka Hana, Barbora Hladka, Tomas Jelinek, Svatava Skodová, and Barbora Stindlová. 2020. Compiling and annotating a learner corpus for morphologically rich language CreSL, a corpus of non-native Czech.
Ellis Page. 1966.
The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238-243,
Nicholas Parslow. 2015a. Automated Analysis of L.2 French Writing: a preliminary study. Master's thesis. Publisher: Unpublished.
Rex Dajun Ruan. 2020. Neural Network Based Auto- matic Essay Scoring for Swedish. page 61.
Lawrence M. Rudner and Tahung Liang. 2002. Au- tomated essay scoring using Bayes' theorem. The Journal of Technology, Learning and Assessment. 1(2).
Nicholas Lynton Parslow. 2015b. Automated Analysis of L2 French Writing: a preliminary study. Ph.D. thesis, Master's thesis). University of Paris Diderot. doi: 10.13140/RR. 2.1, 2833.5204.
André A. Rupp. Jodi M. Casabianca, Maleika Krüger, Stefan Keller, and Olaf Köller. 2019. Automated es- say scoring at scale: a case study Switzerland and Germany. ETS Research Report Series, 2019(1):1- 23. Publisher: Wiley Online Library.
Clive Perdue. 1993. Comment rendre compte de la "logique" de "'aqquiiition d' unu langue étrangère par l'adulte. Études de Linguistique Appliquée, 92(1):8- 23.
Kumiko Sakoda and Yoko Hosoi. 2018. International
11dikó Pilán and Elena Volodina. 2018. Investigating the importance of linguistic complexity features across different datasets related to language learning. In Proceedings of the Workshop on Linguistic Complex- ity and Natural Language Processing, pages 49-58.
Corpus of Japanese as a Second Language (I-JAS); 4 TH t o) a0 liff 3L 2 ji 1- [For Language Research and Teaching of Japanese Learn- ers). In y 2 /
y Ln än a ー 1 2 0) fit
I6 &I [Symposium: Construction and Use of Spoken Language Corpus].
íldikó Pilán and Elena Volodina. 2018. Investigating the importance of linguistic complexity features across different datasets related language learning. In Proceedings of the Workshop on Linguistic Complex- ity and Natural Language Processing, pages 49-58.
Larry Selinker. 1972. Interlanguage. Publisher: Walter de Gruyter, Berlin/New York Berlin, New York.
Mark D Shermis, Jill Burstein, and Sharon Apel Bursky. 2013. Introduction to automated essay evaluation. In Handbook of automated essay evaluation, pages 23-37. Routledge.
Alice Pintard and Thomas François. 2020. Combining expert knowledge with frequency information to infer cefr levels for words. In Proceedings of the Work- shop on Tools and Resources to Empower People with REAding Difficulties (READI), pages 85-92.
Peter Siemen, A. Lüdeling, and F.H. Müller. 2006. FALKO An error-annotated German learner cor- pus. Proceedings of KONVENS 2006 Konferenz zur Verarbeitung Naturlicher Sprache, pages 130-136.
Nives Preradovic, Monika Berać, and Damir Boras. 2015. Learner Corpus of Croatian as Second and Foreign Language.
Christian Stab and Tryna Gurevych. 2014. Identifying Argumentative Discourse Structures in Persuasive Essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 46-56. Association for Computa- tional Linguistics.
Ekaterina V. Rakhilina, Anastasia Vyrenkova, Elmira Mustakimova, Alina Ladygina, and Ivan Smirnov. 2016. Building learner corpus for Russian. In Pro- ceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition, pages 66-75.
Kaveh Taghipour and Hwee Tou Ng. 2016, A neural approach automated essay scoring. In Proceed- ings of the 2016 conference on empirical methods in natural language processing, pages 1882-1899 18911
Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: system- atic literature review. Artificial Intelligence Review, 5593):2495-2577.
Kari Tenfjord, Paul Meurer, and Knut Hofland. 2006a.
The ASK corpus language learner corpus of Nor- wegian as second language. In Proceedings of the Fifth International Conference on Language Re- sources and Evaluation (LREC), Genoa, Italy. Euro- pean Language Resources Association (ELRA).
Bojana Ranković, Sarah Smirnow, Martin Jaggi, and Martin J. Tomasik. 2020. Automated Essay Scoring in Foreign Language Students Based on Deep Con- textualised Word Representations. In LAK20- 10th International Conference on Learning Analyties & Knowledge. Issue: CONF.
Kari Tenfjord, Paul Meurer, and Knut Hofland. 2006b. The ASK Corpus-a Language Learner Corpus of Nor- wegian as Second Language. In LREC, volume 6. pages 182 1821-1.24.
Randi Reppen. 2010. Using Corpora in Language Classroom. Cambridge University Press.

---

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nip pipeline. arXiv preprint arXiv: 1rxiv:9905.05900.
of NLP for Building Educational Applications. pages 118-123, Denver, Colorado. Association for Compu- tational Linguistics.
Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013. report on the first native language identification shared task. In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 48-57, Atlanta, Georgia. Associ- ation for Computational Linguistics.
Valentin Werner, Robert Fuchs, and Sandra Götz. 2020. LI influence vs. universal mechanisms: An SLA- driven corpus study on temporal expression. In Learner Corpus Research Meets Second Language Acquisition, pages 39 39-66. Cambridge University Press.
Masaki Uto. 2021. A review of deep-neural automated essay scoring models. Behaviormetrika, 48(2):499- 484.
Rodrigo Wilkens, David Alfter, Xiaoou Wang, Alice Pintard, Anaïs Tack, Kevin P Yancey, and Thomas François. 2022. Fabra: French aggregator- readability assessment toolkit. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1217-1233.
Sowmya Vajjala. 2018. Automated Assessment of Non- Native Learner Essays: Investigating the Role of Lin- guistic Features. International Journal of Artificial Intelligence Education, 28(1):79-105.
Katrin Wisniewski, Karin Schöne, Lionel Nicolas, Chiara Vettori, Adriane Boyd, Detmar Meurers, An- drea Abel, and Jirka Hana. 2013. MERLIN: An online trilingual learner corpus empirically grounding the European Reference Levels in authentic learner data.
Sowmya Vajjala and Taraka Rama. 2018. Experiments with Universal CEFR Classification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 147-153, New Orleans, Louisiana. Association for Computational Linguistics.
Edward W Wolfe, Tian Song, and Hong Jiao. 2016. Fea- tures of difficult-to-score essays. Assessing Writing, 27:1-10.
Freiderikos Valetopoulos and Jolanta Zajac. 2012. Les compétences en progression: un défi pour la didac- tique des langues.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A New Dataset and Method for Automatically Grading ESOL Texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pages 180-189, Portland, Oregon, USA. Association for Computational Linguistics.
Gudrun Sanderbauwhede. 2012. The integrated con- trastive model evaluated: The french and dutch demonstrative determiner in 11 and 12. International Journal of Applied Linguistics, 2293)3392-413.
Helmut Vollmer, J and John B. Caroll. 1983. Psycho- metric theory and language testing. In John W. Oller, editor, Issues language testing research, pages 29-79. Newbury House, Rowley, Mass. 00000.
Tal Yarkoni, David Balota, and Melvin Yap. 2008. Mov- ing beyond coltheart's n: A new measure of ortho- graphic similarity. Psychonomic bulletin & review, 15:971-979.
Helmut Vollmer, and Fritz Sang. 1983. Competing hypotheses about second language ability: plea of caution. In John W. Oller, editor, Issues in lan- guage testing research. Newbury House, Rowley, Mass. 00000.
Wajdi Zaghouani. 2002. AUTO-ÉVAL vers un modèle d' é'évllaation automatique des textes. In Actes du col- loque des étudiants en sciences du langage, page 16, Montréal, Canada. Université du Québec à Montréal.
Elena Volodina, Lena Granstedt, Arild Matsson, Beáta Megyesi, Iidikó Pilán, Julia Prentice, Dan Rosén, Lisa Rudebeck, Carl-Johan Schenström, Gunlög Sundberg, and Mats Wirén. 2019. The SweLL Lan- guage Learner Corpus: From Design to Annotation. The Northern European Journal of Language Tech- nology, 6:67-104.
Elena Volodina, lldikó Pilán, Ingegerd Enström, Lorena Llozhi, Peter Lundkvist, Gunlög Sundberg, and Mon- ica Sandell. 2016. SweLL on the rise: Swedish Learner Language corpus for European Reference Level studies. In Proceedings of the Tenth Interna- tional Conference on Language Resources and Eval- uation, pages 206-212.
Maolin Wang. Shervin Malmasi, and Mingxuan Huang. 2015. The Jinan Chinese Learner Corpus. In Pro- ceedings of the Tenth Workshop on Innovative Use

---

A Description of the learner and candidate corpora
Table 6 in this appendix provides summary of candidate corpora collected from L2 certification exams and learner corpora targeting French for comparison with TCFLE-S.
CANDIDATE CORPUS FROM 1.2 CERTIFICATIONS
CANDIDATE CORPUS
L2
NUMBER
LEVEL
NUMBER
TESTING INSIITUTION
ANNOTATION
OF LI
OF TEXTS
Open Cambridge Learner
ENG
all levels
10,000
Cumbridge
Assessment
auto: POS
Corpus (Open CLC)
English
CLC Pirst Certificate En-
ENG
all levels
1,238
Cambridge Assessment
POS and syntactic
glish (CLC-FCE)
English
manual:
ETS Corpus Non-
ENG
11
levels
12,100
Education Testing Sor-
Native Written English
vices (TORFL.)
MERLIN
CZE
A1-B2
2,290
ÚTOP
auto: POS
GER ITA
A1-CI
TELC
manual: errors, syntactic,
A1-B2
CEPR related
ASK
NNO
10
B1-82
1,936
Norwegian Language Test
auto: POS
manual errors, syntactic
COPLE2
POR
14
A1-CI
966
CAPLE
auto: POS
ICLP
manual: erross
TCFLE-8
FRE
levels
6,500
France Educution Interna-
tional
CORPORA
FRENCH
LEARNER CORPUS
L2
LEVEL
NUMBER
COLLECTION CONTEXT
ANNOTATION
OF WORDS
French Interlanguage
FRE
ENG
int adv
450,000
language class (univeristy)
manual: errors
Dutabuse (FRIDA)
DUT
others
Learner Corpus French
PRB
DUT
B2-C1
500,000
language class (university)
Chy-FLE
FRB
GRE
adv
150,000
language class (university) 1.2 high-school read aloud and inter- views from 115 students (university)
manual: grammatical
Hellla-FLL
stituents order annotution
Corpus Interlangue (CIL)
FRE
ARA
B1-C1
ENG
Madarin
tx))
ENG
SPA
Corpus Ecrit Français
FRE
deb
100,000
language class (high- school)
auto:
Langue Etrangère
manual: errors
Word Reference Corpus
FRE
not evalu- ated
FFL: 4M.
forum posts of Word refer-
no anmotation
ENG
ence website
SPA
ITA
Dire Autrement
FRE
ENG
int adv
50,000
language class (university) written production of TCF certificution (France Edu- cation International)
manual: lexical errors text-level annolation
TCELES
HRB
ENGLARA
A1-C2
580,000
SAP.RS,
POR.KAB
CHIJPN
Table 6: Candidate corpora and French learner corpora
B (Pseudo-)anonymizza)
we start by applying the MAPA tool in the entire corpus. Next, we followed Volodina et al. (2019) by selecting 200 random texts and evaluating them manually to assess MAPA's output. In addition, we controlled the same amount of text from each level because different levels can affect the system differently. However, contrarily to Volodina et al. (2019), we also evaluate whether anonymization is appropriate. The reason for this stricter approach was to measuring the of distorting caused by the peeuudd--anonymizzttin step.
For
The MAPA's evaluation was carried out by independent French native speaker. Each evaluated 100 essays. Later, third evaluator double-checked the 200 essays searching for inconsistencies, which were fixed after discussion with the other evaluators. During this assessment, we identify standard errors. These errors, described in Section 3.3, were automatically corrected after we identified their patterns occurrence.

---

The results of this evaluation is shown in Table 7. The first observation is about the ability fully identify an entity where MAPA presents difficulty. However, we point out that partial anonymization is already considered correct. In addition, small number of errors are caused by the entity type, as exemplified by the close scores in the partial matching columns the table, where the only distinction is the consideration of the entity and span or just the span. this evaluation, we highlight two scores: accuracy and F2. The first takes into account the words that have been correctly identified as non- anonymized. The second, on the other hand, is a variation of the F-score where recall receives a greater weight. F2 represents the interest of coverage but without significant loss in precision. Given the difference between these scores, and the recall and precision values, notice that MAP tends overdo, identifying more terms than needed for anonymization. Although the anonymization method generates an abundance edits in the text, ensures quality in the process and in the protection of the privacy of the writers.
span-based
entity-based Partial match
Partial match
Exact match
Accuracy
0.97
0.96
0.97
Precision
0.55
0.47
0.50
Recall
0.86
0.68
0.85
F1
0.67
0.56
0.63
F2
0.77
0.63
0.75
Table 7: Result of the anonymization evaluation
C
Confusion matrices of transformer-based AES model
Table shows the results for each of the 5 repetitions (see Section 5) of the AES model based transformers (column CamemBert Table 5).

---

Prediction
Prediction
A1
A2
B1
B2
C1
C2
A1
A2
B1
B2
C1
C2
A1
48
26
0
0
A1
51
21
0
0
0
0
A2
10
57
65
2
0
A2
30
87
19
0
0
0
Gold
B1
0
11
100
37
0
Gold
B1
29
76
38
2
B2
0
0
99
22
1
B2
24
63
55
C1
0
44
11
C1
1
13
87
C2
0
0
1
21
C2
0
31
19
(a) Prediction of Run
(b) Prediction of Run 2
Prediction
Prediction
A1
A2
B1
B2
C1
C2
A1
A2
B1
B2
C1
C2
A1
37
30
5
0
A1
20
49
0
0
0
A2
8
68
54
0
A2
93
41
0
Gold
B1
84
40
2
Gold
B1
28
99
17
2
B2
80
42
B2
40
31
C1
0
2
17
62
30
C1
3
30
71
10
C2
0
0
17
29
C2
0
1
32
10
(c) Prediction of Run 3
(d) Prediction of Run 4
Prediction
A1
A2
B1
B2
C1
C2
A1
47
16
3
0
A2
25
84
27
0
Gold
B1
44
66
32
3
B2
21
85
35
C1
0
0
29
70
18
C2
0
0
18
28
(e) Prediction of Run 5
Table Predictions repetions the Camem models fine-tuned to TCFLE-8
C.1 Correlated features
In this section, we list the features identified by the feature selection method presented in Section 4.2. Table 9 list as features and their correlations, while Figure plots the distribution of some feature across the six CEFR levels.

---

(a) Average number of syllables per
(b) Average ratio of affixes
(c) Squared type/token ration
word
054
(d) 1st quartile of Words in FLELex Average of Al level lemmas accord- source for Al CEFR level
(f) Average automatically identi- fied Language Tool
ing (Pintard and François, 2020)
2000
9000
(g) Average mean orthographic Average number surface form (i) Word probability, based on Lexique3 Levenstein distance (based Lexique3)
words in top 1000 words Lexique3
Figure 2: Violin plot of some of the top correlated features through the 6 CEFR levels

---

Variable
List features and their correlations with essay CEFR level
Family
Graded fea-
LEXgrdFMLAI sud (0,463), LEXgrdFSOOUA Top (-0,41), LEXgrdFA2.mn (-0,416),
tures
LEXgrdFAl (-0,529), LEXgrdFSOOUCL, var (0,402). LEXgrdFA22or (-0.532).
kurtuss LEXgrdBAI kurtosis (-0,418), LEXgrdBAI skewness (0,417), LEXgrdFSOOUAI std
(0,457), LEXgrdrSOOUCIssd (0,402), LEXgrd (0,463), LEX 2 (-
0,56), LEXgrdFA221 (-0,531), LEXgrdFB2 (-0,441), LEXgrdFCI 20p (-0,403),
LEXgrdFA2 (-0,497), LEXgrdFBI lOp (-0,45), LEXgrdFSOOUA I avg (-0,464),
LEXgrdFSOOUA var (0,457), LEXgrdFB220P (-0,483), LEXgrdFB 20P (-0,497),
LBXgrdIBB24 (-0,485), LEXgrdEMLA (-0,411)
Lexical diversity
LEXdvrWLRR, (0,545), LEXdvrFSS (0,559),
LEXdvrWSS (0,587),
LEXdvIFSRarv
(0,559).
LEXdvIVLRW avg
(0,433),
LEXdvrWLC
(0,545),
LEXdvINSR
(0,54),
LEXdvINSC
(0,54),
LEXdvrVLRayg
(0,478),
LEXdvrVSU avg
(0,451),
LEXdvIVLS avg
(0,478),
LEXdvrWSR avg
(0,587),
LEXdvrVLSW avg
(0,433),
LEXdvrVLCW avg
(0,433),
LEXdvrVLC. avg
(0,478),
LEXdvrWLSav
(0,545),
LEXdvrVSS avg
(0,482),
LEXdv1ELR avg
(0,558),
LEXdvrFLC avg
(0,558),
LEXdvrVSRW avg
(0,449),
LEXdvrFSC avg
(0,559),
LEXdvrFLS uvg
(0,558),
LEXdvINLS avg LEXdvINLC avg (0,544),
(0.544),
LEXdvrVSSW
(0,449), LEXdvINSS (0.54), LEXdvrVSR.., (0,482)
Lexical
ERRaIISUM (-0,771)
errors
Lexical Fre-
LEXfrqLNN (0,446), LEXfrQLNS (0,46), LEXfrqFCL (0,448),
quency
LEXfrgFNL2op (0,41), LEXfrQLCSq (0,496), LEX (0,464), LEXfrqLCL20P (0,488)
Lexical so-
LEXsopl var (0,419), LEXsopLWKIang (-0,443), LEXsopLWKI stteness skewness (0,417),
phistication
LEXsopLWKI (0,427), LEX (0,427), LEXsopFKIay (-0,429).
LEXsopFKI sid (0,419)
Lex.oorhholog
EXafxAas (0,442)
features
Orthographic
LEXnghNUM1Pp (-0,448), LEXnghPHOigg (0,426), LEXnghPHOşar (0,417),
neighbors
LEXnghORT median
median (0,407), LEXnghNUM20p (-0,526), LEXnghNUMF2op (-
0,436), LEXnghhRRQ.t (-0,431), LEXnghPHO (0,432), LEXnghPHO.s (0,425),
LEXnghORT iqr (0,504), LEXnghAVGF2op (-0,42), LEXnghORT sop (0,526),
LEXnghORT 90 (0,492), LEXnghORT avg (0,519), LEXnghNUMF ToP (-0,436),
LEXnghPHO.., avg (0,481), LEXighNUM, (-0,515), LEXnghPHO,... max (0,472)
Word length
LENwrdSYL.coo (0,416). LENWrdSYL (0,449), LENWidLETTERS... (0,415),
LENwrdS YL_ max (0,468), LENWILEETTERS (0,406), LENwrdSYL (0,414),
LENwrdS (0,517), LENwrdSYL (0,558)
Tense
fea-
SYNunsINDPP.o (-0,422)
tures
Text likeli-
DISIkh Kurtosis
(0,452), DISIKHVSML avg (-0,411), DISIkhWLM (-0,438),
hood
DISIhhFMMpp CP (-0,41), DISIKh (-0,424), DISIKhWLGM (-0.469).
DISIkhFSM Median
(-0,401), DISIKANSM (0,401), DISIkhWSMPo (-0,48),
DISIKhVLGM,
(-0,4), DISIKhFSM skowesss
(0,409). DISIkh (-0,451),
DISIkhWSMI
(-0,509), D (-0,416), DISIKhFL.M skewness
(0,455),
DSSI (-0,453), DISIkhALM (-0,424), DISIkhASM
(-0,422),
DISIkhFLM
(0,438), DISIKh
(-0,417),
DISIkhASM,in
(-0,406),
DISIKhWLMmedimn (-0,419), DISIKhWLMMa (0,4),
V
(-0,424),
DISIkhALM20p (-0,402), DISIKhWSMmelinn (-0,455)
Table 9: Correlation between features and CEFR level grouped by linguistic variable family. For the name of the features, see httpps:/cent:: uclouvain. wwwpuwawiib.baabbbraeeee/e www.abra/daooaooooo html.


Reviews:
[{"review_id": "TqFJtRqVZ5", "reviewer": "Reviewer_p5TN", "summary": "", "questions": "\nA) In lines 527-529 the manuscript states \"For example, measures of word length have been known to be good predictors of proficiency levels\", followed by two references (Ferris, 1994 and Grant and Gither, 2000). These reviewer is not familiar with these cited references, unfortunately, but I have the following three questions for clarity:\n\nA) The two references appear rather dated, are there any other more recent studies backing up this statement?\n\nB) Do these two references refer to French specifically, to the exclusion of other languages (e.g. English), so that the statement is applicable and relevant here?\n\nC) Do these two references apply to learner language (i.e. French, as noted in the previous question b), and written essays in certification examinations in particular (regardless of the CEFR level)? This seems to be a very specific setting in which this statement must apply.\n\n\n\n\nD) It is unclear to me what the author(s) mean(s) in line 349 when they mention \"atypical candidates\": this should be explained and potentially exemplified, although I wonder whether it is a badly chosen phrase, that may carry some negative overtones if not outright stigma with it. In particular, I would argue that being \"on the borderline between levels\" is not (necessarily or always) the same as being an \"atypical candidate\", i.e. there's no identity and possibly also no correlation between these two concepts. The author(s) would do well to consider a suitable (and clearer, more intuitive) alternative term here that does not raise controversy and does not appear to stigmatize candidates, whatever their circumstances and results.\n\n\n\nE) Statement in lines 542-544: \"To conclude, if this analysis confirms existing research findings, it also points out that TCFLE-8 may be helpful for SLA studies.\": Yes, of course, I would agree in principle with the claim, but how can this be done, exploring which issues specifically?? Otherwise like this the statement is rather empty.\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough and insightful review of our paper. Your feedback and meticulous attention to detail are valuable in refining our work, and we are grateful for the positive assessment you've provided. We would like to address the points you raised:\n\nYour recommendations for using \"language of habitual use\" instead of \"usual language,\" resolving acronym explanations, and addressing the inconsistencies in the language coverage descriptions are duly noted. Additionally, we will implement your suggestions to enhance the manuscript's readability and coherence.\n\n> A) In lines 527-529 the manuscript states \"For example, measures of word length have been known to be good predictors of proficiency levels\", followed by two references (Ferris, 1994 and Grant and Gither, 2000). These reviewer is not familiar with these cited references, unfortunately, but I have the following three questions for clarity:\n> A) The two references appear rather dated, are there any other more recent studies backing up this statement?\n> B) Do these two references refer to French specifically, to the exclusion of other languages (e.g. English), so that the statement is applicable and relevant here?\n> C) Do these two references apply to learner language (i.e. French, as noted in the previous  question b), and written essays in certification examinations in particular (regardless of the CEFR level)? This seems to be a very specific setting in which this statement must apply.\n\nWe have selected these references to show that this is already known in the domain. Ferris (1994) and Grant and Gither (2000) address AES for English. And, for the sake of completeness, we’ll include the following references, addressing AES for Swedish, Japanese, and French languages, respectively.\n- Pilán, I., & Volodina, E. (2018, August). Investigating the importance of linguistic complexity features across different datasets related to language learning. In Proceedings of the Workshop on Linguistic Complexity and Natural Language Processing (pp. 49-58).\n- Hirao, R., Arai, M., Shimanaka, H., Katsumata, S., & Komachi, M. (2020, May). Automated essay scoring system for nonnative japanese learners. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 1250-1257).\n- Parslow, N. L. (2015). Automated Analysis of L2 French Writing: a preliminary study (Doctoral dissertation, Master’s thesis). University of Paris Diderot. doi: 10.13140/RG. 2.1. 2833.5204).\n\nFerris (1994) applies to certification examinations in particular (i.e., TSOL), while the other references address learner corpora.\n\n> D) It is unclear to me what the author(s) mean(s) in line 349 when they mention \"atypical candidates\": this should be explained and potentially exemplified, although I wonder whether it is a badly chosen phrase, that may carry some negative overtones if not outright stigma with it. In particular, I would argue that being \"on the borderline between levels\" is not (necessarily or always) the same as being an \"atypical candidate\", i.e. there's no identity and possibly also no correlation between these two concepts. The author(s) would do well to consider a suitable (and clearer, more intuitive) alternative term here that does not raise controversy and does not appear to stigmatize candidates, whatever their circumstances and results.\n\nThank you for the suggestion. We'll change it for candidates on the borderline between levels.\n\n\n> E) Statement in lines 542-544: \"To conclude, if this analysis confirms existing research findings, it also points out that TCFLE-8 may be helpful for SLA studies.\": Yes, of course, I would agree in principle with the claim, but how can this be done, exploring which issues specifically?? Otherwise like this the statement is rather empty.\n\nWe'll precise it by replacing it with “In addition, while this analysis confirms existing research findings in AES, it also points out that TCFLE-8 may be helpful for new SLA studies.”\n\n\n> The paper uses the term \"usual language\" a number of times, which is duly explained in footnote 5 as follows: \"The usual language is the language the candidate indicates as the one they usually use.\". As far as I know, the standard (and probably preferred, also on the grounds of being more transparent and intuitive) term to indicate this same concept is \"language of habitual use\" (of the candidates / students / examinees). It is understood that this is not (necessarily / always) the same as first / native language, which is fine\n\nWe selected this term because it reflects how the term used by FEI is typically translated to English in the scientific literature. However, we do agree that language of habitual use makes a clear statement. We'll update it.\n\n\n> The manuscript explains the meaning and full form of CEFR in line 146, but the acronym is already used in the abstract (line 018) and in line 093 before then: acronyms/abbreviations should be explained on first occurrence with their full form, and then be used consistently thereafter.\n\nThanks for pointing it out. We'll correct this mistake.\n\n\n> Footnote 2: I appreciate that the title of the referenced learner corpus collection is clickable and has a built-in link to the online resource, but probably the footnote should rather (or also?) give the full URL.\n\nWe add it as a built-in link due to space constraints as it's a long URL, but we'll include the entire URL for accessibility of people with software issues or using a paper version.\n\n\n> Section 2.1.1 on candidate corpora focuses on English, so what is the relevance of this to the study presented in the paper, which focuses on French? Probably the same information can be given in a more compact from in a table, if it is deemed relevant?\n\nSection 2.1.1 presents several well-known candidate corpora to situate TCFLE-8 among similar corpora. There are 2 English corpora, one targeting German, Italian and Czech, one Norwegian and one Portuguese. We’ll add a reference to Table 3 in the Appendix for an overlook of learner corpora comparable to TCFLE-8. \n\n\n> Similarly, Section 2.1.2 on corpora from language classes focuses again on English; while it's clear that this is the dominant language, the direct relevance of this information for the paper that is devoted to French is largely unclear. As suggested above for similar reasons for section 2.1.1, probably the same information (about English, as background for comparison purposes) can be presented more succinctly in table format.\n\nSection 2.1.2 presents learner corpora that have been used for AES, with one targeting the English language, the biggest learner corpus used for AES, and three others targeting Spanish, Swedish and Japanese. We’ll also add a reference to Table 3\n\n\n> Lines 168-169 mention \"three recent candidate corpora\", but they go as far back as 2013 (at least), so the use of the adjective \"recent\" does not seem (equally) justified for all of them.\n\nWe'll remove the \"recent\" as we agree it is needless here.\n\n\n> Line 346 mentions that an \"empirical evaluation\" was peformed, but it remains unclear how this was conducted; it would be useful and interesting for the readers to give at least a brief description of this evaluation and what it main findings were that justified removing all essays with a standard residual value greated than 4. In particular, how many essays from the whole initial collections were dropped, as a result of this decision?\n\nWe’ll provide a brief description of the final version.\n\n\n> Line 530 mentions the acronym \"TTR\", without explanation. I assume this is \"type/token ratio\", and in any case, the full form of the term/concept should be given, to avoid ambiguity and potential confusion on the part of the readers, especially if the acronym/concept is not used elsewhere.\n\nYes, it's type/token ratio. We'll precise it.\n\n\n> Table 5: the label in the left-hand column should probably be \"(CEFR) Level\", rather than \"Language\" for clarity. Again Table 5: I would add to the caption (the rest stays as is): \", per (CEFR) level\"\n\nWe’ll fix the column head and the caption.\n\nYour careful evaluation adds credibility to our work and reinforces our commitment to delivering robust research outcomes. Your thorough review and insightful comments have been instrumental in guiding our revisions. We are dedicated to addressing your concerns and making the necessary improvements to enhance the overall quality of the paper. Thank you for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "An interesting and well-written paper that addresses a relevant issue with originality and a sound and valid methodology, providing clear and useful results. In addition, the paper describes in detail a novel and very valuable corpus of approximately 6.5k language certification essays in French that has been carefully curated and is accompanied by interesting metadata.", "reasons_to_reject": "None in particular, or that struck me as definitely negative upon reading the manuscript.", "questions_to_authors": "\nA) In lines 527-529 the manuscript states \"For example, measures of word length have been known to be good predictors of proficiency levels\", followed by two references (Ferris, 1994 and Grant and Gither, 2000). These reviewer is not familiar with these cited references, unfortunately, but I have the following three questions for clarity:\n\nA) The two references appear rather dated, are there any other more recent studies backing up this statement?\n\nB) Do these two references refer to French specifically, to the exclusion of other languages (e.g. English), so that the statement is applicable and relevant here?\n\nC) Do these two references apply to learner language (i.e. French, as noted in the previous question b), and written essays in certification examinations in particular (regardless of the CEFR level)? This seems to be a very specific setting in which this statement must apply.\n\n\n\n\nD) It is unclear to me what the author(s) mean(s) in line 349 when they mention \"atypical candidates\": this should be explained and potentially exemplified, although I wonder whether it is a badly chosen phrase, that may carry some negative overtones if not outright stigma with it. In particular, I would argue that being \"on the borderline between levels\" is not (necessarily or always) the same as being an \"atypical candidate\", i.e. there's no identity and possibly also no correlation between these two concepts. The author(s) would do well to consider a suitable (and clearer, more intuitive) alternative term here that does not raise controversy and does not appear to stigmatize candidates, whatever their circumstances and results.\n\n\n\nE) Statement in lines 542-544: \"To conclude, if this analysis confirms existing research findings, it also points out that TCFLE-8 may be helpful for SLA studies.\": Yes, of course, I would agree in principle with the claim, but how can this be done, exploring which issues specifically?? Otherwise like this the statement is rather empty.\n\n\n", "paper_topic_and_main_contributions": "The paper contributes to the improvement automated essay scoring for French and introduces a new valuable corpus consisting of 6.5k essays collected in a well-known certification exam for French (the official French Knowledged Test, TCF) that are related to the CEFR levels. The paper also discusses how certain key linguistic properties of the essays relate to the learners’ proficiency as reflected in the corpus data.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nMinor comments and suggestions that the authors might consider to improve the manuscript upon revision:\n\n\n- The paper uses the term \"usual language\" a number of times, which is duly explained in footnote 5 as follows: \"The usual language is the language the candidate indicates as the one they usually use.\". As far as I know, the standard (and probably preferred, also on the grounds of being more transparent and intuitive) term to indicate this same concept is \"language of habitual use\" (of the candidates / students / examinees). It is understood that this is not (necessarily / always) the same as first / native language, which is fine.\n\n- The manuscript explains the meaning and full form of CEFR in line 146, but the acronym is already used in the abstract (line 018) and in line 093 before then: acronyms/abbreviations should be explained on first occurrence with their full form, and then be used consistently thereafter.\n\n- Footnote 2: I appreciate that the title of the referenced learner corpus collection is clickable and has a built-in link to the online resource, but probably the footnote should rather (or also?) give the full URL.\n\n- Typo in line 138: \"Corpora built for AES can focus _of_ specific dimensions\" >>> \"Corpora built for AES can focus _ON_ specific dimensions\"\n\n- Section 2.1.1 on candidate corpora focuses on English, so what is the relevance of this to the study presented in the paper, which focuses on French? Probably the same information can be given in a more compact from in a table, if it is deemed relevant?\n\n- Similarly, Section 2.1.2 on corpora from language classes focuses again on English; while it's clear that this is the dominant language, the direct relevance of this information for the paper that is devoted to French is largely unclear. As suggested above for similar reasons for section 2.1.1, probably the same information (about English, as background for comparison purposes) can be presented more succinctly in table format.\n\n- Lines 168-169 mention \"three recent candidate corpora\", but they go as far back as 2013 (at least), so the use of the adjective \"recent\" does not seem (equally) justified for all of them.\n\n- Typo in line 202: \"Appendix A provides _a_ additional detailed description\" >>> \"Appendix A provides _AN_ additional detailed description\"\n\n- Typo in line 342: \"which _humans_ raters\" >>> \"which _HUMAN_ raters\"\n\n- Line 346 mentions that an \"empirical evaluation\" was peformed, but it remains unclear how this was conducted; it would be useful and interesting for the readers to give at least a brief description of this evaluation and what it main findings were that justified removing all essays with a standard residual value greated than 4. In particular, how many essays from the whole initial collections were dropped, as a result of this decision?\n\n- It is unclear to me what the author(s) mean(s) in line 349 when they mention \"atypical candidates\": this should be explained and potentially exemplified, although I wonder whether it is a badly chosen phrase, that may carry some negative overtones if not outright stigma with it. In particular, I would argue that being \"on the borderline between levels\" is not (necessarily or always) the same as being an \"atypical candidate\", i.e. there's no identity and possibly also no correlation between these two concepts. The author(s) would do well to consider a suitable (and clearer, more intuitive) alternative term here that does not raise controversy and does not appear to stigmatize candidates, whatever their circumstances and results.\n\n- Potential typo in lines 560-561, to be checked: \"Any essay that does not fit XXX of these two criteria...\" >>> \"Any essay that does not fit _EITHER_ of these two criteria...\"???\n\n- There's a clear and serious inconsistency (that can be easily resolved) between line 368 (which states \"As the top six [languages] were all European ones,,,\"), which does not correspond to the statement in line 454, which also mentions Arabic, Russian and Kabyle (in addition to Chinese and Japanese), as follows: \"As for the usual language, the corpus covers 8 languages, as described in Section 3.2: English, Arabic, Spanish, Russian, Portuguese, Kabyle, Chinese, and Japanese...\". These explanations of the languages covered by the corpus in terms of \"usual languages\" (or, perhaps preferably, \"languages of habitual use\", as noted above) of the learners/examinees seem to be inconsistent and in conflict. Also make sure that there are no other incompatible or confusing statements in this regard elsewhere in the manuscript.\n\n- In lines 527-529 the manuscript states \"For example, measures of word length have been known to be good predictors of proficiency levels\", followed by two references (Ferris, 1994 and Grant and Gither, 2000). These reviewer is not familiar with these cited references, unfortunately, but I have the following three questions for clarity:\n\na) The two references appear rather dated, are there any other more recent studies backing up this statement?\n\nb) Do these two references refer to French specifically, to the exclusion of other languages (e.g. English), so that the statement is applicable and relevant here?\n\nc) Do these two references apply to learner language (i.e. French, as noted in the previous question b), and written essays in certification examinations in particular (regardless of the CEFR level)? This seems to be a very specific setting in which this statement must apply.\n\n\n- Line 530 mentions the acronym \"TTR\", without explanation. I assume this is \"type/token ratio\", and in any case, the full form of the term/concept should be given, to avoid ambiguity and potential confusion on the part of the readers, especially if the acronym/concept is not used elsewhere.\n\n- Statement in lines 542-544: \"To conclude, if this analysis confirms existing research findings, it also points out that TCFLE-8 may be helpful for SLA studies.\": Yes, of course, I would agree in principle with the claim, but how can this be done, exploring which issues specifically?? Otherwise like this the statement is rather empty.\n\n- Typo in line 582: \"between the ratings of one of the two _evaluator_ and the reference CEFR levels\" >>> \"between the ratings of one of the two _EVALUATORS_ and the reference CEFR levels\"\n\n- Typo in footnote 14: \"... because the rating assigned by each rater _contribute_ to the final CEFR score...\" >>> \"... because the rating assigned by each rater _CONTRIBUTES_ to the final CEFR score...\"\n\n- Line 634: \"anonymization\" >>> \"(pseudo-)anonymization\"\n\n- Possible typo in line 657: \"Given the weak complementary observed between both solutions\" >>> should \"complementary\" be \"complementarity\" instead?\n\n- Typo in line 687: \"However, this _lead_ to the removal\" >>> \"However, this _LED_ to the removal\"\n\n- Generally, Appendix B on (pseudo-)anonymisation is not well written, there are several obvious typos: that whole part should be carefully proofread to resolve the problems.\n\n- Typo in line 1277: \"Finally, Table 7 picture the amount of essays\"??? This seems incorrect: proofread careafully and amend as required.\n\n- Other typos are found in lines 1284-1285: proofread carefully and amend as required.\n\n- Table 5: the label in the left-hand column should probably be \"(CEFR) Level\", rather than \"Language\" for clarity.\n\n- Again Table 5: I would add to the caption (the rest stays as is): \", per (CEFR) level\"\n\n- Typo in Table 17: \"Arab\" >>> \"Arabic\" (for the language)\n"}}, {"review_id": "36xLy7FxA1", "reviewer": "Reviewer_5vDV", "summary": "", "questions": "0. What does \"LE\" of TCFLE-8 stand for? Does \"8\" mean eight L1 languages?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased to note your recognition of the main contributions of our paper. Furthermore, we are encouraged by your positive evaluation of our work's strengths. \n\nRegarding your question about \"LE\" in \"TCFLE-8,\" we appreciate your inquiry and will precise it in the final version. TCF stands for “Test de connaissance du français” and FLE stands for “français langue étrangère” (french as a foreign language) and \"8\" indeed refers to the eight different L1 languages included in the corpus. \n\nWe thank you for your attention to detail in identifying the correct naming convention for the authors. We will make the necessary correction to accurately cite the reference as \"Ono et al\" rather than \"Masumi Ono et al.\" \n\nWe sincerely appreciate your thorough evaluation and positive feedback. Your insights have contributed to improving the quality and impact of our paper. Thank you for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- According to Table 3, the proposed corpus is superior to previous corpora, which are not suitable for AES because of the rating span.\n- Rich metadata and annotations.\n- Comprehensive related work.\n- The baselines are provided.", "reasons_to_reject": "I did not find any flaw that prevents the paper from being published.", "questions_to_authors": "0. What does \"LE\" of TCFLE-8 stand for? Does \"8\" mean eight L1 languages?\n", "paper_topic_and_main_contributions": "This work presents a new corpus for automated essay scoring for French.\nThe corpus contains the followings:\n- 6,569 essays\n- CEFR ratings (A1-C2)\n- L1 labels\n- gender labels\n- CEFR level assigned to examinees\n- many annotations automatically assigned\n\nIt provides the way of creating the corpus including explanation of the test, cleaning, and distribution of L1.\n\nThe paper also provides the base line results using CamemBERT, XGBoost, and logistic regression.\nThe correlation of annotated variables and CEFR levels is also reported.\n\nLast but not least, the related work section a comprehensive review on AES corpora including those for English and French.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l.319 Masumi Ono et al -> Ono et al (Masumi is her first name)"}, "extra_text": {}}, {"review_id": "bIRwEMnBnn", "reviewer": "Reviewer_cs99", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful review of our paper. Your feedback is invaluable in enhancing the quality and impact of our work, and we are pleased to see that the overall assessment is positive. Your acknowledgment of the potential impact on AES research, particularly in the context of the French language, motivates us to continue our efforts in facilitating advancements in this field.\n\nWe appreciate your observation regarding the variability in performance between methods in Table 2. Your concern about potential majority-class bias in the Camembert model is valid, and we apologize for not including the predicted distribution statistics in the current version of the paper. We will promptly address this by providing the statistics on the predicted distribution (i.e., confusion matrix) to enhance the transparency and understanding of the model's performance. Additionally, we highlight that we tried to make a stratified sample as much as possible. However, the efforts to stratify the corpus ended up interfering with the final size of the corpus. We therefore opted for a larger corpus with an imbalance in the rare classes in the corpus.\n\nYour agreement with our note in Section 7 about the criteria for the exclusion of essays is noted. We acknowledge that this represents a gap, and your feedback reinforces the importance of this aspect that may be explored in the future. We will further elaborate on the considerations and implications of this exclusion criterion in the revised version of the paper.\n\nWe're deeply grateful for your thoughtful review and constructive feedback. Your observations will enhance the clarity and comprehensiveness of our paper. Thank you once again for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The corpus as described is sizeable and of high quality.  The rigorous scoring protocol and useful metadata will make this an important resource for researchers working on AES, especially those focused on the French language\n- The baseline scoring results described here will be a helpful starting point for researchers wishing to apply novel methods to this task\n- The literature review is comprehensive and on-point\n- The paper is well-crafted and clear in its presentation and organization", "reasons_to_reject": "I have no significant concerns about the paper that would lead me to recommend rejection.  However\n\n- I was surprised by the variability in performance between methods in Table 2.  The high accuracy for Camembert, coupled with much lower QWK, suggests that it may have a serious majority-class bias.  Statistics on the predicted distribution would be useful information.\n- I concur with the authors' note in section 7 (Limitations) that the criteria for exclusion of essays that are not clear examples of one score/level or another represent a significant gap to consider\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes a new resource for French-language AES research to be released publicly.  The corpus consists of essays across a range of CEFR levels, from the Test de Connaissance de Francais certification exam.  The paper describes distributional statistics of the corpus, criteria for inclusion of essays, available metadata, and linguistic features made available as part of the resource.  The paper also presents baseline AES results using deep learning and feature-based machine learning", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an original and valuable corpus for automatic essay scoring for French. The content and presentation of this paper is excellent, as reflected by three Soundness ratings of 5 and three Excitement ratings of 4. (Two reviewers rated its Reproducibility a 4 and one rated it a 5.) Other aspects of note are the paper's clarity, its comprehensive literature review, the quality of the corpus presented, as well as the inclusion of baseline results that set the stage for future work. None of the reviewers cited reasons to reject the paper, though all raised minor issues (e.g. grammatical errors, typos, unclear terminology) that were resolved during the author rebuttal period.

In light of this, only minor revisions, addressing reviewers’ comments and questions, need to be made to ensure this paper is camera ready.
Decision: Accept-Main