# The more proficient the learners, the less sophisticated their L2 vocabulary?

Raffaella Bottini, Elen Le Foll

<figure><img src="image_1.png" /></figure>

► To cite this version:

Raffaella Bottini, Elen Le Foll. The more proficient the learners, the less sophisticated their L2 vocabulary?. International Journal of Learner Corpus Research, 2024, 10.1075/ijlcr.23029 bot. hal-04799095

HAL Id: hal-04799095

https://hal.science/hal-04799095v1

Submitted on 22 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# The more proficient the learners, the less sophisticated their L2 vocabulary?

## The curious effect of the reference corpus on mean-frequency measures of lexical sophistication

Raffaella Bottini and Elen Le Foll

Lancaster University

University of Cologne

Mean-frequency scores of lexical sophistication are used to evaluate written and spoken language production. They are calculated using word frequencies extracted from a reference corpus. Using mixed-effects regression models, we analyse the strength of the relationship between L2 proficiency and mean-frequency scores in spoken and written texts using reference corpora representing different modes and registers. We control for task and topic effects. We observe that mean-frequency measures of lexical sophistication are considerably more influenced by the mode and register of the reference corpus used to calculate these scores than by language users' proficiency level. Advanced language users produce more frequent vocabulary, typical of the target register, in both spoken monologues and written essays. These results provide evidence in favour of a conceptual and terminological shift from *lexical sophistication* to *register appropriateness* (as suggested by Durrant & Brenchley, 2019) to refer to the construct captured by mean-frequency scores of vocabulary use.

**Keywords:** lexical complexity, L2 proficiency, reference corpora, register appropriateness, language testing

## 1. Introduction

Lexical complexity refers to the structure of a text in terms of its vocabulary, i.e. the number and type of words that constitute the text and their relational patterns (cf. Pallotti, 2020). Lexical complexity indices are widely used in applied linguistics, for example in research on language acquisition, language impairment,

<figure><img src="image_3.png" /></figure>
stylistics, and register variation (e.g., Dawson et al., 2021; Fergadiotis et al., 2013). Studies in psycholinguistics and second language acquisition (SLA) analyse lexical complexity to investigate how second language (L2) learners process and produce vocabulary (e.g., Balota et al., 2007; Berger et al., 2019) and to explore the relationship between lexis and L2 proficiency (e.g., Lu, 2012; McNamara et al., 2010). In learner corpus research, lexical complexity measures are also used to assess vocabulary use in learners' written and spoken production, combining insights from corpus linguistics, language testing, and computational linguistics. Within the construct of lexical complexity, lexical sophistication has been investigated extensively in L2 production, both in the written and spoken modes (e.g., Crossley et al., 2011; Crossley & McNamara, 2012; Eguchi & Kyle, 2020; Kim et al., 2018; Kyle & Crossley, 2016; Saito et al., 2021; Salsbury et al., 2011).

Lexical sophistication is a multidimensional construct. Read (2000: 200–203) defines it as “the proportion of relatively unusual or advanced words” in a text, “appropriate to the topic and style” of the communicative event. This definition is based on the frequency of individual words and their appropriateness to the target register. Recent studies have analysed aspects of lexical sophistication beyond frequency, such as range, phonological, orthographic and register-related features, hypernymy, polysemy, psycholinguistic properties of words (e.g., concreteness, familiarity, imageability) (e.g., Eguchi & Kyle, 2020), as well as n-grams and multi-word units, extending the conceptualisation of lexical sophistication to phraseological sophistication (Paquot, 2019; Paquot et al., 2022). In this paper, we focus on the frequency of individual words.

Lexical sophistication based on frequency is typically operationalised using two types of indices. Lexical frequency profiler are indices based on frequency bands in a reference corpus (e.g., high/mid/low-frequency; Nation, 2001). They assess the vocabulary of a text by either providing a percentage for each frequency band or the number of words in the text that are not included in a predefined core wordlist. Conversely, mean-frequency metrics compute the average frequency score of a text by extracting a frequency value for each of its words from a reference corpus. The assumption is that the lower the mean-frequency score of a text, the less frequent (therefore, the more sophisticated) its vocabulary. Crossley et al. (2013) have shown that mean-frequency measures of lexical sophistication are more fine-grained than lexical frequency profiler. They can predict the L2 proficiency level categorisation of a text with a 58% accuracy rate compared to the 36–48% accuracy rate of lexical frequency profiler. Hence, mean-frequency indices have been preferred in recent studies which have measured the association between learners’ lexical sophistication and L2 proficiency (e.g., Crossley et al., 2019; Eguchi & Kyle, 2020).
To date, the literature on lexical sophistication in learner language has mainly focused on written L2 English, analysing its relationship with L2 proficiency. These studies generally show a negative correlation between mean-frequency scores of content words<fnref n="1" /> and learners’ L2 proficiency (e.g., Crossley et al., 2011, 2012; Kim et al., 2018; Monteiro et al., 2020), indicating that advanced learners produce less frequent content words than learners at lower proficiency levels. These findings are in line with SLA theories (e.g., Bell, 2003; Ellis, 2002; Ellis et al., 2008) which suggest that beginners produce more frequent vocabulary as they are foremost exposed to high-frequency words.

Fewer lexical sophistication studies have been conducted on L2 speech, either on cross-sectional (e.g., Bottini, 2022; Crossley & McNamara, 2013; Eguchi & Kyle, 2020; Kyle & Crossley, 2015) or longitudinal corpora (e.g., Crossley et al., 2010; Kim et al., 2018). Interestingly, the results of these studies point to the opposite trend: the more proficient the learners, the more frequent their content words. For example, Eguchi and Kyle (2020) analysed a cross-sectional L2 corpus of oral proficiency interviews and reported that proficient speakers produce more content words which are frequent, abstract, and widely used (e.g., *chance*, *different*, *know*, *thing*). Similarly, Bottini (2022) reported that advanced learners use more frequent content words compared to less proficient speakers in a corpus of L2 spoken English produced under exam conditions. Among possible explanations for these findings, the effects of topic, content familiarity, and planning have been discussed (e.g., Kormos, 2011; Ong & Zhang, 2010), as well as the use of infrequent cognate words by low proficiency learners (e.g., Horst & Collins, 2006). Bottini (2022) also suggested that advanced L2 speakers might make more informed lexical choices, producing words that are more commonly used in spoken language, thus showing awareness of register-related linguistic features. By contrast, Crossley and McNamara (2013) found that proficient L2 learners produce less frequent content words in speech, which they hypothesised reflects greater vocabulary size. In sum, existing studies on lexical sophistication across L2 proficiency levels point to contrasting results when mean-frequency scores are computed.

One possible reason for these partially contradictory results may be the choice of the reference corpus used to compute mean-frequency scores. Most studies on L2 production have relied on a mixture of written and spoken reference corpora,

<footnote n="1">1. Studies generally operationalise content words as all nouns, adjectives, adverbs, and lexical verbs (but Monteiro et al., 2020, for instance, excluded adverbs). Modal and auxiliary verbs are often excluded (e.g., Lu, 2012). However, these methodological details are not always specified in publications or in the documentation of tools that compute lexical sophistication scores. We also found that decisions about tokenization, lemmatization, POS-tagging, and frequency normalisation are often underreported for both the texts to be analysed and the reference corpora.</footnote>
without matching their mode or register to that of the learner texts. For example, Crossley and Kyle (2015) combined written and spoken reference corpora (the Brown corpus representing written American English, the written and spoken subcorpora of the BNC1994 and COCA, and SUBTLEX-us which comprises American TV series subtitles) to investigate lexical sophistication in L2 production, analysing both written essays and spoken samples. Kim et al. (2018) used the same combination of written and spoken reference corpora to explore lexical sophistication in a longitudinal corpus of L2 speech. By contrast, Eguchi and Kyle (2020) drew on a group of spoken reference corpora (SUBTLEX-us and the spoken subcorpora of the BNC1994 and COCA) to match the learner corpus mode, thus addressing Tidball and Treffers-Daller's (2008) call for research on L2 speech to use spoken reference corpora.

Few studies have commented on the impact of the reference corpus when computing frequency-based lexical measures. Kim et al. (2018) and Durrant et al. (2019) reported that mean-frequency scores based on different reference corpora correlate strongly. This led Durrant et al. (2019:57) to conclude that the choice of the reference corpus “does not appear to be an important variable”. By contrast, Durrant (2014) observed that the strength of the correlation between collocation frequency and learners’ knowledge of collocations varies across different reference corpora. More recently, focussing on written L1 English, Durrant and Brenchley (2019) and Durrant and Durrant (2022) introduced the notion of *register appropriateness* in research on children’s L1 vocabulary development. They analysed the frequency distribution of words across four written registers (academic, fiction, magazine, newspaper) and used these to compute “register scores”, i.e., the normalized frequency of each word in each register. They used these register scores to compare the vocabulary produced by children at different developmental stages across school subjects. The authors observed unexpected patterns (i.e., the high prevalence of magazine-type vocabulary in pupils’ science writing and the decrease of this type of vocabulary in older children’s writing) and concluded that researchers using frequency-based lexical measures must carefully consider the impact of the reference corpora used to compute such measures.

## 2. The problem

We demonstrate the importance of selecting a suitable reference corpus with an example. Extract (1) is taken from the *International Corpus Network of Asian Learners of English* (ICNALE; Ishikawa, 2023). It is part of the transcription of a spoken monologue produced by an upper-intermediate L2 English learner (corresponding to B2 level on the CEFR scale; Council of Europe, 2020).
(1) Having a part-time job is the main reason to have extra money for some people. It is also people's way to have an experience how to work the real job. Personally, I agree with doing a part-time job. First for getting the extra money is very important for college students since they don't want to burden their parents. I have a friend who did a part-time job before because she wanted to buy something. It was actually quite expensive so she decided to spend her own money. Second, the experience itself when doing a part-time will make the college students easier to have a real job after they graduate. Of course, they will get another friend. Third to get over the burdensome of college life, may become one of the reason too because college students already having a hard time working on their home works presentations, or simply read their textbooks the whole night for exams preparation. At least they need something to refresh their minds (file IDN87_PTJ_B2)

To measure the lexical sophistication of this extract using a mean-frequency score, we retrieve a frequency value for every lemma<fnref n="2" /> from a reference corpus. The Written BNC1994 (BNC Consortium, 2007) is often used as a reference corpus in studies on lexical sophistication as it is integrated in many of the existing automated tools that compute lexical scores (e.g., Kyle et al., 2018). Hence, in this example, we first assign each lemma in the extract to its corresponding frequency per thousand words (ptw) in the Written BNC1994³ – e.g., *have_VERB* 12.78; *a_DETERMINER* 22.78; *part-time_ADJECTIVE* 0.02. Note that we are using relative frequencies normalised ptw to be able to compare mean-frequency scores computed with reference corpora of different sizes. Next, we compute the mean-frequency score of all words (AW), content words (CW) and function words (FW) in the text. For extract (1), these are AW=9.16, CW=3.15, FW=18.01. However, if we change the reference corpus to match the language mode of the learner text, e.g., using the Spoken BNC2014 (Love et al., 2017), the scores change substantially: AW=8.60, CW=5.89, FW=12.59. The large difference between these two sets of scores is due to the relative frequency of individual words in the two reference corpora. For instance, the adjective *part-time* occurs four times in extract (1) and each of its occurrence is assigned either 0.02 ptw or 0.004 ptw depending on whether the Written BNC1994 or the Spoken BNC2014 is used as the reference corpus. In the same extract, the verb *burden* and the adjective *burdensome* correspond to 0.003 ptw and 0.001 ptw, respectively, based on the Written BNC1994. However, these

<footnote n="2">2. We define lemma as a headword and all its inflections, e.g., the lemma *train_NOUN* includes the headword *train* and the plural inflection *trains*.</footnote>

<footnote n="3">3. In this study, and as is currently standard in lexical sophistication research using mean-frequency scores, normalised frequencies are extracted from the reference corpora without considering the distribution of each word across each text file.</footnote>
two lemmas do not occur in the Spoken BNC2014 and are therefore assigned a frequency value of zero when this reference corpus is used. This straightforward comparison, using a single text from a single speaker, highlights a potential issue in terms of the reliability of lexical sophistication measures based on different reference corpora.

Even more concerning is what happens when we consider not just one speaker, but rather compare learners across different proficiency levels. Tables 1 and 2 display the mean-frequency scores of two sample monologues from the ICNALE corpus, produced by a pre-intermediate (CEFR A2) and an upper-intermediate (CEFR B2) learner of L2 English. The scores have been computed using the Written BNC1994 (Table 1) and the Spoken BNC2014 (Table 2).

<table><caption>**Table 1.** Mean-frequency score of all words (AW), content words (CW) and function words (FW) of two monologues produced by an A2 and a B2 learner of L2 English computed using the Written BNC1994 as the reference corpus</caption><thead><tr><th>Learner ID</th><th>AW</th><th>CW</th><th>FW</th></tr></thead><tbody><tr><td>TWN45 (CEFR A2)</td><td>8.29</td><td>3.21</td><td>15.30</td></tr><tr><td>IDN87 (CEFR B2)</td><td>9.16</td><td>3.15</td><td>18.01</td></tr></tbody></table>

*Note.* Shaded cells indicate the lowest mean-frequency score in each column.

<table><caption>**Table 2.** Mean-frequency score of all words (AW), content words (CW) and function words (FW) of two monologues produced by an A2 and a B2 learner of L2 English computed using the Spoken-BNC2014 as the reference corpus</caption><thead><tr><th>Learner ID</th><th>AW</th><th>CW</th><th>FW</th></tr></thead><tbody><tr><td>TWN45 (CEFR A2)</td><td>9.56</td><td>5.42</td><td>15.27</td></tr><tr><td>IDN87 (CEFR B2)</td><td>8.60</td><td>5.89</td><td>12.59</td></tr></tbody></table>

*Note.* Shaded cells indicate the lowest mean-frequency score in each column.

Table 1 is based on frequency scores extracted from the Written-BNC1994. It shows that the more proficient learner at B2 level produces a text characterised by a lower mean-frequency score for content words than the A2 learner. Low mean-frequency scores are usually equated with high lexical sophistication. Conversely, the A2 learner's text is regarded as the most sophisticated when all words and function words are considered. However, Table 2 shows that, when the Spoken BNC214 is used, the trend is reversed (see shaded cells). This time, the mean-frequency scores of content words suggest that the A2 learner's text is more sophisticated than the one produced by the B2 learner, while the B2 learner's
monologue is rated as the most sophisticated when function words and all words are considered.

These examples show that the choice of the reference corpus potentially affects the reliability of mean-frequency measures of lexical sophistication and their use in assessing learners' proficiency level. The comparisons in Tables 1 and 2 show that the reference corpus affects which level is highest/lowest, i.e. the difference among proficiency levels. This raises serious concerns in terms of both the replicability of empirical studies and the comparability of results across studies conducted with different reference corpora. It also highlights an important methodological issue which could have major implications in language testing, given that lexical sophistication scores are often used (together with other lexical indices) to predict L2 learners' proficiency level (e.g., Crossley et al., 2011; Eguchi & Kyle, 2020; Nakatsuhara et al., 2021). We are not the first to raise this issue: Tidball and Treffers-Daller (2008) and Brysbaert and New (2009) argue that the choice of a reference corpus that matches the register of the texts to be analysed is crucial. Similarly, Egbert (2017); Durrant and Brenchley (2019), and Durrant and Durrant (2022) highlight the importance of a reference corpus' situational variables. However, most studies on lexical sophistication have selected reference corpora without fully considering their representativeness in terms of mode and register, or their comparability to the texts to be analysed (but see Monteiro et al., 2018 who compared reference corpora representing L1 and L2 production). Previous studies largely focused on the predictive validity of lexical sophistication (i.e., investigating which of its component(s) are the stronger predictor(s) of L2 proficiency) and may have underestimated the effect of the mode and register of the reference corpora used to compute lexical sophistication measures.

### 3. Hypotheses

Previous research on mean-frequency scores of lexical sophistication has produced contradictory results. Mean-frequency scores of content words (CW) correlate negatively with L2 proficiency in writing (i.e., the more proficient the learners, the less frequent the words they produce, thus, the smaller the CW scores of their texts), while most studies on L2 speech point to the opposite pattern. In previous studies, there is often a mismatch between the learner data and the reference corpora regarding mode, register, and other situational variables. Our examples in Section 2 demonstrate the potential impact of the reference corpus used to measure mean-frequency scores, which might explain some of the contradictory findings in the literature. Based on our literature review, we formulate the following hypotheses:
1. In speech, the more proficient the language users, the more frequent the content words they produce and, thus, the higher the mean-frequency scores of their content words (CW). By contrast, in written production, the more proficient the language users, the less frequent their content words and, thus, the lower the CW scores of their texts.

2. The nature of the reference corpus (i.e., the mode and register it represents) has a statistically significant effect on lexical sophistication measures based on mean-frequency scores of content words (CW).

Our hypotheses apply to all types of mean-frequency scores for all words (AW), content words (CW), and function words (FW). For reasons of space, we focus on CW scores (our supplementary materials,<fnref n="4" /> however, include AW and FW scores).

We aim to ensure that our analyses are reproducible and replicable: we rely on open-access corpora, open-source software, and publish our code under a CC BY-SA license. We thus show how a cumulative approach to knowledge building can be embraced following Open Science principles to boost the practical significance, replicability, and generalisability of learner corpus research findings.

# 4. Methodology

## 4.1 Data

Our learner data comes from the ICNALE corpus. We selected ICNALE to address Hypothesis 1 because it comprises (1) over 3.5M words produced by L1 and L2 users of English (CEFR A2 to B2) and (2) both spoken and written data, which were collected controlling for the effects of task, topic, and production time, thus allowing a direct comparison across language modes (on the importance of controlling for these factors, see, e.g., Kormos, 2011; Ong & Zhang, 2010). The spoken component consists of monologues and dialogues. The written component features essays, available both in the original format and in an error-annotated version. In the present study, only the monologic speech and the unedited essays were analysed.

The ICNALE Spoken Monologues subcorpus (version 2.1) contains transcripts of speech recorded over the phone. Instructions were delivered via audio recording. Participants were asked to give their opinions on two statements: “It is important for college students to have a part-time job” (hereafter referred to as

<footnote n="4">4. The materials are accessible on: https://osf.io/mne6w/.</footnote>
PTJ) and “Smoking should be completely banned at all restaurants in the country” (hereafter referred to as SMK) (Ishikawa, 2023: 44). For each topic, participants had 20 seconds to plan their answer and 60 seconds to speak. The data collection system automatically stopped the recording after the allocated time. Participants were asked to answer the same questions again, thus each respondent spoke twice about each topic. We merged participants’ responses so that we could analyse two files per participant: one for each of the two topics.

The ICNALE Written Essays subcorpus (version 2.5) consists of two argumentative essays per participant. Participants were asked to give their opinion on the same two statements as for the monologue task. The instructions were: “Do you agree or disagree with the following statements? Use reasons and specific details to support your opinion” (Ishikawa, 2023: 41). For each essay, participants had 20–40 minutes to write 200–300 words using a word processor. The use of a spell checker was mandatory but no other reference tools were allowed.

Descriptive statistics about ICNALE are presented in Table 3. L2 English data was collected from college students in ten countries and regions in Asia (China, Hong Kong, Indonesia, Japan, Korea, Malaysia, Pakistan, the Philippines, Singapore, Taiwan, Thailand). L1 English data was collected from “college students, teachers and business persons in more than five countries” (Ishikawa, 2023: 18).

<table><caption>Table 3. Descriptive statistics of the ICNALE data used in this study</caption><thead><tr><th rowspan="2">Task</th><th rowspan="2">Proficiency</th><th rowspan="2">Number of texts</th><th colspan="5">Number of tokens</th></tr><tr><th>Min.</th><th>Max.</th><th>M</th><th>SD</th><th>Total</th></tr></thead><tbody><tr><td rowspan="5">Spoken Monologues</td><td>A2</td><td>200</td><td>63</td><td>332</td><td>170.27</td><td>59.45</td><td>34,054</td></tr><tr><td>B1.1</td><td>442</td><td>52</td><td>370</td><td>179.21</td><td>55.57</td><td>79,211</td></tr><tr><td>B1.2</td><td>938</td><td>55</td><td>458</td><td>218.49</td><td>68.27</td><td>204,944</td></tr><tr><td>B2</td><td>320</td><td>54</td><td>398</td><td>213.75</td><td>66.13</td><td>68,399</td></tr><tr><td>L1</td><td>300</td><td>165</td><td>477</td><td>305.88</td><td>55.91</td><td>91,763</td></tr><tr><td colspan="2">Total</td><td>2,200</td><td>52</td><td>477</td><td>217.44</td><td>74.50</td><td>478,371</td></tr><tr><td rowspan="5">Written Essays</td><td>A2</td><td>960</td><td>180</td><td>399</td><td>221.34</td><td>24.05</td><td>212,485</td></tr><tr><td>B1.1</td><td>1,904</td><td>176</td><td>440</td><td>227.88</td><td>28.59</td><td>433,880</td></tr><tr><td>B1.2</td><td>1,872</td><td>180</td><td>340</td><td>236.58</td><td>32.98</td><td>442,873</td></tr><tr><td>B2</td><td>464</td><td>178</td><td>475</td><td>241.55</td><td>35.64</td><td>112,081</td></tr><tr><td>L1</td><td>400</td><td>187</td><td>320</td><td>224.29</td><td>23.97</td><td>89,714</td></tr><tr><td colspan="2">Total</td><td>5,600</td><td>176</td><td>475</td><td>230.54</td><td>30.47</td><td>1,291,033</td></tr></tbody></table>
To test Hypothesis 2, we selected six reference corpora ensuring they repre-
sented different language modes (spoken, written, spoken+written) and two main
registers (general and academic English). We choose three reference corpora of
general English:

a. BNC1994-W: the Written *British National Corpus* BNC1994⁵ (85M tokens; BNC Consortium, 2007), representing L1 written English across different registers (e.g., fiction, newspapers, magazines);

b. BNC2014-S: the Spoken *British National Corpus* BNC2014 (11M tokens; Love et al., 2017), the most recent and largest general corpus of L1 spoken English, representing everyday conversations;

c. BNC-S+W: the combination of BNC2014-S and BNC1994-W.

We selected three reference corpora of academic English:

a. BAWE: the *British Academic Written English corpus* (7.2M tokens; Nesi & Gardner, 2012), consisting of essays and research reports produced by proficient students at UK universities across four disciplinary areas (Arts and Humanities, Social Sciences, Life Sciences, Physical Sciences);

b. BASE-L: the monologic component of the *British Academic Spoken English corpus*<fnref n="6" /> (855K tokens; Nesi, 2001) consisting of transcripts of lectures (L) in Arts and Humanities and in Life Sciences;

c. BASE-L+BAWE: the combination of the BASE-L and BAWE.

## 4.2 Procedure

For each of the six reference corpora, we extracted the normalised frequency per thousand words (ptw) of all occurring lemmas following the steps outlined in Table 4.

We measured lexical sophistication in the learner texts using LexComplexity-Tool v.2,<fnref n="7" /> a Python script which includes a wide range of lexical complexity metrics (including lexical diversity, sophistication, and density indices) and allows customisation of the wordlists extracted from the reference corpora (Bottini, 2022). For each text, we computed a mean-frequency score for all content lemmas. Lemmas not occurring in the reference corpora were included in the com-

<footnote n="5">5. Since the text files of the Written BNC2014 (Brezina et al., 2021) were not fully accessible when this study was conducted, the Written BNC1994 was used instead, excluding the written-to-be-spoken texts (i.e., speeches and news transcripts) and e-mails.</footnote>

<footnote n="6">6. To match the register of the ICNALE Spoken Monologues, in this study, we only used the monologic component of the BASE corpus.</footnote>

<footnote n="7">7. To access this tool, please contact the first author.</footnote>
<table><caption>**Table 4.** Reference corpora pre-processing workflow</caption><thead><tr><td>Step</td><td>Tool</td></tr></thead><tbody><tr><td>1. Data cleaning (e.g., deleting transcribing conventions) and tokenization</td><td>R script</td></tr><tr><td>2. Lemmatization and POS tagging</td><td>TreeTagger (Schmid, 1994), PENN tagset (Santorini, 1991)</td></tr><tr><td>3. Simplifying the POS tags (see supplementary materials) and processing the data to the format lemma_POS</td><td>R script</td></tr><tr><td>4. Extracting wordlists based on lemma_POS</td><td>Shell script</td></tr><tr><td>5. Normalising frequency values ptw</td><td>Shell script</td></tr></tbody></table>

putation by assigning them a frequency value of zero. Previous research often excludes auxiliary verbs, copular verbs, and adverbs which are not based on adjectives from the analysis of content lemmas (e.g., Eguchi & Kyle, 2020). We opted for including all content lemmas (operationalised here as all noun, adjective, adverb, and verb lemmas, including auxiliaries and modals) to limit tagging errors in identifying features such as copular verbs and thus facilitate research replicability.<fnref n="8" /> Although we compute content lemmas, for consistency with previous research, we hereafter refer to them as content words. Throughout this paper, the abbreviation CW refers to the mean-frequency score of these content words.<fnref n="9" /> LexComplexityTool was also used to obtain a complete list of all the lemmas not included in the reference corpora. This allowed us to check for any methodological issues in our processing pipeline and supported the qualitative analysis of the results.

We tested the two hypotheses using mixed-effects linear regression modelling with the *lme4* package (Bates et al., 2015). For each ICNALE subcorpus, we first computed the following maximal model:

CW ~ RefMode + RefRegister + RefMode*RefRegister + Proficiency + Topic + (1|ID)

This model attempts to predict the mean-frequency scores of content words (CW) of the ICNALE texts. The fixed effects are the mode of the reference corpus

<footnote n="8">8. Following one of the reviewers’ comments, we compared different operationalizations of content lemmas. These did not significantly affect our results (see supplementary materials).</footnote>

<footnote n="9">9. Mean-frequency scores were not log transformed as they follow a normal distribution. This decision was also informed by Durrant et al. (2019) who compared raw and log-transformed mean-frequency scores concluding that they offer similar information.</footnote>
used to compute the CW scores (RefMode; three levels: spoken, written, spo-
ken+written) and its register (RefRegister; two levels: academic, general), the
participants' L2 proficiency (five levels: A2, B1.1, B1.2, B2, L1), and the topic of
participants' production (two levels: part-time job [PTJ], smoking [SMK]). The
first three fixed effects were included to test our hypotheses. We added topic as
a predictor to control for the effect of this variable. The baseline values used
in the regression model are: written mode, general register, L1 proficiency, PTJ
topic. We tested Hypothesis 1 by comparing the regression coefficient estimates
of the five proficiency groups. The reference proficiency level is always L1 speak-
ers. To better discern differences across proficiency groups, we provide visualisa-
tions of the CW scores predicted by our models (Figures 2 and 5). For Hypothesis
2, we compared the estimated coefficients of all predictors. The alpha level was
set at *p* ≤ .01. We added participant ID as random effect because each participant
contributed two texts per subcorpus (one on each topic). We followed the pro-
cedure described in Gries (2015) to compare the maximal model to more parsi-
monious ones using likelihood ratio tests, selecting the model with the lowest AIC
as the best-fitting model. We checked the assumptions of normally distributed
and homogeneous residuals by inspecting diagnostic plots (histogram of residu-
als, Q-Q plots of residuals, and residuals plotted against fitted values). We also
checked variance inflation factors (VIF) and concluded that collinearity was not
an issue. For details of the model selection and diagnostics procedures, see sup-
plementary materials.

5. Results

We present the results of our analysis of the ICNALE Spoken Monologues in 5.1.
Those of the Written Essays follow in 5.2. In both subsections, we begin with a
description of the observed mean-frequency scores of content words (CW) before
analysing the results of the mixed-effects models. It is worth emphasising once
more how CW scores should be interpreted in the results: the larger the mean-
frequency score of a text, the more frequent its content words.

5.1 Results from the ICNALE spoken monologues

Figure 1 displays the observed mean-frequency scores of content words (CW) for the ICNALE Spoken Monologues. Each panel displays the CW scores calculated using one of the six reference corpora representing general English (in the top row of panels) and academic English (in the bottom row). Boxplots show the median and interquartile range of the CW scores.
Figure 1 shows that, across all six reference corpora, the CW scores for L1 speakers are consistently higher than those of L2 participants, thus, on average, L1 speakers use more frequent content words. Only very slight differences in median CW values can be observed across L2 proficiency levels. The use of a spoken reference corpus (BNC2014-S and BASE-L) leads to a considerable increase in CW scores in the ICNALE Spoken Monologues. Moreover, we observe a much wider range of CW scores using spoken rather than written reference corpora, as illustrated by the elongated distribution shapes. The patterns observed in Figure 1 were explored further using mixed-effects linear regression modelling. A summary of the best-fitting model is displayed in Table 5.

<figure><img src="image_4.png" /><figcaption>**Figure 1.** CW scores of the ICNALE Spoken Monologues computed using six different reference corpora</figcaption></figure>
<table><caption>**Table 5.** Summary of the best-fitting model to predict the CW scores of the ICNALE Spoken Monologues: CW ~ RefMode\*RefRegister + Proficiency + Topic + (1|ID)</caption><thead><tr><th rowspan="2">Predictors</th><th colspan="3">Response</th></tr><tr><th>Estimates</th><th>CI</th><th>p</th></tr></thead><tbody><tr><td>Intercept [RefMode: Written, RefRegister: General, Proficiency: L1, Topic: PTJ]</td><td>4.80</td><td>4.59, 5.01</td><td>&lt; 0.001</td></tr><tr><td>RefMode Mixed</td><td>0.41</td><td>0.34, 0.47</td><td>&lt; 0.001</td></tr><tr><td>RefMode Spoken</td><td>3.54</td><td>3.47, 3.60</td><td>&lt; 0.001</td></tr><tr><td>RefRegister Academic</td><td>0.27</td><td>0.20, 0.33</td><td>&lt; 0.001</td></tr><tr><td>Proficiency B2</td><td>-0.68</td><td>-0.96, -0.39</td><td>&lt; 0.001</td></tr><tr><td>Proficiency B1.2</td><td>-0.84</td><td>-1.08, -0.61</td><td>&lt; 0.001</td></tr><tr><td>Proficiency B1.1</td><td>-0.90</td><td>-1.16, -0.63</td><td>&lt; 0.001</td></tr><tr><td>Proficiency A2</td><td>-1.17</td><td>-1.49, -0.85</td><td>&lt; 0.001</td></tr><tr><td>Topic SMK</td><td>1.05</td><td>1.01, 1.08</td><td>&lt; 0.001</td></tr><tr><td>RefMode Mixed × RefRegister Academic</td><td>-0.12</td><td>-0.21, -0.03</td><td>0.010</td></tr><tr><td>RefMode Spoken × RefRegister Academic</td><td>-1.75</td><td>-1.84, -1.66</td><td>&lt; 0.001</td></tr><tr><td>Random Effects</td><td></td><td></td><td></td></tr><tr><td>σ<sup>2</sup></td><td>1.21</td><td></td><td></td></tr><tr><td>τ<sub>oo</sub> ID</td><td>1.53</td><td></td><td></td></tr><tr><td>ICC</td><td>0.56</td><td></td><td></td></tr><tr><td>N<sub>ID</sub></td><td>1,100</td><td></td><td></td></tr><tr><td>Marginal R<sup>2</sup> / Conditional R<sup>2</sup></td><td>0.417 / 0.743</td><td></td><td></td></tr></tbody></table>

*Note.* RefMode = mode of the reference corpus, RefRegister = register of the reference corpus, Mixed = spoken+written, PTJ = part-time job, SMK = smoking, σ² = within-person variance, τ₀₀ ID = between-person variance, ICC = intra-class correlation.

The model summarised in Table 5 describes 74% of the variance in CW scores in the ICNALE Spoken Monologues. The fixed effects (mode and register of the reference corpus, proficiency, and topic) describe 42% of the variance in CW scores, whilst the random effect variable (participants) accounts for an additional 33%. The estimate for the intercept (4.80) represents the average CW score for the spoken monologues on the PTJ topic produced by L1 speakers, measured using a written reference corpus of general English (BNC1994-W).

In Table 5, the negative sign of the estimates for the proficiency groups indicates a decrease in CW scores from L1 to L2 speakers. This means that the lower the proficiency level, the lower the CW scores (thus, the less frequent the content words). The estimate for the SMK topic, which is 1.05 units higher than the intercept, indicates that more frequent vocabulary is used to talk about this topic than PTJ. Most importantly, differences in the estimated coefficients are much greater
across different reference corpora than across different proficiency levels. Also, the model highlights a statistically significant interaction between the register and mode of the reference corpus on CW scores. This can clearly be seen in Figures 2 and 3 which display the CW scores predicted by the model, with red bars representing the mean predicted CW scores.

<figure><img src="image_5.png" /><figcaption>**Figure 2.** CW scores (by mode of reference corpus and participant proficiency level) predicted by the model summarised in Table 5 for the ICNALE Spoken Monologues</figcaption></figure>

Figure 2 confirms the patterns observed in Figure 1. On average, in monologic speech, L1 speakers are predicted to produce higher CW scores (thus, more frequent content words) than those produced by L2 learners. Also, CW scores based on the spoken reference corpora (BNC2014-S and BASE-L) are predicted to be the highest and display a wider distribution of data points compared to the written and spoken+written (mixed) reference corpora.<fnref n="10" />

Figure 3 shows that CW scores are predicted to be higher when measured using the general spoken (BNC2014-S) rather than the academic spoken reference

<footnote n="10">10. The wider distribution of data points is due to the fact that the spoken reference corpora used in this study are smaller than the written reference corpora (see Section 4.1). For this reason, some of the words contained in the texts of the ICNALE corpus may not occur in the spoken reference corpora, and are assigned a frequency score of zero when the spoken reference corpora are used for computation.</footnote>
corpus (BASE-L). Interestingly, this trend is not maintained for written and spoken+written (mixed) reference corpora.

<figure><img src="image_6.png" /><figcaption>**Figure 3.** CW scores (by mode and register of reference corpus) predicted by the model summarised in Table 5 for the ICNALE Spoken Monologues</figcaption></figure>

## 5.2 Results from the ICNALE Written Essays

Figure 4 displays the distributions of mean-frequency scores of content words (CW) in the ICNALE Written Essays calculated using each of the six reference corpora. As with the Spoken Monologues, we observe that the use of a spoken reference corpus generates considerably higher and more dispersed CW scores. This effect is, however, less pronounced with the spoken academic English reference corpus (BASE-L) than with the general English one (BNC2014-S). Table 6 displays a summary of the best-fitting mixed-effects regression model.

In Table 6, the estimate for the intercept (4.09) corresponds to the average CW score for the written essays on the PTJ topic produced by L1 speakers, measured using a written reference corpus of general English (BNC1994-W). The negative sign of the coefficient estimates for the proficiency groups indicates a decrease in CW scores from L1 to L2 speakers. As with the Spoken Monologues, the lower the proficiency level, the smaller the CW scores (thus, the less frequent the content words produced). The estimate of CW scores for the SMK topic, which is 0.48 units higher than the intercept, indicates that more frequent vocab-
<figure><img src="image_7.png" /><figcaption>**Figure 4.** CW scores of the ICNALE Written Essays computed using six different reference corpora</figcaption></figure>

ulary is used to talk about this topic compared to the PTJ topic. Crucially, Table 6 indicates that the estimated coefficients corresponding to the different reference corpora vary much more than those modelling differences across proficiency levels. Here, too, we find that the combination of the register and mode of the reference corpus significantly contributes to the prediction of the CW scores of the ICNALE essays. This can also be seen in Figures 5 and 6 which display the CW scores of the essays predicted by the model. Figures 5 and 6 confirm the trends observed in Figure 4 and those described in Section 5.1 for the ICNALE Spoken Monologues.
<table><caption>**Table 6.** Summary of the best-fitting model to predict the CW scores of the ICNALE Written Essays: CW ~ RefMode*RefRegister + Proficiency + Topic + (1|ID)</caption><thead><tr><th rowspan="2">Predictors</th><th colspan="3">Response</th></tr><tr><th>Estimates</th><th>CI</th><th>p</th></tr></thead><tbody><tr><td>Intercept [RefMode: Written, RefRegister: General, Proficiency: L1, Topic: PTJ]</td><td>4.09</td><td>3.95, 4.23</td><td>&lt;0.001</td></tr><tr><td>RefMode Mixed</td><td>0.32</td><td>0.29, 0.35</td><td>&lt;0.001</td></tr><tr><td>RefMode Spoken</td><td>2.83</td><td>2.80, 2.86</td><td>&lt;0.001</td></tr><tr><td>RefRegister Academic</td><td>0.26</td><td>0.23, 0.29</td><td>&lt;0.001</td></tr><tr><td>Proficiency B2</td><td>-0.33</td><td>-0.51, -0.14</td><td>0.001</td></tr><tr><td>Proficiency B1.2</td><td>-0.35</td><td>-0.50, -0.20</td><td>&lt;0.001</td></tr><tr><td>Proficiency B1.1</td><td>-0.39</td><td>-0.54, -0.24</td><td>&lt;0.001</td></tr><tr><td>Proficiency A2</td><td>-0.27</td><td>-0.43, -0.11</td><td>&lt;0.001</td></tr><tr><td>Topic SMK</td><td>0.48</td><td>0.46, 0.49</td><td>&lt;0.001</td></tr><tr><td>RefMode Mixed × RefRegister Academic</td><td>-0.10</td><td>-0.14, -0.06</td><td>0.001</td></tr><tr><td>RefMode Spoken × RefRegister Academic</td><td>-1.42</td><td>-1.46, -1.38</td><td>&lt;0.001</td></tr><tr><td colspan="4">Random Effects</td></tr><tr><td>σ<sup>2</sup></td><td>0.65</td><td></td><td></td></tr><tr><td>τ<sub>oo</sub> ID</td><td>0.91</td><td></td><td></td></tr><tr><td>ICC</td><td>0.58</td><td></td><td></td></tr><tr><td>N<sub>ID</sub></td><td>2,800</td><td></td><td></td></tr><tr><td>Marginal R<sup>2</sup> / Conditional R<sup>2</sup></td><td>0.407 / 0.753</td><td></td><td></td></tr></tbody></table>

Note. RefMode = language mode of the reference corpus, RefRegister = register of the reference corpus, Mixed = written+spoken, PTJ = part-time job, SMK = smoking, σ² = within-person variance, τ₀₀ ID = between-person variance, ICC = intra-class correlation.
<figure><img src="image_8.png" /><figcaption>**Figure 5.** CW scores (by mode of reference corpus) predicted by the model summarised in Table 6 for the ICNALE Written Essays</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 6.** CW scores (by mode and register of reference corpus) predicted by the model summarised in Table 6 for the ICNALE Written Essays</figcaption></figure>
# 6. Discussion

## 6.1 Hypothesis 1: The relationship between L2 proficiency and mean-frequency scores of content words (CW)

We hypothesised that the more proficient the language users, (a) the more frequent their content words in speech (i.e., the higher the mean-frequency scores of content words (CW) of their spoken texts), and (b) the less frequent their content words in writing (i.e., the lower the CW scores of their written texts). Our results partially support our hypothesis. According to our analysis of the ICNALE Spoken Monologues, proficient speakers produce more frequent content words. This is in line with the first part of our hypothesis. The visualisation of predicted CW scores (Figure 2) for this subcorpus highlights the same pattern regardless of the reference corpus used. This trend is in line with most research on lexical sophistication in L2 speech (e.g., Bottini, 2022; Eguchi & Kyle, 2020; Kyle & Crossley, 2015) and corresponds to proficient language users' ability to produce vocabulary that is typical of (and therefore frequent in) the mode and register represented in the reference corpus. By contrast, less proficient learners may be less aware of the specific linguistic, contextual, functional, and pragmatic factors that distinguish spoken from written communicative events and might use words which are less characteristic of the mode and register represented in the reference corpus. A possible explanation for this is that L2 production may be influenced by the language input that learners are exposed to. Proficient learners are more likely to access a wide range of examples of real-world language use (e.g., newspapers, online communication, university lectures) and will therefore be exposed to patterns of vocabulary use that vary across registers. Less proficient learners may be foremost exposed to the language of English as a Foreign Language (EFL) textbooks, which is likely to affect their L2 production. Various studies have shown that EFL textbooks rarely focus on lexical variation across registers, especially spoken registers (e.g., Gablasova & Bottini, 2022; Le Foll, 2022a). Indeed, as a variety, “Textbook English” is characterised by considerably less register variation than English outside the EFL classroom, and didactised representations of spoken English often share many linguistic features of written English (Le Foll, 2021, 2022b).

The results from the ICNALE Written Essays also show a similar pattern, with texts produced by the L1 writers characterised by the highest CW scores (i.e., frequent content words). This is contrary to the second part of our hypothesis and to previous research on lexical sophistication in (L2) writing, which has found that advanced writers produce content words with lower frequency scores compared to less proficient learners (e.g., Crossley et al., 2011, 2012; Kim et al., 2018; Monteiro et al., 2020). These unexpected results may be due to the specificities
of the learner data used in this study and, in particular, the influence of the task instructions which are discussed in Section 6.2.

These results bring us to the issue of interpreting CW scores across proficiency levels as a measure of language users' *lexical sophistication*. Indeed, the traditional interpretation of mean-frequency measures of vocabulary assumes that the lower the mean-frequency score, the more sophisticated and advanced the vocabulary. However, our analysis shows that advanced language users tend to produce vocabulary that is typical of the target language mode and register, and which is therefore characterised by higher frequency values in the reference corpus. This suggests a need for a conceptual shift from the traditional construct of *lexical sophistication*. The etymology of the adjective *sophisticated* is associated with something rare and artificial, while the noun *sophistication* refers either to something altered, corrupted, and deceptive, or something complex, demonstrating knowledge and expertise (OED, 2023) — this latter meaning highlights the link with the superordinate concept of lexical complexity. When used in relation to lexis, both *sophisticated* and *sophistication* involve a subjective evaluation of the text to be analysed, which highly rates the use of infrequent lexical items without considering their communicative function within a specific context (i.e., register). Pallotti (2020) expresses similar concerns and suggests that the importance of complexity in the target domain needs to be critically evaluated: “Complex constructions seem to be highly valued in academic contexts, but may not be so in others; in any case, complexity cannot grow indefinitely and beyond a certain level it may hamper functional communication” (2020:207; see also Leńko-Szymańska, 2019).

Our analysis of the ICNALE (and most research on L2 speech, see Section 2) suggests that the more proficient the language users, the more frequent their vocabulary — providing that the reference corpus is representative of the mode and register of language users' production. This means that proficient language users “select words that are characteristic of the register they are attempting to invoke” (Durrant & Durrant, 2022: 2). Our results therefore lend support to Durrant and Brenchley's (2019) suggestion to adopt the more transparent term *register appropriateness* (rather than *lexical sophistication*) to describe the construct measured with mean-frequency scores. This would have the advantage of accounting for both frequency and register-related aspects of vocabulary use. This shift is also in line with previous calls to consider *lexical sophistication* as a superordinate term defining a multidimensional construct. The notion of *register appropriateness* could thus be employed more narrowly to specifically denote a quality that can be calculated using mean-frequency scores.
## 6.2 Hypothesis 2: The effect of mode and register of the reference corpus on mean-frequency scores of content words (CW)

We predicted that the nature of the reference corpus — i.e., the mode(s) and register(s) it represents — has a statistically significant effect on mean-frequency scores of content words (CW). Our results support this hypothesis. First, we discuss the results from the ICNALE Spoken Monologue (6.2.1), then those of the Written Essays (6.2.2).

### 6.2.1 *The ICNALE spoken monologues*

In the ICNALE Spoken Monologues, the use of a spoken reference corpus produces higher CW scores in comparison to a written or written+spoken reference corpus. This suggests that the content words produced in the ICNALE Spoken Monologues are characteristic of the spoken mode and, therefore, more frequent in the spoken than in the written reference corpora. Also, the general English spoken corpus (BNC2014-S) produces the highest CW scores for the ICNALE Spoken Monologues. Hence, the ICNALE speakers use content words that are more typical of informal conversations (as represented in the BNC2014-S) than academic lectures (as represented in the BASE-L). Examples (2) to (4) are extracts taken from the ICNALE Spoken Monologues produced by a Chinese B1 speaker of L2 English (Extract 2) and two L1 speakers of English (Extracts 3 and 4). They exemplify the informal register used in the ICNALE Spoken Monologues. The language features in bold are typical of informal, interactive speech as if the participants were talking to an interlocutor on the phone. Content words (underlined) include adverbs (e.g., *maybe*, *well*), adjectives (e.g., *right*), and verbs (e.g., *know*, *agree*, *think*) — comprising delexicalized verbs (e.g., *get*) and phrasal verbs (e.g., *mess up*) — which are more frequent in the spoken than in the written mode and in the general rather than the academic register.

(2) Maybe you're right. Uh, the young — beyond going — the young beyond going to the communities and this — taking a part-time job, students can know the — society — society — society better and — uh — learn how to get along well with people.
(CHN_PTJ1_051)

(3) And also there is, you know, financial aid itself but most people aren't qualified so that's more complex that either, um, even if someone you know is they don't consider themselves well up financially...
(ENS_PTJ1_093)

(4) These days some people say that it is important for a university student to have a part-time job, I agree with that because... Oh, I — I think I messed up, can I fix it?
(ENS_PTJ1_065)
Participants' use of content words typical of informal conversations (as repre-
sented in the BNC2014-S) could be due to the audio instructions used to collect
the data for the ICNALE Spoken Monologues: these include linguistic features
typical of conversational speech (Biber et al., 2021), such as imperatives (e.g., "lis-
ten to the topic carefully") and direct question (e.g., "Do you agree or disagree
with this statement?"). The data collection system via telephone calls could also
have prompted the use of linguistic features typical of conversations. Therefore,
both the informal register of the instructions and their format (i.e., recording over
the telephone) could have elicited informal, conversation-like answers, charac-
terised by interactive features of language use as if the participants were address-
ing an imaginary interlocutor. Gablasova et al. (2023) made similar observations,
finding that L2 English learners frequently use interactional and interpersonal
discourse features in monologues produced in a computer-based language test.
These findings align with research on human-computer interaction which has
found linguistic similarities with human-human interaction (e.g., Dombi et al.,
2022; Ockey & Chukharev-Hudilainen, 2021).

In addition to language mode, the register represented in the reference corpus
also affects CW scores. The ICNALE Spoken Monologues were produced in
response to opinion-giving questions and therefore present features typical of
argumentative texts. The academic spoken reference corpus (BASE-L) used in
this study only contains lectures which are more likely to be informative rather
than argumentative. This might explain why predicted CW scores were higher
using BNC2014-S rather than BASE-L (see Figure 3).

When written reference corpora are used, the trend is reversed: the reference
corpus of general English (BNC1994-W) produces marginally lower CW scores
than the academic English corpus (BAWE). This can likely be explained by the
composition of these reference corpora. BAWE contains a range of academic
text types (e.g., essays, reports, critiques, case studies, literature surveys) which
are mainly argumentative and informative. By contrast, the BNC1994-W com-
prises mostly informative and imaginative texts (e.g., fiction, magazines, newspa-
pers) (BNC Consortium, 2007). This may explain why the largely argumentative
ICNALE Spoken Monologues are characterised by vocabulary more typical of
the academic register represented in BAWE (especially, its argumentative section)
rather than of the imaginative and informative texts in the general English refer-
ence corpus (BNC1994-W). The same pattern is observed when written+spoken
reference corpora are used. This is because the written reference corpora in this
study are larger than their spoken counterparts so that, when the written and spo-
ken corpora are combined, the influence of the written corpora has a considerably
larger effect. This imbalance reflects the composition of many general language
corpora currently used as reference corpora in lexical sophistication research.
**bollocks. [...] I honestly can't understand** what all the **kerfuffel** is about when they **rave on** about **character building** either. I **mean**, if I **want character building**, then I'll join the **military** or a **religious sect**. [...] Where I **come** from, if I **was** in a **job interview** and **told** the **interviewer** that I **delivered newspapers** in the **mornings** they **would probably say**, **straight off the bat**, "Oh I **see**, no **cash** hey?" It's **not important** to **have a part-time job** at all for **college purposes** unless you **need the cash**. **Period!**

(ENS_PTJ_002)

In the ICNALE Written Essays, the interaction between language mode and register of the reference corpus has the same effect observed in the ICNALE Spoken Monologues. Namely, using the spoken general English reference corpus (BNC2014-S) produces higher CW scores than the academic spoken reference corpus (BASE). This trend is not observed with written and spoken+written reference corpora. As discussed in Section 6.2.1, this is probably due to the composition of the reference corpora and the communicative functions expressed in their text files (e.g., argumentative, informative, imaginative texts). This further demonstrates the importance of carefully considering the suitability of the reference corpora used to measure mean-based frequency scores.

Our results also confirm how important it is to control for topic when comparing learners' language production using CW scores. Indeed, in both the spoken and the written ICNALE subcorpora, participants' answers in the smoking (SMK) tasks were characterised by higher CW scores compared to the part-time job (PTJ) tasks. This is because, on average, the content words used to discuss the SMK topic tend to be more frequent in the reference corpora used in this study.

Overall, we found that the CW scores of the ICNALE texts are considerably more influenced by the mode and register of the reference corpus than by the proficiency level of the participants or by the topic of the task. This has major methodological implications for (learner) corpus research and studies in (second) language acquisition, and practical implications for language testing. In line with previous studies (e.g., Brysbaert & New, 2009; Durrant & Brenchley, 2019; Durrant & Durrant, 2022; Egbert, 2017; Tidball & Treffers-Daller, 2008), we recommend that the reference corpora used to calculate mean-frequency scores match the mode and register of the texts to be analysed (in terms of their communicative context and functions) to ensure the validity and reliability of the measure. This also applies to language testing when mean-frequency scores are used for the (automated) assessment of learner production. Overlooking the effect of the reference corpus could potentially lead to unreliable evaluations of L2 proficiency (as demonstrated in Section 2). We also recommend normalising mean-frequency scores to facilitate the comparison of studies that use different reference corpora.
## 6.3 Limitations and future research

Several limitations should be mentioned to inform future research. First, this study focused on content lemmas and did not consider function words (but see supplementary materials). It also analysed the frequency of individual words rather than multi-word units. Future research could investigate the effect of the reference corpus on mean-frequency scores of function words and on measures of phraseological sophistication. Moreover, this study relied on frequency values extracted from reference corpora without taking dispersion into account, as is currently the norm in research on lexical sophistication. Considering the dispersion of words in the reference corpora is, however, likely to alter the picture and is certainly an avenue worth pursuing in future research.

Second, the use of the ICNALE presents some limitations. Our analysis only includes Asian learners of L2 English. This study could be replicated comparing the vocabulary produced by learners from a wider range of L2s and L1 varieties. The learners' proficiency levels in the ICNALE also constitute a potential limitation since these were assigned based on diverse tests and exams and are all self-reported. Nonetheless, the trends in all our analyses are clear and consistent. As for the language tasks, our study focused exclusively on informal monologues and essays. Future studies could investigate other tasks and registers. Also, in the ICNALE Spoken Monologues, learners were asked to talk about the same topic twice. While some speakers repeat similar content and ideas with a substantial overlap of vocabulary across the two attempts, other participants produce two different monologues characterised by different content words (Ishikawa, 2023: 44–45). This may have influenced the CW scores and the comparability of the transcripts across speakers (though learner-specific variability was accounted for in our mixed-effects models). Despite these limitations, in the context of our study, we believe that using the ICNALE strengthened the robustness of the findings. As explained in 4.1, this corpus was collected under rigorously controlled conditions which, among other aspects, allowed us to test our hypotheses whilst controlling for several key variables (e.g., topic, task, planning, time).

As for the reference corpora, this study used data representing British English and a limited range of registers. Our research could be replicated on corpus data representing other varieties and registers, specifically considering the effect of contextual and functional features of language use. The relatively small size of the reference corpora in this study may also be a limitation, especially for computing mean-frequency scores. However, whilst the Spoken BNC2014 only consists of 11 million words, it is currently the largest corpus of spontaneous spoken English available to the research community. Follow-up research could consider using subtitle corpora (e.g., SUBTLEX) as reference corpora for spoken English,
though, these too bring about their own biases (see, e.g., Van Halteren, 2020 on topic bias). Finally, the different size and composition of the six reference corpora used in this study may have affected the results. Follow-up replication studies could compile more comparable reference corpora.

## 7. Conclusions

This study aims to contribute to cumulative knowledge building in research on lexical sophistication. It has methodological implications for (learner) corpus research and practical applications in language testing. Our first hypothesis focused on the correlation between mean-frequency scores of content words (CW) and proficiency across spoken and written production. Our findings only partially matched our predictions. Based on previous studies, we hypothesised that the more proficient the language users, (a) the more frequent their content words in speech and (b) the less frequent their content words in writing. Our results showed this to be true in the Spoken Monologues, but not in the Written Essays of the ICNALE. We suggested that this was likely due to the nature of the essay task. Overall, our findings show that proficient learners tend to use vocabulary that is typical of the register represented in the reference corpus. Based on this, we suggest a conceptual and terminological shift from *lexical sophistication* to *register appropriateness* following Durrant and Brenchley (2019) to refer to the construct captured by mean-frequency measures of vocabulary.

Second, we hypothesised that the choice of the reference corpus could be responsible for some of the contrasting results reported in previous research on lexical sophistication in L2 English. Our findings confirm our hypothesis demonstrating that the use of a reference corpus that does not match the learner production register and mode, or combines different language modes, impacts CW scores. This raises concerns about the validity and reliability of mean-frequency measures. It also makes it difficult to replicate previous research and compare the results of studies that rely on different reference corpora. Follow-up research could aim to design and validate a diagnostic procedure to evaluate the language represented in the data to be analysed — considering, among others, register-related features due to, for instance, task instructions and communicative context (e.g., speaking over the telephone). A similar procedure could be envisaged for the selection of a suitable reference corpus (i.e., one that best represents the mode and register of the texts to be analysed) to compute mean-frequency scores.
Open data badge and open code badge

This article has been awarded an open data badge and an open code badge. All
data and code used in this article are publicly accessible via ORS at https://osf.io
/mne6w/

Learn more about the Open Practices badges from the *Center for Open Science*: https://osf.io/tvyxz/wiki

Funding

Open Access publication of this article was funded through a Transformative Agreement with Lancaster University.

References

<figure><img src="image_10.png" /></figure>

Balota, D.A., Yap, M. J., Hutchison, K.A., Cortese, M. J., Kessler, B., Loftis, B., Neely, J. H., Nelson, D. L., Simpson, G. B., & Treiman, R. (2007). The English lexicon project. *Behavior Research Methods, 39*(3), 445-459.

<figure><img src="image_11.png" /></figure>

Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. *Journal of Statistical Software*, 67(1), 1-48.

Bell, H. M. (2003). Using frequency lists to assess L2 texts [PhD thesis, University of Wales]. https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.636075

<figure><img src="image_12.png" /></figure>

Berger, C., Crossley, S., & Skalicky, S. (2019). Using lexical features to investigate second language lexical decision performance. *Studies in Second Language Acquisition, 41*(5), 911-935.

Biber, D. (2023). Writing and speaking. In R. Horowitz (Ed.), *The Routledge International Handbook of Research on Writing* (2nd ed., pp. 535–548). Routledge.

<figure><img src="image_13.png" /></figure>

Biber, D., Johansson, S., Leech, G. N., Conrad, S., & Finegan, E. (2021). *Grammar of spoken and written English*. John Benjamins.

BNC Consortium. (2007). *The British National Corpus (XML) [Data set]*. Oxford Text Archive. http://www.natcorp.ox.ac.uk/corpus/

Bottini, R. (2022). Lexical complexity in L2 English speech: Evidence from the Trinity Lancaster Corpus [PhD thesis]. Lancaster University.

<figure><img src="image_14.png" /></figure>

Brezina, V., Hawtin, A., & McEnery, T. (2021). The written British National Corpus 2014 – design and comparability. *Text & Talk, 41*(5–6), 595–615.

<figure><img src="image_15.png" /></figure>

Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. *Behavior Research Methods, 41*(4), 977-990.
Council of Europe (Ed.). (2020). *Common European framework of reference for languages: Learning, teaching, assessment*. Companion volume. Council of Europe Publishing.
https://rm.coe.int/common-european-framework-of-reference-for-languages-learning-teaching/16809ea0d4

doi Crossley, S., Cobb, T., & McNamara, D. (2013). Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. *System*, *41*(4), 965–981.

doi Crossley, S., Kyle, K., & Römer, U. (2019). Examining lexical and cohesion differences in discipline-specific writing using multi-dimensional analysis. In T. B. Sardinha & M. V. Pinto (Eds.), *Multi-dimensional analysis: Research methods and current issues* (pp. 189–216). Bloomsbury Academic.

doi Crossley, S., & McNamara, D. (2012). Predicting second language writing proficiency: the roles of cohesion and linguistic sophistication. *Journal of Research in Reading*, *35*(2), 115–135.

doi Crossley, S., & McNamara, D. (2013). Applications of text analysis tools for spoken response grading. *Language Learning & Technology*, *17*(2), 171–192.

doi Crossley, S., Salsbury, T., & McNamara, D. (2010). The development of polysemy and frequency use in English second language speakers: Polysemy and frequency use in English L2 speakers. *Language Learning*, *60*(3), 573–605.

doi Crossley, S., Salsbury, T., & McNamara, D. (2012). Predicting the proficiency level of language learners using lexical indices. *Language Testing*, *29*(2), 243–263.

doi Crossley, S., Salsbury, T., McNamara, D., & Jarvis, S. (2011). What is lexical proficiency? Some answers from computational models of speech data. *TESOL Quarterly*, *45*(1), 182–193.

doi Dawson, N., Hsiao, Y., Tan, A., Banerji, N., & Nation, K. (2021). *Features of lexical richness in children's books: Comparisons with child-directed speech*.

doi Dombi, J., Sydorenko, T., & Timpe-Laughlin, V. (2022). Common ground, cooperation, and recipient design in human-computer interactions. *Journal of Pragmatics*, *193*, 4–20.

doi Durrant, P. (2014). Corpus frequency and second language learners' knowledge of collocations: A meta-analysis. *International Journal of Corpus Linguistics*, *19*(4), 443–477.

doi Durrant, P., & Brenchley, M. (2019). Development of vocabulary sophistication across genres in English children's writing. *Reading and Writing*, *32*(8), 1927–1953.

doi Durrant, P., & Durrant, A. (2022). Appropriateness as an aspect of lexical richness: What do quantitative measures tell us about children's writing? *Assessing Writing*, *51*, 100596.

doi Durrant, P., Moxley, J., & McCallum, L. (2019). Vocabulary sophistication in First-Year Composition assignments. *International Journal of Corpus Linguistics*, *24*(1), 33–66.

doi Egbert, J. (2017). Corpus linguistics and language testing: Navigating uncharted waters. *Language Testing*, *34*(4), 555–564.

doi Eguchi, M., & Kyle, K. (2020). Continuing to explore the multidimensional nature of lexical sophistication: The case of oral proficiency interviews. *The Modern Language Journal*, *104*(2), 381–400.

doi Ellis, N. C. (2002). Frequency effects in language processing. *Studies in Second Language Acquisition*, *24*(2), 143–188.

doi Ellis, N. C., Simpson-Vlach, R., & Maynard, C. (2008). Formulaic language in native and second language speakers: Psycholinguistics, corpus linguistics, and TESOL. *TESOL Quarterly*, *42*(3), 375–396.
doi Fergadiotis, G., Wright, H. H., & West, T. M. (2013). Measuring lexical diversity in narrative discourse of people with aphasia. *American Journal of Speech-Language Pathology / American Speech-Language-Hearing Association*, 22(2).

doi Gablasova, D., & Bottini, R. (2022). Spoken learner corpora for language teaching. In R. Jablonkai & E. Csomay (Eds.), *The Routledge handbook of corpora and English language teaching and learning* (pp. 296–310). Routledge.

Gablasova, D., Harding, L., Brezina, V., & Dunlea, J. (2023, July). *Talking to an imagined interlocutor: Interactional and interpersonal features of discourse in computer - mediated semi-direct speaking assessment*. Corpus Linguistics 2023 Conference, Lancaster University (UK).

doi Gries, S. Th. (2015). The most under-used statistical method in corpus linguistics: multi-level (and mixed-effects) models. *Corpora*, 10(1), 95–125.

doi Horst, M., & Collins, L. (2006). From faible to strong: How does their vocabulary grow? *The Canadian Modern Language Review*, 63(1), 83–106.

doi Ishikawa, S. (2023). *The ICNALE guide: An introduction to a learner corpus study on Asian learners' L2 English*. Routledge.

doi Kim, M., Crossley, S., & Kyle, K. (2018). Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. *The Modern Language Journal*, 102(1), 120–141.

doi Kormos, J. (2011). Task complexity and linguistic and discourse features of narrative writing performance. *Journal of Second Language Writing*, 20(2), 148–161.

doi Kyle, K., & Crossley, S. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. *TESOL Quarterly*, 49(4), 757–786.

doi Kyle, K., & Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. *Journal of Second Language Writing*, 34, 12–24.

doi Kyle, K., Crossley, S., & Berger, C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): version 2.0. *Behavior Research Methods*, 50, 1030–1046.

doi Le Foll, E. (2021). Register variation in school EFL textbooks. *Register Studies*, 3(2), 207–246.

doi Le Foll, E. (2022a). *Making tea and mistakes: The functions of make in spoken English and textbook dialogues*. In Z. Yin & E. Vine (Eds.), *Multifunctionality in English: Corpora, language and academic literacy pedagogy* (pp. 157–178). Routledge.

doi Le Foll, E. (2022b). Textbook English: A corpus-based analysis of the language of EFL textbooks used in secondary schools in France, Germany and Spain [PhD thesis, Osnabrück University].

doi Leńko-Szymańska, A. (2019). *Defining and assessing lexical proficiency*. Routledge.

doi Love, R., Dembry, C., Hardie, A., Brezina, V., & McEnery, T. (2017). The spoken BNC2014. *International Journal of Corpus Linguistics*, 22(3), 319–344.

doi Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners' oral narratives. *The Modern Language Journal*, 96(2), 190–208.

doi McNamara, D., Crossley, S., & McCarthy, P. (2010). Linguistic features of writing quality. *Written Communication*, 27(1), 57–86.

doi Monteiro, K., Crossley, S., & Kyle, K. (2020). In search of new benchmarks: Using L2 lexical frequency and contextual diversity indices to assess second language writing. *Applied Linguistics*, 41(2), 280–300.
doi Nakatsuhara, F., Khabbazbashi, N., & Inoue, C. (2021). Assessing speaking. In G. Fulcher & L. Harding (Eds.), *The Routledge handbook of language testing* (pp. 209–222). Routledge.

doi Nation, I. S. P. (2001). *Learning vocabulary in another language*. Cambridge University Press.

Nesi, H. (2001). A corpus-based analysis of academic lectures across disciplines. In J. Cotterill & A. Ife (Eds.), *Language across boundaries* (pp. 201–218). British Association for Applied Linguistics in association with Continuum Press.

doi Nesi, H., & Gardner, S. (2012). *Genres across the disciplines: Student writing in higher education*. Cambridge University Press.

doi Ockey, G. J., & Chukharev-Hudilainen, E. (2021). Human versus computer partner in the paired oral discussion test. *Applied Linguistics, 42*(5), 924–944.

doi OED. (2023). sophistication, n. | sophisticated, adj. In *Oxford English Dictionary*. Oxford University Press; Oxford English Dictionary.

doi Ong, J., & Zhang, L. J. (2010). Effects of task complexity on the fluency and lexical complexity in EFL students' argumentative writing. *Journal of Second Language Writing, 19*(4), 218–233.

doi Pallotti, G. (2020). Measuring complexity, accuracy, and fluency (CAF). In P. Winke & T. Brunfaut (Eds.), *The Routledge handbook of Second Language Acquisition and language testing* (pp. 201–210). Routledge.

doi Paquot, M. (2019). The phraseological dimension in interlanguage complexity research. *Second Language Research, 35*(1), 121–145.

doi Paquot, M., Gablasova, D., Brezina, V., & Naets, H. (2022). Phraseological complexity in EFL learners' spoken production across proficiency levels. In S. Götz & A. Leńko-Szymańska (Eds.), *Complexity, accuracy and fluency in learner corpus research* (pp. 115–136). John Benjamins.

doi Read, J. (2000). *Assessing vocabulary*. Cambridge University Press.

doi Saito, K., Suzuki, S., Oyama, T., & Akiyama, Y. (2021). How does longitudinal interaction promote second language speech learning? Roles of learner experience and proficiency levels. *Second Language Research, 37*(4), 547–571.

doi Salsbury, T., Crossley, S., & McNamara, D. (2011). Psycholinguistic word information in second language oral discourse. *Second Language Research, 27*(3), 343–360.

doi Tidball, F., & Treffers-Daller, J. (2008). Analysing lexical richness in French learner language: What frequency lists and teacher judgements can tell us about basic and advanced words. *Journal of French Language Studies, 18*(3), 299–313.

doi Van Halteren, H. (2020). Domain bias in distinguishing Flemish and Dutch subtitles. *Natural Language Engineering, 26*(5), 493–510.
Address for correspondence

Raffaella Bottini
Lancaster University
Department of Linguistics and English Language
County South, Office C54
LA14YL, Lancaster
United Kingdom
r.bottini@lancaster.ac.uk
https://orcid.org/0000-0003-2142-2184

<figure><img src="image_16.png" /></figure>

Co-author information

Elen Le Foll
University of Cologne
Department of Romance Studies
elefoll@uni-koeln.de

Publication history

Date received: 8 September 2023
Date accepted: 16 July 2024
Published online: 21 October 2024