# Exploring learner knowledge with Large Language Models fine-tuned with the EFCAMDAT

Nicolas Ballier, Bernardo Stearns

► To cite this version:

Nicolas Ballier, Bernardo Stearns. Exploring learner knowledge with Large Language Models fine-tuned with the EFCAMDAT. LCR2024 Learner Corpus Research conference, University of Tartu; Learner Corpus Association, Sep 2024, Tartu (Estonie), Estonia. hal-04878135

HAL Id: hal-04878135

https://hal.science/hal-04878135v1

Submitted on 9 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Exploring learner knowledge with Large Language Models fine-tuned with the EFCAMDAT

Nicolas Ballier (Université Paris Cité / CLILLAC-ARP & LLF) and Bernardo Stearns (University of Galway)

Large Language Models (LLMs) like BERT (Bidirectional Encoder Representations from Transformers) have been trained to predict a masked token in a sentence (a bit like a cloze test in English). These LLMs have proved to be successfully retrained (fine-tuned) for specific tasks, such as question answering (cf. ChatGPT). Our paper explores the opportunities for learner corpus research of retraining an LLM like BERT (Devlin et al, 2016) with learner data to explore learner lexical knowledge.

We fine-tuned BERT models with samples from the EFCAMDAT (Geertzen et al, 2013) to investigate how these models may simulate learner behaviours of a given L1 or CEFR level. As an exploratory validation procedure, we used a subset of the FCE dataset (Yannakoudakis et al. 2011) that has been annotated for errors to evaluate the ability of our retrained model to predict the token produced by the learners. Only 57.5% of the erroneous tokens according to the FCE annotators were predicted by our model with the first prediction, but 82,2% of the tokens were correctly predicted among the top 10 predictions.

We explain why analyzing the probabilities (prob in Figure 1) contribute to computational learner modeling (Abyaa et al, 2019). Examining the first ten predictions that we extracted from our fine-tuned model can be used as a window on lexical and grammatical knowledge, a virtual exploration of what a learner of the same level could have said. We highlight some of the challenges raised by this new way to revisit paradigmatic knowledge.

<figure><img src="image_2.png" /><figcaption>Figure 1: The first 10 predictions of the native (uncased) BERT model for the masked token (learner choice : taken) in the sentence *I was shoked because I had alredy spoken with them and I had [MASK] two autographs*.</figcaption></figure>

## References

Abyaa, A., Khalidi Idrissi, M., & Bennani, S. (2019). Learner modelling: systematic review of the literature from the last 5 years. *Educational Technology Research and Development*, 67, 1105-1143.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. *arXiv preprint arXiv:1810.04805*.

Geertzen, J., Alexopoulou, T., & Korhonen, A. (2013, October). Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCAMDAT). In *Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project* (pp. 240-254).

Housen, A., & Kuiken, F. (2009). Complexity, accuracy, and fluency in second language acquisition. *Applied linguistics*, 30(4), 461-473.

Khetan, V., Ramnani, R., Anand, M., Sengupta, S., & Fano, A. E. (2022). Causal BERT: Language models for causality detection between events expressed in text. In *Intelligent Computing: Proceedings of the 2021 Computing Conference, Volume 1* (pp. 965-980). Springer International Publishing.

Yannakoudakis, H., Briscoe, T., & Medlock, B. (2011). A New Dataset and Method for Automatically Grading ESOL Texts. In D. Lin, Y. Matsumoto, & R. Mihalcea (Éds.), *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies* (p. 180-189). Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=2002472.2002496