Evaluating Large Language Models for Native Language Identification
Authors: Dhiman Goswami, Kai North, Shervin Malmasi, Antonios Anastasopoulos, Marcos Zampieri


Reviewer paper summaries: The paper presents a comprehensive benchmark study comparing traditional statistical machine learning models, BERT-based transformer models, and large language models (LLMs) for the task of Native Language Identification (NLI) based on second language (L2) text production. Utilizing datasets such as ICNALE, FCE, and TOEFL11, the study finds that fine-tuning GPT-4 achieves the highest performance across all datasets, significantly outperforming other models. The paper highlights the effectiveness of LLMs in identifying linguistic patterns, syntax variations, cultural references, grammatical errors, and spelling mistakes indicative of the writer's native language (L1). It also notes that while zero-shot prompting of LLMs shows variable performance, fine-tuning consistently yields superior results. The study underscores the potential of LLMs for NLI tasks and calls for further exploration with more diverse datasets and additional L2 contexts.
Summary of strengths: 1. Comprehensive benchmarking of different models for NLI.1. 
2. In-depth analysis of L1 markers, providing valuable linguistic insights.
Summary of weaknesses: 1. The study focuses primarily on English as the L2, limiting its applicability to other L2 contexts.
2. Limited diversity in datasets, primarily relying on well-known corpora, which may not fully capture global linguistic variations.
Comments, suggestions and typos: Question
1. How is the impact of different types of L2 texts (e.g., informal writing, social media posts) on NLI performance.
2. Did you investigate the scalability and practical deployment of fine-tuned LLMs like Llama2 or Llama3.
3. Could you provide examples of common errors or challenges faced during the annotation process?
Overall assessment scores: 2.5