How Good are Humans at Native Language Identification? A Case Study on Italian L2 writings
Authors: Elisa Di Nuovo, Elisa Corino, Cristina Bosco (Università di Torino, 2020)
Topic: Learner Corpora and Language Acquisition

Abstract
In this paper we present a pilot study on human performance for the Native Language Identification task. We performed two tests aimed at exploring the human baseline for the task in which test takers had to identify the writers' L1 relying only on scripts written in Italian by English, French, German and Spanish native speakers. Then, we conducted an error analysis considering the language background of both test takers and text writers.

Introduction
Native Language Identification (NLI) is a task usually performed by machines consisting in identifying the mother tongue (henceforth L1) of a person based only on their writings in another language (e.g. L2 or L3). To date, the majority of the existing studies have focused on English as L2, that is English used by people who are acquiring English as a second or foreign language (Tomokiyo and Jones, 2001; Koppel et al., 2005; Malmasi et al., 2015; Kulmizev et al., 2017; Markov et al., 2017; Cimino and Dell’Orletta, 2017, among others). Three editions of the NLI shared task had been organized (Tetreault et al., 2013; Schuller et al., 2016; Malmasi et al., 2017) in which systems had to correctly identify the L1 among 11 L1s. The basic assumption of this task is that when we learn a new language (henceforth Target Language, TL), our L1 interferes within the learning process introducing in the TL productions clues that can be automatically detected. Nevertheless, another issue to be investigated within this task is the interference in the TL learner’s productions also of other languages previously learned as L2 or L3. In fact, L1 may not be the only language playing a role in the acquisition of a TL, since “bi- or multilingualism is as frequent in the population of the world as pure monolingualism, perhaps even more frequent” (Hammarberg, 2001, p. 21). This issue is especially relevant performing the NLI task in languages other than English. For instance, when someone learns Italian, it is likely their L3, since English is the language taught worldwide as L2 (with more that 100 million learners, as stated in the British Council annual report 2018-19). In this paper, we investigate the human performance for NLI applied on productions of learners Italian, thus focusing not only on the issues related to second language acquisition (Ellis, 2015; Slabakova, 2016), but also to third language acquisition (Cenoz et al., 2001; Picoral, 2020). We asked human Test Takers (TTs) to perform a simplified NLI task on learner Italian scripts extracted from VALICO (Corino and Marello, 2017), selecting only four L1s —i.e. English (EN), French (FR), German (DE) and Spanish (ES). This simplified task will be challenging since all the selected languages are Indo-European languages sharing typological similarities. Moreover, we performed an error analysis of the test results considering learners’ L1 and L2(s) and learning stage (i.e. year of study), in addition to text features and TTs’ language background. Test results could be useful for the improvement of the error annotation scheme introduced in Di Nuovo et al. (2019). Our research questions therefore are: 1. How good are humans in performing the NLI task? 2. What features share the most problematic texts? We try to answer to these questions in this paper organized as follows: in Section 2 we briefly describe previous work on the subject; in Section 3 we describe the tests performed and discuss the results; in Section 3.2 we conduct the error analysis; and in Section 4 we conclude the paper.

Conclusion
In this paper we described two human NLI tests for Italian. Although it was a simplified NLI task, tailored bearing in mind human skills, it proved to be a difficult task even for experts. The error analysis showed that ES was the easiest L1 to identify—correctly identified 60% of the time—while DE the hardest. L2 transfer was misleading, even when L1 clues were present. TTs knowledge of the involved L1s proved not to be a discriminant factor. It would have been interesting to ask the TTs to point out the clues that supported their answers to be less hypothetical in the discussion, especially when dealing with texts featured by L1 and L2 transfer. For this reason, we asked our TTs to take part in another test based on the same texts in which they have to highlight the clues. At the moment, we are collecting the answers. In the future, we will test a machine learning system on the same texts to compare its results with those of our TTs.