Venue: NAACL 2024

# Native Language Identification in Texts: A Survey

Dhiman Goswami1, Sharanya Thilagan1, Kai North1, Shervin Malmasi2, Marcos Zampieri1

###### Abstract

We present the first comprehensive survey of Native Language Identification (NLI) applied to texts. NLI is the task of automatically identifying an author's native language (L1) based on their second language (L2) production. NLI is an important task with practical applications in second language teaching and NLP. The task has been widely studied for both text and speech, particularly for L2 English due to the availability of suitable corpora. Speech-based NLI relies heavily on accent modeled by pronunciation patterns and prosodic cues while text-based NLI relies primarily on modeling spelling errors and grammatical patterns that reveal properties of an individuals' L1 influencing L2 production. We survey over one hundred papers on the topic including the papers associated with the NLI and INLI shared tasks. We describe several text representations and computational techniques used in text-based NLI. Finally, we present a comprehensive account of publicly available datasets used for the task thus far.

languageresource

100 papers on the topic published primarily at the ACL Anthology, but also at other repositories such as the ACM Digital Library. As evidenced in this survey, most work in NLI has relied on feature engineering and traditional machine learning (ML) classifiers (e.g., SVMs) Jarvis et al. (2013); Malmasi and Dras (2014); Vajjala and Banerjee (2017). Recent advances in deep learning, and the introduction of Large Language Models (LLMs) have brought renewed interest in NLP, opening exciting new avenues for new and existing tasks. We believe that by reviewing prior work, this survey can identify knowledge gaps and open research questions that can help shape future research.

The remainder of this paper is organized as follows: Section 2 presents an overview of text-based NLI. Section 3 provides a brief description of available NLI datasets. Section 4 discusses the two most well-known NLI shared tasks. Section 5 describes features used in NLI while Section 6 discusses the different models that have been used to approach NLI. Lastly, Section 7 presents concluding remarks and discusses avenues for future work.

## 2 NLI Task Overview

Text-based NLI is typically modeled as a supervised multi-class classification task evaluated using widely-used metrics in text classification such as accuracy, precision, recall, and F-score Malmasi (2016). Given a collection of texts in a particular L2 written by a set of speakers from \(n\) different L1s, NLI systems are trained to assign a class label (L1) to each document in the text collection. Figure 1 illustrates the NLI task with an English L2 and four different L1s.

As discussed in this survey, NLI systems rely on a variety of features such as words, characters, morphosyntactic information, dependency relations, and spelling errors. While word-based features (e.g., word n-grams) have shown to deliver high performance in most text classification tasks, _topic bias_ is a significant confounder. Topic bias occurs when thematic references in a text reveal the true L1 label, for example, English texts written by L1 speakers of Japanese are more likely to talk about Tokyo or Kyoto than texts from Chinese or Italian L1 speakers. Therefore, to decrease topic influence, several NLI studies have incorporated linguistically-motivated features that can capture morphological and syntactic variation between different L1s Malmasi and Dras (2014); Malmasi et al. (2018); Malmasi et al. (2019); del Rio (2020).

## 3 Datasets

We present a comprehensive account of all available datasets created for and/or used in NLI research to date. The list is presented in Table 1 and summarized below.

ArabicThe Arabic Learner Corpus (ALC) includes 329 in-class essays written by speakers of seven different L1's who studied Arabic as L2 in Saudi Arabia.

ChineseThe Chinese Learner Corpus (CLC) features 3,216 essays of 600 token on an average and written in Chinese by university students from 11 countries, representing diverse proficiency levels.

CzechCzeSL-SGT is a subset of the publicly available Czech as a Second Language with Spelling, Grammar, and Tags corpus, comprising Czech 8,617 essays written by 54 different L1s. The used dataset includes 47% Non-Indo European, 29% Slavic and 24% Indo European L1's after excluding texts of unknown language group.

EnglishThis is the most well-studied L2 in NLI. There are ten datasets presented in Table 1. The most popular is the TOEFL11 dataset that contains 12,100 essays of eight different prompts and three score levels written by speakers of eleven L1s.

FinnishThe Corpus of Advanced Learner Finnish (LAS2) comprises 204 Finnish L2 writings from speakers seven different L1s collected as part of a project at the University of Turku.

GermanThe FALKO (fehlerannotierien Lernerkorpus) corpus is the largest publicly available selection of German learner texts. It contains 221 essays and text summaries written by eight different non-native speakers of German.

Figure 1: The general concept of an NLI system as depicted by Malmasi (2016).

 



**Italian** VALICO (Varieta di Apprendimento della Lingua Italiana Corpus Online) includes approximately one million tokens of learner Italian writing from a wide range of L1s along with the associated metadata. A subset containing 2,531 texts from 14 different L1s is used for NLI.

NorwegianThe ASK Corpus is a learner corpus composed of the 2158 essays of learners of Norwegian who are native speaker of 10 different L1s. These texts are essays written as part of a test of Norwegian as a second language.

PortugueseNLI-PT was collected from three different learner corpora of Portuguese: (i) COPLE2; (ii) Leiria corpus, and (iii) PEAPL2, containing 148 topics including written exercises on lessons, official Portuguese proficiency test and different stimuli from learners of Portuguese with different proficiency levels and fifteen L1s.

RussianThe Russian Learner Corpus (RLC) is a collection of 7831 - both academic and non-academic texts produced by speakers of fifteen L1s.

SpanishThe Anglia Ruskin University (ARU) Spanish learner corpus contains 206 texts written by speakers of six L1s who are mainly undergraduate students of the university and some ERASMUS students. The texts that were produced by students either as course work or as part of exams.

TurkishThe Turkish Learner Corpus (TLC) covers 284 essays written by L1 speakers of Afghan, Albanian, Arabic, Azeri Turkish and Farsi.

\begin{table}
\begin{tabular}{l l l l c} \hline
* & 54 different L1’s (5 most frequent languages
- Chinese, Russian, SGT & Ukrainian, Korean, English
- 3,715 instances) & 8,617 & [Tydilitatova, 2016] \\ \hline  & CLC & Catalan, Chinese, Dutch, French, German, Greek, Italian, Japanese, FCE & 1,244 & [Malmasi, 2016] \\  & Estival & English, Spanish, Arabic & 9,836 & [Estival et al., 2007] \\  & ESL & Chinese, Czech, Italian, Russian, Spanish & 4,185 & [Rozovskaya and Roth, 2011] \\  & ICLE & Bulgarian, Chinese, Czech, Dutch, Finnish, French, German, Italian, v2.0, Japanese, Norwegian, Polish, Russia, Spanish, Swedish, Turkish, & 6,085 & [Granger et al., 2009] \\  & INLI & Bengali, Hindi, Kannada, Malayalam, Tamil, Telugu & 10,647 & [Anand Kumar et al., 2017] \\  & 2017 & ELNL (US, UK, Australia etc.), ESL (Hong Kong, Pakistan), EFL & 4,428 & [Ishikawa, 2011] \\ English & ICNALE & ENL (US, UK, Australia etc.), ESL (Hong Kong, Pakistan), EFL & 4,428 & [Ishikawa, 2011] \\  & Lang-8 & 65 different native languages included, with 14 of those languages having 1000 entries (Japanese, Chinese (Mandarin and Cantonese), Korean, Russian, Spanish, French, German, Polish, Italian, Vietnam) & 154,702 & [Brooke and Hirst, 2011] \\ OGI-TS & English, Farsi, French, German, Hindi, Korean, Japanese, Mandarin, Spanish, Tamil, Vietnamese & 1,236 & [Zissman, 1996] \\ Reddit
- & English, German, Dutch, French, Polish, Romanian, Finish, & 200 million & [Rabinovich et al., 2018] \\ L2 & Swedish, Spanish, Greek, Portugues, Estonian, Czech, Italian, Russia, Turkish, Bulgarian, Croatian, Norwegian, Hungarian, Lithuanian, Slovenian and Serbian & 12,100 & [Blanchard et al., 2013] \\  & TOEEL11 & Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, Turkish & 204 & [Malmasi and Dras, 2014c] \\ \hline Finnish & LAS2 & Russian, Japanese, Lithuanian, Czech, German, Hungarian, Polish, Komi, English & 204 & [Malmasi and Dras, 2014a] \\ \hline German & FALKO & Chinese, Danish, English, French, Polish, Russian, Turkish, Uzbek & 221 & [Malmasi and Dras, 2017a] \\ Italian & VALLCO & Albanian, Chinese, Czech, English, French, German, Hindi, Japanese, Polisl, Portugal, Romanian, Russian, Serbian, Spanish & 2,531 & [Malmasi and Dras, 2017a] \\ \cline{2-5}  & Norwegian ASR & German, Dutch, English, Spanish, Russian, Polish, Bosnian-Croatian-Serbian, Albanian, Vietnamese, Somal & 2,158 & [Malmasi et al., 2015a] \\ \cline{2-5}  & Portuguese NL1-PT & Chinese, English, Spanish, German, Russian, French, Japanese, Italian, Dutch, Tetum, Arabie, Polish, Korean, Romanian, Swedish & 1,868 & [del Rio et al., 2018] \\ \hline Russian & RLC & Chinese, Danish, English, Estonian, Finnish, French, Deutsch, Italian, Japanese, Kazakh, Korean, Norwegian, Serbian, Swedish, Thai & 7,831 & [Remnev, 2019] \\ \hline Spanish & ARU & English, French, German, Greek, Italian, Japanese & 206 & [Malmasi and Dras, 2017a] \\ \hline Turkish & TLC & Arabic, Albanian, Azeri Turkish, Farsi, Afghanistan & 284 & [Uluslu, 2023] \\ \hline \end{tabular}
\end{table}
Table 1: A list of LI datasets with L2, L1s, number of instances, and a reference to the paper describing the resource.


Shared Tasks

Shared tasks are competitions in which multiple participating teams develop systems to address a task using the same benchmark dataset typically provided by the shared task organizers. The goal is to foster research on a topic while encouraging wide collaboration (Escartin et al., 2017).

Shared tasks provide important benchmarks and datasets to the community. Several related shared tasks on language and dialect identification have been organized (Zubiaga et al., 2016; Gaman et al., 2020; Aepli et al., 2022), but only two shared tasks on text-based NLI have been organized thus far. The two editions of the NLI shared tasks co-located with the workshop on Innovative Use of NLP for Building Educational Applications (BEA) (Tetreault et al., 2013; Malmasi et al., 2017b) included English texts written by speakers of 11 L1s. The Indian Native Language Identification (INLI) shared task at the Forum for Information Retrieval Evaluation (FIRE) 2017 (Anand Kumar et al., 2017) and 2018 (Kumar et al., 2018) featured an English dataset with comments from social media written by speakers of multiple L1s spoken in India such as Bengali, Hindi, Kannada, Malayalam, Tamil, and Telugu. We describe the two editions of each of these shared tasks in detail in this section.

**Native Language Identification 2013 (NLI-2013)** NLI-2013 (Tetreault et al., 2013) comprised three sub-tasks using the TOEFL11 dataset containing English texts written by speakers of 11 L1s: Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish. The submitted systems were evaluated on the TOEFL11 test set while training data varied according to the sub-task. In (1) closed-training, participants could use only the TOEFL11 training set and no additional external data; in (2) open-training-1: participants could use any external dataset for training but not TOEFL11; finally in (3) open-training-2 participants could used the TOEFL11 training set in combination with any other dataset. Each participant could submit up to five systems for each of the three sub-tasks. This allowed teams to experiment with different variations of their core system. The best performing teams approached the task using classic ML systems such as SVMs, Logistic Regression, String Kernels, Local Rank Distance, and Maximum Entropy. The top systems were trained on features such as word and Part-of-Speech (POS) n-grams (Goutte et al., 2013; Gebre et al., 2013; Tsvetkov et al., 2013; Jarvis et al., 2013).

**Native Language Identification 2017 (NLI-2017)** NLI-2017 (Malmasi et al., 2017b) featured text and speech. The organizers provided participants with a corpus including written essays and transcriptions of spoken responses from TOEFL proficiency test takers. The same 11 L1s as NLI 2013 were included in NLI 2017. The task featured three tracks: essay-only, speech-only, and fusion. In the essay-only track, participants predicted the candidate's L1 based only on essays. In the speech-only track, participants predicted the L1 based on a transcription of a 45-second spoken response. The raw audio for the spoken responses was not distributed, but i-vectors, low-dimensional representations of acoustic measurements (Dehak et al., 2011; Martinez et al., 2011), were provided. Finally, in the fusion track, participants predicted the L1 using a combination of written essays and spoken responses. Both open and closed competitions were held, allowing participants to use additional NLI training data in the open competition but limiting the use of external data in the closed competition. Systems featuring ensemble of multiple traditional ML classifiers (e.g. SVMs) trained on lexical and syntatic features were the most effective approaches in all tracks (Goutte and Leger, 2017; Li and Zou, 2017).

**Indian Native Language Identification 2017 (Inli-2017)** INLI-2017 (Anand Kumar et al., 2017) provided participants with a corpus containing over 10,600 Facebook comments in English written by native speakers of Bengali, Hindi, Kananda, Malayam, Tamil, and Telugu. The comments were retrieved from local newspaper pages. Instances containing code-mixing, a wide-spread linguistic phenomenon in the region, have been excluded. Most high-performing approaches in this competition used TF-IDF vectors with SVMs (Kosmajac and Keselj, 2017; Lakshmi and Shambhavi, 2017). Deep learning approaches have not obtained competitive performance in INLI-2017 displaying inferior performance compared to the SVM baseline system released by the organizers.

**Indian Native Language Identification 2018 (INLI-2018)** INLI-2018 (Kumar et al., 2018) was the second iteration of the INLI shared task. It features English Facebook comments written by speakers of the same six L1s included in the previous edition. The organizers provided participants with the same training data as the 2017 edition and 

two test sets. Test set 1 was the same test set from the 2017 edition while test set 2 was a novel test set compiled one year after the training data. The best performing systems in NLI-2018 used TF-IF vectors as features along with neural networks and ensemble approaches combining predictions from both traditional ML and deep learning classifiers (Ajees and Idicula, 2018; Nayel and Shashirekha, 2018; Markov and Sidorov, 2018). The use of neural network architectures with word embeddings has not been explored in this shared task.

## 5 Text Representation and Features

Most approaches to NLI have relied on statistical ML classifiers. These classifiers take a variety of features as input such as characters and, most commonly, words extracted from the training corpora often as part of n-gram models. In addition to character and word-based features, POS tags are widely used in NLI along with several additional lexical and syntactic features. These different feature classes have been shown to provide complementary information (Malmasi and Cahill, 2015). In the following paragraphs we define each feature and provide a list of studies that use each group of feature in Table 2.

Lexical FeaturesLexical features used in NLI include word frequencies, word type frequencies, spelling errors, number of phonemes and syllables, and more. These features provide insight into the linguistic fingerprint of an individual and can therefore be used to identify their native language. Swan and Smith (2001) and Gebre et al. (2013) used spelling and grammatical errors to predict learners' L1 influences for NLI. del Rio (2020) and Malmasi and Dras (2014c) used frequencies of letters, phonemes, syllables, morphemes and suffixes respectively for Portuguese and Finnish NLI. Wong and Dras (2009) and Malmasi and Dras (2017a) explored the use of function words for NLI, including the use of stopword lists for multilingual NLI.

N-gramsN-grams are a sequence of \(n\) items in texts, most often letters or words, and are subsequently defined at either the character-level or word-level. Wong and Dras (2009) utilized word-level n-grams and character bigrams and trigrams, and the 100 most frequent unigrams found within their dataset. Gebre et al. (2013) also explored word-level unigrams and bigrams both separately and in combination. Tydlitatova (2016) employed character n-grams extracted from individual words for \(n\) = 1, 2, 3 for Czech. N-grams therefore have been frequently used throughout NLI research to identify character or word combinations that are indicative of an individual's native language. N-grams have also been used to model sequences of POS tags as described in this section.

POS tagsPart-of-Speech (POS) tags are used to denote a word's grammatical function, such as noun, verb, adjective, etc. POS tags are assigned at the word-level most commonly by an automatic POS tagger. Several studies have used POS tag frequencies to help differentiate between speakers of various L1s. Malmasi et al. (2015a) used Mixed POS function word n-grams (\(n\) = 1-3) influenced by Wong et al. (2012) for NLI of Norwegian texts. Malmasi et al. (2018) and del Rio (2020) used fine grained POS tags as topic independent features of NLI including verbs, nouns and adjectives for Portuguese. Mechti et al. (2016) and Mechti et al. (2020) used POS n-gram (\(n\) = 1..3) and applied Alkhalil Arabic-specific morphological analyser (Boudtal et al., 2010) to tags word for Arabic NLI.

TF-IDF WeightingWhile the aforementioned features can be represented by binary or normalized feature vectors, Term Frequency-Inverse Document Frequency (TF-IDF) weighting takes into account feature frequency across the entire training corpus, helping to identify rare features that might be highly discriminative. TF-IDF vectors have been frequently used for information retrieval and various NLP tasks such as text categorization and authorship identification. Their ability to characterize a given text along with their predictive capabilities has made their use popular in NLI, often leading to improved performance (Gebre et al., 2013; Zampieri et al., 2017).

Syntactic Features Syntactic transfer from L1 to L2 has been widely studied in various language pairs (Liu et al., 2022). Syntactic features for NLI range from the use of Context Free Grammar (CFG) structures to the frequency of noun or verb phrases within a text. Mechti et al. (2020) explored several syntactic features including the use of the CFG production rule. Wong and Dras (2011) proposed a model treating parse tree horizontal slices as CFG production rule sets, using them as binary features. In addition, several studies have investigated topic-independent features alongside CFG rules to capture relationships useful for NLI, includ 

ing for Chinese (Malmasi and Dras, 2014b), Arabic (Malmasi and Dras, 2014a), Finnish (Malmasi and Dras, 2014c), Norwegian (Malmasi et al., 2015a) and Portuguese NLI (Malmasi et al., 2018). Swanson and Charniak (2012) and Tetreault et al. (2012) alternatively explored the use of Tree Substitution Grammars (TSGs).

## 6 Computational Models

A variety of traditional ML classifiers trained on the features described in Section 5 have been explored in NLI, most notably SVMs and Logistic Regression. The use of multiple base classifiers as part of ensemble or meta-classifiers have also been a popular approach for NLI yielding competitive performance in NLI-2017. Finally, various papers have approached NLI using deep learning architectures. Unlike in other NLP tasks, the performance of these approaches is, however, not always clearly superior to the performance of traditional ML classifiers.

Support Vector MachinesLinear SVM classifiers are the most widely used in NLI due to their high performance and ability to deal with large and

\begin{table}
\begin{tabular}{r c c c c c c l} \hline
* & ✓ &
- & ✓ &
- & (Koppel et al., 2005) \\ ICLE & ✓ & ✓ &
- & (Wong and Dras, 2009) \\ ICLE &
- & ✓ & (Swanson and Charniak, 2012) \\ ICLE &
- & ✓ & (Tetreault et al., 2012) \\ ICLE &
- & ✓ & ✓ &
- & (Wong et al., 2012) \\ TOEFL11 &
- & ✓ &
- & (Wu et al., 2013) \\ TOEFL11 & ✓ & ✓ &
- & ✓ & ✓ &
- & (Lynn, 2013) \\ TOEFL11 &
- & ✓ & (Milmaroton et al., 2013) \\ TOEFL11 & ✓ & ✓ & ✓ & ✓ &
- & ✓ & (Gebre et al., 2013) \\ TOEFL11 & ✓ & ✓ & ✓ &
- & (Jarvis et al., 2013) \\ Multiple & ✓ & ✓ & ✓ &
- & ✓ & ✓ & (Bykh et al., 2013) \\ TOEFL11 & ✓ & ✓ & ✓ &
- & ✓ & (Goutte et al., 2013) \\ TOEFL11 & ✓ & ✓ &
- & (Henderson et al., 2013) \\ TOFEL11 &
- & ✓ &
- & (Tvetkov et al., 2013) \\ ALC &
- & ✓ &
- & ✓ & (Malmasi and Dras, 2014a) \\ CLC &
- & ✓ &
- & ✓ & (Malmasi and Dras, 2014b) \\ LAS2 &
- & ✓ & ✓ &
- & (Malmasi and Dras, 2014c) \\ ASK &
- & ✓ & (Malmasi et al., 2015a) \\ TOEFL11 & ✓ & ✓ & ✓ &
- & ✓ & (Mallasi et al., 2015b) \\ ALC &
- & ✓ & (Mechti et al., 2016) \\ CzeSL-SGT &
- & ✓ &
- & (Tydiliatova, 2016) \\ TOEFL11 &
- & ✓ &
- & ✓ &
- & (Mohammadi et al., 2017) \\ Multiple &
- & ✓ &
- & (Malmasi and Dras, 2017a) \\ TOTEI11 & ✓ & ✓ &
- & ✓ &
- & (Zampieri et al., 2017) \\ INLI &
- & ✓ &
- & (Nayel and Shashirekha, 2017) \\ INLI &
- & ✓ &
- & (Bharathi et al., 2017) \\ Multiple & ✓ & ✓ & ✓ &
- & (Ircing et al., 2017) \\ TOEFL11 & ✓ & ✓ & ✓ &
- & (Vajala and Banerjee, 2017) \\ Multiple & ✓ & ✓ & ✓ &
- & ✓ &
- & (Markov et al., 2017) \\ Multiple &
- & ✓ & ✓ &
- & ✓ & ✓ & (Kulmizer et al., 2017) \\ Multiple & ✓ & ✓ &
- & (Rama and Coltekin, 2017) \\ Multiple & ✓ & ✓ &
- & ✓ & ✓ & (Li and Zou, 2017) \\ Multiple & ✓ & ✓ & ✓ &
- & (Goutte and Leger, 2017) \\ Multiple & ✓ & ✓ &
- & (Oh et al., 2017) \\ Multiple & ✓ &
- & ✓ & ✓ & (Bjerva et al., 2017) \\ INLI &
- & ✓ & ✓ & (Anand Kumar et al., 2017) \\ INLI &
- & ✓ & ✓ &
- & (Jain et al., 2017) \\ INLI & ✓ & ✓ & ✓ &
- & (Markov and Sidorov, 2018) \\ INLI &
- & ✓ & ✓ & ✓ &
- & (Gupta, 2018) \\ INLI &
- & ✓ & ✓ &
- & (Modalal et al., 2018) \\ INLI &
- & ✓ & ✓ & ✓ &
- & (Thennozhi et al., 2018) \\ RLC & ✓ & ✓ &
- & ✓ &
- & (Remev, 2019) \\ NI-PT &
- & ✓ &
- & (del Rio, 2020) \\ ALC &
- & ✓ &
- & ✓ & (Mechti et al., 2020) \\ TLC & ✓ & ✓ & ✓ &
- & ✓ &
- & (Uluslu, 2023) \\ \hline \end{tabular}
\end{table}
Table 2: Primary features used in ML and feature engineering approaches to NLI along with references.
 

sparse feature spaces (Jarvis et al., 2013; Malmasi and Dras, 2014b, 2017a). SVMs have been used in early work (Koppel et al., 2005) and in the highest performing entries of NLI-2013 (Gebre et al., 2013; Jarvis et al., 2013; Bykh et al., 2013; Goutte et al., 2013; Henderson et al., 2013), INLI-2017 (Markov et al., 2017; Kulmizev et al., 2017; Rama and Coltetkin, 2017; Li and Zou, 2017), and INLI-2018 (Mondal et al., 2018; Markov and Sidorov, 2018). SVM implementations available at LIBLINEAR (Fan et al., 2008) and scikit-learn (Pedregosa et al., 2011) have been widely used (Malmasi and Dras, 2014a,c; Malmasi et al., 2015a; Uluslu, 2023; Remnev, 2019; Malmasi and Dras, 2017a). Some studies (Mechti et al., 2016, 2020) have used the SVM implementation available in the LIBSVM package (Chang and Lin, 2011) whereas Tydlitarova (2016) applied SVMs with linear and polynomial kernel functions to NLI.

Logistic RegressionLogistic Regression is another popular model used in NLI. Di Nuovo et al. (2020) performed guided logistic regression among four L1s in an Italian L2 dataset. In NLI-2013 various teams used Logistic Regression (Tsvetkov et al., 2013; Popescu and Ionescu, 2013). In NLI-2017 Ircing et al. (2017) used a Logistic Regression meta-classifier trained on words characters and POS base models and Vajjala and Banerjee (2017) used Logistic Regression model trained on word n-grams (1-3) for essays and transcripts of speech. Finally, Logistic Regression has also been used in INLI-2018 (Gupta, 2018).

Ensembles and Meta-classifiersAn ensemble classifier combines the predictions of multiple individual classifiers, referred to as base classifiers or weak learners, to create more accurate predictive models. This combination is done by aggregating the predictions using methods such as voting. The objective of ensemble classification is to use the diversity among these base classifiers to achieve better prediction than any individual classifier. Researchers have employed ensemble methods to integrate a variety of models and features in NLI (Zampieri et al., 2017; Malmasi et al., 2018). In the NLI shared task 2017, for example, several teams used ensemble methods with competitive performance taking advantage of the multimodal nature of the dataset that contained both text and speech Goutte and Leger (2017); Oh et al. (2017); Bjerva et al. (2017); Ircing et al. (2017); Anand Kumar et al. (2017); Thenmoghi et al. (2017); Jain et al. (2017). Malmasi et al. (2015b) combined the predictions from all NLI Shared Task 2013 systems in a voting ensemble, showing that this approach can achieve very high accuracy results. Similar to voting ensembles, meta-classifiers also leverage the prediction of multiple base classifiers. By training a model that uses the base classifier predictions as inputs, such systems are better able to combine this information by learning patterns in base model predictions and thereby yielding more accurate predictions compared to any individual base classifier. Malmasi and Dras (2017b) presented a thorough examination of meta-classification models for NLI, achieving state-of-the-art results on three datasets from different languages. In another study, Malmasi and Dras (2018) presented a system with 200 meta-classifiers merged in a bagging ensemble, where all models are trained on different subsets of the base classifiers.

Deep LearningDeep learning approaches such as deep neural networks (DNN) (Oh et al., 2017) have also been applied to NLI with various levels of success. Examples of architectures used include Gated Recurrent Unit (GRU) (Shargava et al., 2017), Rectified Linear Unit (ReLU) (Thenmoghi et al., 2017), Convolutional Neural Network (CNN) (Ajees and Idicula, 2018), and Long Short-Term Memory (LSTM) (Mundotiya et al., 2018). These architectures typically take an embedding representation of hundreds of dimensions instead of the n-gram models and linguistically-inspired features described in Section 5. Transformer architectures such as BERT (Devlin et al., 2019) had great impact in NLP (Rogers et al., 2021) paving the way for the latest generation of pre-trained LLMs. However, they have not had the same impact on NLI with only a few studies proposing the use of transformers for the task (Steinbakken and Gamback, 2020; Visu, 2023). The study by Lotfi et al. (2020) showed that the performance of LSTMs and BERT on TOEFL11 and ICLE was much lower than the performance of SVMs and that of GPT-2 as described next.

LLMs and Recent AdvancesRecently proposed LLMs (e.g., GPT, Mistral, Llama-2) have displayed state-of-the-art performance on various NLP tasks (Minaee et al., 2024). One of the first papers to explore LLMs in NLI is the work by Lotfi et al. (2020). They have fine-tuned a GPT-2 model showing performance slightly superior to SVMs on both the TOEFL11 and ICLE corpora. More recently, a 

couple studies (Uluslu and Schneider, 2022; Zhang and Salle, 2023) evaluated the performance of LLMs on the TOEFL11 dataset, including GPT-2, GPT-3.5 (Brown et al., 2020) and GPT-4 (Achiam et al., 2023). Zhang and Salle (2023) demonstrated that GPT-4 achieved state-of-the-art performance of 91.7% accuracy on this dataset using zero-shot prompting without any task-specific fine-tuning, surpassing all previously proposed ensemble approaches. Zhang and Salle (2023) further explored the interpretability of models by prompting GPT-4 to provide explanations for its predictions including linguistic features such as spelling errors, syntactic patterns, and translated phrases as cues. This study highlights the potential of generative models in achieving strong performance in NLI while also enabling explainable predictions. However, as this analysis has been done only on a single English L2 dataset, the impact of LLMs in NLI is still unclear, particularly for L2s other than English where LLMs support is still very limited.

## 7 Conclusion and Future Directions

This paper presented the first comprehensive survey on text-based NLI. We surveyed over 100 papers and discussed features, models, and datasets used for NLI. We collected information about all available datasets created for or used in NLI research to date. Moreover, we also discussed the NLI shared tasks organized in 2013 and 2017 and the INLI shared tasks organized in 2017 and 2018.

This survey shows that apart from a few studies (Lotti et al., 2020; Uluslu and Schneider, 2022; Zhang and Salle, 2023) the bulk of NLI research has focused on approaches that rely on feature engineering and traditional ML classifiers, most notably SVMs. Traditional ML approaches have shown to deliver competitive performance in NLI shared tasks and other available NLI datasets. This indicates that text-based NLI is essentially a pattern-matching task that requires little to no semantic understanding of language. These findings are similar to what we observe in related language identification tasks such as dialect and language variety identification where classical ML approaches often outperform deep learning approaches (Jauhianen et al., 2019; Zampieri et al., 2020, 2024). The similarity between NLI and other language identification has been noted in previous work as discussed in Malmasi et al. (2017b).

Amidst exciting developments in AI and NLP such as the introduction of pre-trained LLMs, we believe that this survey is an important information source for future research in text-based NLI. We expect it to be a useful resource for new and well-established researchers alike. We conclude by discussing future directions and open challenges in text-based NLI including the use of LLMs, explainability, and low-resource scenarios.

Low-resource domainsAs described in Section 3, most corpora used in NLI are collected within educational settings (e.g., TOEFL exams, classroom activities). L2 production, however, is not restricted to education. Texts on social media, blogs, online reviews, etc. are also written by L2 speakers and can be used in NLI (Anand Kumar et al., 2017; Rabinovich et al., 2018). More research should be carry out to produce suitable datasets that allow us to investigate L2 production in non-educational domains. This is important as prior work has shown that NLI models may not generalize across datasets (Malmasi and Dras, 2015).

Low-resource languagesThere is scarcity of corpora with data from L2 speakers of low-resource languages particularly from Africa, Southeast Asia, and Oceania. This hinders the development of language technology for low-resource languages, an issue addressed by initiatives such as Masakhane Orife et al. (2020) and No Language Left Behind (NLLB) Costa-jussa et al. (2022). The datasets presented in Section 3 mostly contain widely-spoken languages such as Arabic, Chinese, English Spanish, and Portuguese and a few mid-resource languages with millions of speakers such as Finnish and Norwegian. There is ample scope for NLI research in low-resource languages and it must start by data curation.

Large Language ModelsWith the exception of very few studies Lotfi et al. (2020); Zhang and Salle (2023), LLMs have not been substantially explored for NLI. With recent advances in LLMs, we believe that more attention should be devoted to LLM-based approaches in NLI. We see task finetuning and prompt engineering, potentially using prompts with linguistic features indicative of L1, as two promising avenues to improving NLI performance with LLMs. Results described in Lotfi et al. (2020) indicate that task fine-tuning is a promising approach for NLI. Finally, LLMs can also be incorporated into ensemble systems which perform well with classic ML in NLI 

ExplainabilityExplainability of models is an important area that can reveal insights about both the data and the models. The clear majority of studies discussed in this survey, however, focus on exploring which features and/or computational models work best for text-based NLI. Only a few studies have explored data-driven methods to gain insights on language transfer and/or on computational models of L2 production Liu et al. (2022); Berti et al. (2023). With the availability of LLMs, we see ample room for more studies on model explainability in text-based NLI.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Aepli et al. (2022) Noemi Aepli, Antonios Anastasopoulos, Adrian-Gabriel Chifu, William Domingues, Fahim Faisal, Mihaela Gaman, Radu Tudor Ionescu, and Yves Scherrer. 2022. Findings of the vardial evaluation campaign 2022. In _Proceedings of VarDial_.
* Ajees and Idicula (2018) AP Ajees and Sumam Mary Idicula. 2018. lnli@ fire-2018: A native language identification system using convolutional neural networks. In _Proceedings of FIRE_.
* Kumar et al. (2017) M Anand Kumar, HB Barathi Ganesh, Shivkaran Singh, KP Soman, and Paolo Rosso. 2017. Overview of the inil pan at fire-2017 track on indian native language identification. In _Proceedings of CLEF_.
* Anastasopoulos et al. (2019) Antonios Anastasopoulos, Alison Lui, Toan Q Nguyen, and David Chiang. 2019. Neural machine translation of text from non-native speakers. In _Proceedings of NAACL_.
* Berti et al. (2023) Barbara Berti, Andrea Esuli, and Fabrizio Sebastiani. 2023. Unravelling interlanguage facts via explainable machine learning. _Digital Scholarship in the Humanities_, 38(3):953-977.
* Bharathi et al. (2017) B Bharathi, M Anirudh, and J Bhuvana. 2017. Bharathi ssn@ inili-fire-2017: Svm based approach for indian native language identification. In _FIRE_.
* Bhargava et al. (2017) Rupal Bhargava, Jaspreet Singh, Shivangi Arora, and Yashvardhan Sharma. 2017. Bits_pilani@ inili-fire-2017: Indian native language identification using deep learning. In _FIRE_.
* Bjerva et al. (2017) Johannes Bjerva, Gitare Grigonyte, Robert Ostling, and Barbara Plank. 2017. Neural networks and spelling features for native language identification. In _Proceedings of BEA_.
* Blanchard et al. (2013) Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. Toefl11: A corpus of non-native english. _ETS Research Report Series_.
* Boudal et al. (2010) Abderrahim Boudal, Abdelhak Lakhouaja, Azzeddine Mazroui, Abdelouaf Meziane, MOAO Bebah, and Mostafa Shoul. 2010. Alkhalil morphto sys1: A morphosyntactic analysis system for arabic texts. In _Elsevier IACIT_.
* Brooke and Hirst (2011) Julian Brooke and Graeme Hirst. 2011. Native language detection with 'cheap' learner corpora. In _Proceedings of LCR_.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Proceedings of NeurIPS_.
* Bykh et al. (2013) Serhiy Bykh, Soomya Vajjala, Julia Krivanek, and Detmar Meurers. 2013. Combining shallow and linguistically motivated features in native language identification. In _Proceedings of BEA_.
* Chang and Lin (2011) Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: a library for support vector machines. _ACM transactions on intelligent systems and technology (TIST)_.
* Costa-jussa et al. (2022) Marta R Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elaine Kalbassi, Janinee Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint arXiv:2207.04672_.
* Dehak et al. (2011) Najim Dehak, Pedro A Torres-Carrasquillo, Douglas Reynolds, and Reda Dehak. 2011. Language recognition via i-vectors and dimensionality reduction. In _In Proceedings of ISCA_.
* del Rio (2020) Ira del Rio. 2020. Native language identification on l2 portuguese. In _Springer CPPL_.
* de Rio et al. (2018) Iria del Rio, Marcos Zampieri, and Shervin Malmais. 2018. A portuguese native language identification dataset. In _Proceedings of BEA_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL_.
* Du Novo et al. (2020) Elisa Di Nuovo, Cristina Bosco, Elisa Corino, et al. 2020. How good are humans at native language identification? a case study on italian l2 writings. In _Proceedings of ICCL_.
* Ellis (2015) Rod Ellis. 2015. _Understanding second language acquisition 2nd edition_. Oxford university press.
* Escartin et al. (2017) Carla Parra Escartin, Wessel Reijers, Teresa Lynn, Joss Moorkens, Andy Way, and Chao-Hong Liu. 2017. Ethical considerations in nlp shared tasks. In _Proceedings of EthNLP_ 

* Estival et al. (2007) Dominique Estival, Tanja Gaustad, Son Bao Pham, Will Radford, and Ben Hutchinson. 2007. Author profiling for english emails. In _Proceedings of PACLING_.
* Fan et al. (2008) Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. _the Journal of machine Learning research_, 9:1871-1874.
* Gaman et al. (2020) Mihaela Gaman, Dirk Hovy, Radu Tudor Ionescu, Heidi Jauhiainen, Tommi Jauhainen, Krister Linden, Nikola Ljubesic, Niko Partanen, Christoph Purschke, Yves Scherrer, et al. 2020. A report on the vardial evaluation campaign 2020. In _Proceedings of Varma Dial_.
* Gass (1988) Susan M Gass. 1988. Second language acquisition and linguistic theory: The role of language transfer. In _Linguistic theory in second language acquisition_, pages 384-403. Springer.
* Gebrekidanegtre et al. (2013) Binyam Gebrekidanegtrek 

* Lynum (2013) Andre Lynum. 2013. Native language identification using large scale lexical features. In _Proceedings of BEA_.
* Major (2007) Roy C Major. 2007. Identifying a foreign accent in an unfamiliar language. _Studies in second language acquisition_, 29(4):539-556.
* Malmasi (2016) Shervin Malmasi. 2016. _Native Language Identification: Explorations and Applications_. Ph.D. thesis, Macquarie University.
* Malmasi and Cahill (2015) Shervin Malmasi and Aoife Cahill. 2015. Measuring feature diversity in native language identification. In _Proceedings of BEA_.
* Malmasi et al. (2018) Shervin Malmasi, Iria del Rio, and Marcos Zampieri. 2018. Portuguese native language identification. In _Proceedings of PROOPOR_.
* Malmasi and Dras (2014a) Shervin Malmasi and Mark Dras. 2014a. Arabic native language identification. In _Proceedings of ANLP_.
* Malmasi and Dras (2014b) Shervin Malmasi and Mark Dras. 2014b. Chinese native language identification. In _Proceedings of EACL_.
* Malmasi and Dras (2014c) Shervin Malmasi and Mark Dras. 2014c. Finnish native language identification. In _Proceedings of ALTA_.
* Malmasi and Dras (2015) Shervin Malmasi and Mark Dras. 2015. Large-scale native language identification with cross-corpus evaluation. In _Proceedings of NAACL_.
* Malmasi and Dras (2017a) Shervin Malmasi and Mark Dras. 2017a. Multilingual native language identification. _Natural Language Engineering_, 23(2):163-215.
* Malmasi and Dras (2017b) Shervin Malmasi and Mark Dras. 2017b. Native language identification using stacked generalization. _arXiv preprint arXiv:1703.06541_.
* Malmasi and Dras (2018) Shervin Malmasi and Mark Dras. 2018. Native language identification with classifier stacking and ensembles. _Computational Linguistics_, 44(3):403-446.
* Malmasi et al. (2017a) Shervin Malmasi, Mark Dras, Mark Johnson, Lan Du, and Magdalena Wolska. 2017a. Unsupervised text segmentation based on native language characteristics. In _Proceedings of ACL_.
* Malmasi et al. (2015a) Shervin Malmasi, Mark Dras, and Irina Temnikova. 2015a. Norwegian native language identification. In _Proceedings of RANLP_.
* Malmasi et al. (2017b) Shervin Malmasi, Keelan Evainini, Aoife Cahill, Joel Tetreault, Robert Pugh, Christopher Hamill, Diane Napolitano, and Yao Qian. 2017b. A report on the 2017 native language identification shared task. In _Proceedings of BEA_.
* Malmasi et al. (2015b) Shervin Malmasi, Joel Tetreault, and Mark Dras. 2015b. Oracle and human baselines for native language identification. In _Proceedings of BEA_.
* Markov et al. (2017) Ilia Markov, Lingzhen Chen, Carlo Strapparava, and Grigori Sidorov. 2017. Cic-fbk approach to native language identification. In _Proceedings of BEA_.
* Markov and Sidorov (2018) Ilia Markov and Grigori Sidorov. 2018. Cic-ipn@inli2018: Indian native language identification. In _FIRE (Working Notes)_.
* Martinez et al. (2011) David Martinez, Oldrich Plchot, Lukas Burget, Ondrej Glembek, and Pavel Matejka. 2011. Language recognition in ivectors space. In _In Proceedings of ISCA_.
* Mechti et al. (2016) Seifeddine Mechti, Ayoub Abbassi, Lamia Hadrich Belguith, and Rim Faiz. 2016. An empirical method using features combination for arabic native language identification. In _proceedings of AICSA_.
* Mechti et al. (2020) Seifeddine Mechti, Nabil Khoufi, and Lamia Hadrich Belguth. 2020. Improving native language identification model with syntactic features: Case of arabic. In _Proceedings of ISDA_.
* Minaee et al. (2024) Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. _arXiv preprint arXiv:2402.06196_.
* Mizumoto et al. (2013) Tomoya Mizumoto, Yuta Hayashibe, Keisuke Sakaguchi, Mamoru Komachi, and Yuji Matsumoto. 2013. Naist at the nli 2013 shared task. In _Proceedings of BEA_.
* Mohammadi et al. (2017) Elham Mohammadi, Hadi Veisi, and Hessam Amini. 2017. Native language identification using a mixture of character and word n-grams. In _Proceedings of BEA_.
* Mondal et al. (2018) Soumik Mondal, Athul Harilal, and Alexander Binder. 2018. Corplab inli @ fire-2018: Identification of indian native language using pairwise coupling. In _Proceedings of FIRE_.
* Mundotiya et al. (2018) Rajesh Kumar Mundotiya, Manish Kumar Singh, and Anil Kumar Singh. 2018. Nlprl @ inli-2018: Hybrid gated lstm-cnn model for indian native language identification. In _Proceedings of FIRE_.
* Nayel and Shashirekha (2017) Hamada A Nayel and HL Shashirekha. 2017. Mangalore-university@ inili-fire-2017: Indian native language identification using support vector machines and ensemble approach. In _Proceedings of FIRE_.
* Nayel and Shashirekha (2018) Hamada A Nayel and HL Shashirekha. 2018. Mangalore university inli@ fire2018: Artificial neural network and ensemble based models for inli. In _Proceedings of FIRE_.
* Oh et al. (2017) Yoo Rhee Oh, Hyung-Bae Jeon, Hwa Jeon Song, Yun-Kyung Lee, Jeon-Gue Park, and Yun-Keun Lee. 2017. A deep-learning based native-language classification by using a latent semantic analysis for the nli shared task 2017. In _Proceedings of BEA_.
* Orfe et al. (2020)Iroro Orfe, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, et al. 2020. Masakhane-machine translation for africa. _arXiv preprint arXiv:2003.11529_.

 

Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. _the Journal of machine Learning research_, 12:2825-2830.
* Popescu and Ionescu (2013) Marius Popescu and Radu Tudor Ionescu. 2013. The story of the characters, the dna and the native language. In _Proceedings of BEA_.
* Rabinovich et al. (2016) Ella Rabinovich, Sergiu Nisioi, Noam Ordan, and Shuly Wintner. 2016. On the similarities between native, non-native and translated texts. In _Proceedings of ACL_.
* Rabinovich et al. (2018) Ella Rabinovich, Yulia Tsvetkov, and Shuly Wintner. 2018. Native language cognate effects on second language lexical choice. _Transactions of the Association for Computational Linguistics_.
* Rama and Coltekin (2017) Taraka Rama and Cagri Coltekin. 2017. Fewer features perform well at native language identification task. In _Proceedings of BEA_.
* Remnev (2019) Nikita Remnev. 2019. Native language identification for russian. In _Proceedings of ICDMW_.
* Rogers et al. (2021) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A primer in bertology: What we know about how best works. _Transactions of the Association for Computational Linguistics_, 8:842-866.
* Rozovskaya and Roth (2011) Alla Rozovskaya and Dan Roth. 2011. Algorithm selection and model adaptation for esl correction tasks. In _Proceedings of ACL_.
* Steinbakken and Gamback (2020) Stian Steinbakken and Bjorn Gamback. 2020. Native-language identification with attention. In _Proceedings of ICON_.
* Swan and Smith (2001) Michael Swan and Bernard Smith. 2001. _Learner English: A teacher's guide to interference and other problems_. Cambridge University Press.
* Swanson and Charniak (2012) Ben Swanson and Eugene Charniak. 2012. Native language detection with tree substitution grammars. In _Proceedings of ACL_.
* Tetreault et al. (2013) Joel Tetreault, Daniel Blanchard, and Aoife Cahill. 2013. A report on the first native language identification shared task. In _Proceedings of BEA_.
* Tetreault et al. (2012) Joel Tetreault, Daniel Blanchard, Aoife Cahill, and Martin Chodow. 2012. Native tongues, lost and found: Resources and empirical evaluations in native language identification. In _Proceedings of COLING_.
* Thenmozhi et al. (2017) D Thenmozhi, Kawshik Kannan, and Chandrabose Aravindan. 2017. Ssn\(\_\)nlp@ nli-fire-2017: A neural network approach to indian native language identification. In _Proceedings of FIRE_.
* Themozhi et al. (2018) D Thenmozhi, S Kayalvizhi, and Chandrabose Aravin-dan. 2018. A machine learning approach to indian native language identification. In _FIRE (working notes)_.
* Tsvetkov et al. (2013) Yulia Tsvetkov, Naama Twitto, Nathan Schneider, Noam Ordan, Manal Farauqui, Victor Chahuneau, Shuly Wintner, and Chris Dyer. 2013. Identifying the 11 or non-native writers: the cmu-haifa system. In _Proceedings of BEA_.
* Tydlitatova (2016) Ludmila Tydlitatova. 2016. Native language identification of l2 speakers of Czech. _Univerzita Karlova, Matematicko-fyzikadmi fkkulta_.
* Uluslu (2023) Ahmet Yavuz Uluslu. 2023. Turkish native language identification. In _Proceedings of ICNLSP_.
* Uluslu and Schneider (2022) Ahmet Yavuz Uluslu and Gerold Schneider. 2022. Scaling native language identification with transformer adapters. In _Proceedings of ICMISP_.
* Vajjala and Banerjee (2017) Sowmya Vajjala and Sagnik Banerjee. 2017. A study of n-gram and embedding representations for native language identification. In _Proceedings of BEA_.
* Vian (2023) Matias Johansen Vian. 2023. A study of transformers for cross-corpus native language identification. Master's thesis, NTNU.
* Wong and Dras (2009) Sze-Meng Jojo Wong and Mark Dras. 2009. Contrastive analysis and native language identification. In _Proceedings of ALTA_.
* Wong and Dras (2011) Sze-Meng Jojo Wong and Mark Dras. 2011. Exploiting parse structures for native language identification. In _Proceedings of EMNLP_.
* Wong et al. (2012) Sze-Meng Jojo Wong, Mark Dras, and Mark Johnson. 2012. Exploring adaptor grammars for native language identification. In _Proceedings of EMNLP_.
* Wu et al. (2013) Ching-Yi Wu, Po-Hsiang Lai, Yang Liu, and Vincent Ng. 2013. Simple yet powerful native language identification on toefll11. In _Proceedings of BEA_.
* Zampieri et al. (2017) Marcos Zampieri, Alina Maria Ciobanu, and Liviu P. Dinu. 2017. Native language identification on text and speech. In _Proceedings of BEA_.
* Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, and Yves Scherrer. 2020. Natural language processing for similar languages, varieties, and dialects: A survey. _Natural Language Engineering_, 26(6):595-612.
* Zampieri et al. (2024) Marcos Zampieri, Kai North, Tommi Jauhainen, Mariano Felice, Neha Kumari, Nishant Nair, and Yash Bangera. 2024. Language variety identification with true labels. In _Proceedings of LIREC-COLING_.
* Zhang and Salle (2023) Wei Zhang and Alexandre Salle. 2023. Native language identification with large language models. _arXiv preprint arXiv:2312.07819_.
* Zissman (1996) Marc A Zissman. 1996. Comparison of four approaches to automatic language identification of telephone speech. _IEEE Transactions on speech and audio processing_, 4(1):31.
* Zubiaga et al. (2016) Arkairitz Zubiaga, Inaki San Vicente, Pablo Gamallo, Jose Ramm Pichel, Inaki Alegreia, Nora Aranberri, Aitzol Ezeiza, and Victor Fresno. 2016. Tweetlid: a benchmark for tweet language identification. _Language Resources and Evaluation_, 50:729-766