Generalization in Native Language Identification: Learners versus Scientists
Authors: Sabrina Stehwien, Sebastian Padó (Universität Stuttgart, 2015)
Topic: Learner Corpora and Language Acquisition

Abstract
Native Language Identification (NLI) is the task of recognizing an author's native language from text in another language. In this paper, we consider three English learner corpora and one new, presumably more difficult, scientific corpus. We find that the scientific corpus is only about as hard to model as a less-controlled learner corpus, but cannot profit as much from corpus combination via domain adaptation. We show that this is related to an inherent topic bias in the scientific corpus: researchers from different countries tend to work on different topics.

Introduction
Native Language Identification (NLI) is the task of recognizing an author's native language (L1) from text written in a second language (L2). NLI is important for applications such as the detection of phishing attacks AUTHOR or data collection for the study of L2 acquisition AUTHOR. State-of-the-art methods couch NLI as a classification task, where the classes are the L1 of the author and the features are supposed to model the effects of the author's L1 on L2 (language transfer). Such features may be of varying linguistic sophistication, from function words and structural features AUTHOR on one side to N-grams over characters, words and POS tags AUTHOR on the other side. Like in many NLP tasks, there are few large datasets for NLI. Furthermore, it is often unclear how well the models really capture the desired language transfer properties rather than topics. The widely-used International Corpus of Learner English (ICLE, AUTHOR) has been claimed to suffer from a topic bias AUTHOR: Authors with the same L1 prefer certain topics, potentially due to the corpus collection strategy (from a small set of language courses). As a result, AUTHOR question the generalization of NLI models to other corpora and propose the use of domain adaptation. In contrast, AUTHOR report their ICLE-trained models to perform well on other learner corpora. This paper extends the focus to a novel corpus type, non-native scientific texts from the ACL Anthology. These are substantially different from learner corpora: (a) most authors have a good working knowledge of English; and (b) due to the conventions of the domain, terminology and structure are highly standardized AUTHOR. A priori, we would believe that NLI on the ACL corpus is substantially more difficult. Our results show, however, that the differences between the ACL corpus and the various learner corpora are more subtle: The ACL corpus is about as difficult to model as some learner corpora. However, generalization to the ACL corpus is more difficult, due to its idiosyncratic topic biases.

Conclusion
This study investigated the generalizability of NLI models across learner corpora and a novel corpus of scientific ACL documents. We found that generalizability is directly tied to corpus properties: well-controlled learner corpora (TOEFL-11, ICLE) generalize well to one another AUTHOR. Together with the minor effect on performance of removing topic-related features, we conclude that topic bias within a similar text type does not greatly affect generalization. At the same time, ``classical'' learner corpora do not generalize well to less-controlled learner corpora (Lang-8) or scientific corpora (ACL). Lang-8 and ACL show comparable performance, which seems surprising given the small size of the ACL corpus and its quite different nature. Our analysis shows that the ACL corpus exhibits an idiosyncratic topic bias: scientists from different countries work on different topics, which is reflected in the models. As a result, the improvements that Lang-8 can derive from domain adaptation techniques carry over to the ACL corpus only to a limited extent. Nevertheless, the use of mSDA can halve the amount of ACL data necessary for the same performance, which is a promising result regarding the generalization to other low-resource domains.