DETERMINING THE ETHNO-NATIONALITY OF WRITERS USING WRITTEN ENGLISH TEXT
Year: 2021

Abstract:
Ethno-nationality is where nations are defined by a shared heritage, for instance it can be a membership of a common language, nationality, religion or an ethnic ancestry. The main goal of this research is to determine a person’s country-oforigin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques. The current literature mainly focuses on determining the native language of English writers and a minimal number of researches have been conducted in determining the country-of-origin of English writers. Further, most experiments in the literature are mainly based on the TOEFL, ICLE datasets which were collected in more controlled environments (i.e., standard exam answers). Hence, most of the writers try to follow some guidelines and patterns of writing. Subsequently, the creativity, freedom of writing and the insights of writers could be hidden. Thus, we believe it hides the real nativism of the writers. Further, those corpora are not freely available as it involves a high cost of licenses. Thus, the main data corpus used for this research was the International Corpus of English (ICE corpus). Up to this point, none of the researchers have utilised the ICE corpus for the purpose of determining the writers’ countryof-origin, even though there is a true potential. For this research, an overall accuracy of 0.7636 for the flat classification (for all ten countries) and accuracy of 0.6224∼1.000 for sub-categories were received. In addition, the best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model.

text: Ethno-nationality is where nations are defined by a shared heritage, for instance it can be a membership of a common language, nationality, religion or an ethnic ancestry. The main goal of this research is to determine a person’s country-oforigin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques. The current literature mainly focuses on determining the native language of English writers and a minimal number of researches have been conducted in determining the country-of-origin of English writers. Further, most experiments in the literature are mainly based on the TOEFL, ICLE datasets which were collected in more controlled environments (i.e., standard exam answers). Hence, most of the writers try to follow some guidelines and patterns of writing. Subsequently, the creativity, freedom of writing and the insights of writers could be hidden. Thus, we believe it hides the real nativism of the writers. Further, those corpora are not freely available as it involves a high cost of licenses. Thus, the main data corpus used for this research was the International Corpus of English (ICE corpus). Up to this point, none of the researchers have utilised the ICE corpus for the purpose of determining the writers’ countryof-origin, even though there is a true potential. For this research, an overall accuracy of 0.7636 for the flat classification (for all ten countries) and accuracy of 0.6224∼1.000 for sub-categories were received. In addition, the best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model.

heading: 1 INTRODUCTION
text: Ethno-nationality can be determined as the identity in which jointly defined by ethnicity and nationality (Sapp, 2012). Identifying a writer’s country-of-origin is a part of identifying his ethnonationality (Jerry, 2008). Foremost focus of ethno-nationality in this research is to identify a writer’s country-of-origin based on their writings in a non-native or second language. In recent years, identification of ethno-nationality of a writer has gained a growing interest. In author profiling demographic features (such as age, gender, education, native language, country of origin, etc.,) of an author from a written text will be identified, which are commonly needed in forensic linguistics. For instance, Intelligence to build a profile of their suspect, to identify the author of an anonymous email threat. Therefore, this will enable to limit the search space as well (Estival et al., 2007). Moreover, for business applications this can be useful; for example, in marketing where the demographic features as stated above of customers is important to predict behaviors, upgrade the current products and to develop new products.
People from different ethno-nationality make various language errors when learning a language. Identification of writer’s country of origin could have an impact on educational applications designed towards non-native speakers of a language. Besides, this can be used as a plug-in to online tutor systems to provide more tailored feedback to the students about their mistakes (Tetreault et al., 2013). This can help researchers to identify specific teaching and learning issues in different ethnonationalities. This will enable them to develop pedagogical learning materials to address and solve those issues.
In this research we focus on English as the second language. As English has become a ‘universal communication’ language due to the globalization. It is no longer restricted to the native countries such as England and United States (Eric, 2013). English language is now available in numerous emerging fields, and has become an essential requirement of labour market and further it is considered to have a cultural importance (Marko, 2009). We define, A controlled environment as a place or area where rules, regulations and norms are subject to strict enforcement. For instance, in an examination setting, candidates are required to produce answers limiting the scope of the question.

heading: 2 RELATED WORK
text: Distribution of English language around the world can be identified in two ways, mainly; based on the geographic distribution and based on the Kachru’s theory (Braj B, 2004). In this study when segmenting according to the geographic distribution we have considered clusters namely; ‘Asia’, ‘non-Asia’, ‘South Asia’, ‘non-South Asia’ and ‘North America’. According to the Kachru’s ‘Concentric Circles’, English speakers were segmented in to three main categories, namely; ‘inner circle’, ‘outer circle’ and ‘expanded circle’.
As stated by his model, the inner circle comprises of the countries where the English is the native language (i.e., UK, USA, Canada, New Zealand, etc.). The outer circle includes countries where there are small communities of native English speakers however English is used as the second language in education and official purposes (i.e., Sri Lanka, India, Singapore, Nigeria, etc. ). The expanding circle contains countries which considered English as foreign language (i.e., China, Indonesia, Japan, Saudi Arabia, etc.).
The work of Koppel et al. (2005) is one of the first ethno-nationality identification work where it involves categorizing users based on their native language employing Support Vector Machines (SVMs) on various stylistic features focused on identification of common. Authors achieved 0.802 accuracy using five chosen languages (i.e., Czech, French, Bulgarian, Russian and Spanish) from International Corpus of Learner English (ICLE)(Granger, 2014). The ICLE dataset contains argumentative essays writings of the university students and less nativism and creativity of the writer involved.
Bykh & Meurers (2012) proposed use of recurring n-grams on three different classes (word based, POS based, Open-Class-POS-based) as features for training SVMs. Out of other SVM implementations LIBLINEAR produced best results. For this study, random data from seven native languages selected from the ICLE corpus. The highest performance was obtained for word-level n-grams with an accuracy of 0.8971.
Gebre et al. (2013) employs linear SVM, logistic regression and perceptron (as baseline) for the native language identification (NLI) and achieved accuracy of 0.814 for eleven languages of ‘The Test of English as a Foreign Language’ (TOEFL) data set(Blanchard et al., 2013). Features used includes; word n-grams, POS n-grams, character n-grams and spelling errors. TOEFL11 has become standard benchmark in NLI tasks since its introduction for the NLI Shared Task 2013. The main limitation of the TOEFL11 dataset is that it is collected in a more controlled environment (i.e., exam for English).
Cimino & Dell’Orletta (2017) utilizes a novel stacked classifier approach where linear logistic regression based sentence feature classifier is stacked with a SVM based document feature classifier with standard lexical, stylistic and syntactic features. However stacked classification approach has gained a minor gain compared to unstacked. Best results of the NLI shared task 2017 reported for this approach with F1-score of 0.8818 for the TOEFL11 dataset.
Kulmizev et al. (2017) introduced ‘Groningen’ system for the NLI Shared Task 2017 which out performs employing linear SVM for character 1-9 n-grams with the F1-score of 0.8756. Authors have reported that several experiments done with ensemble approach and other features such POS, word, lemma n-grams, skip-grams; and those failed to match the performance of character 1-9 ngram system.
Goutte & Léger (2017) explored use of voting ensemble SVM models with character, word and POS n-grams. Authors confirms that ensemble methods provide minor but systematic predictive perfor-
mance gains. Highest F1-score performance was with best-vote approach consisting 10 models is 0.8740 for TOEFL11 dataset.
Markov et al. (2018) has used punctuation-based features with POS n-grams for his experiments and accuracies of 0.4883 (TOEFL11) and 0.6948 (ICLE) reported for best performing settings. SVM with one-vs-all (OvA) multi-class classification approach has been used to conduct these experiments.
Malmasi & Dras (2018) employed a supervised multi-class classification approach and incorporated several corpora, including; TOEFL11, EF Cambridge Open Language Database Corpus (EFCAMDAT), ASK Corpus (Andresprȧkskorpus, Second Language Corpus), and Jinan Chinese Learner Corpus. The features extracted were; word/lemma n-grams, character n-grams, function word ngrams, POS n-grams, dependencies, CFG rules, adaptor Grammars and TSG fragments. Highest accuracy of 0.871 is reported for the TOEFL11 dataset.
According to the literature it is evident that most of the researchers in the computational linguistic community have employed ICLE in the early stages and TOEFL in the latest NLI tasks (refer Table 1) which both were collected in more controlled environments. Finally, due to above controlled environments; being unable to capture real nativism in written texts certainly inflate the performance of ethno-nationality identification. On the other hand, identifying the nation of the English writer opposed to his native language will be equally beneficial for author profiling as well. Thus, we
introduce use of International Corpus of English (ICE) for the ethno-nationality identification. To the best of our knowledge, none of the researches have been conducted to determine the writer’s country of origin based on the International Corpus of English (ICE) corpus. Moreover, very limited number of researchers have identified the significant features which helps to distinguish Sri Lankan English writers using a large corpus like ICE. One limitation of these researches raised by the authors Tofıghı et al. (2012) is that, since for most of the web-based applications, automatic spell-checker has been applied, idiosyncratic features including misspelling and other anomalies are ignored. This may hide some of the features which will be useful in identifying the real categories. Nevertheless, in current context, usage of spell checkers and grammar checkers can be seen frequently. Thus, this will be a common limitation in similar types of researches.

heading: 3 RESEARCH METHODOLOGY
text: Supervised learning approach was chosen as it is suitable in classifying text documents into classes more accurately if the classes are known and the data set is labelled (Slotte, 2018). Hence, according to the literature most promising ML algorithms for the text classification such as; Support Vector Machines with Stochastic Gradient Descent Optimizer (linear SVM with SGD), Multinomial Naı̈ve Bayes (MNB), Decision Tree (DT) and Random Forest (an ensemble approach) have been employed. For this research mostly Scikit-learn(Pedregosa et al., 2012) has been used for preprocessing, feature extraction and classification tasks. Scikit-learn is a Python module for machine learning which provides state-of-the-art implementations of many well-known machine learning algorithms, and maintains an easy-to-use interface(Pedregosa et al., 2012). The workflow of the research is depicted in Figure 1 (Left).
The main assumptions of this research are authors from the same ethno-nationality share the same linguistic features in their writing and will often have an influence on the way they express themselves in writings (Jain et al., 2017).
The research questions addressed in this research are;

heading: Q1. How can texts produced by English writers in a given ethno-nationality be captured from existing corpora?

heading: Q3. Which machine learning techniques can gainfully employ the extracted data to identify countryof-origin of English writers?
text: #Docs = number of documents, #Words = number of words

heading: 3.1 DATA COLLECTION
text: The main dataset employed in this study is International Corpus of English (ICE) (Greenbaum & Nelson, 1996; Kirk & Nelson, 2018) which comprises of several corpora from different countries. This data-set was built with the intention of providing a resource to conduct comparative studies of English used in different countries where English is the native language or the second language. In order to maintain consistency among each country corpus, common corpus design, corpus size and a common scheme for grammatical annotation have been followed (Kirk & Nelson, 2018). This dataset consists of both written and spoken English texts. Only written texts have been considered for this study. Written texts were gathered from many areas, such as; student writings, letters, academic writing, news reports, instructional writing, persuasive writing and creative writing (Kirk & Nelson, 2018).
The authors and speakers of the texts are aged 18 or above, educated through the medium of English, and were either born in the country in whose corpus they are included or spent the majority of their lives there, or moved there at an early age, and received their education through the medium of English in the country concerned. Written English corpora from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA have been collected and used for this research. Each country is consisted of ≈200 text documents (≈2,000 words per document) (refer Figure 1).

heading: 3.2 DATA ANALYSIS
text: As the initial step, 30% (testing data) of the total data of each country corpus was kept as unseen or un-touched data. The remaining 70% of data (training data) was analyzed in order to identify the pre-processing requisites.

heading: 3.3 DATA PRE-PROCESSING
text: Raw text-files with specific markups are used as input. These files are cleaned as per the ICE-Corpus markup guide Nelson (2002). In addition, ‘strip accents’ parameter for the TF-IDF vectorizer and ‘Unicode data normalize NFKD’ were used. Moreover, URLs, XML tags, e-mail addresses, censored data, line feeds were removed. The data set contained HTML entity encodings and those were decoded(refer Figure 1). In order to avoid country names, nationality, currency and popular cities being trained as features, all the country specific nationality, cities and country names are removed as a pre-processing step.

heading: 3.4 CLASSIFICATION MODEL BUILDING
text: The training data set were used for model building, parameter tuning, training and selection. This labelled training data was tokenized using the TF-IDF vectorizer. Vectorized output was fed to ML classifier to train the model. Further, k-fold (k=3) cross validation technique was used on the training data. Testing data set was kept hold to feed and evaluate the classification model in later stages. Models were built based on two classification strategies;
• Flat Classification Strategy Flat classification refers to a single classifier at the root level as the decision point as depicted in Figure 2(Top). This classifier will handle the all classes as per its classification approaches such as one-vs-rest, one-vs-one...etc. In this study ‘one-vs-all’ approach has been followed. Hence, the dataset was trained for 10 classes (countries) together.
• Sub-Category Classification Strategy Sub-category classification employs N number of classifiers as depicted in Figure 2(Bottom Left) and Figure 2(Bottom Right). For each sub-category specialized in solving a subset of the problem in which each classifier is trained. Each sub-category was trained separately. As depicted, some are binary (marked with black boxes in Figure 2) and some are multi-class classification models (marked with grey boxes in Figure 2). All the experiments of the sub-categories which were tested are depicted in Figure 3.

heading: 4 TESTING AND EVALUATION
text: A combination of both ‘hold-out validation’ and k-fold cross-validation was used in order to reduce biasness for training and testing data set (Slotte, 2018). Hence, on the 70% of the training data set, 3-fold cross validation technique was used for model building, training and selection and 30% of the data was kept for hold-out validation to validate the machine learning model. For each decision point (where a single classifier needs to be selected) 16 experiments (4 ML x 4 n-grams) have been carried out. For each ML algorithm, word level (1-1,2,3,4) n-grams were tested. Further, performance accuracy, F1-score, precision and recall were calculated to select the best performing ML model.

heading: 5 RESULTS AND DISCUSSION

heading: Q1: How can texts produced by English writers in a given ethno-nationality be captured from existing corpora?
text: • Flat classification strategy The best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model. Furthermore, overall balanced accuracy of 0.7620 and macro average F1-Score of 0.76 was obtained. Score breakdown for each country is depicted in
Figure 4(Left). In addition, India has the highest F1-score of 0.94 while USA & Sri Lanka holding the lowest F1-score of 0.68. These statistics are further verified by the confusion matrix depicted in Figure 4(Right). Accordingly, all the documents of the India have been classified correctly. The confusion matrix further depicts that the model makes most of the mistakes at the classification of USA vs Jamaica pair. Certainly, one reason could be the geo-graphical proximity between these two countries.
• Sub-category classification strategy In sub-category classification, each sub-category can be perceived as a Decision Point (a single classifier) in the hierarchy of a decision tree. Figure 5 (Left) depicts the test results of the selected ML models for each sub-category. Accuracies of different sub-categories have varied in between 0.6224∼1.000 and F1-score ranges between 0.49∼1.00. Based on the Kachru’s concentric theory, lowest F1-Score of 0.71 is for “Outer vs Expanded” sub-category. On the other hand, ‘Expanded‘ sub-category (Hong Kong vs Jamaica) has the highest F1-score of 0.92. In geo-graphical distribution-based model, lowest F1-score of 0.49 is for “Non-Asia” sub-category, while ‘South Asian countries‘ sub-category holding the highest F1-score of 1.0.
• Generalisability of the model Assessing the generalizability of the ML models is crucial. Over-fitting leads to poor generalizability of the model. Hence, in order to detect and prevent over-fitting, ‘hold-out validation’ and k-fold cross-validation used. Further, learning curves on the training data set have been drawn to diagnose whether the model is over-fitting or under-fitting. The red line represents the training score (trained and tested using same data) and the green line represents the cross-validation score(k=3) obtained
for varying number of samples. Cross validated score curve is flattening against the training score curve, as depicted in Figure 5(Right). Thus, this model does not over-fitting or under-fitting for the dataset.
• Most significant features of the model Most significant features identified for the flat classification is depicted in Figure 6. These features are computed base on the coefficients assigned. Some of the features are driven by the cultural and geo-graphical differences of the particular country. For instance, Ireland: northern, christmas, groundwater, queen. and Singapore: business, batik, population. Nigeria: donut(for do not), federal, god..etc.

heading: Q2: Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?
text: Comparison of ML algorithms based on the selected best parameters are analysed on top of the test data and results are depicted in Figure 7. Linear SVM with SGD optimizer seems to be outperforms in most cases. Moreover, this behavior is also verified in the previous work by Kulmizev et al. (2017), Koppel et al. (2005), Ekaterina (2011), Gebre et al. (2013), Bykh & Meurers (2012). Furthermore, it is noticeable that the DT is under-performing for most cases.

heading: 6 CONCLUSION AND FUTURE WORK
text: With various accuracy levels the literature has proven that the ethno-nationality of a person can be identified using their written English texts and this area of research has lot of practical applications and usage. However, as discussed in the literature review still those researches are comprised with lot of limitations as stated above. Therefore, those identified limitations have re-framed this research to obtain solutions to the identified research problem and defined research questions.
Our work on Ethno-nationality Identification confirms that linear SVM with SGD optimizer trained with word n-grams can yield a higher level of performance. When determining country-of-origin it is essential to identify set of features which are unique to each author or a particular group of authors.
One spectacular restraint would be the usage of spelling and grammar checkers when writing English. This limitation can be overwritten when using spoken English text. Hence, as a future work transcribed text can be considered to identify the country-of-origin of the English writers. Further, needs to focus more on features which have more pedagogical value and cross corpus generalizability should be assessed to examine the extendibility of the model.

heading: ACKNOWLEDGMENT
text: We are deeply grateful for the International English of Corpus (ICE) project owners and maintainers of the respective countries for making available this corpus to the research community without any license fee.
ETHICS STATEMENT
The Contributors of the respective ICE Corpus countries have informed about the data collection procedures and objectives. All the identifiable named entities of the data have been anonymized in order to disjoint any relation of individuals and organizations to the data. Hence, no one, including the researchers, will be able to link data to a specific individual. Outcomes of these type of researches on categorization of individuals based on ethno-nationality can raise concerns over its usage and discrimination. However, author-profiling is gaining pace and the authors of this paper admire the true potential of such categorizations.
REPRODUCIBILITY STATEMENT
The implementation details this experiment is available as a supplementary material along with the submission. Kindly note that the supplementary submission does not include the complete dataset. Authors can produce the complete data-set used upon a formal request.
All the raw-data for each country in ICE corpora is available as text files. Files are pre-processed through a jupyter notebook and results were written into a csv called ‘ice-merged.csv‘. These preprocessing details are also available with the supplementary materials and for further details please refer to README.txt file in the root level.

Review summaries:
This paper presents a simple classification-based approach to a newly defined problem: determining ethno-nationality of writers of English text data. The problem of determining ethno-nationality can be seen as a direct extension of the research problem of identifying the writers' native language based on written English text. The authors define ethno-nationality mostly as country-of-origin and aim to distinguish this demographic feature from native language as another feature. The empirical validations are mostly applications of standard (and even obsolete) classification models on the newly collected data for the introduced problem.

The work aims to automatically determine the country of origin of authors given their English texts.  Experiments are conducted using the International Corpus of English (ICE corpus), which includes texts of authors from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA; there are 200 text documents per origin country.   It is assumed that authors from the same ethno-nationality share the same `linguistic features'.  The documents are represented as TF-IDF weighted vectors of words, and a classifier is applied (it is not very obvious which classifier exactly, perhaps I missed this detail?).  In classification, the authors apply text classification, either directly into the 10 countries of origin, or using two different class hierarchies.  In addition to multi-class classification, they also conduct binary classification (e.g., native vs. non-native English speakers).  The paper then presents the classification results. Salient features (words) are detailed. 

This paper aims to determine the country of origin of English language writers. It uses documents from the International corpus of English (ICE) and employs various learning algorithms and text-based features. The findings are that linear SVM with unigram features works best, that some countries are better distinguished than others, and that some features are more informative than others.