Determining the Ethno-nationality of Writers Using Written English Text
Authors: Deenuka Niroshini Perera
Ruvan Weerasinghe
Randhil Pushpananda
Venue: ICLR 2022 Submitted (2022)
Keywords: Ethno-nationality
Native Language Identification
Natural Language Processing
Machine Learning
Linear SVM
Less-controlled environments
ICE corpus

Abstract: Ethno-nationality is where nations are defined by a shared heritage, for instance it can be a membership of a common language, nationality, religion or an ethnic ancestry. The main goal of this research is to determine a person’s country-of-origin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques. The current literature mainly focuses on determining the native language of English writers and a minimal number of researches have been conducted in determining the country-of-origin of English writers.

Further, most experiments in the literature are mainly based on the TOEFL, ICLE datasets which were collected in more controlled environments (i.e., standard exam answers). Hence, most of the writers try to follow some guidelines and patterns of writing. Subsequently, the creativity, freedom of writing and the insights of writers could be hidden. Thus, we believe it hides the real nativism of the writers. Further, those corpora are not freely available as it involves a high cost of licenses. Thus, the main data corpus used for this research was the International Corpus of English (ICE corpus). Up to this point, none of the researchers have utilised the ICE corpus for the purpose of determining the writers’ country-of-origin, even though there is a true potential. 

For this research, an overall accuracy of 0.7636 for the flat classification (for all ten countries) and accuracy of 0.6224~1.000 for sub-categories were received. In addition, the best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model.

Under review as a conference paper at ICLR 2022
DETERMINING THE ETHNO-NATIONALITY OF WRITERS USING WRITTEN ENGLISH TEXT
Anonymous authors Paper under double-blind review
ABSTRACT
Ethno- nattonnttioonltty where nations are defined by a shared heritage, for instance it can be membership of common language, nationality, religion ethnic ancestry. The main goal of this research is to determine a person's country-of- origin using English text written in less controlled environments, employing Ma- chine Learning (ML) and Natural Language Processing (NLP) techniques. The current literature mainly focuses writers and a minimal number of researches have been conducted in determining the country-of ofriiinn of English writers.
determining the native language of English
Further, most experiments in the literature mainly based on the TOEFL, ICLE datasets which were collected in more controlled environments (i.e., standard answers). Hence, most of the writers try to follow some guidelines and patterns of writing. Subsequently, the creativity, freedom of writing and the in- sights of writers could be hidden. Thus, believe it hides the real nativism of the writers. Further, those corpora are not freely available as it involves a high cost of licenses. Thus, the main data corpus used for this research national Corpus of English (ICE corpus). Up to this point, none of the researchers have utilised the ICE corpus for the purpose of determining the writers' country- of-origin, even though there is true potential.
the Inter-
For this research, an overall accuracy of 0.7636 for the flat classification (for all ten countries) and accuracy of 0.6224-~1.00 1.000 for sub-categories were received. In addition, the best ML. model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model.
INTRODUCTION
Ethno-nationality can be determined as the identity which jointly defined by ethnicity and na- tionality (Sappl 2012). Identifying writer's country-ofoord is part of identifying his ethno- nationality (Jerry 2008). Foremost focus of nationallin in this research is to identify a writer's cofntry-of-origin based their writings in non-native second language. In recent years, iden- tification of ethno-nationality of a writer has gained a growing interest. author profiling demo- graphic features (such as age, gender, education, native language, country of origin, etc..) of an author from a written text will be identified, which are commonly needed in forensic linguistics. For instance, Intelligence build profile of their suspect, identify the author of anonymous email threat. Therefore, this will enable to limit the search space as well (Estival et al. (2007). Moreover, for business applications this can be useful; for example, in marketing where the demographic fea- tures as stated above of customers is important to predict behaviors, upgrade the current products and to develop new products.
People from different ethno-nationality make various language errors when learning a language. Identification of writer's country of origin could have an impact on educational applications designed towards non-native speakers of language. Besides, this can be used plug-in to online tutor systems to provide more tailored feedback to the students about their mistakes (Tetreault et al. 2013). This help researchers to identify specific teaching and learning issues in different ethno- nationalities. This will enable them to develop pedagogical learning materials to address and solve those issues.

---

Under review as a conference paper at ICLR 2022
In this research focus English the second language. As English has become 'universal communication language due to the globalization. no longer restricted to the native countries such as England and United States (Eric 2013). English language is now available in numerous emerging fields, and has become an essential requirement of labour market and further it is consid- ered to have cultural importance (Marko 2009). We define, A controlled environment as a place or area where rules, regulations and norms are subject to strict enforcement. For instance, in an examination setting, candidates are required to produce answers limiting the scope of the question.
2 RELATED WORK
Distribution of English language around the world can be identified in two ways, mainly; based on the geographic distribution and based on the Kachru's theory (Braj B 2004). In this study when segmenting according to the geographic distribution we have considered clusters namely; 'Asia', inon-Asia', 'South Asia', 'non-South Asia' and 'North America'. According to the Kachru's 'Con- centric Circles', English speakers were segmented in three main categories, namely; 'inner circle', 'outer circle' and "expanded circle'.
As stated by his model, the inner cirele comprises of the countries where the English is the native lan- guage (i.e., UK, USA, Canada, New Zealand, etc.). The outer circle includes countries where there small communities of native English speakers however English is used the second language in education and official purposes (i.e., Sri Lanka, India, Singapore, Nigeria, etc. The expand- ing circle contains countries which considered English as foreign language (i.e., China, Indonesia, Japan, Saudi Arabia, etc.).
The work of Koppel et al. (2005) is one of the first nationaati identification work where it involves categorizing users based on their native language employing Support Vector Machines (SVMs) on various stylistic features focused on identification of common, Authors achieved 0.802 accuracy using five chosen languages (i.e., Czech, French, Bulgarian, Russian and Spanish) from International Corpus of Learner English (ICIE))Granger) (014)). The ICLE dataset contains argu- mentative essays writings of the university students and less nativism and creativity of the writer involved.
Bykh & Meurers 2012)
proposed use of recurring n-grams on three different classes (word based, as features for training SVMs. Out of other SVM implementa-
POS based, Open-Class Class-POS tions LIBLINEAR produced best results. For this study, random data from seven native languages selected from the ICLE corpus. The highest performance was obtained for word-level n-grams with an accuracy of 0.8971.
Gebre et al. 2013) employs linear SVM, logistic regression and perceptron (as baseline) for the native language identification (NLI) and achieved accuracy of 0.814 for eleven languages of 'The Test of English as a Foreign Language" (TOEFL) data Bet(Bhanchard et al., 2013). Features used includes; word n-grams, POS n-grams, character n-grams and spelling errors. TOEFLI1 has become standard benchmark in NLI tasks since its introduction for the NLI Shared Task 2013. The main limitation of the TOEFL1 dataset is that is collected in more controlled environment (i.e., exam for English).
Cimino & Dell'Orlette (017)) utilizes a novel stacked classifier approach where linear logistic re- gression based sentence feature classifier is stacked with SVM based document feature classifier with standard lexical, stylistic and syntactic features. However stacked classification approach has gained minor gain compared to unstacked. Best results of the NLI shared task 2017 reported for this approach with Fl-score 0.8818 for the TOEFLII dataset.
Kulmizev et al. (017)) introduced 'Groningen' system for the NLI Shared Task 2017 which out performs employing linear SVM for character 1-9 n- n-grams with the Fl-score of 0.8756. Authors have reported that several experiments done with ensemble approach and other features such POS, word, lemma n-grams, skip-grams; and those failed to match the performance of character 1-9 n- gram system.
Goutte & Léger (017)) explored use of voting ensemble SVM models with character, word and POS n-grams. Authors confirms that ensemble methods provide minor but systematic predictive perfor-
2

---

Under review as a conference paper at ICLR 2022
Table Summary of the current literature
Accuracy
Research
Corpus
/FI- Score
#Classes
Approach
Koppel
ICLE
0.802
5
Support Vector Machines (SVMs) on var- ious stylistic features namely; function words, letter n-grams, and errors and id- iosyncrasies
(2005)
Bykh Meurers
& ICLE
0.8971
7
SVMs with recurring n-grams of three dif- ferent classes (word based, POS based, Open-Class-POS-benn
(2012)
as features Linear SVM, logistic regressions and per- ceptron as the linear classifiers with word n-grams, POS n-grams, character n-grams and spelling errors
Gebre et al.
TOFFLI1
0.814
11
2013)
Cimino & TOEFLI1 /0.8818
0.8818
11
Two-stacked sentence- and document- feature based classifier architecture. Out- put of the sentence-level linear regression model is being used by document-level SVM.
Dell' Orletta 2017)
Kulmizev
TOFFL11
0.8755
11
Linear SVM and character 1-9 n-grams
al.
2017
10.8756
Goutte
TOEFLI1
0.8736
11
Voting ensemble SVM models approach with character, word and POS n-grams fea- tures
Léger
/0.8740
(2017)
Markov
TOEFL11
0.4883
SVM with one-vs-all (OvA) multiclass ap- proach. Abstract POS n-gram and punctu- ation marks (PM) features have been used.
al. (2018)
and ICLE
(TOEFLI))
(TOEFLI))
& 0.6948
&
7
(CI.E)
(CLE) 11
Malmasi & Dras (2018)
TOEFLII, EFCAM- DAT, ASK, JCLC
0.871
Supervised multi-class classification ap- proach with feature including character, function word, POS n-grams, dependen- cies, CFG rules, adaptor grammars and TSG fragments used
(TBFLII))
(TOBFIII)
mance gains. Highest Fl-score performance was with best-vote approach consisting 10 models is 0.8740 for TOEFL11 dataset.
Markov al. (2018) has used punctuation-basel features with POS n-grams for his experiments and accuracies of 0.4883 (TOEFL! and 0.6948 (ICLE) reported for best performing settings. SVM with one-vs-all (OvA) multi-class classification approach has been used to conduct these experi- ments.
Malmasi & Dras 2018 employed a supervised multi-class classification approach and incorporated several corpora, including; TOEFL11, EF Cambridge Open Language Database Corpus (EFCAM- DAT), ASK Corpus (Andrespräkskorpus, Second Language Corpus), and Jinan Chinese Learner Corpus. The features extracted were; word/lemma n-grams, character n-grams, function word n- grams, POS n-grams, dependencies, CFG rules, adaptor Grammars and TSG fragments. Highest accuracy of 0.871 is reported for the TOEFLII dataset.
According to the literature evident that most of the researchers in the computational linguistic community have employed ICLE in the early stages and TOEFL in the latest NLI tasks (refer Table T which both were collected more controlled environments. Finally, due to above controlled environments; being unable to capture real nativism in written texts certainly inflate the performance of ethno nationaati identification. On the other hand, identifying the nation the English writer opposed his native language will be equally beneficial for author profiling well. Thus,
3

---

Under review as a conference paper at ICLR 2022
introduce use of International Corpus of English (ICE) for the ethno- nationnttin identification. To the best of our knowledge, none of the researches have been conducted to determine the writer's country of origin based the International Corpus of English (ICE) corpus. Moreover, very limited number researchers have identified the significant features which helps to distinguish Sri Lankan English writers using large corpus like ICE. One limitation of these researches raised by the authors Tofight et al. (2012) is that, since for most of the web-based applications, automatic spell-checker has been applied, idiosyncratic features including misspelling and other anomalies are ignored. This may hide some of the features which will be useful in identifying the real categories. Nevertheless, in current context, usage of spell checkers and grammar checkers can be seen frequently. Thus, this will be common limitation in similar types of researches.
RESEARCH METHODOLOGY
Supervised learning approach was chosen as it is suitable in classifying text documents into classes more accurately if the classes are known and the data cording to the literature most promising ML algorithms for the text classification such as; Support Vector Machines with Stochastic Gradient Descent Optimizer (linear SVM with SGD), Multino- mial Naïve Bayes (MNB), Decision Tree (DT) and Random Forest (an ensemble approach) have been employed. For this research mostly Scikit-leam(Pedregosaa et al., 2012) has been used for pre- processing. feature extraction and classification tasks. Scikit-learn is a Python module for machine learning which provides state-of-the-ar implementations of many well-known machine learning algorithms, and maintains easy-to-use interface(Pedregosa al. 2012). The workflow of the research is depicted in Figure (Left).
labelled (Slotte, 2018). Hence, ac-
The main assumptions of this research are authors from the same ethno-nationality share the same linguistic features in their writing and will often have an influence on the way they express them- selves in writings (Jain et al. 2017).
The research questions addressed this research are;
Ql. How can texts produced by English writers in given ethnoo-ationallit be captured from existing corpora?
Q3. Which machine learning techniques can gainfully employ the extracted data to identify country- of-origin of English writers?
Data Colection
Chutwing
Training (Seen Data) 70%
Testing
Pre Procesiing
Hu Nuwreetia USl,, top Frall dimmmm
Language
Total
(Unseen Data) 30%
#Docs
#Words
#Docs
#Words
#Docs
#Words
Nigoria
177
342,144
123
237,053
54
105,091 091
Fattire Extraction Wecooriation
EF Vicerie
Philippines
178
401,871
124
281,000
54
120,871
Jamaica
200
412,138
286,651
Singspore
200
308,637
Charthastan Bulding
Velschu
USA
200
428,943
140
299,509
60
129,434
Canada
200
427.064
140
300,106
60
126,948
Irelard
200
420,515
140
294,497
60
126,018
Parmeter taing Made lnkkng in whicier
$-6d Dan Wiltatio
India
200
421,071
140
297,259
60
123,812
Sri Lankca
200
467,711
140
320,142
60
147,569
10
Hong Kong
200
537,153
140
375,704
60
161,440
Baiiig
Model Exiludionn
Total
1,955
4,297,450
1,367
3,000,558
588
1,296,992
Figure 1: (Left) Workflow of the research methodology. (Right) Document word distribution of ICE corpus
#Docs number of documents, #Words number of words
4

---

Under review as a conference paper at ICLR 2022
3.1
DATA COLLECTION
The main dataset employed in this study is International Corpus of English (ICE) (Greenbaum & Nelson 1996; Kirk & Nelson, 2018) which comprises several corpora from different countries. This data-set built with the intention of providing resource conduct comparative studies of English used in different countries where English is the native language or the second language. In order maintain consistency among each country corpus, common corpus design, corpus size and common scheme for grammatical annotation have been followed (Kirk & Nelson 2018). This data- consists of both written and spoken English texts. Only written texts have been considered for this study. Written texts were gathered from many areas, such as; student writings, letters, academic writing, news reports, instructional writing, persuasive writing and creative writing (Kirk Nelson 2018).
The authors and speakers of the texts are aged 18 or above, educated through the medium of English, and were either born in the country in whose corpus they are included or spent the majority of their lives there, or moved there an early age, and received their education through the medium of English in the country concerned. Written English corpora from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA have been collected and used for this research. Each country is consisted of ==200 text documents (=2,,00 words per document) (refer Figure D..
3.2
DATA ANALYSIS
As the initial step, 30% (testing data) the total data each country corpus was kept as unseen or un-touched data. The remaining 70% of data (training data) was analyzed in order to identify the pre- pre-processing requisites.
3.3 DATA PRE-PROCESSING
Raw text-files with specific markups are used as input. These files are cleaned as per the ICE-Corpus markup guide Nelson (002). In addition, strippaccents' parameter for the TF-IDF vectorizer and 'Unicode data normalize NFKD' were used. Moreover, URLs, XML tags, e-mail addresses, cen- sored data, line feeds were removed. The data set contained HTML entity encodings and those were decoded(refer Figure [T. In order to avoid country names, nationality, currency and popular cities being trained features, all the country specific nationality, cities and country names are removed as pre- pre-proccesing step.
3.4 CLASSIFICATION MODEL BUILDING
The training data set were used for model building, parameter tuning, training and selection. This labelled training data was tokenized using the TF-IDF vectorizer. Vectorized output was fed to ML classifier to train the model. Further, k-fold (k=3) cross validation technique was used on the training data. Testing data set was kept hold to feed and evaluate the classification model later stages. Models were built based on two classification strategies;
Flat Classification Strategy
Flat classification refers to a single classifier at the root level as the decision point depicted in Figure Top). This classifier will handle all classes as per its classification approaches such as one- mne-vs-rest, one-vs-one..eetc. In this study 'one-vs-all' approach has been followed. Hence, the dataset was trained for 10 classes (countries) together.
Sub-Category Classification Strategy
Sub-category classification employs number of classifiers as depicted in Figure 2 Bottom Left) and 2 Bottom Right). For each sub-category specialized in solving subset of the problem in which each classifier trained. Each sub-category was trained separately. As depicted, some binary (marked with black boxes in Figure 2 and some multi-class classification models (marked with grey boxes in Figure 2) All the experiments the sub-categories which were tested depicted in Figure
5

---

Under review as a conference paper at ICLR 2022
TOME
Imeorkk
CAPCIC
Nan SATTT
Ramppa
Harphhn
Figure 2: (Top) Flat classification strategy setup. (Bottom Left) Sub-category classification setup based on the Kachru's 's concentric circles. (Bottom Right) Sub-category classification setup based on the geographical distribution
DPs/tbb Categiry
Type
Destription
Based the Kachen's Theary
Netive va Non--istive
Beary
Nathe fineer didck Nen (Duter circle
Byarded dide
Costtries
Iner moustree (aaada treand)
Uvjer ixpardbol unle
Espanded Eountrias
Sparded dide coustries
ve Non-Ada
Beary
daise Hon-daa clams
Eourtries
Non-Seuth folan
Mubiclas
Nan-eurh As
Kang us Singiaar))
Non-Aite
Mili-clan
Nav-bala das Purope to Africa vo North Amarical
North Ameriion
Mubi-clas
Worth-Ame- countrees (USA Casada
Figure 3: Identified sub-categories based on Kachru's Theory and Geographical distribution
TESTING AND EVALUATION
combination of both 'hold-out validation' and k-fold cross-validation was used in order to reduce biasness for training and testing data set (Slotte 2018). Hence, the 70% of the training data set, 3-fold cross validation technique was used for model building, training and selection and 30% of the data was kept for hold-out validation validate the machine learning model. For each decision point (where a single classifier needs to be selected) 16 experiments (4 ML x 4 n-grams) have been carried out. For each ML algorithm, word level (1-1,2,3,4) n-grams were tested. Further, performance accuracy, FI F1-score, precision and recall were calculated to select the best performing ML model.
RESULTS AND DISCUSSION
Q1: How can texts produced by English writers given ethno-nationality be captured from existing corpora?
Flat classification strategy
The best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni umi-gram model. Furthermore, overall balanced accuracy of 0.7620 and macro average F1-Score of 0.76 obtained. Score breakdown for each country is depicted in

---

Under review as a conference paper at ICLR 2022
Figure 4[ In addition, India has the highest Fi-score of 0.94 while USA & Sri Lanka holding the lowest Fl-score of 0.68. These statistics are further verified by the confusion matrix depicted in Figure Rightt) Accordingly. all the documents of the India have been classified correctly. The confusion matrix further depicts that the model makes most of the mistakes at the classification of USA vs Jamaica pair. Certainly, reason could be the geo-graphical proximity between these two countries.
CLar SYM
Accuracy: 0.7636
Belanced Accuracy' 0.7620
Analyeert Word mamm Q
Training Time 1.ds
Teeting T'mee 0,07g
Caneda
Precision
Recall
F1-score
Support
Hinghang
Keng Indix
0.94
Philippines
Singapoce
Segapn
30
Lanke
USA
Heighh
Macro
Weighted
Figure 4: (Left) Test Results for the Flat Classification Strategy. (Right) Confusion matrix for the Flat Classification Strategy
Sub-category classification strategy
In sub-category classification, each sub-category can be perceived as a Decision Point (a single classifier) in the hierarchy of decision tree. Figure 5 (Left) depicts the test results of the lected ML models for each sub-category. Accuracies of different sub-categories have varied in between 0.6224~1.000 and Fl-score ranges between 0.49~~1.00. Based the Kachru's concentric theory, lowest F1-Score of 0.71 is for "Outer vs Expanded" sub-category. On the other hand, 'Ex- panded sub-category (Hong Kong vs Jamaica) has the highest Fl-score of 0.92. In geo-graphical distribution-bar model, lowest Fl-score of 0.49 is for "Non-Asia" sub-category, while 'South Asian countries' sub-category holding the highest Fl FI-score of 1.0.
Ang
Bonge
Time
Tims
Beund the Kachnu's Theary
Mothe
Word
9.3h
0.03
Inner Countries
DST
Word
0.85
0.55
Expandad
MWN
0.81
637
Outer Countrio
5WM
Word
0.80
077
Caerties
Word
5.5
Beaed the Geographical Diaribution
Asts Nan-
SWM
111
0.44
Non South Adia
South Asio
SWM
WWord
011
0.06
9.
0,9957
0.90
2.5
0.00
South Autin
Aslan Countries
Word
0.7074
0.83
032
North American
Figure 5: (Left) Test results of the sub-category classification strategy based selected ML model. (Right) Learning curve for the Flat Classification to assess the generalisability
Generalisability of the model
Assessing the generalizability of the ML models is crucial. Over-fitting leads to poor generalizability of the model. Hence, in order to detect and prevent over-fitting, 'hold-out validation' and k-fold cross-validation used. Further, learning curves the training data set have been drawn to diagnose whether the model is over- fitting or under fitting. The red line represents the training score (trained and tested using same data) and the green line represents the cross-validation score(k=3) obtained
7

---

Under review as a conference paper at ICLR 2022
for varying number of samples. Cross validated score curve is flattening against the training score curve, as depicted in Figure Rightt. Thus, this model does not over- wver-fitting or under-fitting for the dataset.
Most significant features of the model
Most significant features identified for the flat classification is depicted in Figure These features are computed base on the coefficients assigned. Some of the features are driven by the cultural and geo-graphe -ro-graphicall differences of the particular country. For instance, Ireland: northern, christmas, groundwater, queen. and Singapore: business, batik, population. Nigeria: donut(for do not), federal, god..etc.
>=Canada top festizes
y ye:Bengkeng top Seatures
y-ladia tup features
y-Ireland top fxctares
yejomaica top festires
+2. 120
conallaio
+0 151
+3 1444
+2. 44:44
Jasn
+2.725
lockic
3005
ertp
con
milel
chrsstaar
12:085
moockiiioo
1-7753
coureatioos
921
bods
10:2 08.75
12 0600
Coanacts
12
11:51
peztzto
+1 053
Jojjap
+0.3 02722
Oblloge
givindivuter
13.50 11,503
2000
capep
jeponisss
+0.009
13.085
+1.330
+1 11:0
mrechotion
+0.075
dult
+3. 61
In
+3.704
Seoge
+1 263
concondia
+1.324
polnisa
+0.07]
grof
3.23
garda
+3 +3.710
persons
mcclia
classe
community
835
+6.678
beoks
1.5
key
11.41
zairo
11:501
Lowloon
+0.68
bas
134635
QEoca
13.615
IDL
+1 405
DoYa
446
captalsn
+0.635
0ar5
40.40
Jode
+3 10000
CTED
wylh
Internatioon
+0.626
11.555
groups
2-4
po
+0.000
l0tter
bdress
+3.512
Staprang
1,070
stAre postive
25353
more posthhe
17022
RoY posttise
14335
rore pocmiee
14147
three posttive
uore sugative
marc nogatte
more segetive
roget
esurc
1.250
3.030
1.,011
3-103
stali
260
was
this
0.526
o8
000
programa
150
.611
hopo
1 1.50
notumnl
0.911
t2at
-3 1655
1.351
V0
that
1.243
+BUSS>
y=Nigeela top
P=PPhllpppam top
y g=Niggggeme
top
y=
tog
top fraturm
Wedght?
Postum
Welght
Desture
Weight
Welght"
Teetur
Weight
Pestum
12:25
state
3ta
that
11.702
collle
bedik
planlottes
coportabt
+1.091
+1301
41/073
prpuiaticn
11:456
8203
+1.77 172773
kelinde
+3 E.55
+3. 1.304
Staeet
2000
(33
Speliteeer
43-750
+3.027
43 2500
10:00
Samila
43 11.50
43.00 11.661
Gonat
+3 1.554
11,523
ken03
11.775
ago
4.00
Cancer
41.00 13.3313
Tenexifeeding
+3 4.4
melf
63. E27
baby
2013
photoggrppyy
43.385
god
+1.407
+2:407
czen:
11:525
111342
Cluse
+3.570
Iesdii-opped
+3 114444
+3.470
150
asies ac1
+3 531
Ieillg
43.501
comrugus
+3.43
pecceal
43-153
poslacr.p
41.551
Cholectand
43.323
Inkaad
+3. 155
govertor
+3 12394
Jeinas
1.115
S7A5
11.212
Solla
272
excupe
+3.80
LEDO0O
obeect
TUES
92 23:276
+3.445
+3,300
43.501
41,440
lockd
13770
mome postive
15021
more pwutthot
15356
IEIE posttive....
more possthun
mome pas
42412
thona
pengiomas
Wisst
201
dun
11.08
MAL
3.260
11408
Jur
3 0.000
eniieeiit
-3,00
Stadonts
3,503
5.59
Figure 6: Most significant features of the flat classification
Q2: Which machine learning techniques can gainfully employ the extracted data identify country-of-origin of English writers?
Comparison of ML algorithms based on the selected best parameters are analysed on top of the test data and results are depicted in Figure 7 Linear SVM with SGD optimizer seems to be outper- forms in most cases. Moreover, this behavior also verified in the previous work by Kulmizev et al. (2017), Koppel et al. (2005), Ekaterina (201T), Gebre et al. (0133), Bykh & Meurers (2012). Furthermord Furthermore, it is noticeable that the DT is under-performing for most cases.
CONCLUSION AND FUTURE WORK
With various accuracy levels the literature has proven that the ethno-nationality of a person can be identified using their written English texts and this area of research has lot of practical applications and usage. However, as discussed in the literature review still those researches comprised with lot of limitations as stated above. Therefore, those identified limitations have re-fre this research to obtain solutions to the identified research problem and defined research questions.
8

---

Under review as a conference paper at ICLR 2022
Performance Comparison of ML Techniques
04
02
countres
Nathe?
Outervs Expanced
Outer?
Depanded?
Ads?
SouthAsst!
Lanka?
Non-Ada?
America
North
0.7036
0.8912
0.7822
0.8418
MNB
0.7333
0.8384
0.8333
0.148
0.7228
0.0022
0.709
0.7511
0.7245
0.7923
0.9677
0.6795
Figure 7: Performance comparison of each ML technique
Our work on Ethno-nationality Identification confirms that linear SVM with SGD optimizer trained with word n-grams can yield a higher level of performance. When determining country-of-orde it is essential to identify set of features which are unique to each author or particular group of authors.
One spectacular restraint would be the usage of spelling and grammar checkers when writing En- glish. This limitation can be overwritten when using spoken English text. Hence, as future work transcribed text be considered to identify the country-of-origin of the English writers. Further, needs to focus more on features which have more pedagogical value and cross corpus generalizabil- ity should be assessed to examine the extendibility of the model.
ACKNOWLEDGMENT
are deeply grateful for International English of Corpus (ICE) project owners and maintainers of the respective countries for making available this corpus to the research community without any license fee.
ETHICS STATEMENT
The Contributors of the respective ICE Corpus countries have informed about the data collection procedures and objectives. All the identifiable named entities of the data have been anonymized in order disjoint any relation of individuals and organizations to the data. Hence, one, including the researchers, will be able to link data to a specific individual. Outcomes of these type of researches categorization of individuals based ethno-nationality discrimination. However, author-profiling is gaining pace and the authors of this paper admire the true potential of such categorizations.
raise concerns its usage and
REPRODUCIBILITY STATEMENT
The implementation details this experiment available as a supplementary material along with the submission. Kindly that the supplementary submission does not include the complete dataset. Authors can produce the complete data-set used upon a formal request.
All the raw-data for each country in ICE corpora is available text files. Files preprrcessed through a jupyter notebook and results were written into a csv called 'ice-merged.csv'' These pre- processing details also available with the supplementary materials and for further details please refer to README.tt txt file in the root level.
REFERENCES
Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. Toefl A corpus of non-native english. Ets research report series, ETS Research Report Series, 2013.

---

Under review as a conference paper at ICLR 2022
Kachru Braj B. Asian Englishes: beyond the canon. Hong Kong 2004.
Hong Kong University Press,
Serhiy Bykh and Detmar Meurers. Native language identification using recurring n-grams in- vestigating abstraction and domain dependence. In Proceedings of COLING 2012, pp. 425-440, Mumbai, 2012.
Andrea Cimino and Felice Dell'Orletta, Stacked sentence-document classifier approach for improv- ing native. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educa- tional Applications, pp. 430 437, Copenhagen, Denmark, 2017. Association for Computational Linguistics.
Kochmar Ekaterina. Identification of a writer's native language by error analysis (mphil's thesis). Master's thesis, University of Cambridge, Cambridge, United Kingdom, 2011.
Brown Eric. Native and non-native english speaking esl/efl teachers in sweden: A study on students' attitudes and perceptions towards the teaching behavior of native and nonnative english speaking teachers. Technical report, English C, 2013.
Dominique Estival, Tanja Gaustad, Son Pham, Will Radford, and Ben Hutchinson. Author pro- filing for english emails. In Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics, pp. 263-272, Melbourne, Australia, 2007.
Binyam Gebrekidan Gebre, Marcos Zampieri, Peter Wittenburg, and Tom Heskes. Improving native language identification with TE-IDF weighting. Proceedings of the Eighth Workshop on Inno- vative Use of NLP for Building Educational Applications, pp. 213-223, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
Cyril Goutte and Serge Léger. Exploring optimal voting in native language identification. In 7 Association for Computational Linguistics, pp. 367-373, Copenhagen, Denmark, 2017. doi: 10. 18653/v1/W17-50411
Sylviane Granger. ICLE international corpus of learner english, 2014. LINDAT/CLARIAH-CZ digital library at the Institute Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Sidney Greenbaum and Gerald Nelson. The international corpus of english (ice) project. World Englishes, 15(1)33-15, 1996.
R. Jain, Venkatesh Duppada, and S. Hiray. native language identification. In FIRE, pp. 224-231, 2017.
Hierarchical ensemble for indian
Muller Jerry. Us and them: The enduring power of ethnic nationalism. Technical report, The
Catholic University of America, 2008
John Kirk and Gerald Nelson. The international corpus of english project: A progress report. World Englishes, 37(4)6697-716, 2018. doi: https://doi.org/1..I1 hww.l//www....3
Moshe Koppel, Kfrr Zigdon, and Schler Jonathan. Determining an author's native language by mining text for errors. In KDD' '05 Chicago, II,, USA, 2005.
Artur Kulmizev, Bo Blankers, Johannes Bjerva, Malvina Nissim, Gertjan Noord, van, Barbara Plank, and Martijn Wieling. The power of character n- n-grams in native language identification. In Pro- ceedings of the 12th Workshop Innovative Use of NLP for Building Educational Applications, pp. 382-389, Copenhagen, Denmark, 2017. Association for Computational I. Linguisties.
Shervin Malmasi and Mark Dras. Native language identification with classifier stacking and ensem- bles. Computational Linguistics, 44(3):403 446, sep 2018. doi: 10. 10.1162/coli
Modiano Marko. Language Learning in the Multicultural Classroom: English in a European and Global Perspective. Studentlitteraurr AB; UK ed. edition, 2009.
Ilia Markov, Vivi Nastase, and Carlo Strapparava. Punctuation native language interference. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 3456-3466, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.
10

---

Under review as a conference paper at ICLR 2022
Gerald Nelson. Markup manual for written texts. Technical report, ICE, 2002.
Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Coumapeau, Matthieu Brucher, Matthieu Perrot, Edouard Duchesnay, and Gilles Louppe. Scikit-leamn: Machine learning in python. Journal of Machine Learning Research, 12, 01 2012.
Jan Sapp. Race finished. American Scientist, 104, 01 2012. doi: 10.151 1/2012.9..
Hans Olav Slotte. A shallow neural network architecture for native language identification (mas- ter's thesis). Master's thesis, Norwegian University of Science and Technology, Department of Computer Science, Trondheim, Norway, 2018.
J. Tetreault, D. Blanchard, and Aoife Cahill. A report on the first native language identification shared task. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educa- tional Applications, pp. 48-57, 01 2013.
Parham Tofighı, Cemal Köse, and Leila Rouka. Author's native language identification from web- based texts. International Journal of Computer and Communication Engineering, pp. 47-50, 2012. doi: 10.7763/IICCE.2012. V1.14.


Reviews:
[{"review_id": "ajUWelUdnjF", "reviewer": "Reviewer_bgYR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The task is of interest\n\nCons:\n- Representing the documents as bags of words is overly simplistic. The experiments and results would normally be considered as baseline.\n- There is a line of works by Shuly Wintner that concerns authorship and style modeling in non-native languages, and there may be more related works that are missed here.\n- Results must be compared against other approaches, include ablation study etc.", "summary_of_the_paper": "The work aims to automatically determine the country of origin of authors given their English texts. \nExperiments are conducted using the International Corpus of English (ICE corpus), which includes texts of authors from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA; there are 200 text documents per origin country. \n\nIt is assumed that authors from the same ethno-nationality share the same `linguistic features'.  The documents are represented as TF-IDF weighted vectors of words, and a classifier is applied (it is not very obvious which classifier exactly, perhaps I missed this detail?).\n\nIn classification, the authors apply text classification, either directly into the 10 countries of origin, or using two different class hierarchies. \nIn addition to multi-class classification, they also conduct binary classification (e.g., native vs. non-native English speakers).\n\nThe paper then presents the classification results. Salient features (words) are detailed.\n", "summary_of_the_review": "The paper is technically below ICLR threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NPzh06HjWJ", "reviewer": "Reviewer_hVHm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The research problem itself is quite intriguing (but brings many connotations which should have been better explored in the paper), but I really fail to see how this paper fits ICLR's scope and expected submission quality. In other words, there are several key weaknesses to the paper.\n\n1) Starting from the motivation, the authors provide a very vague motivation why detecting ethno-nationality should be more (or equally) important than detecting a native language of the English text writers - how is this problem exactly different than the previous one? Why do we need both, how can both be combined? Do we actually need a different methodology (since the approach is a range of simple classification-based methods) to tackle this problem?\n-- The authors claim: \"On the other hand, identifying the nation of the English writer opposed to his native language will be equally beneficial for author profiling as well.\", but that is such a vague statement - there's no evidence or citations to previous literature underpinning this claim.\n\n2) Suitability to ICLR: I don't see how this paper aligns with ICLR at all. This is in its essence a straightforward application of simple ML-based techniques to a socio-linguistic problem. In essence, this makes it a truly computational linguistics paper, even leaning towards more human-centered linguistic research. It uses some computational tools to study language and linguistic text. There's nothing in the paper advancing representation learning methodology or even advancing our computational tools to study linguistics - such work is tangentially relevant also for conferences such as EMNLP, not to mention more general ML conferences. A good venue for this type of work (not judging its quality, but relevance only) is e.g., COLING.\n\n3) There is no methodological contribution at all: the main contributions of the paper are mostly a redesign of the research problem, and the use of another dataset (which wasn't even collected by the authors, but simply used for a new purpose).\n\n4) The paper puts a lot of emphasis on Related Work, but still bypasses all the more recent work on using Transformer-based neural models for the problem of native language identification from English texts. The coverage of related work stops in 2018, while there are newer papers such as: https://aclanthology.org/2020.coling-main.159.pdf (Lotfi et al., COLING 2020) or another paper: https://aclanthology.org/2020.icon-main.35.pdf\n\nThere are more concerns, but these are already sufficient to signal that the paper cannot be accepted in its current form and format.", "summary_of_the_paper": "This paper presents a simple classification-based approach to a newly defined problem: determining ethno-nationality of writers of English text data. The problem of determining ethno-nationality can be seen as a direct extension of the research problem of identifying the writers' native language based on written English text. The authors define ethno-nationality mostly as country-of-origin and aim to distinguish this demographic feature from native language as another feature. The empirical validations are mostly applications of standard (and even obsolete) classification models on the newly collected data for the introduced problem.", "summary_of_the_review": "This paper, while tackling an interesting computational linguistic problem, does not bring anything new, and there are detected issues with its thematic fit with ICLR, its lack of novelty and methodological, theoretical or empirical improvements, some problems with presentation (a stronger motivation is needed, the results are presented as snapshots of figures!), etc.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "MrA0k7To6fD", "reviewer": "Reviewer_Ak7m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the motivation for studying ethnonationality is a bit weak. The main argument is forensic linguistics and marketing. Both motivations are, in my view, ethically questionable, but the study of ethnonationality also has direct applications for *improving* the fairness of NLP systems. \n\nI like the idea of moving away from TOEFL/ICLE datasets for native language identification - in theory. I previously did research on social media data, and the task suddenly becomes a lot harder. The challenge leaving the TOEFL/ICLE domain is that suddenly a lot of open class words become give-aways of the target. People in Jamaica talk about Jamaica. People in Australia talk about Melbourne, surfing, and kangaroos. In the context of forensic linguists, Australians are not gonna talk about surfing or kangaroos, especially not if they want to hide where they’re from. For this reason, it is common methodology when moving out of a controlled setting (like TOEFL/ICLE), to remove open class words or rely only on character bigram or trigram statistics. The authors fail to do so. \n\nOther problems: a) The protocol is weak; split data temporally rather than randomly to make sure your results will generalize. b) The description of state of the art is out-dated, e.g., see [0-2]. \n\n[0] https://aclanthology.org/2020.coling-main.159.pdf\n[1] https://aclanthology.org/2020.icon-main.16.pdf\n[2] https://arxiv.org/pdf/2010.01869.pdf", "summary_of_the_paper": "The authors rely on the International Corpus of English to evaluate document classification algorithms on the task of recovering ethnonationality. They evaluate standard algorithms such as decision trees and support vector machines. ", "summary_of_the_review": "Several methodological problems in how data was preprocessed. Motivation is weak, and models are out-dated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Kj2bAjkiEK", "reviewer": "Reviewer_k2RW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to determine the country of origin of English language writers. It uses documents from the International corpus of English (ICE) and employs various learning algorithms and text-based features. The findings are that linear SVM with unigram features works best, that some countries are better distinguished than others, and that some features are more informative than others. \n\nThe main premise of this work is that previous work in this space used controlled settings such as English learners texts or standardized examples, while this work examines more naturalistic settings. This is a reasonable motivation. However, besides that, the contributions of this work are very limited: it uses standard algorithms and features, there isn't much analysis of the results, and so the reader does not learn much new from this study. It also doesn't seem like a very good fit for ICLR. ", "summary_of_the_paper": "This paper aims to determine the country of origin of English language writers. It uses documents from the International corpus of English (ICE) and employs various learning algorithms and text-based features. The findings are that linear SVM with unigram features works best, that some countries are better distinguished than others, and that some features are more informative than others. \n\n", "summary_of_the_review": "This paper is an empirical study of standard methods on the problem of classifying a writer's country of origin. It has limited technical contributions or insights. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use longstanding statistical learning techniques to identify the nationality of the author of a text.

Reviewers agreed that this work is a poor fit for ICLR, as there is nothing here that advances our understanding of representation learning. Reviewers were further concerned about the soundness of the claims, raising issues about data contamination and comparison with prior work. 

Finally, reviewers pointed out (correctly in my view) that work that aims to infer protected identity characteristics of non-user human subjects should be held to an especially high ethical standard, and needs a highly persuasive cost-benefit analysis that defends why the problem is ethical to study at all. The available discussion of ethics is not up to this standard.
Decision: Reject