DEIM Forum 2016 E6-1

Robust Chinese Native Language Identification with Skip-gram
Lan WANG†

Hayato YAMANA‡†Graduate School of Fundamental Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku Tokyo, Japan‡Faculty of Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku Tokyo, Japan‡National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, Japan
E-mail:†wanglan0605@yama.info.waseda.ac.jp,‡yamana@yama.info.waseda.ac.jp

Abstract Native Language Identification (NLI) is a task in which an author’s native language can be identified by his/her
essays written in a second language. In this work, a supervised model is built to accomplish this task based on a Chinese learnercorpus. This research is based on the authors’ previous work, which highlighted the significance of noisy data elimination andterm weighting techniques in pursuing higher accuracy. In this work, the innovation points include: (1) it is the first work to
explore skip-gram as features in NLI field. (2) By dividing the dataset into training, tuning and test subset, evaluation is morerobust than that in other Chinese NLI works. By employing a hierarchical structure of linear SVM classifiers, a state-of-the-art
accuracy of 75.3% is achieved by our proposed model.
Keyword Text Mining，Author Profiling，Machine Learning
Firstly, as a large number of people have shown their

1. INTRODUCTION
There is no doubt that a Chinese native speaker can judge

interest in learning Chinese, the demand of Chinese

whether a person is a native speaker or not , easily based onteaching services is increasing progressively. Secondly,

his/ her speaking patterns. In addition, sometimes, it issince Chinese is an ideographic language, in which strokes

easier to guess the mother to ngue of a person from his/her

and their combination need to be remembered, it is morepronunciation than his/her speaking patterns. A large

difficult to learn than an alphabet ic language, such as

number of studies have indicated that the same principle

English [3]. Thirdly, most of the current NLI studies focus

can be employed in texts [1]. Further, since native language

on essays written in English. Therefore, research on

is one of the authorship attributes, it is possible to estimate

Chinese NLI becomes indispensable.

traits of different native language speakers ba sed on their

However, there are two major issues in the modern NLI
research field. First, features exploited are restricted toessays written in a second language.
Native Language Identification (NLI) is a task in which

eitherwriter ’s

beinformative enough [7] [9] [10]. Second, it is not robust

recognized by his/her essays written in a second language

enough to adopt current evaluation methods of Chinese(hereafter, L2). In a nutshell, it can be considered as a

NLI on the ground; applying test data to tune parametersclassification task where machine learning methods assign

will reduce the applicability of the algorithm [12].

anativelanguage(hereafter,

L1)

can

labels (native languages) to different objects (essays).
While NLI has been widely used in security, such as

lexicalorsyntactic

features,

which

are

not

In order to solve the two problems mentioned above,
besides

automatic

noisy

data

elimination

and

term

identifying false information distributors on SNS andweighting technique proposed in the authors’ previous

detecting phishing websites [2], many applications canwork, we integrate (1) skip-gram feature, which contains

adopt Second Language Acquisition (SLA). In general,

both lexical (word level) and syntactic (grammatical level)

SLA can shed light on whether a person’s L2 learni ng is

information, and (2) a more robust evaluation method of

influenced by his/her L1 background. Moreover, SLA

dividing the dataset into training, tuning and test subset .

investigates the difference between L2 writings (such as

The paper is structured as follows: Section 2 describes

error patterns) of learners with different L1 background,

representative prior works related with NLI; Section 3

which can be applied to provide language teachers withshows the corpus used in this work; Section 4 presents our

instructive advice to guide students in a more sophisticatedproposed methodology for NLI; Section 5 discusses theway.

evaluation of our experiments; Section 6 gives our

The motivation of this work is based upon three aspects.

conclusion and forecast for future work.

syntactic grammar constructions. They applied their
method to Chinese Learn Corpus (CLC) [11] and achieved

2. RELATED WORKSNLI was first proposed by Koppel et al. [4] in 2005.

70.6% accuracy.

Combining a feature set of character n -grams, POS bigrams,

Wang et al. [12] first highlighted the significance offunction words, and errors, they trained linear support

noisy data elimination and term weighting technique invector machines (SVMs) to classify English essays of

pursuing high NLI accuracy. They put forward a modelInternational Corpus of Learner English (ICLE) 1

Ver.1 tointegrating both automatic noisy data elimination andachieve 80.2% accuracy. They showed that linear SVMs are

BM25 [13] [14] term weighting method. After training

well suited to an NLI task.

hierarchical linear SVM classifiers to classify Chinese

Even though NLI was initiated around 10 years back,notable works have been published in the recent years.

essays of Jinan Chinese Learner Corpus (JCLC) [15],77.1% accuracy was gained.

Bykh and Meurers [5] defined the n-grams occurring in at

However, there are two problems to be solved. To begin

least two essays as “recurring n -grams.” A recurring n-

with, features in existing NLI studies are restricted togram has the advantage to reduce noisy data caused byeither lexical or syntactic features. Apart from this, by

spelling errors. After mapping essays into the vector space,using test data to tune parameters, the evaluation method

they trained linear SVMs to classify English essays of theof current Chinese NLI [12] is not robust enough to verify

ICLE Ver.2 dataset into seven different native languages,

the applicability of the algorithm.

achieving 89.7% accuracy. Our proposed method is
inspired by their paper, i.e., adopting another form of

3. DATASET

recurring n-grams. That is, in order to shorten the training

In order to compare the performance of our proposed

time by dimensionality reduction, we adopt n -grams

model with the author ’s previous work directly, we

appearing in more than ten essays instead of at least twocontinue adopting the Jinan Chinese Learner Corpus

essays.

(JCLC) dataset as same as [12].

However, the ICLE dataset used in the above works

JCLC contains a total of 8,739 essays written by students

faces the problem of topic bias. Thus, the TOEFL11 dataset

in an examination or as homework. We extract all essays[6] was published during the first NLI shared task [7] inwith explicit native language metadata in the dataset. The

2013. It is designed particularly for an English NLI task

L1 distribution in JCLC is listed in Table 1. Besides native

without topic bias compared with other corpora, such as

language, metadata, such as proficiency level and gender,

ICLE. In this task, in order to improve the accuracy, Gebre

are included in this corpus, however, only native language

et al. [8] proposed a new perspective to tackle an NLI task,

metadata is taken into consideration in our research.

i.e., adopting a term weighting technique before training.
In their study, by scoring on the features that combine n gramcharacters/words/POS

tags

with

TF -IDF,

they

Table 1. Essay distribution of JCLCNative Language# of essays

Indonesian

3,381

obtained 84.55% accuracy with the above -mentioned linear

Thai

1,307

SVMs.

Vietnamese

822

Korean568

rules

Burmese410(CFGR) and obtained 84.8% by using TOEFL11, higher

Laotian398

than those reported in all works in [7]. Since CFGR is an

Khmer

329

Filipino

293

Japanese

270

Bykh and Meurers [9] defined three new features
accordingtocontext-freegrammar-production

informative feature in English NLI, we wonder whether
CFGR can be used in Chinese NLI to obtain higher
performance.

Total

Malmasi and Dras [10] first developed an NLI method
for a Chinese dataset. Not only did their feature set involve
POS-tag-n-grams and function

words butthey also

exploited CFGR, which can capture the structure of1

Mongolian

http://www.uclouvain.be/en-cecl-icle.html

119
7,897

4. METHODOLOGY
The task of NLI can be regarded as a multiclass textclassification. In our proposed method, four elements havebeen taken into consideration to achieve high accuracy: 1)

How we should select effective essays to extract features

which the highest tuning accuracy can be obtained.

before training, 2) what kinds of features we should choose,3) how we can estimate the importance of each feature, and4) how we should construct machine -learning models.

4.2 Features
4.2.1 Skip-gram

In order to solve the abovementioned four problems,

Although skip-gram is a technique commonly used inbesides a noisy data cleaning method and a term weighting

speech processing [16], it was first applied to modeltechnique of BM25 proposed in the authors’ previous

context in [17] and achieved high performance. In view ofwork[12], 1-skip-bi-gram feature is employed in ourthis, the usefulness of skip-grams is explored in our work.

method.

Figure 2 a) shows a Chinese sentence segmented withword-base. Figure 2 b) presents the corresponding 1-skip-

4.1 Noisy data elimination [12]bi-grams of sentence in a).

Figure 1 shows the length distribution of essays written
by students of Myanmar in JCLC. As shown in Figure 1,
we can see that there are some very short and very longessays. For example, a short essay consists of only one
sentence. Such an essay has the tendency to consist ofsmall L2-related characteristics because the essay is too

Figure 2. Example of a Chinese sentence with corresponding

short. Further, we assume that very long essays consist of

1-skip-bi-gram

the small characteristics of L2, because students who could
write long essays might be the top level of L2 that results

As is shown in Figure 2 b), in 1-skip-bi-grams, not onlyin the small appearance of the L2 -related characteristics.

can adjacent tokens contain lexical information (the same

These essays, i.e., both very short and very long essays,

with word bi-gram), but skipped tokens can also provide us

might not be suitable for the training dataset.

withsyntactic

information(similartogrammaticaldependency). 1-skip-bi-gram is liable to be a useful feature
for Chinese NLI. Additionally, in k-skip-n-gram model, it
is obvious to note that the skip-gram feature dimension willgrow enormously as k and n increase, which will result inpoor performance due to the generation of considerable
useless n-grams. As a result, only 1-skip-bi-gram is taken
into consideration in our proposed method.

4.2.2 Feature combinationIn

additionto

1-skip-bi-gram,otherlexicalandsyntactic features are adopted in our proposed method.
Lexical features such as character n -gram, word n-gram,part-of-speech (POS) tag n-gram, and function words havebeen commonly utilized in NLI. As for syntac tic features,
Figure 1. Essay length distribution of students from Indonesia

we examine the context-free grammar production rules(CGPR), the same as what we explored in [12] .

In order to select effective training essays by discarding

The feature combination exploited is listed in Table 2.

both very long and very short ones, we still employ

As is explained in Section 2, we adopt recurring n-grams,

equation (1) in [12] to filter out such essays:

selecting n-grams that occur in more than ten essays as

i  n1   i  essaylengt h  i  n2   i
where

i represents native language

i ,

i

(1)
and

i

denote the mean and the variance of the length of essays
whose native language is i , respectively.

n1 andn2

represent the parameters used for controlling the numberof discarded essays.
Our goal in this step is to find a suitable pair ( n1 , n 2 ) by

informative features.

Table 2. Our adopted features

In the JCLC dataset, the number of essays corresponding

Features
Character 1,2,3-gram
Word 1-gram
POS-tag 1,2,3-gram
Function words
CGPR
1-skip-bi-gram

to each native language is considerably different from eachother.

For this reason, we continue exploiting theapplication of hierarchical linear SVMs to Chinese NLI.
The structure of our hierarchical linear SVM classifiers isshown in Figure 3. In our classification, we determine
whether a test essay belongs to the label “Indonesian” byusing classifier 1 first. If so, then the classification is

4.3 Term weighting method of BM25 [12]complete. Otherwise, continue to identify which label

Wang et al. [12] first demonstrated that term weighting

should be assigned among the remaining nine native

method of BM25 played a significant role in pursuing

languages, such as Thai, Khmer, and Korean by using

higher NLI accuracy. In this work, we continue utilizingclassifier 2 (one-vs-the-rest). Here, one-vs-the-rest method

the application of BM25, which is defined as equation (2).

chooses the label (native language) which classifies the testwij 

tf ij  k1  1len(d j ) 
tf ij  k1  1  b  b 
avgdl 


 log

|D|
df i

essay with greatest margin.
(2)

where wij represents term t i ’s weighting in document j .

tfij shows the term frequency of term t i in document j . dfi
shows the document frequency of term t i , i.e., the numberof documents that consist of term t i . |D| represents the total
number of Chinese essays . avgdl denotes the average
document length and len(d j )represents the length of

5. EXPERIMENT AND EVALUATION
5.1 Evaluation method
In the authors’ previous work [12], we ran 10-fold crossvalidation experiment: the JCLC dataset was divided
randomly into ten subsets of equal size, nine of which wereused for training with varying n1 and n 2 , and the tenth wasused for testing. This process was repeated ten times witheach subset being held out for test exactly once. However,document j . According to Robertson and Zaragoza [13],in our previous paper [12], test data were used for tuning

high performance can be obtained under the conditio nparameters that results in unfairness of the evaluation.

1.2  k1  2 and b  0.75 . In our proposed method, weassigned 1.2 to k1 and 0.75 to b .

Therefore, in order to execute fair evaluation, besides
test dataset, we divide the remaining data into training
dataset and tuning dataset at a ratio of 4:1 while carrying

4.4 Hierarchical classifiers [12]out 10-fold cross validation, as is shown in Figure 4. In

Most of the works on the first NLI shared task [7]this case, after tuning phase, parameters will be fixed soverified that the application of linear SVMs to NLI canthat they can be applied to the machine learning model forachieve higher performance than that of the other machine

test particularly.

learning methods. In [12], the authors put forward a newhierarchical structure of linear SVM classifiers t o classify
Chinese essays in JCLC dataset and yielded higher
accuracy.

Figure 4. 10-fold cross validation in this work

5.2 Tuning parameters n 1 and n 2
In parameter tuning phase, classifiers are trained withvarying pair of ( n1 , n 2 ) where n1 =0, 1, 2, 3 and n 2 =0, 1, 2,
3, respectively. After training, we test the trained modelwith the tuning dataset to select best parameters. Table 3
summarizes the candidate pairs of ( n1 , n 2 ) in each roundFigure 3. Structure of our hierarchical linear SVM classifiersof 10-fold cross validation (left column), by which highesttuning accuracy can be obtained (right column).
Table 3. Result of tuning parameters(n 1 ,n 2 )Best tuning accuracy(2,3)(3,3)(2,3)(3,3)(2,3)(3,3)(3,2)(2,3)(3,3)(2,3)(3,3)(2,3)(3,3)(2,1)(2,3)(3,1)(3,3)(2,3)(3,3)(2,3)(3,3)

0.75630.75420.75490.75840.75630.76680.75980.75210.75000.7437
Figure 5. Accuracy of each native language

5.3 Test result
Even though candidate pairs of ( n1 , n 2 ) are obtained, thenext issue is, in a multiple candidates c ase, which pair of( n1 , n 2 ) we should choose. In this step, a pair of ( n1 , n 2 )is selected at random and is applied to the proposed modelfor test.

Table 4 presents the test result of 10 -fold crossvalidation. As is shown, the left column is the randomly
selected pair of ( n1 , n 2 ), with its corresponding test
accuracy in the right column. Finally, afte r averaging all ofthem, an accuracy of 0.753 is achieved by our proposedmodel.
Table 4. Test result of 10-fold cross validation(n 1 ,n 2 )Test accuracy

Figure 6. Relationship between accuracy and # of training

(3,3)
(2,3)
(3,3)
(3,2)
(2,3)
(2,3)
(3,3)
(2,3)
(3,3)
(3,3)

0.74650.74520.76930.72620.74650.77690.75790.74520.74900.7630essaysFinal score

0.753

5.4 Analysis
Figure 5 shows the accuracy of each native language in
our experiment. As is shown, essays written by Thai
achieves the best accuracy of 0.8447, whereas essays
written by Mongolian gains the poorest performance of
0.3361 accuracy. Such low accuracy arises from the factthat there is not sufficient training data so that effective
training cannot be carried out.

Figure 6 illustrates the relationship between accuracy
and the number of training essays. In order to demonstrate
the pattern intuitively, we calculate the logarithm of thenumber of training essays as x-axis. As is shown, when x ≤6.3, i.e., the number of essays is smaller than or around 550 ,there seems a liner upward trend that the accuracy
increases in direct proportion to the number of essays .
When x > 6.3, the accuracy seems to reach a limit around
0.8 with a constant forward trend.

5.5 Comparison with the authors’ previous work
Besides 70.6% accuracy in [10], a supervised model
based on JCLC corpus was built in the authors’ previous
work [12] and 77.1% accuracy was achieved. As baselines,
we implemented those two algorithms with the evaluation
method proposed in this work and obtained an accuracy of
0.653 and 0.748. We compare it with that of our proposed
supervised model shown in Figure 7. As is illustrated in
Figure 7, our proposed method outperforms the baselines

by 0.5% and 10%.
0.8
0.748

0.75
0.7

0.753

0.653

0.65
0.6
Accuracy
Baseline[10]Baseline[12]Our method

Figure 7. Accuracy compared with baseline

6. CONCLUSION AND FUTURE WORKIn conclusion, in this paper, we put forward a new
Chinese NLI method based on JCLC dataset and achieved
the state-of-the-art accuracy of 75.3% with a robust
evaluation method, which can contribute to second
language education and identification of phishing. Above
all, our proposed model is cross lingual, which can also be
applied to accomplish English NLI.
On a basis of our previous work, which highlighted thesignificance of noisy data elimination and term weighting
techniques, in this work, not only is our supervised model
the first work to verify skip -gram is an informative feature,
but we also propose a robust evaluation method to v alidate
the applicability of the algorithm.
In the future, in the light of the high -dimensional feature
vector in this task, an efficient feature selecti on methodneeds to be further investigated.

REFERENCE[1] Joel Tetreault, Daniel Blanchard, Aoife Cahil l andMartin Chodorow, “Native Tongues, Lost and Found:Resources and Empirical Evaluations in Native
Language Identification,” Proc. of 24th COLING,
pp.2585-2602, 2012.
[2] Ria Perkins, “Native Language Identification (NLID)
for Forensic Authorship Analysis o f Weblogs,” New
Threats and Countermeasures in Digital Crime andCyber Terrorism, IGI Global, pp.213-234, 2015.
[3] Bernd H. Schmitt, Yigang Pan and Nader T. Tavassoli,“Language and Consumer Memory: The Impact ofLinguistic Differences between Chinese and English,”
J. of Consumer Research, vol. 21 (3), pp.419 -431,
1994.
[4] Moshe Koppel, Jonathan Schler and Kr Zigdon,“Determining an Author's Native Language by Mininga Text for Errors,” Proc. of the 11th KDD, pp.624-628,
2005.
[5] Serhiy Bykh and Detmar Meurers, “Nat ive Language
Identification
Using
RecurringN-grams
Investigating Abstraction and Domain Dependence,”

Proc. of the 24th COLING, pp.425-440, 2012.
[6] Daniel Blanchard, Joel Tetreault, Derrick Higgins,
Aoife Cahill and Martin Chodorow, “TOEFL11: A
Corpus of Non-Native English,” ETS RR-13-24, 2013.
[7] Joel Tetreault, Daniel Blanchard and Aoife Cahill, “A
Report on the First Native Langu age Identification
Shared Task,” Proc. of the 8th Workshop on
Innovative Use of NLP for Building Educational
Applications, pp.48-57, 2013.
[8] Binyam Gebrekidan Gebre, Marcos Zampieri, Peter
Wittenburg and Tom Heskes, “Improving Native
Language Identification with TF-IDF Weighting,”
Proc. of the 8th Workshop on Innovative Use of NLP
for Building Educational Applications, pp.216 -223,
2013.
[9] Serhiy Bykh and Detmar Meurers, “Exploring
Syntactic
Featuresfor
NativeLangu age
Identification: A Variationist Perspective on Feature
Encoding and Ensemble Optimization,” Proc. of the25th COLING, pp.1962-1973, 2014.
[10] Shervin Malmasi and Mark Dras, ”Chi nese Native
Language Identification,” Proc. of the 14th Conf. of
the European Chapter of the Association for
Computational Linguistics, pp.95 -99, 2014.
[11] Maolin Wang, Qi Gong, Jie Kuang, and Ziyu Xiong,“The Development of a Chinese Learner Corpus,”
Proc. of Int’l Conf. on Speech Database andAssessments (Oriental COCOSDA), pp.1 -6, 2012.
[12] Lan Wang, Masahiro Tanaka, and Hayato Yamana,“What is your Mother Tongue?: Improving Chinese
Native Language Identification by Cleaning Noisy
Data and Adopting BM25,” Proc. of IEEE Int’l Conf.
on Big Data Analysis, 2016.
[13] Stephen Robertson and Hugo Zaragoza, “TheProbabilistic Relevance Framework: BM25 andBeyond,” J. of Foundations and Trends in Information
Retrieval, vol. 3 (4), pp. 333 -389, 2009.
[14] John S. Whissell and Charles L. A. Clarke,“Improving Document Clustering Using Okapi BM25
Feature Weighting,” J. of Information Retrieval, vol.
14 (5), pp.466-487, 2011.
[15] Maolin Wang, Shervin Malmasi, and Mingxuan Huang,“The Jinan Chinese Learner Corpus, ” Proc. of 10th
Workshop on Innovative Use of NLP for Building
Educational Applications, pp.118 -123, 2015.
[16] Manhung Siu and Mari Ostendorf, “Variable n -gramsand extensions for conversational speech language
modelling,” IEEE Trans. on Speech and Audio
Processing, 8:63-75, 2000.
[17] David Guthrie, Ben Allison, Wei Liu, Louise Guthrie,
and Yorick Wilks, ”A Closer Look at Skip-gram
Modelling,” Proc. of the 5th Int’l Conf. on Language
Resources and Evaluation, pp.1222–1225, 2006.