Lexical Bias In Essay Level Prediction

Abstract: Automatically predicting the level of non-native English speakers given their written essays is an interesting machine learning problem. In this work I present the system"balikasg"that achieved the state-of-the-art performance in the CAp 2018 data science challenge among 14 systems. I detail the feature extraction, feature engineering and model selection steps and I evaluate how these decisions impact the system's performance. The paper concludes with remarks for future work.

Automatically predicting the level of English of non-native speakers from their written text is an interesting text mining task. Systems that perform well in the task can be useful components for online, second-language learning platforms as well as for organisations that tutor students for this purpose. In this paper I present the system balikasg that achieved the state-of-the-art performance in the CAp 2018 data science challenge among 14 systems. In order to achieve the best performance in the challenge, I decided to use a variety of features that describe an essay's readability and syntactic complexity as well as its content. For the prediction step, I found Gradient Boosted Trees, whose efficiency is proven in several data science challenges, to be the most efficient across a variety of classifiers.
The rest of the paper is organized as follows: in Section 2 I frame the problem of language level as an ordinal classification problem and describe the available data. Section 3 presents the feature extaction and engineering techniques used. Section 4 describes the machine learning algorithms for prediction as well as the achieved results. Finally, Section 5 concludes with discussion and avenues for future research.
In order to approach the language-level prediction task as a supervised classification problem, I frame it as an ordinal classification problem. In particular, given a written essay INLINEFORM0 from a candidate, the goal is to associate the essay with the level INLINEFORM1 of English according to the Common European Framework of Reference for languages (CEFR) system. Under CEFR there are six language levels INLINEFORM2 , such that INLINEFORM3 . In this notation, INLINEFORM4 is the beginner level while INLINEFORM5 is the most advanced level. Notice that the levels of INLINEFORM6 are ordered, thus defining an ordered classification problem. In this sense, care must be taken both during the phase of model selection and during the phase of evaluation. In the latter, predicting a class far from the true should incur a higher penalty. In other words, given a INLINEFORM7 essay, predicting INLINEFORM8 is worse than predicting INLINEFORM9 , and this difference must be captured by the evaluation metrics.
In order to capture this explicit ordering of INLINEFORM0 , the organisers proposed a cost measure that uses the confusion matrix of the prediction and prior knowledge in order to evaluate the performance of the system. In particular, the meaures uses writes as: DISPLAYFORM0 
where INLINEFORM0 is a cost matrix that uses prior knowledge to calculate the misclassification errors and INLINEFORM1 is the number of observations of class INLINEFORM2 classified with category INLINEFORM3 . The cost matrix INLINEFORM4 is given in Table TABREF3 . Notice that, as expected, moving away from the diagonal (correct classification) the misclassification costs are higher. The biggest error (44) occurs when a INLINEFORM5 essay is classified as INLINEFORM6 . On the contrary, the classification error is lower (6) when the opposite happens and an INLINEFORM7 essay is classified as INLINEFORM8 . Since INLINEFORM9 is not symmetric and the costs of the lower diagonal are higher, the penalties for misclassification are worse when essays of upper languages levels (e.g., INLINEFORM10 ) are classified as essays of lower levels.
In this section I present the extracted features partitioned in six groups and detail each of them separately.
As the class distribution in the training data is not balanced, I have used stratified cross-validation for validation purposes and for hyper-parameter selection. As a classification1 algorithm, I have used gradient boosted trees trained with gradient-based one-side sampling as implemented in the Light Gradient Boosting Machine toolkit released by Microsoft.. The depth of the trees was set to 3, the learning rate to 0.06 and the number of trees to 4,000. Also, to combat the class imbalance in the training labels I assigned class weights at each class so that errors in the frequent classes incur less penalties than error in the infrequent.
In this work I presented the feature extraction, feature engineering and model evaluation steps I followed while developing balikasg for CAp 2018 that was ranked first among 14 other systems. I evaluated the efficiency of the different feature groups and found that readbility and complexity scores as well as topic models to be effective predictors. Further, I evaluated the the effectiveness of different classification algorithms and found that Gradient Boosted Trees outperform the rest of the models in this problem.
While in terms of accuracy the system performed excellent achieving 98.2% in the test data, the question raised is whether there are any types of biases in the process. For instance, topic distributions learned with LDA were valuable features. One, however, needs to deeply investigate whether this is due to the expressiveness and modeling power of LDA or an artifact of the dataset used. In the latter case, given that the candidates are asked to write an essay given a subject BIBREF0 that depends on their level, the hypothesis that needs be studied is whether LDA was just a clever way to model this information leak in the given data or not. I believe that further analysis and validation can answer this question if the topics of the essays are released so that validation splits can be done on the basis of these topics.
I would like to thank the organisers of the challenge and NVidia for sponsoring the prize of the challenge. The views expressed in this paper belong solely to the author, and not necessarily to the author's employer.

Figure and table captions:
Table 1: Cost matrix used to calculate the miscalssification error described in Eq. (1).
Figure 1: The distribution of essays according to the CERF levels in the training data.
Table 2: Basic statistics for the released essays.
Figure 2: The accuracy scores of each feature set using 3-fold cross validation on the training data.
Table 4: Ablation study to explore the importance of different feature families.
Table 3: Stratified 3-fold cross-validation scores for the official measure of the challenge.
Table 5: Confusion matrix of the 3-fold stratified cross validation. The Ci,j value is the number of predictions known to be in group i and predicted to be in group j. Notice how most of the mis-classification errors occur between close categories.

Questions about this paper:
what features of the essays are extracted?
what were the evaluation metrics?
what future work is described?

Answers:
Numerical features, Language Models, Clusters, Latent Dirichlet Allocation, Part-Of-Speech tags, Bag-of-words
Accuracy
Investigate the effectiveness of LDA to capture the subject of the essay.