A Readable Read: Automatic Assessment of Language Learning Materials based on Linguistic Complexity

Abstract: Corpora and web texts can become a rich language learning resource if we have a means of assessing whether they are linguistically appropriate for learners at a given proficiency level. In this paper, we aim at addressing this issue by presenting the first approach for predicting linguistic complexity for Swedish second language learning material on a 5-point scale. After showing that the traditional Swedish readability measure, L\"asbarhetsindex (LIX), is not suitable for this task, we propose a supervised machine learning model, based on a range of linguistic features, that can reliably classify texts according to their difficulty level. Our model obtained an accuracy of 81.3% and an F-score of 0.8, which is comparable to the state of the art in English and is considerably higher than previously reported results for other languages. We further studied the utility of our features with single sentences instead of full texts since sentences are a common linguistic unit in language learning exercises. We trained a separate model on sentence-level data with five classes, which yielded 63.4% accuracy. Although this is lower than the document level performance, we achieved an adjacent accuracy of 92%. Furthermore, we found that using a combination of different features, compared to using lexical features alone, resulted in 7% improvement in classification accuracy at the sentence level, whereas at the document level, lexical features were more dominant. Our models are intended for use in a freely accessible web-based language learning platform for the automatic generation of exercises.

Linguistic information provided by Natural Language Processing (NLP) tools has good potential for turning the continuously growing amount of digital text into interactive and personalized language learning material. Our work aims at overcoming one of the fundamental obstacles in this domain of research, namely how to assess the linguistic complexity of texts and sentences from the perspective of second and foreign language (L2) learners.
There are a number of readability models relying on NLP tools to predict the difficulty (readability) level of a text BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . The linguistic features explored so far for this task incorporate information, among others, from part-of-speech (POS) taggers and dependency parsers. Cognitively motivated features have also been proposed, for example, in the Coh-Metrix BIBREF2 . Although the majority of previous work focuses primarily on document-level analysis, a finer-grained, sentence-level readability has received increasing interest in recent years BIBREF6 , BIBREF7 , BIBREF8 .
The previously mentioned studies target mainly native language (L1) readers including people with low literacy levels or mild cognitive disabilities. Our focus, however, is on building a model for predicting the proficiency level of texts and sentences used in L2 teaching materials. This aspect has been explored for English BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , French BIBREF13 , Portuguese BIBREF14 and, without the use of NLP, for Dutch BIBREF15 .
Readability for the Swedish language has a rather long tradition. One of the most popular, easy-to-compute formulas is LIX (Läsbarthetsindex, `Readability index') proposed in BIBREF16 . This measure combines the average number of words per sentence in the text with the percentage of long words, i.e. tokens consisting of more than six characters. Besides traditional formulas, supervised machine learning approaches have also been tested. Swedish document-level readability with a native speaker focus is described in BIBREF4 and BIBREF17 . For L2 Swedish, only a binary sentence-level model exists BIBREF8 , but comprehensive and highly accurate document- and sentence-level models for multiple proficiency levels have not been developed before.
In this paper, we present a machine learning model trained on course books currently in use in L2 Swedish classrooms. Our goal was to predict linguistic complexity of material written by teachers and course book writers for learners, rather than assessing learner-produced texts. We adopted the scale from the Common European Framework of Reference for Languages (CEFR) BIBREF18 which contains guidelines for the creation of teaching material and the assessment of L2 proficiency. CEFR proposes six levels of language proficiency: A1 (beginner), A2 (elementary), B1 (intermediate), B2 (upper intermediate), C1 (advanced) and C2 (proficient). Since sentences are a common unit in language exercises, but remain less explored in the readability literature, we also investigate the applicability of our approach to sentences, performing a 5-way classification (levels A1-C1). Our document-level model achieves a state-of-the-art performance (F-score of 0.8), however, there is room for improvement in sentence-level predictions. We plan to make our results available through the online intelligent computer-assisted language learning platform Lärka, both as corpus-based exercises for teachers and learners of L2 Swedish and as web-services for researchers and developers.
In the following sections, we first describe our datasets (section SECREF2 ) and features (section SECREF3 ), then we present the details and the results of our experiments in section SECREF4 . Finally, section SECREF5 concludes our work and outlines further directions of research within this area.
Our dataset is a subset of COCTAILL, a corpus of course books covering five CEFR levels (A1-C1) BIBREF19 . This corpus consists of twelve books (from four different publishers) whose usability and level have been confirmed by Swedish L2 teachers. The course books have been annotated both content-wise (e.g. exercises, lists) and linguistically (e.g. with POS and dependency tags) BIBREF19 . We collected a total of 867 texts (reading passages) from this corpus. We excluded texts that are primarily based on dialogues from the current experiments due to their specific linguistic structure, with the aim of scaling down differences connected to text genres rather than linguistic complexity. We plan to study the readability of dialogues and compare them to non-dialogue texts in the future.
Besides reading passages, i.e. texts, the COCTAILL corpus contains a number of sentences independent from each other, i.e. not forming a coherent text, in the form of lists of sentences and language examples. This latter category consists of sentences illustrating the use of specific grammatical patterns or lexical items. Collecting these sentences, we built a sentence-level dataset consisting of 1874 instances. The information encoded in the content-level annotation of COCTAILL (XML tags list, language_example and the attribute unit) enabled us to include only complete sentences and exclude sentences containing gaps and units larger or smaller than a sentence (e.g. texts, phrases, single words etc.). The CEFR level of both sentences and texts has been derived from the CEFR level of the lesson (chapter) they appeared in. In Table TABREF3 , columns 2-5 give an overview of the distribution of texts across levels and their mean length in sentences. The distribution of sentences per level is presented in the last two columns of Table TABREF3 . COCTAILL contained a somewhat more limited amount of B2 and C1 level sentences in the form of lists and language examples, possibly because learners handle larger linguistic units with more ease at higher proficiency levels.
We developed our features based on information both from previous literature BIBREF9 , BIBREF3 , BIBREF13 , BIBREF4 , BIBREF8 and a grammar book for Swedish L2 learners BIBREF20 . The set of features can be divided in the following five subgroups: length-based, lexical, morphological, syntactic and semantic features (Table TABREF6 ).
Length-based (Len): These features include sentence length in number of tokens (#1) and characters (#4), extra-long words (longer than thirteen characters) and the traditional Swedish readability formula, LIX (see section SECREF1 ). For the sentence-level analysis, instead of the ratio of number of tokens to the number of sentences in the text, we considered the number of tokens in one sentence.
Lexical (Lex): Similar to BIBREF8 , we used information from the Kelly list BIBREF21 , a lexical resource providing a CEFR level and frequencies per lemma based on a corpus of web texts. Thus, this word list is entirely independent from our dataset. Instead of percentages, we used incidence scores (IncSc) per 1000 words to reduce the influence of sentence length on feature values. The IncSc of a category was computed as 1000 divided by the number of tokens in the text or sentence multiplied by the count of the category in the sentence. We calculated the IncSc of words belonging to each CEFR level (#6 - #11). In features #12 and #13 we considered difficult all tokens whose level was above the CEFR level of the text or sentence. We computed also the IncSc of tokens not present in the Kelly list (#14), tokens for which the lemmatizer did not find a corresponding lemma form (# 15), as well as average log frequencies (#16).
Morphological (Morph): We included the variation (the ratio of a category to the ratio of lexical tokens - i.e. nouns, verbs, adjectives and adverbs) and the IncSc of all lexical categories together with the IncSc of punctuations, particles, sub- and conjunctions (#34, #51). Some additional features, using insights from L2 teaching material BIBREF20 , captured fine-grained inflectional information such as the IncSc of neuter gender nouns and the ratio of different verb forms to all verbs (#52 - #56). Instead of simple type-token ratio (TTR) we used a bilogarithmic and a square root TTR as in BIBREF3 . Moreover, nominal ratio BIBREF4 , the ratio of pronouns to prepositions BIBREF13 , and two lexical density features were also included: the ratio of lexical words to all non-lexical categories (#48) and to all tokens (#49). Relative structures (#57) consisted of relative adverbs, determiners, pronouns and possessives.
Syntactic (Synt): Some of these features were based on the length (depth) and the direction of dependency arcs (#17 - #21). We complemented this, among others, with the IncSc of relative clauses in clefts (#26), and the IncSc of pre-and postmodifiers (e.g. adjectives and prepositional phrases) BIBREF4 .
Semantic (Sem): Features based on information from SALDO BIBREF23 , a Swedish lexical-semantic resource. We used the average number of senses per token as in BIBREF8 and included also the average number of noun senses per nouns. Once reliable word-sense disambiguation methods become available for Swedish, additional features based on word senses could be taken into consideration here.
The complete set of 61 features is presented in Table TABREF6 . Throughout this paper we will refer to the machine learning models using this set of features, unless otherwise specified. Features for both document- and sentence-level analyses were extracted for each sentence, the values being averaged over all sentences in the text in the document-level experiments to ensure comparability.
We explored different classification algorithms for this task using the machine learning toolkit WEKA BIBREF24 . These included: (1) a multinomial logistic regression model with ridge estimator, (2) a multilayer perceptron, (3) a support vector machine learner, Sequential Minimal Optimization (SMO), and (4) a decision tree (J48). For each of these, the default parameter settings have been used as implemented in WEKA.
We considered classification accuracy, F-score and Root Mean Squared Error (RMSE) as evaluation measures for our approach. We also included a confusion matrix, as we deal with a dataset that is unbalanced across CEFR levels. The scores were obtained by performing a ten-fold Cross-Validation (CV).
We trained document-level classification models, comparing the performance between different subgroups of features. We had two baselines: a majority classifier (Majority), with B2 as majority class, and the LIX readability score. Table TABREF9 shows the type of subgroup (Type), the number of features (Nr) and three evaluation metrics using logistic regression.
Not only was accuracy very low with LIX, but this measure also classified 91.6% of the instances as B2 level. Length-based, semantic and syntactic features in isolation showed similar or only slightly better performance than the baselines, therefore we excluded them from Table TABREF9 . Lexical features, however, had a strong discriminatory power without an increase in bias towards the majority classes. Using this subset of features only, we achieved approximately the same performance (0.8 F) as with the complete set of features, All (0.81 F). This suggests that lexical information alone can successfully distinguish the CEFR level of course book texts at the document level. Using the complete feature set we obtained 81% accuracy and 97% adjacent accuracy (when misclassifications to adjacent classes are considered correct). The same scores with lexical features (Lex) only were 80.3% (accuracy) and 98% (adjacent accuracy).
Accuracy scores using other learning algorithms were significantly lower (see Table TABREF10 ), therefore, we report only the results of the logistic regression classifier in the subsequent sections.
Instead of classification, some readability studies (e.g. BIBREF10 , BIBREF14 ) employed linear regression for this task. For a better comparability, we applied also a linear regression model to our data which yielded a correlation of 0.8 and an RMSE of 0.65.
To make sure that our system was not biased towards the majority classes B1 and B2, we inspected the confusion matrix (Table TABREF11 ) after classification using All. We can observe from Table TABREF11 that the system performs better at A1 and C1 levels, where confusion occurred only with adjacent classes. Similar to the findings in BIBREF13 for French, classes in the middle of the scale were harder to distinguish. Most misclassifications in our material occurred at A2 level (23%) followed by B1 and B2 level, (20% and 17% respectively).
To establish the external validity of our approach, we tested it on a subset of LäSBarT BIBREF4 , a corpus of Swedish easy-to-read (ETR) texts previously employed for Swedish L1 readability studies BIBREF4 , BIBREF17 . We used 18 fiction texts written for children between ages nine to twelve, half of which belonged to the ETR category and the rest were unsimplified. Our model generalized well to unseen data, it classified all ETR texts as B1 and all ordinary texts as C1 level, thus correctly identifying in all cases the relative difference in complexity between the documents of the two categories.
Although a direct comparison with other studies is difficult because of the target language, the nature of the datasets and the number of classes used, in terms of absolute numbers, our model achieves comparable performance with the state-of-the-art systems for English BIBREF9 , BIBREF12 . Other studies for non-English languages using CEFR levels include: BIBREF13 , reporting 49.1% accuracy for a French system distinguishing six classes; and BIBREF14 achieving 29.7% accuracy on a smaller Portuguese dataset with five levels.
After building good classification models at document level, we explored the usability of our approach at the sentence level. Sentences are particularly useful in Computer-Assisted Language Learning (CALL) applications, among others, for generating sentence-based multiple choice exercises, e.g. BIBREF25 , or vocabulary examples BIBREF26 . Furthermore, multi-class readability classification of sentence-level material intended for second language learners has not been previously investigated in the literature.
With the same methodology (section SECREF7 ) and feature set (section SECREF3 ) used at the document level, we trained and tested classification models based on the sentence-level data (see section SECREF2 ). The results are shown in Table TABREF13 .
Although the majority baseline in the case of sentences was 7% higher than the one for texts (Table TABREF9 ), the classification accuracy for sentences using all features was only 63.4%. This is a considerable drop (-18%) in performance compared to the document level (81.3% accuracy). It is possible that the features did not capture differences between the sentences because the amount of context is more limited on the fine-grained level. It is interesting to note that, although there was no substantial performance difference between Lex and All at a document level, the model with all the features performed 7% better at sentence level.
Most misclassifications occurred, however, within a distance of one class only, thus the adjacent accuracy of the sentence-level model was still high, 92% (see Table TABREF14 ). Predictions were noticeably more accurate for classes A1, A2 and B1 which had a larger number of instances.
In the next step, we applied the sentence-level model on the document-level data to explore how homogeneous texts were in terms of the CEFR level of the sentences they contained. Figure FIGREF15 shows that texts at each CEFR level contain a substantial amount of sentences of the same level of the whole text, but they also include sentences classified as belonging to other CEFR levels.
Finally, as in the case of the document-level analysis, we tested our sentence-level model also on an independent dataset (SenRead), a small corpus of sentences with gold-standard CEFR annotation. This data was created during a user-based evaluation study BIBREF27 and it consists of 196 sentences from generic corpora, i.e. originally not L2 learner-focused corpora, rated as being suitable at B1 or being at a level higher than B1. We used this corpus along with the judgments of the three participating teachers. Since SenRead had only two categories - INLINEFORM0 and INLINEFORM1 , we combined the model's predictions into two classes - A1, A2, B1 were considered as INLINEFORM2 B1 and B2, C1 were considered as INLINEFORM3 B1. The majority baseline for the dataset was 65%, INLINEFORM4 B1 being the class with most instances. The model trained on COCTAILL sentences predicted with 73% accuracy teachers' judgments, an 8% improvement over the majority baseline. There was a considerable difference between the precision score of the two classes, which was 85.4% for INLINEFORM5 B1, and only 48.5% for INLINEFORM6 B1.
Previously published results on sentence-level data include BIBREF6 , who report 66% accuracy for a binary classification task for English and BIBREF7 who obtained an accuracy between 78.9% and 83.7% for Italian binary class data using different kinds of datasets. Neither of these studies, however, had a non-native speaker focus. BIBREF8 report 71% accuracy for Swedish binary sentence-level classification from an L2 point of view. Both the adjacent accuracy of our sentence-level model (92%) and the accuracy score obtained with that model on SenRead (73%) improve on that score. It is also worth mentioning that the labels in the dataset from BIBREF8 were based on the assumption that all sentences in a text belong to the same difficulty level which, being an approximation (as also Figure FIGREF15 shows), introduced some noise in that data.
Although more analysis would be needed to refine the sentence-level model, our current results indicate that a rich feature set that considers multiple linguistic dimensions may result in an improved performance. In the future, the dataset could be expanded with more gold-standard sentences, which may improve accuracy. Furthermore, an interesting direction to pursue would be to verify whether providing finer-grained readability judgments is a more challenging task also for human raters.
We proposed an approach to assess the proficiency (CEFR) level of Swedish L2 course book texts based on a variety of features. Our document-level model, the first for L2 Swedish, achieved an F-score of 0.8, hence, it can reliably distinguish between proficiency levels. Compared to the wide-spread readability measure for Swedish, LIX, we achieved a substantial gain in terms of both accuracy and F-score (46% and 0.6 higher respectively). The accuracy of the sentence-level model remained lower than that of the text-level model, nevertheless, using the complete feature set the system performed 23% and 22% above the majority baseline and LIX respectively. Misclassifications of more than one level did not occur in more than 8% of sentences, thus, in terms of adjacent accuracy, our sentence-level model improved on previous results for L2 Swedish readability BIBREF8 .
Most notably, we have found that taking into consideration multiple linguistic dimensions when assessing linguistic complexity is especially useful for sentence-level analysis. In our experiments, using only word-frequency features was almost as predictive as a combination of all features for the document level, but the latter made more accurate predictions for sentences, resulting in a 7% difference in accuracy. Besides L2 course book materials, we tested both our document- and sentence-level models also on unseen data with promising results.
In the future, a more detailed investigation is needed to understand the performance drop between document and sentence level. Acquiring more sentence-level annotated data and exploring new features relying on lexical-semantic resources for Swedish would be interesting directions to pursue. Furthermore, we intend to test the utility of this approach in a real-world web application involving language learners and teachers.

Figure and table captions:
Table 1. The distribution of items per CEFR level in the datasets.
Table 2. The complete feature set.
Table 3. Document-level classification results.
Table 4. Accuracy scores (in %) for other learning algorithms.
Table 5. Confusion matrix for feature set ALL at document level.
Table 6. Sentence-level classification results.
Table 7. Confusion matrix for feature set ALL at sentence level.
Fig. 1. Distribution of sentences per CEFR level in the document-level data.

Questions about this paper:
what combination of features helped improve the classification?
what linguistics features did they apply?

Answers:
Using all the 61 features helped them improve the classification
Sentence length
Modal verbs to verbs
Average token length
Particle IncSc
Extra-long words
SG pronoun IncSc
Number of characters
Punctuation IncSc
LIX
Subjunction IncSc
S-verb IncSc
A1 lemma IncSc
S-verbs to verbs
A2 lemma IncSc
Adjective IncSc
B1 lemma IncSc
Adjective variation
B2 lemma IncSc
Adverb IncSc
C1 lemma IncSc
Adverb variation
C2 lemma IncSc
Noun IncSc
Difficult word IncSc
Noun variation
Difficult noun and verb IncSc
Verb IncSc
Out-of-Kelly IncSc
Verb variation
Missing lemma form IncSc
Nominal ratio
Avg. Kelly log frequency
Nouns to verbs
Function word IncSc
Average dependency length
Lexical words to non-lexical words
Dependency arcs longer than
Lexical words to all tokens
Longest dependency from root node
Neuter gender noun IncSc
Ratio of right dependency arcs
Con- and subjunction IncSc
Ratio of left dependency arcs
Past participles to verbs
Modifier variation
Present participles to verbs
Pre-modifier IncSc
Past verbs to verbs
Post-modifier IncSc
Present verbs to verbs
Subordinate IncSc
Supine verbs to verbs
Relative clause IncSc
Relative structure IncSc
Prepositional complement IncSc
Bilog type-token ratio
Square root type-token ratio
Avg. nr. of senses per token
Pronouns to nouns
Noun senses per noun
Pronouns to prepositions