Abstract: Corpora and web texts can become a rich language learning resource if we have a means of assessing whether they are linguistically appropriate for learners at a given proficiency level. In this paper, we aim at addressing this issue by presenting the first approach for predicting linguistic complexity for Swedish second language learning material on a 5-point scale. After showing that the traditional Swedish readability measure, L\"asbarhetsindex (LIX), is not suitable for this task, we propose a supervised machine learning model, based on a range of linguistic features, that can reliably classify texts according to their difficulty level. Our model obtained an accuracy of 81.3% and an F-score of 0.8, which is comparable to the state of the art in English and is considerably higher than previously reported results for other languages. We further studied the utility of our features with single sentences instead of full texts since sentences are a common linguistic unit in language learning exercises. We trained a separate model on sentence-level data with five classes, which yielded 63.4% accuracy. Although this is lower than the document level performance, we achieved an adjacent accuracy of 92%. Furthermore, we found that using a combination of different features, compared to using lexical features alone, resulted in 7% improvement in classification accuracy at the sentence level, whereas at the document level, lexical features were more dominant. Our models are intended for use in a freely accessible web-based language learning platform for the automatic generation of exercises.

Introduction: Linguistic information provided by Natural Language Processing (NLP) tools has good potential for turning the continuously growing amount of digital text into interactive and personalized language learning material. Our work aims at overcoming one of the fundamental obstacles in this domain of research, namely how to assess the linguistic complexity of texts and sentences from the perspective of second and foreign language (L2) learners. There are a number of readability models relying on NLP tools to predict the difficulty (readability) level of a text BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . The linguistic features explored so far for this task incorporate information, among others, from part-of-speech (POS) taggers and dependency parsers. Cognitively motivated features have also been proposed, for example, in the Coh-Metrix BIBREF2 . Although the majority of previous work focuses primarily on document-level analysis, a finer-grained, sentence-level readability has received increasing interest in recent years BIBREF6 , BIBREF7 , BIBREF8 . The previously mentioned studies target mainly native language (L1) readers including people with low literacy levels or mild cognitive disabilities. Our focus, however, is on building a model for predicting the proficiency level of texts and sentences used in L2 teaching materials. This aspect has been explored for English BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , French BIBREF13 , Portuguese BIBREF14 and, without the use of NLP, for Dutch BIBREF15 . Readability for the Swedish language has a rather long tradition. One of the most popular, easy-to-compute formulas is LIX (Läsbarthetsindex, `Readability index') proposed in BIBREF16 . This measure combines the average number of words per sentence in the text with the percentage of long words, i.e. tokens consisting of more than six characters. Besides traditional formulas, supervised machine learning approaches have also been tested. Swedish document-level readability with a native speaker focus is described in BIBREF4 and BIBREF17 . For L2 Swedish, only a binary sentence-level model exists BIBREF8 , but comprehensive and highly accurate document- and sentence-level models for multiple proficiency levels have not been developed before. In this paper, we present a machine learning model trained on course books currently in use in L2 Swedish classrooms. Our goal was to predict linguistic complexity of material written by teachers and course book writers for learners, rather than assessing learner-produced texts. We adopted the scale from the Common European Framework of Reference for Languages (CEFR) BIBREF18 which contains guidelines for the creation of teaching material and the assessment of L2 proficiency. CEFR proposes six levels of language proficiency: A1 (beginner), A2 (elementary), B1 (intermediate), B2 (upper intermediate), C1 (advanced) and C2 (proficient). Since sentences are a common unit in language exercises, but remain less explored in the readability literature, we also investigate the applicability of our approach to sentences, performing a 5-way classification (levels A1-C1). Our document-level model achieves a state-of-the-art performance (F-score of 0.8), however, there is room for improvement in sentence-level predictions. We plan to make our results available through the online intelligent computer-assisted language learning platform Lärka, both as corpus-based exercises for teachers and learners of L2 Swedish and as web-services for researchers and developers. In the following sections, we first describe our datasets (section SECREF2 ) and features (section SECREF3 ), then we present the details and the results of our experiments in section SECREF4 . Finally, section SECREF5 concludes our work and outlines further directions of research within this area.

Question: what linguistics features did they apply?
Answer: Sentence length
Modal verbs to verbs
Average token length
Particle IncSc
Extra-long words
SG pronoun IncSc
Number of characters
Punctuation IncSc
LIX
Subjunction IncSc
S-verb IncSc
A1 lemma IncSc
S-verbs to verbs
A2 lemma IncSc
Adjective IncSc
B1 lemma IncSc
Adjective variation
B2 lemma IncSc
Adverb IncSc
C1 lemma IncSc
Adverb variation
C2 lemma IncSc
Noun IncSc
Difficult word IncSc
Noun variation
Difficult noun and verb IncSc
Verb IncSc
Out-of-Kelly IncSc
Verb variation
Missing lemma form IncSc
Nominal ratio
Avg. Kelly log frequency
Nouns to verbs
Function word IncSc
Average dependency length
Lexical words to non-lexical words
Dependency arcs longer than
Lexical words to all tokens
Longest dependency from root node
Neuter gender noun IncSc
Ratio of right dependency arcs
Con- and subjunction IncSc
Ratio of left dependency arcs
Past participles to verbs
Modifier variation
Present participles to verbs
Pre-modifier IncSc
Past verbs to verbs
Post-modifier IncSc
Present verbs to verbs
Subordinate IncSc
Supine verbs to verbs
Relative clause IncSc
Relative structure IncSc
Prepositional complement IncSc
Bilog type-token ratio
Square root type-token ratio
Avg. nr. of senses per token
Pronouns to nouns
Noun senses per noun
Pronouns to prepositions