AutomAted text ReAdAbility Assessment foR RussiAn second lAnguAge leARneRs

words list coverage of a text 0.58 3.9e-57 0.60 2.3e-63 Percentage of neuter words per text 0.55 1.3e-49 0.60 5e-68 Median number of punctuation per sentence 0.55 1.4e-49 0.55 7.3-50 Percentage of words in genitive case per text 0.50 2.8e-39 0.60 6.1e-60 Table 2. Examples of correlation coefficient for different groups of features level A1 A2 B1 B2 C1 C2 C2+ number of text 108 120 106 97 39 75 48 Table 1. Corpus content distribution There are 6 basic CEFR levels, translated into numerical form (A1 = 0, A2 = 1 etc) at the core of our scale. So the complexity of the text is presented as an increasing value, that reflects the concept of process of language acquisition more naturally, 4 https://lexile.com 5 https://en.wikipedia. org/wiki/Common_European_Framework_of_Reference_for_Languages Automated Text Readability Assessment for Russian Second Language Learners 5 than 6 closed classes. Our corpus contains about 600 texts from the CIE resource6 and several textbooks. Authors of these books provided the information about the target level. The content distribution is shown in Table 1. For the C2 (the level of an educated native speaker) texts from news portals and articles from popular magazines on various subjects were used. However, it is obvious that the reading difficulty of texts marked as native speaker level can also differ greatly. Therefore, we added to our scale the C2+ level, which will include texts supposedly perceived as difficult by Russian native speakers: texts of laws, articles from the popular science magazine N+17, noted by the editors as complicated (they defined complexity as an amount of the scientific background in this field which is needed to understand the article). We have faced several issues while collecting corpus: a) information about level could be absent in the textbook; b) this information may not contain a clear indication of the CEFR levels (B1-B2, «advanced», «for the second semester»); c) the difficulty level reflects the author’s subjective evaluation. Therefore, in the future we are planning to perform expert or crowdsourcing annotation of our text collection to fix these limitations and to get more objective information about complexity of given texts using average score from several annotators. 2.2. Feature extraction and selection We select the features to extract taking into account the following principles: 1) the features should reflect the information provided in the regulatory documents. 2) following [Sharoff et al. 2008], we believe that the features should be quite simple and reproducible if we are talking about the real usage of this system in language learning. First, we extract some basic text metrics such as average and median word length, sentence length, average number of syllables per word, percentage of ‘long’ words (more than 4 syllables), average number of punctuation marks per sentence. This group of features is easy to get but it is still capable to show high correlation with a difficulty level in an obvious way: the longer the text and the words in it, the more likely it is difficult to read. [François and Miltsakaki 2012] in their readability study have found that the best prediction performance was obtained using both classic (readability formulas) and non-classic features. Considering this, we have applied as a feature 5 commonly used readability formulas, which are using following parametres: 1. Flesch–Kincaid: (words / sentences) + (syllables / sentences); 2. Сoleman Liau index: (characters / words) + (sentences / words); 3. Automated Readability Index: (characters / words) + (words / sentences); 4. Dale-Chall formula: (‘difficult’ words that are out of Dale’s 3000 simple words list / all words) + (words / sentences); 5. Simple Measure of Gobbledygook: (words more than 4 syllables / sentences). 6 http://texts.cie. ru