SlowGuess/NAACL_2021_OCR huggingface dataset:
Sian Gooding
Dept of Computer Science and Technology  
University of Cambridge
shg36@cam.ac.uk
Ekaterina Kochmar
Dept of Computer Science University of Bath
ek762@bath.ac.uk
Seid Muhie Yimam
Language Technology group  
Universität Hamburg, Germany
yimam@informatik.uni-hamburg.de
Lexical complexity is a highly subjective notion, yet this factor is often neglected in lexical simplification and readability systems which use a "one-size-fits-all" approach. In this paper, we investigate which aspects contribute to the notion of lexical complexity in various groups of readers, focusing on native and non-native speakers of English, and how the notion of complexity changes depending on the proficiency level of a non-native reader. To facilitate reproducibility of our approach and foster further research into these aspects, we release a dataset of complex words annotated by readers with different backgrounds.
Complex word identification (CWI) is the first step in a lexical simplification (LS) pipeline, concerned with identification of words in text that are in need of further simplification (Shardlow, 2013). For instance, in example (1) a CWI system might identify engulfed as a complex word, which would allow an LS system to replace it with a simpler alternative, e.g. flooded, in the next step (Paetzold and Specia, 2016a; Gooding and Kochmar, 2019b):
(1) Water engulfed Beringia.
Water flooded Beringia.
It has been shown that accurate CWI can significantly reduce errors in simplification (Shardlow, 2014), thus improving the quality of an LS system output (Lee and Yeung, 2018). In addition, CWI has been shown to be an important component in readability assessment systems (Maddela and Xu, 2018) and in vocabulary acquisition modules of educational applications (Zaidi et al., 2020). However, an important aspect of CWI and LS that is often neglected is that text complexity is not an objective notion homogeneous across various target populations: what is challenging for a
reader with a particular background (for example, a non-native reader at a lower level of language proficiency) would not necessarily be challenging for readers with other backgrounds (for example, more proficient readers) (Bingel, 2018). A number of factors may contribute to that, including the reader's age and level of language proficiency, among others (Paetzold and Specia, 2016c). LS systems often aim to address the needs of specific reader populations, such as children, non-native speakers, or readers with particular cognitive impairments. Thus, personalization in LS typically results in specialized simplification tools aimed at certain groups of readers (Carroll et al., 1998; Rello et al., 2013; Evans et al., 2014), with only a few systems addressing adaptation to the readers' needs in a more dynamic way (Bingel et al., 2018; Yimam and Biemann, 2018a,b; Scarton and Specia, 2018).
Despite CWI being one of the key steps in an LS pipeline in need of adaptation to readers' profiles, this is rarely addressed in practice (Lee and Yeung, 2018; Bingel, 2018). For instance, existing and widely used datasets on CWI present a homogeneous view on word complexity, merging annotations from various groups of readers (Paetzold and Specia, 2016c; Yimam et al., 2018). From the cognitive perspective, little is still known about the challenges that particular readers face when developing their reading skills and about the factors contributing to their vocabulary acquisition.
In this paper, we investigate factors focusing on the two key background aspects in the development of reading abilities: whether a reader is a native speaker of the language, and if not, what is the reader's level of language proficiency. We use the data from Yimam et al. (2017a), which contains English sentences where complex words are annotated by native and non-native speakers of English, spanning three different levels of language proficiency. We investigate which aspects contribute to the notion of lexical complexity for readers with
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4439-4449 June 6-11, 2021. ©2021 Association for Computational Linguistics
different backgrounds and how the notion of complexity changes depending on the proficiency levels of the non-native readers.
In our paper we make the following contributions:
- We show that the best models for predicting complexity are trained using the annotations of the target audience.
- We perform feature analysis by observing which correlate most with the notion of complexity for native and non-native audiences.
- We analyse the distribution of features for complex words across differing proficiency levels.
- Finally, we release a CWI dataset annotated by readers with different backgrounds.
2.1 Models of Complex Word Identification
CWI was established as an essential step in LS in Shardlow (2013), which demonstrated that without this step, LS systems tend to over- or undersimplify, thus rendering the output less useful for the readers. Early approaches to this task considered simplification of all words (Devlin and Tait, 1998; Bott et al., 2012) and use of frequency-based thresholds (Zeng et al., 2005; Biran et al., 2011), however Shardlow (2013) shows that classification algorithms are more precise in identification of complex words than both these approaches. Recent shared tasks on CWI (Paetzold and Specia, 2016c; Yimam et al., 2018) helped it gain popularity in the NLP community as they provide researchers with shared data and benchmarks. Most systems participating in the shared tasks addressed CWI with classical machine learning algorithms, with the best-performing systems using ensemble-based approaches. Current state-of-the-art results on CWI are achieved by a sequence-labeling model of Gooding and Kochmar (2019a), however models of such type are less easily interpretable.
2.2 Aspects of word complexity
The question of what contributes towards the notion of word complexity has been investigated before, for example in readability studies. Word length is commonly believed to correlate with text complexity and is included as a component in a wide range
of readability formulas (Dale and Chall, 1948; Kincaid et al., 1975; Dubay, 2004). Frequency, another factor often considered in readability and text simplification approaches (Rudell, 1993; De Belder and Moens, 2010), was shown to correlate and cause word familiarity, which in its turn contributes to higher word recognition and lower reaction times (Connine et al., 1990; Morrel-Samuels and Krauss, 1992). Notably, word length and frequency have been widely used in CWI systems, and are reported to be good, cross-linguistic predictors of complexity (Bingel and Bjerva, 2018). Other factors considered important for word complexity include a variety of psycholinguistic properties, including word's age of acquisition, concreteness, and imagability (Carroll and White, 1973; Zevin and Seidenberg, 2002; Begg and Paivio, 1969). At the same time, not all factors are equally applicable to all groups of readers: for instance, while frequency may be an important factor for second language learners, other populations may be more affected by the length of a word or the occurrence of certain character combinations (Rudell, 1993; Rello et al., 2013). Yet, little is still known about the factors contributing to word complexity for native vs non-native readers as well as for non-native readers at different levels of language proficiency.
The most comprehensive CWI dataset to date was released by Yimam et al. (2017a) and further used in the CWI shared task 2018 (Yimam et al., 2018). This dataset has been annotated for complex words across a number of languages, including English, German, and Spanish. In this paper, we use the English portion of the data with the information about annotators' backgrounds<sup>1</sup>. The dataset contains texts from 3 different sources: professionally written news articles (NEWS), amateurishly written news articles (WIKINEWS), and WIKIPEDIA articles. The annotation was performed using the Amazon Mechanical Turk platform, where a total of 20 annotators, 10 native speakers and 10 nonnative speakers, were asked to mark words that they deemed complex for a given target readership, particularly children, language learners, and people with reading impairments. The workers were presented with text, consisting of 5 to 10 sentences (Figure 1), and were asked to select lexical items that they found complex (Figure 2). Workers use
$^{1}$ CWI Dataset with Language levels
their mouse pointer to highlight the complex units. The complex words or phrases included content words (e.g., nouns, verbs, adjectives, and adverbs) and phrases up to 50 characters in length. In this dataset, the complex units are considered if they are selected by at least one worker (Yimam et al., 2017a,b). Non-native speakers of English were asked to report their proficiency levels (beginner, intermediate, advanced). For our experiments, we concentrate on complex words only and disregard complex phrases. A break-down of proficiency labels for words (across all genres) is presented in Table 5, with label 1 denoting complex words and label 0 used for non-complex words. It is worth noting that the groups of annotators labelling portions of the dataset were not fixed. Within each group, the proficiency distribution varied, with some containing no annotators from a given class.
We firstly show that when predicting word complexity, the needs of sub-groups differ and are best predicted using models targeting them specifically. We demonstrate that the best performing models for a sub-group are trained with the annotations of that group using a classical machine learning approach. Secondly, we analyse the correlation of features with the number of annotators who found the word complex for both native and non-native groups. Finally, we investigate how the distributions of features vary for words marked as complex across audiences.
4.1 Complexity Features
To gain fundamental insights into the performance across proficiency groups, we run experiments using the CAMB system by Gooding and Kochmar (2018) as it achieved the best results across all binary and two probabilistic tracks in the CWI 2018 shared task (Yimam et al., 2018). Furthermore, the code for this system has been made publicly available by the authors. The CAMB system relies on 27 features in total. Feature types include lexical, syntactic, frequency-based and other aspects of information about individual words, outlined below.
Lexical Features: For each target word, the word itself as well as the length and number of syllables (obtained using the Datamuse API) is included. Additionally, the number of senses, hybernyms and hyponyms are collected for the word lemma using WordNet (Fellbaum, 2005). Fi
nally, the number of phonemes for the word are included sourced from the MCR Psycholinguistic Database (Wilson, 1988).
POS & Dependency Parse Relations: The target sentence is parsed using the NLPCore pipeline. Following this, the number of dependency relations are counted to produce a feature. The part-of-speech tag for the word is additionally included.
List-Based Features: A set of binary features are used that indicate the presence of the target word in a given list. The source of each list is outlined below:
- SubIMDB: using the SubIMDB corpus (Paetzold and Specia, 2016b), the word frequencies are calculated from the 'Movies and Series for Children' section. The top 1,000 most frequent words are then included.
- Simple Wikipedia (SimpWiki): a list of the top 6,368 words contained in the Simple Wikipedia (Coster and Kauchak, 2011).
- Ogden's Basic English: the top 1,000 words from Ogden's Basic English list (Ogden, 1968).
- Cambridge Advanced Learner's Dictionary (CALD):<sup>2</sup> the entries contained in the Cambridge Advanced Learner's Dictionary.
Word Frequency: The frequency of the target word is estimated using the Google dataset of n-grams (Goldberg and Orwant, 2013). Additionally, the Thorndike-Lorge written frequency derived from Thorndike and Lorge (1944) is obtained from the MCR Psycholinguistic Database (Wilson, 1988).
Psycholinguistic Features: Finally, the following features are extracted from the MCR Psycholinguistic Database (Wilson, 1988):
- Word familiarity rating (FAM)
- Imagability rating (IMG), representing the ease of associating the word with an image.
- Concreteness rating (CNC) represents the degree to which the word refers to a tangible entity, based on the norms of Gilhooly and Logie (1980).
- The number of categories (KFCAT) and samples (KFSMP) are derived from Kucera and Francis (1967).
- Age of acquisition (AOA) is based on the norms of Gilhooly and Logie (1980)
Figure 1: Complex word identification instruction with examples
Highlight hard words or phrases
(see instructions below)
#9-27 Camila Nunes, a sociologist of the Federal University of ABC, told the AFP "medium- and long-term policies to reduce the vulnerability of certain social groups [and] to prioritize prevention rather than repression" are needed.
#9-28 Reuters reported Alexandre de Moraes, minister of the Justice Department, recently authorized the state of Rio Grande do Norte to spend 13 million Brazilian reals to upgrade and expand prison equipment.
#9-29 De Moraes promised to prevent more prison riots by increasing funds and prison security.
#9-30 Meanwhile, Luiz Alberto Cartaxo, the prison chief for the southern Paraná state, said an explosion on Sunday broke a guarding wall of a Piraquara prison, prompting at least 21 inmates to escape.
#9-31 Cartaxo also reported that two other inmates were killed by police during their escape attempt.
Your selections:
Figure 2: Complex word identification annotation interface
4.2 Experimental Framework
The CAMB system uses the sklearn machine learning framework and achieves best results using an ensemble of algorithms. In our experiments, we use the logistic regression classifier as this was the best performing classifier for proficiency prediction due to the reduced number of annotations. As shown in Table 5, the number of annotations for each subgroup varies and the ratio of non-complex to complex words is highly skewed. For the data in our experiments, we firstly convert all proficiency annotations to a binary format, where if at least one annotator has marked the word as complex the word is given a binary label of 1. For our initial experiments the aim is to see if the needs of a proficiency group are best predicted by that target group. In order to make a fair comparison, we control for the number of binary annotations by restricting all groups to the same amount of labels as in the beginner class (2, 263).
The annotations are ordered by the highest class agreement and the top 2, 263 values are selected. Additionally, we remove  $20\%$  of non-complex labels, where no proficiency groups had marked the word as complex, to re-balance the class distribution to that of the original binary shared task. This resulted in a dataset containing 9,828 non-complex words and 4,423 words marked with at least one proficiency annotation. Stratified 5-fold cross-validation was used resulting in a test size of 2,850 and total training size of 11,400 per fold.
In all experiments, 5-fold stratified cross validation is performed and the average scores across folds presented. Table 1 shows the results of training the system using the annotations of one proficiency subgroup and the subsequent model performance across subgroups. Columns represent the training annotations used and the rows represent the results on the respective test sets. As a result of the small training size, the overall F1-Score achieved across classes is low. For instance, when all avail
2Publicly available here
<sup>3</sup>http://scikit-learn.org/stable/
TRAINING DATA
TEST	Beginner	Intermediate	Advanced
PRECISION	RECALL	F1-Score	PRECISION	RECALL	F1-Score	PRECISION	RECALL	F1-Score
Beginner	0.649	0.245	0.356	0.425	0.289	0.344	0.433	0.270	0.333
Intermediate	0.529	0.201	0.291	0.669	0.452	0.538	0.596	0.423	0.494
Advanced	0.513	0.196	0.283	0.594	0.398	0.477	0.659	0.476	0.552
Table 1: Results of models trained and tested with differing proficiency labels
TRAINING DATA
TEST	Native	Non-native
PRECISION	RECALL	F1-SCORE	PRECISION	RECALL	F1-SCORE
Native	0.794	0.801	0.797	0.761	0.796	0.773
Non-native	0.766	0.730	0.748	0.785	0.792	0.788
Table 2: Results of models trained and tested with native and non-native annotations
TEST	Native
PRECISION	RECALL	F1-Score
Beginner	0.232	0.789	0.359
Intermediate	0.539	0.794	0.642
Advanced	0.623	0.803	0.702
Table 3: Results of model trained with native annotations across non-native proficiency
Binary Labels
1	0
Beginner	2,263	27,433
Intermediate	5,203	24,493
Advanced	5,849	23,847
Table 5: Binary label distribution for words per proficiency class,  $I$  is complex and  $O$  is simple.
PREC	REC	F1-SCORE
Beginner(2263)	0.62	0.22	0.33
Intermediate(5203)	0.80	0.80	0.80
Advanced(5849)	0.76	0.77	0.78
Table 4: Results showing PRECISION, RECALL and F1-SCORE using all sub-group annotations
able labels are used for intermediate and advanced classes an F1-Score of over  $75\%$  is achieved as shown in Table 4. However, the results are still highly informative, as we observe that in all cases the best F1-Score is obtained when the original sub-group annotations are used. This finding supports the case that the needs of such sub-groups differ and are best predicted using models targeting them specifically. The PRECISION, RECALL and F1-Score across all categories are best when the model is trained using the annotations of the target subgroup. The only exception is RECALL for beginner, where the intermediate and advanced models perform the best (results underlined). However, it is worth noting that if an intermediate or advanced learner considers a word to be complex, it is highly likely that a beginner will too. This observation is further supported by the finding that whilst the
advanced and intermediate models perform adequately on the beginner test set, the beginner model performs very poorly when predicting the needs of intermediate or advanced users. The advanced and intermediate models achieve higher F1-Scores than the beginner model. These results support the case that beginner word acquisition is more idiosyncratic than at an intermediate or advanced level where the concept of word complexity converges.
Table 2 additionally shows that the complex annotations of a subgroup are the best predictors for that class. We observe that the best results for the native group occur when trained with native only annotations and the same holds for the non-native class.
We perform experiments by training with native complexity annotations and observe the performance across non-native proficiency groups. The results of these are shown in Table 3, and as there is a larger training set the scores are higher than those in Table 1. We see that the native annotations perform best when predicting the advanced non-native word complexities. However, this is not the case for the beginner class. We also observe a pattern in native annotations being preferential for higher
Figure 3: Graphs showing the top 5 correlated features against the absolute number of annotations for the native and non-native classes all values are significant ( $N = 17250$ ;  $p < .001$ )
Figure 4: The average percentage of complex words as identified by CWI models trained with advanced and beginner annotations on the Newsela dataset
proficiency levels.
Newsela results
(2) His frequent use of prepositions suggests he was rigorously educated in grammar.
(3) The way he wrote shows he was very educated in grammar.
We apply our beginner and advanced CWI models on an additional dataset, Newsela.4 Newsela contains articles which are rewritten by professional editors at differing levels of simplicity with each grade level as defined by the Common Core Standards (Porter et al., 2011). We take the highest, intermediate and lowest level of each article and perform CWI using the models trained with all advanced and beginner annotations. Our aim is to see if these models are able to differentiate between levels as CWI has been shown to be an important component in readability assessment systems (Maddela and Xu, 2018). In Figure 4, we see that the model trained with the annotations from beginners identifies a higher percentage of words as complex across levels when compared to the advanced model. Additionally, both models identify
more complex words in the advanced texts than in the intermediate or beginner. These results show that models trained for specific audiences can result in a different concept of complexity. For instance, examples 2 and 3 show a sentence from an advanced and simplified article. Words in bold are identified as complex by the advanced model and italicised if found complex by the beginner model. We see that in the higher level sentence (2), two words are identified as difficult by both models and one word is identified as complex by only the beginner model. In the lower level article, the words identified as complex by both models have been simplified. This results in only one word being identified as complex by the model tailored for beginners. We know that text begins to be accessible for non-native readers if they are familiar with at least  $90\%$  of word content (Nation, 2006). Therefore, being able to model text understanding across audiences relies on audience specific models of word complexity as demonstrated in our example.
Feature Correlations
As the absolute number of native and non-native annotators remained constant across annotations (i.e. 10), we explore the feature correlations for these subgroups. For instance, the word vowed in a given context has been marked as complex by 10 non-native and 1 native annotator. This indicates that the word might be more challenging for a non-native audience than for native in the given context.
Figure 3 shows the highest correlated features for the native and non-native groups, all of which are significant  $(p < .001)$ . Overall, the correlations for the native class are higher than for non-native which is likely due to a more united perspective of complexity. This follows as individuals with a similar first language or educational background are more likely to annotate the same words as complex (Specia et al., 2012).
For both classes, the feature with the highest
<sup>4</sup>https://newsela.com
correlation is that of word length: the positive correlation shows that the longer the word, the more likely it will belong to the complex class. Following this, for the native class we see that the number of syllables is second. Whilst the length of a word and the number of syllables are highly correlated (0.64), it is interesting to note that the number of syllables correlates more highly with the native notion of complexity than for non-native. This may be explained by the fact that syllable and phoneme awareness plays an independent role in the processing of text (Engen and Hoien, 2002). This impact is especially pronounced in lower skilled readers, where due to a reduced vocabulary set, the development of precise phonological representations are not yet formed (Elbro, 1996).