SlowGuess/NAACL_2022_OCR huggingface dataset:
Sian Gooding  
University of Cambridge  
shg36@cam.ac.uk
Manuel Tragut Google mtragut@google.com
Complex Word Identification (CWI) aims to detect words within a text that a reader may find difficult to understand. It has been shown that CWI systems can improve text simplification, readability prediction and vocabulary acquisition modelling. However, the difficulty of a word is a highly idiosyncratic notion that depends on a reader's first language, proficiency and reading experience. In this paper, we show that personal models are best when predicting word complexity for individual readers. We use a novel active learning framework that allows models to be tailored to individuals and release a dataset of complexity annotations and models as a benchmark for further research.
Illiteracy affects at least 773 million people globally. For these individuals, reading information from unfamiliar sources or on unfamiliar topics can be extremely difficult. Furthermore, there are a reported 1.5 billion English-language learners worldwide, spanning all proficiency levels. Statistics such as these illustrate that the comprehension of written text across the world varies substantially. Even those who are deemed functionally literate have differing degrees of text comprehension. There are many characteristics of a written text that impact how difficult it is to read. However, one of the main aspects contributing to overall text difficulty is lexical complexity, i.e. word complexity (Dubay, 2004). Lexical content has been shown to be more important than syntactic features when predicting readability (Vajjala and Meurers, 2012) and a study performed by Nation (2006) found that a reader needs to be familiar with at least  $95\%$  of the words contained within a text for them to understand the content.
(1) The cantankerous cat.  $\downarrow$  The grumpy cat.
Complex word identification (CWI) focuses on identifying words in a text that may be difficult for a reader to understand and therefore may benefit from simplification (Shardlow, 2013a). For instance, in example (1) a CWI system may identify cantankerous as a complex word, then a lexical simplification system is able to substitute it with a simpler alternative, e.g. grumpy (Paetzold and Specia, 2016a; Gooding and Kochmar, 2019c).
Many downstream tasks benefit from the identification of complex words. For example, CWI has been shown to improve readability assessment systems (Maddela and Xu, 2018) as well as vocabulary acquisition modules in educational applications (Zaidi et al., 2020). Furthermore, CWI can significantly reduce errors in simplification systems (Shardlow, 2014) and result in higher quality simplifications (Lee and Yeung, 2018).
Previous work on modelling CWI, including two shared tasks, has relied on the collection of large corpora containing words that have been annotated for their complexity (Paetzold and Specia, 2016c; Yimam et al., 2018). However, annotating the difficulty of words is a subjective task, and previous data collection has yielded low levels of annotator agreement (Specia et al., 2012; Paetzold and Specia, 2016c). As a way of mitigating individual differences, such datasets typically present a homogeneous view on word complexity, by merging annotations across readers (Gooding et al., 2021). Further attempts to improve annotator agreement have included offering bonus incentives for annotators who select words matching other annotations (Yimam et al., 2017), as well as providing guidelines for annotators to mark words that they assume would be complex for audiences such as children and those with learning difficulties. Such incentives
$^{1}$ Dataset is available at: https://github.com/siangooding/personal_CWI
2https://en.unesco.org/themes/literacy
Findings of the Association for Computational Linguistics: NAACL 2022, pages 353 - 365 July 10-15, 2022 ©2022 Association for Computational Linguistics
and instructions change the motivation of annotators and undermine the validity of the resulting complexity labels (Gooding, 2022).
It is evident that lexical complexity is a highly subjective and idiosyncratic notion. In this paper, we argue that contrary to the one-size-fits-all paradigm currently dominating many natural language processing tasks, the modelling of individual differences in word complexity is critical. Many of the downstream tasks that benefit from CWI, such as text simplification, would be more effective if tailored to the individual reader. In order to facilitate personalised CWI, and therefore personalised text simplification and readability systems, audience specific complexity annotations are required (Bingel et al., 2018). Whilst it has been shown that the concept of word complexity, and thus the level of agreement, is aligned between individuals sharing a common background (Gooding and Kochmar, 2018a; Gooding et al., 2021), we argue that the best CWI model for each individual is trained with them 'in the loop'.
In our paper, we use active learning to produce per-individual models of word complexity. We collect complexity annotations from 1,225 English as a second language (ESL) readers. Using a novel active learning framework, we show that individual models of word complexity can be trained without extensive annotation. Our experiments demonstrate that bespoke models outperform baselines and a state-of-the-art system when predicting word complexity for individuals, thereby emphasising the importance of viewing the task at a personal level. We present the following contributions:
- We are the first to train personalised word complexity models and show that training can be achieved in real-time using active learning.
- We show that per-individual models outperform baseline and state-of-the-art approaches when predicting word complexity for individuals, most notably for lower proficiency levels.
- We demonstrate the benefits of bespoke CWI models in downstream tasks such as text simplification and proficiency prediction.
- We release a novel dataset containing 55,125 English word complexity annotations from 1,225 ESL participants, as well as all trained models, as a benchmark for further research.
2.1 Complex Word Identification
The first dataset collected for the evaluation and benchmarking of CWI was the CW corpus (Shardlow, 2013b). The dataset contained 731 instances, collected using simplifications made by human editors when adapting content for Simple Wikipedia articles. This dataset was extended with further Wikipedia examples, including data collected by Horn et al. (2014), for the first CWI shared task. The resulting dataset contained a total of 9,200 sentences which were annotated by 400 non-native speakers for complex words. The shared task organised by Paetzold and Specia (2016b) required the prediction of binary word complexity for non-native speakers. However, as the dataset had low annotator agreement, the merged notion of complexity was difficult to model, and all systems resulted in low F-scores.
Following this, Yimam et al. (2017) collected a CWI dataset which was used in a second shared task (Yimam et al., 2018). The annotations for this data were collected using the Amazon Mechanical Turk platform. For each sentence, ten non-native and ten native English speakers were asked to mark words or phrases as complex. Words and phrases were annotated as complex (label 1) if at least one of the 20 annotators annotated them as such, and simple (label 0) otherwise. The winning system for the binary track was a feature based ensemble approach (Gooding and Kochmar, 2018b). The system used traditional features associated with word complexity such as the frequency, word length and psycholinguistic attributes of words. The performance on this dataset was subsequently improved using a neural sequence labelling approach by Gooding and Kochmar (2019a).
The prior datasets and systems largely focused on binary approaches to modelling CWI. Recognising the limitation of viewing word complexity in this way, Shardlow et al. (2021) arranged a shared task on lexical complexity prediction based on the Shardlow et al. (2020) dataset. The dataset contains judgements on word complexity using a Likert scale of 1-5, allowing an indication of the annotators' familiarity with the term. The framing of word complexity in a continuous fashion allows for more subjective approaches of CWI to be introduced. However, overall judgements of difficulty in this dataset are still produced by averaging the labels across multiple annotators. Whilst the data
gives an insight into complexity as a continuous measure, it does not provide insight into the individual differences across annotators.
Closest to our work, Gooding et al. (2021) make the case for audience specific CWI by showing that the best models for a given proficiency group are produced using annotations from the target audience. They use proficiency annotations from the Yimam et al. (2017) dataset and find that the features contributing to word complexity differ depending on the audience background. This work illustrates the importance of considering the target audience in CWI. However, the populations investigated still represent coarse groups, such as native vs non-native, or beginner, intermediate and advanced. Whilst this work is a step in the right direction for subjective complexity, there still remain many differences in complexity judgements even within such groups.
Finally, a similar task to CWI is that of vocabulary prediction, whereby the size and extent of a learner's vocabulary is estimated (Meara and Buxton, 1987; Laufer and Nation, 1999). Whilst word difficulty is subtly different to vocabulary size, as a learner may recognise a word but still find it difficult to understand, the areas are closely aligned. Empirical studies have found strong correlations between receptive vocabulary size tests and reading comprehension tests for ESL readers with learners from different proficiency levels (e.g. Laufer, 1992; Qian, 1999, 2002; Henriksen et al., 2004). In fact, vocabulary size has been found to predict as much as  $72\%$  of the variance in reading, leading many researchers to emphasise that receptive vocabulary size is the determinant factor for reading success in L2 (Staehr, 2008). Therefore, being able to predict a readers' vocabulary size, with a small number of samples, is very useful. The effectiveness of active learning for vocabulary prediction was investigated by Ehara et al. (2014), who showed that graph-based active learning improves vocabulary prediction for Japanese speakers. However, the active learning process is simulated and not performed in real-time with participants.
Annotated data can be time consuming and expensive to obtain, notably in specialised domains where only experts can provide reliable labels (Konyushkova et al., 2017). By enabling a classifier to interactively query data points, active learn
ing allows machine learning classifiers to achieve higher accuracies with fewer training instances. It is well-motivated in many modern machine learning problems where data may be abundant, but labels are scarce or expensive to acquire (Settles, 2009). For instance, active learning has been used in text classification (Hoi et al., 2006), information extraction (Settles and Craven, 2008) and financial applications (Gooding and Briscoe, 2019).
The task of personalised CWI is a good candidate for active learning for two reasons:
Labelling time Previous CWI datasets contain thousands of annotated instances to train word complexity models. It is unfeasible to collect this amount of data, in a reasonable time frame, for an individual reader.
Learner lifetime In complex word identification we can expect the needs of a user to change over their 'learner lifetime'. Whilst a model may be effective for a reader at one point in time, as their proficiency improves, the model will require retraining.
2.2.1 Uncertainty Sampling
In uncertainty sampling an active learner queries the instances it is least certain about how to label. This is implemented using an uncertainty measure and is one of the most commonly used active learning frameworks (Lewis and Gale, 1994; Settles, 2009). For many real-world learning problems, large collections of unlabelled data can be gathered easily. For instance, in our setting the unlabelled data represents all possible words. This motivates an uncertainty sampling approached called pool-based sampling (Lewis and Gale, 1994) which assumes that there is a small set of labelled data  $\mathcal{L}$  and a large pool of unlabelled instances  $\mathcal{U}$ . Instances are drawn from this unlabelled pool for labelling in a greedy fashion, according to an informativeness measure calculated by applying the classifier and measuring the uncertainty.
In our paper, we employ active learning for the task of complex word identification. We train classifiers to identify whether a word would be considered difficult for a reader based on 23 word familiarity questions.
3 Experiment Design
In our experiment, participants are presented with vocabulary and asked whether they could confidently define the meaning. There are two stages to
the data collection; initially, word annotations are requested in tandem with a classifier as part of an active learning process. During this stage, participants are shown a total of 23 words to annotate. $^{3}$  These annotations are used to train a personal word complexity model. Following this, annotations are collected for a test set, per participant, which can be used to validate the trained classifier. The test set contains 22 words across CEFR levels and is presented in a randomised order. Annotators are not aware of the switch from training to test data collection. The study was hosted online and implemented using the Flask framework for Python (Grinberg, 2018).
3.1 Participants
We collected annotations via the Prolific platform from 1,255 ESL readers spanning 57 different first languages.4 Participants were required to fill out a demographic questionnaire which is available to view in our data repository. The questionnaire was informed using guidelines on judging reading ability (Acheson et al., 2008) and included the self-reported English proficiency, native language, hours spent reading per week and highest level of formal education. Demographic statistics are available in Appendix A.
3.2 Materials
We created a test set containing 22 words spanning different levels of the Common European Framework of Reference for Languages (CEFR) (North and Piccardo, 2020) shown in Figure 1. The items were selected by sampling those with the fewest dictionary entries to minimise word ambiguity.
To bootstrap the active learning process, we train an initial logistic regression classifier with 150 instances from the CWI dataset of Yimam et al. (2017). The features used are those found to correlate most with non-native judgements of word complexity according to Gooding et al. (2021), namely the word length, word frequency and psycholinguistic properties of the words such as the familiarity, concreteness, and imagability, which have been shown as important when predicting word complexity (Carroll and White, 1973; Begg and Paivio, 1969; Zevin and Seidenberg, 2002). This classifier is then tuned to the individual during the active learning process.
3.2.1 Pre-clustering
A typical approach for collecting annotations, in an active learning setting, is to select samples close to the classification boundary. However, it has been shown that taking into account the prior data distribution can improve performance (Settles, 2009). One way to achieve this, is by modelling the input distribution using clustering, and propagating label information to instances in the same cluster (Nguyen and Smeulders, 2004). In our experiment, we use the Yimam et al. (2017) dataset as our unlabelled pool and perform clustering on the 7,476 uniquely labelled words prior to active learning.
We use agglomerative clustering, a hierarchical clustering approach which performs clustering in a bottom up fashion (Gowda and Krishna, 1978). Each word initially belongs to its own cluster and the algorithm successively merges these to form larger groups. This type of unsupervised clustering has been used across multiple natural language processing tasks including word clustering, co-reference resolution and word sense disambiguation (Chen and Ji, 2010). The features used for clustering are the same as employed for the task of the complexity prediction.
We set the number of clusters to 7 in an attempt to align with the CEFR categories chosen in our test set. One disadvantage of hierarchical clustering methods, is that they have large storage requirements, and so can be computationally intensive to compute (Gan et al., 2020). However, as we pre-calculate and cache clusters the time complexity is that of base informativeness measures, e.g. uncertainty sampling (Settles and Craven, 2008). The time complexity is a necessary consideration, as we are conducting active learning interactively with oracles in real-time.
An example of words from the resulting clusters is displayed in Figure 2a. Words of length 5 were sampled randomly and have been colour coded according to the corresponding CEFR level. $^{5}$
3.2.2 Cluster Evaluation
We evaluate our clusters using two techniques; the first employs EFLLEX which is a graded lexical resource for learners of English as a foreign language. The dataset, collected by Dürlich and François (2018), contains 15,280 entries from 1,971 graded texts. The frequency of the lemma at each level of
323 annotations were found to be the minimum number required for model convergence during trial runs.
<sup>4</sup>https://prolific.co
5CEFR levels were obtained via https://languageeresearch.cambridge.org/wordlists/text-inspector
Figure 1: Test set
Cluster	Word Sample
1	first	water	woman	A1
2	cases	grown	joins	A2
3	false	inner	risky	B1
4	brief	votes	stock	B2
5	major	index	blend	C1
6	stake	coral	plank	C2
7	glyph	overt	trove	C2+
(a) Sample of words from clusters
(b) Frequency of vocabulary across levels per cluster present in EFLLEX
(c) Histogram showing the density of annotations present per cluster
the CEFR (C2 excepted) is provided. A total of 5,983 entries were present in both EFLLEX and our clusters. We plot the average frequencies of words in each cluster across CEFR levels in Figure 2b. Generally, the plot shows that there are clear differences in the levels of vocabulary across clusters. When we consider A1 texts, the cluster with the highest average frequency of words is Cluster 1. There is a linear decrease in the frequency of words at this level as the cluster level increases. When we consider the proportion of higher level vocabulary (i.e. B1 to C2), a clear trend emerges where Clusters 5-7 contain words that appear much more frequently at this level. Cluster 7, contains words at the highest level of CEFR for instance glyph, and is shown to contain words that occur much more commonly at C1 and B2 level.
The second way we evaluate clusters is using the annotations of the Yimam et al. (2017) dataset. As we use this data for our unlabelled pool, we have the complexity annotations for each item from both native and non-native annotators. Our work focuses on complexity prediction for ESL readers and so we evaluate clusters using the non-native annotations. This results in a complexity score per word ranging from 1-10 which represents how many non-native annotators considered the word to be complex. A histogram showing the density of non-native complexity annotations present for words in each cluster is shown in Figure 2c. The average of these scores per cluster  $(\bar{x})$  is shown in Figure 2c,
Figure 2: Clustering statistics
Figure 3: Figure showing the stages of our active learning framework
as well as the number of words present in each cluster  $(n)$ . Clusters are colour coded according to the same key in Figure 2c. Cluster 7 contains the highest average complexity labels (5.39) and Cluster 1 the lowest (0.01). As the cluster levels increase, we see a shift in the density of complexity annotations present, supporting the notion that these clusters contain vocabulary at differing levels.
3.3 Active Learning Loop
Figure 3 shows the four stages that take place during the active learning process. Each of these steps is described below.
(1) Annotation In the initial step, the annotator is presented with a word form and asked whether they could confidently define the meaning. Whilst the context of a word impacts its meaning and com
plexity, we use word forms as this allows for a pool-based active learning approach. Furthermore, the use of word forms alone has been shown as an effective measure of proficiency and vocabulary prediction (Lemhöfer and Broersma, 2012; Ehara et al., 2014). The user interface showing the instructions for this experiment are provided in Appendix B. When the annotator selects an answer this provides a binary annotation for the word.
(2) Label propagation The chosen label is then propagated to the closest 150 instances from the clustered pool.
(3) Re-train model We train a log-linear model (i.e. logistic regression) per individual, for the following reasons: the model can provide a class ownership probability which is required for active learning, log-linear models are flexible and interpretable, finally the algorithm is efficient and able to train in  $\mathcal{O}(nd)$  time, where  $n$  is the number of samples and  $d$  the feature dimensions (Gujarati and Porter, 2009; Bulso et al., 2019). As each model is retrained every time an annotation is provided, the speed of training was a critical consideration.
(4) Apply model and re-rank Once the model has been re-trained, it is applied to the unlabelled pool so that the samples can be ranked according to the model uncertainty. The uncertainty sampling technique we use is entropy (Shannon, 1948). In the following notation  $x^{*}$  denotes the most informative instance from an unlabelled vocabulary pool:
$$
x _ {H} ^ {*} = \underset {x} {\operatorname {a r g m a x}} - \sum_ {i} P _ {\theta} (y _ {i} | x) \log P _ {\theta} (y _ {i} | x)
$$
where  $y_{i}$  ranges over the possible labellings. The instance from the pool that the model is least certain about is then presented to the annotator for the next annotation step.
In our experiment, each participant produces a gold test set containing annotations for words across CEFR levels. This varies for each individual, as word complexity is subjective. For instance, consider the test set samples shown in Table 1; words marked with a 1 have been annotated as difficult by participants. Participant  $a$ , who is self-reported to be intermediate, has found three words in this sample difficult, whereas participant  $b$ , self-reported at a near native level, only reports one.
World	Realm	Merit	Alms
Pa	0	1	1	1
Pb	0	0	0	1
Table 1: Test set samples from participants  ${P}_{a}$  and  ${P}_{b}$
We calculate the F-score and Kappa agreement across all annotations to evaluate the performance of bespoke models. The F-score represents the harmonic mean between the precision and recall of the classifier, whereas the Kappa coefficient measures pairwise agreement, correcting for expected chance agreement (Cohen, 1960).
We additionally run an experiment to evaluate the effectiveness of active learning compared to randomly sampling vocabulary. In the experiment, we require annotators to label a sample of 22 random words, as well as 22 words sampled using active learning. As this doubles the annotation effort we perform this with a subset of 100 annotators. The results are shown in Table 3. We compare with models trained using random sampling as well as random sampling with cluster-based label propagation. The best results are achieved using the active learning framework.