SlowGuess/NAACL_2022_OCR huggingface dataset:
We evaluate the performance of the active learning models and compare to the following baselines. (1) Group Average: we calculate the average of the test set complexity annotations across participants, if a word has been labelled as complex by more than  $10\%$  of the target group it is labelled as difficult. This is a competitive baseline as we use the direct annotations of the target audience on the test sets. (2) Seq-CWI: we compare to a state-of-the-art system for complex word identification (Gooding and Kochmar, 2019a). However, we note that this is an unfair comparison as the system is one-size-fits-all and not audience specific. (3) Frequency: relies on frequency thresholding, where all words below a given frequency are marked as complex. (4) All Simple: all words are marked as simple.
Table 2 shows the results of the active learning models compared to baselines. We group results according to the proficiency of participants. The beginner results are shown for completeness, but should be interpreted with caution due to the smaller test set size (176). Kappa values in the range of 0.40-0.60 represent moderate agreement (Cyr and Francis, 1992), although it has been argued that scores above 0.45 represent substantial agreement (Munoz and Bangdiwala, 1997).
Model	Proficiency level
All	Beginner	Intermediate	Advanced	Near Native	Native
F-score
Active Learning	0.751	0.763	0.720	0.752	0.764	0.761
Group Average	0.669	0.608	0.624	0.679	0.726	0.754
Seq-CWI	0.677	0.620	0.688	0.689	0.660	0.642
Frequency	0.534	0.461	0.529	0.541	0.530	0.514
All Simple	0.441	0.450	0.436	0.437	0.447	0.454
Kappa
Active Learning	0.503	0.525	0.441	0.504	0.529	0.524
Group Average	0.379	0.296	0.304	0.395	0.473	0.523
Seq-CWI	0.395	0.300	0.409	0.414	0.371	0.347
Frequency	0.189	0.083	0.173	0.194	0.196	0.182
Test size	26,136	176	4,180	12,122	7480	2,156
Table 2: Model results grouped by annotator proficiency
Model	F-score	Kappa
Random	0.605	0.218
Cluster	0.649	0.361
AL	0.740	0.482
Table 3: Results showing differing sampling techniques applied to 2,200 annotations from 100 participants
When we consider the active learning models, we see that the F-score calculated across all test annotations is 0.751, and the Kappa agreement is 0.503. The active learning models outperform the competitive group-average baseline which supports the hypothesis that individual models are preferable to aggregate group-level approaches. The best scores for all proficiency groups are achieved using the active learning framework. However, for the native annotations, the difference between the group-based approach and active learning models is small. When considering the group-average baseline we see that both the F-score and Kappa values increase in line with proficiency. This supports the notion that the concept of word complexity stabilises for higher proficiency levels (Krstic et al., 2018; Gooding et al., 2021). As such, aggregate models may work well for highly proficiency audiences and individual models best for lower levels where word knowledge is more variable.
Each annotator produces a bespoke model based on their personal perception of word difficulty. We
tested whether there was a significant difference in complexity predictions of models depending on the annotator's self-reported proficiency. To investigate this, we selected 100 models from near native, advanced and intermediate annotators. We control for the number of classifiers due to having an unequal number of annotators across levels.
These models are applied to the EFLLEX dataset, removing any vocabulary that may have been seen during train time. This results in complexity predictions for approximately 10,000 words per model. The average number of words found complex by these classifiers is shown in Table 5. The intermediate classifiers, on average, identify more complex words across all vocabulary levels, whereas the near native classifiers predict fewer words as difficult across all levels compared to both the intermediate and advanced classifiers. We calculate the significance using Satterthwaite's method (Kuznetsova et al., 2017), applied to a mixed-effects model that treats the first language of annotators and the trained model as random effects. We find that the difference between model predictions depending on the proficiency of the annotator is highly significant  $(p < 1 \times 10^{-10})$ .
Being able to distil the vocabulary experience of an individual, using a limited number of annotations, is extremely useful for many applications.
$^6$ Using R formula notation, the model is: modelpredictions  $\sim$  annotator_profiency + (1 | annotator_L1) + (1 | model)
Near Native:	0.34	0.32
Advanced:	0.51	0.47
Intermediate:	0.59	0.59
(1) The	current	cuts	in	public	expenditure	will	exacerbate	this situation.
↓
(2) The	current	cuts	in	public	spending	will	exacerbate	this situation.
(3) The	current	cuts	in	public	spending	will	worsen	this situation.
Table 4: Example of how simplification output changes depending on audience specific CWI models. The complexity probability for audiences is the result of aggregating output from all trained models across proficiencies. A word is considered complex if the probability is above 0.50.
A1	A2	B1	B2	C1
Intermediate	333	881	1208	2109	2474
Advanced	277	754	1016	1773	2062
Near Native	232	643	866	1505	1735
Table 5: The average number of words identified as complex across CEFR levels. Results are averaged from a sample of 100 classifiers from each proficiency group.
For instance, when judging the proficiency of a reader. Using our trained classifiers, we run a preliminary experiment to see if it is possible to predict the proficiency of the annotator using their model predictions. We train a linear model to predict whether the annotator self-reported at an intermediate, advanced or near native level. We use the total number of C1 words identified as complex as a feature and test using 5-fold stratified cross-validation. By training a simple model with this feature alone, we can obtain a precision of 0.70. We release the trained models with our paper to encourage further work on the applications and evaluation of personal CWI models for proficiency prediction.
Finally, a further application that benefits from individualised models of word complexity is text simplification. To illustrate, we provide an example in Table 4 which was produced using an open source lexical simplification system by Gooding and Kochmar (2019c). The system is designed to accommodate differing models of word complexity so that simplification output can be personalised. In this example, we use the mean probability that a word is complex by averaging the output from our classifier groups. Sentence (1) shows the result of using near native CWI models, no words are considered complex and so no simplifications are performed. The advanced models produce a probability larger than 0.50 for the word expenditure
and it is therefore simplified. Finally, the intermediate models result in two words being identified as complex which are subsequently simplified. This example provides further motivation for why considering word complexity in a subjective manner is beneficial.
We show that with active learning it is feasible to build individual models of word complexity. Our models are able to be trained efficiently, in real-time, using only 23 word annotations per individual. Models trained in this fashion are able to accommodate the subjective nature of word complexity, thereby facilitating personalised readability and simplification systems. We release our dataset containing 55,125 word complexity annotations, participant metadata and models, providing a new benchmark for the task of human-centered word complexity prediction.
Our work is a first step towards the development of personal complexity models. In future work, we aim to more extensively investigate the relationship between a learner's perceived and actual knowledge of words, as it has been reported that there is a misalignment between what a learner thinks they know, and what they are actually able to define (Laufer, 1997). Additionally, the context of a word impacts its perceived difficulty (Gooding and Kochmar, 2019b; Strohmaier et al., 2020); considering token-level complexity as opposed to type-level complexity is another important avenue for future work.
Daniel J Acheson, Justine B Wells, and Maryellen C MacDonald. 2008. New and updated tests of print exposure and reading abilities in college students. Behavior research methods, 40(1):278-289.
Ian Begg and Allan Paivio. 1969. Concreteness and imagery in sentence meaning. Journal of Verbal Learning and Verbal Behavior, 8(6):821-827.
Joachim Bingel, Gustavo Paetzold, and Anders Søgaard. 2018. Lexi: A tool for adaptive, personalized text simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 245-258, Santa Fe, NM, USA.
Nicola Bulso, Matteo Marsili, and Yasser Roudi. 2019. On the complexity of logistic regression models. Neural computation, 31(8):1592-1623.
John B Carroll and Margaret N White. 1973. Word frequency and age of acquisition as determiners of picture-naming latency. The Quarterly Journal of Experimental Psychology, 25(1):85-95.
Zheng Chen and Heng Ji. 2010. Graph-based clustering for computational linguistics: A survey. In Proceedings of TextGraphs-5-2010 Workshop on Graph-based Methods for Natural Language Processing, pages 1-9.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Louis Cyr and Kennon Francis. 1992. Measures of clinical agreement for nominal and categorical data: the kappa coefficient. Computers in biology and medicine, 22(4):239-246.
William H. Dubay. 2004. The Principles of Readability. Costa Mesa, CA: Impact Information.
Luise Dürlich and Thomas François. 2018. Efflex: A graded lexical resource for learners of english as a foreign language. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Yo Ehara, Yusuke Miyao, Hidekazu Oiwa, Issei Sato, and Hiroshi Nakagawa. 2014. Formalizing word sampling for vocabulary prediction as graph-based active learning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1374-1384.
Guojun Gan, Chaoqun Ma, and Jianhong Wu. 2020. Data clustering: theory, algorithms, and applications. SIAM.
Sian Gooding. 2022. On the ethical considerations of text simplification.
Sian Gooding and Ted Briscoe. 2019. Active learning for financial investment reports. In Proceedings of the Second Financial Narrative Processing Workshop (FNP 2019), pages 25-32, Turku, Finland. Linköping University Electronic Press.
Sian Gooding and Ekaterina Kochmar. 2018a. CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting. In Proceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, pages 184-194, New Orleans, Louisiana.
Sian Gooding and Ekaterina Kochmar. 2018b. CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 184-194, New Orleans, LA, USA. Association for Computational Linguistics.
Sian Gooding and Ekaterina Kochmar. 2019a. Complex word identification as a sequence labelling task. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1148-1153.
Sian Gooding and Ekaterina Kochmar. 2019b. Complex Word Identification as a Sequence Labelling Task. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1148-1153, Florence, Italy. Association for Computational Linguistics.
Sian Gooding and Ekaterina Kochmar. 2019c. Recursive context-aware lexical simplification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4855-4865, Hong Kong, China. Association for Computational Linguistics.
Sian Gooding, Ekaterina Kochmar, Seid Muhie Yimam, and Chris Biemann. 2021. Word complexity is in the eye of the beholder. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4439-4449.
K Chidananda Gowda and G Krishna. 1978. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):105-112.
Miguel Grinberg. 2018. Flask web development: developing web applications with python. "O'Reilly Media, Inc."
DN Gujarati and DC Porter. 2009. How to measure elasticity: The log-linear model. *Basic Econometrics*, McGraw-Hill/Irwin, New York, pages 159-162.
BIRGIT Henriksen, DORTE Albrechtsen, and KIRSTEN Haastrup. 2004. The relationship between vocabulary size and reading comprehension in the 12. Angles on the English-speaking World, 4(1):129-140.
Steven CH Hoi, Rong Jin, Jianke Zhu, and Michael R Lyu. 2006. Batch mode active learning and its application to medical image classification. In Proceedings of the 23rd international conference on Machine learning, pages 417-424.
Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a lexical simplifier using Wikipedia. In Proceedings of the 52nd ACL, pages 458-463.
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. 2017. Learning active learning from data. In Advances in Neural Information Processing Systems, pages 4225-4235.
Ksenija Krstic, Anela Šoskić, Vanja Ković, and Kenneth Holmqvist. 2018. All good readers are the same, but every low-skilled reader is different: an eyetracking study using pisa data. European Journal of Psychology of Education, 33(3):521-541.
Alexandra Kuznetsova, Per B Brockhoff, Rune HB Christensen, et al. 2017. lmertest package: tests in linear mixed effects models. Journal of statistical software, 82(13):1-26.
Batia Lafer. 1992. How much lexis is necessary for reading comprehension? In Vocabulary and applied linguistics, pages 126-132. Springer.
Batia Laufer. 1997. The lexical plight in second language reading: Words you don't know, words you think you know, and words you can't guess. Second language vocabulary acquisition: A rationale for pedagogy, 1:20-34.
Batia Laufer and Paul Nation. 1999. A vocabulary-size test of controlled productive ability. Language testing, 16(1):33-51.
John Lee and Chak Yan Yeung. 2018. Personalizing lexical simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 224-232.
Kristin Lemhöfer and Mirjam Broersma. 2012. Introducing lextale: A quick and valid lexical test for advanced learners of english. Behavior research methods, 44(2):325-343.
David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR'94, pages 3-12. Springer.
Mounica Maddela and Wei Xu. 2018. A word-complexity lexicon and a neural readability ranking model for lexical simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3749-3760, Brussels, Belgium. Association for Computational Linguistics.
Paul Meara and Barbara Buxton. 1987. An alternative to multiple choice vocabulary tests. Language testing, 4(2):142-154.
Sergio R Munoz and Shrikant I Bangdiwala. 1997. Interpretation of kappa and b statistics measures of agreement. Journal of Applied Statistics, 24(1):105-112.
I. S. Paul Nation. 2006. How Large a Vocabulary Is Needed For Reading and Listening? The Canadian Modern Language Review/La Revue canadienne des langues vivantes, 63(1):59-82.
Hieu T Nguyen and Arnold Smeulders. 2004. Active learning using pre-clustering. In Proceedings of the twenty-first international conference on Machine learning, page 79.
Brian North and Enrica Piccardo. 2020. Compan- ion volume COMMON EUROPEAN FRAMEWORK OF REFERENCE FOR LANGUAGES: LEARNING, TEACHING, ASSESSMENT COMMON EUROPEAN FRAMEWORK OF REFERENCE FOR LANGUAGES: LEARNING, TEACHING, ASSESSMENT Companion volume Language Policy Programme Education Policy Division Education Department Council of Europe.
Gustavo Paetzold and Lucia Specia. 2016a. Benchmarking lexical simplification systems. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3074-3080, Portoorž, Slovenia. ELRA.
Gustavo Paetzold and Lucia Specia. 2016b. SemEval 2016 Task 11: Complex Word Identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 560-569, San Diego, CA, USA. Association for Computational Linguistics.
Gustavo Paetzold and Lucia Specia. 2016c. Sv000gg at semeval-2016 task 11: Heavy gauge complex word identification with system voting. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 969-974, San Diego, CA, USA. Association for Computational Linguistics.
David Qian. 1999. Assessing the roles of depth and breadth of vocabulary knowledge in reading comprehension. Canadian modern language review, 56(2):282-308.
David D Qian. 2002. Investigating the relationship between vocabulary knowledge and academic reading performance: An assessment perspective. Language learning, 52(3):513-536.
F. E. Satterthwaite. 1946. An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6):110-114.
Burr Settles. 2009. Active learning literature survey.
Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1070-1079.
Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423.
Matthew Shardlow. 2013a. A comparison of techniques to automatically identify complex words. In Proceedings of the Student Research Workshop at the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 103-109, Sofia, Bulgaria. Association for Computational Linguistics.
Matthew Shardlow. 2013b. The cw corpus: A new resource for evaluating the identification of complex words. In Proceedings of the 2nd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 69-77, Sofia, Bulgaria. Association for Computational Linguistics (ACL).
Matthew Shardlow. 2014. Out in the open: Finding and categorising errors in the lexical simplification pipeline. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1583-1590, Reykjavik, Iceland. ELRA.
Matthew Shardlow, Michael Cooper, and Marcos Zampieri. 2020. CompLex — a new corpus for lexical complexity prediction from Likert Scale data. In Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIficulties (READI), Marseille, France. European Language Resources Association.
Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021. Semeval-2021 task 1: Lexical complexity prediction. arXiv preprint arXiv:2106.00473.
Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, pages 347-355. Association for Computational Linguistics.
Lars Stenius Staehr. 2008. Vocabulary size and the skills of listening, reading and writing. Language Learning Journal, 36(2):139-152.
David Strohmaier, Sian Gooding, Shiva Taslimipoor, and Ekaterina Kochmar. 2020. SeCoDa: Sense complexity dataset. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5962-5967, Marseille, France. European Language Resources Association.
Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163-173. Association for Computational Linguistics.
Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo H Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack, and Marcos Zampieri. 2018. A report on the complex word identification shared task 2018. arXiv preprint arXiv:1804.09132.
Seid Muhie Yimam, Sanja Štajner, Martin Riedl, and Chris Biemann. 2017. Multilingual and Cross-Lingual Complex Word Identification. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2017, pages 813-822, Varna, Bulgaria. INCOMA Ltd.
Ahmed Zaidi, Andrew Caines, Russell Moore, Paula Buttery, and Andrew Rice. 2020. Adaptive forgetting curves for spaced repetition language learning. arXiv preprint arXiv:2004.11327.
Jason D Zevin and Mark S Seidenberg. 2002. Age of acquisition effects in word reading and other tasks. Journal of Memory and language, 47(1):1-29.
A Participant demographics
The demographic information of participants is shown in Table 6. A more detailed breakdown including first language statistics is available on our data repository.
B Experiment interface
The experiment interface is shown in Figure 4. Annotators were paid 15 cents per minute in accordance with Prolific guidelines. Annotators were informed that the study and their subsequent annotations were for research purposes. Data is fully anonymised and both this and the models will be released under the Affero General Public License 3 (AGPL-3.0). All participant instructions as well as the demographic questionnaire are included in our GitHub data repository.
Proficiency (%)	Education (%)	Age range (%)	Hours reading English per week (%)
Native	8.29	Graduate	31.16	18 - 24	39.24	0 - 10	40.40
Near native	30.50	Undergraduate	29.83	25 - 34	22.17	10 - 20	22.83
Advanced	45.51	High School	33.37	35 - 44	14.52	20 - 30	14.95
Intermediate	15.14	Vocational Training	5.42	45 - 54	11.64	30 - 40	9.82
Beginner	0.66	No formal education	0.22	55+	9.53	40 +	11.98
Table 6: Background statistics for the 1225 participants
Below you will be shown 67 words.
Please click the 'yes' and 'no' buttons to answer the following question:
Can you confidently define this word?
Figure 4: Experiment interface