SlowGuess/NAACL_2021_OCR huggingface dataset:
For the non-native class, the second highest correlated feature is KFCAT which represents the number of categories of text in which the word was present as given in the norms of Kucera and Francis (1967). The negative correlation shows that the more categories of text a word appears in, the less likely it is to be considered complex. This measure can also be considered as the specificity of the word. For instance, we see that the word grounds is found across a wide range of text categories and is rarely considered complex. Whereas words like altimeter and aneroid, which are highly specific to a particular domain, are considered complex in all contexts by both native and non-native readers. The number of categories that a word occurs in is correlated with the word's frequency (0.35). However, when you control for the word frequency, the effect of this correlation is even higher:  $-0.40$  and  $-0.41$  for non-native and native respectively. Therefore, the narrower the scope of application for a word the more likely it will be considered difficult.
Finally, we see that psycholinguistic measures such as the word familiarity and imagability are highly correlated with both the native and non-native absolute number of annotations. When considering imagability, the larger the img score the higher the imagability, for instance 'dog' has a high img factor whereas 'decision' has a low score as it cannot be easily associated with an image. The negative correlation shows that the higher the score the less likely the word is considered complex. Intuitively, it makes sense why this feature would be influential in determining word complexity. In fact, research on children's reading has shown that words high in imagability are easier to read than
words low in imagability (Coltheart et al., 1988). It has been suggested that this occurs because low imagability words are acquired later in life than high imagability words. Finally, concreteness is one of the top five features correlated with the nonnative annotations. It has been found that the higher the concreteness of a word, the more likely it is to be comprehensible (Sadoski et al., 2000).
Feature Distributions
Word length and frequency have been widely used in CWI systems and are reported to be good cross-linguistic predictors of complexity (Bingel et al., 2018). Additionally, psycholinguistic properties are considered important in word complexity estimation (Carroll and White, 1973). When investigating the feature importance for our binary models in Section 5, we find that the features with the highest importance across models are word length, frequency and imagability. We investigate whether the distribution of the feature values is dependent on the intended audience.
Figure 5 contains two histograms presenting binned word lengths across proficiency classes. Words that have been marked as complex are grouped into 20 bins and the distribution of lengths plotted. We observe that beginners mark more shorter words as complex than either the intermediate or advanced class do. Generally, the distribution of lengths shifts to the right as proficiency increases. This same pattern is observed for the native and non-native classes, where non-native annotators are more likely to mark shorter words as complex than native.
Figure 6 contains histograms presenting the binned frequencies for complex words (20 bins). For frequencies, we observe a clear difference between the beginner and intermediate/advanced classes. The beginner sub group has marked many more low frequency words as complex. For the advanced class, the range (difference in largest and smallest frequency value) is 259 whereas for beginners the range is 569. Furthermore, the mean frequency values show that the advanced and intermediate classes, on average, are more likely to consider words with lower frequencies to be complex (15.09 and 16.22) whereas for beginners the mean is higher (22.63). As the advanced and intermediate classes have a narrower spread and lower mean, it is likely frequency based thresholding techniques would work well for these groups.
(a)
(b)
Figure 5: Word length histograms with 20 bins
(a)
(b)
Figure 6: Frequency histograms with 20 bins
(a)
(b)
Figure 7: Imagability histograms with 3 bins
When we consider the native and non-native frequency distributions, we notice the same pattern emerging between classes. The non-native class has many more low frequency words annotated as complex and the relationship between native and non-native closely resembles the one between advanced and beginner. Word frequency provides signal on the likelihood of an individual being exposed to the word. However, the actual likelihood of exposure will depend on whether an individual is a native or non-native speaker as well as their experience of the language.
Finally, in Figure 7 we group imagability ratings into 3 bins representing high, medium and low scores. We see that for the advanced and intermediate classes most complex annotations fall in the middle range. However, for the beginner class there are still many high imagability words that are deemed as complex. It is worth noting, that the coverage of imagability is limited and therefore results should be considered more cautiously. Regarding the native and non-native imagability, we again see that the non-native class has slightly more higher imagability words marked as complex.
To conclude, the relative relationships between beginner and advanced feature distributions very closely mirror the relationship between native and non-native. There is a clear trend across features based on the proficiency and experience the reader. Furthermore, the feature profiles of advanced non-native speakers are more similar to that of a native
speaker. As far as we are aware, this is the first work exploring how the thresholds of features vary across audiences for complexity. Investigating this is insightful, as there are numerous threshold based approaches to CWI (Zeng et al., 2005; Elhadad, 2006; Biran et al., 2011), therefore understanding how these thresholds differ for audiences can produce more informed techniques.
Textual complexity is a subjective phenomenon that is dependent on the intended audience. We show that when considering lexical complexity, the best performing CWI models for a target proficiency level are trained with the labels of that sub-group. We investigate which features correlate most with the absolute number of native and non-native annotations as well as observe how the distributions of classic complexity features are dependent on the intended audience. We find strong similarities between the notion of word complexity for advanced non-native readers and native readers. Finally, we release a dataset for CWI with proficiency subgroup annotations. In future work we plan to collect additional annotations across classes, especially concentrating on beginners. We would also like to investigate how effective informed-thresholding techniques for CWI are compared to high resource systems.
This work has been done while the second author was a Senior Research Associate at the University of Cambridge. We thank Cambridge English for supporting this research via the ALTA Institute. We are also grateful to the anonymous reviewers for their valuable feedback.
Ian Begg and Allan Paivio. 1969. Concreteness and imagery in sentence meaning. Journal of Verbal Learning and Verbal Behavior, 8(6):821-827.
Joachim Bingel. 2018. Personalized and Adaptive Text Simplification. Ph.D. thesis, University of Copenhagen.
Joachim Bingel and Johannes Bjerva. 2018. Crosslingual complex word identification with multitask learning. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pages 166-174, New Orleans, LA, USA. Association for Computational Linguistics.
Joachim Bingel, Gustavo Paetzold, and Anders Søgaard. 2018. Lexi: A tool for adaptive, personalized text simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 245-258, Santa Fe, NM, USA. Association for Computational Linguistics.
Or Biran, Samuel Brody, and Noemie Elhadad. 2011. Putting it Simply: a Context-Aware Approach to Lexical Simplification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 496-501, Portland, OR, USA. Association for Computational Linguistics.
Stefan Bott, Luz Rello, Biljana Drndarevic, and Horacio Saggion. 2012. Can Spanish Be Simpler? Lex-SiS: Lexical Simplification for Spanish. In Proceedings of COLING 2012: Technical Papers, pages 357-374, Mumbai, India. COLING.
John Carroll, Guido Minnen, Yvonne Canning, Siobhan Devlin, and John Tait. 1998. Practical simplification of english newspaper text to assist aphasic readers. In Proceedings of the AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology, pages 7-10, Madison, WI, USA.
John B Carroll and Margaret N White. 1973. Word frequency and age of acquisition as determiners of picture-naming latency. The Quarterly Journal of Experimental Psychology, 25(1):85-95.
Veronika Coltheart, Veronica J Laxon, and Corriene Keating. 1988. Effects of word imageability and age of acquisition on children's reading. *British journal of psychology*, 79(1):1-12.
Cynthia M Connine, John Mullennix, Eve Shernoff, and Jennifer Yelen. 1990. Word familiarity and frequency in visual and auditory word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(6):1084.
William Coster and David Kauchak. 2011. Simple English Wikipedia: a new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 665-669, Portland, OR, USA. Association for Computational Linguini.
Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin, pages 37-54.
Jan De Belder and Marie-Francine Moens. 2010. Text simplification for children. In *Prceedings of the SIGIR workshop on accessible search systems*, pages 19-26, Geneva, Switzerland. ACM; New York.
Siobhan Devlin and John Tait. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic Databases, pages 161-173.
William H Dubay. 2004. The Principles of Readability. Costa Mesa, CA: Impact Information.
Carsten Elbro. 1996. Early linguistic abilities and reading development: A review and a hypothesis. Reading and Writing, 8(6):453-485.
Noemie Elhadad. 2006. Comprehending Technical Texts: Predicting and Defining Unfamiliar Terms. In AMIA Annual Symposium Proceedings, pages 239-243, Washington, DC, USA.
Liv Engen and Torleiv Hoien. 2002. Phonological skills and reading comprehension. Reading and Writing, 15(7-8):613-631.
Richard Evans, Constantin Orăsan, and Iustin Dornescu. 2014. An evaluation of syntactic simplification rules for people with autism. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), pages 131-140, Gothenburg, Sweden.
Christiane Fellbaum. 2005. Encyclopedia of Language and Linguistics, Second Edition, chapter WordNet and wordnets. Oxford: Elsevier.
Ken J Gilhooly and Robert H Logie. 1980. Age-of-acquisition, imagery, concreteness, familiarity, and ambiguity measures for 1,944 words. Behavior research methods & instrumentation, 12(4):395-427.
Yoav Goldberg and Jon Orwant. 2013. A dataset of syntactic-ngrams over time from a very large corpus of English books. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, volume 1,
pages 241-247, Atlanta, GA, USA. Association for Computational Linguistics.
Sian Gooding and Ekaterina Kochmar. 2018. CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 184-194, New Orleans, LA, USA. Association for Computational Linguistics.
Sian Gooding and Ekaterina Kochmar. 2019a. Complex Word Identification as a Sequence Labelling Task. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 1148-1153, Florence, Italy. Association for Computational Linguistics.
Sian Gooding and Ekaterina Kochmar. 2019b. Recursive context-aware lexical simplification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4855-4865, Hong Kong, China. Association for Computational Linguistics.
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch.
Henry Kučera and Winthrop Nelson Francis. 1967. Computational analysis of present-day American English. Dartmouth Publishing Group.
John Lee and Chak Yan Yeung. 2018. Personalizing lexical simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 224-232.
Mounica Maddela and Wei Xu. 2018. A word-complexity lexicon and a neural readability ranking model for lexical simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3749-3760, Brussels, Belgium. Association for Computational Linguistics.
Palmer Morrel-Samuels and Robert M Krauss. 1992. Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(3):615.
I. S. Paul Nation. 2006. How Large a Vocabulary Is Needed For Reading and Listening? The Canadian Modern Language Review/La Revue canadienne des langues vivantes, 63(1):59-82.
Charles Kay Ogden. 1968. *Basic English: international second language*. Harcourt, Brace & World.
Gustavo Paetzold and Lucia Specia. 2016a. Benchmarking lexical simplification systems. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3074-3080, Portoorž, Slovenia. ELRA.
Gustavo Paetzold and Lucia Specia. 2016b. Collecting and exploring everyday language for predicting psycholinguistic properties of words. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1669-1679, Osaka, Japan.
Gustavo Paetzold and Lucia Specia. 2016c. SemEval 2016 Task 11: Complex Word Identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 560-569, San Diego, CA, USA. Association for Computational Linguistics.
Andrew Porter, Jennifer McMaken, Jun Hwang, and Rui Yang. 2011. Common Core Standards: The New U.S. Intended Curriculum. Educational Researcher, 40:103-116.
Luz Rello, Ricardo Baeza-Yates, Laura Dempere-Marco, and Horacio Saggion. 2013. Frequent words improve readability and short words improve understandability for people with dyslexia. In IFIP Conference on Human-Computer Interaction, pages 203-219, Cape Town, South Africa.
Alan P Rudell. 1993. Frequency of word usage and perceived word difficulty: Ratings of kucera and francis words. Behavior Research Methods, Instruments, & Computers, 25(4):455-463.
Mark Sadoski, Ernest T Goetz, and Maximo Rodriguez. 2000. Engaging texts: Effects of concreteness on comprehensibility, interest, and recall in four text types. Journal of Educational Psychology, 92(1):85.
Carolina Scarton and Lucia Specia. 2018. Learning simplifications for specific target audiences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 712-718, Melbourne, Australia. Association for Computational Linguistics.
Matthew Shardlow. 2013. A comparison of techniques to automatically identify complex words. In Proceedings of the Student Research Workshop at the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 103-109, Sofia, Bulgaria. Association for Computational Linguistics.
Matthew Shardlow. 2014. Out in the open: Finding and categorising errors in the lexical simplification pipeline. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1583-1590, Reykjavik, Iceland. ELRA.
Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In *SEM 2012: The First Joint Conference
on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 347-355, Montreal, Canada. Association for Computational Linguistics.
Edward L Thorndike and Irving Lorge. 1944. The teacher's wordbook of 30,000 words. New York: Columbia University, Teachers College.
Michael Wilson. 1988. The MRC Psycholinguistic Database: Machine Readable Dictionary, Version 2. Behavioural Research Methods, Instruments and Computers, pages 6-11.
Seid Muhie Yimam and Chris Biemann. 2018a. Demonstrating Par4Sem - A Semantic Writing Aid with Adaptive Paraphrasing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 48-53, Brussels, Belgium. Association for Computational Linguistics.
Seid Muhie Yimam and Chris Biemann. 2018b. Par4Sim-Adaptive Paraphrasing for Text Simplification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 331-342, Santa Fe, NM, USA. COLING.
Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Šažner, Anaïs Tack, and Marcos Zampieri. 2018. A Report on the Complex Word Identification Shared Task 2018. In Proceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, pages 66-78, New Orleans, LA, USA. Association for Computational Linguistics.
Seid Muhie Yimam, Sanja Štajner, Martin Riedl, and Chris Biemann. 2017a. CWIG3G2 - Complex Word Identification Task across Three Text Genres and Two User Groups. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages 401-407, Taipei, Taiwan. Association for Computational Linguistics.
Seid Muhie Yimam, Sanja Šajner, Martin Riedl, and Chris Biemann. 2017b. Multilingual and Cross-Lingual Complex Word Identification. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2017, pages 813-822, Varna, Bulgaria. INCOMA Ltd.
Ahmed Zaidi, Andrew Caines, Russell Moore, Paula Buttery, and Andrew Rice. 2020. Adaptive forgetting curves for spaced repetition language learning. arXiv preprint arXiv:2004.11327.
Qing Zeng, Eunjung Kim, Jon Crowell, and Tony Tse. 2005. A Text Corpora-Based Estimation of the Familiarity of Health Terminology. In Biological and Medical Data Analysis. ISBMDA 2005. Lecture Notes in Computer Science Vol. 3745, ISBMDA 2005, Aveiro, Portugal. Springer.
Jason D Zevin and Mark S Seidenberg. 2002. Age of acquisition effects in word reading and other tasks. Journal of Memory and language, 47(1):1-29.