Venue: NAACL 2019

# Strong Baselines for Complex Word Identification

across Multiple Languages

Pierre Finnimore\({}^{1}\), Elisabeth Fritzsch\({}^{1}\), Daniel King\({}^{1}\), Alison Sneyd\({}^{1}\),

**Aneeq Ur Rehman\({}^{1}\)**, Fernando Alva-Manchego\({}^{1}\) and Andreas Vlachos\({}^{2}\)

\({}^{1}\)Department of Computer Science, University of Sheffield

\({}^{2}\)Department of Computer Science and Technology, University of Cambridge

{pmfinimore,fritzsch.elisabeth,danielking1903}@gmail.com, a.sneyd@shef.ac.uk,a.neeq8394@gmail.com, f.alva@shef.ac.uk,andreas.vlachos@cst.cam.ac.uk

###### Abstract

Complex Word Identification (CWI) is the task of identifying which words or phrases in a sentence are difficult to understand by a target audience. The latest CWI Shared Task released data for two settings: monolingual (i.e. train and test in the same language) and cross-lingual (i.e. test in a language not seen during training). The best monolingual models relied on language-dependent features, which do not generalise in the cross-lingual setting, while the best cross-lingual model used neural networks with multi-task learning. In this paper, we present monolingual and cross-lingual CWI models that perform as well as (or better than) most models submitted to the latest CWI Shared Task. We show that carefully selected features and simple learning models can achieve state-of-the-art performance, and result in strong baselines for future development in this area. Finally, we discuss how inconsistencies in the annotation of the data can explain some of the results obtained.

## 1 Introduction

Complex Word Identification (CWI) consists of deciding which words (or phrases) in a text could be difficult to understand by a specific type of reader. In this work, we follow the CWI Shared Tasks Paetzold and Specia (2016); Yimam et al. (2018) and assume that a target word or multi-word expression (MWE1) in a sentence is given, and our goal is to determine if it is complex or not (an example is shown in Table 1). Under this setting, CWI is normally treated using supervised learning and feature engineering to build monolingual models Paetzold and Specia (2016); Yimam et al. (2018). Unfortunately, this approach is infeasible for languages with scarce resources of annotated data. In this paper, we are interested in both monolingual and cross-lingual CWI; in the latter, we build models to make predictions for languages not seen during training.

Footnote 1: We consider n-grams with \(n\geq 2\) as MWEs, while Yimam et al. (2018) used \(n\geq 3\).

While monolingual CWI has been studied extensively (see a survey in Paetzold and Specia (2017)), the cross-lingual setup of the task was introduced only recently by Yimam et al. (2017b), who collected human annotations from native and non-native speakers of Spanish and German, and integrated them with similar data previously produced for three English domains Yimam et al. (2017a): News, WikiNews and Wikipedia.

For the Second CWI Shared Task Yimam et al. (2018), participants built monolingual models using the datasets previously described, and also tested their cross-lingual capabilities on newly collected French data. In the monolingual track, the best systems for English Gooding and Kochmar (2018) differed significantly in terms of feature set size and the model's complexity, to the best systems for German and Spanish Kajiwara and Komachi (2018). The latter used Random Forests with eight features, whilst the former used AdaBoost with 5000 estimators or ensemble voting combining AdaBoost and Random Forest classifiers, with about 20 features.

In the cross-lingual track, only two teams achieved better scores than the baseline: Kaijiwara and Komachi (2018) who used length and frequency based features with Random Forests, and

\begin{table}
\begin{tabular}{l l c} \hline \hline Sentence & Target word/MWE & Complex? \\ _Both China and the Philippines flexed their muscles on Wednesday._ & fixed & Yes \\ \hline \hline \end{tabular}
\end{table}
Table 1: An annotated sentence in the English dataset of the Second CWI Shared Task.



Bingel and Bjerva (2018) who implemented an ensemble of Random Forests and feed-forward neural networks in a multi-task learning architecture.

Our approach to CWI differs from previous work in that we begin by building competitive monolingual models, but using the same set of features and learning algorithm across languages. This reduces the possibility of getting high scores due to modelling annotation artifacts present in the dataset of one language. Our monolingual models achieve better scores for Spanish and German than the best systems in the Second CWI Shared Task. After that, we focus on language-independent features, and keep those that achieve good performance in cross-lingual experiments across all possible combinations of languages. This results in a small set of five language-independent features, which achieve a score as high as the top models in the French test set. Finally, we analyse the annotation of the datasets and find some inconsistencies that could explain some of our results.

Code for all our models can be found at: [https://github.com/sheffieldnlp/cwi](https://github.com/sheffieldnlp/cwi)

## 2 Problem Formulation

We tackle the binary classification task in the Second CWI Shared Task Yimam et al. (2018), in which a model decides if a target word/MWE in a sentence is complex or not. Following common practice, we extract features from the target word/MWE and its context, and then use a supervised learning algorithm to train a classifier. For training and testing our models, we use the annotated datasets provided for the Second CWI Shared Task (see Table 2 for some statistics).

## 3 Monolingual Models

### Features Description

Our feature set consists of 25 features that can be extracted for all languages considered (English, German, Spanish and French). They can be divided into three broad categories: features based on the target word/MWE, sub-word level features, and sentence-level features to capture information from the target's context. As we intended that our features be applicable across languages, we drew on features found to be useful in previous work on CWI Yimam et al. (2017b, 2018). We made use of the python libraries spaCy2 Honnibal and Montani (2017) and NLTK3 Loper and Bird (2002). Details on the resources used for extracting each feature can be found in Appendix A.

Footnote 2: [https://spacy.io/](https://spacy.io/)

Footnote 3: [https://www.nltk.org/](https://www.nltk.org/)

At the **target word/MWE level**, we experimented with features such as Named Entity (NE) type, part-of-speech, hypernym counts, number of tokens in the target, language-normalised number of characters in each word, and simple unigram probabilities. These features are linguistically motivated. The perceived complexity of a MWE may be higher than that of a single word, as each component word can be complex, or simple component words can be synthesised into a complex whole. Similarly, infrequent words are less familiar, so we would expect low-probability target words to be found more complex. Along these lines, proper nouns could be more complex, as there is a vast number of NEs, and the chance that a person has encountered any one of them is low. We would expect this trend to reverse for the NE type of organisations, in combination with the Enlighi-News dataset, as organisations mentioned in news articles are frequently global, and so the chance that a person has encountered a proper noun _that is an organisation_ is often higher than for proper nouns in general. In total, 14 features were used at the target word/MWE level.

Our **sub-word level** features include prefixes, suffixes, the number of syllables, and the number of complex punctuation marks (i.e. punctuation within the target word/MWE, such as hyphens, that could denote added complexity). We would expect certain affixes to be useful features, as language users use sub-word particles like these to identify unknown words: by breaking up a word like "granted" into "grant-" and "-ed", readers can fall back on their knowledge of these component pieces to clarify the whole. A total of 9 sub-word features were used in the monolingual models.

Finally, **sentence level** features with linguistic

\begin{table}
\begin{tabular}{l r r r} \hline \hline Dataset & Train & Dev & Test \\ \hline English (EN) - News & 14,002 & 1,764 & 2,095 \\ English (EN) - WikiNews & 7,746 & 870 & 1,287 \\ English (EN) - Wikipedia & 5,551 & 694 & 870 \\ Spanish (ES) & 13,750 & 1,622 & 2,232 \\ German (DE) & 6,151 & 795 & 959 \\ French (FR) & N/A & N/A & 2,251 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Number of annotated samples in each dataset for each language.

 

motivations were also considered. Long sentences could be harder to understand, which makes it more difficult to figure out the meaning of unknown words contained within them. Also, long sentences are more likely to include more unwron words or ambiguous references. Therefore, we considered sentence length (i.e., number of tokens in the sentence) as a feature. In addition, we extracted N-grams (unigrams, bigrams and trigrams) from the whole sentence, since certain sentence constructions can help a reader understand the target word/MWE. For example, "A of the B" suggests a relation between A and B. We used 2 sentence-level features in total.

### Experiments and Results

Following Yimam et al. (2018), we used MacroF1 score to evaluate performance and for comparison with previous work on the datasets. We used Logistic Regression for all our experiments, as it allowed for easy exploration of feature combinations, and in initial experiments we found that it performed better than Random Forests. We evaluated both using the full feature set described before, as well as a two-feature baseline using the number of tokens of the target and its language-normalised number of characters. Results of our monolingual experiments are shown in Table 3.

In the test set, our baseline results (BL in Table 3) are strong, especially in German. Our full 25-features model improves on the baseline in all cases, with the biggest increase of over 16 percentage points seen for the EN-News dataset. Our system beats the best performing system from the Shared Task in Spanish (77.0) and German (74.5), both obtained by Kajiwara and Komachi (2018). However, the state of the art for German remains the Shared Task baseline (75.5) (Yimam et al., 2018). The best results for all three English datasets were obtained by Gooding and Kochmar (2018); ours is within two percentage points of their News dataset score. Furthermore, the mean score for our system (79.2) is close to the mean of the best performing models (81.0), which are different systems, while using simpler features and learning algorithm. The best-performing model in English, for example, used Adaboost with 5000 estimators (Gooding and Kochmar, 2018).

## 4 Cross-lingual Models

### Features Description

Linguistically, the cross-lingual approach can be motivated by the relation between certain languages (such as French and Spanish both being Romance languages). In addition, there may be features identifying complex words that are shared even across language families.

To be able to test a model on a language that was unseen during training, the features the model works with must be cross-lingual (or language-independent) themselves. For example, the words themselves are unlikely to transfer across languages (apart from those that happen to be spelled identically), but the popularity of the words would transfer. This rules out some of the features we used for the monolingual approach (see Sec. 3.1), as they were language-dependent. One such feature is N-grams for the target word/MWE, which depend on the language, and so will only occur with extreme sparsity outside of their source language. For example, if applying a system trained on English to unseen French, the English phrases "a la mode" or "film noire" might reoccur in the French, since they originate from that language, but these are rare exceptions. What is more, a French loan-phrase may have different complexity characteristics to the same N-grams occurring in their native language. Therefore, we did not use these features in the cross-lingual system.

### Experiments and Results

To find out which features were best suited for the cross-lingual approach, we performed an iterative ablation analysis (see Appendix B for details). Using this process, we arrived at our final cross-lingual feature set: number of syllables in the target, number of tokens in the target, number of complex punctuation marks (such as hyphens), sentence length, and unigram probabilities.

Furthermore, we analyse the effect of different

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Dataset & \multicolumn{2}{c}{Dev} & \multicolumn{3}{c}{Test} \\ \cline{2-6}  & BL & MA & BL & MA & SotA \\ \hline EN - News & 83.6 & 85.5 & 69.7 & 86.0 & **87.4** \\ EN - WikiNews & 80.4 & 82.8 & 65.8 & 81.6 & **84.0** \\ EN - Wikipedia & 74.2 & 76.6 & 70.1 & 76.1 & **81.2** \\ ES & 78.0 & 77.1 & 69.6 & **77.6** & 77.0 \\ DE & 79.5 & 74.6 & 72.4 & 74.8 & **75.5** \\ \hline Mean & 79.1 & 79.3 & 69.5 & 79.2 & N/A \\ \hline \hline \end{tabular}
\end{table}
Table 3: Macro-F1 for the baseline (BL), our monolingual approach (MA), and the state of the art (SotA) on the Dev and Test splits of each dataset.

 

language combinations on the performance of the cross-lingual model in order to investigate how the relationship between the languages trained and tested on would influence model performance. Recall that we only have training data for English, Spanish and German, but not French. We train models using all possible combinations (each language independently, each pairing, and all three) and evaluate on each of the four languages that have test data (i.e. the former three and French), excluding training combinations that include the test language. Results are shown in Table 4.

When testing on French, we achieved the highest performance by training on German only (75.8), followed closely by training on a combination of German and Spanish (75.7) and only Spanish (75.5). The worst performance was achieved by training only on English (69.2), and the performance also noticeably decreased for all training combinations that included English.

When testing on German, language choice had a weaker effect. The highest score came from combining English and Spanish (73.4), but using only one of those languages gave comparable results (72.6 for Spanish, 73.0 for English).

For Spanish, the best results were achieved when training only on German (72.6). Adding English to the training languages decreased the performance (70.8), which was even lower when training only on English (69.1).

It is noteworthy that adding English to the training languages noticeably decreases performance for both Spanish and French, but not for German. One possible reason for Spanish and French not benefiting from English when German does is that both English and German are Germanic languages, whereas Spanish and French are Romance languages. Another possible explanation for the decrease of performance caused by training with English is that there are inconsistencies in the way MWEs in the datasets were labelled across languages, which we explore in Sec. 5.

We finally compare our cross-lingual models against the state of the art: the best monolingual system for Spanish and German, and the best cross-lingual system for French, where no monolingual systems exist. As Table 5 shows, our cross-lingual models come close to the best monolingual models for Spanish and especially for German. This is remarkable given how simple our model and features are, and that the approaches we compare against train complex models for each language. Furthermore, this points towards the possibility of extending CWI to more languages which lack training data.

Finally, Table 6 compares the coefficients for models trained on Romance and Germanic languages. Notably, use of complex punctuation (such as the hyphenation in "laser-activated" or "drug-related") and the number of tokens are inversely correlated w.r.t. the word or MWE being complex. More words in the target was correlated with complexity for English and German, and inversely correlated for Spanish.

## 5 Dataset Analysis

While examining our models' incorrect predictions, we observed inconsistencies in labelling in the datasets between target MWEs and their subwords/sub-expressions (SWs).

\begin{table}
\begin{tabular}{l c r l l l l} \hline \hline EN & ES & DE & Eval & Source & Test & Dev \\ \hline  & ✓ & ✓ & EN & WikiNews & 61.8 & 63.7 \\  & & ✓ & EN & WikiNews & 62.3 & 63.6 \\  & ✓ & ✓ & EN & WikiNews & 61.6 & 63.8 \\  & ✓ & ✓ & EN & Wikipedia & 62.8 & 64.4 \\  & ✓ & ✓ & EN & Wikipedia & 62.6 & 64.4 \\  & ✓ & & EN & Wikipedia & 63.1 & 65.2 \\  & ✓ & ✓ & EN & News & 67.1 & 65.6 \\  & ✓ & ✓ & EN & News & 67.0 & 65.6 \\  & ✓ & & EN & News & **67.2** & **65.9** \\ \hline ✓ & & ✓ & ES & N/A & 70.8 & 71.3 \\  & & ✓ & ES & N/A & **72.6** & **74.1** \\ ✓ & & & ES & N/A & 69.1 & 70.0 \\ \hline ✓ & ✓ & & DE & N/A & **73.4** & **78.3** \\  & ✓ & & DE & N/A & 72.6 & 77.4 \\ ✓ & & & DE & N/A & 73.0 & 76.0 \\ \hline ✓ & ✓ & ✓ & FR & N/A & 73.1 & N/A \\  & ✓ & ✓ & FR & N/A & 75.7 & N/A \\ ✓ & ✓ & & FR & N/A & 73.4 & N/A \\  & & ✓ & FR & N/A & 70.5 & N/A \\  & ✓ & & FR & N/A & **75.8** & N/A \\  & ✓ & & FR & N/A & 73.4 & N/A \\  & & & ER & N/A & 69.2 & N/A \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of Test and Dev results for all permutations of training languages.

\begin{table}
\begin{tabular}{l r r r} \hline \hline  & Spanish & German & French \\ \hline Monolingual SotA & **77.0** & **75.5** & N/A \\ Cross-lingual SotA & N/A & N/A & **76.0** \\ Our cross-lingual & 72.6 & 73.4 & 75.8 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison between the monolingual and cross-lingual state of the art (SotA), and our cross-lingual system.

 

The First CWI Shared Task (Paetzold and Specia, 2016) used the annotations of a group (i.e. ten annotators on the training data) to predict the annotation of an individual (i.e. one annotator on the test data). The resulting inconsistencies in labelling may have contributed to the low F-scores of systems in the task (Zampieri et al., 2017). Although the Second CWI Shared Task improved on the first by having multiple annotators for all splits of the data, it contains some labelling inconsistencies arising from annotators now being able to label phrases, and words within them, separately.

More concretely, we found that across all datasets, 72% of MWEs contain at least one SW with the opposite label (see Table 7). While this makes sense in some cases, every SW in 25% of MWE instances has the opposite label. For example, "numerous falsifications and ballot stuffing" is not annotated as complex, despite its SWs "numerous", "numerous falsifications", "falsifications", "ballot", "ballot stuffing" and "stuffing" all being complex. Conversely, "crise des marches du credit" is complex, despite "crise", "marches" and "credit" being labelled non-complex. It is difficult to see how classifiers that extract features for MWEs from their individual SWs could predict the labels of both correctly.

Furthermore, every target MWE in the Spanish, German and French datasets is labelled complex. This may bias a classifier trained on the Spanish or German datasets towards learning MWEs and long individual words (if length is a feature) are complex. In particular, this observation may help explain why adding English as a training language decreased the performance of our cross-lingual system when testing on French and Spanish (where all MWEs are complex). An analysis in Bingel and Bjerva (2018) further found that their cross-lingual French model was effective at predicting long complex words/MWEs but had difficulty predicting long non-complex words.

It is also worth noting that considering a word or MWE as complex is subjective and may differ from person to person, even within the same target audience. Bingel et al. (2018) investigated predicting complex words based on the gaze patterns of children with reading difficulties. They found a high degree of specificity in misreadings between children, that is, which words they found complex when reading aloud. This variety of complexity judgements even within one target group points towards the high degree of subjectivity in the task, which may also partly explain the inconsistencies in the dataset.

## 6 Conclusion and Future Work

The monolingual and cross-lingual models presented achieve comparable results against more complex, language-specific state-of-the-art models, and thus can serve as strong baselines for future research in CWI. In addition, our analysis of the dataset could help in the design of better guidelines when crowdsourcing annotations for the task. Dataset creators may wish to only allow single words to be chosen as complex to avoid labelling inconsistencies. In case MWEs are being permitted, we suggest instructing annotators to chose the smallest part of a phrase they find complex (French annotators for the Second CWI Shared Task sometimes grouped individual complex words into a complex MWE (Yimam et al., 2018)).

### Acknowledgements

This work was initiated in a class project for the NLP module at the University of Sheffield. The authors would like to acknowledge the contributions of Thomas Dakin, Sanjana Khot and Harry Wells who contributed their project code to this work. Andreas Vlachos is supported by the EPSRC grant eNeMILP (EP/R021643/1).

\begin{table}
\begin{tabular}{l r r} \hline \hline Feature & Train & Coefficient \\ \hline number of complex punctuation marks & EN & -0.693 \\  & DE & -0.559 \\  & ES & 1.111 \\ \hline number of tokens & EN & -2.200 \\  & DE & -0.534 \\  & ES & 1.420 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Coefficients for cross-lingual models trained on Germanic and Romance languages.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & C & NC & \(\geq\) 1 Irreg. & All Irreg. \\ \hline English & 3,750 & 982 & 3,315 & 950 \\ Spanish & 2,309 & 0 & 1,747 & 760 \\ German & 502 & 0 & 374 & 178 \\ French & 242 & 0 & 192 & 82 \\ \hline Total & 6,803 & 982 & 5,628 & 1,970 \\ \hline \hline \end{tabular}
\end{table}
Table 7: MWE annotation analysis: numbers of MWEs labelled complex (C) and non-complex (NC), numbers with at least one SW (\(\geq 1\) Irreg) and all SWs (All Irreg.) having the opposite label.

 



## References

* [1]M. Honnibal and I. Montani (2017)SpaCy2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. Cited by: SS3.4.
* [2]E. Loper and S. Bird (2002-06) Nltk: the natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 63-70. External Links: Link, Document Cited by: SS5.
* [3]G. Paetzold and L. Specia (2016-09) SemEval 2016 task 11: complex word identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 560-569. Cited by: SS1, SS2.
* [4]S. Muhie Yimam, C. Biemann, S. Malmasi, G. Puetzold, L. Specia, S. Stajner, A. Tack, and M. Zampieri (2018-06) A report on the complex word identification shared task 2018. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 66-78. Cited by: SS1, SS2.
* [5]J. Bingel and J. Bjerva (2018) Crosslingual complex word identification with multitask learning. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 166-174. Cited by: SS1, SS2.
* [6]S. Gooding and E. Kochmar (2018-06) Camb at cwi shared task 2018: complex word identification with ensemble-based voting. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 184-194. Cited by: SS1, SS2.
* [7]M. Honnibal and I. Montani (2017) SpaCy2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. Cited by: SS1, SS2.
* [8]T. Kajiwara and M. Komachi (2018-06) Complex word identification based on frequency in a learner corpus. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 195-199. Cited by: SS1, SS2.
* [9]R. Lee, D. R. Lee, and P. B. Lee (2018-06) The role of a new era of deep learning. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [10]A. Lin, X. Li, Z. Wang, W. Liu, Q. Zhang, and F. Yang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. arXiv preprint arXiv:1802.01880. Cited by: SS1, SS2.
* [11]D. Lin, X. Li, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [12]C. Lin, X. Li, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [13]P. Lin, X. Li, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [14]K. Lin, Z. Wang, W. Liu, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [15]Y. Luo, Z. Wang, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [16]X. Luo, Z. Wang, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [17]H. Lu, Z. Wang, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.
* [18]L. Meng, J. Sun, and J. Shi (2018-06) Learning phrase representations from context-aware text generation. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-122. Cited by: SS1, SS2.
* [19]W. Zhao, Z. Wang, and Z. Wang (2018-06) Deep learning: a large-scale hierarchical approach to boosting neural networks. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 115-120. Cited by: SS1, SS2.





## Appendix A Detailed Feature Set

\begin{table}
\begin{tabular}{l l p{142.3pt} p{85.4pt}} Level & Name & Description & Resource \\  & NER\_tag\_counts & Counts of each Named Entity tag in target & spaCy \\  & pos\_tag\_counts & Counts of each part-of-speech tag in target & spaCy \\  & hypernym\_count & Number of hypernyms & WordNet (NLTK) \\  & len\_tokens & Absolute length in tokens & N/A \\  & len\_tokens\_norm & Normalised length in tokens & N/A \\  & len\_chars\_norm & Normalised length in characters & N/A \\  & unigram\_prob & Log of the product of unigram probabilities & EN: Brown Corpus (NLTK) \\  & &bilities & ES: CESS-ESP (NLTK) \\  & & DE: TIGER Corpus4  \\ Target word/MWE & bag\_of\_shapes & Bag of morphological shapes & spaCy \\  & rare\_word\_count & Count of rare words in target & EN: subset of Google’s Trillion Word Corpus6 \\  & & DE: list of the most common 3,000 words7  \\  & rare\_trigram\_count & Count of rare trigrams in target & Same as rare\_word\_count \\  & is\_stop & Frequency of stopwords in target & NLTK, Ranks NL9  \\  & is\_nonphrase & If target is a noun phrase & spaCy \\  & avg\_char s\_per\_word & Avg. word length (in characters) of the target & N/A \\  & iob\_tags & Count of BIO tags in target & spaCy \\ \hline  & lemma\_feats & Bag of lemmas for target sentence & spaCy \\  & len\_sysls & Length of target in syllables & Pyphen10  \\  & num\_complex\_punct & Count of complex punctuation in target & N/A \\  & char\_n\_gram\_feats & Character N-Grams, incl. prefixes and suffixes & N/A \\  & char\_tri\_sum & Sum of character trigrams’ corpus frequencies & EN: Brown Corpus (NLTK) \\ Sub-word & & quencies & ES: CESS-ESP (NLTK) \\  & char\_tri\_avg & Average of character trigrams’ corpus frequencies & DE: TIGER Corpus \\  & consonant\_freq & Count of consonants in target & same as char\_tri\_sum \\  & gr\_or\_lat & If target has Greek or Latin affixes & N/A \\  & is\_capitalised & If target's first letter is uppercased & N/A \\ \hline Sentence & sent\_length & Number of tokens in the sentence & N/A \\  & sent\_n\_gram\_feats & Unigrams, bigrams and trigrams in the sentence & N/A \\ \end{tabular}
\end{table}
Table 8: Monolingual and Cross-lingual Feature Set Summary



## Appendix B Cross-lingual Features Ablation

\begin{table}
\begin{tabular}{l l c p{142.3pt}} \hline \hline Iteration & Current features & Features increasing performance & Features decreasing performance \\ \hline  & len\_tokens & num\_complex\_punct len\_sylls sent\_length & is\_nonunphrase len\_tokens\_norm constant\_freq \\
1 & & & consomat\_freq \\  & & is\_capitalised bag\_of\_shapes pos\_tag\_count \\ \hline  & len\_tokens & unigram\_prob & gr\_or\_lat len\_sylls sent\_length \\ \hline  & len\_tokens & char\_ngram\_feats iob\_tags \\
3 & num\_complex\_punct sent\_length & lemma\_feats NER\_tag\_counts \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ablation analysis for the cross-lingual features